The ^>= operator in .cabal files

sclv · July 16, 2023, 6:15am

If there are painful experience reports of a time before(?) the upper bound requirement, I would love to hear them. It might go a long way towards making it more palatable.

The article motivating the pvp was published in 2005, right as cabal and hackage were really coming into being. The first pvp page on the haskell wiki reached a recognizable form in 2007: Package versioning policy - HaskellWiki

Here are all 15 packages that were on hackage in 2006:
https://web.archive.org/web/20060303165044/http://hackage.haskell.org/ModHackage/Hackage.hs?action=view

What I will note is that we do have experience with upper-bounds and not-upper-bounds that is extensive nonetheless, because there have been plenty of packages on hackage that do not follow the pvp with regards to upper bounds, and we have experienced the difficulties in keeping things working with them.

As jackdk describes, if a package has many releases, and no upper bounds, then a new release of a dependency that causes breaks necessitates adding upper bounds revisions to all prior versions as well, lest the solver fall back to them. In the converse case, it requires relaxing upper bounds on at most the few most recent releases.

Most end-users do not see the pain of this, and only the maintainers who do not put upper bounds do, as well as the trustees who have to go fix the ecosystem when problems occur. We never did create a database of all the reported issues to run quantitative analysis on, but the old reddit threads arguing about this stuff all had a fair accounting of stories as I recall (and also covered most all of the discussion here, repeatedly).

A sampling:

Skimming btw, I see a number of proposals to improve things have been already implemented (including something like the carat operator, as well as the --allow-newer flag [which can be very granular in project files]).

Ideas for how to collect more quantitative data on the effects of different bounds policies are very welcome – and indeed one of the threads has some pretty neat analysis derived from the 01-index.tgz hackage tarball and analyzing metadata revisions.

hasufell · July 16, 2023, 6:33am

Here is an alternative angle that would allow the haskell community or hackage to create their own policies and take away the burden from PVP maintainers to decide on such far reaching consequences: [RFC] Make PVP ecosystem/language agnostic · Issue #54 · haskell/pvp · GitHub

tomjaguarpaw · July 16, 2023, 7:25am

@jackdk’s comment is an example of painful experience due to missing upper bounds, isn’t it? Or did I misunderstand your question?

chreekat · July 16, 2023, 7:58am

You’re right, it is.

So am I correct in understanding that the trade-off is, at the extremes,

An author releases a new major version of a package for reasons other than a breaking change. No downstream user can use it until any/all intermediate packages are revised to mark compatibility.

Versus

An author releases a breaking change in a new major version. Any current or old version of an intermediate package will satisfy the build plan, but will fail to produce correct results.

(Of course, most situations are not at either extreme, i.e. releases are made for a mix of reasons and breakage only affects some percentage of consumers. I think a full analysis would need to account for the factors to be accurate, but I just want to be sure I understand the fundamental trade-off first.)

Bodigrim · July 16, 2023, 9:16am

@chreekat a cost-benefit analysis of upper bounds from the viewpoint of Hackage Trustees is available at https://github.com/haskell-infra/hackage-trustees/blob/master/cookbook.md#best-practice-for-managing-meta-data.

andreabedini · July 16, 2023, 11:56am

(cross-posting from the GitHub issue linked above)

IMHO the essence of upper bounds is this: being able to compile an old unmaintained package should be a no-op.

Of course using a newer GHC might not be possible, but with upper bounds one is guaranteed to have at least one build plan that works, for some version of GHC.

As a test, someone tell me what dependency versions I need to make wai-middleware-authwork. It’s not a trick, I did spend a couple of days trying to figure it out and gave up

andreabedini · July 16, 2023, 12:01pm

Another angle: package authors report that bumping upper bounds is time-consuming but this effort saves a ton of time to package consumers! Without upper bounds they will have no idea how to make your package work.

Also, let’s be honest:

CIs and automations will eventually disappear (burnout is real)
Authors and maintainers will disappear too, moving to other jobs, or other lives.
If you don’t put upper bounds in a package, your package will bit rot quickly, and I will have lost something valuable

hasufell · July 16, 2023, 12:07pm

Maybe…

What you actually would want to know is: for every dependency (of the old package) that doesn’t have an upper bound, what is the latest dependency version available at the time the old package was added?

This is similar to a freeze file, but not quite the same. You basically fall back to the index of the time the package was added, but only for its dependencies lacking upper bounds.

I believe this isn’t too hard to achieve maybe, although I am not too knowledgeable about the hackage index format.

andreabedini · July 16, 2023, 1:02pm

I am not sure whether this works or not but it is an idea worth exploring. Should we care if the package does not work with those versions?

brandonchinn178 · July 16, 2023, 7:47pm

I’m not sure upper bounds are sufficient though, since the package might not work with older versions of GHC. IMO if you care about this use case, use Stack snapshots

hasufell · July 17, 2023, 3:15am

Yes, this is another feature that’s necessary to implement: asking cabal “what is the latest version of GHC you can find a build plan for?”

That can’t be too hard: instead of pinning base (and boot library) versions, pass it to the solver without bounds, then extract the base version it comes up with, then map that to a GHC version.

Also see 'ghcup satisfy' command · Issue #109 · haskell/ghcup-hs · GitHub

andreabedini · July 17, 2023, 7:25am

I know you are repying to Julian, from my pov using an older GHC is not a problem.

andreabedini · July 17, 2023, 7:40am

I think a POC should not be too hard but I believe the cabal-install codebase has to go through some changes before we can use it this way (the ghcup ticket you mention links to haskell/cabal#6885 which I believe is relevant).

E.g. pre-installed boot packages always exist and their version is pinned (because they cannot be reinstalled anyway), so you cannot leave it “floating”.

I did try --allow-boot-library-installs but I am not sure what the error is telling me:

❯ cabal build --dry-run --allow-newer=aeson:base --allow-boot-library-installs -v3 -w ghc-9.4
...
Component graph for ghc-bignum-1.3:
Component graph for ghc-prim-0.7.0: component lib
component ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3
    include rts-1.0.2
unit ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3
    include rts-1.0.2
    GHC.CString=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.CString,GHC.Classes=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.Classes,GHC.Debug=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.Debug,GHC.IntWord64=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.IntWord64,GHC.Magic=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.Magic,GHC.Prim.Exception=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.Prim.Exception,GHC.Prim.Ext=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.Prim.Ext,GHC.Prim.Panic=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.Prim.Panic,GHC.PrimopWrappers=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.PrimopWrappers,GHC.Tuple=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.Tuple,GHC.Types=ghc-prim-0.7.0-85190006ed57da9be8e03ab6d533fb5dbf69877426d00246e104507bd30abbc3:GHC.Types
Component graph for base-4.15.1.0: component lib
CallStack (from HasCallStack):
  withMetadata, called at src/Distribution/Simple/Utils.hs:368:14 in Cabal-3.10.1.0-inplace:Distribution.Simple.Utils
Error:
    Dependency on unbuildable library from ghc-bignum
    In the stanza 'library'
    In the package 'base-4.15.1.0'

PS: This is ignoring any external depdenency, which would add another set of problems.

sclv · July 17, 2023, 8:47pm

Similarly – just ran into this trying to help someone.

Does anyone care to explain what constraints need to be added to make the following work?

cabal install hoogle-5.0.18.3 --constraint='crypton-x509 < 0.0'

Note that this runs into something mysterious with constructors appearing in warp's Transport type somewhere in the 3.3.x series.

jackdk · July 18, 2023, 2:58am

To be precise: fixing http-conduit's bounds for aeson-2.2.0.0 necessitated revisions to 25 releases, which I made in my capacity as Hackage Trustee. This is why I support the current recommendation of “have upper bounds and raise them in response to new releases”: it is my experience that omitting upper bounds not only amplifies the amount of work required to respond to breakage, it tends to push that work onto people other than the package maintainers.

sclv · July 18, 2023, 4:11am

note that in the above, that constraint which I already added (giving part of the solution) is necessary (and took people quite some time to figure out – maybe an hour on irc of back-and-forth) because warp-tls itself does not have an upper-bound on the version of tls it uses (cf warp-tls: HTTP over TLS support for Warp via the TLS package) and so the older version does not bound itself to the cryptonite-using-tls and allows mixing-and-matching with the forked-crypton-using-tls. Once this situation occurs, trustees can, and probably should, figure out what revisions need to be made to avoid this sort of problem.

However, figuring out those revisions, as these examples show, is highly nontrivial, and involves a ton of prior releases all of which have missing bounds. Figuring out a bounds-bump revision, on the other hand, is very easy and rapid work.

see also can't install warp-tls.3.3.6 in macOS · Issue #936 · yesodweb/wai · GitHub for more on this current situation.

hasufell · July 18, 2023, 4:38am

So you agree that hackage revision are a terrible hack and hackage should actually reject packages without upper bounds? This is one of the possible solutions, but status quo is broken.

glguy · July 18, 2023, 4:59am

Part of the challenge here is that it’s easy to forget that just because you released a new version with a fix or better bounds that all the old, unbounded dependencies on your old releases are still in play. Each release with no upper-bound is some lingering debt. Unfortunately it’s easy to have a mindset that only the most recent release “matters”.

hasufell · July 18, 2023, 5:06am

Yes, in fact both semver and PVP are terrible, because they encourage frequent API iteration and so you naturally end up with many actively used “branches”.

But that ship has sailed.

andreabedini · July 18, 2023, 6:16am

Sorry but I don’t understand where this negativity comes from. It’s not an “terrible hack”.

Even if Hackage rejected outright packages without upper bounds, the ability to adjust the bounds is what allow us to extend the life of a package.

A newer version of a dependency will eventually come up; and, as I have been told many times, there’s a good chance it will be still compatible after all. So what do we do?

Release a new minor version? (as I think most ecosystems do). Redistributing a single file is a much cheaper solution.

And who does that? The author or maintainer might be MIA or have just moved on. This is when a group of “repository curators” can step in.

(Needless to say: automated testing could make all this easier and less error prone. There have been cases of bad revision but they can be fixed quickly)

At some point some code change would be required to keep the package working with newer versions of its dependencies. At this point a new version is necessary, and if the author is unavailable that might be the end of the story (but there is a process for adopting a package).

I don’t want to state that this is perfect, but that, at least from some point of view, it makes sense.

E.g. I feel this supports perfectly the “I wrote some Haskell packages during my PhD” kind-of scenario: original author will disappear, content is valuable, we still want to use it even if very few people understand it.