Dependency version bounds are a lie

[maralorn]

The problem with people testing against nixpkgs is, ironically, that it doesn’t help nixpkgs maintenance at all. We are kind of victims of our own success, when that happens. Projects which test (only) against current nixpkgs very often break in nixpkgs when we update nixpkgs, because they never had an incentive to fix any build errors with newer packages. They will only start testing against the new versions of their dependency, when we release a new version of nixpkgs, but at that point we already needed the fix.

That’s more about upper bounds, though, where my fear with cabal-plan-bounds is more about lower bounds. So it might not be that bad. Actually testing against the newest versions available on hackage, stackage-nightly and stackage-lts will probably bring us a long way.

[maralorn]

It probably wouldn’t be hard to create a nix derivation which generates a cabal config file like that.

Having CI which tries to figure out the loosest possible lower bounds and then autogenerating the bounds seems fine fore me. But then the bounds should really be convex.

[NicolasT]

IMHO, this is a case where downstream is pushing a problem they might have upstream.

The intent of something like cabal-plan-bounds is to ensure a package author who wishes to do so, can publish a package with (lower & upper) bounds set to something that’s known to work. Validated by CI or otherwise (manually), which in any case has a cost (CI may be somewhat-free’ish nowadays, but still), hence, the package author might decide to bump lower bounds because keeping them low (and tested) is costly when new upper bounds need to be supported.

Even if there’s no technical reason to bump said lower bounds, i.e., the package may still work with older versions of the dependency.

If downstream (a distribution, a company,…) wants to ship/use the package with older versions of dependencies, its on to them to validate things work properly, and apply bounds relaxations as appropriate (e.g., by applying a vendor patch during the package build process, or maybe using allow-older definitions in cabal.project).

[nomeata]

Nice!

You can pass multiple cabal files to the tool in one invocation, then the plans are parsed only once.

@NicolasT

cabal-plan-bounds formats build-depends different than how cabal-fmt does it,

Can you just run cabal-fmt after it?

[NicolasT]

@nomeata

Can you just run cabal-fmt after it?

Of course.

[nomeata]

@maralorn

Actually testing against the newest versions available on hackage, stackage-nightly and stackage-lts will probably bring us a long way.

That’s what I meant! Add nixpkgs as an additional settings you use, to keep that lower bound alive and tested. It’s not perfect yet (maybe someone creates a tool that calculates a set of “full range covering build plans”…) but might work most of the time.

[NicolasT]

@nomeata

Integration of that idea with haskell-ci would be amazing! I use haskell-ci whenever I can (although I found it limiting whenever I need to do something slightly non-standard in my build scripts – the old problem of convenient framework vs. composable building blocks I think.)

Thinking of it: haskell-ci could create build plans automatically (derive constraint-sets, I guess), based on bound definitions in build-depends, to ensure all bounds are validated (and, as a safety-check, have something like cabal-plan-bounds at the end).

[maralorn]

I see your point. But downstream can also be another word for “user” here.

A new user would probably start shouting “cabal hell!!!” when they try to build a simple webserver and the sql and the http library don’t have a valid common build plan because neither of the upstreams tested them together. It’s not likely to happen, but certainly possible if maintainers test only always the newest configuration at time of release and only accept exactly those bounds.

One other thing: The biggest reason for upstream to maintain bounds instead of downstream is that the upstream has more information. Downstream can (and nixpkgs does) run the test suite as well. But only upstream knows about breaking changes which don‘t cause build and test failures and for those bounds are really important and downstream packages need to be careful with them. If upstream only creates the bounds based on builds and tests succeeding this becomes less relevant.

I think there is reasonable agreement here. Using tests to verify and create bounds is really cool. I just want to urge everyone to not forget that giving reasonably wide bounds is important. So if you make a CI setup which creates bounds you should keep that in mind. Especially I want to argue for checking upper and lower bounds and then assume that values in between are fine, I think that’s a good enough approximation of testing everything.

If we take the “only release tested bounds thing” seriously, we couldn’t reasonably create the bounds from more than one build plan at once because of the possible combinatorial explosion of valid build plans for bounds created from just two build plans. Accepting that, making bounds generally convex seems reasonable to me.

[parsonsmatt]

I think I generally agree that it’s bad for a version bound to be wrong - in either direction. Either allowing a build plan that won’t work, or disallowing a build plan that would work. Both are really bad!

I deal with redundant upper bounds a lot. I occasionally deal with overly lax lower bounds. But if there’s a tool that’s automatically pushing version bounds on packages up, and up, and up, then that’s gonna make “overly strict lower bounds” go from being “a problem no one has right now” to “a problem everyone has to deal with occasionally”

[nomeata]

I fully agree that bounds are a service to the downstream user, including distributios (else I’d just use a freeze file), and that wider bounds are better here. And I’d say a tools like cabal-plan-bounds doesn’t change that - it just mean I have to do different things to achieve these bounds (e.g. think about the build plans I want to test, use --prefer-oldest to keep testing the lower bound, or simply pin the exact versions you want to keep tested in the cabal config).

Yes, that’s more work than just leaving stale lower bounds in place. But if the sanity of distro maintainers and other users so far relied on accidental, hope-for-the best stale lower bounds, then I think there can be room for improvement

[atravers]

@jaror

[…] the problem of an exponential amount of CI jobs remains if you want to be completely sure every combination of dependencies works.

@Underlap

I’ve worked on non-Haskell projects which attempted to do lots of automation in a CI pipeline. This had the effect that any flakiness in complex tests […] was amplified such that a large proportion of the team’s efforts were swallowed up improving tests and making the CI pipeline more robust.

@turion

If I remember correctly, my packages got thrown out of stackage every now and then because of the restrictive bounds.

@ChShersh

I’ve spent unbelievably crazy amount of time in dozens of my open-source Haskell projects on supporting both cabal and stack, maintaining proper lower and upper bounds, compiling with multiple GHC’s (and maybe even multiple Stack snapshots), and fixing various bounds-related errors.

[…]

Asking people to perform extra steps only works for people who care deeply about some issue so they’re eager to invest more time or for people willing to pay money for doing this kind of work.

…at least for Haskell, continuous integration looks more like continuous irritation (or perhaps insanity). To someone like me who hasn’t built anything big in Haskell for a few years now, it just seems more and more excruciating.

[Vlix]

I just use stack LTS-es… which were, I think, in part created because of things like this.

My libraries I build using stackage LTS-es, and I take the versions from the lowest LTS I support for lower bounds, and the highest for the higher bounds. And then I guess I sometimes have to adjust when new packages come out for the higher bounds, but other than that I haven’t really had any issues and none were raised in the GitHub issue section either

But I can see how YMMV and I sympathise with my fellow Haskellers who prefer cabal.

(Getting your package thrown out of stackage is my way of knowing when new updates to libraries have been published, and I, at the moment, don’t mind updating my libraries when that happens. But then again, I have like 2 that I maintain, so…)

[atravers]

@tomjaguarpaw

This reinforces my longstanding belief that bounds don’t belong to packages but to package collections

The proposal is to define a Haskell Platform – a collection of high quality, versioned packages from Hackage.
[…]
There would be regular release cycles. The initial suggestion is major releases every six months with minor bug-fix updates in between.

^{Haskell: Batteries Included.}

…then most Haskell users switched to ghcup, stack, and others.

[nomeata]

We currently encode positive information in the cabal file - these dependencies are (very likely) gonna work. Hence lower and upper bounds.

What would happen if we’d encode only negative information, i.e. only exclude versions that are semantically not compatible or are known to not compile? Presumably that means few upper bounds, and maybe fewer lower bounds as well? Who’d benefit, and for whom would it be worse?

It seems it would be much less busywork for maintainer, and from what @maralorn says it sounds it might also be better for packe set maintainers. It likely doesn’t matter for users of package sets. Would it be worse for users using hackage directly?

Maybe in that world semantic breaking changes that don’t break the build would be even worse, and maybe avoided more?

[sclv]

@nomeata

It likely doesn’t matter for users of package sets. Would it be worse for users using hackage directly?

I think it would be basically an unmitigated disaster. In particular, “known not to compile” is a huge question – known not to compile by who? By maintainers who update those bounds? By anybody who attempts to compile? If the default assumption is that things will work, then the solver will find lots of plans that don’t work, and things will break constantly.

If, on the other hand, we had tooling that automatically tested nearly every config and then introduced bounds for everything based on whether it compiled or not, then that would not be so bad, but also, at that point the “gap” between “positive” and “negative” information would narrow almost entirely (leaving out semantic changes with no type reflection) because we would in fact have “perfect” information.

[atravers]

@jaror

But the problem of an exponential amount of CI jobs remains if you want to be completely sure every combination of dependencies works. Because we can never be sure that an old version of one dependency works together with a new version of another dependency.

…Haskell Central is going to need a lot more hardware!

[NicolasT]

@atravers

…at least for Haskell, continuous integration looks more like continuous irritation (or perhaps insanity). To someone like me who hasn’t built anything big in Haskell for a few years now, it just seems more and more excruciating.

Even though that may be a conclusion of this thread, I don’t believe the situation is that bad (actually, I think it’s much better now than say 8 years ago!). When comparing with other languages/communities, keep in mind

• Haskell folks tend to be much less YOLO about this kind of things
• In several popular languages, it’s somewhat easier to maintain backward compatibility, e.g., in JavaScript when the params (dict) argument pattern is used, one can easily add new fields in a backwards-compatible way, as long as there are sane (backwards-compatible) defaults for all ‘new’ fields. In Haskell, adding a field to a record, or a new constructor to a datatype, is a breaking change, and it’s that kind of changes which make dependency tracking a bit harder.

[atravers]

@NicolasT

In Haskell, adding a field to a record, or a new constructor to a datatype, is a breaking change […]

Yes, that’s the expression problem - it also appears in many OO languages. I’m now curious to know if JS’s params features was specifically introduced to deal with that problem, or another…along with whether or not the feature could work with static typing as implemented by Haskell.

[NicolasT]

@nomeata

Very nice! I think you should be able to easily copy the second half of my suggestes workflow (collect build results and validate/update bounds) and combine it with yours.

Here’s a diff on my project to get all of this integrated: Comparing 09ba50598d2f7117..e7f94692d840550 · NicolasT/landlock-hs · GitHub

It relies on a (fairly hefty and conflict-prone) patch on top of what haskell-ci generates. I added a constraint-set pinning all dependencies to the lowest (supposedly) supported version (except for base), on the oldest supported GHC version, to ensure full coverage.

One tricky thing I ran into (and hack around for now) is multiple packages in the same repo. Say, there’s packages A and B, both pointed to by cabal.project. When B depends on A ^>= 0.1.0.0, and A (in the repo) is at 0.1.2.3, then adding a constraint on A ==0.1.0.0 doesn’t work, the solver will bail out because all it wants to install is 0.1.2.3.

[nomeata]

Very nice, I hope it proves useful.

@NicolasT

then adding a constraint on A ==0.1.0.0 doesn’t work

I don’t fully follow here. It’s not my tool adding that constraint, is it (it only adds ^>= constraints)?