This has been practice, but is a mistake.
See Internal convention is a mistake – Functional programming debugs you
As part of GHC's base libraries: Combining stability with innovation by Ericson2314 · Pull Request #51 · haskellfoundation/tech-proposals · GitHub CLC and GHC devs have agreed that it’s a bad strategy and GHC internals will be following PVP.
Likewise, some of the boot libraries will start following this pattern as well.
Can we at least try not to make the same mistake twice here:
GHC extensions are not regular Haskell features,
and “internal”/“low-level” libraries are not regular libraries.
…otherwise we are practically inviting everyone (again) to rely on their preferred set of such libraries as though they were regular ones (with a similar reaction to anyone contemplating major changes to them e.g. their deprecation).
Having 2N combinations of GHC extensions is bad enough - do we really then want 2M combinations of internal libraries pretending to be regular ones? Forget “explosion”, that would have to be a combinatorial “big bang!” I recall this thread:
…has anyone tried using this guardian system yet?
Sorry, I have no idea what you are talking about. What does this have to do with GHC extensions?
That is absolutely fine, because PVP is guaranteed, but major versions of internals might increase VERY rapidly and constantly break API. That’s what you get.
Like internal libraries, GHC extensions aren’t normally intended for novices e.g. not all 2N combinations of extensions work.
No - “what you get” is lots of unhappy users of those libraries, and therefore lots of unhappy maintainers left to deal with their complaints. If the solution was as simple as telling novices to "avoid using libraries labelled internals", then we could also just tell novices to "avoid using anything labelled unsafe or inline".
Or is Haskell now only intended for “power users”?
I have still no idea what you mean. I’m not the GHC extension police.
So you’re suggesting to hide internals, so that power users (e.g. those that write alternative preludes) have an extra hard time achieving anything?
I’m not sure that’s a sensible thing to do.
We’re already discussing options to emit e.g. warnings when you depend on an internal package. Documentation for now will be enough.
…because telling novices to "avoid using anything labelled unsafe or inline" is working so well now. Based on that experience, finding a way to direct novices away from using internal libraries/packages first (e.g. with warnings) seems the more sensible option here.
I wonder whether there’s a meaningful split between users of Internal modules for published library writers and application writers?
A study could probably be done on Hackage to determine use of Internal modules within published libraries.
If it’s substantial then the debate is the same. If it’s minimal, then perhaps we just need to make it really really trivial for application writers to use a patched version of a library or to use it with all modules exposed.
My POV is that we are all “novices”. I surely am, which is why I like to use Haskell to help me write correct code.
I believe it is easy, even for experts, to make wrong assumptions about the internal working of a library and cause bugs (I have seem this happening a few times).
IMHO every exposr api would be properly versioned. The perceived need to access “internal” functions should perhaps lead to adding those functionality to the library, rather than “crossing the API border”.
If you want to discourage people from using Internal modules, put deprecation pragmas on them. People hate generating warning messages.
More seriously, but not completely seriously, what do you think about functionality becoming exposed based on version constraint. Consider some examples like:
@since 1.2.3 – if you depend on a lower version than that, you won’t see it even when building against a higher version.Is there any precedent for this kind of thing in other (even niche) languages?
Interesting proposition. This would help to identify broken lower bounds.
How about an {-# INTERNAL #-} pragma instead for generating an appropriate warning?
(heh) …so something like {-# INTERNAL #-} has ben contemplated before:
You can just use a freeze file that serves as a company wide blacklist and has constraints like
constraints: ghc-internals < 0
, filepath-internals < 0
I don’t see the problem. This is not the job of hackage or GHC devs. It’s a company policy. Other companies might have different policies or blacklist other packages (like foundation and its entire suite).
Those constraints won’t work. They’ll blacklist packages not only from being immediate deps, but also transitive deps, and obviously internals will be transitive deps of the packages they are the internals of.
Yes, that was the idea, but I didn’t think it through properly. It would only work for an internal package that isn’t used by anything else (unlikely to exist), so you effectively also blacklist filepath and base.
I’m guessing the solver doesn’t know the origin of a dependency? Would there be a way to express “reject (transitive) dependency, unless coming from packages x, y and z?”.
Otherwise I guess you’re left with identifying packages that abuse internals and blacklisting those. That still seems like something feasible to do.
Or… if hackage has an API of querying reverse deps… you could query that for internals, identify all non-core packages depending on them and generate a blacklist.
I am keen on Hackage reporting on PVP compliance but I have no idea how to implement that reliably :-/
I’ve been thinking about this, and I think I have a defense for internal modules/internal sublibraries. What matters most for me personally, is how much work it is to upgrade the dependencies of my software to the latest version. With the separation between internal/external, there is a promise: the external API is more stable. Stick to that, and it will likely be less work to upgrade. Besides that, changes are rare, and if they are there, the PVP version will notify you.
At the core, maintainers don’t want to be forced to make inner workings a part of the stable API. At the same time, users want access to either or both the stable API and the unstable inner workings. In the latter case, accepting that upgrading is more work.
I think that tradeoff is a good one, and I personally don’t have a strong opinion on how that is implemented or even whether it breaks PVP.
At the end of the day, when I upgrade the dependencies of my projects, I barely even look at version changes. I just try to compile it, see where the errors are and see if it still runs nicely.
Are we all on the same page what we want? My understanding so far is:
So for PVP compliance internal modules should not be exposed. At the same time dealing with a too restrictive surface of a library (as you only expose the interface you are happy to PVP version), yet providing the internal modules currently serves as a form of escape hatch, should you need something that isn’t in the PVP versioned exposed API. This to me is basically something we should track with metadata: I expose this, even though I don’t want to bother with providing a stable API for this and this does not follow any PVP versioning, if you need to use it, feel free to but, proceed at your own risk.
We have this informal convention thing with .Internal modules right now.
Hence again, I think we should be able to have GHC support something like {-# INTERNAL #-} markers to attach this metadata to the bindings. This could be picked up by haddock, and we could have some logic to enforce that INTERNAL tagged bindings do not leak out of module unless explicitly permitted.
This would allow us to get the benefits of the PVP, while still allowing for the practical escape hatch. (And similar to the ghc base library discussion, one would expect that usage of internal bindings would hopefully spur some petitioning for making those internal bindings part of the public API).
This would also allow automated tools (to inform PVP bounds) to ignore INTERNAL marked symbols. The alternative is to just not expose INTERNAL symbols, and then just have people fork/source-repository-package dependnecie they need ITNERNAL symbols from (which imo is pretty bad and cumbersome UX), especially if you are in the exploratory phase. Passing some -permit-internal or -Wno-internal. Would allow to just sidestep this temporarily.
At the same time it would allow you to compile with -Werror -Winternal or (-prohibit-internal), to make sure you don’t have any of these cross-package internal leakage, that would likely result in painful breakage (not caught, contained by the PVP).
The other view I guess is to solve this at the package level, don’t have Internal or other modules. Stick them into a separate package, and version that with PVP as well. This would basically mean we enforce PVP across all packages, and the exposure of internal components thus forces the existence of -internal packages, as they have to be PVP compliant as well.
Thus I believe the contention(?) might be between:
I’m not a fan of {-# INTERNAL #-}. Handling this in GHC seems like the wrong place.
I think it makes more sense to handle this in cabal. Perhaps add a stanza for exposed-internal-modules? Using internals is “safe” if you’ve specified an exact version of your dependency, and unlike GHC, cabal (or another build tool that wishes to support this) will know if this is the case.