Library companion site tools

To grow the Haskell user base one area to improve is documentation. My impression is that while Haddock is great for reference documentation it often leads to scattered and hard to navigate guides. For example, one important Persistent guide page is buried in Database.Persist.Quasi.

I would rather like a “companion site” (maybe there’s a better term?) like many Rust projects have that serves as an entry point for the library, shows common developer tasks and links to the reference docs. One good example is the restructured Welcome to the Cabal User Guide — Cabal 3.13.0.0 User's Guide.

A few reasonable requirements could be

  • Easy to setup (ideally not more than a single static executable)
  • Doc source file can live next to library source code.
  • Markdown syntax (simple, and most widespread)
  • Haskell examples can be embedded and get type checked
  • Easy cross-references to Haddock.
  • Nice to have: Image and Math embeddings, Versioning

Can you recommend such tool that could work across the whole Haskell ecosystem?

I can imagine GitHub - rust-lang/mdBook: Create book from markdown files. Like Gitbook but implemented in Rust and with hosting at *.github.io would work for most libraries. But maybe Haddock could be extended to to read in Markdown and host such sites at hackage.org; then all docs could look the same and be easy to find. Or maybe Flora.pm…

7 Likes

This reminds me of @Kleidukos’s talk at the 2024 Haskell ecosystem workshop on documentation. There is a blog post which briefly covers the ideas discussed in the talk:

I don’t know if there are recordings of the talk itself.

5 Likes

Until Haddock gains such capabilities (which is going to be a long work), I have done this dirty little hack of shipping an mdbook as part of my Haddocks distribution on Hackage: See text-display/text-display.cabal at main · haskell-text/text-display · GitHub and Introduction - text-display

16 Likes

I can’t remember in which package I saw it first, but I like the idea of having a Tutorial module in packages whose sole purpose is to have a user guide right on Hackage.

Here’s an example for a package of mine, javelin, which has a documentation-only module, Data.Series.Tutorial.

The advantage of doing it this way is two-fold:

  1. It is trivial to use doctest on such a module, just like you might already be doing on your codebase, which ensures that the tutorial is always correct;
  2. You can easily link to reference documentation right in the tutorial, since it’s based on Haddock.

I agree that having something more than reference documentation is needed. I stumbled upon this documentation structure framework a while back, and I now recognize that user guides / tutorials are extremely useful.
My fear is splitting the reference documentation (e.g. on Hackage) from the user guide (e.g. on some custom domain). In practice, I usually consult both types of documentation simultaneously.

18 Likes

The lens package has a Tutorial module: Control.Lens.Tutorial.

I really like this approach since you can load it on the REPL and try stuff out like examples, and from a maintainers perspective the code in such module is always compiling.

5 Likes

Thanks for those great suggestions.

@jaror @LaurentRDC @emlautarom1 This diataxis framework is exactly what i have in mind. I really like this documentation module(s) idea, because it ticks all the boxes except Markdown syntax for me and is ready to be used and promoted today. Do you have an opinion how to structure multiple shorter guides and tutorials such that the sections become more prominent in Haddocks content overview? I was thinking about a top-level Documentation module, e.g.
applied to Javelin

Documentation
    Documentation.Guides
         Documentation.Guides.How-To-Create-Series
         Documentation.Guides.How-To-Access-Series-Data
         Documentation.Guides.How-To-Aggregate-Series-Data
    Documentation.Tutorials
...real code modules

or shorter

Doc
    Doc.Guide
         Doc.Guide.How-To-Create-Series
         Doc.Guide.How-To-Access-Series-Data
         Doc.Guide.How-To-Aggregate-Series-Data
    Doc.Tutorial
        ...
...usual reference doc for real code modules

@Kleidukos What made you reach for for mdbook? The doc module approach supports versioning, which i imagine can be fiddly to setup. Do you think the Haddock maintainers would be open to add Markdown support at some point? Maybe at least a documentation top-level module could become a first-class thing (like e.g. being shown as the first content element).

2 Likes

FWIW Pandoc is happy to convert Markdown to Haddock format.

5 Likes

I’m not sure this is a technical problem.

It’s just that we lack a convention. There are packages that have great READMEs. Other packages have a top level module with great haddocks.

I think it’s mostly an attitude/priority issue.

The only way to improve in such a case is to lead by example.

3 Likes

Ah, i wasn’t aware. That opens up a nice migration path. First, manually call Pandoc before Haddock. Later, Haddock may support Markdown out of the box with Pandoc as a library.

A convention is what i’m rooting for here. For the many overloaded single-maintainer libraries we have in the Haskell ecosystem at the moment, the friction must be minimal to be adopted.

What i like about the Rust ecosystem is that there’s this really nice convention of having a small usage example of a library on the entrypoint page (Github Readme, crates.io) and quite often a companion page created with the easy-to-setup mdbook tool.

As i have learned from this thread the latter could be achieved with a low-hanging fruit of existing tools in Haskell, e.g. Haddock, doctest and doc modules. It may be just missing a few structuring and setup recommendation, which i am searching for here.

Regarding examples, i plan to try a simple setup out with Persistent (the maintainer is open to restructure the guide sections). I already pushed e.g. Cabal documentation to adopt this Diataxis structure that @LaurentRDC mentioned, which seems to work out great; except using Asciidoc and Readthedocs hosting in my opinion come with too much friction to be adopted ecosystem-wide.

5 Likes

Do you think the Haddock maintainers would be open to add Markdown support at some point?

I personally will work towards having something better than markdown, which has less footguns and enables real error messages instead of the “produce HTML at all cost” mentality of Markdown. Maybe something inspired by Djot, or more probably something inspired by Lean4’s Verso.

Maybe at least a documentation top-level module could become a first-class thing (like e.g. being shown as the first content element).

References are not guides. We shouldn’t try to fit more things inside modules. Modules are not the endgame of Haskell documentation, and things like mdbook should be doable from within Haddock.

Regarding the talk I gave at the Haskell Ecosystem Workshop of 2024, here are the slides: Haskell and its documentation. I encourage you to read them.

3 Likes

By and large the biggest issue with Haddock markup is that it’s not Markdown :wink: I’m not quite sure that introducing yet another markup language can solve more issues than create.

9 Likes

I agree, can’t beat markdown plus a smooth optional path to djot.

PS well yes Verso is also awesome. Carry on! Just ideally markdown is always included as an option, because of its massive network-effect benefits.

1 Like

I don’t feel like we should add more modules just for documentation.

The first thing that came to my mind is the duckling package Facebook made a while ago that in itself is pretty useful, but has an entire snap dependency because one of the example executables uses it.

IIRC this did end up in my dependency stack, even though I never used it. (not sure if it’s better nowadays)

But the idea of adding more time to compilation, when documentation has nothing to do with compilation doesn’t sit well with me.
I do like the mdbook setup that @Kleidukos showed in text-display.

Haddock is already something like that, but there is also the possibility of literate Haskell. One of my projects consists entirely of .lhs files with one main LaTeX file that

  1. imports all the modules that are exported in the Cabal file
  2. can import even more hidden modules to give more examples

Instead of LaTeX one could possibly also use Pandoc to produce other kinds of documents, but LaTeX is effortless. Major downside: The source becomes much less readable and many editors have problems with syntax highlighting two languages in one file.

Could such modules be part of a separate compilation goal that’s off by default?

I agree it wouldn’t be very nice to get an extra huge dependency just because the examples show how easy it is to integrate with some third party library or whatever (which can be super useful in a howto guide! But you don’t want to actually compile that example on every CI build of some app where this well-documented library is perhaps deep in your dependency tree).

So if one is to have a convention of using FooLib.Tutorial modules, I guess there also has to be a strong convention of making their compilation default-off.

If not, the extra-doc-files trick looked neat. It doesn’t give the compile-check on tutorial examples, but I’m not sure that’s worth the cost if everyone has to do it on build.

2 Likes

It doesn’t by default but actually you can have that too. A previous version of my documentation setup would use @hackage/literatex to produce markdown from literate haskell Add a mdbook setup (#43) · haskell-text/text-display@a7460b3 · GitHub

2 Likes

I looked into adding Markdown support to Haddock once: Support Markdown syntax via `commonmark-hs` · Issue #794 · haskell/haddock · GitHub It looked to me like Haddock’s native format is richer and you’ll have to make arbitrary choices on how to map Markdown to Haddock’s IR, which felt unsatisfying. That said, I think Markdown should be supported, because it’s a reasonable expectation from anyone coming to the language today.

2 Likes

Thanks for the additional comments. It’s great to see a shared understanding that improving the documentation tooling is worthwhile.

Regarding documentation modules i learned that they are not the right approach, because they open the possibility to pull in unintended additional (example-) dependencies. So something like mdbook as part of Haddock seems to be the way to go.

What i am missing is consensus on the documentation format. If we were to decide to choose Markdown, initiatives like @artem may be revived and the ecosystem could start producing guides in this format. @Kleidukos do you do you have a roadmap when such a decision can be made? Do you work as a Haskell Foundation representative on this topic?

1 Like

I think this sounds similar to Tintin. It’s unmaintained but I think the basic idea is there

Hasktorch’s tutorial page is built using tintin

2 Likes