Installing a library with C dependencies on Windows

This is a short description of how I managed to install a cabal package with C library dependencies on Windows. I hope this is useful information for others and for myself in the future.

Just now I wanted to install the hackage-cli package under windows. Normally that should be as simple as opening powershell and running cabal install hackage-cli but doing that result in an error:

Resolving dependencies...
Error: cabal-3.10.2.1.exe: Could not resolve dependencies:
[__0] trying: hackage-cli-0.1.0.1 (user goal)
[__1] trying: http-io-streams-0.1.6.3 (dependency of hackage-cli)
[__2] trying: http-io-streams:+brotli
[__3] trying: brotli-streams-0.0.0.0 (dependency of http-io-streams +brotli)
[__4] next goal: brotli (dependency of brotli-streams)
[__4] rejecting: brotli-0.0.0.1, brotli-0.0.0.0 (conflict: pkg-config package
libbrotlidec-any, not found in the pkg-config database)
[__4] fail (backjumping, conflict set: brotli, brotli-streams)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: base, http-io-streams, brotli,
http-io-streams:brotli, brotli-streams, hackage-cli
Try running with --minimize-conflict-set to improve the error message.

This is an intimidating wall of text, but the important part is:

conflict: pkg-config package libbrotlidec-any, not found in the pkg-config database

If we dig a bit deeper by making cabal show verbose output with the cabal install hackage-cli -v option then we see an even bigger wall of text, but somewhere hidden inside it there is a more informative message:

Failed to query pkg-config, Cabal will continue without solving for pkg-config
constraints: Cannot find pkg-config program

Ah, so the pkg-config program is not even installed at all! But how do we install it? On Linux you can just install the package with the package manager of your distribution, but Windows has no standard package manager. Luckily the Haskell infrastructure on Windows uses a Linux campatibility environment called MSYS. To install packages in this environment, you can start mingw64.exe which is usually found in C:\ghcup\msys64. That gives you a shell in which you can run pacman commands to install packages.

First we need to install pkg-config. A simple search will list a bunch of options. We can use pacman -Ss pkg-config to search for packages, but it will list many results including these:

mingw64/mingw-w64-x86_64-pkg-config 0.29.2-6
    A system for managing library compile/link flags (mingw-w64)
ucrt64/mingw-w64-ucrt-x86_64-pkg-config 0.29.2-6
    A system for managing library compile/link flags (mingw-w64)
clang64/mingw-w64-clang-x86_64-pkg-config 0.29.2-6
    A system for managing library compile/link flags (mingw-w64)
msys/pkgconf 2.1.0-1
    pkg-config compatible utility which does not depend on glib

Which one should you choose? I believe the right answer is mingw-w64-x86_64-pkg-config which is what works for me. However, I donā€™t know exactly why.

So letā€™s install it:

$ pacman -S mingw64/mingw64-w64-x86_64-pkg-config

And we already know from the first error message that we also need the brotli library so letā€™s add that too:

$ pacman -S mingw64/mingw64-w64-x86_64-brotli

Now letā€™s try installing hackage-cli again. Unfortunately we see the same:

conflict: pkg-config package libbrotlidec-any, not found in the pkg-config database

And with more details again:

Failed to query pkg-config, Cabal will continue without solving for pkg-config
constraints: Cannot find pkg-config program

The reason is that cabal canā€™t find the pkg-config executable, even though it is installed. You can make it visible to cabal by adding it to your Path environment variable, but that is not recommended because it might interfere with other programs and toolchains. Instead you can use GHCUp to run a command with the right directories on the Path temporarily:

> ghcup run --mingw-path -- cabal install hackage-cli

When running this command it will start installing a bunch of packages, but in the end we see another error message:

Configuring HsOpenSSL-0.11.7.6...
Error: setup.exe: Missing dependencies on foreign libraries:
* Missing (or bad) header file: openssl/asn1.h
* Missing (or bad) C libraries: ssl, crypto
This problem can usually be solved by installing the system packages that
provide these libraries (you may need the "-dev" versions). If the libraries
are already installed but in a non-standard location then you can use the
flags --extra-include-dirs= and --extra-lib-dirs= to specify where they are.If
the library files do exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

Error: cabal-3.10.2.1.exe: Failed to build HsOpenSSL-0.11.7.6 (which is
required by exe:hackage-cli from hackage-cli-0.1.0.1). See the build log above
for details.

Now we know how to solve this issue. The only slightly difficult part is finding the name of the package. In this case there are quite a few clues pointing us to openssl, so letā€™s install that:

$ pacman -S mingw64/mingw64-w64-x86_64-openssl

And now finally running cabal install hackage-cli does work and installs the hackage-cli command.

But when we try it in PowerShell we see nothing happening:

> hackage-cli --help
>

The reason this doesnā€™t work is that it now cannot find the dynamic linker dependencies, inspecting with ldd (which you can install in the mingw64 environment, it is in the mingw-w64-x86_64-python-mingw-ldd package) shows us these dependencies:

        libbrotlicommon.dll => /mingw64/bin/libbrotlicommon.dll (0x7ffaf2c00000)
        libbrotlidec.dll => /mingw64/bin/libbrotlidec.dll (0x7ffaf2d80000)
        libbrotlienc.dll => /mingw64/bin/libbrotlienc.dll (0x7ffa6b330000)
        libcrypto-3-x64.dll => /mingw64/bin/libcrypto-3-x64.dll (0x7ffa6b3f0000)
        libssl-3-x64.dll => /mingw64/bin/libssl-3-x64.dll (0x7ffa6b240000)

You can still run the executables built in this way under ghcup --mingw-path:

> ghcup --mingw-path -- hackage-cli --help
hackage-cli - CLI tool for Hackage

Usage: hackage-cli.exe [--version] [--verbose] [--hostname HOSTNAME] COMMAND

Available options:
  -h,--help                Show this help text
  --version                Output version information and exit.
  --verbose                Enable verbose output.
  --hostname HOSTNAME      Hackage hostname (default: "hackage.haskell.org")

Available commands:
  pull-cabal               Download .cabal files for a package.
  push-cabal               Upload revised .cabal files.
  sync-cabal               Update/sync local .cabal file with latest revision on
                           Hackage.
  push-candidate           Upload package candidate(s).
  list-versions            List versions for a package.
  check-revision           Validate revision.
  index-sha256sum          Generate sha256sum-format file.
  add-bound                Add bound to the library section of a package, unless
                           the bound is redundant. The .cabal file is edited in
                           place.

Each command has a sub-`--help` text. Hackage credentials are expected to be
stored in an `${HOME}/.netrc`-entry (or `.netrc.gpg`) for the respective Hackage
hostname. E.g. "machine hackage.haskell.org login MyUserName password TrustNo".
All interactions with Hackage occur TLS-encrypted via the HTTPS protocol.

So Iā€™m not sure how to fix this without adding the mingw directory to your Path permanently.

6 Likes

This is not recommended, since PATH on windows affects the linker. This can cause issues for other programs/toolchains.

Instead, this directory can be added to extra-prog-path in cabal config. This is what ghcup already does.

You can also do ghcup run --mingw-path <your-binary>.


Did you check the linking of your binary? Are mingw paths hardcoded? Can you execute your binary without changed PATH in powershell? Can you ship the binary?

5 Likes

Thatā€™s good to know. Iā€™ll do some more tests and update my post.

I saw that this had happened when I ran the verbose cabal command, but it still didnā€™t recognize the pkg-config program when I had installed it. Or maybe I could have gotten that wrong because I took a brief detour to try to use the clang64 environment, because I thought that GHC had moved to that. I will check this again.

These are good questions, which Iā€™d hoped people more qualified than me would have answered already. I donā€™t even know that well how to check the linking, I guess I could use lld but maybe that only works on Linux?

Iā€™ve tried your suggestions hereā€™s what worked and what didnā€™t:

  • If I remove mingw from the path then cabal can no longer find pkg-config even though the right directory has been added to the extra-prog-path.
  • Building it with ghcup run --mingw-path -- cabal install hackage-cli does work, but produces an executable that only works when you run it under ghcup run --mingw-path -- ... too.

Checking the linking yields these (and a bunch more windows specific dlls with an absolute path):

        libbrotlicommon.dll => /mingw64/bin/libbrotlicommon.dll (0x7ffaf2c00000)
        libbrotlidec.dll => /mingw64/bin/libbrotlidec.dll (0x7ffaf2d80000)
        libbrotlienc.dll => /mingw64/bin/libbrotlienc.dll (0x7ffa6b330000)
        libcrypto-3-x64.dll => /mingw64/bin/libcrypto-3-x64.dll (0x7ffa6b3f0000)
        libssl-3-x64.dll => /mingw64/bin/libssl-3-x64.dll (0x7ffa6b240000)

I donā€™t know how to fix this.

1 Like

It seems this might be related to `cabal` doesn't inherit the path from `cabal/config` in PowerShell Ā· Issue #9519 Ā· haskell/cabal Ā· GitHub

2 Likes

The issue appears to be that the dlls are added to bin, and hence they can be found while building the executable but not running it. In my experience, the typical (unfortunate) way that relocatable executables are built on windows is that the dlls are just copied into the directory of the executable itself, no?

I donā€™t think this is a problem with building or installing a library at all. It is simply that executables that require dlls will also require they be in the path, and you can choose to alter the path to allow this or not. And if you donā€™t, then you need to copy them to where they will be found regardless (the same directory).

Unless Iā€™m missing something, thereā€™s nothing particularly cabal or haskell specific at all?

Iā€™m not a Windows expert, but if thatā€™s the case then It would be nice if cabal could put the required dlls next to the executable.

3 Likes

I ran into an offshoot of these problems while trying to build Agda on a new laptop today. Cabal couldnā€™t find the zlib.cp config file, despite it being able to use pkg-config, and this caused the zlib package build to fail. What I eventually realized was that there was a discrepancy between what cabal wanted and what my install of msys2 wanted.

When you install msys2 on its own, as opposed to through ghcup, it asks you which environment you want to set as your default. It puts the default in your system PATH, and as of today it recommends ucrt64 over mingw64 (and explains why on that page). But regardless of which you choose as default, it still installs each of the environments as their own directories. The issue I had was that despite selecting ucrt64 as my default, using that environmentā€™s bash to install ghcup, and running cabal in that environment, the cabal config file defaulted to using C:\msys64\mingw64\lib instead of C:\msys64\ucrt64\lib.

I didnā€™t realize that there was this conflict at first, so I tried a bunch of things like adding C:\msys64\mingw64\lib\pkgconfig to the extra-prog-path, and even to my system PATH, but it still wouldnā€™t recognize the zlib.pc file in that directory. Once I realized the discrepancy between cabal using the ucrt64 pkg-config but looking in the mingw64 directories, I replaced each instance of C:\msys64\mingw64\ in the cabal config with C:\msys64\ucrt64\ and zlib successfully built.

Iā€™m sure that most of this comes down to msys2 being sensitive to which files are connected to which environments, since there actually was a zlib.cp in the mingw64 directories, but there is one aspect of this which I think is Haskell specific and goes towards answering:

The reason why, if Iā€™m right about any of this, is because cabal defaults to ~\msys2\mingw64 regardless of anything else going on. If you had changed the cabal config file to point to any of the other environments, mingw-w64-x86_64-pkg-config would not have been the one that worked. I confirmed this by trying all combinations of which environment was in my system PATH and which was in the cabal config.

Initially this made me want to look for a way for cabal (or ghcup?) to choose whichever environment someone who had their own msys2 install set as default. I thought that the environment variable in msys2.ini which the page I linked mentions might be something that could be used, but all that variable does is change which environment is used by the generic msys2 bash shell. Changing that variable never impacted the success of finding zlib.cp, the only thing that did was matching/mismatching the system PATH and the cabal config.

A lot of this makes complete sense in retrospect, and I guess Iā€™m not really suggesting that anything be changed, and Iā€™m definitely not trying to criticize anyone working on these tools for mingw64 being chosen as the default (itā€™s a really subtle issue, if I had chosen it instead of ucrt68 Agda would have successfully built and I wouldnā€™t have found this thread). I just thought that given how often problems related to this have come up this year, it might be worth writing all this down in case anyone else runs into it. Although, again, in retrospect I think my error had an obvious cause and Iā€™m a little embarrassed about how long it took me to realize the mismatch.

1 Like

I donā€™t think any of those tools should try to guess which environment you want. Is this good UX? Most certainly not. We need a global entry point where the user can say ā€œscrew it, set up msys2 env for all those tools to XY and donā€™t bother meā€. That would be the ghcup boostrap script, I guess: ghcup-hs/scripts/bootstrap at master Ā· haskell/ghcup-hs Ā· GitHub
And it needs an overhaul (e.g. an ā€œadvanced installationā€ mode). Itā€™s hard to strike a good balance between ā€œdonā€™t ask too many weird questionsā€ and ā€œallow extensive up-front configurationā€.

3 Likes

For Stack users, or for comparison, the corresponding Stack experience might be as follows:

Stack has stack install but I am not sure if the hackage-cli package provides its own Stack project-level configuration file or will build with my default ā€˜globalā€™ configured snapshot. So, I command:

stack unpack hackage-cli
cd hackage-cli-*
dir

It turns out that the package does provide stack-9.6.yaml, so I command:

stack --stack-yaml stack-9.6.yaml install

That fails because (the output tells me) Haskell package brotli needs program pkg-config version >= 0.9.0. So, I look on MSYS2 (its package search), and find mingw-w64-x86_64-pkg-config. I am interested in the mingw-w64 version because I know from developing on Windows that is the MSYS2 environment that Stack uses by default (as @hasufell has noted, that can be configured differently, from Stack 3.1.1). I command:

stack exec -- pacman -S mingw-w64-x86_64-pkg-config
stack --stack-yaml stack-9.6.yaml install

That fails because Haskell package brotli needs pkg-config package libbrotlienc. That looks like it could be C library-related, so I look on MSYS2 and find that mingw-w64-x86_64-brotli supplies /mingw64/lib/pkgconfig/libbrotlienc.pc. I command

stack exec -- pacman -S mingw-w64-x86_64-brotli
stack --stack-yaml stack-9.6.yaml install

That fails because Haskell package HSOpenSSL is missing openssl/ans1.h and C libraries ssl and crypto. I look on MSYS2 and find mingw-w64-x86_64-openssl, which looks like it might supply those missing things. I command:

stack exec -- pacman -S mingw-w64-x86_64-openssl
stack --stack-yaml stack-9.6.yaml install

Bingo! That works. Stack has put hackage-cli.exe on the PATH.

EDIT: But I spoke too soon. hackage-cli --help brings up dialog boxes warning things are missing. I should have read developing on Windows more closely - it warns me that I will need to put those things on the PATH or, alternatively, use the executable in Stackā€™s environment. To do the latter, I command:

stack exec -- hackage-cli --help
2 Likes