Cross-Compiling to 32-bit arm; weird syscall behaviour

This is probably a bit of a niche use-case that is not well-supported, as I understand, but I wanted to work on a small haskell application for my e-book reader, for which I would have to target a 32-bit ARM architecture.

I was using haskell.nix for cross compilation in different variants (including fully statically linked builds against glibc and musl and a dynamically linked build where I would deploy all dynamic dependencies onto the device). I have had varying degrees of success with this. For example, computing fibonacci numbers and printing out the results was working fine. However, as soon as I introduced some code interacting with the filesystem, the stuff got a bit wonky. For example, when I am having the equivalent of readFile "SomeExistingFile" >>= putStrLn in my file, the output would be just the name of my executable, I guess, followed by a colon.

Originally, I aimed for making some syscalls via FFI (which work locally and also when I cross-compile a similar C program for my reader), but the syscalls return weird stuff (like open returning -1, even though it returns a valid file descriptor in the equivalent C code).

For now, I have failed to set up a cross-compiler toolchain for anything newer than GHC 9.2.8 (haven’t tested with GHC 9.4.x yet), but my basic setup is the following haskell.nix-based flake, sorry it is a bit messy:

My flake.nix
  inputs.haskellNix.url = "./haskell.nix";
  inputs.nixpkgs.follows = "haskellNix/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  outputs = { self, nixpkgs, flake-utils, haskellNix }:
    let
      supportedSystems = [
        "x86_64-linux"
      ];
    in
      flake-utils.lib.eachSystem supportedSystems (system:
      let
        drv' = pkgs': extraModules:
          let
            isCrossStatic = pkgs'.stdenv.hostPlatform != pkgs.stdenv.hostPlatform;
          in pkgs'.haskell-nix.project {
            src = ./.;
            compiler-nix-name = (if isCrossStatic then "ghc928" else "ghc964");
            modules = [
              {
                 packages.koboworld.components.exes.koboworld =
                   let p = nixpkgs.lib.strings.makeSearchPath "include" (with pkgs'; [
                     linuxHeaders
                   ]);
                   in {
                     configureFlags = ["--extra-include-dirs=${p}"];
                   };
              }]
              ++ (if isCrossStatic then
                  [{
                    packages.koboworld.components.exes.koboworld = {
                      configureFlags = [
                        "--disable-executable-dynamic"
                        "--disable-shared"
                        "--ghc-option=-optl-pthread"
                        "--ghc-option=-optl-static"
                        "--ghc-option=-optl-lffi"
                        "--ghc-option=-optl-lnuma"
                        "--ghc-option=-optl-lgmp"
                        "--ghc-option=-optl-lz"
                        "--disable-executable-stripping"
                        "--disable-library-stripping"
                      ] ++ map (l: "--ghc-option=-optl=-L${l}/lib") (with pkgs'; [
                        gmp6
                        zlib
                        libffi
                        numactl
                      ]);
                    };
                  }]
                else [])
              ++ extraModules;
            shell.tools = {
              cabal = {};
            } // (if isCrossStatic then {} else {
              hlint = {};
              haskell-language-server = {};
            });
          };
        drv = pkgs': drv' pkgs' [];
        pkgs = haskellNix.legacyPackages.${system};
        pkgsArm = haskellNix.legacyPackages.${system}.pkgsCross.armv7l-hf-multiplatform.pkgsMusl;
      in {
        p = (drv pkgs);
        packages = {
          "exe:koboworld" = (drv pkgs).koboworld.components.exes.koboworld;
          "armv7l-hf-multiplatform:exe:koboworld" = (drv pkgsArm).koboworld.components.exes.koboworld;
        };
        devShells = {
          default = (drv pkgs).shell;
        };
      }
    );
}```

For the musl-based build I built GHC with ld.gold, even though this is disabled in haskelll.nix, but since the referenced ld.gold bug is fixed, I still wanted to give it a try. The behaviour, I have described above, however, is the same as if I do a dynamically linked build against glibc and deploy all dependencies onto the device (or a statically linked one against glibc, I think, but I can’t confirm this easily anymore right now and this does not seem to be super-safe?).

Before I give up on this foolish undertaking, I wanted to ask whether any of you has any hints on how I could possibly proceed?

2 Likes

Let me have a look today. I remember something similar from the aarch64 backend.

@marmayr I’ve tried to reproduce this, but was unable to.

git clone git@github.com:angerman/nixpkgs-static-repo.git --branch angerman/discourse-32bit-arm-repro
nix build .#hydraJobs.x86_64-linux.armv7-musl.myprogram

this will produce result/bin/myprogram, using qemu to run it results in:

nix-shell -p qemu
qemu-arm ./result/bin/myprogram 
foo

where foo is the content of test.txt

I’m afraid there must be something else going on in your code, I can’t reproduce right now. Do note that this points at a custom haskell.nix branch right now, which also sets ld.gold (though I’m thinking we might want to use lld when using the llvm toolchain anyway).

Thanks a lot, I will try to reproduce your results with qemu and on my target device and report back. In any case, it is super helpful to have a working example, I can start from. Thank you for your help!

If you can provide a broken reproducer, I’m happy to try to debug it.

Everything is working now, thanks!

After running through the steps in your post, I’ve basically swapped out my flake.nix file for your one and it started working immediately. I’m also very happy that I can use GHC9.6 now and won’t investigate how exactly I have messed up. I am impressed that it is actually much easier to create those binaries than I thought, couldn’t get this from reading the haskell.nix page on cross compilation.

Thanks again!

Yes, I’m stretched very thin, and the documentation isn’t where I’d like it to be :frowning: Any help is appreciated!