Facing "mmap 4096 bytes at (nil): Cannot allocate memory"? You're not alone!

#1

You! yes, you!

Are you facing a segfault with the following error?

GHCi, version 9.2.7: https://www.haskell.org/ghc/  :? for help
ghci> 40000 / 9
4444.444444444444
ghc: mmap 4096 bytes at (nil): Cannot allocate memory
ghc: Try specifying an address with +RTS -xm<addr> -RTS
fish: Job 1, 'ghci' terminated by signal SIGSEGV (Address boundary error)

You will be pleased to know that migrating to GHC 9.4 solves this problem, as explained in #19421: internal error: m32_allocator_init: Failed to map · Issues · Glasgow Haskell Compiler / GHC · GitLab.

11 Likes
Cabal's foreign-library stanza
#2

and if you can’t upgrade your ghc, upgrade your linux kernel to 6.2.12 or something (technically 6.2.9 or later with this patch)

7 Likes
#3

I’m afraid the problem is present even with 6.3.1 linux kernel. So it seems GHC 9.4 is the only way.

Why wouldn’t the fix be backported to 9.2 though?

Funnily, this issue did get worse with newer kernels.

Btw, can someone confirm that adding +RTS -xp -RTS (or export GHCRTS='-xp') actually fixes the problem (or makes it much more unlikely)?

7 Likes
#4

It’s present in 6.3.1 for some reason, but not in 6.2.12 (or maybe others in the 6.2.x line, I haven’t tested them all). I think it was even fixed in 6.3.0-rc7 but for some reason not in 6.3.1.

1 Like
#5

The kernel people promised to backport the fix for all the affected versions.
But I’m not seeing that in my Ubuntu mainline archive yet…

1 Like
#6

Are people hitting this on 6.3 also able to reproduce with the C programs from int-e (@int-index) on Memory allocation errors on 6.1.1 / Kernel & Hardware / Arch Linux Forums ? I am curious whether it is a separate bug. But it would be odd if it wasn’t also related to the new maple trees.

1 Like
#7

Yes, the test program stops after relatively few requests using kernel version 6.3.1 on my machine. The shortest of multiple runs was only 23 requests, shown below!

mmap-test.c:

#include <sys/mman.h>
#include <stddef.h>
#include <stdio.h>

int main()
{
    for (int i = 0; i < (1 << 16); i++) {
        void *p = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0);
        printf("%p\n", p);
        if (p == (void *)-1) {
            printf("%d\n", i);
            return 0;
        }
    }
}

Test:

$ uname -srvmo
Linux 6.3.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 01 May 2023 17:42:39 +0000 x86_64 GNU/Linux
$ gcc -o mmap-test mmap-test.c
$ ./mmap-test
0x40898000
...
0x417e2000
0xffffffffffffffff
23
2 Likes
#8

Unfortunately, upgrading to ghc 9.4 isn’t an option for many, including those of us who need to check that library code is compiling on older versions of ghc.

I’m experiencing this with both the current and LTS versions of the linux kernel on ArchLinux (6.3.1-arch2-1 and 6.1.27-1). Do we know if the upstream kernel is going to be fixing this promptly or should we be looking into compiling our own kernels and running them (or downgrading)? 6.2.9 works for me.

1 Like
#9

using
export GHCRTS='-xp'
gives me a

R_X86_64_32S relocation out of range: (noname) = 7fdf70318050

on ghc-8.10.7 compiled apps, vs
mmap 4964352 bytes at (nil): Cannot allocate memory when I don’t use any flags.

1 Like
#10

Liam Howlett just mailed in 35 patches that seem related: '[PATCH v3 00/35] Maple tree mas_{next,prev}_range() and cleanup' - MARC

#11

Robert Hensing reports (possibly using bisect?) that an issue was reintroduced into the 6.1 series in commit 58c5d0d, titled mm/mmap: regression fix for unmapped_area{_topdown}. Googling this commit, I found it in the 6.1.26 changelog and the 6.3 changelog.

So it’s likely that 6.1.25 is ok.

The title of commit 58c5d0d was not listed in the rc7 changelog (compare 6.3 announcement, where it is present). So this is consistent with your observation.

2 Likes
#12

Unfortunately the ghci 9.2 crash is happening sometimes in current Fedora releases now (~10% of the time in kernel-6.2.15-300; 6.2.9-300 is okay) and more often with kernel-6.3.3-200 and 6.4.0 rc’s (rawhide).
So I am starting to wonder if this behavior will really change (back) in the Linux kernel or not?
(Though I am hoping to ship ghc-9.4 in Fedora 39…)

The crash happens for me only sometimes and before ghci even shows the initial prompt
(the core files are about 4MB apparently).

Also worth mentioning Sylvain’s draft backport MR too (not a small patch though by any definition). If you are using Fedora 38 you can update to this temporary scratch build which includes the patches, eg using koji-tool install 101285362.

2 Likes
#13

I don’t know if all those patches are included, but I can confirm that 6.3.2 does NOT fix the problem. I’m holding onto 6.2.9 for now.

1 Like
#14

Is there a minimal reproducer in Haskell?

#15

I am working on a Reflex FRP project, where upgrading to GHC 9.4 is not an option. Using export GHCRTS='-xp' causes Obelisk to fail, reporting that it does not support that RTS option, but adding +RTS -xp -RTS to baseGhciOptions as follows makes the ob run and ob repl commands work for me. Thank you @fintara for the hint about using that RTS option!

diff --git a/lib/command/src/Obelisk/Command/Run.hs b/lib/command/src/Obelisk/Command/Run.hs
index 0e2b6b3d..6fce8551 100644
--- a/lib/command/src/Obelisk/Command/Run.hs
+++ b/lib/command/src/Obelisk/Command/Run.hs
@@ -552,6 +552,7 @@ baseGhciOptions =
   , "-no-user-package-db"
   , "-hide-all-packages"
   , "-package-env", "-"
+  , "+RTS", "-xp", "-RTS"
   ]

 -- | Run ghci repl

For Nix builds that have caused problems, I am simply doing those in a VM running an older kernel.

$ uname -a
Linux ghc810 5.15.110 #1-NixOS SMP Sun Apr 30 23:23:24 UTC 2023 x86_64 GNU/Linux
3 Likes
#16

Just checked 6.3.3, no good.

1 Like
#17

this is now fixed (again) in linux kernel 6.3.4

3 Likes
#18

Thanks @fommil

The maple_tree fix also appears to be in Linux kernel 6.4-rc3, so that is also good news.
I tested it too successfully already in Fedora Rawhide.

Also I have pushed Fedora updates for the ghc and ghc9.2 packages with the backport into testing.

#19

We have released ghc-9.2.8 with a fix to this issue.

11 Likes
#21

For anyone still stuck on a ghc prior to 9.2.8, adding the following to your cabal component works:

  ghc-options:
    +RTS -xp -RTS

You could do cabal repl --ghc-options='+RTS -xp -RTS' some-target but that will also rebuild dependencies.