I am currently experiencing a mysterious memory leak. Profiling with -hc reports stable 8MB of memory residency, so there does not seem to be apparent leaking memory.
However, the executable is gradually eating up more memory. Initially it was using 100MB, then increases to 200MB, then 300MB, … it never stops. I don’t get what is happening… Does anyone know how such anomaly could happen?
My favourite thing to diagnose! If you post a small sample that exhibits the problem I will try to track down the cause. Otherwise my advice is
foldl' not foldl)-fno-full-laziness
Thing is, it is a program involving GUI. I don’t think I could water it down into simple components.
If it was caused by lazy thunks not being evaluated, surely it would be caught by profiler, no?
One would think so, but given that you are seeing memory increases that never stop, and the profiler shows 8MB constant memory, I guess something is wrong with the profiler.
Oh, is profiler not reliable? Hmm, I wish there is anything else to . The codebase, while being very small, is rather large to me.
I certainly never fold anything, and most of my data has strictness annotation.
I would first try to make sure that you are measuring the right thing. What does -s report about maximum residency? Maybe also try eventlog2html.
Maybe someone knows of a reason why the profile shows less memory usage than you are telling us the OS (I think) reports.
The only reason for mismatch in memory usage that I know of is memory fragmentation, but I don’t think that can cause such a large difference in a relatively normal program.
Let me check that when I am back to the computer.
The profiler shows only data stored on the Haskell heap, so another possibility is that the leak exists in memory managed on the C side (e.g. via FFI). You mentioned a GUI - which library is that using?
GTK. Now you mention it, that sounds quite likely. Thing is, how do I profile such C memory?
I’m no expert on C memory profiling, alas. AFAIK it should be possible to use standard tools (e.g. Valgrind) to look for memory allocated with malloc() and never freed. That might at least help confirm the presence of a leak and perhaps point to the C allocation site, although it wouldn’t directly tell you which Haskell code was responsible. (C tools generally won’t understand the Haskell call stack or heap structure… but in principle the RTS is “just” a C program.)
Verify that the issue is not fragmentation related. The easier way to do this is to render a profile using eventlog2html and look at how the “live bytes” relates to the “heap size”. You should expect that heap size is approximately twice the amount of live bytes (if you are using the default copying GC). If the heap size is much larger than live bytes (and corresponds to OS memory usage) then you probably have an issue with fragmentation. Otherwise, it is probably off-heap memory which is being leaked.
If the heap profile looks standard then you can try investigating C memory leaks using gperftools. I have an example project here which explains how to set up a project to use this profiler.
Got the following with -s option. Duh, it requires normal termination, not killing via signal, but my program normally never terminates.
84,624,352,560 bytes allocated in the heap
7,374,360 bytes copied during GC
14,800,208 bytes maximum residency (20 sample(s))
129,656 bytes maximum slop
49 MiB total memory in use (0 MB lost due to fragmentation)
14MB maximum residency, while memory consumption was increasing to near 200MB.
Guess it is likely the FFI side, then.
eventlog2html also confirms that (ghc) heap size remains at constant 42MB. Far higher than maximum residency (or live bytes lying at 14MB) but eh. At least no leak there, this one is constant. (OS Memory was exceeding 200MB at this point)
Trying C debugging.
valgrind so far does not give any specific signals. Only 38MB is “possibly” lost on exit. (Think it might be tripped on that haskell runtime is unusual)
Even scarier, memory usage with presence of valgrind is stable.
++
Now I am in horror, somehow at some point it stopped taking more memory!
I don’t know what is happening, perhaps -optc-g is lessening the memory leak?
Hmmm.
++
Just found that opening VSCode is causing (not my code) to emit icon not found errors, causing strange phenomenon of memory explosion! I don’t know how “icon not found” could cause such problems.
How do I work with gperftools? I can’t find any way to install e.g. pprof.
How are you measuring the constant memory increases?
Reports from OS. If it were in small scale, I ofc wouldn’t consider the deal to be too much.
However, it increases starting from 60MB to reaching over 500MB sometimes!
It can even increase further, IIRC I’ve seen it taking 3GB (although I don’t recall it well)
Are you running your app on a wayland or XOrg sytem? Was just curious if they had the same behaviour, while reading the latest GNOME release notes where they mention having a memory leak issue with GTK3 that is only manifesting on wayland (with a similar symptom of unaccounted for memory) memory leaks under wayland (#4092) · Issues · GNOME / GIMP · GitLab
It is running on XOrg system, so I don’t think that is related.
Yea, identified and fixed the problem. I love when the memory leak issue is caused by a dependency, not my own code. What can I do when the dependency library has foldl (not foldl') littered around, duh.