I am currently experiencing a mysterious memory leak. Profiling with -hc reports stable 8MB of memory residency, so there does not seem to be apparent leaking memory.
However, the executable is gradually eating up more memory. Initially it was using 100MB, then increases to 200MB, then 300MB, … it never stops. I don’t get what is happening… Does anyone know how such anomaly could happen?
Thing is, it is a program involving GUI. I don’t think I could water it down into simple components.
If it was caused by lazy thunks not being evaluated, surely it would be caught by profiler, no?
One would think so, but given that you are seeing memory increases that never stop, and the profiler shows 8MB constant memory, I guess something is wrong with the profiler.
Oh, is profiler not reliable? Hmm, I wish there is anything else to . The codebase, while being very small, is rather large to me.
I certainly never fold anything, and most of my data has strictness annotation.
The only reason for mismatch in memory usage that I know of is memory fragmentation, but I don’t think that can cause such a large difference in a relatively normal program.
The profiler shows only data stored on the Haskell heap, so another possibility is that the leak exists in memory managed on the C side (e.g. via FFI). You mentioned a GUI - which library is that using?
I’m no expert on C memory profiling, alas. AFAIK it should be possible to use standard tools (e.g. Valgrind) to look for memory allocated with malloc() and never freed. That might at least help confirm the presence of a leak and perhaps point to the C allocation site, although it wouldn’t directly tell you which Haskell code was responsible. (C tools generally won’t understand the Haskell call stack or heap structure… but in principle the RTS is “just” a C program.)
Verify that the issue is not fragmentation related. The easier way to do this is to render a profile using eventlog2html and look at how the “live bytes” relates to the “heap size”. You should expect that heap size is approximately twice the amount of live bytes (if you are using the default copying GC). If the heap size is much larger than live bytes (and corresponds to OS memory usage) then you probably have an issue with fragmentation. Otherwise, it is probably off-heap memory which is being leaked.
If the heap profile looks standard then you can try investigating C memory leaks using gperftools. I have an example project here which explains how to set up a project to use this profiler.
Got the following with -s option. Duh, it requires normal termination, not killing via signal, but my program normally never terminates.
84,624,352,560 bytes allocated in the heap
7,374,360 bytes copied during GC
14,800,208 bytes maximum residency (20 sample(s))
129,656 bytes maximum slop
49 MiB total memory in use (0 MB lost due to fragmentation)
14MB maximum residency, while memory consumption was increasing to near 200MB.
Guess it is likely the FFI side, then.
eventlog2html also confirms that (ghc) heap size remains at constant 42MB. Far higher than maximum residency (or live bytes lying at 14MB) but eh. At least no leak there, this one is constant. (OS Memory was exceeding 200MB at this point)
Trying C debugging. valgrind so far does not give any specific signals. Only 38MB is “possibly” lost on exit. (Think it might be tripped on that haskell runtime is unusual)
Even scarier, memory usage with presence of valgrind is stable.
++
Now I am in horror, somehow at some point it stopped taking more memory!
I don’t know what is happening, perhaps -optc-g is lessening the memory leak?
Hmmm.
++
Just found that opening VSCode is causing (not my code) to emit icon not found errors, causing strange phenomenon of memory explosion! I don’t know how “icon not found” could cause such problems.
Reports from OS. If it were in small scale, I ofc wouldn’t consider the deal to be too much.
However, it increases starting from 60MB to reaching over 500MB sometimes!
It can even increase further, IIRC I’ve seen it taking 3GB (although I don’t recall it well)
Are you running your app on a wayland or XOrg sytem? Was just curious if they had the same behaviour, while reading the latest GNOME release notes where they mention having a memory leak issue with GTK3 that is only manifesting on wayland (with a similar symptom of unaccounted for memory) memory leaks under wayland (#4092) · Issues · GNOME / GIMP · GitLab
Yea, identified and fixed the problem. I love when the memory leak issue is caused by a dependency, not my own code. What can I do when the dependency library has foldl (not foldl') littered around, duh.