How to start collecting data in the eventlog after a signal is sent to a process

Learn
#1

Hey all!

I have a long running server like program that memory leak in a condition related to DB access. I normally use -prof, set SCC at the interesting call sites and then launch the program with ./program +RTS -l -hc --RTS to generate a eventlog to be processed by eventlog2html.

This works fine small short running programs. On my case, this long running server has an specific network specific condition that trigger this leak. We notice it by checking the RSS on htop not being released after 15 major collections. The server still works though. The eventlog file generated is huge, in the order of 25GB and eventlog2html cannot process it without the OOM triggering. In fact I am not event interested in most of the data, I only want to collect data on the eventlog after I notice the condition.

I was think on instrumenting a signal handler for SIGUSR1 and then enable collection programmatically. I don’t see any module on base under the GHC. hierarchy to do so nor blogposts about it. I did noticed about profiling eras, is this the best approach to do this with the current setup?

#2

Perhaps you want to use GHC.Profiling along with the RTS flag described in the docstring

#3

Maybe I am doing something wrong, but I get the process killed after 10 seconds

main :: IO ()
main = do
   runFilteredLoggingT $ runThreads [apiThread, liftIO delayProf]

delayProf :: IO ()
delayProf = threadDelay 10_000_000 *> startHeapProfTimer

I am running with

stack exec --profile program --rts-options '-l -hc --no-automatic-heap-samples'
#4

What’s the definition of runThreads? I’m wondering if it terminates when either thread finishes

1 Like
#5

runThreads was foldr1 race, that explains it all, thank you!

1 Like