Catching (async) exceptions

In my attempts at improving how we use Haskell at work I’m trying to make sure k8s doesn’t silently kill our services when running out of memory. So I’ve added +RTS -MxM, where xM is a reasonable amount less than the available memory. That lead to some services throwing HeapOverflow exceptions. We have some rules about the format of our logs and the default exception handler (defaultHandler, defined in uncaughtExceptionHandler) doesn’t come close. So, I thought I’d play around a little to see what my options are, but I very quickly got confused. I started looking at forkIO.

It seems that the default exception handler (defaultHandler, defined in uncaughtExceptionHandler) doesn’t handle async exceptions at all. Even though the documentation of forkIO says it does. Well, it does say it “passes all other exceptions to the uncaught exception handler”, I assumed that’d be the one I can set using setUncaughtExceptionHandler, but setting it has no effect.

I’ve found that this code (mostly) works

module Main where

import Control.Concurrent (forkFinally, forkIO, myThreadId, threadDelay)
import Control.Exception (SomeException, handle, throwIO)
import GHC.Conc (setUncaughtExceptionHandler)

main :: IO ()
main = do
    handle myExceptionHandler $ do
        putStrLn "13: trigger a sync exception in a forked thread - with my explicit exception handler"
        thrid <- forkIO threadFunc
        putStrLn $ "forked thread ID: " <> show thrid
        threadDelay 2_500_000
    putStrLn "time to exit"

threadFunc :: IO ()
threadFunc = do
    putStrLn "running thread"
    let f x = 1 + f x
    print $ f 1

myExceptionHandler :: SomeException -> IO ()
myExceptionHandler e = do
    myId <- myThreadId
    putStrLn $ str myId
  where
    str tid = "caught exception on " <> show tid <> ": " <> show e

I compiled it using ghc --make -threaded -rtsopts -o exp13 exp13.hs.

I write “mostly” because most times I get the expected output when running it

> ./exp13 +RTS -M50M; print $?
13: trigger a sync exception in a forked thread - with my explicit exception handler
running thread
forked thread ID: ThreadId 5
caught exception on ThreadId 4: heap overflow
time to exit
0

But then, every now and then I get this

./exp13 +RTS -M50M; print $?
13: trigger a sync exception in a forked thread - with my explicit exception handler
running thread
forked thread ID: ThreadId 5
caught exception on ThreadId 4: heap overflow
exp13: Heap exhausted;
exp13: Current maximum heap size is 52428800 bytes (50 MB).
exp13: Use `+RTS -M<size>' to increase it.
251

It looks like both my exception handler and the default where triggered. Surely that should never happen, right?

ghc --version
The Glorious Glasgow Haskell Compilation System, version 9.12.3

I just realised that I ought to run with -N too, and now I can’t seem to reproduce the behaviour from above.

I did see some even stranger behaviour when I’d forgotten the -threaded flag, So is that the cause, I must use -threaded when compiling and at least 2 capabilities, -N2, when running it to guarantee the expected behaviour?

The GHC user’s guide docs for -M are helpful here:

If the maximum heap size is exceeded the runtime will raise an exception on the main thread. To give a chance for a clean shutdown or recovery GHC will hold off on rethrowing the exception after it has been raised for a grace period determined by -Mgrace=⟨size⟩. After which it will be raised again if the heap size limit is still exceeded.

So I think what is happening in your example is this:

  • You have two threads, the main thread (ThreadId 4) which is waiting in threadDelay, and the forked thread (ThreadId 5) which is allocating wildly in threadFunc.
  • Eventually, an allocation hits the memory limit, so GHC bumps the limit by -Mgrace and throws HeapOverflow to the main thread.
  • In the “expected” case, the main thread catches this exception in myExceptionHandler, prints it, then quickly exits (killing thread 5 in the process).
  • In the “unexpected” case, the main thread catches the exception and leaves the scope of the handler, but thread 5 is still allocating, so another HeapOverflow occurs and this time the default handler is triggered.

It seems that the default exception handler (defaultHandler, defined in uncaughtExceptionHandler) doesn’t handle async exceptions at all. Even though the documentation of forkIO says it does. Well, it does say it “passes all other exceptions to the uncaught exception handler”, I assumed that’d be the one I can set using setUncaughtExceptionHandler, but setting it has no effect.

Can you give an example program that illustrates this problem? I believe the default exception handler, forkIO and async exceptions work “as expected”, but these things are subtle to understand, and there are definitely dark corners with other features (e.g. FFI or unsafePerformIO)!

By the way, depending on what you are doing you may find enableAllocationLimit helpful. This makes it possible to limit total allocations on a per-thread basis, although depending on your program’s retention behaviour this may or may not correlate with a limit on total live memory; it’s more like a limit on how much work the thread can perform.

Here’s a modified version of the program above

module Main where

import Control.Concurrent (forkFinally, forkIO, myThreadId, threadDelay)
import Control.Exception (SomeException, handle)
import GHC.Conc (setUncaughtExceptionHandler)

main :: IO ()
main = do
    putStrLn "13: trigger a sync exception in a forked thread - with my uncaught exception handler"
    setUncaughtExceptionHandler myExceptionHandler
    thrid <- forkIO threadFunc
    putStrLn $ "forked thread ID: " <> show thrid
    threadDelay 2_500_000

threadFunc :: IO ()
threadFunc = do
    putStrLn "running thread"
    let f x = 1 + f x
    print $ f 1

myExceptionHandler :: SomeException -> IO ()
myExceptionHandler e = do
    myId <- myThreadId
    putStrLn $ str myId
  where
    str tid = "caught exception on " <> show tid <> ": " <> show e

I compile it with

ghc --make -threaded -rtsopts -o exp exp.hs

and then run it with

./exp +RTS -M128M

The output is always

13: trigger a sync exception in a forked thread - with my uncaught exception handler
forked thread ID: ThreadId 6
running thread
exp13: Heap exhausted;
exp13: Current maximum heap size is 134217728 bytes (128 MB).
exp13: Use `+RTS -M<size>' to increase it.

Which clearly isn’t the exception handler I registered.

Oh, and adding -N2 when running it makes no difference, neither does large -M values (I tried up to 512M which I think ought to be more than enough for the program).

Yes, that makes perfect sense. It also highlights the need to be able to replace the default exception handler used for async exceptions.

I’ll make sure to look closer at -Mgrace, though after quick glance I think that its default value ought to be good enough for more reasonable cases than the code I’m using to explore the runtime behaviour above.

Huh. Looking at the implementation of real_handler it seems that GHC does indeed treat HeapOverflow (and certain other async exceptions) specially, and refrains from calling reportError (which in turn triggers the default exception handler). This seems very strange: why would it make any sense to allow HeapOverflow to be caught by a normal exception handler but not by the default handler? I think it’s worth a GHC ticket at least.

As a workaround, I suppose one could install an exception handler at the start of main, then have that handler immediately re-add another handler when it is invoked. I don’t know if this guarantees that another async exception can’t arise in between, though.