State monad - memory exhausted

[tomjaguarpaw]

@Ambrose

I thought CPS Writer basically was State under the hood?

Yes, that’s right.

@jaror

Edsko de Vries from Well-Typed has written the nothunks library

nothunks was the inspiration for my article Make Invalid Laziness Unrepresentable. After I learned about that library I thought “after we’ve checked a value for thunks it would be nice to indicate that in the type, so that we know we don’t have to ever check it again” (I was imagining introducing a ThunkFree newtype wrapper). Then I realised that the simpler approach was to forbid thunks from occurring in the first place, through a better definition of the type in question!

@ajbarber

it just seems too important to be relegated to some third party tool.

Yes, that’s why you should make invalid laziness unrepresentable. Then you don’t have to rely on any tool. The impossibility of thunks just becomes part of your program.

@ajbarber

say a PR lands on my desk with a small change where, in reviewing this change, I am guilty of not doing a thorough top down analysis of the problem, and approve the seemingly innocuous change, which passes CI, but … leads to a horrible server crashing memory leak.

Do you think it is acceptable, that ghc provides no warning (albeit noisy), of this situation occuring?

We’re all about type safety, but I wonder have we paid enough attention to memory safety in ghc.

I don’t think this situation is acceptable, but it’s not GHC’s fault. The author of loopM chose to allow thunks to build up in the state that is passed between iterations (similar to foldl). Why should GHC object to that? Maybe that’s necessary for some applications. If the author wanted different behaviour he should have implemented it differently. If the user wanted different behaviour she shouldn’t have used loopM. It’s not GHC’s job to decide! (This is why I have opened the discussion Why shouldn't I make my monads "value strict"? - #2 by sgraf)

So what exactly do I find unacceptable? Our ecosystem has 100 laziness footguns, foldl, modifyIORef, Control.Monad.Trans.State.Strict.modify, all of Control.Monad.Trans.State (i.e. not .Strict), all of Data.Map.Lazy, … . It is so easy to create catastrophic space leaks using them that they should only be used by experts in very specific circumstances (generally to eke out performance). But we don’t do a very good job of educating users about this in general, nor of finding way of discouraging use of the footguns through other means.