I’m learning haskell, and there has something that has been bothering me about naive vs. advanced haskell.
I came across this great somewhat old blog post (but I am able to repro with a ghc 8.10.1).
The gist is that optimizing for speed and memory usage is not always about strictness. The author gives an example of calculating the mean of a sequence of numbers. Once the list of numbers grows large (1e8), the naive implementation exhausts memory and crashes. It’s only upon looking at the Core output that the illuminating detail is found: that the garbage collector can’t clean up a list (even a lazily evaluated list) if that list is used in a subsequent computation. So, sum myLazyList / length myLazyList
fails because myLazyList
stays in memory, because the compiler assumes that it can’t clean up during the sum because length
also needs the list.
The (first) solution, which the author says is the bread and butter of functional programmers for 40 years, is to do the sum and the length together:
mean :: [Double] -> Double
mean = go 0 0
where
go :: Double -> Int -> [Double] -> Double
go s l [] = s / fromIntegral l
go s l (x:xs) = go (s+x) (l+1) xs
My question (finally) is that even for this seemingly trivial example, the details of an efficient solution seem to only be revealed when looking at Core output, and reasoning very carefully about what is happening. This implies (to me) that for much larger programs, finding the performance problems seems like it would be a very difficult undertaking. Is writing performant haskell really this challenging?
Are there tools that would have revealed something like the above quickly, or at a unittest level? Can discrete parts of a large haskell program be thusly performance tested?
Can the naive/advanced patterns (the above is an example) be mapped to the same kind of naive/advanced patterns of other programming languages? In other words, is it more/less/same difficulty in optimizing, say, C++, if one were new to it?
Thanks!