You can fix that leak by adding bangs:
f3 :: Vector Double -> Vector Double
f3 v = let !x = v V.! 0
!y = v V.! 1
!z = v V.! 2 in
V.fromList [2*(x-1),2*(y-2),2*(z-3)]
1 Like
That fixes the toy example 
but not the real code, even after turning ps into a vector and using V.map, V.sum, and adding even more bangs. Is there some way to more systematically determine what is leaking other than eye-balling?
Did you try forcing your thunks? Bangs only take care of the top level. Or you could try unboxed vectors, but force is the make-sure-it-works option.
An eye-watering number of methods 
Haskell Optimization Handbook is still a work in progress, but already has lots of good info.
3 Likes
Ok I’ve “fixed” it. If I add the following anywhere it results in th' getting evaluated on each recursion and the memory not growing.
(trace $ show $ V.maximum th')
Now is there some way to do that @jaror without causing work?
<tap> <tap> Is this thing on?
force. force is the way to do that.
1 Like
Hmm…I’m guessing this one:
O(n) Yield the argument, but force it not to retain any extra memory, possibly by copying it. [...]
…but be careful when using functions like these more generally - if ⊥ is lurking anywhere inside the given argument, it usually stops the program e.g. no output, being unresponsive to new input, a runtime error, etc:
Thank you, yes others had mentioned force. I’m just surprised that I have to do this sort of thing given the simplicity of the code. I had hoped there would be a more out-of-the-box tweak.
PS: force from Data.Vector does not work – I’ll try the deepseq one.
EDIT: deepseq worked as advertised. Thank you all.
OK, I now understand what’s going on.
Right, and the reason the problem doesn’t appear on your original example but it does in a higher-dimensional version is because of @rhendric’s observation:
The space leak occurs if the original example is changed from V.fromList [-2000] to V.fromList [-2000, -2000] (so the second coordinate can leak), plus abs x < 1e-8 to abs x < 0 (so that it runs long enough to see the leak).
Yes, make invalid laziness unrepresentable. I disagree with others who are saying that there are too many methods, that you should sprinkle bang patterns, or that you should use force. There is no need to read a book, there is no need to apply the blunt hammer of deepseq, you just need to choose the correct data structure. Here is your original code, fixed by simply choosing the correct data structure:
{-# LANGUAGE GHC2021 #-}
import Numeric.AD (grad)
import qualified Data.Vector as V
import qualified Data.Vector.Generic as G
import qualified Data.Strict.Vector as SV
-- With lazy V.Vector it leaks, with strict SV.Vector it does not
-- leak.
type AdaVector = SV.Vector
-- type AdaVector = V.Vector
adaMax f' start alpha beta1 beta2 = adaMax' start 0 zeros zeros
where
zeros = G.fromList (replicate n 0)
n = length start
adaMax' th t m u
| all (\x-> abs x < 0) g = th'
| otherwise = adaMax' th' (t+1) m' u'
where g = f' th
m' = G.map (\(m,g)->beta1 * m + (1-beta1) * g) (G.zip m g)
u' = G.map (\(u,g)->max (beta2*u) (abs g)) (G.zip u g)
th' = G.map (\(th,m,u)->th - (alpha / (1-beta1^(t+1))) * m/u) $ G.zip3 th m' u'
fTest = grad (\xs -> (G.head xs)^2 - (G.head xs))
test1 = adaMax fTest (G.fromList @AdaVector [-2000, -2000]) 0.002 0.9 0.999
main = test1 `seq` pure ()
All I have done is generalized your code to the Data.Vector.Generic interface so we can compare what happens when we use Data.Vector to what happens when we use Data.Strict.Vector (from strict-containers). With Data.Vector.Strict there is no space leak. No need for !, seq, force, or any form of careful analysis. That is the benefit of making invalid states unrepresentable through a powerful type system!
In practice you probably wouldn’t even bother with this generalization, you’d just write your code against Data.Strict.Vector directly.
Why is this the correct way to write the code? Because in your particular use case there is no purpose to being able to store unevaluated thunks as vector elements. Therefore, let’s statically ensure that the vector cannot possibly leak space by choosing the correct vector data type. (I would actually guess that in 99% of practical use cases there’s no need to store unevaluated thunks either.)
13 Likes
Thank you – this is exactly what I had hoped for: a “no, use X data structure instead” solution. I was feeling quite bad about having to use a function called “force”. Sort of like saying, no it doesn’t work so just bash it into shape.
7 Likes
Add strict vector by Shimuuar · Pull Request #488 · haskell/vector · GitHub 
the future has fewer footguns (you won’t have to know already that strict-containers exists, just notice from the module list of vector that there is one named Strict)
6 Likes
Is there some way to more systematically determine what is leaking other than eye-balling?
Yes, perhaps take a look at 8. Profiling — Glasgow Haskell Compiler 9.12.1 User's Guide, the memory profiling section.
Here’s a writeup about memory profiling A First Look at Info Table Profiling - Well-Typed: The Haskell Consultants, focusing on one of the profiling methods (info table).
1 Like