Effectful is better at running unspecialized mtl-style code than transformers

Show and Tell
1

This is something I’ve discovered a year ago and wanted to include in a refined version of benchmarks, but since I didn’t manage to do that in such a long time, let me just share this particular bit.

A reminder:

  1. countdown - a microbenchmark that effectively measures performance of monadic binds and the effect dispatch.
  2. filesize - a more down to earth benchmark that does various things, including I/O.

Each benchmark has two flavours that affect the amount of effects available in the context:

  • shallow - only effects necessary for the benchmark.

  • deep - necessary effects + 5 redundant effects put into the context before and after the relevant ones (10 in total). This simulates a typical scenario in which the code uses only a portion of the total amount of effects available to the application.

Here are the results of cabal run bench -- -p countdown.1000.mtl and cabal run bench -- -p filesize.1000.mtl from the effectful repo on my machine (Ryzen 9950x3D) with GHC 9.10.3, neatly put in a table.

Countdown

Benchmark Duration Allocated Copied by GC
effectful (shallow) 35.8 μs ± 1.3 μs 259 KB 57 B
transformers (shallow) 49.0 μs ± 4.5 μs 407 KB 53 B
change +36.9% +57.1% −7.0%
effectful (deep) 36.7 μs ± 2.7 μs 266 KB 190 B
transformers (deep) 391 μs ± 34 μs 3.0 MB 603 B
change +965.1% +1027.8% +217.4%

Filesize

Benchmark Duration Allocated Copied by GC
effectful (shallow) 801 μs ± 25 μs 6.5 MB 192 KB
transformers (shallow) 938 μs ± 90 μs 7.0 MB 267 KB
change +17.1% +7.7% +39.1%
effectful (deep) 805 μs ± 49 μs 6.5 MB 198 KB
transformers (deep) 2.12 ms ± 172 μs 14 MB 694 KB
change +163.4% +115.4% +250.5%
Raw data 
$ cabal run bench -- -p countdown.1000.mtl
All
  countdown
    1000
      mtl (effectful)
        shallow: OK
          35.8 μs ± 1.3 μs, 259 KB allocated,  57 B  copied, 6.0 MB peak memory
        deep:    OK
          36.7 μs ± 2.7 μs, 266 KB allocated, 190 B  copied, 6.0 MB peak memory
      mtl (transformers)
        shallow: OK
          49.0 μs ± 4.5 μs, 407 KB allocated,  53 B  copied, 6.0 MB peak memory
        deep:    OK
          391  μs ±  34 μs, 3.0 MB allocated, 603 B  copied, 6.0 MB peak memory

All 4 tests passed (0.87s)

$ cabal run bench -- -p filesize.1000.mtl
All
  filesize
    1000
      mtl (effectful)
        shallow: OK
          801  μs ±  25 μs, 6.5 MB allocated, 192 KB copied, 7.0 MB peak memory
        deep:    OK
          805  μs ±  49 μs, 6.5 MB allocated, 198 KB copied, 7.0 MB peak memory
      mtl (transformers)
        shallow: OK
          938  μs ±  90 μs, 7.0 MB allocated, 267 KB copied, 7.0 MB peak memory
        deep:    OK
          2.12 ms ± 172 μs,  14 MB allocated, 694 KB copied, 7.0 MB peak memory

All 4 tests passed (0.90s)

effectful always runs faster and allocates less, even in the shallow countdown microbenchmark which compares Eff vs State (which was pretty surprising to me). This is primarily because State is StateT Identity, so every monadic bind is 2 function calls vs 1 call with Eff. Basically, for a transformer stack of depth N, each monadic bind of an unspecialized mtl code is N function calls. That’s why results with a deep stack (not that deep really, easily reachable by a non-trivial application) are so bad.

Moreover, it’s not only about raw speed. Running mtl code via Eff also puts less pressure on the garbage collector.

Given this info, I’d like to announce that recently I’ve released effectful 2.7.0.0 and 2.7.1.0 with a lot of goodies, so it’s never been a better time to try it out :wink:

13 Likes