# One Billion Row challenge in Hs

**URL:** <https://discourse.haskell.org/t/one-billion-row-challenge-in-hs/8946>\
**Category:** Links\
**Created:** [March 3, 2024, 8:24am UTC](https://discourse.haskell.org/t/one-billion-row-challenge-in-hs/8946 "2024-03-03T08:24:38Z")\
**Posts on this page:** 1\
**Showing post:** 176

<div class="post-metadata">

**Author:** ![tomjaguarpaw](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/tomjaguarpaw/32/1230_2.png) [@tomjaguarpaw](https://discourse.haskell.org/u/tomjaguarpaw)\
**Post date:** [March 20, 2024, 3:59pm UTC](https://discourse.haskell.org/t/one-billion-row-challenge-in-hs/8946/176 "2024-03-20T15:59:03Z")

</div>

> [@emiruz](#):
>
> There is no such thing as “declarative all the way down”

I’m inclined to agree, but I still don’t understand what you’re asking for. Is an Opaleye-like solution satisfactory? Or a `bytestring`-like solution?

> [@emiruz](#):
>
> I suppose I’m saying it didn’t take long in this case for a very simple problem.

I suspect that’s _because_ it’s a simple problem. Once the problem becomes sufficiently complicated the challenge becomes managing complexity in the implementation rather than the speed of bit twiddling. I suspect Haskell to eclipse C in complex scenarios.

> the naive solutions literally don’t work because of the way thunks accumulate in lazy evaluation

I don’t think we’ve seen any problem in this thread that was related to thunks accumulating, have we? There are indeed some strictness annotations, but they are about specifying whether individual thunks are allocated or not, not about whether long chains of thunks are allocated.

> [@emiruz](#):
>
> I’ve run into this same problem in 3 different data processing use cases

I remember one case (resolved at [Optimiser performance problems - #30 by tomjaguarpaw](http://discourse.haskell.org/t/optimiser-performance-problems/8906/30)). Could you share the two others?

> [@emiruz](#):
>
> I was advised by the experts to either use force from deepseq or adopt a specialised strict container structure

The suggestion to use `force` from `deepseq` is absolutely wrong, and adopting the strict container structure (I wouldn’t call it “specialised”, I’d just call it “correct”) is absolutely the right solution (i.e. [make invalid laziness unrepresentable](http://h2.jaguarpaw.co.uk/posts/make-invalid-laziness-unrepresentable/)). I am willing to die on this hill! In brief: if you don’t want thunks to appear in your data type then choose a data type that doesn’t allow thunks to appear.

> [@emiruz](#):
>
> told that the `ad` package is basically doomed to be slow because of the way it deals with some aspect of thunk management

No, that’s not right. [I explained](http://discourse.haskell.org/t/optimiser-performance-problems/8906/2) that the approach to AD that the `ad` package takes is doomed to be slow on vector problems. That has nothing to do with thunks.

It’s a curious thing. There’s one large group of people (mostly outside the Haskell world) that thinks that “Haskell is hard to use because laziness and thunks make it hard to reason about performance and space usage” and another group of people (mostly Haskell insiders) who think “Haskell is great, laziness has all sorts of benefits, and no problems in practice”. They’re both wrong! Laziness has severe implications when used incorrectly, but the solution is simple, not difficult: [make invalid laziness unrepresentable](http://h2.jaguarpaw.co.uk/posts/make-invalid-laziness-unrepresentable/). Unfortunately this message has not really percolated very far in the Haskell world. I will keep on trying.

> [@emiruz](#):
>
> a noob like me obviously wonders what that means at scale or with more complex problems

To reiterate, Haskell will be _better_ for more complex problems, not worse! After all, the most simple problems will always be fastest with hand-crafted assembly.

---

_[View the full topic](https://discourse.haskell.org/t/one-billion-row-challenge-in-hs/8946)._