If I recall correctly, their core engine and several microservices were built entirely in Haskell, but they seemed to be facing a lot of memory challenges and even had to rope in Well Typed to help them investigate. Not surprised by this move at all.
7 Likes
If that is true, well it is a shame. Understandable. Rust starts looking like a holy grail when you have memory issues.
I think Haskell projects get a pretty short lease in commercial “systems” projects. Which is a shame because it’s an exceedingly good systems language.
What’s interesting is I think the imperative language most comparable to Haskell out there today for systems work is Golang. And GHC in many ways has a superior RTS to Golang (especially when you consider the potential engineering-hour difference due to Golang being more commercially successful). And personally, I’ve shot myself in the foot due to Golang’s RTS and semantics more than I have GHC’s, despite having more experience with GHC in prod.
Is it just that GHC has rougher corners - or even steep valleys you can fall into? Maybe there are simple and elegant ways out.
I remember the Pusher article from 2016 (time flies). There was a lot of discussion about how Haskell is fundamentally flawed for certain workloads. Years go by and we now have two great new pieces of systems programming available to us: Compact regions and the non-moving GC. Not to say those were easy to implement (they were huge efforts), but they resulted in a stepwise increase in GHC’s viability in different situations. The Pusher of today would be able to make small changes to their code and config and potentially never had issues.
9 Likes
There were other factors too, but the Haskell team being understaffed was indeed one of the major problems. Over time engineers left and business struggles led to layoffs. Positions that needed to be backfilled weren’t for long periods of time, and when they were there was a refusal to hire below senior levels. I think it’s a testament to Haskell that a much smaller remaining team could continue maintaining everything and developing new features. But eventually there wasn’t enough bandwidth to improve core infrastructure and those strengths turned into bottlenecks.
Python
6 Likes
Yeah, it’s hard to compete with that initial simplicity of having effects everywhere, rather than e.g being seemingly-trapped behind an abstract interface which still defies a simple explanation. It will be interesting to see if they stay with Python in the long-term though, as the task of programming gets more difficult, not less.
1 Like
I would like to counter the gloom of this thread with a happy tale of conversion.
My work codebase was 100% Python until a few months ago, for the usual reasons:
- There is a large pool of Python developers;
- Python is easy to get started, so it feels productive at first;
- There is a large ecosystem of third-party libraries ( e.g. how many programming languages have a large weather science toolkit available ?
But of course, slowly, cracks in the foundation appeared. Python is slow for most things. People started dreading refactorings. Surprise “one-off” bugs in production due to a sentinel None happen at least once a week. All of this has contributed in limiting our technical velocity as the business scales.
After some experimentation, we’ve started using Haskell for a new key area where we want to grow, and our results are simply stellar. This area plays into Haskell’s strengths:
- Refactoring is a breeze, which has made us bolder. This is a huge plus for a startup;
- Parallel performance is unmatched, especially given that I put very little effort in it. We’ve been able to tackle problems at a scale we would never have touched before;
- The single-threaded performance bump has allowed us to re-wire our brain and express ourselves better, instead of trying to fit everything in a
numpyandpandascontext; - An architecture based on monad transformers has made it possible to run the same code in separate contexts (e.g. experimentation vs production) in ways which has been a pain in our previous Python efforts;
- No sentinel
Nonehas crashed our system yet, which I’m very happy about
Sure we bumped against a nasty memory leak once, but to be fair I was being silly (import Data.Map as Strict instead of import Data.Map.Strict as Strict).
Overall I would say that there are many good reasons for a start-up to use Haskell, as long as the tradeoffs match the use-case.
34 Likes
Re: monads:
I would say it’s more the problem of the Haskell community:
https://tomasp.net/academic/papers/monads/monads-programming.pdf softly accuses Haskellers of gatekeeping with monads by calling it a “rite of passage”, and a different, similarly negative interpretation, is that our inability to explain monads quickly and clearly implies that we don’t really understand what monads are (“to really get a concept, you have to be able to teach it”).
On the other hand, there are multiple production Haskellers who claim they can explain monads to a reasonably competent programmer in 15 minutes, usually via the typeclass definition, and I gave it a whirl to explain monads are to a Clojurian (have to think in types, methods defined by types and laws, interfaces, useful to provide a standard interface to a diverse quantity of types, which in other languages would require language primitives, see Rust, provides the basis for an effect system, provides the basis for a standard eDSL interface, using Blaze and parser combinators as examples, what “a monoid in the category of endofunctors” means, and in their context, tradeoffs of monadic eDSL vs macro eDSL).
TBH I do have to wonder whether Hasura is actually ditching Haskell. I’ve done a bunch of searches, but I can find no evidence that their version 3 will be in Rust. It makes sense for them to bring it into Rust; they are providing server infrastructure and most Haskell-capable developers are potentially Rust-capable developers, and being ultra-high performance is good for their competitiveness.
Where can I find actual confirmation that Hasura is ditching Haskell?
4 Likes
and being ultra-high performance is good for their competitiveness
I think for ultra-high performance, C++ is still the gold standard, no matter the safety features rust has. And there is a large market for C++ engineers, especially since the layoffs.
So, kinda odd.
Maybe it has more to do with the fact that both languages have an academic vibe, that you might convince your Haskell devs to switch (no re-hiring needed) and that it solves some scaling issues they couldn’t figure out how to solve easily with Haskell.
1 Like
Surely both languages would have comparable performance?
4 Likes
A big question I do want to ask, though, is why do Haskell start-ups fail?
Start-ups in general fail, but the lack of successful pure Haskell start-ups serving outside communities >(i.e, not FPComplete, not DigitallyInduced) is pretty alarming, considering that Haskell has much of >what’s needed to succeed.
Contrary to public opinion, there are loads of successful Haskell startups that don’t make too much noise about it. My favorite among them, Co-star doesn’t even look like your typical Haskell shop. They have a small team ( 2 - 3 Haskell devs, though you could argue Pavan Rikhi is worth an extra 3) and over ~7m active users.
6 Likes
The key quote is in:
Introducing Hasura version 3, Rahul Agarwal, at 6:30-7:00
New Engine Architecture
New Rust Based Codebase
- Refactored to use Rust
- Leverage best in class server technology
- Better baseline performance than V2
- Helps engage with the open source community, given that Rust has such a large and growing ecosystem
So Hasura rewrote the core system in Rust, which is a strategic decision that requires a big investment. Hasura is betting that Rust has a brighter future than Haskell, and they put real money on their bet.
11 Likes
I just think that for the application chosen, Rust is a better option, since they need performance.
I suspect that their mistake was trying to go with the junior dev Haskell route, but that’s actually a good experiment for the rest of us. What they showed was that they didn’t have the experienced engineers on staff to guarantee performance in time and space, and by the time they realized the problem, their codebase was likely too big and too filled with junior code that a rewrite, in this case, in Rust, was the correct solution.
On the other hand, let’s say they were building their application in Rust in the first place. Would it have been able to take off? Possibly, but they’d have been slower to get to the starting line since Haskell is more expressive and the FP paradigm is good for prototyping.
I hope Hasura will at least let us have the courtesy of suggesting to us that Haskell was the best choice at their initial stage of development, but between moving to linear types and going with Rust, Rust was the more mature option and more performant to boot.
4 Likes
I think there are a least a few good haskellers there. I am pretty sure Alexis King worked in Hasura for sometime, for example… Also, the code in github doesn’t look “noob” at all. So I don’t think this is the reason.
My guess is that there is a combo of “Rust fits our performance needs better” + “devs are eagger to work with rust”.
They have contribute a lot to the Haskell ecosystem (with ghc-debug). So that’s something we should be gratefull. It would be very good to know if some parts of the Haskell code base remains, and a little bit of ups and downs of using Haskell.
5 Likes
- Total funding $136.5M per Crunchbase
- Invested in the Haskell OSS ecosystem
- Hired and trained many people, thus creating more Haskellers in the world (great use of some of that dosh)
- Built a pretty popular bit of infrastructure (“10B+ API calls served / day”)
Sounds like a Haskell success story to me!
Again I want to reiterate - we as a community really shouldn’t over-fixate on corporate decisions as a form of validation for Haskell. The language is well beyond the point of “prove it” imo - I can confidently say I could build most things in Haskell, for instance.
Programming language choice really isn’t a key factor for business success. And PL choice is not purely technical. There are many less-than-technical reasons to pick a PL. Morale, attrition, fun, culture, ecosystem, office politics, hiring concerns, etc. And even if every Haskell and ex-Haskell company told you their 5 Whys behind their Haskell decisions, you can’t really take it at face value. Companies aren’t oracles who make optimized decisions. They’re messy webs of people usually primarily motivated in part by financial concerns.
10 Likes
I also underrated their revenues; I took the casual search of a Haskell basher at face value, Hasura is around 300 million in revenues via Zoominfo: https://www.zoominfo.com/c/hasura-inc/404445228
@Lsmor: What I heard was that they started out by hiring Indian IIT grads (which, note multiple, tend to be more selective than MIT, but not well-ranked globally) with no Haskell experience, then transitioned off toward hiring senior devs.
The code we see on Github is not necessarily all of their code, and it’s possible they have huge balls of mud hidden in their servers that a combination of Simon Marlow, Alexis King, and SPJ wouldn’t be able to fix.
But I agree with you on the positives of Rust. One of the big deals of Rust, is that while there are over 2 million Rustacaeans, there’s also historically a dearth of Rust jobs. I think in 2021, at least, the Haskell job market actually looked better than the Rust job market. Combine that with Rust ranking #1 in developer happiness, when Haskell is only middling now, and you have a huge arbitrage opportunity.
@Elorm
Well, Co-star, as an astrology firm, is likely to throw the ETO > IOHK crowd into even greater paroxysms.
There’s also Obsidian Systems (around 15m revenues), which is interesting to me because from what I know of them, their main selling point is pure prototyping speed; i.e, between their FRP libraries (Obelisk, Reflex) and Haskell’s safety, they can get applications built very quickly, and with Haskell’s safety and correctness guarantees to boot.
4 Likes
It’s not just about the language. It never is.
It’s about ecosystem, community knowledge, resources, reliability of performance, etc.
C++ has been used for these things for decades. Rust is nowhere near that. Yes, you can probably do similar things, but you’ll likely end up ffi’ing into C/C++ libs or writing your own wrapper libs (you can do that with Haskell anyway).
Some time ago I also remember talks that the generated llvm IR is inefficient and has a lot of unnecessary allocations. But that might have been fixed.
1 Like
…perhaps because it was the least-worst option available - if it arrives as scheduled, the new GCC Rust front-section could change that in a big way.
Having said that…this isn’t the first attempt at a second Rust implementation. For Hasura to make their announcement now, before Rust is fully supported in GCC, is a rather speculative decision - if the current attempt at a GCC implementation also failed, they will be moving from one single-implementation language to another.
instead of remarks like e.g:
New All-Rust Codebase
- Full Rewrite in Rust
…tend to suggest they haven’t switched entirely over to Haskell. Did Hasura give an estimated completion-date for this migration?
Thanks for saving us from sheer speculations.
Still, some of these are important to us Haskellers, e.g. ecosystem and hiring concerns. How can I start a Haskell project at my company if I can’t guarantee them a sufficiently large ecosystem and sufficient applicants? And if these aren’t given, then there is something that we as a community can improve on.
3 Likes