Commercial Haskell should go after Python / Julia, not Rust

1

I’ve been reading up a lot on Haskell production, as well as translating some Python toy scripts into idiomatic Haskell, and it’s provided some ideas as to how Haskell can find more commercial adoption.

Here’s the big problem with Haskell in an age of Rust. Rust, while more verbose and significantly more imperative than Haskell, is more performant and provides more memory / space safety. While Haskell can be tweaked and forced to get within 70-80% of C, at least with toy examples, Rust can, first, do so more easily, second, when it comes to high-performance use cases, it is easier to hire or train a Rustacaean than it is to hire or train a Haskeller.

Haskell has tried to compete via its type system, but type-level Haskell is, first, arcane, and second, first-class dependent types look as though they’ll be very late, with rae having admitted that GHC needs substantial refactoring before dependent types can be added.

We are seeing Haskell losing in competition with Rust as a consequence; Meta, while still supporting Haskell Foundation, is more supportive to Rust (it is an approved language for new projects), Tsuru Capital moved from Haskell to Rust, and we are all familiar with Hasura’s migration from Haskell to Rust.

On the other hand, going after Python’s use cases is closer to seal-clubbing. Going after Julia’s use cases is not so one-sided, with Julia winning in terms of pure performance, but Haskell winning in maturity and reliability of ecosystem (Julia is going to be notorious for buggy libraries for quite some time), however, Julia in expressivity is a downgrade from Python, whereas Haskell, excepting the ecosystem problem, is almost a pure upgrade.

In terms of performance, Haskell knocks the socks off Python; even the slightly worn Haskell server libraries promise at least a 3x performance improvement over comparable Haskell.

When it comes to code correctness, the Pythonistas themselves are moving to gradual typing (with type-hinting, or static typing, being idiomatic for Python code), but their type system, as a bolt-on, is not going to be as expressive or powerful as Haskell’s.

In terms of memory safety, Haskell’s propensity to space leak at scale is probably equal to Python’s propensity to leak at scale, but we’ve known of this problem for a long time and have been developing workarounds, and the nuclear -XStrict option is still available.

Any time you go into more complicated data processing code, Haskell’s FP-optimized syntax is much more succinct and understandable than any imperative Python implementation, and even “functional” Python code loses out easily because Python is not designed to be a functional language and has no spare performance for paying the lambda tax.

The problem with this, of course, is that Haskell’s library ecosystem is still quite problematic; there is a survivorship bias where every time Haskell’s ecosystem is found wanting, and the persons involved lack the skill to write their own library, they cease to be Haskellers, so Haskell ends up being about library writers and people working within the cases where the Haskell ecosystem is satisfactory.

A further problem involved is marketing. Haskell, of course, is a “pure” (in the technical, not vernacular sense, by being referentially transparent with the overlap of call-by-value, call-by-name, and call-by-need semantics) functional programming language. It is foreboding to the average Pythonista (anecdotally, a few friends checked out the Python and Javascript Discords, and the average member of either community did not know what a pure function was), but when you trial monadic Haskell code to imperative programmers with zero Haskell experience, a common response is “I can’t believe it’s not Python”.

Historically, we’ve had the fake sieve and fake quicksort scandals. For pushing production use, it’s not that morally different from hiding Haskell’s fundamentally functional nature, limiting newbies to an eDSL (Gabriella Gonzalez suggests that Haskell’s killer app is as an eDSL builder in https://www.youtube.com/watch?v=fNpsgTIpODA ), then gradually explaining how to do computation functionally, building and extending the eDSL themselves.

This basically just amounts to how to sneak Haskell into the workplace; i.e, it is hard to sell based on effect systems and type-level, considering that the average IT manager likely has no idea what they are and will be hostile to those features for that reason, but it is easy to sell Haskell as “Python that’s fast, concurrent, and scales way better to medium-sized projects” when the Haskell ecosystem is suitable.

The only caveat is, of course, you need experienced Haskellers to protect the junior and newbie Haskellers from space leaks, IO monad abuse wrecking the code architecture, lazy IO giving you single-threaded concurrency bugs, but that’s sort of a win, isn’t it? It means that the population of experienced Haskellers becomes indispensable to make Haskell projects actually work.

And if this sounds vaguely Faustian, well, it’s about a way to get a starting point. The goal is that, if the Haskell projects actually take off within your organization, the production idiom aims at eventually upgrading to Simple Haskell with TyDD. Once that’s achieved, depending on the scale and the skill level of the developer base, type-level could be deployed in future projects, but that assumes that the developers involved will get to that skill level, and realistically, by the time you get to that point, ergonomic dependent types will probably be available and you’d rather go with dependent types rather than older Haskell type-level programming.

Ultimately, it just comes out to, have you ever heard of a Rust project being rewritten in Haskell? Even C++ will end up getting wrapped in Haskell, as with Meta’s Sigma and Standard Chartered’s Mu project (strictly a dialect of Haskell). Meanwhile, while the sample size is small, I’ve never heard of a Haskell project being rewritten in Python, simply because while Rust vs Haskell is a difficult challenge, Haskell vs Python, as long as the ecosystem is there, is seal-clubbing.

Key.me’s successful Python to Haskell transition:

9 Likes
2

Marketing based on comparing yourself with other brands is a poor strategy.

I’m not usually one to link to CEO speeches, but this one struck me: Best marketing strategy ever! Steve Jobs Think different / Crazy ones speech (with real subtitles) - YouTube

Never did apple tell you in one of their ads/campaigns why they’re better than Microsoft. Never did Nike talk how they’re better than Adidas. Instead, they sell an idea, a vision.

As for programming languages, I think there are two ideas that are worth to market and Haskell can do those:

  • for the engineer: the conflict between expressivity and confidence
  • for the manager: the conflict of aggressive iteration and keeping technical debt low

Engineers love high expressivity, but they will soon regret it if they lose confidence in the things they built. This happens with languages like C++ quickly, in my opinion. Soon you’ll need very disciplined engineers, so your project doesn’t derail and becomes unmaintainable.

Managers love the idea of iteration (agile is all over the industry) and yet they’re scared of technical debt for long-running projects as it’s slowly regressing productivity, burning out employees and requiring a rewrite down the line.

Haskell has a unique stand in both conflicts, IME.

22 Likes
3

Thanks for deigning to respond, hasufell.

First, I think comparison is useful as a way to identify potential use cases. Rust will always beat Haskell in certain fields (systems programming, for instance) because of its core values, but in fields where it should be more of a toss-up, Haskell’s advantages (expressivity, ease of refactoring, correctness) can’t cancel out the deeper Rustacaean developer base and depth of ecossytem.

However, for Pythonic or Julian use cases, the relative strength of Haskell is such that, barring issues with the ecosystem, Haskell’s greater performance, correctness, and expressivity cancels out the relative dearth of Haskell developers.

Second, the point of comparing Haskell to Python is more that, Haskell currently has a reputation for being extremely difficult to learn. If, say, you want to understand all the language extensions, all the possible design patterns and architectures in Haskell, this is 100% true.

However, if we’re talking about the minimum dialect of Haskell that’s useful and idiomatic (i.e, does not have IO eating up the entire program, does not abuse mutable references, has proper division between effect-free and effectful code), it is not that much more challenging than Python.

Python is notorious for just having libraries and library functions for everything. Ironically, this maps pretty well to the Haskell idea of pure-impure separation; the functional pattern of having effectful code that calls effect-free code is pretty much just Python “just use a library function” except that you are writing the libraries yourself.

If the ecosystem supports the use case, and there are competent Haskellers on the team, for a junior developer, the Python to Haskell transition does not need to be painful.

And in reality, there’s like multiple blog posts already talking about individuals transitioning from Python to Haskell:

How to read Haskell like Python : ezyang’s blog ← Thanks for backpack!

Haskell for Python Programmers. Python is a functional programming… | by Noah Hradek | Medium <— the assertion in the first few paragraphs are technically wrong, but the point is that Python has declarativeness as a goal, and Haskell’s pure FP is a subset of declarative programming, so the confusion should at least be understandable.

4

@Liamzy Your mention of Python (ultra-popular) and Julia (pretty niche) makes me think of the following: the target user of both languages is not a formally-trained software engineers.

The following observation is limited to my experience, i.e. physical sciences and finance, so take this with a grain of salt.

The reason I didn’t keep using Julia during my graduate studies (like many of my fellow students) was that there was too much friction to get started and keep going. This includes a lack of appropriate learning material aimed at non-software people and language-specific issues (in the case of Julia, time-to-first-plot).

As the demand for software knowledge has increased – and the supply of formally-trained software engineers is somewhat constrained – many software developers aren’t knowledgeable about the existence of many technologies, let alone their pros and cons. As an example, at my work, <15% of developers are software engineers, who presumably have had a class on type systems. The others are smart cookies who are no less effective, but have little-to-no knowledge of type systems / mathematical purity / algorithmic complexity.

Therefore, adoption of a technology goes by the path of least resistance. In this optic, Julia hasn’t reached great penetration because of a thousand paper cuts.

In order to increase industrial Haskell adoption, we need to remove friction from the perspective of non-formally-trained software developers. The breadth of the ecosystem is a pain point, but there are easier wins:

  • Having good tooling (which has improved so much recently!);
  • Appropriate learning material (I’m a big fan of ‘Haskell Programming from First Principles’ and ‘Production Haskell’, but buying a book requires some level of motivation);
  • A focus on documentation, including API documentation but also tutorials and examples;
8 Likes
6

I’m sorry for the selective quoting, but I don’t think I’m taking you out of context here.

I see a discrepancy between these two statements:

From my perspective, monads are an extremely useful way to structure a wide variety of APIs. Abstracting over arbitrary monads is good not only because it’s fun and enjoyable, but also because it enables a great deal of reliable code re-use. Programming languages that don’t make it convenient to abstract over arbitrary monads are missing a proven, useful tool. The slow spread of the concept to a variety of other programming languages is evidence that it’s not just for programming language enthusiasts. I think your post implies otherwise, though, and that really doesn’t match my experience.

4 Likes
7

Re: LaurentRDC: I’m worried about how well the Haskell community might take an influx of developers with less formal software developer training, but I think both yours and maxigit’s post has brought up interesting perspectives.

First, Haskell is unusually valuable to outsider developers, because learning Haskell, to an extent, is trying to pick up parts of a formal computer science education, given that many pre-existing Haskellers are very experienced, knowledgeable, and skilled software developers and engaging with Haskell is engaging the culture.

Second, I think of this blog post:

Outsider developers are often competitive in software engineering because they are domain experts and understand a necessary domain better than formally-trained developers, who know how to develop software but not necessarily the domain the software is being developed for.

Incidentally, these domains are often mathematical, and Haskell’s mathematical culture matches better to these non-formally trained developers than things like SOLID or Agile. A notorious meme concerning Haskell is the OOP-trained developer, who is often very experienced, rejecting Haskell because their domain knowledge concerning imperative and OOP programming doesn’t translate well to a functional programming paradigm. This issue doesn’t apply to domain experts, who don’t know SOLID or agile in the first place. And for a non-formally trained developer, Haskell helps them address imposter syndrome, fill lacuna in their software development knowledge, and gain respect within the software development community.

Likewise, for formally-trained software developers working in shops with lots of domain experts, Haskell’s tendency toward “pit of success” design provides benefits in that the domain experts, without software engineering training, are more likely to use proper software engineering practices.

That said, yeah, documentation is written by and for experienced Haskellers. Consider Network.HTTP.Simple for instance. You and I can read the documentation and figure out how to use the “simple” library, but it’d take a substantial amount of time, or at least dedicated instruction, for the average Haskell newbie to figure out how to get access to the body of an HTTP response.

“Why can’t I just strip the IO type off? The example isn’t in do notation! What the hell is a ByteString?” What does the Response a → a type of getResponse body mean? What’s the >>= operator?"

Another example might be Wreq, whose documentation uses lenses.

https://hackage.haskell.org/package/wreq-0.5.4.0/docs/Network-Wreq.html

1 Like
8

“Late” is good. The alternative is “never”. The entire initiative hangs by a thread, as there aren’t many people working on adding dependent types to GHC.

I’m rather lucky to be able to work on Dependent Haskell at Serokell, but if I get hit by a bus or if the funding runs out, I can only hope that someone else will pick up where I left off.

14 Likes
10

I’m more oriented on breaking down resistance to Haskell; i.e, there can be dialects of Haskell that resemble what people are used to, without alienating industrial users by telling them they have to complete a master’s degree in functional programming (which, of course, doesn’t exist) before they can start using Haskell.

The idea is more slow infiltration; i.e, start with main = do, wire together library functions, then learn how an FP language can achieve the same things an imperative language can, as opposed to throwing HaskellBook at someone and hoping it sticks.

Given the average knowledge of industrial users (Haskell has been attacking at the high-end, but Rust is starting to block that off), showing Haskell that the average developer or IT manager can understand is useful.

Re: @hasufell: I agree with your points concerning how Haskell can be branded. I just disagree strongly with the notion of introducing and pushing Haskell through its most cutting edge and experimental features; i.e, the average developer or IT manager would feel professionally threatened by what is really a completely different paradigm, rendering much of their existing knowledge useless, and infiltrating Haskell by its similarity to what they already know would be more successful.

1 Like
11

Are you sure pythonistas are willing to move to Haskell (or any other static typed lang)?. I’ve been a python developer (exclusively in the field of data science) for 8 years, and whenever I proposed to my colleghes to use a different language (commonly scala's spark) or to use python’s type hints I’ve got the response: “I prefer to quit than using types”

Moreover, not just one, but two managers have told me “stop proposing typed language” (they did kindly, written down seems more dramatic that the actual conversation). The simple reason: most developers I work with do only know python. They are completely unaware that languages with types exists, and when they see type hints in a python code, the tend to frustrate because they add verbosity to the language.

Notice that me background is exclusively data science/data engineering, and my sample of coworkers are phicisits, mathematicians, electric engineers, etc… probably just 5% of my coworkers have had a degree in CS. So probably I am biased, but I’d say pretty much 0 pythonistas would like work with Haskell

6 Likes
13

I’ve noticed the same thing. Large amount of resistance to consider anything else but the status quo.

However, I refuse to believe that there are people out there who are so set in their ways that they wouldn’t consider an alternative. Rather, there needs to be a very high upside for some people.

I can’t help with data scientists, quants and data engineers, but here’s something for managers.

I work in proprietary trading, and I’ve been tasked with assessing the cost of production bugs. Every bug in production costs two things:

  • A loss in case we’ve traded the wrong thing;
  • An opportunity cost; the loss of the money we would have made, had the bug not happened.

We have a failure rate that’s pretty low. I can’t give the numbers exactly, but think in units of bugs / month. And yet, in the past year, just in terms of Python’s TypeError and related errors, i.e. things that static typing would have caught at compile-time, we could have saved something on the order of person-years. Assuming that new bugs pop up as new ones get squashed – which is likely given that we keep expanding to new products and markets --, there is a tangible financial incentive to consider static typing. In fact, we have been experimenting with Haskell for this purpose, and now we have firm numbers backing our decision.

5 Likes
14

TBH, while this isn’t proper TyDD, a funny thing you can do with HLS is just write a function or value, and watch the shadowed type signature change. When you are doing something wrong, the types make no sense. When you’ve completed your definition, just click on the code shadow and your type signature shows up.

One of the big advantages of Haskell’s approach to type signatures is that we don’t actually need the type signature in code most of the time. The type inference engine is powerful enough to guess the types most of the time, and an type-sigless style can work, albeit be smelly.

Do your Pythonista coworkers accept docstrings? Knowing what I know of the Python community, I wouldn’t be surprised if there were Pythonistas who hate docstrings, but introducing a type signature, something that strictly exists outside the code and comments existing code, might work.

I hope this isn’t too shocking, and I hope no one actually ships code into production without type signatures, but as a way of getting crap under the radar, the ability to avoid type signatures 95-99% under the time IS a way to sell strict static typing to dynamic typing addicts. Gradually, they’ll discover that it’s better to have a type signature most of the time, just to get the compiler to give better errors on what went wrong, just as some Haskellers go so far as to type sig all their where-clause code.

Also, for Python code golfers, one of the interesting parts of Haskell to me is that it is actually quite easy to get idiomatic Haskell code to be so terse as to be unreadable. We literally introduce garbage and unnecessary names just so that someone reading the code knows what the hell is going on. That’s a sellable feature.

15

I wouldn’t be surprised if there were Pythonistas who hate docstrings, but introducing a type signature, something that strictly exists outside the code and comments existing code, might work.

If we’re being pedantic (pydantic?) “type signatures” (I think the normal nomenclature in Python is “type hints”) are available at run time in Python, and are able to be used to determine behaviour. I have worked with Python programmers who never run a type checker like mypy over their code, but write types, just so they can use libraries like pydantic or fastapi.

1 Like
16

No type inference, way less useful typing errors, way more cryptic messages, I think during these 20 years there has been great progress in static typing and compilers (and not only in Haskell/FP).

1 Like
17

Haskell has type sigs. We use them all the time because they’re idiomatic, but your code will often run if you leave them off. This idiom mostly happens with newbies and people who are learning Haskell, because taking the type signature off makes Haskell a bit simpler to read and write.

18

I think the main problem comparing with Python is that Python has many target users. For example, Python is a great scripting language, and a great language for prototyping. The thing I love about Haskell is it forces you to think about the types you actually want and handle all the cases, but that’s not always what you want. If I want a quick script that’s mostly portable, fairly small/limited in scope, and I’m fine with making assumptions, Python is perfect.

If we’re talking about production systems, etc etc, I’m 100% on board with evangelizing Haskell more. But we should recognize that there are valid use cases for keeping things in Python, even as a die-hard Haskell enthusiast as myself

3 Likes
19

I just wanted to both expose a class of users you may not have been aware of, as well as to clarify that type hints in Python are very much things that are used at run time (which seemed at odds with the statement “strictly exists outside the code”), in contrast to languages like Typescript or Haskell (after typeclass dictionaries have been inserted) where they are erased.

Taking the type signatures off often makes Haskell a lot harder to read IMO, but that’s a discussion for another time. However, even ignoring ambiguity and all the language extensions that interfere with inference, there are cases where they are genuinely mandatory, for example polymorphic recursion.

data Splitting a = Val a | Split (Splitting (a,a)) deriving Show
collect :: Splitting a -> [a] -- See what happens when you remove this
collect (Val x) = [x]
collect (Split x) = concatMap (\(a,b) -> [a,b]) (collect x)
2 Likes
20

Why do you … “want” more commercial adoption ?

  • As maxigit said you might be interested in the langage per se/for language research
  • Would you accept waiting for more adoption instead of actively seeking it ?

IIRC : In this talk Uncle Bob says “Memory is cheap, dirt cheap - everyone has in its pocket a thumb drive or a phone with an sdcard orders of magnitude larger than our wildest dream in 2000 … the real next fight of moore’s law is multicore programming … and Haskell (and FP) shines at it”

21

Good informative post, thanks !

A small typo.

Did you mean to write Haskell there ? I didn’t quite understand the point being made.

22

Thanks for noticing.

What I mean by the latter is that Haskell can displace C++ by wrapping C++; i.e, instead of the interface being written in C++ for Sigma and SC’s Mu codebase, it’s written in Haskell instead. It is not wholly replacing C++; Haskell does not have the high and predictable performance needed for that, but it is replacing part of a C++ system.

You can consider this, to an extent, being Haskell replacing C++ by having a powerful type system that can increase code correctness and expressivity.

In contrast, Haskell is not sufficiently better than Rust for these purposes that it’s worthwhile building a Haskell interface for a Rust application.

@satoshi778

I have been criticized in the past for wanting too much Haskell adoption, and I think people new to Haskell tend to go through such a phase, where they’re incredibly besotted with Haskell and would like everyone to use it.

However, given current trends, encouraging Haskell adoption is somewhat similar to simply wanting Haskell adoption to stay at current levels, since there are Haskell users dropping the language for Rust.

The thesis of this thread is that Haskell has a hard time competing with Rust right now because for high-performance applications, Rust is has more and more predictable performance, while having a larger ecosystem, a larger pool of developers, and being easier to train for.

However, performance and safety is not the only benefit of Haskell’s nature as a pure, strongly and expressively typed language; Haskell is also capable of being extremely terse and being, for a developer trained in the FP paradigm, easy to write.

That is to say, Haskell is suitable to attack Python’s use cases, and in doing so, maintain its social and library ecosystem. Haskell isn’t the best qualified for this purpose; half of Python’s strength is its extensive and easy-to-use ecosystem, which Haskell lacks. However, I think Haskell still has sufficient advantages in performance, safety, and expressiveness to compete with Python, and moreover, trying to compete with Python will ultimately come down to whether or not Haskell has a competitive ecosystem that is both broad and easy to use.

I would say that if Python’s use cases get targeted, the side effect will be a broader and easier-to-use Haskell ecosystem, which is a net gain.

@brandonchinn178

As far as scripting goes, I wouldn’t consider Haskell optimal because Haskell is typically compiled (interpreter without O2 tends to perform relatively badly, although generally it’ll outperform Python substantially), and because Haskell’s syntax requires names wrapping your script, whereas OCaml and Python can execute commands on the top-level. However, Haskell’s terseness makes it competitive enough, and the scripting ecosystem, while not ideal, seems sufficient. If you absolutely demand top level scripts, script :: IO (); script = {…;} using the brackets and semicolon to override significant whitespace and write everything on the first column.

For prototyping, I think most are aware of the old Aegis code prototype study that Hudak (RIP) published? https://www.cs.yale.edu/publications/techreports/tr1049.pdf

You can use Ada as a benchmark in this regard; Haskell took less than half the time of the equivalent Ada code to prototype (C++ never completed), and around 3 times that of a logic-programming Lisp, while creating code with greater maturity than either.

It’s just that, while, apparently, there was interest in Haskell as a prototyping language in the 90s and 00s, Haskell didn’t evolve into that direction, and I’d consider the standard TyDD idiom neither excellent nor terrible for prototyping (write your types, build up a program as undefined, then fill in the blanks is pretty competitive, but it’s easy to get the types wrong when you’re doing exploratory programming) . On inquiries in /r/Haskell, some Haskellers seem to have obtained a level of success with Haskell prototyping, but the advice seems contrary to standard TyDD.

While I’d consider Haskell as a scripting language relatively explored (it’s easy to try, there are libraries for it), the prototyping direction is interesting precisely because while some Haskellers have had success, many Haskellers regard Haskell as unsuited for such. That suggests that the Haskell prototyping space is ripe for exploration and development.

1 Like
23

As a small, final comment, I talk to a Clojurian who hates Haskell and I like our Socratic conversations as a way to refine ideas and get out of the Haskell bubble.

He’s a staunch dynamic typing advocate, and while he’ll use type hinting and mypy in Python, he’s viewing it as for “inferior” programmers as we’ve agreed that he’s a professional “s-code” cleaner, and mypy gets bugs out of his codebases quite well.

The effective question we’ve been discussing is whether or not there’s a space for purely type-inferred code in Haskell; of course, as Probie has shown, there are cases where even Haskell98 / Haskell2010 type inference will fail.

Considering prototyping, I think there is.

Put another way, every red-blooded Haskeller is pretty addicted to having type signatures to their code and can list 50 different reasons skipping top-level type signatures is a terrible idea.

However, when it comes to prototyping, getting your types wrong is a real risk, and writing in a type-inferred idiom is useful. There’s obviously readability tradeoffs, but there’s a fairly easy way to handle this; put the types in comments.

This has two benefits.

First, the absence of top-level type signatures is an obvious “DO NOT SHIP” warning signal for prototype code; it makes prototype code distinct.

Second, when you’re trying to get dynamic typing advocates to embrace the advantages of static typing, it helps for them to think of type signatures as comments; my point about docstrings is that while you can have a substantial community of Pythonistas who hate static typing, the population of people who object to docstrings is much, much smaller. Having them think of type signatures as comments to improve the readability of code is a useful starting point to have them experience the advantages of modern static typing over dynamic typing as well as more antiquated forms of static typing.

It also helps because all their code, as they adapt to Haskell, will start off as prototype code anyways. Their code is unlikely to pass code review, and moving into a “no type sigs, commented out type sigs mean prototype code” idiom is both useful for indicating what should not hit production, as well as segueing them into a statically-typed idiom.