Why use an effect system?

[atravers]

Why don’t you all go and build something with an effect system, and come back with actual production insights?

I think I’ve heard this before…yes, that’s what it was:

Unfortunately, no one can be told what the Matrix is. You have to see it for yourself.

Morpheus

• I see… a decades-old pop culture reference. Disappointing. By the way, Morpheus could have just said "The Matrix is a machine-made virtual world that your real body, currenly residing in a vat full of snot, has been plugged into and receiving stimuli from all your life."

  Right. There’s no need to get overly mystical, when good explanations are all that more satisfactory.

  Continuations FAQ (epoq diary 013) :: Strangely Consistent

…can we just wait until the GHC developers choose one, so it can be deemed as recommended workable?

[tomjaguarpaw]

The second one is tightly coupled to the effect system. It’s impossible to convert to a pure IO action and pass to a non-Eff subsystem … It’s only reusable within Eff framework and unusable outside. Doesn’t look too modular.

This is not true. There is no coupling to the effect system (assuming we’re talking about Bluefin). To see this, note that this function is safe:

tag :: Untagged.Connection -> Tagged.Connection e

so you can define the former write in terms of the latter:

untaggedWrite :: Connection -> Foo -> IO ()
untaggedWrite conn foo = runEff $ \io ->
  taggedWrite (tag conn) foo

@vshabanov

I said a similar thing before: Haskell lacks a body of industrial usage wisdom .

Agreed with this. It would be good to establish one.

@vshabanov

Effect systems are highly experimental, most of them are not production ready and/or have very serious flaws.

I disagree with this. Effect systems as a whole are an extremely well understood area of the Haskell world. They’re so well known, in fact, that their significant weaknesses, and potential approaches to ameliorate them, is extensively covered and recovered ground.

However, until the progression from ReaderT IO to effectful that I covered in my talk there had been no approach developed that resolved all the significant weaknesses. effectful does address all the issues of existing effect systems (with one exception: it doesn’t directly support multi-shot continuations – that’s probably fine!) and we know it doesn’t have any additional weaknesses of its own relative to IO because it is just IO. Bluefin inherits these properties.

I suppose one could argue: “but the additional interface that effectful and Bluefin put on IO is too complex”. But firstly, IOE :> es => Eff es a is (almost) just IO a, so you’re never far from the lowest common denominator, and secondly, I don’t believe there is a simpler way to carve out effects from IO. Can you think of one? If not then that’s evidence that carving out effects requires a certain level of additional complexity. If you don’t want that complexity then so be it, but that’s not the same as a proof that effectful and Bluefin are experimental or flawed.

@vshabanov

Yet somehow they became a necessary pre-requisite for writing basic programs. They’re not.

If a new experimental thing appears, it’s probably better to advertise it as “look, what an interesting way to program” than “this is the future of Haskell”

I don’t recall anyone saying effect systems were a pre-requisite for writing basic programs. Can you point out such a claim?

Regarding “this is the future of Haskell”, perhaps you’re referring to my slide “Bluefin is the future of Haskell!” at timestamp 40s of my talk. To be clear, that is not advertising. That is simply my belief. The point of the talk was to justify that belief based on properties of Bluefin. To summarise why that is my belief:

To justify effect systems per se: I believe it is useful in practical programs to “make invalid operations unrepresentable”, i.e. use types to tightly delimit what externally-visible effects a function can perform. This must include, at minimum, state, exceptions and I/O, and it must do so in a composable manner.

To justify IO-based effect systems: it is essential for practical programming that an effect system provide resource safety and easy reasoning about behaviour. I don’t believe this is possible outside IO-based effect systems.

To justify Bluefin (i.e. value level effect arguments) versus effectful (i.e. type level effect arguments), I think it’s simpler and more approachable. (I wouldn’t really be surprised or disappointed if effectful won out over Bluefin. It’s a matter of taste and I’m happy to let the market decide. But the Haskell ecosystem really does IO-based effect systems to displace all others, and I think that’s inevitable.)

EDIT:

@vshabanov

any data structure or function that uses Connection e becomes polymorphic despite no real polymorphism is involved. It’s only reusable within Eff framework and unusable outside. Doesn’t look too modular.

Do you also feel the same way about ST, which has the same “polymorphic” property?

[tomjaguarpaw]

@vshabanov

Agreed. The more pure code and the less IO the better.

If one wants better control, it should be a discreet decision, not a jump on a bandwagon.

That’s good news for effectful and Bluefin then, because you can use them in internal components without effecting external APIs, which can either be pure (if the internal components didn’t use IO) or use IO (if they did).

@vshabanov

the most obvious option – providing a pure `IO’ interface – is missing. This would be the easiest for library users to integrate into whatever framework they’re using.

I agree. Bluefin and effectful can do this, no problem.

@BurningWitness

Regarding lack of modularity in effects libraries, that’s pretty much how all of Haskell’s ecosystem works: instead of providing the minimal tools and letting users stitch things together, it’s instead expected that you use one of fifteen convenient runner functions and for anything beyond that you have to dive into non-PVP-compliant internals.

This is completely false for Bluefin and effectful because they are just IO wrappers: they are minimal and you can always just unwrap to IO.

[jeukshi]

@vshabanov

It’s only reusable within Eff framework and unusable outside.

I agree, but not everything has to be reusable outside the context it is important in. I mentioned codebase, but that is not good enough. I’d draw a distinction between application/library code. Then I’d agree that for library code (API to be precise), Eff is not it. But for application code, you don’t care for unusable outside.

@vshabanov

I re-read my points and can’t find which of them apply to a record of functions.

Noisy types, leaks details, hides details, unmodular, Java factory, infects code. I think those could arguably work. I don’t know if it is worth discussing, though. Just wanted to point out that Bluefin is really close to a record of functions, which is what I like about it.

[atravers]

Because I also can’t remember everything:

The issues with effect systems

Hello, I’ve been rather interested in effect systems for months at this point and I’ve been researching their benefits and issues. I do know that their benefits include: Potentially being faster than traditional MTL (put a pin on this, I’lll come back to it later…) Being easier to write & read (?) They compose nicer than monads (could be a subpoint of 2?) I might’ve missed some other potential benefits with effect systems, so I’d love to hear about them if you have any in mind. But while the…

…to avoid further gratuitous repetition here.

[lortabac]

As someone who doesn’t have a strong opinion on this topic I’d say ultimately it depends on the kind of applications that you write and on personal preference.

However I’d like to point out another aspect, that I don’t see mentioned in this discussion, namely that Haskell is not a “batteries-included” language. If you consider a language like PHP, which has built-in support for HTTP, database clients and whatnot, Haskell is on the other end of the spectrum. It’s probably the only (almost) mainstream language without even built-in booleans!

This is both the strength and the curse of Haskell. It’s a strength because it makes it flexible and general-purpose. It helped the language to remain relevant for more than 30 years despite not being super-popular. It’s a curse because more functionality is delegated to the libraries, so for each feature that other languages would provide as built-in, we have a myriad of competing solutions and libraries. The whole ecosystem is more distributed compared to other programming language communities.

So, just like the lack of proper built-in record manipulation encouraged the development of optics (which ultimately became much more than just record selectors), we can say that effect libraries were also born from the lack of certain built-in features (and they probably provide more than what built-in features could have provided, at the cost of additional complexity/boilerplate).

Part of the discussion here seems to be just the latest reiteration of this same issue. The tension between vanilla Haskell being “too simple” and libraries being “too powerful”. Finding the sweet spot is hard. It may even be a wrong idea, and maybe the decision should be taken on a case-by-case basis.

[vshabanov]

@BurningWitness

In my view the only true benefit to effect systems is that it’s the only sane way of getting … a mirror test system that behaves exactly the same, but calls test versions of all real-world things it links to.

There are many ways to achieve mocking without effect systems:

• use original services but with test data
• mock services
• use functions: mockConnection :: IO Connection where Connection has functions inside (Connection {write :: ...} or Connection :: Service a => a -> Connection which is less convenient to use)
• built-in mocking: data Connection = Real Socket | Mock (IORef (Input, Output))

Mocking worked long before effect systems. Ironically, it’s especially easy to mock with first-class functions.

@BurningWitness

Regarding lack of modularity in effects libraries, that’s pretty much how all of Haskell’s ecosystem works: instead of providing the minimal tools and letting users stitch things together, it’s instead expected that you use one of fifteen convenient runner functions and for anything beyond that you have to dive into non-PVP-compliant internals.

Hm, if I look at base, text, async, network, unix, warp or http-client packages, I usually see pure functions or IO. What parts or ecosystem have these “convenient” runners?

@Kleidukos

I see a lot of speculation here and people speaking past each-other. Why don’t you all go and build something with an effect system, and come back with actual production insights? The conversation would certainly be more productive.

I worked on a moderate sized production project (~150k lines) with effect system, so I do have some experience which I’m sharing here. It wasn’t effectful, but switching to effectful will still keep a lot of problems (though indeed it will improve performance and simplify the implementation of new effects).

@tomjaguarpaw

This is not true. There is no coupling to the effect system (assuming we’re talking about Bluefin). To see this, note that this function is safe:

tag :: Untagged.Connection -> Tagged.Connection e

It will work if you have Untagged.Connection. What if you only have Tagged.Connection e? And that’s what usually happen (at least in my experience).

That’s why write :: ... IO () is modular – you can use it standalone, or embed into an effect system, and write :: Eff ... is not (unless it uses a modular IO underneath).

@tomjaguarpaw

@vshabanov

Effect systems are highly experimental, most of them are not production ready and/or have very serious flaws.

…
However, until the progression from ReaderT IO to effectful … there had been no approach developed that resolved all the significant weaknesses.

I think you confirmed my “most of them are not production ready or have flaws”. Yes, effectful is a pretty good implementation. But there’s already a Bluefin that tries to replace it. A lot of other systems to come.

@tomjaguarpaw

If you don’t want that complexity then so be it, but that’s not the same as a proof that effectful and Bluefin are experimental or flawed.

I think that Bluefin is still in its early stages. I was mostly talking about myriad of other systems.

@tomjaguarpaw

I don’t recall anyone saying effect systems were a pre-requisite for writing basic programs. Can you point out such a claim?

That’s my general feeling. I see far too many discussions about effect systems, I see newbies asking which system to use. It’s becoming a matter of course to use an effect system, whereas it’s an advanced and experimental thing with no consensus on which one to use.

@tomjaguarpaw

Regarding “this is the future of Haskell”, perhaps you’re referring to my slide “Bluefin is the future of Haskell!” at timestamp 40s of my talk.

Yes, I was referring to this. Maybe if it was “the future of Haskell effect systems” I wouldn’t start all this discussion

Yes, for performance considerations and to be able to use exceptions, state and concurrency properly, IO-based system looks inevitable.

Though there are many other use cases (like verification) where interpreting a program might be more convenient.

@tomjaguarpaw

Do you also feel the same way about ST, which has the same “polymorphic” property?

In a sense, yes. But ST is local and has a very strong guarantee – function that uses runST is still pure, I can easily use its pure results everywhere.

Effect systems are usually for managing IO, and there can be a need to connect systems that use Eff and systems that use IO. Here I can have issues connecting the two.

@tomjaguarpaw

That’s good news for effectful and Bluefin then, because you can use them in internal components without effecting external APIs, which can either be pure (if the internal components didn’t use IO) or use IO (if they did).

Hm, much like runST. It may work, but it will lead to a bizzare mixed codebase (one part is IO, another is Eff), and may complicate extracting parts of Eff system to a separate IO libraries.

@jeukshi

Then I’d agree that for library code (API to be precise), Eff is not it. But for application code, you don’t care for unusable outside.

Many libraries are initially part of the application and are extracted later. It’s a sign of good modularity if the application code can be extracted into a library. So it’s worth aiming for IO. Otherwise the Eff code, which could be a future library, would remain coupled to the application.

@jeukshi

@vshabanov

I re-read my points and can’t find which of them apply to a record of functions.

Noisy types, leaks details, hides details, unmodular, Java factory, infects code. I think those could arguably work.

Hm, it’s still an abstract Connection data type, that is not noisy and doesn’t leak much details. Though yes, it might hide details as it’s not clean which functions are used.

@lortabac

so for each feature that other languages would provide as built-in, we have a myriad of competing solutions and libraries.

And it’s great. It makes Haskell a vehicle for innovation, and sometimes we get amazing results.

This is one of the things I don’t like about effect systems – they are often proposed as the one true way to write Haskell.

[BurningWitness]

@vshabanov

There are many ways to achieve mocking without effect systems

• Production services with test data is QA territory;

• Built-in mocking means your declarations and implementations for both the production and the test versions have to live in the same module, whereas they most probably belong to two different executables.

• Mocking downstream services is something I’d much rather have the service do itself via a sandbox, so that’s QA again;

• Passing records of functions makes you responsible for juggling all that as extra arguments instead of constraints. It’ll definitely work, the question is just how much inline visual clutter the team you’re working in is willing to endure.

  Though it’s hard to say just how more convenient effect systems are because there definitely is a problem of sharing overlapping effects between components and I don’t know if anyone has solved it yet.

@vshabanov

What parts or ecosystem have these “convenient” runners?

That part of my reply vaguely gestures at the mentality that produces duplications like this bunch and package after package with no refined constituent pieces, like most web-servers, but as it relates to this discussion specifically I think it’s a non-sequitur. (unless “I think everyone should agree on one effect system” is a point worth discussing)

[jeukshi]

@atravers

Then here’s the challenge for you and everyone else who thinks IO a is now bunglesome and needs diluting:

type IO a = Eff All a

…use your preferred system of effects to provide a Haskell declaration for All .

I think I finally understood this concern. Effect systems (those named in this topic at least) are not about breaking IO into pieces, they are not even about IO at all. They are about built-in/custom effects and restricting functions to specific effects. Of course, there are performance and ergonomics issues too. One of the ergonomic considerations is providing a way to make effects out of the existing code. Which happens to be Haskell code, and a lot of it lives in IO. So those libraries provide an easy way to make effect use IO, to use IO functions.

Just like effectful provides an easy way to integrate mtl code to make effects. Nothing profound about that at all.

[atravers]

@jeukshi

So those libraries provide an easy way to make effect use IO, to use IO functions. […] Nothing profound about that at all.

The latest entry to Philip Wadler’s list of notable quotes:

We are entering an era where someone might use a large language model to generate a document out of a bulleted list, and send it to a person who will use a large language model to condense that document into a bulleted list. Can anyone seriously argue that this is an improvement?

Ted Chiang

Now another era is being foisted upon Haskell, an era where someone might have to generate an I/O action out of a library using fused-effects and provide it to a library using speff. Can anyone seriously argue that this is an improvement?

[tomjaguarpaw]

@atravers

Then here’s the challenge for you and everyone else who thinks IO a is now bunglesome and needs diluting:

type IO a = Eff All a

…use your preferred system of effects to provide a Haskell declaration for All .

I don’t get it. Is this a trick question? In effectful the answer is type All = '[IOE]. Why did you ask the question and what’s the consequence now that you have an answer?

[atravers]

Then IO a can be superseded in a future Haskell standard without breaking every Haskell program in existence. (can we assume that there are no more Haskell 1.0-1.2 programs still in use ? :-)

So for example, future Haskellers could use Eff All ... directly in FFI calls:

foreign ... thisNewEffect :: ... -> Eff All ()

(or use a subset of All).

But as I recall…IOE is connected to the current definition of IO a, so you would need to sever that connection somehow to avoid a vacuous cycle between the two.

[tomjaguarpaw]

@atravers

But as I recall…IOE is connected to the current definition of IO a, so you would need to sever that connection somehow to avoid a vacuous cycle between the two.

Sure, just define effectful's Eff directly in terms of RealWorld rather than IO.

[atravers]

@tomjaguarpaw

Sure, just define effectful's Eff directly in terms of RealWorld rather than IO.

You mean the RealWorld mentioned here?

---

• data State# a :: ZeroBitType
  

  State# is the primitive, unlifted type of states. It has one type parameter, thus State# RealWorld, or State# s, where s is a type variable. The only purpose of the type parameter is to keep different state threads separate. It is represented by nothing at all.

  GHC.Prim (State#)

• type ZeroBitType = TYPE ZeroBitRep
  

  The kind of the empty unboxed tuple type (# #)

  GHC.Types (ZeroBitType)

• type Void# = (# #)
  

  Deprecated: Void# is now an alias for the unboxed tuple (# #)

  GHC.Types (Void#)

---

…how exactly should that be explained to future beginners?

[Bodigrim]

How did we get into discussion of changing the definition of IO (which is obviously not going to happen)? Why would it be relevant to the topic?

[tobz619]

I will preface this that my understanding through reading this thread is that effect systems seem better for doing things with IO; therefore my pure logging endeavours below might not put them in the best light but I’d like to share what I’ve learned so far.

===================================================

I’m currently making Snake in Haskell via Brick as a learning opportunity and decided I wanted to learn about effect systems and see what it was like to combine effect systems together with mtl/transformer-style. To understand 1) Can I make a logging system with a known name? and 2) if I can, how does it work alongside Brick?

Brick’s EventM type is a big ReaderT of stuff and I wanted to do logging on the side but without adding to the overall stack of Monad Transformers.

I choose Bluefin as the one to learn and came up with this for my logging solution.

My initial plan was to have the whole Eff es Eventlist as a record in my overall GameplayState type because then later, I can call runPureEff and extract the Eventlist out of it. I couldn’t get this work by adding Eff e Eventlist to the GameplayState type as then it becomes GameplayState e and I struggled to get logInGame function to be happy when called in my eventHandler function (some type stuff, couldn’t figure it out why it wasn’t happy).

So right now I just build the list at the time of usage.

I thread it into my main program like so:

runLogger :: (forall e. Logger g e -> Eff (e :& es) r) -> g -> Eff es EventList
runLogger f gps =
  execWriter $ \writ -> do
    runReader gps $ \rea -> do
      useImplIn f $ Logger (mapHandle writ) (mapHandle rea)
...
handleMovement :: (e1 :> es) => ([a1] -> Either a2 (K.KeyDispatcher KeyEvent m)) -> BrickEvent n e2 -> Logger GameplayState e1 -> Eff es ()
handleMovement disp ev (Logger writ readstate) = do
  gameplaystate <- ask readstate
  let logaction = getKeyEvent disp altConfig ev
      tick = gameplaystate ^. tickNo
  addToLog writ tick logaction
...
logInGame :: BrickEvent n e2 -> GameplayState -> EventList
logInGame ev gs = runPureEff $ runLogger (handleMovement gameplayDispatcher ev) $ gs
...
eventHandler :: BrickEvent MenuOptions Tick -> EventM MenuOptions GameplayState ()
eventHandler  ev = do
  glf <- use gameLog
  gps <- get
  case gs of
...
    Playing _ -> do
      zoom gameState $ handleGameplayEvent' ev
      gameLog .= (logInGame ev gps <> glf)
      tickNo %= (+1)
...

However, for me this solution feels like cheating because at the end of the day, all my Logger es is doing is generating a singleton event to append onto the front of my EventList which is a linked list and I’m just appending them onto the front of the list by calling a function ==> a lot of work for (++) - which really should become a (:) in this current state - and function to pattern match on KeyEvents. Still, a lot has been learned so far.

Ideally, it would be nice to get the Logger e or the Eff e into the GameplayState type but I can’t figure it out right now.

I’m still experimenting around and trying to understand it and I have nowhere near the capability as anyone used to working on this stuff on a regular basis :’) so apologies if I have butchered anything.

[atravers]

@Bodigrim

How did we get into discussion of changing the definition of IO a […]

The impressions I acquired from the majority of other’s posts on this topic was that:

• IO a is now more hindrance than help.

• the use of a “piecemeal” monadic type for effects have been hindered by adverse attitudes towards earlier (and allegedly-defective) implementations (with frequent criticism of Java’s checked-exception mechanism) - apparently it can be done correctly in Haskell.

So I proposed this challenge:

type IO a = Eff All a  -- what is "All"?

…with a view to a future Haskell standard - if IO a could be defined as a special case of a more general system of effects, doing this would mean moving closer to having one standard system of effects that everyone can improve together, as an alternative to everyone continuing to build their own.

But here’s something else I’ve just realised - by being able to switch between IO a's current definition and that special-case definition right now (for example by reusing some of the alternate Haskell preludes) it would be possible for the authors of effect systems to more-or-less use the entire Haskell ecosystem to test and refine their offerings.

---

@Bodigrim

[…] (which is obviously not going to happen)?

As I have now mentioned a few times already, a more permanent change for the management of effects would be a matter for a future Haskell standard. But it will be one of many matters that the writers of that standard will have to contend with; this being another:

• (pages 18-19 of 217)

  This no-intervention-needed parallelism is usually called implicit parallelism and it is very important for scalable machines: one cannot expect a programmer to pre-plan the execution of hundreds of thousands of independent processing tasks “by hand.”

  Graph reduction without pointers (1989).

  • https://www.amd.com/en/newsroom/press-releases/2023-10-19-amd-introduces-new-amd-ryzen-threadripper-7000-ser.html

    …which “feature up to 96 cores and 192 processing threads of incredible performance for the workstation market.”

  • Intel shows off a maths defying 8-core chip with 528 threads and optical interconnects | PC Gamer

    …interesting: that’s 528/8 = 66 threads/core.

  • Light-Based Quantum Computer Exceeds Fastest Classical Supercomputers | Scientific American

    It is difficult to estimate just how much time would be needed for a supercomputer to solve a distribution with 76 detected photons—in large part because it is not exactly feasible to spend 2.5 billion years running a supercomputer to directly check it.

  • Thanks to RISC-V International and SOPHGO we got a Milk-V Pioneer Box […]

    Even though it has 64 cores, single core performance isn’t very fast […]

    RISC-V Pioneer Box for builder.sourceware.org gcc CI - Mark Wielaard

• (page 2 of 9)

  As Amdahl’s law states (Amdahl 1967), the overall speedup that can be achieved is determined not by the parallel parts of a program but by its sequential segments. This becomes even more true with increasing numbers of processing units.

  Controlling Chaos: On Safe Side-Effects in Data-Parallel Operations (2009).

Hence the increasing relevance of this edited comment:

@darkxero

I think the universal conclusion we can draw from all these cases is that it’s better to push I/O, the allmother of exceptions and other external effects, to the edge of your program, as far away as possible.

[tomjaguarpaw]

@atravers

by being able to switch between IO a's current definition and that special-case definition right now (for example by reusing some of the alternate Haskell preludes) it would be possible for the authors of effect systems to more-or-less use the entire Haskell ecosystem to test and refine their offerings.

Yes, this is exactly the case currently.

@atravers

…how exactly should that be explained to future beginners?

I wouldn’t explain it to beginners. It’s an abstract type. Its internals are irrelevant to most people, including beginners. I’m not an educator, so maybe I’m missing something, but I don’t understand the need some people feel to “explain” IO (for example, by saying it’s a “pure program which produces an impure program for the RTS to execute”).

[BurningWitness]

@atravers

by being able to switch between IO a's current definition and that special-case definition right now … it would be possible for the authors of effect systems to more-or-less use the entire Haskell ecosystem to test and refine their offerings.

If the intent is for multiple libraries to share Eff on the type level, what would the term-level definition for Eff be? Also I don’t know how you’d prove All is the same as “all effects in scope” with current Haskell type-level features.

I can definitely imagine a future version of GHC that ships an effect system as a boot library plus an extension that switches between presenting all effects through Eff or through IO (for beginners), but that’s about it.