Haskell wlroots bindings

[bradrn]

This is split off from the Waymonad thread in the hopes of coordinating some new bindings for wlroots.

Starting off with my thoughts on design questions from @anarchymatt:

@anarchymatt

• should we fork hsroots?
• should we start from scratch?

I have no idea. I suspect it depends on how complete hsroots is already, how out-of-date it is, and whether the design is sensible or not.

• which version of wlroots should we target releases

Is there any reason to not target the latest one?

• should the bindings be very low-level or should they try to provide a nicer extraction? There are some projects that were started and then looked like they died because of the imperative nature of what they were doing

From what I’ve seen, a lot of Haskell bindings have two components: a set of low-level Haskell wrappers around the individual functions, and then a higher-level functional interface built on top of that. I’d strongly suggest starting with the low-level bindings, since they’re easier.

• I don’t think this involves just wlroots, I think it involves bindings to wayland too, which may already be done.

Are you sure? I thought the whole point of wlroots was to provide a wrapper around wayland.

[anarchymatt]

I was looking at this example WM and it seemed like it was calling libwayland: tinywl/tinywl.c · master · wlroots / wlroots · GitLab

/* The Wayland display is managed by libwayland. It handles accepting
	 * clients from the Unix socket, manging Wayland globals, and so on. */
	server.wl_display = wl_display_create();

I was looking at their TinyWM example to see what a usage of wlroots looks like.

It would be odd if we had to do a lot of work binding to Wayland, but I was wondering if some would be necessary.

[Solid]

FWIW this now-also-abandoned fork of waymonad features slightly newer hsroots bindings than can be found in the wlroots repo. I did want to look at wlroots bindings at some point as well, so I appreciate this effort!

[bradrn]

I’ve been looking a little more at wlroots (and especially their tinywl example compositor), and I think the following APIs are most important to prioritise:

• From libwayland: wl_display, wl_list, wl_listener
• Basic interfaces: wlr_backend, wlr_renderer, wlr_allocator, wlr_output
• Compositing: wlr_compositor, wlr_subcompositor, wlr_scene
• Window management: wlr_xdg_shell
• Input: wlr_input_device, wlr_keyboard
• Pointer: wlr_pointer, wlr_cursor, wlr_xcursor_manager
• Clipboard: wlr_data_device
• Logging: wlr_log

If we implement these, we should be able to reimplement tinywl in Haskell! It looks like a lot, but most wlroots interfaces look reasonably small and self-contained, so it should be doable.

I’ve also been looking at existing C bindings, in particular sdl2 and botan. (Incidentally, thanks to @ApothecaLabs and @BurningWitness for their work there!) Both are based on low-level, simple wrappers around individual C functions, which is very reasonable. Insofar as I can tell, sdl2 implements a more Haskelly interface directly on top of these, whereas botan has an intermediate ‘safer low-level’ layer; I guess it’s hard to know ahead of time which approach would be best for wlroots.

For actually writing the bindings, it would probably be easiest to use hsc2hs: wlroots makes use of a lot of big structs, and writing Storable instances manually would be an utter pain. There’s two approaches to the FFI itself, namely ccall and capi. It looks like the latter is more modern and more featureful, whereas ccall is simpler. sdl2 uses ccall; botan started off with ccall, but is busy switching to capi. I’m not entirely sure which would be best for wlroots, but I’m leaning towards capi given the experience of botan.

[BurningWitness]

@bradrn

ccall and capi

As per the blog, you should use capi for the bindings. You’re not limited to just one choice though, you can use a CPP macro to switch between the two conventions in the .cabal file.

@bradrn

wrappers around individual C functions

“Raw bindings as a separate package” approach is something I advocate for because it allows other people to use the FFI interface without any wrappers or extra dependencies. From what I’ve seen C library interfaces are already as good as it gets in terms of abstraction, so adding anything on top is a waste of time and resources (don’t even bother with MonadIO, not everyone uses effect systems).

The bonus of this approach is that you don’t need to think about boundaries when porting, you can just export the entire header top to bottom and choose which definitions you need later.

---

Regarding writing raw FFI in general:

• Storable is a bad abstraction for structs. It forces the read to go over the entire data structure; it requires an all-encompassing write definition (try storing C union types); it is impossible to do partial updates with it.

  The far more powerful approach is to generate sizeOf and alignment for the data structure, and the offset of every single field. Note that you do have to be diligent about zero-initializing memory.

  I made a storable-offset package a while ago just for the offset storing and it did the job well enough. (note that the base < 4.17 requirement is solely due to it relying on GHC.Records, I don’t feel like bumping it every half a year)

  I wish base had this figured out, but alas.

• foreign imports should be safe by default. unsafe variations can be added later by just copying the function and naming it *_unsafe.

  The reason for this is that unsafe calls stall garbage collection, so you should rely on benchmarks to prove you’re actually speeding the application up.

• I prefer to export constants as patterns (:: (Eq a, Num a) => a) and types as type synonyms. Constraining types and/or wrapping into newtypes isn’t all that helpful and may clutter the code.

• Sometimes callbacks are unmarshallable, but you can still have the types for them. Library users are expected to make their own C wrappers in those cases.

• C strings can be embedded with MagicHash (GHC.Ptr.Ptr "honk\0"#).

• For statically linking libraries: Cabal currently does not have a proper concept of an FFI library, so the only way to do that without baking code in is with pkg-config.

See vulkan-raw for all of these things used in an auto-generated leviathan of an API.

[bradrn]

@BurningWitness

As per the blog, you should use capi for the bindings.

Ah, thanks! (I actually do recall having seen that article recently, I just forgot about it.)

@BurningWitness

“Raw bindings as a separate package” approach is something I advocate for because it allows other people to use the FFI interface without any wrappers or extra dependencies.

With regards to this, I quite like sdl2’s approach: provide both the raw FFI interface and the idiomatic Haskell interface within a single package. This does add a few dependencies, but they’re nearly all boot packages, so that’s unproblematic.

@BurningWitness

From what I’ve seen C library interfaces are already as good as it gets in terms of abstraction, so adding anything on top is a waste of time and resources (don’t even bother with MonadIO, not everyone uses effect systems).

I’m not even talking about MonadIO, though: just basic things like, for instance, using ByteStrings rather than raw CStrings. For an example of what I mean, look at how botan-low wraps botan-bindings.

Also, I think that if it’s possible to make a proper higher-level interface, that is desirable. It may not be — as you note, some C libraries are pretty good already (and it looks like wlroots may be in that category). But that’s something to sort out later after everything else is stable.

@BurningWitness

I made a storable-offset package a while ago just for the offset storing and it did the job well enough.

This does look nice, thanks! And I like its minimalism too. In any case I’m going to start off with Storable for simplicity, but I’ll keep this in mind if that works poorly.

@BurningWitness

I prefer to export constants as patterns (:: (Eq a, Num a) => a)

Hmm, what role does Eq a play here? Can’t it just be Num a => a?

@BurningWitness

Sometimes callbacks are unmarshallable, but you can still have the types for them. Library users are expected to make their own C wrappers in those cases.

I don’t quite understand what you mean by this… could you provide an example please?

@BurningWitness

For statically linking libraries: Cabal currently does not have a proper concept of an FFI library, so the only way to do that without baking code in is with pkg-config.

Of course, this is exactly what I’m planning to do (via extra-libraries).

[bradrn]

Now that I look at it, it’s also worth noting these comments from L-as/waymonad:

hsroots needs a redesign so that it at least exposes a semi-safe interface, where you don’t have to juggle pointers around like now. It also needs to run all wlroots functions from a single thread since it’s not thread safe. Right now waymonad just isn’t compiled with -threaded to avoid thread safety issues. The same is possibly true for hayland.

The design I’m thinking of:

• All wlroots function calls are delegated to a bound thread using a channel.
• All “wlr objects” returned from the API will be wrapped in something like an IORef Ptr.
• A signal handler on “wlr objects” will be installed onto the destroy event such that the above IORef is set to null atomically when the object is destroyed.
• There will also have to be a finalizer on this that removes the signal handler from the destroy signal handler linked list.

AFAICT there shouldn’t be any thread-safety issues with the above.

I don’t yet know if that kind of thing is the right direction or not, but on the face of it, these ideas doesn’t look too unreasonable. (IORef or similar is likely to be especially critical, since IIRC wlroots uses mutation extensively.)

[BurningWitness]

@bradrn

I quite like sdl2’s approach

This approach also lead them to adding a -nolinear flag to the Cabal file because that’s extra dependencies.

@bradrn

using ByteStrings rather than raw CStrings.

Haskell has [at least] three distinct ways of allocating memory: malloc, mallocForeignPtr, newPinnedByteArray#. ByteString uses ForeignPtrs underneath and Text uses ByteArrays, so even converting between these two isn’t free.

@bradrn

if it’s possible to make a proper higher-level interface, that is desirable

You’re free to build however many intermediate libraries you need, I’m talking about the raw library at the very bottom of your dependency graph.

@bradrn

what role does Eq a play here? Can’t it just be Num a => a?

Pattern matching (because it’s a pattern).

ghci> pattern Foo = 1
ghci> :info Foo
pattern Foo :: (Eq a, Num a) => a

@bradrn

Sometimes callbacks are unmarshallable / could you provide an example

Chipmunk2D has a few callback functions that pass vectors and bounding boxes on the stack.

[bradrn]

@BurningWitness

This approach also lead them to adding a -nolinear flag to the Cabal file because that’s extra dependencies.

Ah, I hadn’t noticed that.

@BurningWitness

You’re free to build however many intermediate libraries you need, I’m talking about the raw library at the very bottom of your dependency graph.

Of course, for that one I need to keep it as simple and low-level as possible; no disagreements on that!

@BurningWitness

Chipmunk2D has a few callback functions that pass vectors and bounding boxes on the stack.

I don’t understand this example you’ve linked… it looks to me like this CpSpacePointQueryFunc is marshallable, since it exists as a function on the Haskell side too!

[BurningWitness]

@bradrn

it looks to me like this CpSpacePointQueryFunc is marshallable, since it exists as a function on the Haskell side too!

FunPtr a is marshallable, but you can’t write

foreign import ccall "dynamic"
  mkCpSpacePointQueryFunc :: CpSpacePointQueryFunc -> FunPtr CpSpacePointQueryFunc

, since cpVect is not marshallable. One way of solving that would be providing a C function that reads cpVect from stack to a pointer, but the user could also choose to write the entire thing in C.

[BurningWitness]

It also needs to run all wlroots functions from a single thread since it’s not thread safe.

I’d expect a giant “thread safety” disclaimer in their GitLab docs if that were the case.

Inevitably some things will need to be run in bound threads, so you will need a good architectural approach to this, otherwise you’ll get mired in race conditions and that’s just not fun. I don’t know if IORefs are a good way to structure anything multithreaded, in my personal videogame project I’m heavily leaning towards an actor model.

[bradrn]

Hold on, that code makes no sense to me… surely it should be:

foreign import ccall "wrapper"
  mkCpSpacePointQueryFunc :: CpSpacePointQueryFunc -> IO (FunPtr CpSpacePointQueryFunc)

?

But either way, I think I understand the issue now: you can create a cpVect from Haskell and pass it to C, but a C cpVect on the stack cannot be marshalled to Haskell as-is. That means that it’s impossible to make a FunPtr CpSpacePointQueryFunc using only Haskell code. Is my understanding here correct?

[BurningWitness]

Yes, you got it right, and yes, I screwed up the type signature.

[anarchymatt]

Thank you both for the knowledge sharing.

I started reading about the FFI, with Real World Haskell. The book encouraged using newtypes and it seemed elegant to me.

I’m excited to help even though I’m not as knowledgeable as the both of you.

When I took a brief stab at writing some bindings a few weeks ago, I noticed that some of the Wlroots functions accepted function pointers, and I wasn’t sure how to do that. I imagined we’d need to write the callbacks on the C side. That’s why I gotta keep studying I guess.

What should we do about the repo?

[bradrn]

@anarchymatt

I started reading about the FFI, with Real World Haskell. The book encouraged using newtypes and it seemed elegant to me.

Oh dear… RWH is nice, but sorely out of date. I wouldn’t rely on it for anything.

@anarchymatt

What should we do about the repo?

Given the discussion above, I think I’ve formed a fairly good idea of how I want to approach this. So I’m planning to make one sometime in the next day.

[bradrn]

Well, I got the first little baby step working:

This is admittedly from a grand total of 4 lines of code (7 if you include blanks), but it at least shows I can get something to work. It’s 1:30am here and I’m tired, but tomorrow I’ll extend this to support some actually useful functions, and make a repo for it.

[bradrn]

I’ve created a repo here:

GitHub - bradrn/wlhs: Haskell bindings to wlroots (and libwayland)

So far it contains (very incomplete) bindings to wl_display, wl_list, wl_listener, wlr_backend and wlr_renderer. Using GHCi I’ve tested what’s defined so far; it all seems to work correctly, though I haven’t got it to the state of actually displaying anything on the screen yet.

So far I’ve been prioritising the functions which are used in the tinywl demo. Firstly, because I presume those are the most important ones. But secondly, because that means we can get to a working example application as quickly as possible, which would be a really great milestone to achieve.

I would encourage anyone who’s interested to pitch in and help write bindings! The code is very repetitive, and some help would be much appreciated. It should be pretty easy to add new bindings — it basically involves just copying what’s already there and changing the names around. (Note that I’m trying to stay as close to the C as possible for these low-level bindings, so there’s plenty of naming conventions and so on which don’t look like Haskell at all.)

One note: I’m specifically targeting wlroots-0.17.1 here, since that’s the latest version and that’s what’s installed on my Arch Linux system. (Running a wlroots-based compositor, so clearly this version works well!) I therefore presume that you’ll need to get the header files for that specific version, if you want to work on this repo.

[BurningWitness]

I’m trying to stay as close to the C as possible for these low-level bindings

I brought up the naming question quite a while ago, keeping it 1:1 to C seems like the correct solution (edwardkmett comments on Should raw library bindings drop the namespace prefix?)

@bradrn

I’m specifically targeting wlroots-0.17.1

You most probably want to put that at the top of the README.

[bradrn]

@BurningWitness

@bradrn

I’m specifically targeting wlroots-0.17.1

You most probably want to put that at the top of the README.

Well… I’m specifically targeting that version for now. Of course I intend to keep it updated when the next version is released.

(For that matter, I don’t even have a README yet!)

EDIT: Just noticed and merged slotThe’s PR which adds a README with precisely this information!

[bradrn]

After making a few more bindings, I’m now at struct wlr_output… and it’s huge: more than 50 fields! I don’t fancy writing a Storable instance for that thing manually. Does anyone happen to know if there’s any less laborious way to do it?

(My usual options would be CPP or Template Haskell, but alas, to my understanding these can only run after hsc2hs…)

EDIT: on further examination, apparently hsc2hs lets me define custom CPP macros for preprocessing! I’ll have to see if I can automate it that way.