The topic is a little exotic but hopefully worth an announcement just in case.
YCHR is a Constraint Handling Rules compiler that targets procedural languages instead of Prolog.
The motivation is threefold:
Building the first production-grade CHR compiler outside of a Prolog ecosystem.
Exploring various compiler design trade-offs.
Experimenting with LLMs.
Yes, the code has been entirely generated by large language models.
The project is still in active development but the implemented features are fully functional and usable. Currently the supported backends are Haskell and Scheme.
The binary is a standalone compiler, but the library can be used to compile and run CHR programs from Haskell.
I didn’t review it line by line. That’s the “experiment” I was talking about.
In my reviews I focused mostly on getting the types right and on test coverage. My experience so far is that, as long as the types are descriptive enough and enforce the most important invariants (which doesn’t happen automatically; the LLM needs guidance), the implementation is always correct.
This is an early release. So far I focused on getting the core compilation algorithm right and implementing the full pipeline. The library API is, admittedly, an afterthought and probably needs a redesign. The DSL should probably be redesigned from scratch.
Once the API gets more stable, the plan is to rewrite most of the public documentation in my own words.
Same as any other software you install: you don’t know unless you review the whole code base.
Regarding this particular project, I can give you a little more context, so you can decide yourself whether to trust the software or not:
I reviewed a good part of the code, focusing on the parts that I deemed more important, and skimming over the rest.
Every single change has been reviewed by multiple independent AI agents, also by models of different companies.
I perform regular adversarial QA tests, both manual and agent-driven.
I am familiar with the domain and would be able to rewrite the same compiler without AI (given enough time). If the code contained something blatantly anomalous I think I would spot it quickly.
I wrote such a bug myself (arbitrary file deletion) and it took me months to spot it… and I accidentially found it while reading my own code base out of boredom.
This type of overconfidence is exactly what I find concerning about AI use: The output is ten fold in quantity, you are less familiar with the code than ever before, but you still think it’s probably fine.
Maybe one thing I can do to mitigate the problem is to isolate the unsafe parts (TH and Generic) and mark all the other modules with the Safe pragma. This should at least guarantee that the pure parts are really pure, so a user who is worried about unexpected IO only needs to review the IO functions. I can make the IO-pure segregation sharper at module level to help with that.
That seems like probably a very different failure mode to me?
Like, the kind of bug that allows arbitrary file deletion in a program that’s intended to be able to delete specific files, is very different from the kind of bug that allows arbitrary file deletion in a program that’s not intended to be able to delete files. I’m imagining you were working on the first kind of program, and ychr is the second.
I’m imagining that ychr has a handful of uses for IO: reading files, mutable variables (IORef/MVar/STM/…), exceptions, printing to stdout/stderr. Maybe one or two I’m forgetting. But no writing to the filesystem, and no network access. Assuming that’s right (which it may well not be!) then I think “ruling out hidden destructive bugs to sufficient degree of confidence” isn’t all that difficult.
I think our imagination is irrelevant when dealing with slop. And it doesn’t matter what you prompted. It’s known that LLMs sometimes ignore instructions and that unrelated code can make it into the implementation.
I don’t think any amount of effects system or safe haskell would convince me.
Either you own the output and understand it or you’re playing code roulette.