Safety of LLM generated code

philh · August 15, 2026, 11:27am

That seems like probably a very different failure mode to me?

Like, the kind of bug that allows arbitrary file deletion in a program that’s intended to be able to delete specific files, is very different from the kind of bug that allows arbitrary file deletion in a program that’s not intended to be able to delete files. I’m imagining you were working on the first kind of program, and ychr is the second.

I’m imagining that ychr has a handful of uses for IO: reading files, mutable variables (IORef/MVar/STM/…), exceptions, printing to stdout/stderr. Maybe one or two I’m forgetting. But no writing to the filesystem, and no network access. Assuming that’s right (which it may well not be!) then I think “ruling out hidden destructive bugs to sufficient degree of confidence” isn’t all that difficult.

hasufell · August 15, 2026, 11:42am

I think our imagination is irrelevant when dealing with slop. And it doesn’t matter what you prompted. It’s known that LLMs sometimes ignore instructions and that unrelated code can make it into the implementation.

I don’t think any amount of effects system or safe haskell would convince me.

Either you own the output and understand it or you’re playing code roulette.

philh · August 15, 2026, 12:24pm

I’m not imagining details about what the LLM generated, I’m imagining details about how to check what the LLM generated.

Like, if you know that a codebase shouldn’t have any IO at all, then checking it doesn’t is fairly straightforward. Do you agree with that?

gregbm · August 16, 2026, 2:53am

How do I know your program doesn’t delete my home directory?

Same way you’d know for human-generated code on hackage, probably.
Speaking of which, what defenses does hackage have against malicious packages?

hasufell · August 16, 2026, 3:18am

That’s a pretty strange way to dismiss the “lack of agency” issue. And I don’t think it can be taken seriously.

Empirical research across Fortune 50 enterprises found that AI-assisted developers produce commits at three to four times the rate of their peers but introduce security findings at 10x the rate, creating a security debt that accumulates faster than organizations can remediate it [7].

Our study suggests that securing vibe-coded appli-
cations requires treating vibe coding as a workflow-level security problem more than a code-generation problem

Ambrose · August 16, 2026, 3:55am

LLMs change the game here fundamentally sadly. They rework distributed trust because there is less of a hurdle of effort to get something compiling and plausibly working.

I see this myself in industry with junior devs. Before, they would get stuck and be unable to cut a PR that looked reasonable. Now, the LLM can give them that by default. But the LLM PRs have some pretty silly issues that trace back to the juniors’ lack of understanding.

gregbm · August 16, 2026, 4:14am

With regards to trusting code on hackage, I am more worried about intentionally-introduced malware than I am about vibecoded packages, so I would be interested in a verification mechanism that isn’t just scanning for signs of AI-generated code.

hasufell · August 16, 2026, 5:03am

Models can very well be compromised by state actors.

lortabac · August 16, 2026, 11:51am

My two cents:

The same models that empower inexperienced developers to ship sloppy code, are also extremely good at discovering bugs. This week alone, with the help of AI I caught and fixed 5 different subtle bugs at work. These bugs had been introduced by competent Haskell developers before LLMs were popular.
Yes, models can be compromised, even by state actors. I believe independent reviews between models of different companies should become more common. E.g. an American model implements, a Chinese model reviews, and viceversa. This is not only for paranoia, but also because different models tend to focus on different aspects.

Ambrose · August 16, 2026, 3:51pm

Yeah I am a fan of LLM code review and using it to navigate a large codebase. Those two things feel like free cheese to me.

I fix the bugs by hand though. So it kind of feels like a minigame. And because I’m in there by hand, it’s also just some grease for me to polish things up in general.

github.com/hawkw/mycelium

CONTRIBUTING.md

b77e854bc

# contributing to mycelium

thanks for your interest in contributing to Mycelium, an alleged "operating
system"! please bear in mind that this is a hobby project that i work on in my
spare time, so pull requests may not always be reviewed in a timely manner.

## AI policy

when contributing to Mycelium, an alleged "operating system", you are free to
use any tools you deem appropriate. this includes the LLM of your choosing,
agents, loops, gastown, AI code review tools, Amazon Rufus, or even `vim`.
however, all contributions to this project (including my own) must abide by
exactly one rule:

**your code (and any associated prose, such as commit messages and
documentation) must not _look_ like it was written by a large language model.**

basically, the game is simple: contributors are free to use whatever AI tools
they wish when contributing to mycelium and associated projects, but in order to
have your PRs merged, you have to be able to *trick me* into believing it was

This file has been truncated. show original

^ that’s why i really like this AI policy. “if i can’t smell the LLM on the code, i’ll accept it.” it’s a nice forcing function.

(and the package in the thread this one forked one sadly did not pass that sniff test for me upon opening Hackage not knowing LLM or nah)

hasufell · August 17, 2026, 4:40am

I’m not sure you read the studies I linked above, but “Empirical research across Fortune 50 enterprises […]” doesn’t read like “inexperienced developers” to me.

Sure, I’ve also considered that use case: ask an LLM to find bugs in my codebase prior to a release. If it doesn’t find any bugs, I didn’t really lose anything (LLM not finding bugs doesn’t mean your application is secure… this is not how it works).

But your calculation doesn’t really compute… this is for cases where there’s actually an author who understands the code and is familiar with it. You can’t claim that the bug-searching capabilities of AI is enough to replace the need for someone to understand the code. And the research I linked supports this.