For example: no matter what I do, I can't prevent Codex from introducing linter errors.
I use tabs instead of spaces for indentation. It seems like the model is massively weighted on code written using spaces (duh). Despite having a very well articulated styleguide (that Codex helped me write after examining my codebase!) that clearly specifies that tabs are used for indentation, the model will happily go off and make a bunch of changes that incorporate spaces seemingly at random. It will use tabs correctly in certain places, but then devolve back to using spaces later on in the same files.
I have a linter that I've taught the model to run to catch these things, but A) that feels like such a waste of tokens and B) the model often forgets to run the linter anyway.
It's like having a junior developer who has never used the tab key before. They remember to ctrl-f their spaces->tabs sometimes before opening a PR, but not all the time. So I wind up giving them the same feedback over and over.
This example -- tabs instead of spaces -- is just one specific case where the model seems to invariably converge on a local maximum that is dictated by the center of the software bell curve. But in general I'm finding it to be true of just about any "quirky" style opinion I want to enforce:
- Avoid Typescript's `any` or `unknown`: the model will still happily throw these in to make the compiler happy
- Avoid complex ternaries: nope, the model loves these
- Avoid inscrutable one-letter variables versus longer, self-descriptive ones: nope, `a` it is
It just seems like I'm using a tool that is really good at producing the average software output by the average software developer. It's not able to keep my aesthetic / architectural desires at the front of its thinking as it's making my desired changes. It's as if I hired the guy from Memento as an intern.
How do people get around this, other than slathering on even more admonitions to use tabs no matter what (thereby wasting valuable tokens)?
Beyond syntax, I'm still able to use the models to crudely implement new features or functionality, but the way they write code feels... inelegant. There's often a single unifying insight within grasp that the model could reach for to greatly simplify the code it wrote for a given task, but it's not able to see it without me telling it that it's there.
LLMs by nature are not very good at peeing against the wind. Also on average they are only as good as the average codebase they been trained on. By design.