Tech moves fast, but you're still playing catch-up?

That's exactly why 200K+ engineers working at Google, Meta, and Apple read The Code twice a week.

Here's what you get:

All delivered twice a week in just 2 short emails.

Join 200K+ engineers

I got into auto-research loops the way most people do: someone smarter than me published something impressive, and I figured I'd try it.

The idea is simple. You have a system. You have a scoring function. You run the system, measure the score, make one targeted change, and run it again. If the score improves, keep the change. If it doesn't, revert. Repeat until you hit your target.

I wired it up. It worked. I kept going.

The whole process seems reminiscent of a Brute Force Search algorithm that I wrote in my first attempt at a Chess simulator. A remnant from the olden times when CPU cycles were expensive.

The first place I applied it was improving the quality of AI agent skills — the instructions that tell an AI how to handle specific tasks. I wrote a checklist. Does the skill have a Gotchas section? Does the description say when to use it, not just what it does? Is the content organized instead of dumped inline?

Six rounds later the skills were noticeably better. The loop did exactly what I wanted.

That's because I was solving a policy problem. There was a known standard. The goal was conformance — does this meet the criteria or not? Binary pass/fail is honest when conformance is the actual goal. The checklist wasn't a proxy for quality. Conformance was the quality.

Same logic applies to code reviews, test coverage targets, launch readiness checklists, content quality gates. You know what "done" looks like. You're checking whether you got there. The loop runs until it does.

Then I applied the same approach to something harder — tuning a trading strategy where I was trying to maximize a variable. Not "does this meet the standard" but "is this the best version I can produce."

I wrote criteria. The loop ran. The score improved every round. By round four, everything was green.

The output wasn't what I wanted.

What the loop had actually done was learn to satisfy the letter of each criterion without satisfying the spirit. The outputs got tighter, more technically correct, and progressively more hollow. I'd built a system that was excellent at passing my test and quietly failing at the thing the test was supposed to measure.

The checklist wasn't wrong. I was using it for the wrong problem.

Go back to the original. Andrej Karpathy's autoresearch has no checklist. No criteria. No rubric. There's one float from eval.py, and every iteration either improves that number or it doesn't.

You can't game a float the way you can game a list of criteria — it either went up or it didn't. The loop is disciplined because the measurement is disciplined.

Karpathy reproduced a 1989 LeCun paper a while back — one of the earliest neural nets trained with backpropagation. The interesting finding wasn't about architecture. It was about the loss function. Swap MSE for cross-entropy and the training error drops to zero. The network didn't get smarter. The measurement got more accurate.

Google's ML team ran into the same thing building internal code completion. Six percent reduction in developer iteration time — not from a bigger model or more data, but from a scorer that measured actual completion acceptance instead of syntactic correctness. The signal was the point.

Your loop is only as good as your scoring function.

What's emerged in practice are two distinct modes and getting them confused is where things go sideways.

A policy loop uses a checklist because conformance to a known standard is the actual goal. Code review, test coverage, launch readiness — you know what done looks like, and you're checking whether you got there. The checklist is honest because meeting the criteria is the job.

Optimization is different.

When the goal is finding the best version of something — tuning an algorithm, improving prompt quality, maximizing a trading strategy parameter — you need a signal that can't be satisfied by technicality. Either a hard number that goes up, or a judge model that evaluates whether the output is genuinely better. A separate model, different provider, no shared blind spots with the one doing the work. Slower, costs more per cycle, less consistent. But the outputs are actually better, and for an optimization loop that's the only thing that counts.

The community building these patterns on ClawHub has already figured this out, at least partially. The more rigorous implementations — awesome-autoresearch, the Karpathy-faithful variants — use hard metrics and eval.py patterns with git-native keep/revert. They explicitly say "mechanical verification only — no subjective looks good." The lighter-weight entry points use checklists, which works fine for the problems checklists are suited for.

The gap nobody's really addressing yet is the decision that happens before the loop starts: which type of problem is this, and does my scoring function match it? That's the thing I had to break something to figure out.

Before you wire up the next iteration cycle, one question:

Are you trying to confirm that something meets a standard — or are you trying to find the best version of something?

Conformance: a checklist, binary scoring, loop until it passes.

Optimization: a hard metric or a judge model, keep/revert on a signal that can't be gamed, loop until the number stops going up.

The loop infrastructure is the easy part. The community has built it well. The hard part is the same one that's been hard since backpropagation: what are you actually measuring, and is that measurement honest about what you want?

What problem are you actually trying to solve — and is your scoring function suited for it?