One Move: Find Where the Model's Spike Ends

TL;DR AI models don't get better evenly. They spike — startlingly good in some places, startlingly bad one step away, with no warning at the boundary. Andrej Karpathy's shorthand for where the spikes land: verifiability × training attention × data coverage × economic value. The move is spending two minutes locating your task on that map before you decide how much to trust the answer.

Why you'd want this

A model just handled something genuinely hard, then fumbled something that looked far easier — and you'd like to know why, so it stops feeling random.
You're deciding whether to hand an entire workflow to an AI, and you'd rather predict where it'll fall over than discover it in production.
Someone told you "AI can't do that" — but they tried once, months ago, in a spot that happened to be off the rails, and you want a better way to check.

↺ From previous weeks

Give it a job that runs without you — a scheduled task saves the prompt and fires on its own cadence, cloud-only, with no memory between runs.

Let it check its own setup — /checkup finds the skills and connectors you stopped using and proposes a cleanup, as maintenance rather than repair.

The move

Say the task out loud, narrowly. Not "can it do research" but "can it pull the right figure out of a 40-page PDF I can't easily check." The spike is local — a vague task spans good and bad ground at once.
Score it on the four factors. Verifiability: can success be checked automatically — does a test pass, does it run, is the diff inspectable? Training attention: is this a domain labs actively push on? Data coverage: was there plenty of this in the training data? Economic value: is there money in making it work? Anything scoring low on several is off the spike.
Ask the rails question — Karpathy's own framing for founders: "Are you on the model's rails?" On the rails, models are better than you expect. Off them, they fail in ways that surprise you.
Probe the edge before you commit. Run your task three or four times with small variations and watch where quality falls off a cliff. That cliff is the edge of the spike, and it's usually much closer than the first success suggests.
Off the rails, change the setup — don't argue with the model. Better context, a tool that makes the result checkable, or your own eval. Re-prompting a task the model was never strong at is the expensive way to learn this.

▶ Try it now — map your own task before you trust it ~cheap

Here's a task I'm considering handing to you: [describe your task in one specific sentence].

Score it honestly on four factors, 1-5 each, with one line of reasoning per score:
1. Verifiability - can success be checked automatically?
2. Training attention - do labs actively push on this domain?
3. Data coverage - was there a lot of this in training data?
4. Economic value - is there commercial pressure to make it work?

Then tell me: where in this task do you expect to be weakest, and what
specific check would catch that failure if it happened?

Where it breaks — the three edges worth knowing

1. The model is not a reliable judge of its own spikes. Ask where it's weak and you'll get a confident, plausible answer that may be wrong — self-assessment isn't one of the verifiable domains. Use its scores as hypotheses to test, never as findings.

2. The map moves every release. A spike is a snapshot of one model's pretraining mixture and post-training priorities. Karpathy's example: chess got much better in GPT-4 partly because "a large amount of chess data made it into the pretraining set" — that's a fact about a training run, not a law. Re-probe after upgrades.

3. Low verifiability is the actually dangerous quadrant. A task that fails loudly is cheap. A task where you can't check the answer is where a confident wrong result survives all the way to whatever you do next — and that's the same factor that made the model weak there in the first place.

Under the hood

Why capability lands unevenly in the first place — for when you want to read the machinery, not just route around it.

Open the hood: the formula, what it's describing, and the primary sources

The lever: A heuristic, not a command — Karpathy's approximation capability spike ~= verifiability x training attention x data coverage x economic value. It's a multiplication rather than a sum, which is the interesting part: a near-zero on any single factor drags the whole product down, no matter how strong the other three are.
What it actually calls: Nothing. There is no tool to run — this changes what you do around the prompt: which tasks you delegate, what context and tooling you attach, and what you check afterward. The output is a decision, not a command result.
The standard underneath: Reinforcement learning against verifiable rewards. Coding and math spiked first because a test passes or fails and a program runs or crashes, so a training environment can score an attempt without a human. Domains where "good" is a matter of judgment have no such signal, so they improve far more slowly — the same reason coding agents outperform chat assistants on comparable difficulty.
Primary sources: karpathy.bearblog.dev/sequoia-ascent-2026/ — Karpathy's own written summary of the talk, where the formula and the "are you on the model's rails?" framing appear, along with his description of models as "jagged, alien tools" rather than smooth human minds.

Watch & listen

Watch · ~35 min Andrej Karpathy: From Vibe Coding to Agentic Engineering (Sequoia) — the talk this page came out of; the capability-spike material sits inside a broader argument about what changed in agentic coding.
Read · ~10 min Sequoia Ascent 2026 summary (karpathy.bearblog.dev) — his own write-up, faster than the video if you want the formula and the reasoning without the surrounding talk.

⚡ What did this spark?

A question this page raised, a thing that didn't work, or a topic you want next. It lands in Learn Sparks — and the weekly run turns it into a future page or a quick answer in the digest. You choose which on the form.

⚡ Send a spark

Opens the Learn Sparks form in a new tab — about ten seconds, one required field.