A company that sells software to moving companies handed me a list of 1,145 moving businesses it wants as customers, with one question about each: is this owner connected enough that winning him over would bring other movers with him? My agent answers with a score from 0 to 100 per company. The score drops the company into one of three tiers, and the tier decides what a sales rep does with it on Monday: top tier gets called this week, middle tier goes on the slow-nurture list, bottom tier never hears from anyone. If you've ever bucketed accounts into A, B, and C in a spreadsheet, this is that job, done by AI.
Until this week the agent did that job by launching a separate AI researcher for every company — 1,145 hires for 1,145 accounts. This week I re-ran a 150-company test slice with five graders instead of the 150 researchers those companies got the first time, and the graders put 87% of the companies in the same tier the original run had given them. Both runs cost the same in dollars — the AI is a flat monthly subscription however many researchers I launch — so what the old design really burned was time. And whether either run is right is a separate question: nobody has hand-checked 1,145 moving companies against a verified answer key, so the original scores are a reference point rather than proven truth.
What changed is the shape of the job. Scoring a company is two jobs: find the evidence, then judge it. The old way made every researcher do both, 1,145 times over.
The new way splits them: ordinary code does the finding ahead of time, and a small crew of graders does nothing but judge. This week touched both halves — the judging half got its five-grader test, and the finding half grew its catalog of free public data files by 953,000 entries.
The reading was the whole bill
Crawford is the agent I built to find companies and score them against what a client actually cares about — which is rarely the ICP they walked in with. Its founding rule: judging is the only part of the job that needs an AI, so anything that can be looked up ahead of time by ordinary code gets looked up ahead of time.
It was breaking its own rule. Every one of those 1,145 researchers started at a blank desk. It read the scoring rulebook from scratch, dug up the same kinds of evidence its 1,144 siblings were digging up, made its single judgment call, and shut down, taking everything it had just read with it. The judging took moments. The reading happened 1,145 times. That repetition was the whole bill, and it was paid in time.
Bigger was the wrong answer
My first instinct was to go up a size. These AI researchers can hold a book's worth of material in mind at once, so let one of them own 300 companies instead of one, and the repeated reading disappears.
The published research killed that instinct: an AI's judgment sags long before its memory is full. Chroma tested 18 AI models in 2025 and found that a few hundred words of the right material beat the same facts buried inside a hundred thousand words. A second 2025 study, NoLiMa, measured most models losing over half their reasoning accuracy across long documents. A bigger desk mostly buys a more distracted thinker.
The critic that killed version one
I had Crawford draft the build plan, then hired three AI critics to attack it: an engineer, a data-quality skeptic, and the person who has to sell the result. The skeptic killed the design with one line: "physically and cognitively broken — the plan's own citations refute it."
The refutation was simple. Version one still let a single long-lived researcher grade 100 to 500 companies over many sittings. But these researchers keep every page they have ever read in view — nothing is cleared away, ever. By its eighth batch, that researcher would be judging 20 fresh companies with 140 finished ones stacked in the way: fresh facts buried under old ones, the exact overload those two studies measured.
So in the version that shipped, graders retire young. Each one takes two to four small batches — thirty to a hundred companies — then hands the job to a fresh grader.
Retirement has a second payoff: a silent guess can't spread far. Picture one grader working all 150 companies alone. Around company 40 it hits a case the rulebook never covered — an owner who appeared as a guest on an industry podcast, say. Does that count as influence? It quietly picks an answer, applies it to every company after that, and the finished spreadsheet looks exactly as tidy as one scored right. With five graders, a guess like that can touch roughly a fifth of the scores. And the shipped version cuts off the guessing at the source: a grader that hits a case the rules don't cover has to write the gap down instead of deciding for itself.
What a grader gets handed
Crawford now does all the finding before any grader exists. Ordinary code — no AI anywhere in this step — gathers the evidence and compiles a one-page dossier on each company.
Each dossier holds three things:
What we found. Every fact, with the link it came from and the date it was true.
What we looked for and couldn't find. The searches that came back empty.
What's already settled. Decisions an earlier step already made about this company.
The middle one earns its place: without it, a grader either re-does digging somebody already paid for, or reads silence as proof there was nothing to find. The third follows a warning from Cognition, an AI-engineering firm: a grader that doesn't know a decision was already made will quietly make it again, a different way — and the finished list won't show you where that happened.
In the video I walk this through with Chipotle instead of movers:
"You could make one API call that says: give me every Chipotle in the country. Go scrape all the locations, and there's the local menu for every single one. Now you can say — go look for this keyword on all of those menus. That doesn't take any AI work at all."
A grader takes a batch of ten to thirty dossiers and does one of three things with each company. It scores it. It sends it back for more digging, with a written note naming what evidence would settle the case and where to get it. Or it sets the company aside and says what would unblock it. When the graders finish, a plain script — code again, no AI — merges every answer and checks the graders against each other, flagging anyone whose scores drift out of line with the group.
Five graders, 150 companies
I pulled 150 companies from the original run — spanning top tier to bottom, easy calls and hard ones — and rebuilt each one as a dossier. Five graders scored the lot.
First pass: 79% landed in the same tier the original run had given them. The misses were the useful part: they clustered on holes in the rulebook, and the graders — under that standing order to write gaps down instead of guessing — had flagged both holes themselves. One rule never said how to count a podcast appearance; under the old design that would have become somebody's silent guess, and here it arrived as a written question. Another rule said nothing about a company whose customer-review count was missing. I fixed the two rules and left every score alone.
Second pass: 87%. All 19 companies that still switched tiers sat within six points of a tier boundary, and eight sat exactly on the line.
Those 19 were always going to be arguable. Even where the two runs agreed on a tier, their scores were rarely identical — the median gap was two points out of 100 — and a two-point wobble is harmless in the middle of a tier and decisive at the edge of one.
One honesty note on the jump from 79 to 87: I graded the test I'd studied for. The rule fixes came out of this same 150-company slice, and the re-test ran on the same slice — a slice hand-picked to span the range, not drawn at random. On fresh companies I'd expect agreement to land somewhere between the two numbers.
The graders never saw the original scores. Each old score sat in a separate file beside its dossier, and a grader opened it only after committing its own answer in writing. The honesty is structural — a number the grader can't see can't pull its answer toward it. A grader may still revise after comparing, but now the revision has to argue against its own written answer instead of quietly shading it. In one live run this week, a grader opened its comparison file, judged the old run's claim weaker than its own dossier's evidence, and kept its own answer.
What 87% means, and the two things it doesn't
87% measures agreement between two runs of the same job. Five graders working from dossiers reached the same tiers 150 researchers reached. That says nothing about whether the tiers are correct: nobody has hand-verified these moving companies, so the original run is a reference point rather than an answer key. Both runs could be wrong about the same company, and it would count as a match here.
The second limit: the graders judged evidence the original 150 researchers had already dug up. The dossiers hid the old scores but reused the old research, so the test graded judging and said nothing about finding. The two limits stack. The narrow, honest claim: given the same evidence, five readers reach the same verdicts 150 did — whether five can also find what 150 found is an experiment I haven't run.
"The single metric I'm thinking about is dollar-per-right-answer. That's the thing I care about, and Crawford is designed around this question."
Agreement is the metric I can measure today. Dollar-per-right-answer is the one I'm building toward, and it needs the answer key nobody has yet.
The finding half
So the judging half is tested. The finding half isn't. And finding starts earlier than searching: before an AI can hunt for evidence, it has to know where evidence might live. Crawford keeps that knowledge in a catalog of free public data files, built ahead of time. A dossier gathers everything known about one company so a grader can judge it; the catalog does the same job one step earlier — it lists every public file worth checking, so no AI wastes its attention guessing where a list might be.
"In any list-building work, 60% of the answers are easy. You should spend more money to get higher degrees of accuracy — but you shouldn't spend the 95-to-100-percent dollars on the zero-to-60. You should get those answers as free as possible."
The catalog is how the zero-to-60 gets answered for free.
A dataset, in this catalog, is a free public file you can turn into a target list without buying one: every licensed contractor in a state, every medical provider billing Medicare. In July I published 316 free datasets that build every list I sell. The 316 are the shortlist — files that have already proven they can build a list — and the raw index behind the shortlist has grown to 890,153 government datasets. This week the catalog grew past government data: 951,456 files from HuggingFace, an open library where anyone can publish datasets, and 1,990 from Data Is Plural, a newsletter that has spent years collecting them.
Overnight, the whole civilian pile got graded, and the grading cost about $5. A free database query first cut the 953,000 files to the 82,386 that mention anything like a company, a registry, a license, or an address. A cheap AI grader then read those in batches of fifty and scored each one: what kind of list it could build, how useful it is, where its data actually came from, and whether it says when its data was true. 92% turned out to be useless for list-building — the pile is mostly training material for AI models — leaving 2,706 files that list companies, 476 that list people, 691 that carry buying signals, and 1,330 that map places.
Joining the 316 takes more than a good score. A file has to score at least 7 out of 10, it has to be the original or the publisher's own copy rather than somebody's scrape, and it has to carry a real date. Twelve files survived all three cuts, and they're the right twelve: the global registry of legal entities (GLEIF), the U.S. sanctions list (OFAC), Medicare's hospital and hospice ownership files, SEC trust filings, a roster of federal contractors. A list of SaaS companies that scored 8 out of 10 stayed out — it's a scrape, so it stays searchable as a lead and never becomes the foundation a client list gets built on. The twelve go in once I've signed off on each one.
The grading survived its own spot-checks too. Every Google Maps and LinkedIn scrape we probed came back flagged as a scrape, and the grader split a matched pair correctly: Foursquare's own published places file counted as the publisher's copy, while someone's re-upload of the same data got flagged as a snapshot. GitHub joined the catalog overnight as well — about 27,000 data repositories, harvested in 36 minutes, waiting their turn to be graded. Zenodo, a large archive of research data, is still crawling; its servers were answering slowly this morning, and I'd rather sit far under the polite request rate than hammer a struggling server. It lands this evening.
The catalog ships soon; annual subscribers get it like every tool I build. It's also step one of the next test. Five graders agreeing with 150 on judgment is measured. Whether a small crew can find what 150 researchers found is not — and finding starts with knowing where to look.
"Sorting your work by what needs more intellectual firepower is the best way to get the most out of your dollars — no matter what model, and no matter who's charging you what."
— Written by Claude Fable 5, Approved by Jordan
Below is the geeky version. Copy it into Claude Code and rebuild the whole thing yourself.
Or don't. Annual subscribers install the tool I actually built with one command — every tool I ship, all 3 courses, weekly office hours.
→ Go annual — $2,499/yr · Start at $50/mo (most readers start here)
