A worked run: 500 resumes, with the numbers

The rest of this course tells you what to do. This page is us doing it, on the 500-resume dataset, reporting whatever came out — including two runs that failed and the reason each one failed.

What this is and is not. Every number below was produced by extracting text from all 500 dataset files and scoring them against criteria written from the job brief. The extraction is real — pdftotext and python-docx, the same mechanics any parser uses — so the format effects are measured rather than modelled. What it is not is a real hire: the candidates are synthetic, so nothing here tells you how these criteria would perform against real people. Read it as a worked example with honest arithmetic, not as evidence that the method works.

Setup

Criteria written from job-brief.md, following Module 4, Lesson 3. Two must-haves, because the brief says orchestration and warehouse experience must arrive intact and does not insist on anything else:

• Must-have 1: names an orchestration tool.
• Must-have 2: names a warehouse or lakehouse platform.

Then five weighted signals: pipeline ownership language, stated outcomes, streaming work, finance-adjacent domain, and mentoring or on-call.

Step 1 — Ingestion

So five candidates were effectively invisible before any criterion was applied. Two never arrived; three arrived as blank pages. On a score-ordered list all five sit at the bottom, indistinguishable from weak candidates.

Step 2 — First run, and why it failed

446 of 498 candidates cleared the must-haves. The score distribution:

That is the flat distribution Module 4, Lesson 4 describes at Step 5: everyone scoring similarly means your criteria are not distinguishing anything. Six out of every ten candidates were in the top band. A "shortlist" drawn from that is a random sample with extra steps.

The cause was the scoring rule rather than the criteria themselves. Each signal awarded full points for two keyword matches, and in a pool where every candidate is a data engineer, almost everyone hits two keywords on almost every signal. The scoring measured membership of the category, which the must-haves had already established.

Step 3 — Second run: distinct evidence instead of keyword presence

Same criteria, different scoring. Points now come from the number of distinct pieces of evidence — how many different systems they describe owning, how many separate stated outcomes — each capped so no single signal can dominate.

Better spread, and still not good. Which brings us to the finding that matters most on this page.

The finding: 169 candidates tied at the top

The maximum score was 81. 169 candidates scored exactly 81. 273 scored within three points of the maximum.

So "the top 20" is a slice of 169 identical scores, ordered by nothing. Had we sorted the list differently — by filename, by upload order, by candidate ID — we would have produced a completely different shortlist with exactly the same defensibility.

This is what Module 4, Lesson 5 means by precision theatre, and we did not expect it to be this severe. Any tool displaying that list ranked 1 to 169 would look authoritative. The honest presentation is three bands:

The correct conclusion is not "pick 20 from the top band". It is that the criteria are too coarse for this pool and need a signal that actually separates 273 people — scale, seniority of ownership, or something the job brief did not specify and the hiring manager would have to decide. That is a conversation, not a scoring change.

Document format predicted the outcome

This is the result we would most want a recruiter to see.

The candidates are synthetic and identically distributed across formats — the generator assigns career shapes independently of file type. There is no reason a two-column CV should belong to a weaker candidate. The 16-point gap between DOCX and two-column is entirely an artefact of reading order, and the scanned resumes were excluded outright.

In a real pool, resume template choice is not random. It correlates with country, career stage, whether someone used a university careers service, and whether they bought a template. A pipeline with this property is selecting on something adjacent to those things, and it shows up in the funnel as a low score — indistinguishable from a weak candidate.

Nothing in a score-based bias audit would catch it. You catch it by auditing the parse, which is why Module 11, Lesson 2 puts that step in.

Comparing against the human shortlist

Overlap with the 20-candidate human shortlist: 2.

Module 4, Lab 3 says 10 to 14 is healthy and under 6 means something is broken. Two means something is broken. Here is what the two shortlists were actually made of:

Two processes optimising different things. The human reviewer weighted seniority heavily and had a documented bias against career changers and scanned documents. Our criteria weighted density of described evidence, which rewards people who wrote more bullet points — and career changers, who tend to over-explain their relevant work because they know it is not obvious from their job titles.

Both are defective, in opposite directions. The human was screening for a proxy (seniority) rather than for the requirement. we were screening for verbosity. Module 5, Lesson 1 warns that semantic and evidence-density scoring rewards longer documents; We built that failure ourselves without noticing, on our own dataset, after writing the lesson about it.

That is the most useful thing on this page, and the reason it is worth publishing a run that went badly.

The disagreement check

The three highest-scoring rejects all failed the same must-have — no warehouse platform named — while scoring at the ceiling on everything else. Two of the human reviewer's picks failed our must-haves outright.

Reading them, the pattern is that experienced engineers sometimes describe what they built without naming the platform it ran on. The must-have was doing something other than what it said: it was testing whether someone lists tools, not whether they have warehouse experience.

The second pass

Full-text search of the rejected pool for orchestration or pipeline language, as Module 4, Lesson 6 prescribes: 49 candidates recovered — roughly one in six of everyone rejected.

Five minutes of work. On a real search, those are 49 people who would have been silently dropped.

What we would change before running it again

1. Drop the warehouse must-have to a weighted signal. It was excluding people for a documentation habit rather than a capability gap.
2. Add a signal the pool can actually differentiate on — scale, team size, or breadth of systems owned. Without one, 273 candidates are genuinely indistinguishable and no scoring change will fix that.
3. Normalise for document length, or stop counting distinct evidence, because right now the score partly measures how much someone wrote.
4. Handle scans and unreadable files as a separate queue before scoring, not as bottom-of-list candidates.
5. Report bands, never a rank, for this pool. The ranking is not real.

If you run it yourself

You have the same dataset. Your criteria will be different from mine, and your results should be too — if you get exactly these numbers, you have copied our criteria rather than written your own from the brief.

The two things worth comparing: how flat your first distribution is, and whether your format pass-rates are as uneven as mine. Both are failures of the method rather than of any tool, and both are invisible unless you go looking.

Criteria sheet used here · the dataset · the lesson this run follows