The four pitfalls of developer impact measurement.
In the opening of their talk, O'Brien and Foley lay out the catalogue of failure modes that have made the field of developer productivity measurement quietly notorious. Each of the four is recognisable to anyone who has sat through a quarterly review built on the wrong numbers.
The root cause
The four pitfalls share a single underlying confusion: they conflate activity with outcomes and individuals with systems. A team that ships sixty pull requests a week looks more productive than one that ships twenty — until you notice that the sixty-PR team is churning through bug fixes while the twenty-PR team is shipping a quarter's worth of strategic features.
They also share a tone: punitive. Surveillance creates a defensive posture; defensive teams don't improve. Diagnostics — measurement aimed at understanding the system — creates a collaborative one. Same data, different framing, opposite outcomes.
The baseline — what DXI shows, and where it stops.
DXI is a sentiment instrument. It captures what engineers experience — how they feel about planning, documentation, debugging, ease of release. That experience is invaluable: people who feel blocked usually are blocked. But sentiment lags reality by weeks, and the score alone never reveals the underlying system that produced it.
| Driver | Score | vs P50 | Rank |
|---|---|---|---|
| Documentation | 36 | −8 | #1 |
| Clear direction | 61 | −14 | #2 |
| Planning process | 66 | +1 | #3 |
| Cross-team collaboration | 55 | −4 | #4 |
| Production debugging | 46 | −6 | #5 |
| Build and test | 59 | −1 | #6 |
| Deep work | 61 | −4 | #7 |
| Code maintainability | 70 | −10 | #8 |
| Incremental delivery | 43 | −8 | #9 |
| Customer focus | 45 | −9 | #10 |
| Review turnaround | 72 | −8 | #11 |
| Change confidence | 81 | +1 | #12 |
| Ease of release | 63 | −9 | #13 |
Reading the table
Only two drivers post a positive delta against the industry median — Planning process (+1) and Change confidence (+1). Everything else trails. A score of 59 against a P50 of 64 is not catastrophic, but the pattern of weakness (documentation, customer focus, incremental delivery, ease of release) suggests a system, not just a mood.
The number is the headline. The shape of the deltas is the diagnosis. And the diagnosis is exactly where DXI hands the problem to whatever you build next.
LaterBot — the homegrown layer that closes the gap.
A bespoke layer used to be an expensive choice. AI-native iteration changed the economics. What used to require a six-month internal-tooling sprint can now be drafted in a week and refined in another. That shift is what makes the next section in this report possible at all — and is the reason the speakers stress it as a precondition rather than an option.
The four questions LaterBot exists to answer
- What's actually moving right now?
- What's stuck — and where is the WIP piling up?
- Where is quality degrading?
- Which teams are overloaded?
None of these can be answered by a quarterly survey. None can be answered by a third-party tool that doesn't know your domains. They demand a layer custom to the organisation that uses it.
This system exists for two reasons — to improve the lives of the people who build software, and to improve the software we ship. If it isn't doing both, we change it.The LaterBot Manifesto
The intellectual sources
The system is consciously borrowed from four bodies of work — each a school of thinking about how creative knowledge work actually flows:
The four design principles
- Metrics illuminate, they don't surveil.
- We measure outcomes and flow, not activity.
- Trends matter more than snapshots.
- Everyone sees the same numbers.
The five pillars, one ladder, no black boxes.
The five pillars earn their place because each answers a different question and none can be collapsed into the others. A single number lies by omission. Five numbers tell a story.
≤ 5 issues, single project. Limit work-in-progress; expose the queue rather than hiding it.One ladder, five tiers
Every pillar uses the same five-tier status system, so anyone in the company can read any pillar at a glance without learning a new vocabulary. Learn the scale once, read every dashboard.
The ladder is deliberately verbal rather than numeric. Engineers respond differently to "low traction" than to "score: 2/5". Words carry shame less easily than numbers, and a label invites a question — "what would move us up one rung?" — that a raw score does not.
Stacking it together — measuring AI's precise impact.
The trick isn't a single new metric. It's stacking four lenses — operational, sentiment, system, and threshold — so that a claim of impact has to survive all four to count. Each layer in isolation can be gamed or mis-read. The combination is robust.
GitHub + GetDX
Percentage of AI-assisted lines. Cohort of adopters vs. non-adopters tested against operational metrics.
DXI driver comparison
Quarterly sentiment layered on operational data. Are adopters more confident, more focused?
LaterBot adoption cohorts
High vs. low adoption against all five pillars. WIP health, project health, quality trajectory.
Beyond adoption — impact
Not yes/no adoption. Percentage of engineers using AI above an effectiveness threshold informed by the synthesis of the three prior layers.
Engineers active on tracked AI tools above the effectiveness threshold.
Cursor and Claude Code, fourteen-day rolling window. Twenty-eight of seventy-one engineers are currently using AI effectively — not just installed, not just clicking accept, but operating above a synthesised cross-layer threshold.
Why threshold beats adoption
"Did you install Cursor?" is a binary question. "Is your work measurably better because of it?" is a continuous one. Tab acceptance and accepted-lines rates are informational breakdowns — they describe one modality (tab completions) and miss agent-mode workflows entirely. The effectiveness threshold survives Goodhart's Law because it cannot be gamed by clicking accept more often.
A note from the wider industry
The summit-partners framing — borrowed from Coinbase's internal reporting — quantifies AI output in normalised units of work for an expert engineer, showing the proportion generated by AI versus human-written code.
Eight things that actually worked.
The list is shorter than it looks. Items one through three set the posture. Items four and five set the social contract. Items six and seven set the architecture. Item eight is the kill-switch that prevents the whole system from drifting into surveillance over time.
Developer impact is notoriously challenging to measure, but it's achievable — with the right abstractions and processes that are collaborative, not punitive.Sean O'Brien & Justin Foley · Later.com · March 11, 2026
For DocuSketch — three actions
Later's situation rhymes with ours. Seventy-odd engineers across several domains. AI tools rolling out — Cursor and Claude Code there, Claude Code via Bedrock here. Leadership that wants to know if the investment is paying off, honest enough to admit that the obvious metrics don't work.
One — write the manifesto. Before another dashboard is built, document what we believe about how engineering work works at DocuSketch. One page. Four sentences. Signed.
Two — choose the five pillars. They don't have to be Later's, but they have to be non-collapsible and reportable on one shared ladder.
Three — build the four-layer AI-impact stack. Baseline (GitHub) → sentiment (a four-question pulse, monthly) → system (LinearB cohorts) → threshold. The threshold percentage becomes the KPI for the AI-first transformation, replacing "is the tool installed?" as the lead indicator.