Manufacturing took ten years to learn this. AI gets one budget cycle.

What manufacturing learned the hard way

Four decades of automation, and the metric that broke first

The industrial precedent is not a loose analogy. It is the same structural problem, a new class of machine absorbing work formerly done by people inside a measurement system built when only people did the work, and it produced a documented, expensive failure before it produced a solution.

1.1  The metric that inverted

Post-war manufacturing productivity was measured principally in direct labor hours per unit. It was a good metric for a good reason: labor was the dominant variable cost, and the number moved when the plant genuinely got better.

Robotics inverted it. Direct labor hours per unit falls when you remove a worker, whether or not the robot that replaced them produces a usable part. The metric could not distinguish between a plant that had become more productive and a plant that had merely moved its costs somewhere the metric did not look: into capital, into maintenance, into rework, into scrap. Worse, it actively rewarded the second behavior, because capital and rework were somebody else's line item.

1.2  GM: the $90 billion instrument error

Under CEO Roger Smith, General Motors pursued the "lights-out factory," a plant so automated it could run without illumination, because nobody would be in it. Roughly $90 billion over a decade went into automation and acquisitions.^[1] The Detroit-Hamtramck plant became the emblem of what went wrong; the widely-repeated story of robots painting each other rather than cars is probably embellished, but the substantive failures (robots installing windshields into the wrong position, systems that could not be recalibrated fast enough to keep the line moving) are not in dispute.^[1][2]

The instructive part is what GM had already been shown. In 1984 it entered NUMMI, a joint venture with Toyota in a shuttered Fremont plant with a workforce GM had previously considered its worst. NUMMI reached Toyota-level quality with substantially less automation than GM's flagship plants. The lesson on offer was that Toyota's advantage was a system of work, not a class of equipment. As analyst Maryann Keller put it, Smith "thought Toyota possessed some kind of magic… the trick to cost savings wasn't simply getting rid of people."^[1]

MIT's Nelson Repenning and Don Kieffer have recently made this comparison explicitly as a warning about enterprise AI, noting that the same pattern (automating a process nobody has actually characterized) is producing the same result.^[2]

1.3  Toyota: autonomation, not automation

Toyota's counter-principle, jidoka, is usually rendered as "automation with a human touch," and its defining feature is a machine that stops itself when it detects an abnormality rather than continuing to produce defects at speed. The point is not that the machine is slower. The point is that an unattended machine producing defective output faster is not a productivity gain; it is a defect generator with good throughput statistics.

This has a direct and uncomfortable translation to autonomous agents, which by default do exactly what the pre-jidoka machine did: continue confidently past the point where their output stopped being correct.

1.4  Tesla, 2018: the lesson does not stay learned

Three decades after Hamtramck, Tesla's Model 3 ramp stalled on an over-automated general assembly line. Elon Musk's public conclusion in April 2018 was unusually direct: "Excessive automation at Tesla was a mistake. To be precise, my mistake. Humans are underrated."^[3] Tesla removed automation steps and reinstated human work to restore throughput.

The relevance is that this failure occurred at a company with no legacy process debt, excellent engineers, and full awareness of the GM precedent. The error is not a competence failure. It is what happens when a measurement system counts automated steps as progress.

1.5  The instruments manufacturing built

The resolution was not better robots. It was better instrumentation. Four ideas did most of the work, and all four have direct analogues that AI measurement currently lacks.

Note the common thread. Every one of these instruments exists to defeat a specific way of looking productive while not being productive. That is what a mature measurement system is: an accumulated set of defenses against the ways its predecessor could be fooled.

Six defects in how AI output is measured today

Each one has a manufacturing name and a published counter-example

The metrics currently reported in AI programs (to boards, to finance, and internally) cluster into what is best described as an activity stack: tokens consumed, API calls, monthly active seats, suggestion acceptance rate, percentage of code authored by AI, and self-reported time saved. Every item on that list is either an input, a proxy, or an opinion. None is an output.

2.7  What the aggregate evidence shows

These defects are not academic. MIT's NANDA initiative, reviewing 52 structured interviews, 153 senior-leader surveys, and over 300 publicly disclosed AI initiatives between January and June 2025, reported that roughly 95% of enterprise generative-AI pilots produced no measurable P&L impact, against $30 to $40 billion of enterprise investment.^[7]

The figure deserves care rather than repetition. It counts pilots that stalled before production, the six-month window is short, and the report itself notes ROI attribution is confounded. But a 95% null rate alongside near-universal adoption and 80%-plus perceived productivity gains is not primarily a technology finding. It is a measurement finding. Organizations cannot demonstrate value they have no instrument to detect, and they cannot stop value destruction they have no instrument to see.

There is a legitimate counter-reading, and it should be stated. Brynjolfsson, Rock and Syverson's Productivity J-Curve shows that general-purpose technologies reliably depress measured productivity early, because the complementary intangible investments (process redesign, new skills, reorganization) are costs now and returns later.^[8] Solow's 1987 remark that "you can see the computer age everywhere but in the productivity statistics" preceded the late-1990s IT productivity surge by roughly a decade. Some of today's disappointing numbers are the downstroke of that curve.

That reading strengthens the argument rather than weakening it. You cannot distinguish "we are early on the J-curve" from "this is destroying value" without instruments capable of telling them apart, and the activity stack cannot. The framework below can: a J-curve organization shows rising first-pass yield against flat or negative net value, while a value-destroying one shows flat yield in the task classes that matter.

A corrected measurement framework

Seven layers, from equipment effectiveness to portfolio economics

What follows is a measurement stack, not a scorecard. Each layer answers a different question, and the layers are ordered so that each one's failure mode is caught by the next.

3.1  Layer 1 · Agent Operational Effectiveness

The direct transposition of Nakajima's OEE. Its virtue is multiplicative structure: three factors that each look acceptable produce a product that does not, which is exactly the disclosure property the activity stack lacks.

Nakajima's world-class benchmark is 0.90 × 0.95 × 0.999 ≈ 85%. It is worth internalizing how far agent fleets sit from that: a fleet at 0.93 × 0.70 × 0.61 reports 39.7% AOE, and each individual factor looks survivable in isolation. That is the disclosure property working.

3.2  Layer 2 · Rolled throughput yield across agent chains

Multi-step agent workflows are multi-station production lines, and yield multiplies rather than averages. This is the least intuitive and most consequential arithmetic in the framework.

3.3  Layer 3 · Marginal cost per successful task

The denominator is the entire argument. Cost per API call, cost per run, and cost per seat all divide by things the business does not buy. The business buys completed work.

3.4  Layer 4 · Net value added

MCST is a cost. Turning it into an economic verdict requires an honest counterfactual: what the same completed work would have cost without the agent.

3.5  Layer 5 · The automation viability condition

Layers 1 through 4 measure a program already running. This layer decides, in advance, whether a class of work should be automated at all. It follows from three quantities and one line of algebra.

Let c_p be the cost of a human producing the artifact, c_a the cost of one agent attempt, and c_v the cost of a human verifying an agent-produced artifact to the same confidence. Normalize: α = c_a/c_p (relative inference cost) and ρ = c_v/c_p (verification asymmetry). Let q be first-pass yield.

Under a retry-until-success policy, expected cost per successful output is (c_a + c_v)/q, which beats c_p exactly when (α + ρ)/q < 1. Under a human-fallback-on-failure policy, expected cost is c_a + c_v + (1−q)c_p, which beats c_p exactly when α + ρ + 1 − q < 1. Both reduce to the same inequality. The condition is invariant to failure policy, which is what makes it usable as a planning rule.

3.6  Layer 6 · System throughput

The preceding layers can all be satisfied while end-to-end delivery does not improve. Two classical results bound what a local speedup can do.

The arithmetic is sobering. If authoring is 30% of lead time and AI makes it twice as fast, end-to-end improvement is 1/(0.70 + 0.15) = 1.176×, a 17.6% gain, not 100%. If authoring became instantaneous, the ceiling is 1/0.70 = 1.43×, or 42.9%. Everything beyond that requires relieving a different constraint.

Little's Law then supplies the failure mode. If review capacity absorbs 40 items per week and AI raises arrivals to 55, utilization exceeds 100% and the queue is unstable: work-in-process grows without bound and lead time diverges. Measured as "throughput up, instability up," which is exactly what DORA reports.^[5] The framework predicts the observed industry data.

3.7  Layer 7 · Portfolio normalization

Finally, the guard against Defect 06. Portfolio metrics must be weighted by baseline human hours, not by attempt count, or the easy-task majority dominates the average.

The unweighted figure overstates performance by 26.9 percentage points, a factor of 1.61×. No individual number in the table is wrong. Only the averaging is.

One task class, one quarter, fully costed

Where the money actually is

The following applies the framework to a single task class, "standard bug fix," over one quarter in an organization running roughly 12,000 agent attempts against it. Every input is a quantity an organization can measure from systems it already operates.

This program is genuinely working: Π = 1.93 means the class returns roughly $1.93 of avoided human cost per $1.00 spent. The viability condition agrees independently. Here α = 0.003 and ρ = 0.140, so q* = 0.143 against an actual q of 0.62, comfortably clear. The two methods must agree in sign, and do; that consistency is the framework's internal check.

4.1  The sensitivity that should reset the roadmap

Having built the model, the useful question is which lever moves it.

This is the practical payoff of measuring correctly. An organization whose dashboard shows only token spend will spend its optimization effort on the 1% line, because it is the only line it can see. The framework redirects that effort to yield, where it is worth roughly fifty times more.

Instrumentation and maturity

What to collect, and the order to earn it in

The framework requires no payload inspection, no proxy in the request path, and no new agent behavior. Every input is metadata already emitted by systems the organization runs.

Two of these deserve emphasis because they are where programs fail. The correlation ID is the entire framework's load-bearing element: without a durable join between a model call and the business object it advanced, none of the formulas can be computed and the organization is permanently stuck at activity metrics. It costs almost nothing to add at the start and is expensive to retrofit.

The baseline is the one input that cannot be harvested from logs. It requires either pre-adoption history or a deliberately maintained control group. Most organizations substitute an assumption here, and because NVA is linear in T_baseline, that assumption silently determines the answer. A program that assumes its baseline has not measured its ROI; it has restated its own hypothesis in currency.

5.1  Maturity ladder

The ladder is strictly ordered. Yield cannot be measured before attribution exists, and economics cannot be computed before yield. Programs that attempt to report board-grade ROI from L0 data are not measuring; they are estimating, and Defect 05 governs the accuracy of estimates.

Objections and honest limits

Where this framework is weakest

Goodhart's law applies to this framework too

First-pass yield can be gamed by loosening acceptance criteria, and MCST by routing only easy work. Formula 7 defends against the second; nothing defends against the first except keeping acceptance definitions owned by the function that consumes the output, never by the function whose performance is being measured. This is the same governance separation that keeps a quality department independent of production in manufacturing, and it is a governance answer, not a technical one.

Measurement is not free

Review-time capture is the expensive input, because it is the one least likely to be already instrumented. A reasonable compromise is periodic sampling of review duration by task class rather than continuous capture: ρ is a structural property of a task class and moves slowly, so a quarterly sample is usually sufficient. Continuous capture is warranted only where a class sits near the viability boundary.

The verification asymmetry ρ is genuinely hard to measure

It requires knowing what verification to the same confidence costs, and "the same confidence" is doing real work in that sentence. In practice organizations will estimate ρ from review-time telemetry and refine it from escaped-defect rates. This is the framework's softest input and should be reported as a range, with the viability verdict tested across it.

Task heterogeneity resists stratification

Knowledge work does not partition as cleanly as machined parts. Stratification will be imperfect and will need periodic revision as the work itself changes shape under automation. Imperfect stratification is nonetheless strictly better than none, which is the status quo.

The J-curve confound is real

Early negative NVA may reflect complementary investment not yet paying off rather than genuine value destruction.^[8] The distinguishing signal is the trajectory of q, not the level of NVA: a J-curve organization shows first-pass yield improving quarter over quarter while NVA is still negative. A genuinely failing deployment shows flat q. Reporting q as a time series, not a snapshot, is what makes the two separable, and no activity-stack metric can make that distinction at all.

Not everything valuable is a task

Exploratory work, learning, and the option value of capability that has not yet been applied are real and are not captured by a completed-task denominator. The framework measures production, and should not be extended to govern research. Applying task economics to exploratory work would reproduce, in a new domain, precisely the error this paper describes.

Measure the output, not the machine

The argument reduces to one sentence: an organization cannot manage what its instruments cannot distinguish, and the current instruments cannot distinguish a productive AI deployment from an expensive one.

Manufacturing reached this conclusion between roughly 1985 and 1995, at a cost measured in tens of billions of dollars and several near-death corporate experiences. It responded by building instruments (OEE, first-pass yield, rolled throughput yield, constraint analysis) whose common property is that each defeats a specific way of appearing productive without being productive. Those instruments transfer. The transposition is not metaphorical; the mathematics is the same mathematics, because the structure of the problem is the same structure.

Three claims are worth carrying out of this paper:

• Yield compounds, and chains are worse than their steps. Five 90% steps produce a 59% result at 1.69× the nominal cost. Per-step benchmarks are not merely optimistic about chain behavior; they are the wrong quantity.
• Verification cost, not inference cost, decides viability. The condition q > α + ρ holds regardless of failure policy, and because α is small and falling while ρ is fixed by the nature of the task, cheaper models do not change the answer for judgment-dense work. In the worked case, a 90% price cut moved unit economics by 0.8%; an 18-point yield improvement moved them by 42.9%.
• The measurement gap, not the capability gap, is what most organizations are actually facing. A 95% null-ROI rate coexisting with 80%-plus perceived productivity gains is the signature of a missing instrument, not a missing technology.

The organizations that will defend their AI spend to a board in two years are not the ones with the largest deployments or the cheapest tokens. They are the ones that can produce a number, per class of work, showing what a completed unit of output cost and what it would have cost otherwise, and show the raw records underneath it. That is a measurement capability, and it is built before it is needed, not after it is demanded.

Toyota did not win because it had better robots. It won because it knew, station by station, what it was actually producing.

References

1. Graban, M. "GM's CEO Roger Smith Thought Toyota Had Magic, But This Was the Secret." Lean Blog, June 2016. On GM's ~$90bn automation program, the Hamtramck failures, the NUMMI joint venture, and Maryann Keller's assessment. leanblog.org
2. Repenning, N. and Kieffer, D., MIT Sloan School of Management, reported in Fortune, 3 September 2025. GM versus Toyota in the 1980s as a case study for enterprise AI adoption. fortune.com
3. Musk, E., April 2018, reported in TechCrunch. "Excessive automation at Tesla was a mistake… humans are underrated." techcrunch.com
4. GitClear. AI Copilot Code Quality: 2025 Research. 211 million changed lines, 2020 to 2024; refactoring 25%→<10%, copy-paste 8.3%→12.3%. gitclear.com
5. DORA. State of AI-assisted Software Development, 2025, as reported by InfoQ. ~90% adoption; AI as "amplifier"; throughput up alongside rising instability; 33% trust, 3% high trust, 80%+ perceived productivity gain. infoq.com
6. METR. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. July 2025. 16 developers, 246 tasks; expected +24%, measured −19%, believed +20% afterwards. metr.org · arXiv:2507.09089
7. MIT NANDA. The GenAI Divide: State of AI in Business 2025. 52 interviews, 153 leader surveys, 300+ disclosed initiatives, January to June 2025; ~95% of pilots without measurable P&L impact. State of AI in Business 2025 (PDF)
8. Brynjolfsson, E., Rock, D. and Syverson, C. "The Productivity J-Curve: How Intangibles Complement General Purpose Technologies." American Economic Journal: Macroeconomics 13(1), 2021, pages 333 to 372. NBER w25148
9. Nakajima, S. Introduction to TPM: Total Productive Maintenance. Productivity Press, 1988. Origin of Overall Equipment Effectiveness and the 85% world-class benchmark.
10. Goldratt, E. M. and Cox, J. The Goal: A Process of Ongoing Improvement. North River Press, 1984. Theory of Constraints.
11. Womack, J. P., Jones, D. T. and Roos, D. The Machine That Changed the World. Rawson Associates, 1990. The MIT International Motor Vehicle Program's comparison of mass and lean production.
12. Solow, R. New York Times Book Review, 12 July 1987. "You can see the computer age everywhere but in the productivity statistics."