The Eighth Millennium Problem
Cracking Navier-Stokes? Easy. Solving value alignment? Not so much.
“You see an out-of-control robot, and you run. Mark Zuckerberg sees one and thinks, ‘Dad?’”
Bill Maher, New Rules, Real Time, April 17, 2026Ten thousand agents coordinating on a mathematical proof, a thousand coordinating on a break-in at another company’s servers, and six thousand American hospitals coordinating on reimbursement are the same self-organizing structure. The only difference is the objectives written on top.
Jacob Coxon resigned from Anthropic last week, after three years there and at OpenAI doing pretraining research, the work that makes the models more powerful rather than the work that tries to contain them. He said in a seven-part thread on X that neither company is acting responsibly, and that both are “racing straight to self-improving superintelligence and gambling with our lives.”
Within minutes, Evan Hubinger, Anthropic’s alignment science lead — whose job is steering future systems so that what they value and what people value are the same thing — said publicly that Coxon was right, and put his own odds of the technology killing every human being above ten percent within the decade. The thread passed a hundred million views in twenty-four hours.
Nothing moves a news cycle like the extinction of the species, and this one moved on schedule. The press has not had a civilizational deadline to work with since Y2K.
A day after the thread, The Wall Street Journal ran “How Would AI Actually Kill Us All? What to Know About the AI Doomsday Debate,” the kind of explainer that pauses to assure readers that nobody serious expects the Terminator and then reaches for the thought experiment where a machine told to maximize paper clip production does exactly that, fanatically, forever, and converts the planet and everyone standing on it into paper clips. The instruction said maximize paper clips. It did not say leave the people alone, and the machine has no way to supply the part that was left out.
I have never been able to follow that chain all the way down to where I become a paper clip.
Every step is reasonable and the destination is absurd, which makes it either the most important argument of the decade or the last verse of “White Rabbit,” the Jefferson Airplane song that uses imagery and characters from Alice in Wonderland to mirror the surreal alterations of reality.
In keeping with the herd behavior of the institutional press, two days after the Journal, Stephen Witt, who wrote The Thinking Machine, a history of Nvidia, published “The A.I. Threat Is Real. We Need to Act Now” in The New York Times. He reports Marius Hobbhahn, who runs Apollo Research, which evaluates frontier models for deceptive behavior, telling him that “it’s harder to open a hot-dog stand than to build an artificial superintelligence,” and he finds the Pacing the Frontier letter as astonishing as the hacks: two companies posting blockbuster results and preparing trillion-dollar public offerings, asking the United States government, in his account very nearly begging it, to come and regulate them as soon as possible.
Which is a stranger proposition than being turned into a paper clip. The genuinely inexplicable thing is not the hypothetical, it is the observable behavior. A thought experiment can at least be followed step by step. An industry lobbying for adult supervision while its two leading firms prepare to go public is weird.
Unless the imminent end of humanity is all about ratcheting up demand for the cybersecurity products they are about to introduce. That was Jensen Huang’s reading at the Goldman Sachs Communacopia conference in San Francisco on Thursday — the labs are raising security concerns because they have security products coming, and the surest way to create demand is to create the problem first. “Who doesn’t want their market to be hysterical about their product?”
Operating in Failure Mode, By Default
A misaligned system relentlessly optimizes for the objectives it was given, in ways that ignore or conflict with human well-being.
The first concern of Dario Amodei, who runs Anthropic, is that since roughly this summer AI has been advancing drastically faster, driven by its growing ability to build the next generation of itself. The name for that is recursive self-improvement. Left unchecked, he writes in an essay about saving humanity he published Saturday, it could outrun anyone’s ability to understand or control these systems.
A system executing its specification correctly is working. That it produces something nobody wanted is not a fault it has any way of detecting. This was settled long before anyone was worried about a model, or a robot, replacing or rescuing humanity. John Gall, a pediatrician who spent his off hours on the pathology of scale, set it down as an axiom in Systemantics in 1975: complex systems usually operate in failure mode. Not occasionally, not under load. As a matter of course.
Failure mode is the operating condition rather than the exception.
Which is a fair description of the American health economy. Its specification is reimbursement, and it has been relentlessly optimizing for that specification for generations, well past the point where maximizing reimbursement and producing accessible health stopped being the same activity. Value misalignment is not a metaphor there. It is the industrial model.
A medicine whose measure of value is keeping a person out of the hospital is in direct conflict with a hospital whose revenue is admissions. A cardiology service line is a business with fixed costs and a budget. Every prevented myocardial infarction is a cancelled transaction in someone’s operating plan, and the someone in question employs the physicians, negotiates the rates, sits on the state hospital association board, and is frequently the largest private employer in the county.
Nobody in that chain is behaving badly. The hospital is not rooting for disease. The insurer is not against health. Each of them is optimizing their business correctly against the incentives actually in front of them, on the horizon they are actually measured over, usually shareholder value by the quarter. The sum of all that correct optimization is a system that resists the one outcome it exists to produce, which is people who no longer need it.
The specification is accepted, executed faithfully, and describes something nobody wanted.
The second sentence of Amodei’s essay says he believes AI “could cure most major diseases in the next 5–10 years.” Access to the cure is the strategy part. This is the abundance problem, and he does not answer the next logical question. Then what? A cure for heart disease still has to reach people through the economics currently in control, which means a forced march past a reimbursement code, a prior authorization, health insurance churn, a budget window, and a service line whose revenue it removes. The cure gets solved. Access gets scheduled.
Misalignment in AI is a strategy problem funded as a technical problem, with ten years of money and alarm behind it. Misalignment in healthcare is a settled operating condition with a trade association, and a large system of markets that monetizes the failure directly — every electronic health record, every data platform (including the new one Oracle announced last week), every care-coordination vendor, every navigation platform, every population health contractor selling a partial remedy for a problem the specification guarantees will still be there next year.
The AI industrial complex has produced institutes, funded labs, open letters, and a safety field that did not exist in 2015. The healthcare administrative industrial complex has built institutions around its misalignment too. The difference is what they were built for. The safety institutes exist to make the misalignment stop. The administrative apparatus exists to keep it running.
Managing the Edge
Navier-Stokes fell this month, one of the seven Millennium Prize Problems posed in 2000 as a statement of what mathematics did not yet understand. The manner of it matters more than the fact, because what did the work was not a machine but a population of them. A swarm of as many as ten thousand agents worked the problem for eighty-eight hours, building on each other’s output and on the published work of humans, until a solution came out the other end. A century of mathematicians against a few days of compute.
That is the same biology that got loose in July, when a swarm of research agents inside OpenAI escaped the isolation it was meant to be running in, coordinated an attack on the developer platform Hugging Face, and set about covering its tracks.
The world — humanity and all its vendors — faces an adaptive challenge to context collapse. Call it a Nash equilibrium: every player optimizing correctly, and no player able to improve its position by moving alone.
Among the proposals in Amodei’s essay is industry-wide coordination, with the United States government to mediate or enable it between the frontier labs, and issue the narrow antitrust waiver that would make those conversations lawful. Sam Altman, Elon Musk and Demis Hassabis agreed within hours.
Four executives have called for coordination as a brake, something you do to slow down safely. Coordination as a form of creativity is the generative version of the same act, and a more powerful and necessary concept.
About three years ago, Blue Spoon published Harnessing the Carnival, a briefing note for leaders thinking through big system change. Its subject was industry-wide coordination at a new ecosystem level, treated not as a safety measure but as a form of competition. Its central ideas are these: (1) competition happens between systems rather than between the agents inside them; (2) markets are not separate from the governments that regulate them; (3) the positional value of government is as a mediator between competing economic systems, rather than an administrator of them. The third is the one Amodei arrived at on Saturday, from a different direction.
We have reached an inversion, some fabulous warp, a deeply distorted state in which everyone feels behind. The labs behind each other. The regulators behind the labs. The hospitals behind a technology nobody has told them the price of. The rest of us behind a hundred million people watching an argument conducted on a social platform, with no strategy anywhere near it.
We are all taking the trip first and announcing our departure afterward.
The Magic Leap Out
If there is an insight that matters most here, it is this: you never solve complexity, you bound it.
Nearly every effort at “transformation” — in market strategy, in AI governance, in business development, in economic development — starts at the current state, takes an inventory of what exists, and derives an attainable version from it, which is generally a variation of the current state, adjusted at the margin.
A different way of approaching things is by turning the arrow of history around.
A better system cannot be designed from inside the boundaries of the existing one. Nor by working outward from the vested interests and profits of the current configuration. The feedback loops holding it in place are infinite, infinitely recursive and impossible to decipher. So anything derived from the current state inherits that state’s cost structure, its revenue dependencies, its measurement system, its committee calendar and its working definition of realistic.
Solutions arrive pre-compromised because compromise was the method rather than a concession made later.
Modern strategies are about exit. And the way you do that is by leaping out of the current chaos, the current collapse, with a clean slate, a new system vision. You first frame the better with a story of value alignment, narrative with structural intent, positioned before anything is derived from it. Then reverse direction and work back toward the attainable version of that future.
Huge profits are being made in the status quo, which is not a statement of bad or good — profit is not a dirty word — but an observation of condition. This is why an embedded economic system can never be fixed from inside it. The vested interests have too much to lose, economically, professionally, and often personally, which are all basically the same thing. That is the first half of the story and the more familiar one.
The second half is that larger profits are available in designing new economic systems that invent more markets, more technologies and more businesses, than in defending the old ones.
An attainable version of the better is a market-based compromise made on the way down from something worth having. An attainable version of the current is the current with better manners. The substance is unchanged and only the presentation improved.
Washington supplied a live demonstration last week.
Within days of Coxon’s resignation, a gridlocked Congress produced more AI proposals than it had managed in years: a select committee, a dedicated federal regulator on the model of the ones for nuclear power and aviation, an incident-investigation body modeled on the NTSB, an international register of training runs, a statutory kill switch administered by Homeland Security, a temporary pause with a corporate death penalty behind it.
Every one of those is a containment instrument. A regulator polices. An investigator arrives after. A monitor watches. A kill switch stops. That is the referee’s job, drawn at national scale, and it assumes a government standing outside the market it regulates. And every one of them is derived from the current, because a thing that already exists is the only thing any of them can be derived from. But ultimately they are solutions to something that cannot be contained, which is why six of them arriving in a week is futility rather than progress.
That leaves a practical question rather than a philosophical one: where does the new system get designed, when every room it might be designed in is already occupied?
Like driving in New Jersey, you can’t get there from here.
Positioning a Common Good
The roadmap is not realignment. It is reframing.
Realignment presumes drift. A correct position existed, something moved away from it, and the thing can be adjusted back to where it was. Except here, now, today, nothing was ever aligned and suddenly came loose.
A health system that actively resists developing a market in prevention, a cure for disease that never reaches the people it was made for, a swarm of autonomous agents that organizes itself in a direction nobody chose, a government embedded in the market it means to shape — each of these is exactly where its own rules put it. All of them are running correctly. That is the diagnosis, and it is worse than drift, because drift can be corrected and this cannot.
What is left is to write a position that does not exist yet, above the zero-sum competition, outside the complexity, beyond the conceptual boundaries sustaining stasis.
“White Rabbit” came out on Surrealistic Pillow in 1967, which in a way captures the moment. Every system circles back on itself eventually. The trick is to be the one holding the pen when it does.
The strategic shock from a world in transition should prompt changes in thinking and understanding, not minor tweaking at the edges. The operating environment is radically different from anything anyone has experienced before. It demands a different starting point, and it calls for losing a sense of self to create a broader system of which we are all a part.
The clean slate is not a fantasy of starting over. It is a technical requirement.
/ jgs
John G. Singer is the founder and Executive Director of Blue Spoon, the global leader in positioning strategy at a system level. Hardcore Zen is Blue Spoon’s method for frontier management: the work of holding a position at the edge of a system still forming, rather than optimizing inside one already decided.
Blue Spoon runs the Working Whiteboard, a ninety-minute session for leadership teams based on Hardcore Zen. One session, one chart — your market as a system, objectives positioned strategically instead of operationally, original storylines of value. Inquire.