Who gave AI companies the right to build the future?

In 1954, years after he led the project that created the atomic bomb, physicist J. Robert Oppenheimer was called to testify before the Atomic Energy Commission (AEC). The ostensible subject of the hearings was Oppenheimer’s position on the hydrogen bomb, a far more destructive version of the atomic bomb that the US had developed and first tested two years earlier.

Oppenheimer, who in the years after the war had become increasingly conflicted about atomic weapons, initially opposed work on the hydrogen or thermonuclear bomb, partially for moral reasons and partially because he was skeptical it would work. But he later changed his mind and supported work on it. The lawyers at the AEC wanted to know why.

It wasn’t because Oppenheimer had changed his mind about the morality of city-vaporizing thermonuclear bombs. Rather, it was because American physicists had struck upon a new design for hydrogen bombs that wasn’t just workable, but positively elegant, or “technically sweet” as he called it. For Oppenheimer, that was enough. As he told the AEC hearing: “When you see something that is technically sweet, you go ahead and do it, and you argue about what to do about it only after you have had your technical success.”

What Oppenheimer described was a kind of moral helplessness dressed up as resolve: the pull of a scientifically beautiful answer to an ugly problem, and the accompanying habit of holding the moral accounting until after the technical success. It is one of the most honest things anyone who built the bomb, or any other world-altering thing, has ever said. And it has never stopped being relevant, because the people now building the world-altering technology of our own moment keep saying versions of it too.

You go ahead and do it

Jack Clark, the co-founder and head of policy at Anthropic, the company behind the Claude models, is one such person. So it was worth paying attention last week when Clark sat down for a long public dialogue with Samuel Kimbriel, the founding director of the Aspen Institute’s Philosophy and Society, just six days after the federal government had abruptly cut off access to Anthropic’s two most powerful models, ostensibly over fears of what they could do.

Much of the conversation circled around a single idea that will be familiar to those who read Clark’s work: Powerful AI is coming, and it presents us with a choice, a choice we are actively refusing to make by failing to regulate AI. (Disclosure: Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

We regulate toothbrushes, Clark pointed out, and cars, and nuclear weapons. “But we seem to have this attitude towards technology that it’s impossible to regulate,” he said. “It is not impossible to regulate … we sort of act as though, oh well, the technology industry is just inevitably going to do stuff, which I think is a choice.” His sharpest example was the online platform shift that utterly reshaped the last two decades. “Social media ran an uncontrolled experiment on the world,” he said. “We all now think and talk a bit differently because of social media. That was a choice. We can choose things to be different.”

This is the kind of talk that has long differentiated Anthropic from other major AI companies: Its principals are willing to linger on the serious risks of advanced AI, risks that demand clear and even strong regulation. (About a week before the Aspen dialogue, and just a day before the Trump administration came down hard on Anthropic’s latest models, CEO Dario Amodei published a blog post calling for government authority to legally block or even reverse the deployment of frontier AI models failing safety tests on threats like cyberhacking and bioweapons.)

Anthropic acknowledges that advanced AI is an existential gamble, but argues it’s a gamble we must take. At the Aspen dialogue, Clark spoke of a coming century that will be marked by brutal challenges, aging populations, straining institutions, a warming planet, that apparently can only be addressed with AI. To not go forward with artificial intelligence would be to rob ourselves of medical miracles we can only imagine, and implicitly condemn those who might otherwise be saved.

Clark is right that there is a choice buried in all of this. But the question his framing elides is exactly whose choice it actually is.

Who chooses?

Sure, as Clark said, we regulate cars and toothbrushes and nuclear weapons, but in each case someone built the thing first, and the rest of us were left to decide what to do about a world that already contained it. Nobody voted on whether the atomic bomb should exist. We were handed the consequences and had to write the rules later.

Much the same is true of AI. The choice Clark wants the public to make around governing it only became necessary once his industry created the thing that needs governing. He is offering us a vote on what to do about AI, not a vote on whether it gets made, because that vote was already cast, in private, by him, a few hundred colleagues, and trillions of dollars. But why didn’t we get a say? Why are we stuck in the world where, as in Oppenheimer’s formulation, “you argue about what to do about it only after you have had your technical success”?

I wasn’t the only person in the audience who was wondering this. Near the end of the dialogue, a young woman put a sharper version of this question to Clark directly. Every frontier lab now admits the technology carries enormous risk, even existential risk, she noted. “So my question is, what gives you, Anthropic, and the rest of the frontier labs the right to continue building something that could destroy everybody, when none of us can actually opt out of it?”

Clark, to his credit, did not brush the question away. But neither did he answer it fully. He reframed it, away from the choice to build, toward the need for someone to take responsibility after it is built.

That someone can’t be the companies themselves, he said, describing an ideal future where “outside compliance, regulatory, testing and verification systems” would decide when each lab was allowed to go further. Governments were already moving faster than anyone expected, the US and UK, he said, had built testing agencies whose tools were sometimes better than the companies’ own.

It was a gracious answer, albeit one that sat awkwardly with the reality that President Trump now appears to be regulating AI by whim, but notice what it concedes. Asked what gives his company the right to build something that could destroy everybody, the head of policy at a leading AI lab did not say we have that right. He said the decision shouldn’t rest with companies like his, only to describe a system to take it out of their hands that does not yet fully exist. He and his colleagues are still building, at the frontier, as fast as the science and the compute allows, while telling the room that someone else really ought to be in charge. AI is already loose in the world. The regulation of AI is still mostly the stuff of blog posts.

So why are they really doing this? To bring it back to Oppenheimer: because AI is “technically sweet.” It’s not the race with China, not the trillion-dollar valuations, not even the creditable desire to cure disease, though all of those are real. Underneath them is something simpler and much harder to govern: we are compelled to build what is beautiful. Clark all but said so, marveling that AI is “easier and simpler to build than many other aspects of science,” that his chief scientist jokes they’d have AGI already if they just fixed the bugs in their code.

We humans are a tool-using species, Clark argued, and AI is the ultimate tool. It’s not that AI is inevitable, exactly, but that it is so weirdly simple to build once the foundations are set that “almost any path you go down, [AI] appears.”

Technically sweet

What Clark described is the pull Oppenheimer named in 1954, the pull of an elegant solution that makes the question of whether you should build it feel beside the point.

I can feel it myself, and I’m just a user. Put a capable model at your fingertips, ask it to do something you couldn’t do alone, write the program, find the flaw, untangle the thing you’d been stuck on, then watch it simply do what you requested, and you’ll experience a small electric thrill that has nothing to do with aging populations or the future of democracy. That thrill runs in an unbroken line from the user at the keyboard up through the engineer who trained the model to the executive who shipped it.

That’s why I suspect Clark’s regulation talk, however sincere, is downstream of a decision that was never really in doubt. Like Oppenheimer with the hydrogen bomb, the people building this technology feel they have no choice but to go ahead, and then to hope the rest of us make the right choices to govern what they could not stop themselves from making.

We have been lucky, so far, with the last technically sweet device that could still end the world. The hydrogen bomb has existed for 70 years without being used in anger, not because we solved the politics Oppenheimer warned about, but because the wiser choice won. And because we were lucky.

Clark may be right that the choice is still ours: The bomb did not decide the Cold War, people did, and people can decide this too. But it would help if the people handing us that choice slowed down long enough to let us make it, instead of building as fast as they can and trusting our luck, and theirs, to hold.

A version of this story originally appeared in the Future Perfect newsletter. Sign up here!