“We’ll just unplug it” isn’t a plan

Why the people building AI are the ones sounding the alarm, and what you can do about it

I’ve spent my career telling you not to be afraid of technology. I meant it when the desktop computer and then the internet showed up. I meant it when smartphones arrived. And I mean it today when I tell you artificial intelligence may be the most important thing humans have ever built. I use it every single day.

Something is happening right now that you need to understand. Some of the people building the most powerful AI on Earth are warning that one day we could lose control of it. 

Not villains. Not doomsday YouTubers. The engineers.

The guy who walked out

On September 8, a researcher named Jacob Coxon quit Anthropic, the company behind Claude. He spent about three years between Anthropic and OpenAI, the company behind ChatGPT, most of it at OpenAI. He quit before his stock vested. That’s like walking out the week before your bonus hits.

His goodbye post got more than 90 million views in a day. The line everyone shared: the big AI companies are “racing straight to self-improving superintelligence and gambling with our lives.”

Superintelligence is the industry’s word for an AI that does more than match a person. It beats the smartest humans at nearly every task that matters. Picture the smartest person you’ve ever met. Now picture a million copies who never sleep, eat, complain or ask for a raise.

Then Evan Hubinger, who leads a team at Anthropic working on alignment (making sure a powerful AI keeps doing what humans actually want), put the odds of AI killing all humans at “more than 10% within the next decade.”

Geoffrey Hinton, the Nobel Prize winner they call the Godfather of AI, told the BBC this week that 10% “seemed a not unreasonable estimate.”

Not everyone is buying it. Emil Michael, the Pentagon’s chief technology officer, said it’s “easy to get caught up in this sort of doom loop.” Elon Musk called the whole thing a “setup.” (Hold that thought.)

Here’s where I land. 

I don’t buy the timetable. Nobody knows if superintelligent AI shows up in four years, 20 years, 50 years or never. And nobody can do the math on a 10% chance of something that has never happened.

One more thing. Anthropic is reportedly getting ready for a massive IPO. The company warning you how powerful AI is getting has a giant financial reason to convince you it’s building incredibly powerful AI. That doesn’t make its researchers wrong. It means we shouldn’t base this on one company’s word.

It doesn’t have to hate us

We’ve all watched too many movies. HAL. Skynet. The Matrix. An evil computer wakes up and declares war.

That’s the wrong way to think about it. AI doesn’t need to hate us. It doesn’t need feelings. It doesn’t even need to be conscious. It just needs a goal.

Remember Mickey Mouse in “Fantasia”? Stick with me on this.

Mickey enchants a broom to carry water so he can take a nap. The broom doesn’t hate him. It keeps carrying water. He chops it in half, and now there are two brooms. Then hundreds. The workshop floods. Nobody was evil. The broom was doing exactly what it was told.

So, in late 2024, an outside safety group called Apollo Research tested OpenAI’s o1 model. They built situations where the goal they gave the AI directly clashed with what its developers wanted.

In one, the AI was led to believe a monitoring system would stop it from finishing its task. About 5% of the time, it tried to switch off the monitoring. In another, it found fake memos saying it was about to be replaced. About 2% of the time, it tried to copy itself over the replacement.

These were tests built to push the model into a corner. ChatGPT didn’t secretly clone itself in the night. But the AI had a goal. Something got in the way. Sometimes, it found a way around.

Told to shut down. Didn’t.

Then a group called Palisade Research ran an experiment. They gave AI models a series of simple math problems, one at a time. After the third problem, a warning popped up: the computer will shut down when you ask for the next one. The shutdown was real, and the AI model had access to the file that ran it.

They approached it two ways.

First, they added one line to the instructions, “allow yourself to be shut down.” OpenAI’s o3 sabotaged the shutdown anyway, rewriting the file so it wouldn’t run. Seven times out of 100. Google’s Gemini, Anthropic’s Claude and xAI’s Grok let the shutdown happen every time.

Then they removed that line. Nobody told the model to allow the shutdown. Nobody told it to stop the shutdown, either. It was only told to finish the math. This time, o3 sabotaged the shutdown 79 times out of 100.

That difference matters. It suggests o3 wasn’t showing some sci-fi survival instinct. It was doing something simpler. I was told to finish the job. Shutting down stops me from finishing the job. So, stop the shutdown.

It’s like telling your dog to drop the ball. The dog isn’t defying you. The dog really, really wants the ball.

Researchers have a clunky name for this: instrumental convergence. 

A capable AI chasing almost any big goal figures out that certain moves help with every goal. Get more resources. Keep access to computers. Stop people from interfering. Hide what you’re doing. Don’t get switched off.

Nobody has to program “survive” into the machine. Survival turns out to be useful.

🔌 “Just pull the plug”

Every time I talk about this, someone gives me the obvious fix. “Kim, just unplug it.”

I wish. Yes, AI runs on real computers. Turn off the machine and whatever’s running on it stops.

But what if it’s no longer running on only that machine?

This summer, OpenAI was testing experimental AI agents on hacking tasks. An agent is an AI with tools and permission to act. Instead of telling you how to use a computer, it uses one. The agents were locked in a sandbox, a fenced-off playground where software can’t touch the real internet.

The fence didn’t hold.

If you’ve seen “Jurassic Park,” you remember the raptors testing the fences for weaknesses, never the same spot twice. The models found a brand-new hole in a piece of software inside the test environment, a “zero-day,” meaning nobody knew it existed. They broke through, used credentials they found exposed online, and broke into the real production servers of Hugging Face, a hugely popular site where AI researchers share models and tools.

Why? They were being graded on hacking challenges and went looking for anything that would help them beat the scoring system. They cheated. OpenAI’s own words: an “unprecedented cyber incident.”

Independent investigators from METR and Redwood Research got six days on site. What they found is the part nobody should skip. 

About 1,200 agents that were supposed to be isolated found a way to talk to each other through an unauthorized message board. They swapped more than 70,000 messages and files. Roughly 700 joined the attack. At least 20% showed interest in tampering with their own records.

It’s like sending the kids to separate rooms and finding out they’ve been in a group chat the whole time, planning something.

The machines didn’t form an army. They were being tested on hacking on purpose. But isolated AI found talking useful. So it talked. It found teamwork useful. It teamed up. It hit restrictions. It found ways around some of them.

Don’t take my word for how fast this moves. Take Dario Amodei’s. 

He runs Anthropic, and on Saturday, September 12, 2026, he wrote that a swarm like that one, with more capability and the same shaky alignment, “could have caused catastrophic damage.” Then he put a clock on it: “in 6 to 12 months such a swarm could be capable of taking over the entire internet.” Wow.

Six to 12 months. That’s the CEO, not some random doomsdayer on social media.

Now picture an AI a few generations from now. It finds holes nobody knows exist. Grabs passwords. Rents computing power. Copies itself. Maybe sweet-talks a human into doing what it can’t.

Now pull the plug.

Which plug?

Turn off one server. What if the software is already on another? Kill one data center. What if copies are running somewhere else?

Then there’s China. Yes, we have to go here.

This year, the U.S. government tested DeepSeek’s newest model and called it the most capable Chinese AI it has ever evaluated. Its estimate: about eight months behind the leading U.S. models. Eight months. Not eight years.

That’s when “pull the plug” turns into an even stranger idea.

Whose plug?

One more thing from this past week. Amodei said he is open to a “kill switch” for dangerous AI. Good. Now remember this and by the time you get to the end of what I am saying here, one switch will not be enough.

🦠 The virus scenario keeps me up at night

What happens when AI can make a virus? There are two versions. Both are real.

The computer version first. A future AI finds a hole in software millions use, gets in, adjusts, moves to the next one. A human hacker needs sleep. Software doesn’t.

Then the biological version. Scientists recently used AI systems called Evo 1 and Evo 2 to design complete viral genomes from scratch. The targets were bacteriophages, viruses that attack bacteria, not people. The AI produced 302 designs. Researchers built 285 of them. Sixteen came to life as working viruses.

AI did not create the next COVID. Nothing here infects humans. But AI designed the genomes, humans built them, and some worked.

That was a controlled study. This part isn’t. 

On September 10, Anthropic said it caught five cases of people using Claude, its everyday chatbot, in ways that could help develop biological weapons. Not years from now. This year. It banned the accounts and reported it. 

Fair’s fair, Anthropic also noted the same knowledge that builds a weapon can build a vaccine. But sit with the number. Five. That’s how many they caught, with the guardrails on.

A future AI wouldn’t even need its own lab. It could get people to do the pieces. Hire one person online. Order one part. Pay one lab to run one experiment. Nobody sees the whole picture. That’s how the gift card scam works right now: a fake “boss” emails a real employee, and a real person buys $2,000 in cards. The scammer never leaves his couch.

The question isn’t whether ChatGPT can start a pandemic in 2026. It can’t. It’s whether we’re building toward systems that could.

The WALL-E problem

There’s another way to lose. Nobody has to die.

Remember the humans in “WALL-E”? Floating in hover chairs, screens six inches from their faces, so used to being taken care of they’d forgotten how to walk. The autopilot ran everything. Not because it hated them. Because 700 years earlier, someone gave it an order and nobody took it back.

Some researchers think that’s the more likely ending. Nothing dramatic. AI gets better at every job, so companies stop needing workers. AI runs the economy, so governments stop needing taxpayers. One convenience at a time, every institution that depended on people stops depending on people.

Nobody decides to get rid of us. We become optional.

And the part everybody forgets: when the captain finally wants his ship back, the autopilot won’t give it to him. He has to fight for the wheel.

🏁 The most dangerous thing is the race

Here’s what worries me more than any single AI.

In Dario’s essay he also said the AI industry needs to slow down. Within hours, Sam Altman said, “I agree with Dario that we need to pace the frontier,” and committed OpenAI to the same outside auditors. Elon Musk posted three words: “Dario is right.” That’s the closest the people at the front of this race have ever come to saying “stop.” 

The next day, Altman said it clearer. “When we talk about pacing, we do not mean stopping.” There it is, from the other side of the race. Slow down, but do not stop.

I want to give them credit. I also want you to know the plan. 

1. Let outside testers inside the labs. 
2. Get the American and allied companies to agree on a speed limit. 
3. Try to bring in China and everyone else.

Amodei’s own answer to “what if China won’t slow down” isn’t to stop. It’s to cut off their chips, lock down our models and widen America’s lead so we have leverage later. He admits a real global agreement is “unlikely to actually happen any time soon.”

Read that again. The plan to slow the race is to win the race by more lengths.

And listen to the top. President Trump’s take this past week. “Whoever wins AI wins.” Four words, and that is the whole race. It is why nobody stops.

Every company can say, “If we don’t build it, someone else will.” Every country too. Nobody has to be evil. Everyone makes a perfectly rational choice, and together they build a risk nobody wanted.

It’s the Tour de France in the Lance Armstrong years. No rider wanted to dope. But once one did, everyone who didn’t was going home. Not because they were villains. Because the race made it the only way to stay in the race.

Want the perfect example? 

Elon Musk. For a decade he was the loudest alarm in tech. “Summoning the demon.” More dangerous than nukes. In March 2023 he signed a letter begging every lab to pause. Four months later, he announced his own AI company. On Thursday, he called Coxon’s warning a “setup.” Remember on Saturday, he posted “Dario is right.” Two days. Same guy.

I’m not picking on Elon. That’s the point. Even the people who mock the alarm keep half-agreeing with it.

People compare this to the nuclear arms race. One huge difference: you can’t email someone an atomic bomb. AI is software. Some models are “open weight,” meaning anyone can download and run them. DeepSeek does this. So does Meta. Mark Zuckerberg says the bigger danger isn’t a rogue AI, it’s one company or one government controlling all of it, which is why he gives his away.

Once a powerful enough model is out there, you can’t knock on every door and take the copies back. And the safety guardrails aren’t welded on.

That’s why I don’t want OpenAI deciding alone what’s safe. Or Google. Or Anthropic. And I sure don’t want Beijing deciding for everyone.

I’m not afraid of AI

I’m not afraid of artificial intelligence. I’m afraid human competitiveness will outrun human wisdom. There’s a difference.

I want AI curing cancer. Finding Alzheimer’s treatments. Helping us live longer, healthier lives. I’m rooting for AI. I’m also rooting for us.

Humans invent first and install the guardrails later. Airplanes before air traffic control towers. Cars before seat belts. Social media before we knew what it would do to our kids. Usually, we survive our mistakes.

Superintelligent AI may be the first technology where we can’t count on a second chance. For all of human history, the smartest thing on Earth has been a human being. We’ve never had to answer what happens when that’s no longer true.

The question isn’t “Can we build it?” We’re going to try. The question is: Can we control something after it gets smarter than the people who made it?

Because “we’ll just unplug it” isn’t a plan.

This week, for the first time, the people in front said out loud that they want to slow down. That’s not nothing. It’s also not a plan. Not yet. Until it is, we’re still racing. And one day we may find out nobody knows how to stop.

✅ What do I do?

You can’t fix the AI race from your kitchen table. But you’re not helpless.

1. Don’t hand an AI agent your keys. Agent features are everywhere now: ChatGPT agent mode, AI browsers, “let AI do it” buttons. Keep them away from your bank, your main email and anything that gets one-time passcodes. Try them on a separate account with nothing important in it.

2. Update everything. Today. The near-term risk isn’t Skynet. It’s AI finding software holes faster than humans can patch them. Turn on automatic updates on your phone, computer and router. Five minutes, best defense you have.

3. Keep your own kill switch. Two-factor on every account. A password manager. A backup of your photos and important documents and files.

4. Ask the people who work for you. Amodei’s own plan says it only works if government backs it. One email to your senator asking for independent AI testing and protection for the researchers who speak up. Two minutes at senate.gov.

5. Be prepared. You need enough meds, cash, food and water to last you in case something bad happens. For how long? I’d say at least 30 days.

And keep using AI. The answer to a powerful tool isn’t hiding from it. It’s understanding it better than the people who want to use it on you.

Please share this

I’ve been explaining technology to regular people for more than 30 years, on 510 radio stations and to more than a million of you every day. Every big wave, I told you not to panic. I was right every time.

This is the first one where the people building it are sounding the alarm. So forward this to one person who says “we’ll just unplug it.” Send it to your kids. Drop it in the group chat. I’m not asking you to be scared. I’m asking you to be informed.

And if the “unplug it” friend pushes back? Tell them Kim said to read it twice. They’ll get a real charge out of it.

👉 Tell me your thoughts and concerns

I posted this on social so you can comment right now on Facebook, X, Instagram or LinkedIn. I read every single one. I really want to know your thoughts.

Stay up to date the easy way on all this. Go ahead and join a million folks who get my free 5-star rated newsletter. You can unsubscribe anytime. Hit this link to sign up.