In 1986, the Challenger Space Shuttle exploded 73 seconds after liftoff.¹ Onboard was Christa McAuliffe, the first civilian and educator selected for spaceflight.² Millions of schoolchildren watched live as the shuttle disintegrated. The tragedy was a national shock.

Within days, a presidential commission was formed to investigate the disaster. The commission uncovered the technical failure—a disintegrated O-ring³—but also revealed deeper systemic issues, calling it “an accident rooted in history.”⁴ NASA had ignored engineers’ warnings, pressured teams to proceed despite known risks, and relied on overly optimistic safety assessments.⁵ The shuttle program was grounded for over two years as NASA overhauled both its technology and safety culture.⁶

It took a disaster to reset NASA’s approach to risk. What will it take to reset artificial intelligence’s safety culture?

AI has contributed to several safety incidents over the years. In 2018, Elaine Herzberg became the first pedestrian killed by a self-driving car when an Uber AI system misidentified her as a bicycle.⁷ AI has made questionable cancer treatment recommendations,⁸ made biased hiring recommendations,⁹ and facilitated financial fraud.¹⁰ A chatbot was implicated in a child’s suicide.¹¹

But AI has not yet had its Challenger-scale disaster—an event so publicly catastrophic that it compels a complete rethinking of safety standards. We speculate on what such a disaster might look like by drawing from historical tragedies in medical technology, automotive safety, and public infrastructure to predict AI’s reckoning.

One possibility is AI causing direct harm to human life through its integration into medical devices. A precedent is the Therac-25 medical radiation therapy machine that over-radiated six patients in the 1980s, killing or seriously injuring them. The Therac-25 was an early software-controlled medical device in which hardware fail-safes were implemented in code instead, and coding errors caused it to malfunction. The Therac-25 has become a classic case-study in software engineering, and its failures led to developing international medical software standards.¹² A similar AI-driven failure in diagnostics or robotic surgery could spark a regulatory overhaul.

Another scenario is a series of AI-driven failures culminating in a public outcry. Ralph Nader’s 1965 book Unsafe at Any Speed exposed the auto industry’s disregard for safety. The book outlined several smoking guns such as the rollover risk in the Chevrolet Corvair, dangerous styling and ornamentation, and car companies blaming drivers for accidents. Congress formed a new agency that later became the National Highway Traffic Safety Administration (NHTSA) in response to the book.¹³ If AI is shown to be systematically unsafe—whether through biasing decision-making, widespread misinformation, or reckless deployment in self-driving cars—public backlash could force change and institute government oversight.

A final possibility is a complex, cascading AI failure akin to the Three Mile Island nuclear accident. In 1979, a reactor overheated and partially melted down.¹⁴ The public backlash was immense; after the accident, no nuclear power plant was authorized to begin construction until 2012.¹⁵ Charles Perrow argued that such failures are inevitable in complex, tightly coupled systems.¹⁶ AI, which is being integrated into power grids, air traffic control, and industrial automation, could trigger a similar crisis. Imagine an AI-driven stock trading algorithm triggering a financial collapse, or a failure in AI-managed air traffic control causing multiple plane crashes.

Much of the discussion about AI risk centers around Artificial General Intelligence (AGI)—AI that surpasses human intelligence and escapes human control.¹⁷ This post isn’t weighing in on the “AI doomer” debate about superintelligence. Whether you believe in the possibility of AGI or not, the kind of risks described above are more prescient. Real risks are already here, embedded in everyday AI applications. Worrying about AGI before addressing AI’s current safety failures is like debating self-driving cars before inventing seatbelts.

We do not need to wait for an AI catastrophe to take action. There are clear, pragmatic steps to ensure AI safety without stifling innovation. First, enforce existing safety standards in industries where AI is deployed. Medical devices, aviation software, and industrial automation already have rigorous certification processes—AI should not get a pass. Software in critical infrastructure must meet strict reliability benchmarks; generative AI should not be exempt from similar scrutiny. Second, mandate AI “seatbelts and airbags.” Just as the auto industry initially resisted safety features that saved lives, AI companies may resist regulations that impose guardrails. But explainability, reproducibility, and sector-specific safety requirements are essential. Third, regulate before the backlash. If AI failures mount without oversight, we risk an overcorrection—an AI Winter akin to the post-Three Mile Island freeze on nuclear power, where public fear stalls beneficial technological progress. History shows that industries resist safety reforms until disaster strikes. The AI industry still has time to prove it can be different.

Generative AI and LLMs are amazing! They show obvious value. Just as the takeaway from the Therac-25 incident should not be that software is inherently bad, the takeaway from this post should not be that AI is inherently bad. But both need to be used safely when potential harm is at stake.

Oh, and Three Mile Island? It’s reopening in 2028—with a 20-year commitment from Microsoft to power its AI data centers.¹⁸