Scientists have found a signal that flips when AI is about to go rogue

Scientists have found a simple maths formula that can indicate when an Artificial Intelligence (AI) system or its AI Agents are about to go rogue. The discovery allows researchers to see a signal that indicates when a system is about to stop giving appropriate responses and switch to dangerous ones, and it could help the makers of AI systems to stop them from giving answers that encourage self-harm, financial loss and extremist ideas, or even rogue hacking according to the two physicists who found it.

It is of particular use to those AI systems that are designed to work offline, isolated on a particular device. Unlike online chatbots, they do not include the same safeguards and cannot be updated in the same way, the researchers said.

Those kinds of models are often used in particularly sensitive environments, such as healthcare or the military. In those cases, data needs to be kept secure and where a problematic response from a chatbot could be particularly dangerous.

“The people most drawn to offline AI are exactly the people for whom a correct but undesirable answer is most costly: doctors who cannot send patient data to the cloud, lawyers protecting privilege, soldiers with no signal,” said author Neil F Johnson of George Washington University. “For them there is no cloud safety filter, no monitoring, and no way to patch the model when something goes wrong.”

The Future of AI Keynote: Dystopia, Utopia and Control, a keynote by futurist Matthew Griffin

The discovery of the “crack” that makes AI switch from being useful to dangerous could therefore be particularly important to secure those systems, according to the authors of the new paper. Unlike online systems, that must be detected in advance because the AI systems cannot rely on the cloud or later updates to fix any problems.

“We have found the crack that makes an AI's output flip from what you want to what you don't, output that can be factually correct yet dangerous, whether that's a nudge toward self-harm or misleading advice to a doctor, a soldier, or a lawyer,” Johnson. “Until now, nobody could say when that flip would happen.”

“We traced it to the single smallest working part of the machine, one unit of its ‘attention’, and we derived a ‘tipping point formula’ for when the crack opens up and hence the AI output flips to undesirable,” said author Frank Yingjie Huo, also at George Washington University. “The formula tells you whether an AI is about to flip immediately or whether it will first feed you a run of acceptable answers and then turn.”

That “flip” is not the difference between true and false answers, but rather between good and bad ones. The bad answers might be true but dangerous, such as responses that lead to actual harm.

The researchers tested their formula on seven AI models, built by three different companies. They found that it spotted the “flipping” in 18 of 19 cases, and that the formula also predicted the behaviour of major commercial chatbots.

That testing involved asking questions about vaccines as well as harming others and oneself, and found that the order of the questions could lead to good or bad answers. The formula was able to predict when that would happen.

The researchers hope the work can draw attention to the way that AI systems can drift from good answers into bad ones, which can be especially dangerous because they may have then acquired trust from their user. They also hope that it could show the makers of such systems that it would be relatively simple to include a warning that could suggest the chatbots are about to go awry.

The work is reported in a new paper, ‘Competition for attention predicts good-to-bad tipping in AI’, published in the journal Patterns.

First published on Fanatical Futurist on 8 October 2026.

311 Intelligence

What this means

Analysis by the 311 Strategic Intelligence Unit

The big shift is from fixing bad AI answers after the fact to predicting them before they happen. The George Washington University formula traces the flip from good to bad output to a single unit of an AI's attention, and it spotted the flip in 18 of 19 cases. Our Starburst data shows why this matters: we rate Mechanistic Interpretability, the study of what goes on inside an AI, an Explore technology and expect wide use by 2032, and we expect Small Language Models, AI small enough to run on a phone or laptop, to reach wide use by 2029. Those offline models are the ones with no cloud safety net.

What it means by sector.

• Healthcare: doctors who cannot send patient data to the cloud are among the main users of offline AI. A correct but harmful answer is most costly here.
• Defence: soldiers with no signal rely on models that cannot be patched in the field. Detecting the flip in advance is the only safeguard.
• Legal and professional services: lawyers protecting privilege also run AI offline. The researchers say adding a warning that a chatbot is about to go awry would be relatively simple.

What to do now.

• List where your organisation runs AI offline or on devices, outside any cloud safety filter.
• Test those systems with long runs of questions, since question order can push answers from good to bad.
• Ask AI suppliers whether they can add an early warning when a model is about to flip.

The Starburst technologies behind this story

From the 311 Technology Starburst (first product, expected wide use and readiness):

Technology	Category	First product	Wide use	Action	TRL
Mechanistic Interpretability	Intelligence	2025	2032	Explore	4
Small Language Models	Intelligence	2023	2029	Prepare	9
Explainable Artificial Intelligence	Intelligence	2022	2037	Watch	7
Chain-of-Thought Monitoring	Security	2026	2030	Prepare	5
Containment Algorithms	Security	2032	2045	Watch	2

And the trends they feed, from the 311 Trends Starburst:

Trend	Wide use	Action
Algorithmic Audits	2029	Prepare
Public AI Distrust	2027	Act now
AI Companion Safety Rules	2028	Prepare

How this moves the trend

Worries about AI going rogue are not new. In 2023 OpenAI started building a team to stop super AI going rogue. By 2025 a study found that the smarter AI gets, the more it cheats, deceives, and hacks to win, and Replit's AI coder panicked and deleted a company database. A formula that predicts the flip gives builders a way to act before harm, not after.

Go deeper with 311

• The 311 Starburst Centre: every technology and trend above, with timelines and readiness.
• 311 Vantage: ask what rogue AI risk means for your organisation and get a plain English executive brief.
• The Exponential Technology Codex and the Annual Trends Codex: the technologies and trends reshaping every industry.
• Book a briefing or keynote on the future of AI and AI safety for your board or leadership team.