AI hallucination nearly triggers US military operation

A U.S. military operation against a Chinese vessel was aborted at the last minute this spring after officials discovered that the intelligence driving the mission was the result of an AI chatbot "hallucination," according to reports. The incident occurred when a Special Operations Command analyst queried an AI chatbot to synthesize open-source data with classified signals intelligence. The chatbot misidentified the ship’s cargo manifest, falsely claiming it carried components for a nuclear weapons program. The analyst then used the tool to format these erroneous findings into an official summary, which was circulated through command channels. The near-miss highlights the risks as the U.S. military races to integrate AI to accelerate decision-making. While the Pentagon views AI as a way to maintain a strategic edge, experts warn that the same speed that makes AI attractive can allow hallucinations to propagate without sufficient human oversight. Jake Steckler, a research scholar at GovAI, emphasized the need for service members to understand the uncertainty inherent in LLMs, particularly for decisions involving the use of force, intelligence analysis, or operational planning.