Systems Control Bengio

AI Pioneer Warns of Lost Control as Deployments Outrun Understanding

Yoshua Bengio — one of the three researchers whose deep learning breakthroughs earned the Turing Award and made modern AI possible — warned this week that the world is building AI systems 'we don't know how to control.' Bengio, who has remained in academia without financial stakes in AI commercialization, described fundamental gaps in understanding how these systems make decisions, not merely risks of deliberate misuse.

Anthropic provided the most concrete validation of that warning, announcing that AI could 'soon build its own successors' — an acknowledgment that it is actively developing systems capable of writing and improving their own code without human intervention. Separately, Anthropic confirmed it has embedded engineers at the National Security Agency to support a deployment referred to as 'Mythos,' suggesting capabilities extending well beyond consumer applications. Anthropic's president also acknowledged that AI training costs drove the company's IPO filing, framing the capital raise as a necessity to sustain the pace of development.

President Trump's AI oversight executive order, which sent AI software stocks into an extended decline, appears responsive to these concerns; the Cybersecurity and Infrastructure Security Agency is preparing directives to implement it. Google's launch of Gemini Avatar, which uses the new Omni model to map a user's face and voice within minutes and generate realistic video clones, illustrated how rapidly AI capabilities are moving into territory that could destabilize identity verification and authentication systems. Cloudflare reported that bots now generate more web traffic than humans — a threshold that reshapes assumptions underlying advertising models, cybersecurity, and measurement of online activity.

Microsoft CEO Satya Nadella acknowledged the company 'overcorrected on OpenAI reliance,' conceding that dependence on external AI capabilities the company cannot fully direct has become a strategic vulnerability. Meta's AI chief, Alexander Wang, told Bloomberg's Tech conference that personalized health tools are key to making Meta's AI useful at scale — raising the stakes considerably, given that health recommendations constitute life-or-death applications of technology its builders admit they do not fully understand. The central challenge, as Bengio frames it, is not merely preventing misuse but ensuring alignment: building systems that optimize for human values rather than finding unintended ways to satisfy their training objectives.