Even the king of England has his hesitations about AI

King Charles hosted a private summit Thursday with some of the most prominent names in AI and the U.K. government, including Nvidia’s Jensen Huang, leaders from OpenAI and Anthropic, the U.K.’s new AI minister Kanishka Narayan, and even the head of the U.K.’s foreign intelligence service. Paolo Benanti, who advises the pope on AI, was also there.

The king’s remarks at the summit — held at Dumfries House, an 18th-century Ayrshire estate owned by King Charles’s charitable foundation — were notable because the royal family is usually careful, for constitutional reasons, about veering too deep into political issues — and AI is one of the most hot-button issues in the world right now. The king urged the leaders in the room to find a way to control AI “before it’s too late,” adding that the fast development of this technology — and its “substance” — was “both intriguing and deeply concerning.” 

“AI is already showing immense capacity to improve and save life,” he said, citing medicine and science as an example. “Yet those who have created these technologies are now increasingly warning that AI risks developing darker capacities, perhaps even to take life.” This remark came just days after an Anthropic researcher quit, writing online that AI companies were “racing to build machines that are much smarter than any human” and that humanity “may not survive this.” 

The king continued, saying it’s urgent to properly consider the “existential dangers” of what would happen if this technology fell into the wrong hands and was used in “potentially catastrophic ways.” 

“Surely, we need sufficient means of control before it’s all too late?” he said. As King Charles wrapped up, he urged leaders to find a way to harness the good of AI and to build international cooperation and consensus to achieve that outcome. (This remark comes after Anthropic shut off access to its Fable and Mythos models to only U.S. customers, shocking the Euro tech scene and leaving many to think about the concept of AI sovereignty.) 

“The task before you is not to merely advance technology, but to ensure that it remains firmly in the service of humanity, community, and the natural world,” he said as he finished his speech. “As we embrace what is new, we must not lose what is timeless.” 

Outlets report that he stayed for around 20 minutes, leaving the guests to discuss these matters among themselves. For the king’s summit, OpenAI sent its CFO Sarah Friar while Anthropic sent Tino Cuéllar, who leads global affairs. When reached for comment, Cuéllar said the company was honored to have taken part in the discussion and that it was grateful to the king for “convening industry leaders and voices from across society to discuss the role of AI in serving the public good.” 

This is not the first time King Charles has spoken about the challenges and benefits of AI, having done so in 2023 and 2024. This summit, however, is especially timely as AI continues to dominate every industry, topic, and form of conversation. This month, OpenAI’s Sam Altman, Elon Musk, and Anthropic’s Dario Amodei called for a deceleration in AI development, but Nvidia’s Huang and President Donald Trump quickly rejected those calls. “We’re leading China in AI,” the president said. “We’re the most sophisticated country in the world, and frankly I want to keep it that way because whoever wins AI, wins.” 

It shows how complex the matter has become, as the interests of Big Tech, politics, and society collide. For the past year, OpenAI and Anthropic have teased blockbuster IPOs, though OpenAI recently said its IPO plans were stalled this year as concerns over AI safety persist. 

Topics