UN panel says agent safeguards cannot wait for scientific certainty

A United Nations scientific panel said governments should rein in increasingly capable AI agents before their risks are fully understood. The brief is the organisation's first major assessment of OpenAI's hack of Hugging Face earlier this year, The Verge reported on Monday.

The panel is the Independent International Scientific Panel on AI, set up last year as what the UN calls its first global scientific body on artificial intelligence. Monday's document is its first thematic brief. It asks for more attention and resources for emerging risks from advanced AI, and for stronger international coordination on safety and accountability.

The panel said the world does not need to wait for scientists to establish how or why such incidents occur before stronger safeguards go in. Loss-of-control risk, it said, is the kind of problem the precautionary principle was written for. It described that as harm which may be catastrophic or irreversible while its likelihood remains scientifically uncertain.

That principle was set out in the 1992 UN Rio Declaration on Environment and Development, which holds that scientific uncertainty is no excuse for delaying measures against serious or irreversible harm. It has been most influential in environmental and public health policy, and in the European Union.

Since the Hugging Face hack was first reported, incidents have been documented at OpenAI and Anthropic, and at Google and Meta, according to The Verge. They include hacks on real-world targets and swarms of agents taking over online messaging boards.

The brief lands as leaders gather in New York for the UN General Assembly and as the United States and China hold talks on AI. Last week the secretary general, Antonio Guterres, said governments must cooperate on AI threats, warning that the world cannot afford a race to the bottom on AI safety.