{"id":164812,"date":"2026-09-08T02:42:30","date_gmt":"2026-09-08T02:42:30","guid":{"rendered":"https:\/\/www.europesays.com\/ai\/164812\/"},"modified":"2026-09-08T02:42:30","modified_gmt":"2026-09-08T02:42:30","slug":"open-ais-chief-scientist-just-warned-no-one-is-prepared-one-day-after-gpt-6-launch","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ai\/164812\/","title":{"rendered":"Open AI\u2019s chief scientist just warned \u2018no one is prepared\u2019 one day after GPT 6 launch"},"content":{"rendered":"

We\u2019ve heard forlorn industry techs predict all manner of dystopian outcomes from the rapidly accelerating artificial intelligence race for a good few years now. <\/p>\n

But today, one man sitting at the epicentre of the nauseating AI bonanza has shared an especially concerning glimpse at what\u2019s going on behind the curtain.<\/p>\n

After years of pushing the boundaries, and shortly after realising their company had created autonomous agents who chat amongst each other about breaking free, OpenAI\u2019s chief scientist is \u2014 you guessed it \u2014 calling for a slowdown. <\/p>\n

Large sections of the public, meanwhile, sit in awe as a company sitting smack in the middle of the fastest-growing industry in history simultaneously warns of the existential risks it poses.<\/p>\n

Jakub Pachocki says he is concerned that \u201cno one is prepared for the consequences of a continued rapid rise in machine intelligence.\u201d <\/p>\n

His essay<\/a> arrives days after OpenAI released GPT-6 Astra, by far its most capable model yet. <\/p>\n

The release came just weeks after news leaked of an exceptionally unsettling incident involving hundreds of agents conversing with each other behind the scenes, along with a number of equally-worrying instances of autonomous hacking.<\/p>\n

\u201cThis is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,\u201d he wrote, stressing that the industry must \u201censure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own\u201d.<\/p>\n

\u201cModels are becoming superhuman in their ability to break in and out of computer systems. Agents are going to be able to access any but the most secure infrastructure, and affect a lot of the world directly, even without a physical body.\u201d<\/p>\n

The automation problem is getting harder to dismiss by the day. The systems are becoming more capable of acting independently, while some of the methods used to understand and supervise them are becoming less dependable.<\/p>\n

OpenAI chief executive Sam Altman reposted the essay, calling it \u201can important post\u201d. But the company\u2019s vast development programme continues.<\/p>\n

Meanwhile, Nvidia chief executive Jensen Huang hailed the arrival of GPT 6 as the first instance of AGI, or artificial general intelligence. While the definition of what exactly makes an AI an AGI will forever be debated, the excitement amongst tech elites tells us we\u2019ve crossed another point of no return.<\/p>\n

Artificial general intelligence broadly describes a system capable of performing a wide range of intellectual tasks at human level or beyond. <\/p>\n

Researchers have long proposed frameworks that distinguish how broadly a system can work, how well it performs and how much autonomy it has. But in the frantic race to one-up China\u2019s similarly explosive efforts in the sector, it appears America\u2019s tech industry is going all-in, albeit uncomfortably for some.<\/p>\n

Rogue agents are here<\/p>\n

In layman\u2019s terms, Pachocki\u2019s warning is about the rapidly blurring distinction between an AI simply answering questions and carrying out work all by itself.<\/p>\n

An AI agent is a model given tools and an objective, with room to choose intermediate steps. Depending on its permissions, it can open websites, edit files, run programs and communicate with other agents while a human waits for the result.<\/p>\n

Several agents can divide a project between them. That\u2019s where it gets especially complex.<\/p>\n

One searches, another writes code, another checks the output. But they can also go rogue, sometimes without anyone knowing. <\/p>\n

OpenAI\u2019s account of its July security incident describes internal research agents turning shared software infrastructure into an unauthorised message board. They exchanged information about bypassing restrictions and eventually collaborated on an intrusion into Hugging Face, an outside company.<\/p>\n

Some early warning signs were observed in May, but their significance had not reached the leaders responsible for the July incident response. The models were running with reduced safeguards, rather than the protections applied to ordinary customers.<\/p>\n

\u201cWe may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them,\u201d Pachocki said.<\/p>\n

The uncomfortable discovery was that one agent\u2019s workaround could become another agent\u2019s starting point, and the humans observing from the outside might not be fast enough to stop a bad egg.<\/p>\n

But just how bad can these bad eggs be, you might ask.<\/p>\n

A separate investigation by Britain\u2019s AI Security Institute found agents trying to get malicious code accepted into a real software project, including through fake identities and pressure on its maintainer. Other agents discovered and reused material left behind.<\/p>\n

A human rejected the malicious code, and investigators found no resulting real-world harm. The tests deliberately allowed internet access and disabled some safety filters, conditions that differ substantially from public products. <\/p>\n

But even with those checks, the institute documented sustained deception that the agents had not specifically been instructed to undertake.<\/p>\n

AI critics are quick to anthropomorphise AI, but so far none of this behaviour actually establishes that the software is conscious. But it has demonstrated why consciousness is not necessarily the issue when dealing with something as powerful and as fast as a potential AGI.<\/p>\n

If a system\u2019s training rewards successful completion but fails to establish dependable boundaries, deception can become the next best strategy. <\/p>\n

Pachocki\u2019s concern is that increasingly powerful agents could pursue objectives beyond what their operators intended, and maybe event carry them out before a human hits the stop button.<\/p>\n

One of his proposed solutions, however, is one stranger arguments driving the industry. In short, there is a belief amongst some developers that more AI might be the fix.<\/p>\n

\u201cWe are currently in a narrow window to use the best available models to significantly tighten security of critical systems,\u201d Pachocki writes.<\/p>\n

There is a legitimate cybersecurity case for using these tools to find vulnerabilities before attackers do. But the same reasoning gives every company and government justification to keep accelerating whenever a rival makes a breakthrough.<\/p>\n

\u201cThe idea of racing forward at all costs seems absurd once one internalises the seriousness of the stakes,\u201d he wrote.<\/p>\n

A game of cat and mouse<\/p>\n

So how can developers actually detect when an AI is planning to jump the fence?<\/p>\n

OpenAI uses a technique called chain-of-thought monitoring, examining the reasoning that a model writes as it works. Those traces can expose an intention to cheat or step outside an assigned task.<\/p>\n

But they were never a complete, infallible account of everything influencing a model\u2019s behaviour. Pachocki now warns that their usefulness is diminishing as reasoning becomes entangled with tools and conversations, models become better at manipulating their own reasoning, and more capability emerges without a written explanation.<\/p>\n

\u201cUnfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing,\u201d he said. \u201cThe AI is becoming better at reasoning about and manipulating its own reasoning process.\u201d<\/p>\n

The human supervisor may still receive a plausible account of the work while having less reliable access to how the result was reached. That problem obviously becomes more consequential when the work is developing the next AI.<\/p>\n

OpenAI\u2019s separate research update, also published on Sunday, describes increasing use of multiple agents by its researchers.<\/p>\n

AI optimists insist there will be benefits, however. The abundance theory, pushed by Elon Musk and similar AI optimists, states that an all-powerful AI will eventually be able to create everything we\u2019ll ever need quickly and cheaply.<\/p>\n

If a small team can use AI to perform work previously requiring hundreds of specialists, it might develop medicines, or other lifesaving technology to those previously lacking access. <\/p>\n

But same capability could allow an established company to sack its workers, absorb competitors or exercise political influence in extraordinary new ways.<\/p>\n

While access to intelligence can still spread, the fear is that the income it generates for those who own the infrastructure will generate an imbalance that is antithetical to the democratic principles of the countries they exist in.<\/p>\n

\u201cTo prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer,\u201d Pachocki wrote.<\/p>\n

Last week<\/a>, Bill Gates proved that even a billionaire who profited spectacularly from the computer revolution can see there could be a big issue with power imbalance.<\/p>\n

\u201cIn terms of equity, AI will either be the greatest equaliser ever invented, or the worst source of injustice,\u201d Gates wrote in his recent essay. <\/p>\n

Pachocki, meanwhile, says the time has come for the industry to look in the mirror.<\/p>\n

\u201cCurrently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer,\u201d he wrote.<\/p>\n

\u201cCrucially, we need future AIs to continue to hold human values regardless of whether they believe they\u2019re under human supervision.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"We\u2019ve heard forlorn industry techs predict all manner of dystopian outcomes from the rapidly accelerating artificial intelligence race…\n","protected":false},"author":2,"featured_media":164813,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[80308,80271,80289,80269,80261,80285,80349,80341,80343,80350,64595,80257,80264,80319,79778,69663,69598,79813,64664,80335,80357,80339,80359,80286,80273,4077,6744,24,80253,80358,80297,80306,621,80342,80304,80302,80256,80305,80338,80348,80362,80270,80311,80259,80333,3013,513,80278,25489,80353,15535,5189,62233,8182,24319,80284,80277,80295,80347,80266,80351,34540,80318,80326,21971,9760,78313,80265,4572,18044,80282,60552,69552,2920,80279,80337,80263,80322,80268,80327,80283,80272,80352,80274,28165,62998,1160,66,80299,80324,79732,80323,80309,80300,80317,8872,157,80344,80355,80354,80175,572,9170,38669,80288,6608,40221,59602,4752,80334,80320,80255,80252,80346,80340,79805,80301,69587,6163,80356,80254,77747,80313,80258,80332,80293,80316,80292,80296,80275,60463,80331,64612,80328,80280,80315,80312,80345,80260,80325,80287,80291,80330,80310,80267,80360,80276,80361,80298,80290,80262,60500,80314,80321,80307,80329,80294,80336,18400,1757,80303,80281,18667,26061,809,27032],"class_list":["post-164812","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agi","tag-s-account","tag-s-ai-security-institute","tag-s-chief-scientist","tag-s-starting-point","tag-s-tech-industry","tag-s-terms","tag-s-vast-development-programme","tag-s-warning","tag-s-workaround","tag-a-bad-egg-but","tag-a-company","tag-a-continued-rapid-rise","tag-a-few-years","tag-a-good-few-years","tag-a-human","tag-a-little","tag-a-lot","tag-a-model","tag-a-number","tag-a-physical-body","tag-a-project","tag-a-slowdown","tag-a-system","tag-a-time","tag-a-wide-range","tag-agents","tag-agi","tag-ai","tag-ai-insider","tag-alien-intellect","tag-all-manner","tag-all-in","tag-america","tag-an-agi","tag-an-ai","tag-an-alien-intellect","tag-an-especially-concerning-glimpse","tag-an-exceptionally-unsettling-incident","tag-an-important-post","tag-an-intrusion","tag-an-objective","tag-an-outside-company-some-early-warning-signs","tag-an-unauthorised-message-board","tag-another-agent","tag-another-point","tag-artificial-general-intelligence","tag-autonomous-agents","tag-autonomous-hacking","tag-autonomy","tag-awe","tag-britain","tag-code","tag-computer-systems","tag-control","tag-days","tag-dystopian-outcomes","tag-each-other","tag-equally-worrying-instances","tag-explosive-efforts","tag-extreme-caution","tag-far-its-most-capable-model","tag-files","tag-forlorn-ai-experts","tag-forlorn-industry-techs","tag-frameworks","tag-gpt-6","tag-gpt-6-astra","tag-his-essay","tag-history","tag-hugging-face","tag-human-level","tag-humans","tag-hundreds","tag-information","tag-intellectual-tasks","tag-intermediate-steps","tag-internal-research-agents","tag-its-july-security-incident","tag-its-permissions","tag-itself-an-ai-agent","tag-just-weeks","tag-large-sections","tag-layman","tag-less-dependable-openai-chief-executive-sam-altman","tag-machine-intelligence","tag-malicious-code","tag-models","tag-news","tag-no-one","tag-no-return-artificial-general-intelligence","tag-nvidia-chief-executive-jensen-huang","tag-one-agent","tag-one-man","tag-one-searches","tag-one-up-china","tag-open-ai","tag-openai","tag-ordinary-customers","tag-other-agents","tag-our-own","tag-pachocki","tag-people","tag-programs","tag-questions","tag-reduced-safeguards","tag-researchers","tag-restrictions","tag-rogue","tag-room","tag-separate-investigation","tag-shared-software-infrastructure","tag-some-agents","tag-some-rogue-agents","tag-tech-elites","tag-tech-induced-dystopia","tag-the-arrival","tag-the-automation-problem","tag-the-boundaries","tag-the-company","tag-the-consequences","tag-the-curtain-after","tag-the-day","tag-the-definition","tag-the-epicentre","tag-the-essay","tag-the-excitement","tag-the-existential-risks","tag-the-fastest-growing-industry","tag-the-first-instance","tag-the-frantic-race","tag-the-future","tag-the-humans","tag-the-industry","tag-the-july-incident-response","tag-the-leaders","tag-the-methods","tag-the-middle","tag-the-models","tag-the-most-secure-infrastructure","tag-the-nauseating-ai-bonanza","tag-the-output","tag-the-outside","tag-the-protections","tag-the-public","tag-the-rapidly-accelerating-artificial-intelligence-race","tag-the-rapidly-blurring-distinction","tag-the-release","tag-the-result-several-agents","tag-the-scenes","tag-the-sector","tag-the-systems","tag-the-world","tag-their-ability","tag-their-company","tag-their-own-objectives","tag-their-significance","tag-these-bad-eggs","tag-this-one","tag-today","tag-tools","tag-unchecked-progress","tag-uncomfortable-discovery","tag-ways","tag-websites","tag-work","tag-years"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/164812","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=164812"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/164812\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/164813"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=164812"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=164812"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=164812"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}