important method that shapes Large Language Models (LLMs)<\/a> \u2014 reinforcement learning (RL).
At its core, ChatGPT and other LLMs use the massive amount of data fed into them to predict the next sequence of words and answer a user query.<\/p>\n\u201cDuring the model\u2019s learning process (known as \u201ctraining\u201d), the model might be tasked with completing a sentence like: \u201cInstead of turning left, she turned ___.\u201d Early in training, its responses are largely random. However, as the model processes and learns from a large volume of text, it becomes better at recognising patterns and predicting the most likely next word. This process is repeated across millions of sentences to refine its understanding and improve its accuracy,\u201d explains OpenAI.<\/p>\n
That said, an element of randomness persists, since there are no definitive answers to many questions, like the one mentioned above. Then comes another layer, of reinforcement learning, which involves an agent learning from its environment and making decisions based on \u201crewards\u201d set by its developer.<\/p>\n
According to IBM, \u201cBecause an RL agent has no manually labeled input data guiding its behavior, it must explore its environment, attempting new actions to discover those that receive rewards. From these reward signals, the agent learns to prefer actions for which it was rewarded in order to maximize its gain.\u201d<\/p>\n
Story continues below this ad<\/p>\n
While attempting to understand the goblin issue, OpenAI found that the model designed to encourage the Nerdy personality favoured the use of creature-related words during RL training. \u201cAcross all datasets in the audit, the Nerdy personality reward showed a clear tendency to score outputs to the same problem with \u201cgoblin\u201d or \u201cgremlin\u201d higher than outputs without,\u201d it found.<\/p>\n
It explained this system in the form of a feedback loop:<\/p>\n
Playful style is rewarded
\nSome rewarded examples contain a distinctive lexical tic
\nThe tic appears more often in rollouts
\nModel-generated rollouts are used for supervised fine-tuning (SFT)]
\nThe model gets even more comfortable producing the tic
\nOr, as OpenAI said, \u201cWe unknowingly gave particularly high rewards for metaphors with creatures. From there, the goblins spread.\u201d<\/p>\n
What does all of it mean, ultimately?<\/p>\n
The company called the incident \u201ca powerful example of how reward signals can shape model behavior in unexpected ways, and how models can learn to generalise rewards in certain situations to unrelated ones.\u201d<\/p>\n
Story continues below this ad<\/p>\n
Arguably, the keyword in OpenAI\u2019s earlier statement is \u201cunknowingly\u201d. On one level, the incident points out that a lot about how LLMs function, and how they arrive at their final products, is not fully known by their creators themselves.<\/p>\n
This matters for the process of developing and fine-tuning AI models, which is far from perfect or exact at this stage. And, it serves as a reminder that despite their prevalence and the push for AI to be incorporated in a range of domains, its systems are still very much a work in progress.<\/p>\n","protected":false},"excerpt":{"rendered":"In a somewhat unusual post earlier this week, ChatGPT makers OpenAI said that starting with GPT-5.1, their models…\n","protected":false},"author":2,"featured_media":26281,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[17569,580,15852,17570,17568,16286,157,17567],"class_list":{"0":"post-26280","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-openai","8":"tag-ai-chatbot-personalities","9":"tag-chatgpt","10":"tag-chatgpt-goblins","11":"tag-express-explained","12":"tag-how-are-ai-chatbots-trained","13":"tag-indian-express","14":"tag-openai","15":"tag-why-does-chatgpt-mention-goblins"},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/26280","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=26280"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/26280\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/26281"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=26280"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=26280"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=26280"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}