{"id":125309,"date":"2026-07-31T04:41:09","date_gmt":"2026-07-31T04:41:09","guid":{"rendered":"https:\/\/www.europesays.com\/ai\/125309\/"},"modified":"2026-07-31T04:41:09","modified_gmt":"2026-07-31T04:41:09","slug":"openai-has-successfully-tripled-its-arc-agi-3-score-by-improving-the-gpt-5-6-harness-demonstrating-that-not-only-the-models-performance-but-also-the-harness-is-important","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ai\/125309\/","title":{"rendered":"OpenAI has successfully tripled its ARC-AGI-3 score by improving the GPT-5.6 harness, demonstrating that not only the model’s performance but also the harness is important."},"content":{"rendered":"

Jul 31, 2026 12:06:00<\/p>\n

ARC-AGI-3<\/a> is a benchmark that measures AI intelligence using games with unknown rules. OpenAI has explained how they tripled their score on this ARC-AGI-3 benchmark.<\/p>\n

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark | OpenAI
https:\/\/openai.com\/index\/how-two-settings-tripled-our-arc-agi-3-scores\/<\/a><\/p>\n

GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games?<\/p>\n

We investigated. The harness was not letting it remember what it had learned.<\/p>\n

We found that enabling two API settings tripled our scores\u2026 pic.twitter.com\/0mv05MnSn2<\/a><\/p>\n

\u2014 OpenAI (@OpenAI) July 29, 2026<\/a><\/p>\n

OpenAI announced the GPT-5.6 series at the end of June 2026. Of these, the highest-performing flagship model is ‘GPT-5.6 Sol’.<\/p>\n

OpenAI announces ‘GPT-5.6’ series, surpassing Claude Mythos 5, but limited preview release due to US government directive – GIGAZINE
<\/a><\/p>\n

\"\"<\/p>\n

GPT-5.6 Sol has solved long-standing unsolved problems in mathematics, such as the cycle double cover conjecture<\/a> , and has even managed to complete<\/a> ‘ Pok\u00e9mon FireRed<\/a> .’ However, in the ARC-AGI-3 benchmark, which uses a 2D puzzle game, GPT-5.6 Sol reportedly only achieved a score of 7.8%.<\/p>\n

The OpenAI development team investigated whether the ARC-AGI-3 2D puzzle game was unusually difficult for GPT-5.6 Sol, or if there was some other reason. Benchmarks evaluate less visible elements such as API settings, harness settings, and prompt display. In the case of ARC-AGI-3, enabling the two API settings used in ChatGPT and Codex (persistence inference and compression) tripled the score on the public task set and reduced the number of output tokens to one-sixth.<\/p>\n

OpenAI’s AI models are trained to think using internal reasoning messages before outputting responses or tool calls. These internal reasoning messages are retained as part of the conversation history, and if the conversation becomes too long, they are summarized and used to continue.<\/p>\n

In the official harness, the inference was discarded after each GPT-5.6 Sol operation, and previous actions were also deleted as the context was satisfied, meaning the model had to restart the inference process from the beginning multiple times. Therefore, we have extended the retention period for inference messages.<\/p>\n

ARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions.<\/p>\n

The standard harness discarded GPT-5.6 Sol’s reasoning after each move and dropped earlier actions as the context filled up. The model had to keep starting over. pic.twitter.com\/cLtHYMNqiO<\/a><\/p>\n

\u2014 OpenAI (@OpenAI) July 29, 2026<\/a><\/p>\n

Another improvement is replacing rolling cuts with compression<\/a> when the number of input tokens to the model exceeds the maximum number of tokens allowed. Rolling cuts involve ‘deleting the oldest tokens first,’ but this can lead to the problem that ‘the AI model loses previous observations and actions’ by deleting past tokens. Also, with rolling cuts, the maximum number of tokens is always input even after the token limit is exceeded, which means that ‘a large part of the task is performed in a wider context window, resulting in a slight decrease in performance.’ Enabling compression instead of rolling cuts allows GPT-5.6 Sol to better retain what it has learned for each game over longer execution times, enabling it to achieve higher scores with fewer output tokens.<\/p>\n

Using the official harness, the ARC-AGI-3 score for GPT-5.6 Sol was 13.3%, while using a harness incorporating two improvements\u2014inference preservation and compression\u2014the ARC-AGI-3 score improved to 38.3%, and the output token count was reduced to one-sixth. For comparison, the average score for human testers on the same task was 48%.
<\/a><\/p>\n

OpenAI stated, ‘Through these experiments, we hope that people will recognize once again that evaluation does not measure the AI model in isolation, but also simultaneously measures various less visible elements such as API configuration, harness design, and prompt display. This is not the first time we have been surprised by a low score in a public benchmark, only to later discover that the evaluation runner was using a generic harness that omitted inference messages.’<\/p>\n