![\"Analyzing](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/v3_analysis_gpt5-5-opus4-7.png\")
<\/p>\n<\/p>\n
AI benchmarks can be incredible tools, but they usually only tell you if a model passed or failed. With ARC-AGI-3, however, we can see the thought process behind the score, not just the outcome.<\/p>\n
This week we went through 160 replays and reasoning traces from OpenAI\u2019s GPT-5.5<\/a> and Anthropic\u2019s Opus 4.7<\/a> attempting novel, long-horizon environments. The scores were just one data point, but the interesting story is how they achieved their score.<\/p>\n Today we\u2019re open-sourcing our analysis package.<\/p>\n * Scores tested with the semi-private dataset<\/p>\n With ARC-AGI-3 we can replay every action alongside the model’s reasoning to see where it formed a hypothesis, where it abandoned a correct one, where it locked onto a wrong idea and couldn’t let go.<\/p>\n We found 3 common failure modes:<\/p>\n ARC-AGI-3 as an analysis tool<\/p>\n ARC-AGI-3<\/a> is a series of 135 novel environments. Each was hand-crafted by a human to test the ability of AI models to adapt to novelty. Play them<\/a> yourself or watch our launch video<\/a>.<\/p>\n The test-takers, whether human or AI, are not given instructions on how to play an environment. To make progress they must:<\/p>\n Each environment is built without the cultural knowledge a model would usually lean on. This means the environments isolate abstract reasoning.<\/p>\n You can think of ARC-AGI-3 as the lowest common denominator across novelty, ambiguity, planning, and adaptation. These are the same demands real-world tasks make of agents.<\/p>\n ARC-AGI-3 Public Demo environments: ar25<\/a>, lf52<\/a>, sb26<\/a> ARC-AGI-3 was built with testing and model auditing in mind. Each AI run is recorded along with its reasoning traces. We\u2019ve had over 1,000,000 games<\/a> played on ARC-AGI-3 so far.<\/p>\n For this analysis we:<\/p>\n This led us to discover why each model passed\/failed, which failure modes were shared and which were unique to each model.<\/p>\n Failure mode 1: true local effect, false world model<\/p>\n The first failure mode we saw was the most dominant pattern. Models were able to perceive a local effect:<\/p>\n When I press ACTION3, this object rotates.<\/p><\/blockquote>\n but they weren\u2019t able to translate that into a world model:<\/p>\n ACTION3 rotates the object, and rotation controls which side gets a new value, so I should orient the object to match the target before acting.<\/p><\/blockquote>\n Put another way, the models don\u2019t fail because they don\u2019t observe anything…they fail because they can’t anchor their observation in a world model.<\/p>\n For example, Opus, when playing cd82, knew that ACTION3 rotated the container by step 4<\/a>, and by step 6<\/a> it saw ACTION5 pour\/dip paint but it never converted that into “orient bucket, then dip to recreate the top-left target.”<\/p>\n Or in cn04, Opus found a successful rotate-then-place interaction (this is the correct hypothesis, step 23<\/a>) but optimized for whole-shape overlap (incorrect) and fake top-row progress (step 60<\/a>).<\/p>\n The second failure mode came from an incorrect level of abstraction from a model\u2019s training data. Across the runs, the models repeatedly explained unfamiliar mechanics by mapping them to known games: Tetris, Frogger<\/a>, Sokoban<\/a>, Powder Toy, Flood-It, MiniGrid, CoinRun, Breakout, Pong, Boulder Dash, and others. While recalling abstractions from core prior knowledge is helpful in theory, the literal analogies from the model\u2019s training data hijacked action selection.<\/p>\n The problem is that a local visual resemblance becomes a full gameplay theory, then the model wastes actions testing the wrong affordances.<\/p>\n In cd82, GPT-5.5 anchored on sand\/physics\/Flood-It mechanics. ls20 became Breakout instead of key combinations.<\/p>\n The final failure mode we saw was that even if a model beat a level, that reward did not translate into further success. This shows us that beating a level is not the same as understanding it.<\/p>\n Two Opus runs make this especially clear. On ka59, Opus solved Level 1 in 37 actions<\/a>, but its working theory of the click (teleporting the active character) was wrong. Although it looked like a clean win, this was a coincidence between a misread primitive and a forgiving level.<\/p>\n When Level 2 demanded the real mechanic (shape-matching and pushing), Opus\u2019s mislabel theory hardened into “click each target to fill it<\/a>,” and the run never recovered.<\/p>\n ar25 shows the same pattern at a different abstraction level. Opus cleared Level 1 with a correct read of mirrored movement (step 4<\/a>), then in Level 2 actually discovered the new movable-axis mechanic (step 227<\/a>), but it still drifted into hallucinated rules of punching holes and needing a flip.<\/p>\n In both cases the Level 1 success masked a missing or distorted primitive, and the partial win became a confident scaffold for the wrong Level 2 strategy.<\/p>\n This also shows us that early level progression can be a noisy signal of comprehension. Without an explicit check on why the prior level was won, models will carry their misconception into the next level.<\/p>\n Opus 4.7: wrong compression, GPT-5.5: failure to compress<\/p>\n As we compare the runs of GPT-5.5 and Opus 4.7 we\u2019re able to see they failed in different ways. This is important because aggregate scores alone would hide this distinction<\/strong>.<\/p>\n Opus had the wrong compression, GPT-5.5 failed to compress.<\/p>\n Opus 4.7 is stronger at short-horizon mechanic discovery. On ar25 it identifies the mirror structure almost immediately and clears Level 1. On ka59 it reads the two-character, two-target layout and executes the short Level 1 sequence even with an incomplete world model.<\/p>\n The flipside is that Opus is also more likely to latch onto a false invariant and execute it aggressively. On cn04 it leans into a fake progress\/timer\/conversion theory and spends the opener click-fishing inside that story (step 60<\/a>). It does form a working theory, it’s just the wrong one.<\/p>\n GPT-5.5 sits at the other end. Its hypothesis generation is wider, which makes it more likely to articulate the right idea but less likely to turn it into a plan. On ar25 it names the mirror effect but keeps reopening the genre space, drifting through Tetris, Frogger, Pong, and Tower of Hanoi instead of committing to reflection. On ka59 it reaches the correct object ontology, two target outlines and a switchable second character, but never commits to it.<\/p>\n The difference comes down to compression. Opus compressed its observations into a confident-but-wrong theory. GPT-5.5 had difficulty compressing at all.<\/p>\n ARC-AGI-3 measures agent autonomy<\/p>\nModelARC-AGI-3 Score*Public Demo Replays<\/tr>\n \n GPT-5.5<\/td>\n 0.43%<\/td>\n Link<\/a><\/td>\n<\/tr>\n \n Opus 4.7<\/td>\n 0.18%<\/td>\n Link<\/a><\/td>\n<\/tr>\n \n
\n
\nFailure modes on ARC-AGI-3<\/p>\n\n
![\"Opus](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/Opus-4-7-cd82.png\")
Opus 4.7 understood that ACTION3 rotates the objects but failed to grasp the game concepts.
Opus 4.7 Playing cd82<\/a>. Game Score: 0%<\/p>\n![\"Opus](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/opus-4-7-cn04.png\")
Opus 4.7 Playing cn04<\/a>. Game Score: 0%
\nFailure mode 2: wrong level of abstraction from training data<\/p>\n![\"GPT-5.5](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/gpt-5-5-ls20.png\")
GPT-5.5 mistakes ls20 for other games. GPT-5.5 Playing ls20<\/a>. Game Score: 0%
\nFailure mode 3: solved the level, didn\u2019t learn the game<\/p>\n![\"Opus](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/ka59_opus.gif\")
Opus 4.7 gets stuck in a click-fishing loop. Opus 4.7 playing ka59<\/a>. Game Score: 2.04%<\/p>\n![\"Opus](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2026\/05\/opus-4-7-ar25.png\")
Opus 4.7 Playing ar25<\/a>. Game Score: 0.15%<\/p>\n