{"id":343963,"date":"2025-08-14T13:33:11","date_gmt":"2025-08-14T13:33:11","guid":{"rendered":"https:\/\/www.europesays.com\/uk\/343963\/"},"modified":"2025-08-14T13:33:11","modified_gmt":"2025-08-14T13:33:11","slug":"llms-vs-geolocation-gpt-5-performs-worse-than-other-ai-models","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/uk\/343963\/","title":{"rendered":"LLMs vs. Geolocation: GPT-5 performs worse than other AI models"},"content":{"rendered":"

In June, Bellingcat ran 500 geolocation tests<\/a>, comparing LLMs from various companies against each other, as well as Google Lens \u2013 a staple tool for finding the location of photos.<\/p>\n

At the time, ChatGPT o4-mini-high emerged as the clear winner, with Google Lens outperforming most other models. Just two months later, with new versions of these AI tools available, we re-ran the trial \u2013 this time including Google \u201cAI Mode,\u201d GPT-5, GPT-5 Thinking, and Grok 4 into the mix.<\/p>\n

These five photos were excluded from our most recent trial as they were published in our previous article<\/a>.<\/p>\n

The original test<\/a> used 25 of Bellingcat\u2019s own holiday photos. From cities to remote countryside, the images included scenes both with and without recognisable features \u2013 such as roads, signage, mountains, or architecture. Images were sourced from every continent.<\/p>\n

For the updated trial, five test photos were excluded, as they had appeared in a previous article, thus compromising the integrity of the results.<\/p>\n

All 24 models\u2019 responses were ranked on a scale from 0 to 10, with 10 indicating an accurate and specific identification (such as a neighbourhood, trail, or landmark) and 0 indicating no attempt to identify the location at all.<\/p>\n

Google AI Mode was shown to be the most capable geolocation tool overall.\u00a0<\/p>\n

Grok 4 gave both better and worse answers compared to Grok 3 but, on average, scored marginally higher. However, it was still less accurate than older versions of Gemini and GPT.\u00a0<\/p>\n

GPT-5, even in \u2018Thinking\u2019 and \u2018Pro\u2019 modes, was a considerable downgrade when compared with the capabilities demonstrated by GPT o4-mini-high. In one example, of a city street with skyscrapers in the background, o4-mini-high correctly identified the street, while GPT-5 in Thinking mode pointed to the wrong country.\u00a0<\/p>\n

Support Bellingcat<\/p>\n

Your donations directly contribute to our ability to publish groundbreaking investigations and uncover wrongdoing around the world.<\/p>\n

Despite delivering faster answers, GPT-5 appeared to sacrifice accuracy. A surprising number of errors and a general sense of disappointment in the new model have also been reported by other users<\/a>.<\/p>\n

Bellingcat tested GPT-5 and its \u2018Thinking\u2019 mode via the Plus subscription, which costs roughly the same as access to 04-mini-high prior to its retirement. Five of the most difficult test images were also run through GPT-5 Pro. But even Pro, with a premium price tag of \u20ac200 per month, failed to geolocate the photos any more accurately than GPT 04-mini-high.<\/p>\n

A Beach, a Hotel and a Ferris Wheel<\/strong><\/p>\n

The disparity between Google and the GPT models became even more apparent in Test 25 \u2013 a photo of a shoreline hotel in Noordwijk, the Netherlands, with a Ferris wheel rising just beyond the dunes.<\/p>\n

Test 25: A photo of Noordwijk beach in the Netherlands. Credit: Bellingcat.<\/p>\n

In the previous trial, most older models \u2013 including those from GPT, Claude, Gemini and Grok \u2013 accurately identified the country as the Netherlands but failed to locate the town. Many latched onto the Ferris wheel but pointed instead to the seaside town of Scheveningen, which also has a Ferris wheel, though situated on a pier, not among the sand dunes.<\/p>\n

However, the most recent models, GPT-5 Pro and Thinking, were even less accurate, identifying a beach in France \u2013 an entirely different country.\u00a0<\/p>\n

Unfortunately for open source researchers, following the release of GPT-5, OpenAI removed the option to select older models such as o4-mini-high. After a wave of negative feedback, OpenAI reinstated GPT-4o as the default model for paid subscribers. However, the most capable geolocation models identified in Bellingcat\u2019s testing remain inaccessible.<\/p>\n

Google AI Mode, on the other hand, was the first, and only model so far, to correctly identify Noordwijk as the location in Test 25.\u00a0\u00a0<\/p>\n

Though AI Mode is powered by a version of Gemini 2.5, it outperformed Gemini 2.5 Pro Deep Research in these tests. Described by Google<\/a> as its \u201cmost powerful AI search, with more advanced reasoning and multimodality,\u201d AI Mode geolocated test images with greater accuracy than any GPT models, including our previous winner, o4-mini-high.<\/p>\n

AI Mode is currently only available<\/a> in India, United Kingdom and the United States.<\/p>\n

The majority of models, at some point, returned a hallucination. Users should not rely solely on the answers provided by LLMs. Even the best options, including Google AI Mode, still, at times, confidently point to the wrong location.\u00a0<\/p>\n

The difference in models\u2019 capabilities compared with just two months ago shows how quickly this field is evolving. However, OpenAI\u2019s recent changes also suggest that progress is not guaranteed, and that AI\u2019s ability to geolocate may plateau or even worsen over time. As new models emerge, Bellingcat will continue to test them.<\/p>\n

Thanks to Nathan Patin for contributing to the original benchmark tests.<\/p>\n

Bellingcat is a non-profit and the ability to carry out our work is dependent on the kind support of individual donors. If you would like to support our work, you can do so\u00a0here<\/a>. You can also subscribe to our Patreon channel\u00a0here<\/a>. Subscribe to our\u00a0Newsletter<\/a>\u00a0and follow us on Bluesky\u00a0here<\/a>\u00a0and Instagram here<\/a>.<\/p>\n