‘World War 11’ and other reasons the IWM needed AI

Image:

AI has saved the IWM “20 person years” of time

Switching from human transcription to AI hasn’t just saved the Imperial War Museum time – it’s enabled a new approach to research using its digital archives.

The Imperial War Museum was founded in 1917 with one goal: to record the UK and its Empire’s civil and military war effort during WW1.

The Museum’s remit has grown significantly since then – there have been more wars, for a start – and its archive has expanded even faster.

The IWM found itself sitting on a massive collection of oral histories: more than 20,000 hours of voice recordings taken between 1945 and 2000. That is itself only about two-thirds of the total oral collection; the rest is still being digitised.

In the digital age museum artefacts, especially those with an audio or video component, need to do more than sit on a shelf. Interactivity boosts engagement, and more importantly can help fit recordings into a wider historical context, with huge boons for research.

The IWM has been working to transcribe its recordings for several years, but it’s been slow going.

“Initially we used human transcription, and then we used digital transcription. For the first time, for this project, we worked with Capgemini and Google to use AI transcription,” IWM’s director of digital transformation and engagement, Nick Hodder, told me when we met at Google Cloud London this month.

“We think it’s [saved us] over 20 person years of audio transcription…in a matter of weeks.”

Although a person checks each recording, the AI transcription’s accuracy rate of 99% is “significantly better than a human,” Nick notes.

“It really changes how you conduct research”

A transcript is a simple text-based version of a recording. You can search it for basic information – “the names of ships or the names of people or particular dates or a particular battle” – but it’s very limited as a research tool.

Adding Google’s Gemini AI has changed the game. Now, historians can dig much more deeply.

“You can use an LLM to say, ‘Was this person scared at any point? Do they have any funny stories?’… ‘Why might that person have been scared?’ or ‘What happened next after the funny story?’

“It really kind of changes how you conduct research…[and] one of the biggest takeaways is it’s transformed our thinking about how people might access collections and museum collections in the future.”

So far the system has transcribed more than 8,200 interviews across nearly 46,000 separate recordings and 15 languages. Navigating those, even with AI’s help, is no small task.

That’s where tags, or entities to use their official term, come in. The LLM assigns these when it recognises a key point in a recording, like “World War 2,” “D-Day” or “Purley,” to make it easier to find related artefacts.

The IWM has its own list of about 400,000 entities, but many are synonyms (World War 1, First World War). Running an AI over that would reduce the list by “about 50 or 60%”.

“A surprising amount of our search traffic comes from people Googling World War 11,” Nick notes.

Reimagining research sounds great, and interactive transcripts could completely change how the public uses the IWM’s digital archive, but there is one issue: cost. As a non-profit, the museum has limited resources but a large following: its website records 20 million visitors every year.

“When we make a product accessible, often, a lot of people come and use it straight away – and with an LLM, you’re paying for every for every prompt… In retrospect, building it was probably the cheapest thing [we’ll do].”

Talks are ongoing with Google to find a solution, but as a stopgap the IWM is being flexible with the models it uses: Gemini 1.5 if Gemini 2 is “overkill”, 2.5 or 2.5 Pro if 2 “isn’t cutting it.”

For now, balancing cost with ambition remains a challenge. But what good is a world-class digital archive if it’s not being used? AI might not be a perfect answer, but it’s a promising tool in making sure these voices are heard again.