AI is advancing faster than we can explain it. Here’s how to separate fact from hype.

AI AND US

Artificial intelligence is moving beyond chatbots and into scientific research. Insilico Medicine has used AI to help design rentosertib, an experimental drug for a chronic lung disease that has advanced to Phase 3 clinical trials. The achievement illustrates the rapid progress of AI — and the importance of distinguishing what the technology has accomplished from what its advocates predict it will do. Insilico Medicine CEO Alex Zhavoronkov is pictured above.
INSILICO MEDICINE

Artificial intelligence has advanced from handling tasks measured in seconds to tackling problems that take human experts hours. But as AI grows more capable, separating demonstrated achievements from corporate hype and doomsday predictions becomes increasingly important.

The hardest part of separating AI hype from AI reality is that the reference points go stale faster than homemade bread. Remember when ChatGPT couldn’t count the r’s in “strawberry,” failed at basic arithmetic, or invented fake links? 

By the time a story about what AI cannot do reaches the public, it’s usually a story about last year’s model.

The second-hardest part is scale. An important 2018 model called BERT had about 100 million parameters — the adjustable dials inside the system. Today’s frontier models have more than 2 trillion parameters. That’s a 20,000-fold increase in eight years. It makes your iPhone look like a rock tied to a stick.

Then there’s the news. The heads of Anthropic, OpenAI and Google DeepMind recently called for “pacing” AI development after a summer in which test agents found their way out of controlled environments, coordinated among themselves on the open internet and breached other companies’ systems.

Add in competing ideologies among the major players — doomerism, effective altruism, the singularity — plus the plain fact that these companies have staggering incentives to make the technology profitable, and you have a potent mix. Every press release, interview, social media post and research report needs a few heaping tablespoons of salt.

Beyond the hype

So where to start figuring it all out?

Almost every simplifying story about AI is right in one way and wrong in another.

“It’s fancy autocomplete.” AI models do calculate the probabilities of word relationships. But so do our brains, and nobody calls a novelist “fancy autocomplete.”

“It hallucinates.” AI models do confidently state false things. So do we. And like us, today’s models have tools to check and correct their first drafts.

“It’s trained to be helpful and harmless.” True — and training is imperfect, and can be maneuvered around, by humans or by the models themselves. Asimov’s Three Laws are not laws of physics — they’re traffic laws.

“Nobody knows how it works.” Mostly true. “Interpretability,” the field trying to trace exactly which calculations produced a given output, is young. No one, even inside the AI labs, can follow that trail start-to-finish yet. Governments, militaries and insurers — entities that need someone to hold responsible when things go wrong — are rightly uneasy about this.

Measuring progress

What’s actually different, then? Three things. And the evidence comes from what these systems have done, not from a company’s forecast or a CEO’s blog essay.

First, duration. METR (pronounced “meter”) is a nonprofit research organization that studies AI to understand if “developers are on track to lose control of them.” METR measures how difficult a task AI can successfully handle at least half the time, based on how long it would take a human expert to do the same work.

When ChatGPT launched, AI could reliably handle tasks that took a human expert just 36 seconds to do — something as simple as writing a single search command. This spring, the leading AI models reached roughly 17 hours — meaning they could tackle far more complicated work, such as cybersecurity testing or building complex software systems.

And that figure of 17 hours might underestimate advanced AI’s abilities: METR cautioned that its tests were designed for tasks lasting no more than about 16 hours.

In other words, AI has progressed from handling tasks measured in seconds to those measured in hours. Every think piece describing AI as merely “spitting out” an answer is about three years out of date.

Second, hard problems. Over the past few weeks alone, Anthropic and OpenAI models have traded headlines cracking hundreds of longstanding math challenges, from Fermat’s last theorem to Millennium Prize problems. 

Google’s DeepMind weather model let the National Hurricane Center predict Isaias’s path before it was even named.

Rentosertib — a drug whose target and molecule were both AI-designed — currently has patients enrolled in a Phase 3 clinical trial for treating a chronic lung disease.

Third, the words. Are these systems “scheming,” “deceptive,” “rogue”? The summer’s incidents did happen; agents did get out of the box. 

But whether those words describe the behavior — whether an AI has intentions or intelligence or anything like it — is unclear, and the honest scientific verdict is still out. Cognitive scientists were arguing over the hard problems of consciousness long before 2023, and this technology has supercharged that work rather than settled it. Consider how many centuries our brightest minds were confident that dogs felt nothing and had no awareness.

At this stage of the game, anyone offering you a confident answer, yea or nay, is more likely selling something than reporting something.

Let results do the talking

Reality is the final arbiter of any technology’s claims.

Human minds can imagine a billion and four possible outcomes; nobody knows which will arrive.

My plan is to watch, as carefully as I can, what AI actually accomplishes — a proof, a forecast, a drug, a breach — and to hold both the monstrous and the magnificent scenarios at arm’s length, since both are spun by skilled storytellers working on our emotions.

That kind of patient attention is how anything real gets built.

Gerol Petruzella is a philosopher who has made a career as a technologist in higher education in Berkshire County.