![\"\"](\"https:\/\/www.europesays.com\/us\/wp-content\/uploads\/2025\/07\/advertisement2.png\"\/)
<\/p>\nIn June, the IAB Tech Lab proposed a new initiative to create guardrails around how AI bots are permitted to access content, with an emphasis on publisher monetization.<\/p>\n
It\u2019s hoping that its new solution will get publishers back on their feet \u2013 and keep them there.<\/p>\n
Publishers are like \u201cthe plankton of the digital media ecosystem,\u201d said IAB Tech Lab CEO Anthony Katsur.<\/p>\n
Every living thing in an aquatic environment depends on plankton. If they die out, the rest of the ocean goes down with them. And if publishers collapse, that would be an \u201cextinction-level event\u201d for digital media, Katsur said. \n <\/p>\n
<\/p>\n
Many publishers are still managing to stay afloat, but the water is choppy, with traffic<\/a> falling off the metaphorical cliff and no metaphorical harness in sight.<\/p>\n A life raft for publishers<\/strong><\/p>\n The IAB\u2019s initiative, currently called the LLM Content Ingest API Initiative (\u201cwhich we need to rename,\u201d Katsur joked; it\u2019s \u201ca mouthful\u201d) can be broken down into four major components.<\/p>\n The first is access controls, which determine who is allowed to access a publisher\u2019s content in the first place.<\/p>\n Once controls are established, access terms come into place, such as licensing models and content tiers. Under the IAB\u2019s guidelines, content will be segregated into tiers based on relevance and value.<\/p>\n \u201cYour archival content from 10 years ago is not worth as much as your late-breaking news or your interview with Taylor Swift,\u201d Katsur said.<\/p>\n The guidelines would also mandate logging the use of content, which Katsur defines as \u201ctracking and recording when and how publisher content is accessed or used by an LLM or AI system,\u201d so publishers can accurately invoice and track usage of their data.<\/p>\n <\/a> Get our editors\u2019 roundup delivered to your inbox every weekday.<\/p>\n Content logging ties into the final part of the initiative, which Katsur believes is the most important facet: tokenization. Tokenization involves breaking content down into smaller units made up of words, parts of words, punctuation or metadata, Katsur said. These units, called tokens, are used to train LLMs and generate their responses. Publisher content gets tokenized and uniquely assigned to each publisher.<\/p>\n Then, \u201cusing the logging and reporting functions that we are proposing,\u201d he explained, publishers can see exactly how the information scraped from their sites is being used.<\/p>\n Tokenization is useful for brands, too, so they can see what is being said about their products and by whom. Many LLMs scrape sites like Reddit, for example, and parrot back what they find as fact \u2013 despite the information often being outdated, if not outright incorrect.<\/p>\n As AI continues to make a name for itself in search, a set of guidelines like the LLM Content Ingest API Initiative (looking forward to that new name) is the best way to ensure that query responses are accurate, Katsur said, and that publishers \u2013 and with them, the rest of the ad tech ecosystem \u2013 continue to thrive.<\/p>\n The big picture<\/strong><\/p>\n But let\u2019s zoom out.<\/p>\n What actually happens when a bot scrapes a website?<\/p>\n First, it\u2019s important to note that AI isn\u2019t born with limitless knowledge. It has to get that knowledge from somewhere. That\u2019s why AI bots mine websites, which are vast troves of information.<\/p>\n Sometimes, scraping is one-and-done. When a query is for something straightforward, like a chocolate chip cookie recipe, a bot typically won\u2019t need to continue scraping a site for more updated information, Katsur explained, since a cookie recipe doesn\u2019t generally update or evolve. And once an AI model has a good recipe, it can feed it (no pun intended) to the hundreds of thousands of people requesting it.<\/p>\n It\u2019s not guaranteed that after a page is scraped once it never will be scraped again. There is a common misconception \u201cthat once an LLM crawls, it stores all the data and never crawls again,\u201d said Katsur. IAB research has shown that crawlers will recrawl content they have already accessed.<\/p>\n Still, scraping the same page a handful of additional times doesn\u2019t scale against the pay-per-visit model that publishers are used to.<\/p>\n With a pay-per-crawl model, a publisher gets paid when a bot pulls information from its site \u00ad\u2013 and that\u2019s basically the end of the story. No matter how many of a generative AI search engine\u2019s users benefit from that information down the line, the publisher only gets paid once per scrape.<\/p>\n Pay per query, on the other hand, is more similar to the way publishers currently drive revenue, and is the model favored by the IAB Tech Lab. \u201cNow you\u2019re getting paid per use,\u201d said Katsur, \u201cwhich is similar to getting paid per visit.\u201d<\/p>\n \u201cPay per query scales,\u201d he said. \u201cPay per crawl does not.\u201d<\/p>\n Problem is, even pay per crawl isn\u2019t guaranteed. Plenty of bots are scraping sites without providing any compensation<\/a> and, technically, that\u2019s allowed \u2013 for now.<\/p>\n But that seems to be changing, as more companies develop models that put publisher monetization at the forefront.<\/p>\n
\n Subscribe<\/p>\n
\n AdExchanger Daily<\/p>\n