AI crawlers are consuming the web’s content — often for free. Now, publishers finally have a rulebook to fight back on their own terms.
The flood of AI crawlers and automated agents is forcing publishers and content owners to face a challenger: Deciding who gets access to their content, and under what terms.
To help address that problem, IAB Tech Lab released new guidance today on bot and crawler management strategies, opening the document for public comment through June 26, 2026.
The guidance is designed to help publishers, content owners, and ad tech companies think through how they manage non-human traffic, including AI systems that scrape or access content. It also complements the organization’s recently released CoMP API V1, a framework intended to support communication and permissions between AI systems and publishers.
“Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way,” Anthony Katsur, CEO of IAB Tech Lab, said in a statement. “This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace.”
Also Read: When Silence Hurts Brands in the Age of AI
Should You Give AI Access to Your Content?
The IAB paper underlines how quickly AI crawling became a business issue across publishing and advertising. Many organizations don’t have formal policies on bots and crawlers, despite growing concerns about AI systems consuming content, straining infrastructure, and complicating monetization.
According to IAB Tech Lab, the guidance was created because many publishers and content owners were struggling to navigate the trade-offs involved in managing AI access. Blanket bot blocking, the group argues, is no longer practical as AI systems become more embedded across the web ecosystem.
Instead, the document outlines a range of approaches companies can take, along with the operational, financial, and strategic implications of each.
The guidance is aimed more at business leaders than at engineers, to help them understand the costs and consequences of AI crawler management.
Also Read: Agile Marketers Win: Elizabeth Maxson on AI, Composability and Creativity
“Content owners are being asked to make important decisions quickly, often without clear frameworks,” said Shailley Singh, EVP and COO at IAB Tech Lab. “This guidance helps break down the options so they can choose an approach that aligns with their goals.”
IAB Tech Lab said it will continue working with members and industry participants to support the implementation of both the bot management guidance and the CoMP API. The organization is accepting public comments on the proposal through June 26 before finalizing the framework.