Nearly seven in ten South African news websites take no specific action to block AI crawlers from scraping their content, according to a new report — and the publishers that do act are overwhelmingly the largest and best-resourced.
The Protocol Gap: South Africa, published by the Media Leadership Think Tank at the University of Pretoria’s Gordon Institute of Business Science, the Journalism Relay Project and the International Fund for Public Interest Media, examined the robots.txt files of 263 South African news websites.
It found that while 74.1% have a robots.txt file, only 30.4% use it to block at least one AI crawler as of April 2026 — a modest rise from 29.7% in December 2025. Where publishers do act, they block an average of six bots each, with crawlers from Common Crawl, ByteDance, Google, OpenAI, Amazon and Anthropic among the most frequently restricted.
The gap inside the gap
South Africa’s rate is the highest in the three-country series. Brazil, covered in the first report in December 2025, showed just 7.2% of news sites blocking at least one AI crawler. Indonesia’s early figures stand at 13%, up from 5.8%.
But the report is explicit that the headline figure conceals a deeper imbalance. The capacity to restrict AI crawler access is concentrated among large, well-resourced publishers. Smaller, community and vernacular outlets remain largely exposed.
That distribution matters more than the aggregate. Blocking an AI crawler requires knowing the crawlers exist, knowing which user-agent strings to target, having someone able to edit a file at the site root, and keeping the list current as new bots appear. None of that is expensive. All of it requires technical capacity that a two-person community newsroom publishing in Sesotho or isiXhosa may not have.
The consequence is a compounding one. African-language journalism — some of the scarcest and most valuable text in existence for training models in low-resource languages — is being absorbed into datasets without consent, attribution or payment, by companies whose products serve those same language communities poorly. Publishers are supplying the raw material for systems that do not yet work for their readers.
What robots.txt can and cannot do
Robots.txt is a plain text file publishers place at their site root to signal which automated bots may access their content. It is one of the few tools available, and it is not legally binding. Compliance is voluntary.
It offers no protection against crawlers that simply ignore it. Its value is evidentiary: robots.txt has been cited in ongoing legal complaints against unauthorised scraping in Canada, the United States and the United Kingdom. A publisher that has signalled refusal is in a different position from one that has not.
In the absence of clear legal frameworks, the report argues, robots.txt has become a de facto governance tool — but one only some publishers can currently afford to use well.
A shifting policy backdrop
The findings land in an active regulatory environment. South Africa’s Competition Commission concluded its Media and Digital Platforms Market Inquiry with remedies aimed at strengthening publisher controls, having earlier recommended Google compensate local news media between $16.3 million and $27.2 million annually. The Copyright Amendment Bill was returned to Parliament for further amendment in June following a Constitutional Court ruling.
Internationally, the report points to the UK Competition and Markets Authority’s new requirements on Google Search and Brazil’s competition authority opening its own inquiry into Google’s conduct. Nigeria’s Federal Competition and Consumer Protection Commission is separately investigating global technology firms and generative AI platforms over the unauthorised use of Nigerian media content, following a complaint from the Nigerian Press Organisation.
Comparative evidence for the Global South
Together with the Brazil report and a forthcoming Indonesia study, the series is intended to give Global South publishers evidence of where they stand relative to each other rather than only relative to platforms based in the Global North.
The report was launched at a webinar moderated by MLTT director Michael Markovitz, with Irene Jay Liu of IFPIM, Sérgio Spagnuolo of the Journalism Relay Project and Ompha Tshamano of MLTT, followed by discussion with Daryl Dingley of the PSS publisher group and Sbu Ngalwa of Our City News and the South African National Editors’ Forum.
The underlying argument is that awareness is no longer the constraint. Publishers know their content is being taken. What most of them lack is the technical capacity and resources to do anything about it — and until that changes, the protocol gap will keep widening between the publishers who can defend their archives and those who cannot.
Based on reporting by Bizcommunity: bizcommunity.com/article/new-report-most-south-african-news-sites-still-exposed-to-ai-scraping-186866a
