Newsletter
Reddit has reportedly decided to block AI startups from scraping data from its website. This move prevents third-party companies from using Reddit’s data to train their machine-learning models without permission.
AI startups mostly rely on content available on the web to train the chatbots. This allows them to feed the chatbots and expand the knowledge base without spending any money on producing exclusive content. However, The Washington Post reports that over 535 news organizations are protesting this and want the AI startups to pay for the content. These organizations, including Reddit, have decided to block crawlers from scraping their content.
Reddit might ditch Google and Bing crawlers from scraping data
Reddit’s decision might also affect Google and Bing crawlers. The Washington Post noted that if Reddit fails to reach an agreement with AI companies, it might ditch Google and Bing search crawlers. This means no content from Reddit will be shown in Google and Bing (or OpenAI) search results.
The Post’s report also added that Reddit wants to ditch Google accounts and make users log into the website to read content. However, the platform later denied that part. It seems search crawlers are the only matter of dispute with Google. An anonymous source told the Post, “Reddit can survive without search.”
All evidence suggests that Reddit is strongly pursuing the idea of blocking Google search crawlers if it fails to make the tech giant pay for the content. The company’s spokesperson, Tim Rathschmidt, told The Verge, “In terms of crawlers, we don’t have anything to share on that topic at the moment.”
News organizations are determined to prevent tech companies from using content for free. They first protested against Google and Meta and asked for a fair share. While the tech giants threatened to block news in specific markets, like Canada, news organizations still hope to get compensation in exchange for their content. California news outlets might soon get a fee for content thanks to the state’s AB 886 bill.
X (Twitter) owner Elon Musk has already criticized AI startups for scraping data. He later charged developers a fee for accessing the platform’s API and implemented a reading limit to prevent data scraping.