Then came search engine spiders. (Note: Spiders, browsers and crawlers are interchangeable. They are also all bots.) Sure, they ate up bandwidth, but again, the assumption was that search traffic would be beneficial — it brought in customers and new prospects.
For the most part, search spiders respected robots.txt instructions about which sites they could visit and which pages on those sites they could crawl. Because search providers knew most sites welcomed their visits, they more or less respected the restrictions.
That brings us to today, when we find that the companies behind LLMs — through various sneaky mechanisms — do not respect those do not enter signs. And their crawlers don’t deliver the perceived value of human visitors or even search engine spiders. Instead of bringing with them new prospects to an enterprise site, they steal data, use it for their own apps and then sell it to others.
