15·
8 days agoNephentes that shit. Poison every scraper until they start respecting robot.txt. Purposefully use llm.txt to trap the fuckers.
- 0 Posts
- 2 Comments
Joined 4 months ago
Cake day: March 18th, 2025
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
Robots.txt expects that the client is respecting the rules, for instance, marking that they are a scraper.
AI scrapers don’t respect this trust, and thus robots.txt is meaningless.