robots.txt setting ยท Cloudflare bot solutions docs
Manage a robots.txt file to direct AI bot operators on content scraping permissions.
developers.cloudflare.com