Create search-engine compliant configuration guidelines dynamically. Protect your private endpoints and control AI scraping agents safely from your browser.
Global Rules Core
Applies directly to the catch-all User-agent (*) fallback matrix.
Prevents aggressive crawling from putting heavy load on your server.
Helps crawlers find your index map quickly.
AI & Large Language Model Scrapers
Explicitly disallow modern AI models from scraping your site content to train datasets.