Generate a customized robots.txt file to control search engine crawling. Protect sensitive pages, manage crawl budget, and guide search engine bots.
User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Robots.txt is a file that tells search engine crawlers which pages or sections they can or cannot crawl.
Yes, improper robots.txt can block important pages from indexing or waste crawl budget on unimportant pages.
New tools & features every week.