# Phase 1 crawl policy: let search engines revisit legacy utility URLs so their # X-Robots-Tag noindex directives can be processed. /api/ and /search can be # blocked in phase 2 after Search Console confirms that those URLs are gone. User-agent: * Allow: / Disallow: /private/ Disallow: /desk Disallow: /app Disallow: /Form/ # Block AI training crawlers User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Applebot-Extended Disallow: / # Block common bad bots User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / # Sitemap Sitemap: https://felixtools.in/sitemap.xml