Created
October 19, 2025 12:08
-
-
Save thinkingcap/c28136188ee6f03be0a16659dd38dc8d to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #Block AI Slop | |
| User-agent: AI2Bot | |
| User-agent: Ai2Bot-Dolma | |
| User-agent: aiHitBot | |
| User-agent: Amazonbot | |
| User-agent: Andibot | |
| User-agent: anthropic-ai | |
| User-agent: Applebot | |
| User-agent: Applebot-Extended | |
| User-agent: Awario | |
| User-agent: AwarioRssBot | |
| User-agent: AwarioSmartBot | |
| User-agent: barkrowler | |
| User-agent: bedrockbot | |
| User-agent: bingbot | |
| User-agent: BLEXBot | |
| User-agent: Brightbot 1.0 | |
| User-agent: Bytespider | |
| User-agent: CCBot | |
| User-agent: ChatGPT Agent | |
| User-agent: ChatGPT-User | |
| User-agent: Claude-SearchBot | |
| User-agent: Claude-User | |
| User-agent: Claude-Web | |
| User-agent: ClaudeBot | |
| User-agent: CloudVertexBot | |
| User-agent: cohere-ai | |
| User-agent: cohere-training-data-crawler | |
| User-agent: Cotoyogi | |
| User-agent: Crawlspace | |
| User-agent: Datenbank Crawler | |
| User-agent: DeepSeekBot | |
| User-agent: Devin | |
| User-agent: Diffbot | |
| User-agent: DuckAssistBot | |
| User-agent: Echobot Bot | |
| User-agent: EchoboxBot | |
| User-agent: FacebookBot | |
| User-agent: facebookexternalhit | |
| User-agent: Factset_spyderbot | |
| User-agent: FirecrawlAgent | |
| User-agent: FriendlyCrawler | |
| User-agent: Gemini-Deep-Research | |
| User-agent: Google-CloudVertexBot | |
| User-agent: Google-Extended | |
| User-agent: Google-Firebase | |
| User-agent: Google-NotebookLM | |
| User-agent: GoogleAgent-Mariner | |
| User-agent: GoogleOther | |
| User-agent: GoogleOther-Image | |
| User-agent: GoogleOther-Video | |
| User-agent: GPTBot | |
| User-agent: iaskspider/2.0 | |
| User-agent: ICC-Crawler | |
| User-agent: ImagesiftBot | |
| User-agent: img2dataset | |
| User-agent: ISSCyberRiskCrawler | |
| User-agent: Kangaroo Bot | |
| User-agent: LinerBot | |
| User-agent: meta-externalagent | |
| User-agent: Meta-ExternalAgent | |
| User-agent: meta-externalfetcher | |
| User-agent: Meta-ExternalFetcher | |
| User-agent: meta-webindexer | |
| User-agent: MistralAI-User | |
| User-agent: MistralAI-User/1.0 | |
| User-agent: MJ12bot | |
| User-agent: MyCentralAIScraperBot | |
| User-agent: netEstate Imprint Crawler | |
| User-agent: NovaAct | |
| User-agent: OAI-SearchBot | |
| User-agent: omgili | |
| User-agent: omgilibot | |
| User-agent: OpenAI | |
| User-agent: Operator | |
| User-agent: PanguBot | |
| User-agent: Panscient | |
| User-agent: panscient.com | |
| User-agent: Perplexity-User | |
| User-agent: PerplexityBot | |
| User-agent: PetalBot | |
| User-agent: PhindBot | |
| User-agent: Poseidon Research Crawler | |
| User-agent: QualifiedBot | |
| User-agent: QuillBot | |
| User-agent: quillbot.com | |
| User-agent: SBIntuitionsBot | |
| User-agent: Scrapy | |
| User-agent: SemrushBot | |
| User-agent: SemrushBot-BA | |
| User-agent: SemrushBot-CT | |
| User-agent: SemrushBot-OCOB | |
| User-agent: SemrushBot-SI | |
| User-agent: SemrushBot-SWA | |
| User-agent: ShapBot | |
| User-agent: Sidetrade indexer bot | |
| User-agent: TerraCotta | |
| User-agent: Thinkbot | |
| User-agent: TikTokSpider | |
| User-agent: Timpibot | |
| User-agent: VelenPublicWebCrawler | |
| User-agent: WARDBot | |
| User-agent: Webzio-Extended | |
| User-agent: wpbot | |
| User-agent: YaK | |
| User-agent: YandexAdditional | |
| User-agent: YandexAdditionalBot | |
| User-agent: YouBot | |
| Disallow: / | |
| User-agent: * | |
| Disallow: /wp-content/uploads/wc-logs/ | |
| Disallow: /wp-content/uploads/woocommerce_transient_files/ | |
| Disallow: /wp-content/uploads/woocommerce_uploads/ | |
| Disallow: /wp-admin/ | |
| Allow: /wp-admin/admin-ajax.php | |
| Disallow: /*add-to-cart=* | |
| Disallow: /shop/ | |
| Disallow: /?s= | |
| Disallow: /search | |
| Disallow: /wp-json | |
| Disallow: /cart/ | |
| Disallow: /wishlist/ | |
| Disallow: /checkout/ | |
| Disallow: /my-account/ | |
| Disallow: /*?* | |
| # START YOAST BLOCK | |
| # --------------------------- | |
| User-agent: * | |
| Disallow: /wp-json/ | |
| Disallow: /?rest_route= | |
| Sitemap: https://xxxxx.com.au/sitemap_index.xml | |
| # --------------------------- | |
| # END YOAST BLOCK |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment