# Yahoo! Slurp User-agent: Slurp Crawl-delay: 15 # SemrushBot (General scraping) User-agent: SemrushBot Crawl-delay: 15 # SemrushBot-SA (SEO / technical crawling) User-agent: SemrushBot-SA Crawl-delay: 15 # MJ12bot (https://mj12bot.com/) User-agent: MJ12bot Crawl-delay: 15 # AhrefsBot (https://ahrefs.com/robot) User-agent: AhrefsBot Crawl-delay: 15 # All User-agent: * Disallow: /ab_monitor