User-agent: Abonti Disallow: / User-agent: ADmantX Disallow: / User-agent: AhrefsBot Disallow: / User-agent: AppEngine-Google Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: Baiduspider Disallow: / User-agent: Baiduspider-mobile Disallow: / User-agent: Baiduspider-image Disallow: / User-agent: Baiduspider-video Disallow: / User-agent: Baiduspider-news Disallow: / User-agent: Baiduspider-favo Disallow: / User-agent: Baiduspider-cpro Disallow: / User-agent: Baiduspider-ads Disallow: / User-agent: BPImageWalker Disallow: / User-agent: check_http Disallow: / User-agent: checks.panopta.com Disallow: / User-agent: curl Disallow: / User-agent: DLE_Spider.exe Disallow: / User-agent: dotbot Disallow: / User-agent: EasouSpider Disallow: / User-agent: EMail Exractor Disallow: / User-agent: Exabot Disallow: / User-agent: Ezooms Disallow: / User-agent: GbPlugin Disallow: / User-agent: Go Disallow: / User-agent: gooblog Disallow: / User-agent: gsa-crawler Disallow: / User-agent: HolmesBot Disallow: / User-agent: ichiro Disallow: / User-agent: Infoseek SideWinder Disallow: / User-agent: Infoseek SideWinder/2.0B (Linux 2.4 i686) Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: JikeSpider Disallow: / User-agent: libwww-perl Disallow: / User-agent: linkdex.com Disallow: / User-agent: MJ12bot Disallow: / User-agent: MLBot Disallow: / User-agent: moget Disallow: / User-agent: NaverBot Disallow: / User-agent: New-Sogou-Spider Disallow: / User-agent: OrangeBot Disallow: / User-agent: PEAR HTTP_Request class Disallow: / User-agent: PHP Disallow: / User-agent: PHP/5.4.23 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: PycURL Disallow: / User-agent: python-requests Disallow: / User-agent: Python-urllib Disallow: / User-agent: rmnbot Disallow: / User-agent: Ruby Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: SemrushBot Disallow: / User-agent: SEOENGWorldBot Disallow: / User-agent: seplinkbot Disallow: / User-agent: SeznamBot Disallow: / User-agent: sitebot Disallow: / User-agent: Slurp Disallow: / User-agent: sogou spider Disallow: / User-agent: Sogou web Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sosospider Disallow: / User-agent: Sosospider+ Disallow: / User-agent: Squider Disallow: / User-agent: ssearch_bot Disallow: / User-agent: suchmaschinenoptimierung.de Disallow: / User-agent: Synapse Disallow: / User-agent: Wget Disallow: / User-agent: woopingbot Disallow: / User-agent: WordPress Disallow: / User-agent: Wotbox Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Yandex Disallow: / User-agent: Yeti Disallow: / User-agent: YodaoBot Disallow: / User-agent: YoudaoBot Disallow: / User-agent: * Disallow: /r/categories Disallow: /r/countries Disallow: /r/industries Disallow: /r/lis Disallow: /r/pages Crawl-delay: 15