User-agent: * Disallow: /admin/ Disallow: /confirmer-demande.php Disallow: /?gclid=* Disallow: /faire-une-proposition.php Disallow: /modifier-demande.php Crawl-delay: 10 Request-rate: 1/10 # maximum rate is one page every 10 seconds Visit-time: 2100-0800 # only visit between 21:00 and 08:00 UTC (GMT) User-agent: Slurp Crawl-delay: 20 User-agent: Exabot Crawl-delay: 20 User-agent: msnbot Crawl-delay: 20 User-agent: Fasterfox Disallow: / User-agent: dotbot Disallow: / User-agent: Xenu's link sleuth 1.3c disallow: / User-agent: Xenu's disallow: /