User-agent: * Disallow: / User-agent: Slurp Disallow: /cgi-bin/