User-agent: * Disallow: /admin Disallow: /shared/scripts User-agent: * Crawl-Delay: 20