#This robots.txt file prevents 403s caused by search engines crawling admin pages # #Please place this in the root directory of your site and ensure that the Disallow #contains the correct relative URL. User-agent: * Disallow: /database/ Disallow: /includes/ Disallow: /misc/ Disallow: /modules/ Disallow: /sites/ Disallow: /themes/ Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /awstats/