User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /wiki/ Disallow: /n/ # http://www.webmasterworld.com/robots.txt has a long list of active robots you might want to block. # Some of these (and many others) ignore robots.txt, and are forcibly blocked in .htaccess. User-agent: TurnitinBot User-agent: NPbot User-agent: psbot User-agent: baiduspider User-agent: larbin User-agent: NationalDirectory User-agent: LNSpiderguy User-agent: Teleport User-agent: MIIxpc User-agent: asterias User-agent: lwp-trivial User-agent: LinkWalker User-agent: cosmos User-agent: MSIECrawler User-agent: sitecheck.internetseer.com User-agent: pompos User-agent: curl User-agent: Wget User-agent: Generic Disallow: /