MediaWiki:Robots.txt


 * 1) robots.txt for http://www.wikipedia.org/ and friends
 * 2) Please note: There are a lot of pages on this site, and there are
 * 3) some misbehaved spiders out there that go _way_ too fast. If you're
 * 4) irresponsible, your access to the site may be blocked.
 * 5) Edit at http://en.wikipedia.org/w/index.php?title=MediaWiki:Robots.txt&action=edit
 * 6) Please check any changes using a syntax validator such as http://tool.motoricerca.info/robots-checker.phtml
 * 7) Enter http://en.wikipedia.org/robots.txt as the URL to check.
 * 1) Edit at http://en.wikipedia.org/w/index.php?title=MediaWiki:Robots.txt&action=edit
 * 2) Please check any changes using a syntax validator such as http://tool.motoricerca.info/robots-checker.phtml
 * 3) Enter http://en.wikipedia.org/robots.txt as the URL to check.
 * 1) Please check any changes using a syntax validator such as http://tool.motoricerca.info/robots-checker.phtml
 * 2) Enter http://en.wikipedia.org/robots.txt as the URL to check.

User-agent: Mediapartners-Google* Disallow: /
 * 1) advertising-related bots:

User-agent: IsraBot Disallow:
 * 1) Wikipedia work bots:

User-agent: Orthogaffe Disallow:

User-agent: UbiCrawler Disallow: /
 * 1) Crawlers that are kind enough to obey, but which we'd rather not have
 * 2) unless they're feeding search engines.

User-agent: DOC Disallow: /

User-agent: Zao Disallow: /

User-agent: sitecheck.internetseer.com Disallow: /
 * 1) Some bots are known to be trouble, particularly those designed to copy
 * 2) entire sites. Please obey robots.txt.

User-agent: Zealbot Disallow: /

User-agent: MSIECrawler Disallow: /

User-agent: SiteSnagger Disallow: /

User-agent: WebStripper Disallow: /

User-agent: WebCopier Disallow: /

User-agent: Fetch Disallow: /

User-agent: Offline Explorer Disallow: /

User-agent: Teleport Disallow: /

User-agent: TeleportPro Disallow: /

User-agent: WebZIP Disallow: /

User-agent: linko Disallow: /

User-agent: HTTrack Disallow: /

User-agent: Microsoft.URL.Control Disallow: /

User-agent: Xenu Disallow: /

User-agent: larbin Disallow: /

User-agent: libwww Disallow: /

User-agent: ZyBORG Disallow: /

User-agent: Download Ninja Disallow: /

User-agent: wget Disallow: /
 * 1) Sorry, wget in its recursive mode is a frequent problem.
 * 2) Please read the man page and use it properly; there is a
 * 3) --wait option you can use to set the delay between hits,
 * 4) for instance.
 * 1) for instance.

User-agent: grub-client Disallow: /
 * 1) The 'grub' distributed client has been *very* poorly behaved.
 * 1) The 'grub' distributed client has been *very* poorly behaved.

User-agent: k2spider Disallow: /
 * 1) Doesn't follow robots.txt anyway, but...
 * 1) Doesn't follow robots.txt anyway, but...

User-agent: NPBot Disallow: /
 * 1) Hits many times per second, not acceptable
 * 2) http://www.nameprotect.com/botinfo.html
 * 1) http://www.nameprotect.com/botinfo.html

User-agent: WebReaper Disallow: /
 * 1) A capture bot, downloads gazillions of pages with no public benefit
 * 2) http://www.webreaper.net/


 * 1) Don't allow the wayback-machine to index user-pages
 * 2) User-agent: ia_archiver
 * 3) Disallow: /wiki/User

User-agent: * Disallow: /w/ Disallow: /trap/ Disallow: /wiki/Special:Random Disallow: /wiki/Special%3ARandom Disallow: /wiki/Special:Search Disallow: /wiki/Special%3ASearch Disallow: /wiki/Wikipedia:Checkuser/ Disallow: /wiki/Wikipedia:Spam-Blacklist-Log Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log Disallow: /wiki/MediaWiki:Spam-blacklist Disallow: /wiki/MediaWiki%3ASpam-blacklist Disallow: /wiki/MediaWiki_talk:Spam-blacklist Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist Disallow: /wiki/Wikipedia:Articles_for_deletion/ Disallow: /wiki/Wikipedia%3AArticles_for_deletion/ Disallow: /wiki/Wikipedia:Votes_for_deletion/ Disallow: /wiki/Wikipedia%3AVotes_for_deletion/ Disallow: /wiki/Wikipedia:Pages_for_deletion/ Disallow: /wiki/Wikipedia%3APages_for_deletion/ Disallow: /wiki/Wikipedia:Miscellany_for_deletion/ Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/ Disallow: /wiki/Wikipedia:Miscellaneous_deletion/ Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/ Disallow: /wiki/Wikipedia:Categories_for_discussion/ Disallow: /wiki/Wikipedia%3ACategories_for_discussion/ Disallow: /wiki/Wikipedia:Templates_for_deletion/ Disallow: /wiki/Wikipedia%3ATemplates_for_deletion/ Disallow: /wiki/Wikipedia:Deletion_review/ Disallow: /wiki/Wikipedia%3ADeletion_review/ Disallow: /wiki/Wikipedia:Copyright_problems Disallow: /wiki/Wikipedia%3ACopyright_problems Disallow: /wiki/Wikipedia:Protected_titles/ Disallow: /wiki/Wikipedia%3AProtected_titles/ Disallow: /wiki/Wikipedia:Articles_for_creation/ Disallow: /wiki/Wikipedia%3AArticles_for_creation/ Disallow: /wiki/Wikipedia:Requests_for_arbitration/ Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/ Disallow: /wiki/Wikipedia_talk:Requests_for_arbitration/ Disallow: /wiki/Wikipedia_talk%3ARequests_for_arbitration/ Disallow: /wiki/Wikipedia:Requests_for_comment/ Disallow: /wiki/Wikipedia%3ARequests_for_comment/ Disallow: /wiki/Wikipedia_talk:Requests_for_comment/ Disallow: /wiki/Wikipedia_talk%3ARequests_for_comment/ Disallow: /wiki/Wikipedia:Requests_for_adminship/ Disallow: /wiki/Wikipedia%3ARequests_for_adminship/ Disallow: /wiki/Wikipedia_talk:Requests_for_adminship/ Disallow: /wiki/Wikipedia_talk%3ARequests_for_adminship/ Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/ Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/ Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/ Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/ Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/ Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/ Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/ Disallow: /wiki/Wikipedia_talk:Templates_for_deletion/ Disallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion/ Disallow: /wiki/Wikipedia_talk:Categories_for_discussion/ Disallow: /wiki/Wikipedia_talk%3ACategories_for_discussion/ Disallow: /wiki/Wikipedia_talk:Deletion_review/ Disallow: /wiki/Wikipedia_talk%3ADeletion_review/ Disallow: /wiki/Wikipedia:WikiProject_Spam/ Disallow: /wiki/Wikipedia%3AWikiProject_Spam/ Disallow: /wiki/Wikipedia:Changing_username Disallow: /wiki/Wikipedia%3AChanging_username Disallow: /wiki/Wikipedia:Changing_username/ Disallow: /wiki/Wikipedia%3AChanging_username/ Disallow: /wiki/Wikipedia_talk:Changing_username Disallow: /wiki/Wikipedia_talk%3AChanging_username Disallow: /wiki/Wikipedia_talk:Changing_username/ Disallow: /wiki/Wikipedia_talk%3AChanging_username/ Disallow: /wiki/Wikiquote:Votes_for_deletion/ Disallow: /wiki/Wikiquote%3AVotes_for_deletion/ Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/ Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/ Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/ Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/ Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/ Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/ Disallow: /wiki/Wikipedia:Administrators%27_noticeboard/ Disallow: /wiki/Wikipedia%3AAdministrators%27_noticeboard/ Disallow: /wiki/Wikipedia_talk:Administrators%27_noticeboard/ Disallow: /wiki/Wikipedia_talk%3AAdministrators%27_noticeboard/ Disallow: /wiki/Wikipedia:Community_sanction_noticeboard/ Disallow: /wiki/Wikipedia%3ACommunity_sanction_noticeboard/ Disallow: /wiki/Wikipedia_talk:Community_sanction_noticeboard/ Disallow: /wiki/Wikipedia_talk%3ACommunity_sanction_noticeboard/
 * 1) Friendly, low-speed bots are welcome viewing article pages, but not
 * 2) dynamically-generated pages please.
 * 3) Inktomi's "Slurp" can read a minimum delay between hits; if your
 * 4) bot supports such a thing using the 'Crawl-delay' or another
 * 5) instruction, please let us know.
 * 1) bot supports such a thing using the 'Crawl-delay' or another
 * 2) instruction, please let us know.
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=12111
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=12111
 * 1) https://bugzilla.wikimedia.org/show_bug.cgi?id=13961
 * 1) https://bugzilla.wikimedia.org/show_bug.cgi?id=14075
 * 1) Folks get annoyed when VfD discussions end up the number 1 google hit for
 * 2) their name.
 * 3) http://bugzilla.wikimedia.org/show_bug.cgi?id=#4776
 * en:
 * en:
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=11261
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=11261
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=10288
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=10288
 * 1) https://bugzilla.wikimedia.org/show_bug.cgi?id=13398
 * 1) https://bugzilla.wikimedia.org/show_bug.cgi?id=13398
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=14793
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=14793
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=15095
 * 1) http://bugzilla.wikimedia.org/show_bug.cgi?id=15095