欢迎光临
我们一直在努力

ROBOTS.TXT屏蔽笔记、代码、示例大全

自己网站的ROBOTS.TXT屏蔽的记录,以及一些代码和示例:

屏蔽后台目录,为了安全,做双层管理后台目录/a/xxxx/,蜘蛛屏蔽/a/,既不透露后台路径,也屏蔽蜘蛛爬后台目录

缓存,阻止蜘蛛爬静态缓存文件

下载,阻止蜘蛛爬下载目录,若无用,删除下载目录

编辑器,阻止蜘蛛爬编辑器,也防止编辑器目录被发现产生安全隐患

邮件,阻止蜘蛛爬静态邮件模板

其他页面,无收录价值页面屏蔽

图片,阻止蜘蛛爬除JPG/jpg类文件之外的任何类型图片

核心文件目录,阻止蜘蛛直接爬include及其子目录(函数/类库/模型/模板等)

媒体目录,阻止爬播放类型媒体目录,若无用,删除该目录

附加参数页面,阻止蜘蛛爬带参数的页面

RAR ZIP GZ文件类型

无效蜘蛛、恶意蜘蛛屏蔽

指定sitemap.xml位置

目录屏蔽:

User-agent: *

Disallow: /a/

Disallow: /cache/

Disallow: /download/

Disallow: /editors/

Disallow: /email/

Disallow: /extras/

Disallow: /images/

Disallow: /includes/

Disallow: /media/

Disallow: /pub/

Disallow: /nddbc.html

Disallow: /page_not_found.php

Disallow: /login.html

Disallow: /privacy.html

Disallow: /conditions.html

Disallow: /contact_us.html

Disallow: /gv_faq.html

Disallow: /discount_coupon.html

Disallow: /unsubscribe.html

Disallow: /shopping_cart.html

Disallow: /ask_a_question.html

Disallow: /popup_image_additional.html

Disallow: /product_reviews_write.html

Disallow: /tell_a_friend.html

Disallow: /pages-popup_image.html

Disallow: /popup_image_additional.html

Disallow: /login.html

阻止蜘蛛爬非jpg图片(限制产品图片格式为jpg)

User-agent: Googlebot

Allow: .jpg$

Disallow: .jpeg$

Disallow: .gif$

Disallow: .png$

Disallow: .bmp$

阻止蜘蛛爬压缩文件

User-agent: *

Disallow: .zip$

Disallow: .rar$

Disallow: .gz$

Disallow: .tar $

制定sitemap地址

Sitemap: http://www.xxx.jp/sitemap.xml

其他无效蜘蛛、恶意蜘蛛屏蔽:

User-Agent: almaden

Disallow: /

User-Agent: ASPSeek

Disallow: /

User-Agent: Axmo

Disallow: /

User-Agent: BaiduSpider

Disallow: /

User-Agent: booch

Disallow: /

User-Agent: DTS Agent

Disallow: /

User-Agent: Downloader

Disallow: /

User-Agent: EmailCollector

Disallow: /

User-Agent: EmailSiphon

Disallow: /

User-Agent: EmailWolf

Disallow: /

User-Agent: Expired Domain Sleuth

Disallow: /

User-Agent: Franklin Locator

Disallow: /

User-Agent: Gaisbot

Disallow: /

User-Agent: grub

Disallow: /

User-Agent: HughCrawler

Disallow: /

User-Agent: iaea.org

Disallow: /

User-Agent: lcabotAccept

Disallow: /

User-Agent: IconSurf

Disallow: /

User-Agent: Iltrovatore-Setaccio

Disallow: /

User-Agent: Indy Library

Disallow: /

User-Agent: IUPUI

Disallow: /

User-Agent: Kittiecentral

Disallow: /

User-Agent: iaea.org

Disallow: /

User-Agent: larbin

Disallow: /

User-Agent: lwp-trivial

Disallow: /

User-Agent: MetaTagRobot

Disallow: /

User-Agent: Missigua Locator

Disallow: /

User-Agent: NetResearchServer

Disallow: /

User-Agent: NextGenSearch

Disallow: /

User-Agent: NPbot

Disallow: /

User-Agent: Nutch

Disallow: /

User-Agent: ObjectsSearch

Disallow: /

User-Agent: Oracle Ultra Search

Disallow: /

User-Agent: PEERbot

Disallow: /

User-Agent: PictureOfInternet

Disallow: /

User-Agent: PlantyNet

Disallow: /

User-Agent: QuepasaCreep

Disallow: /

User-Agent: ScSpider

Disallow: /

User-Agent: SOFT411

Disallow: /

User-Agent: spider.acont.de

Disallow: /

User-Agent: Sqworm

Disallow: /

User-Agent: SSM Agent

Disallow: /

User-Agent: TAMU

Disallow: /

User-Agent: TheUsefulbot

Disallow: /

User-Agent: TurnitinBot

Disallow: /

User-Agent: Tutorial Crawler

Disallow: /

User-Agent: TutorGig

Disallow: /

User-Agent: WebCopier

Disallow: /

User-Agent: WebZIP

Disallow: /

User-Agent: ZipppBot

Disallow: /

User-Agent: Xenu

Disallow: /

User-Agent: Wotbox

Disallow: /

User-Agent: Wget

Disallow: /

User-Agent: NaverBot

Disallow: /

User-Agent: mozDex

Disallow: /

User-Agent: Sosospider

Disallow: /

User-Agent: Baidupider

Disallow: /

未经允许不得转载:外贸商城系统,外贸网站模板,php建站教程,zencart模板 » ROBOTS.TXT屏蔽笔记、代码、示例大全

分享到:更多 ()

评论 抢沙发

35PHP 更全 更专业 更方便