内容简介:修改网站目录下的.htaccess,添加如下代码即可(2 种代码任选):可用代码 (1):可用代码 (2):
一、Apache
①、通过修改 .htaccess 文件
修改网站目录下的.htaccess,添加如下代码即可(2 种代码任选):
可用代码 (1):
RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (^$|FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python–urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms) [NC] RewriteRule ^(.*)$ – [F]
可用代码 (2):
SetEnvIfNoCase ^User–Agent$ .*(FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python–urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms) BADBOT Order Allow,Deny Allow from all Deny from env=BADBOT
②、通过修改 httpd.conf 配置文件
找到如下类似位置,根据以下代码 新增 / 修改,然后重启 Apache 即可:
Shell
DocumentRoot /home/wwwroot/xxx <Directory “/home/wwwroot/xxx”> SetEnvIfNoCase User–Agent “.*(FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms)” BADBOT Order allow,deny Allow from all deny from env=BADBOT </Directory>
二、Nginx 代码
进入到 nginx 安装目录下的 conf 目录,将如下代码保存为 agent_deny.conf
cd /usr/local/nginx/conf vim agent_deny.conf
#禁止Scrapy等 工具 的抓取 if ($http_user_agent ~* (Scrapy|Curl|HttpClient)) { return 403; } #禁止指定UA及UA为空的访问 if ($http_user_agent ~* “FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|^$” ) { return 403; } #禁止非GET|HEAD|POST方式的抓取 if ($request_method !~ ^(GET|HEAD|POST)$) { return 403; }
然后,在网站相关配置中的 location / { 之后插入如下代码:
Shell
include agent_deny.conf;
如下的配置:
Shell
[marsge@Mars_Server ~]$ cat /usr/local/nginx/conf/zhangge.conf location / { try_files $uri $uri/ /index.php?$args; #这个位置新增1行: include agent_deny.conf; rewrite ^/sitemap_360_sp.txt$ /sitemap_360_sp.php last; rewrite ^/sitemap_baidu_sp.xml$ /sitemap_baidu_sp.php last; rewrite ^/sitemap_m.xml$ /sitemap_m.php last;
保存后,执行如下命令,平滑重启 nginx 即可:
Shell
/usr/local/nginx/sbin/nginx –s reload
三、 PHP 代码
将如下方法放到贴到网站入口文件 index.php 中的第一个
PHP
//获取UA信息 $ua = $_SERVER[‘HTTP_USER_AGENT’]; //将恶意USER_AGENT存入数组 $now_ua = array(‘FeedDemon ‘,‘BOT/0.1 (BOT for JCE)’,‘CrawlDaddy ‘,‘Java’,‘Feedly’,‘UniversalFeedParser’,‘ApacheBench’,‘Swiftbot’,‘ZmEu’,‘Indy Library’,‘oBot’,‘jaunty’,‘YandexBot’,‘AhrefsBot’,‘MJ12bot’,‘WinHttp’,‘EasouSpider’,‘HttpClient’,‘Microsoft URL Control’,‘YYSpider’,‘jaunty’,‘Python-urllib’,‘lightDeckReports Bot’); //禁止空USER_AGENT,dedecms等主流采集程序都是空USER_AGENT,部分 sql 注入工具也是空USER_AGENT if(!$ua) { header(“Content-type: text/html; charset=utf-8”); die(‘请勿采集本站,因为采集的站长木有小JJ!’); }else{ foreach($now_ua as $value ) //判断是否是数组中存在的UA if(eregi($value,$ua)) { header(“Content-type: text/html; charset=utf-8”); die(‘请勿采集本站,因为采集的站长木有小JJ!’); } }
四、测试效果
如果是 vps,那非常简单,使用 curl -A 模拟抓取即可,比如:
模拟宜搜蜘蛛抓取:
Shell
curl –I –A ‘YisouSpider’ bizhi.bcoderss.com
模拟 UA 为空的抓取:
Shell
curl –I –A ” bizhi.bcoderss.com
模拟百度蜘蛛的抓取:
Shell
curl –I –A ‘Baiduspider’ bizhi.bcoderss.com
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持 码农网
猜你喜欢:- Python爬虫:抓取新浪新闻数据
- Python爬虫:抓取新浪新闻数据
- python爬虫-- 抓取网页、图片、文章
- Python爬虫抓取技术的门道
- 如何使用代理IP进行数据抓取,PHP爬虫抓取亚马逊商品数据
- 利用Python网络爬虫抓取网易云歌词
本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
Writing Windows VxDs and Device Drivers, Second Edition
Karen Hazzah / CMP / 1996-01-12 / USD 54.95
Software developer and author Karen Hazzah expands her original treatise on device drivers in the second edition of "Writing Windows VxDs and Device Drivers." The book and companion disk include the a......一起来看看 《Writing Windows VxDs and Device Drivers, Second Edition》 这本书的介绍吧!