服务器禁用爬虫
2022-09-26 17:05 By
"Powerless"
2708
0
3
【Nginx禁止爬虫访问的方法】
if ($http_user_agent ~* "Scrapy|Baiduspider|Curl|HttpClient|Bytespider|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser
|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSp
ider|Ezooms|^$"){
return 403;
}
如需跳转其他页面，只需要吧return 403 换成对于的地址即可，配置如下:
if ($http_user_agent ~* "Scrapy|Baiduspider|Curl|HttpClient|Bytespider|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser
|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSp
ider|Ezooms|^$") {
return 301 https://yoursite.com;
}
如需禁止特定来源用户，配置如下:
if ($http_referer ~ "baidu\.com|google\.net|bing\.com") {
return 403;
}
如需仅允许GET,HEAD和POST请求，配置如下:
#fbrbidden not GET|HEAD|POST method access
if ($request_method !~ ^(GET|HEAD|POST)$) {
return 403;
}
【Apache禁用爬虫的配置】
mod_rewrite模块确定开启的前提下，在.htaccess文件或者相应的.conf文件,添加以下内容:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (^$|FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms) [NC]
RewriteRule . - [R=403,L]
View in WeChat
Others Discussion
-
浏览器访问网站经历的步骤-Html Posted on 2018-11-28 18:48
-
BASE原则 Posted on 2020-12-17 16:42
-
程序员年中考试题-段子版 Posted on 2021-06-23 15:57
-
前端知识体系精简-Css Posted on 2018-03-28 18:34
-
Mysql联合索引的最左前缀匹配原则 Posted on 2018-08-25 15:00
-
2016年云计算热词 Posted on 2019-06-12 17:53
-
PHP没你想的那么差 Posted on 2021-12-17 15:40
-
分布式架构之「 数据分布」 Posted on 2019-11-14 10:00
评 论