服务器禁用爬虫
2022-09-26 17:05 By
"Powerless"
2577
0
3
【Nginx禁止爬虫访问的方法】
if ($http_user_agent ~* "Scrapy|Baiduspider|Curl|HttpClient|Bytespider|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser
|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSp
ider|Ezooms|^$"){
return 403;
}
如需跳转其他页面，只需要吧return 403 换成对于的地址即可，配置如下:
if ($http_user_agent ~* "Scrapy|Baiduspider|Curl|HttpClient|Bytespider|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser
|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSp
ider|Ezooms|^$") {
return 301 https://yoursite.com;
}
如需禁止特定来源用户，配置如下:
if ($http_referer ~ "baidu\.com|google\.net|bing\.com") {
return 403;
}
如需仅允许GET,HEAD和POST请求，配置如下:
#fbrbidden not GET|HEAD|POST method access
if ($request_method !~ ^(GET|HEAD|POST)$) {
return 403;
}
【Apache禁用爬虫的配置】
mod_rewrite模块确定开启的前提下，在.htaccess文件或者相应的.conf文件,添加以下内容:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (^$|FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms) [NC]
RewriteRule . - [R=403,L]
View in WeChat
Others Discussion
-
分布式服务限流 Posted on 2020-02-07 18:57
-
有状态服务VS无状态服务 Posted on 2020-02-07 18:18
-
企业级PAAS云平台几个关键问题和挑战 Posted on 2019-06-12 18:33
-
MySQL 单库后期分库策略 Posted on 2019-08-19 14:31
-
Redis七大经典问题 Posted on 2021-05-27 11:14
-
关于HTTPS的五大误区 Posted on 2020-02-02 01:10
-
PHP实现精确发布时间 Posted on 2018-12-06 21:00
-
为什么要测量尾部延迟 Posted on 2020-09-18 10:34
评 论