scrapy爬虫防止被禁止 User Agent切换
2015-09-18 13:19
435 查看
http://doc.scrapy.org/en/1.0/topics/practices.html#bans
1. User Agent轮换
2. 禁Cookie
3. 设置大于2s的DOWNLOAD_DELAY
4. 使用Google Cache (不懂)
5. 使用轮换IP(还不会)
6. 使用分布式下载器(不知道scrapy-redis算不算)
User Agent轮换例子
1)新建一个middlewares.py文件,内容如下,文件放在与items.py, settings.py所在的文件夹下。
运行爬虫scrapy crawl spider1 -L WARNING 不打印Debug信息,可以清楚得看到print出来的User Agent不同。
1. User Agent轮换
2. 禁Cookie
3. 设置大于2s的DOWNLOAD_DELAY
4. 使用Google Cache (不懂)
5. 使用轮换IP(还不会)
6. 使用分布式下载器(不知道scrapy-redis算不算)
User Agent轮换例子
1)新建一个middlewares.py文件,内容如下,文件放在与items.py, settings.py所在的文件夹下。
#!/usr/bin/python #-*-coding:utf-8-*- import random from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware class RotateUserAgentMiddleware(UserAgentMiddleware): def __init__(self, user_agent=''): self.user_agent = user_agent def process_request(self, request, spider): ua = random.choice(self.user_agent_list) if ua: print ua, '-----------------yyyyyyyyyyyyyyyyyyyyyyyyy' request.headers.setdefault('User-Agent', ua) #the default user_agent_list composes chrome,I E,firefox,Mozilla,opera,netscape #for more user agent strings,you can find it in http://www.useragentstring.com/pages/useragentstring.php user_agent_list = [\ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"\ "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",\ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",\ "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",\ "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",\ "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",\ "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",\ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\ "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\ "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",\ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",\ "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\ "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\ "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",\ "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",\ "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24" ]2) 然后在settings.py中按如下设置
DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware':None, 'example.middlewares.RotateUserAgentMiddleware':400, }
运行爬虫scrapy crawl spider1 -L WARNING 不打印Debug信息,可以清楚得看到print出来的User Agent不同。
相关文章推荐
- Spring注解方式使用与注册bean
- 实现动态数据生成Word文档并保存
- eclipse常用快捷键
- 【Mockplus教程】复制/克隆
- 关机时windows发的消息WM_QUERYENDSESSION
- 云计算设计模式(六)——命令和查询职责分离(CQRS)模式
- POJ 2718 Smallest Difference(DFS+全排列函数)
- 在ssh框架中注解方式需要注意的几个问题
- 【Mockplus教程】收藏和重用
- Java并发教程(Oracle官方资料)
- 欢迎使用CSDN-markdown编辑器
- Java命令行运行参数说明大全(偷来的)
- js数组操作
- Android用户新危机间谍软件大起底
- 梵塔问题
- 【C/C++学院】0813-C与CPP不同以及命名空间简介/函数重载与函数默认参数/泛型auto/Newdelete
- 二人项目
- Findbugs工具的安装和使用
- 【Mockplus教程】标尺和参考线
- Storm使用手册