使用scrapy爬取域名的whois信息

最近收到个需求:查询几百个域名的whois信息,如域名注册时间,到期时间,注册商,域名状态等.如果手动去一个个查询,效率低,而最近刚好在看scrapy,故用它来爬取相关信息.

首先是想尝试去爬阿里云的whois信息,但尝试失败,需要验证码.(先放一边,有空再看).而站长之家也可以查询,而且没验证码,便选择用站长之家来爬取.

要点

1.理解xpath.在爬去相关内容的时候,要用到

2.理解items和pipelines.使用它们可以更好的来存储,处理爬取后的内容

部署

scrapy的安装,可以看另外一篇: 在centos7和centos6下安装scrapy

安装完成后,就开始吧.

1.新建一个scrapy项目,比如项目名称是chinaz,执行如下命令:

scrapy startproject chinaz

默认整个项目的结构如下:

chinaz

├── chinaz

│   ├── __init__.py

│   ├── __init__.pyo

│   ├── items.py

│   ├── pipelines.py

│   ├── settings.py

│   └── spiders

│       ├── __init__.py

│       └── __init__.pyo

└── scrapy.cfg

scrapy.cfg是整个项目的配置文件,我们这里不用修改.

chinaz/chinaz/spiders/ 这个目录下存放我们爬虫程序

分析

目的有两个:

1.查询多个域名时的交互问题

2.找出对应信息的xpath

打开http://whois.chinaz.com/,随意查询一个域名,比如163.com.点击查询,可以看到相关的whois信息,url变成了http://whois.chinaz.com/163.com,把末尾域替换为其他的,如http://whois.chinaz.com/sina.com,可以看到sina.com的相关whois信息.所以我们只要把后面的域名替换为要查询的域名,便可以获取对应的whois信息了.

使用chrome的检查,找到注册商,找出它的xpath



可以看出,所有的内容都在/ul/ui下面,注册商对应的内容是

<div class="fr WhLeList-right", <div class="block ball"

所以整个路径是:

//ul/li/div[@class="fr WhLeList-right"/div[@class="block ball"]/span/text()

当然,如果一个个找出来拼凑会比较麻烦.我们可以直接用chrome的xpath功能.右击,选择copy,copy xpath,会获得如下路径

//*[@id="sh_info"]/li[2]/div[2]/div/span

爬取

直接上代码吧

1.chinaz_spider.py

#_*_ coding:utf-8 _*_

import scrapy
from chinaz.items import ChinazItem

class DmozSpider(scrapy.Spider):
name = "chinaz"
#allowed_domains = ["chinaz.com"]

dates = file('url').readlines()
start_urls = []
for v in dates:
v = v.strip('\n')
v = 'http://whois.chinaz.com/' + v
start_urls.append(v)

def clean_str(self):
data = self.data
data = [t.strip() for t in data]
data = [t.strip('\r') for t in data]
data = [t for t in data if t != '']
data = [t.encode('utf-8') for t in data]
return data

def parse(self, response):
surl = response.url
domain = surl.split('/')[3].strip()
items = []
regists = []
dates = []
dnss = []
contects = []
status = []
for sel in response.xpath('//ul/li/div[@class="fr WhLeList-right"]'):
#item = ChinazItem()
regist = sel.xpath('./div[@class="block ball"]/span/text()').extract()
date = sel.xpath('./span/text()').extract()
dns = sel.xpath('./text()').extract()
if regist != []:
self.data = regist
regist = self.clean_str()
regists.append(regist)
if date != []:
self.data = date
date = self.clean_str()
dates.append(date)
if dns !=[]:
self.data = dns
dns = self.clean_str()
dnss.append(dns)

for sel in response.xpath('//ul/li/div[@class="fr WhLeList-right block ball lh24"]'):
item = ChinazItem()
contect = sel.xpath('./span/text()').extract()
if contect != []:
self.data = contect
contect = self.clean_str()
contects.append(contect)
#yield item

for sel in response.xpath('//ul/li/div[@class="fr WhLeList-right clearfix"]'):
item = ChinazItem()
state = sel.xpath('./p[@class="lh30 pr tip-sh"]/span/text()').extract()
if state != []:
self.data = state
state = self.clean_str()
status.append(state)
#yield item

dates = [' '.join(t) for t in dates]
if len(dates) == 4:
dates.remove(dates[0])

item = ChinazItem()
item['regist'] = [' '.join(t) for t in regists]
item['date'] = dates
item['dns'] = ' '.join([' '.join(t) for t in dnss])
item['contect'] = [' '.join(t) for t in contects]
item['domain'] = domain
item['status'] = ' '.join([' '.join(t) for t in status])
return item

2.items.py

import scrapy

class ChinazItem(scrapy.Item):
regist = scrapy.Field()
date = scrapy.Field()
dns = scrapy.Field()
contect = scrapy.Field()
domain = scrapy.Field()
status = scrapy.Field()

3.pipelines.py

class ChinazPipeline(object):

def process_item(self, item, spider):
try:
line = [item['regist'][0],item['date'][0], item['date'][1],item['dns'], item['contect'][0],item['domain'],item['status']]
except IndexError:
line = [item['regist'][0],item['date'][0], item['date'][1],item['dns'], ' ',item['domain'],item['status']]
line = '    '.join(line) + '\n'
fp = open('/tmp/domain_new.txt','a')
fp.write(line)
fp.close()
return item

4.setting.py里面要开启这个配置

ITEM_PIPELINES = {
'chinaz.pipelines.ChinazPipeline': 300,
}

以上动作都完成后,就开始爬取吧

scrapy crawl chinaz

后续

站长之家的whois信息没有阿里云的新,有些信息没阿里云的全,所以有时间可以再研究下,如何去获取阿里云的whois信息.