利用Scrapy爬取拉勾网某职位信息

      学习了一段时间的scrapy爬虫框架，也模仿别人的写了几个。最近，在编写爬取拉勾网某职位相关信息的过程中，遇到一些小的问题，和之前一般的爬取静态网页略有不同，这次需要提取的部分信息是js生成的。记录一下，后续备查。

      整个project的文件结构如下所示：

├── lagou_python

│   ├── __init__.py

│   ├── items.py

│   ├── pipelines.py

│   ├── settings.py

│   └── spiders

│       ├── __init__.py

│       ├── lagou_spider.py

│       ├── middleware.py

│       └── rotate_useragent.py

└── scrapy.cfg

      接下来，逐个分析一下。

      首先，分析一下需要抓取的页面信息，招聘信息（城市为上海，职业为python）URL为：

http://www.lagou.com/jobs/list_Python?kd=Python&spc=1&pl=&gj=&xl=&yx=&gx=&st=&labelWords=&lc=&workAddress=&city=%E4%B8%8A%E6%B5%B7&requestId=&pn=1

      通过比较，后续的URL只是pn这个参数的值有变化。显然，只需要修改pn这个参数的值，就可以连续抓取不同网页的信息了。那么，总共有多少页呢，在Firebug中查看网页源码可以得到如下图所示的信息：





      页面的分页是js实现的，应该是开发者修改了一个版本，不太像JQuery中的分页相关代码。js我只懂一点点，所以没有具体分析了。正好有看到scrapy如何处理带js的动态网页相关blog，我就尝试使用scrapy+python-webkit来进行处理。我虚拟机中采用的是Ubuntu14.04.2 Server版本，需要先安装几个包：

apt-get install python-webkit

apt-get install python-jswebkit

apt-get install Xvfb

      完成安装之后，就可以在后面的project中让scrapy结合python-webkit来处理带js的动态网页了。我的初步理解是利用webkit处理带js的网页，然后scrapy抓取处理好的网页内容。拿到我的实例中来看，就是让webkit处理带js的网页，这样分页显示等处理都已经触发了，后续只要用xpath定为到最终呈现的页码处获取尾页的数字，然后自己构造后续的职位信息页码，让scrapy不断抓取就OK了。

      在scrapy中来说，需要自定义下载中间件的处理类。我在project下建立了一个middleware.py的文件，后续在settings.py中把middleware的处理类添加进去。middleware.py中的内容如下：

from scrapy.http import Request, FormRequest, HtmlResponse

import gtk

import webkit

import jswebkit

from lagou_python import settings

class WebkitDownloader( object ):

def process_request( self, request, spider ):

if spider.name in settings.WEBKIT_DOWNLOADER:

if( type(request) is not FormRequest ):

webview = webkit.WebView()

webview.connect( 'load-finished', lambda v,f: gtk.main_quit() )

webview.load_uri( request.url )

gtk.main()

js = jswebkit.JSContext( webview.get_main_frame().get_global_context() )

renderedBody = str( js.EvaluateScript( 'document.body.innerHTML' ) )

return HtmlResponse( request.url, body=renderedBody )

      settings.py中的内容如下：

# -*- coding: utf-8 -*-

# Scrapy settings for lagou_python project

#

# For simplicity, this file contains only the most important settings by

# default. All the other settings are documented here:

#

#     http://doc.scrapy.org/en/latest/topics/settings.html

#

BOT_NAME = 'lagou_python'

SPIDER_MODULES = ['lagou_python.spiders']

NEWSPIDER_MODULE = 'lagou_python.spiders'

WEBKIT_DOWNLOADER=['lagou']

# Crawl responsibly by identifying yourself (and your website) on the user-agent

#USER_AGENT = 'lagou_python (+http://www.yourdomain.com)'

ITEM_PIPELINES = {

'lagou_python.pipelines.LagouPythonPipeline':300

}

COOKIES_ENABLED = False

# Crawl responsibly by identifying yourself (and your website) on the user-agent

#USER_AGENT = 'TBBKAnalysis (+http://www.yourdomain.com)'

DOWNLOADER_MIDDLEWARES = {

'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware' : None,

'lagou_python.spiders.rotate_useragent.RotateUserAgentMiddleware' :400,

'lagou_python.spiders.middleware.WebkitDownloader':543

}

LOG_LEVEL = 'DEBUG'

      已经把自定义的下载类添加进去了。

      为了把爬取的职位信息存在本地为lagou_python.json的文件中，pipelines.py的源码内容如下所示：

# -*- coding: utf-8 -*-

# Define your item pipelines here

#

# Don't forget to add your pipeline to the ITEM_PIPELINES setting

# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html

import json

import codecs

class LagouPythonPipeline(object):

def __init__(self):

self.file = codecs.open("lagou_python.json", encoding="utf-8", mode="wb")

def process_item(self, item, spider):

line = json.dumps(dict(item)) + '\n'

self.file.write(line.decode("unicode_escape"))

return item

      为了防止被ban，建立了一个自动切换User-Agent的文件rotate_useragent.py，源码如下所示：

from scrapy import log

import random

from scrapy.contrib.downloadermiddleware.useragent import UserAgentMiddleware

class RotateUserAgentMiddleware(UserAgentMiddleware):

def __init__(self, user_agent=''):

self.user_agent = user_agent

def process_request(self, request, spider):

ua = random.choice(self.user_agent_list)

if ua:

log.msg('Current UserAgent: '+ua, level=log.INFO)

request.headers.setdefault('User-Agent', ua)

#the default user_agent_list composes chrome,I E,firefox,Mozilla,opera,netscape

#for more user agent strings,you can find it in http://www.useragentstring.com/pages/useragentstring.php

user_agent_list = [

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "

"(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",

"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "

"(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "

"(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "

"(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",

"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "

"(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "

"(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",

"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "

"(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "

"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",

"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "

"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"

]

      最重要的部分，爬虫的编写，在网页中具体内容的定位我就不说明了。职位信息的后续网页是根据尾页的范围，利用xrange产生从第2页到尾页的数字，然后拼接上去的。源码如下：

__author__ = 'sniper.geek'

import re

import json

from scrapy.selector import Selector

from scrapy.spider import Spider

from scrapy.contrib.spiders import CrawlSpider,Rule

from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor as sle

from lagou_python.items import LagouPythonItem

from scrapy import log

from scrapy.http import Request

class LagouSpider(CrawlSpider):

name = "lagou"

download_delay = 2

allowed_domains = ["lagou.com"]

start_urls = [

"http://www.lagou.com/jobs/list_Python?kd=Python&spc=1&pl=&gj=&xl=&yx=&gx=&st=&labelWords=&lc=&workAddress=&city=%E4%B8%8A%E6%B5%B7&requestId=&pn=1"

]

#rules = [

#    Rule(sle(allow=("l/jobs/list_Python?kd=Python&spc=1&pl=&gj=&xl=&yx=&gx=&st=&labelWords=&lc=&workAddress=&city=%E4%B8%8A%E6%B5%B7&requestId=&pn=\d{1}")),follow=True,callback='parse_item')

#    Rule(sle(),follow=True,callback='parse_item')

    #

#]

def parse(self,response):

log.msg("Fetch page: %s"%response.url)

#items = []

sel = Selector(response)

sites = sel.xpath('//div[@class="content"]/ul[@class="hot_pos reset"]/li')

total_page_value = sel.xpath('//div[@class="Pagination myself"]/a[@href="#"][last()]/@title').extract()

log.msg("total_page_value: %s"%total_page_value)

total_page = int(total_page_value[0])

log.msg("page_number: %s"%total_page)

for site in sites:

item = LagouPythonItem()

item['salary'] =site.xpath('div[@class="hot_pos_l"]/span[1]/text()').extract()

item['experience'] =site.xpath('div[@class="hot_pos_l"]/span[2]/text()').extract()

item['education'] =site.xpath('div[@class="hot_pos_l"]/span[3]/text()').extract()

item['occupation_temptation'] =site.xpath('div[@class="hot_pos_l"]/span[4]/text()').extract()

if len(site.xpath('div[@class="hot_pos_r"]/span')) ==3:

item['job_fields'] = site.xpath('div[@class="hot_pos_r"]/span[1]/text()').extract()

item['stage'] = site.xpath('div[@class="hot_pos_r"]/span[2]/text()').extract()

item['scale'] = site.xpath('div[@class="hot_pos_r"]/span[3]/text()').extract()

item['company'] =site.xpath('div[@class="hot_pos_r"]/div[@class="mb10"]/a/text()').extract()

item['url'] = site.xpath('div[@class="hot_pos_r"]/div[@class="mb10"]/a/@href').extract()

item['founder']=[]

else:

item['job_fields'] = site.xpath('div[@class="hot_pos_r"]/span[1]/text()').extract()

item['founder'] = site.xpath('div[@class="hot_pos_r"]/span[2]/text()').extract()

item['stage'] = site.xpath('div[@class="hot_pos_r"]/span[3]/text()').extract()

item['scale'] = site.xpath('div[@class="hot_pos_r"]/span[4]/text()').extract()

item['company'] =site.xpath('div[@class="hot_pos_r"]/div[@class="mb10"]/a/text()').extract()

item['url'] = site.xpath('div[@class="hot_pos_r"]/div[@class="mb10"]/a/@href').extract()

yield item

next_urls=[]

for k in xrange(2,total_page+1):

base_url = "http://www.lagou.com/jobs/list_Python?kd=Python&spc=1&pl=&gj=&xl=&yx=&gx=&st=&labelWords=&lc=&workAddress=&city=%E4%B8%8A%E6%B5%B7&requestId=&pn="+str(k)

next_urls.append(base_url)

for next_url in next_urls:

#log.msg("Next page:%s"%next_url, level=log.INFO)

yield Request(next_url,callback=self.parse)

      尝试抓取，并且在本地生成为log的日志文件，命令如下：

scrapy crawl lagou  --logfile=log

      部分抓取结果：





      整个project的源文件我放到百度云了，感兴趣的可以看看。scrapy还用的不熟练，发现自己python写的也一般，任重道远。加油。

      链接：http://pan.baidu.com/s/1gdjbKUR 密码：hnrk