抓取豆瓣2016年电影/分类_python

Description

嗯，这次简单点

突然很想看电影，于是就抄起了python搞了一发豆瓣的电影年度清单，顺便统计了评分排名和分类之类的。还算简单吧

16年电影都在这个链接(大概)

'https://movie.douban.com/j/search_subjects?type=movie&tag=%E7%83%AD%E9%97%A8&sort=time&page_limit=365&page_start=0'

这里其实是可以get传输直接访问豆瓣的，也能访问这个链接，limit是显示多少条，设一个比较大的数字就能反馈全部电影了

大概长这样



想过用beautifulsoup但是不行，老老实实re匹配去吧

趴下来之后储存在一个dict里面，至于按key排序就比较好玩了。我们可以先记录一下dict的key生成list，然后对list排序，那么遍历这个list对应的dict值就是排好序的了

具体代码

d = {}
d['olahiuj'] = 'handsome'
for key in sorted(d.keys()):
print d[key]

推荐用sorted而不是sort，因为它不改变原本的列表

j接下来就是解析抓到的网址对应找类别，不说了就是re匹配。这一块特别慢可以多线程，但是注意访问避免过频繁尽量像真人一点(笑

r然后呢我们还是用dict来保存类别和对应的计数，输出到一个csv里面保存

0python是自带csv模块的引用就好了

import csv

0之所以选择csv而不是其他主要是因为csv能用excel编辑浏览

0写操作我们这么做

with open('filename.csv', 'wb') as csvfile:
blah = csv.writer(csvfile, dialect = 'excel')
blah.writerow([1, 2, 3])

w为了保证list中的每一个项目都能处在单独的列里，设置dialect为’excel’，还有就是输出一定要是list(大概?

b本来还想着要可视化一下数据建个图什么的，明天再弄吧。话说同性分类有11部电影是什么鬼，排名第一是又是什么鬼

Code

# -*- coding: utf-8 -*-
from requests.packages.urllib3.exceptions import InsecureRequestWarning
import threading
import requests
import time
import csv
import os
import re

def getPage(html, url, headers, params = {}, referer = ''):
flags = True
if (url[:5] == 'https'):
flags = False
headers['Referer'] = referer
response = html.get(url, headers = headers, params = params, verify = flags)
page = response.content
return page

def find(string, page, flags = 0):
pattern = re.compile(string, flags = flags)
results = re.findall(pattern, page)
return results

def work(html, url, headers, cnt):
tmp = ''
for q in url:
if q != '\\':
tmp = tmp + q
url = tmp
page = getPage(html, url, headers)
types = find(r'<span property="v:genre">(.+?)</span>', page)
global mutex, rec
mutex.acquire()
print cnt
for item in types:
if rec.has_key(item):
rec[item] += 1
else:
rec[item] = 1
mutex.release()

def init():
html = requests.session()
doubanUrl = 'https://movie.douban.com'
headers={'User-Agent':'Mozilla/5.0 (compatible; MSIE 5.5; Windows NT)'}
page = getPage(html, 'https://movie.douban.com/j/search_subjects', headers, params = {'type': 'movie', 'tag': '热门', 'sort': 'time', 'page_limit': '400', 'page_start': '0'})
results = find(r'"rate":"(.+?)",.+?"title":"(.+?)","url":"(.+?)"', page)
urls = [item[2] for item in results]
rates = [item[0] for item in results]
titles = [item[1] for item in results]

for i in xrange(len(urls)):
for j in xrange(i + 1, len(urls)):
if (rates[i] < rates[j]):
rates[i], rates[j] = rates[j], rates[i]
urls[i], urls[j] = urls[j], urls[i]
titles[i], titles[j] = titles[j], titles[i]

with open('douban.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, dialect = 'excel')
for i in xrange(len(rates)):
spamwriter.writerow([titles[i], urls[i], rates[i]])
global mutex, rec
mutex = threading.Lock()
rec = {}
jobs = []
cnt = 0
for i in xrange(len(urls)):
cnt += 1
job = threading.Thread(target = work, args = (html, urls[i], headers, cnt))
job.start()
jobs.append(job)

for job in jobs:
job.join()

with open('douban_type.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, dialect = 'excel')
for key in sorted(rec.keys(), reverse = True):
spamwriter.writerow([key, rec[key]])

if __name__ == '__main__':
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
init()