高清图片爬取

目标网站：https://unsplash.com/



目的：爬取一些图片作为Java小程序的背景图

代码如下：

import requests,json,time,sys
from contextlib import closing
next_page =[]#存放下一页面网址
next_page_0='http://unsplash.com/napi/feeds/home'#起始爬取页面
next_page.append(next_page_0)#放入数组
headers = {'authorization':'Client-ID c94869b36aa272dd62dfaeefed769d4115fb3189a9d1ec88ed457207747be626'}
headers_1 = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36'}
filename=1
download_server='https://unsplash.com/photos/xxx/download?force=trues'
k=1
for i in range(40):
photos_id = []#存放图片id
req = requests.get(url=next_page[i],headers=headers,verify=False)
html = json.loads(req.text)#html是经过一系列措施之后得到的网站信息，跟平常的静态网站信息类似，因为是动态态网站，所以不能直接得到
next_page_t = html['next_page']#找到下一个页面的链接
print(next_page_t +'找到了下一页的地址')
next_page.append(next_page_t)
for each in html['photos']:#进入页面的photos标签下寻找该页面下照片id
photos_id.append(each['id'])#照片id存入数组
print(each['id'])
time.sleep(1)
for j in range(len(photos_id)):#照片下载
print('  正在爬取第%d张图片' % (k))
k=k+1
target = download_server.replace('xxx',photos_id[j])
with closing(requests.get(url=target,stream=True,verify=False,headers=headers)) as r:
with open(('E:/照片爬取/'+ '%d.jpg' % filename), 'ab+') as f:
filename=filename+1
for chunk in r.iter_content(chunk_size = 1024):
if chunk:
f.write(chunk)
time.sleep(1)

爬取结果如下：



程序中预计爬取40张，但17张够用了，所以停止了爬取，但是存在一个问题，就是爬取的速度很慢，还没有想好怎么解决这个问题。。。。。