初识 scrapy 框架 - 安装
2018-01-25 07:20
357 查看
前面豆子学习了基本的urllib的模块,通过这个模块可以写一些简单的爬虫文件。如果要处理大中型的爬虫项目,urllib就显得比较low了,这个时候可以使用scrapy框架来实现,很多基本的处理在scrapy里面已经做好了。
首先来安装一下。推荐的流程如下:
首先升级一下pip
安装wheel
3.安装lxml
安装twisted(如果在线安装报错的话,可以考虑离线安装)
从https://www.lfd.uci.edu 上搜索twisted, 可以看见他有很多个版本
执行一下python命令看看当前的版本,可以看见我的版本是3.6.2,32位的
这样的话可以下载对应的版本twisted-17.9.0-cp36-win32.whl就行了。
下载之后,手动安装
最后安装scrapy
安装完成之后,执行一下scrapy, 看看是否工作
首先来安装一下。推荐的流程如下:
首先升级一下pip
C:\WINDOWS\system32>python -m pip install --upgrade pip Requirement already up-to-date: pip in c:\python36\lib\site-packages
安装wheel
C:\WINDOWS\system32>pip install wheel Requirement already satisfied: wheel in c:\python36\lib\site-packages
3.安装lxml
C:\WINDOWS\system32>pip install lxml Collecting lxml Downloading lxml-4.1.1-cp36-cp36m-win32.whl (3.2MB) 100% |████████████████████████████████| 3.2MB 307kB/s Installing collected packages: lxml Successfully installed lxml-4.1.1
安装twisted(如果在线安装报错的话,可以考虑离线安装)
从https://www.lfd.uci.edu 上搜索twisted, 可以看见他有很多个版本
执行一下python命令看看当前的版本,可以看见我的版本是3.6.2,32位的
c:\Users\yuan.li\Downloads>python Python 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
这样的话可以下载对应的版本twisted-17.9.0-cp36-win32.whl就行了。
下载之后,手动安装
c:\Users\yuan.li\Downloads>pip install Twisted-17.9.0-cp36-cp36m-win32.whl Processing c:\users\yuan.li\downloads\twisted-17.9.0-cp36-cp36m-win32.whl Requirement already satisfied: incremental>=16.10.1 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: Automat>=0.3.0 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: zope.interface>=4.0.2 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: hyperlink>=17.1.1 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: constantly>=15.1 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: six in c:\python36\lib\site-packages (from Automat>=0.3.0->Twisted==17.9.0) Requirement already satisfied: attrs in c:\python36\lib\site-packages (from Automat>=0.3.0->Twisted==17.9.0) Requirement already satisfied: setuptools in c:\python36\lib\site-packages (from zope.interface>=4.0.2->Twisted==17.9.0) Installing collected packages: Twisted Successfully installed Twisted-17.9.0
最后安装scrapy
c:\Users\yuan.li\Downloads>pip install scrapy Collecting scrapy Downloading Scrapy-1.5.0-py2.py3-none-any.whl (251kB) 100% |████████████████████████████████| 256kB 2.3MB/s Collecting pyOpenSSL (from scrapy) Downloading pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB) 100% |████████████████████████████████| 61kB 4.5MB/s Collecting cssselect>=0.9 (from scrapy) Downloading cssselect-1.0.3-py2.py3-none-any.whl Collecting parsel>=1.1 (from scrapy)
安装完成之后,执行一下scrapy, 看看是否工作
c:\Users\yuan.li\Downloads>scrapy Scrapy 1.5.0 - no active project Usage: scrapy <command> [options] [args] Available commands: bench Run quick benchmark test fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates runspider Run a self-contained spider (without creating a project) settings Get settings values shell Interactive scraping console startproject Create new project version Print Scrapy version view Open URL in browser, as seen by Scrapy [ more ] More commands available when run from project directory
相关文章推荐
- 初识Scrapy框架——在Pycharm上安装Scrapy
- 初识Scrapy框架——在Pycharm上安装Scrapy
- 爬虫实践之爬虫框架Scrapy安装
- Python爬虫框架Scrapy安装
- Python爬虫进阶三之Scrapy框架安装配置
- Scrapy框架Windows下安装
- Python爬虫进阶一之爬虫框架Scrapy安装配置
- Python2.7下安装Scrapy框架
- 利用Anaconda简单安装scrapy框架的方法
- Windows 系统Scrapy爬虫框架的安装
- python安装scrapy框架
- <scrapy>python 爬虫框架scrapy安装
- 安装Scrapy爬虫框架
- Python之Scrapy爬虫框架安装及简单使用
- 在linux和windows下安装python爬虫框架scrapy
- win7安装python爬虫框架scrapy全纪录
- Python爬虫框架Scrapy实战之安装
- Python爬虫进阶三之Scrapy框架安装配置
- 【Solution】Mac下安装爬虫框架Scrapy出错
- python 3.6.0 scrapy框架安装方法及疑难解决