scrapy安装过程中遇到的问题及解决

pip和easy_install配置好了之后，准备安装scrapy



第一部分

执行pip  install scrapy

You are using pip version 6.0.8, however version 9.0.1 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

    Downloading cryptography-2.0.3.tar.gz (427kB)

   5% |#                               | 24kB 10kB/s eta 0:00:39  Exception:

 Traceback (most recent call last):

   File "C:\Python34\lib\site-packages\pip\_vendor\requests\packages\urllib3\re

ponse.py", line 186, in read

     data = self._fp.read(amt)

   File "C:\Python34\lib\site-packages\pip\_vendor\cachecontrol\filewrapper.py"

 line 49, in read

     data = self.__fp.read(amt)

   File "C:\Python34\lib\http\client.py", line 500, in read

     return super(HTTPResponse, self).read(amt)

   File "C:\Python34\lib\http\client.py", line 539, in readinto

     n = self.fp.readinto(b)

   File "C:\Python34\lib\socket.py", line 374, in readinto

     return self._sock.recv_into(b)

   File "C:\Python34\lib\ssl.py", line 751, in recv_into

     return self.read(nbytes, buffer)

   File "C:\Python34\lib\ssl.py", line 623, in read

     v = self._sslobj.read(len, buffer)

 socket.timeout: The read operation timed out

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):

   File "C:\Python34\lib\site-packages\pip\basecommand.py", line 232, in main

     status = self.run(options, args)

   File "C:\Python34\lib\site-packages\pip\commands\install.py", line 339, in r

n

     requirement_set.prepare_files(finder)

   File "C:\Python34\lib\site-packages\pip\req\req_set.py", line 355, in prepar

_files

     do_download, session=self.session,

   File "C:\Python34\lib\site-packages\pip\download.py", line 782, in unpack_ur

     session,

   File "C:\Python34\lib\site-packages\pip\download.py", line 667, in unpack_ht

p_url

     from_path, content_type = _download_http_url(link, session, temp_dir)

   File "C:\Python34\lib\site-packages\pip\download.py", line 843, in _download

http_url

     _download_url(resp, link, content_file)

   File "C:\Python34\lib\site-packages\pip\download.py", line 615, in _download

url

     for chunk in progress_indicator(resp_read(4096), 4096):

   File "C:\Python34\lib\site-packages\pip\utils\ui.py", line 46, in iter

     for x in it:

   File "C:\Python34\lib\site-packages\pip\download.py", line 580, in resp_read

     decode_content=False):

   File "C:\Python34\lib\site-packages\pip\_vendor\requests\packages\urllib3\re

ponse.py", line 256, in stream

     data = self.read(amt=amt, decode_content=decode_content)

   File "C:\Python34\lib\site-packages\pip\_vendor\requests\packages\urllib3\re

ponse.py", line 201, in read

     raise ReadTimeoutError(self._pool, None, 'Read timed out.')

 pip._vendor.requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnec

ionPool(host='pypi.python.org', port=443): Read timed out.

=================================

第二部分

感觉可能是pip版本过低造成的，先升级下pip版本

pip install --upgrade pip

  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)

    100% |################################| 1.3MB 119kB/s

Installing collected packages: pip

  Found existing installation: pip 6.0.8

    Uninstalling pip-6.0.8:

      Successfully uninstalled pip-6.0.8

  Exception:

  Traceback (most recent call last):

    File "C:\Python34\lib\shutil.py", line 371, in _rmtree_unsafe

      os.unlink(fullname)

  PermissionError: [WinError 5] 拒绝访问。: 'C:\\DOCUME~1\\ADMINI~1\\LOCALS~1\\T

emp\\pip-d1yje83k-uninstall\\python34\\scripts\\pip.exe'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):

    File "C:\Python34\lib\site-packages\pip\basecommand.py", line 232, in main

      logger.debug('Exception information:', exc_info=True)

    File "C:\Python34\lib\site-packages\pip\commands\install.py", line 347, in r

un

      home=temp_target_dir,

    File "C:\Python34\lib\site-packages\pip\req\req_set.py", line 560, in instal

l

      # Then inside find_requirement existing_applicable -> False

    File "C:\Python34\lib\site-packages\pip\req\req_install.py", line 680, in co

mmit_uninstall

      )

    File "C:\Python34\lib\site-packages\pip\req\req_uninstall.py", line 153, in

commit

      self.file = pth_file

    File "C:\Python34\lib\site-packages\pip\_vendor\retrying.py", line 49, in wr

apped_f

      return Retrying(*dargs, **dkw).call(f, *args, **kw)

    File "C:\Python34\lib\site-packages\pip\_vendor\retrying.py", line 212, in c

all

      raise attempt.get()

    File "C:\Python34\lib\site-packages\pip\_vendor\retrying.py", line 247, in g

et

      six.reraise(self.value[0], self.value[1], self.value[2])

    File "C:\Python34\lib\site-packages\pip\_vendor\six.py", line 659, in rerais

e

      _assertCountEqual = "assertItemsEqual"

    File "C:\Python34\lib\site-packages\pip\_vendor\retrying.py", line 200, in c

all

      attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

    File "C:\Python34\lib\site-packages\pip\utils\__init__.py", line 61, in rmtr

ee

      SUPPORTED_EXTENSIONS += BZ2_EXTENSIONS

    File "C:\Python34\lib\shutil.py", line 478, in rmtree

      return _rmtree_unsafe(path, onerror)

    File "C:\Python34\lib\shutil.py", line 368, in _rmtree_unsafe

      _rmtree_unsafe(fullname, onerror)

    File "C:\Python34\lib\shutil.py", line 368, in _rmtree_unsafe

      _rmtree_unsafe(fullname, onerror)

    File "C:\Python34\lib\shutil.py", line 373, in _rmtree_unsafe

      onerror(os.unlink, fullname, sys.exc_info())

    File "C:\Python34\lib\site-packages\pip\utils\__init__.py", line 73, in rmtr

ee_errorhandler

      def import_or_raise(pkg_or_module_string, ExceptionType, *args, **kwargs):

  PermissionError: [WinError 5] 拒绝访问。: 'C:\\DOCUME~1\\ADMINI~1\\LOCALS~1\\T

emp\\pip-d1yje83k-uninstall\\python34\\scripts\\pip.exe'


权限有问题，好吧，以管理员身份打开命令行

=============

运行pip --version ,这时已经是

pip 9.0.1 from C:\Python34\lib\site-packages (python 3.4)了

pip升级成功！！！

[b]第三部分
[/b]

在升级好了Pip之后继续安装scrapy

pip install scrapy


这次运行的时间稍微长了一些，但是仍然没有成功

    copying src\twisted\words\xish\xpathparser.g -> build\lib.win32-3.4\twisted\

words\xish

    running build_ext

    building 'twisted.test.raiser' extension

    error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat).

    ----------------------------------------

Command "C:\Python34\python.exe -u -c "import setuptools, tokenize;__file__='C:\

\DOCUME~1\\ADMINI~1\\LOCALS~1\\Temp\\pip-build-khxnalsf\\Twisted\\setup.py';f=ge

tattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.cl

ose();exec(compile(code, __file__, 'exec'))" install --record C:\DOCUME~1\ADMINI

~1\LOCALS~1\Temp\pip-e0vtso7q-record\install-record.txt --single-version-externa

lly-managed --compile" failed with error code 1 in C:\DOCUME~1\ADMINI~1\LOCALS~1

\Temp\pip-build-khxnalsf\Twisted\


意思就是twisted 没有安装

第四部分

通过查找资料，找到了如下链接

http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted


找到对应的版本并下载

C:\Documents and Settings\Administrator>pip install "C:\Documents and Settings\A

dministrator\My Documents\下载\Twisted-17.5.0-cp34-cp34m-win32.whl"

Processing c:\documents and settings\administrator\my documents\下载\twisted-17.

5.0-cp34-cp34m-win32.whl

Requirement already satisfied: incremental>=16.10.1 in c:\python34\lib\site-pack

ages (from Twisted==17.5.0)

Requirement already satisfied: hyperlink>=17.1.1 in c:\python34\lib\site-package

s (from Twisted==17.5.0)

Requirement already satisfied: constantly>=15.1 in c:\python34\lib\site-packages

 (from Twisted==17.5.0)

Requirement already satisfied: zope.interface>=4.0.2 in c:\python34\lib\site-pac

kages (from Twisted==17.5.0)

Requirement already satisfied: Automat>=0.3.0 in c:\python34\lib\site-packages (

from Twisted==17.5.0)

Requirement already satisfied: setuptools in c:\python34\lib\site-packages (from

 zope.interface>=4.0.2->Twisted==17.5.0)

Requirement already satisfied: attrs in c:\python34\lib\site-packages (from Auto

mat>=0.3.0->Twisted==17.5.0)

Requirement already satisfied: six in c:\python34\lib\site-packages (from Automa

t>=0.3.0->Twisted==17.5.0)

Installing collected packages: Twisted

Successfully installed Twisted-17.5.0


可以看到，Twisted通过手动下载手动安装的方式成功安装了 

第五部分

我们继续安装scrapy

[b]pip install scrapy
[/b]



[b]Successfully installed asn1crypto-0.22.0 cffi-1.10.0 cryptography-2.0.3 cssselec

t-1.0.1 idna-2.6 lxml-3.8.0 parsel-1.2.0 pyOpenSSL-17.2.0 pyasn1-0.3.2 pyasn1-mo

dules-0.0.11 pycparser-2.18 queuelib-1.4.2 scrapy-1.4.0 service-identity-17.0.0
[/b]



执行 scrapy命令

C:\Documents and Settings\Administrator>scrapy

Scrapy 1.4.0 - no active project

Usage:

  scrapy <command> [options] [args]

Available commands:

  bench         Run quick benchmark test

  fetch         Fetch a URL using the Scrapy downloader

  genspider     Generate new spider using pre-defined templates

  runspider     Run a self-contained spider (without creating a project)

  settings      Get settings values

  shell         Interactive scraping console

  startproject  Create new project

  version       Print Scrapy version

  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

C:\Documents and Settings\Administrator>


看来，历经千辛万苦，我们的Scrapy终于安装成功了。。。

==================================================

说明：

之前在另外一台win7-64位系统机器下安装scrapy并没有这么麻烦，

看来对32位的机器的支持越来越不好了。