您的位置：首页 > 其它

nutch-2.0执行过程及hbase下webpage的变化更新

2012-08-16 16:58 267 查看
今天把nutch-2.0整个过程执行了一边，并且分析了hbase下webpage的变化过程。测试环境是本地环境，即hbase只在本地运行。

整个执行过程如下：

1>inject

2>generate

3>fetch

4>parse

5>updatedb

6>solrindex

webpage的表结构可以参考nutch/conf/gora-hbase-mapping.xml，如下所示

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at
  
  http://www.apache.org/licenses/LICENSE-2.0   
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<gora-orm>
    
    <table name="webpage">
        <family name="p" maxVersions="1"/> <!-- This can also have params like compression, bloom filters -->
        <family name="f" maxVersions="1"/>
        <family name="s" maxVersions="1"/>
        <family name="il" maxVersions="1"/>
        <family name="ol" maxVersions="1"/>
        <family name="h" maxVersions="1"/>
        <family name="mtdt" maxVersions="1"/>
        <family name="mk" maxVersions="1"/>
    </table>
    <class table="webpage" keyClass="java.lang.String" name="org.apache.nutch.storage.WebPage">
        
        <!-- fetch fields     name="f"                                  -->
        <field name="baseUrl" family="f" qualifier="bas"/>
        <field name="status" family="f" qualifier="st"/>
        <field name="prevFetchTime" family="f" qualifier="pts"/>
        <field name="fetchTime" family="f" qualifier="ts"/>
        <field name="fetchInterval" family="f" qualifier="fi"/>
        <field name="retriesSinceFetch" family="f" qualifier="rsf"/>
        <field name="reprUrl" family="f" qualifier="rpr"/>
        <field name="content" family="f" qualifier="cnt"/>
        <field name="contentType" family="f" qualifier="typ"/>
        <field name="protocolStatus" family="f" qualifier="prot"/>
        <field name="modifiedTime" family="f" qualifier="mod"/>
        
        <!-- parse fields     name="p"                                  -->
        <field name="title" family="p" qualifier="t"/>
        <field name="text" family="p" qualifier="c"/>
        <field name="parseStatus" family="p" qualifier="st"/>
        <field name="signature" family="p" qualifier="sig"/>
        <field name="prevSignature" family="p" qualifier="psig"/>
        
        <!-- score fields    name="s"                                   -->
        <field name="score" family="s" qualifier="s"/>
        <field name="headers" family="h"/>
        <field name="inlinks" family="il"/>
        <field name="outlinks" family="ol"/>
        <field name="metadata" family="mtdt"/>
        <field name="markers" family="mk"/>
    </class>
    
    <table name="host">
      <family name="mtdt" maxVersions="1"/>
      <family name="il" maxVersions="1"/>
      <family name="ol" maxVersions="1"/>
    </table>
    
    <class table="host" keyClass="java.lang.String" name="org.apache.nutch.storage.Host">
      <field name="metadata" family="mtdt"/>
      <field name="inlinks" family="il"/>
      <field name="outlinks" family="ol"/>
    </class>
    
</gora-orm>


一、inject

在bin目录下创建名为urls的文件，加入一行：http://www.2345.com/

命令：./nutch inject urls

webpage表结构：

< f >

==============================================>

ROW com.2345.www:http/

COL f : fi //fetchInterval

COL f : ts //fetchTime

< h >

==============================================>

< il >

==============================================>

< mk >

==============================================>

ROW com.2345.www:http/

COL mk : _injmrk_ //inject标记

< mtdt >

==============================================>

ROW com.2345.www:http/

COL mtdt : _csh_ //???暂不明含义

< ol >

==============================================>

< p >

==============================================>

< s >

==============================================>

ROW com.2345.www:http/

COL s : s //score

生成的结构已经用红色标出了，含义参见注释。

二、generate

命令：./nutch generate

执行后webpage表的变化：

< f >

==============================================>

ROW com.2345.www:http/

COL f : fi

COL f : ts

< h >

==============================================>

< il >

==============================================>

< mk >

==============================================>

ROW com.2345.www:http/

COL mk : _gnmrk_

COL mk : _injmrk_

< mtdt >

==============================================>

ROW com.2345.www:http/

COL mtdt : _csh_

< ol >

==============================================>

< p >

==============================================>

< s >

==============================================>

ROW com.2345.www:http/

COL s : s

加红的是多出的一行，可见在generate之后做了一个标记

三、fetch

命令：./nutch fetch 1345101942-1642860074

执行后webpage表的变化：

< f >

==============================================>

ROW com.2345.www:http/

COL f : bas //baseUrl

COL f : cnt //content

COL f : fi //fetchInterval

COL f : prot //protocolStatus

COL f : pts //prevFetchTime

COL f : st //status

COL f : ts //fetchTime

COL f : typ //contentType

< h >

==============================================>

ROW com.2345.www:http/

COL h : Accept-Ranges

COL h : Cache-Control

COL h : Connection

COL h : Content-Encoding

COL h : Content-Length

COL h : Content-Type

COL h : Date

COL h : ETag

COL h : Expires

COL h : Last-Modified

COL h : Server

COL h : Vary

COL h : _ip

< il >

==============================================>

< mk >

==============================================>

ROW com.2345.www:http/

COL mk : _ftcmrk_

COL mk : _gnmrk_

COL mk : _injmrk_

< mtdt >

==============================================>

ROW com.2345.www:http/

COL mtdt : _csh_

< ol >

==============================================>

< p >

==============================================>

< s >

==============================================>

ROW com.2345.www:http/

COL s : s

加入了fetch的标记，加入了header信息，加入了fetch的相关信息，具体参见注释（伪的）

四、parse

< f >

==============================================>

ROW com.2345.www:http/

COL f : bas

COL f : cnt

COL f : fi

COL f : prot

COL f : pts

COL f : st

COL f : ts

COL f : typ

< h >

==============================================>

ROW com.2345.www:http/

COL h : Accept-Ranges

COL h : Cache-Control

COL h : Connection

COL h : Content-Encoding

COL h : Content-Length

COL h : Content-Type

COL h : Date

COL h : ETag

COL h : Expires

COL h : Last-Modified

COL h : Server

COL h : Vary

COL h : _ip

< il >

==============================================>

< mk >

==============================================>

ROW com.2345.www:http/

COL mk : __prsmrk__

COL mk : _ftcmrk_

COL mk : _gnmrk_

COL mk : _injmrk_

< mtdt >

==============================================>

ROW com.2345.www:http/

COL mtdt : _csh_

< ol >

==============================================>

ROW com.2345.www:http/

COL ol : http://182.xdwan.com/gg442a/

COL ol : http://2012.cntv.cn/

COL ol : http://2345.qq.okooo.com/shuangseqiu/

COL ol : http://25.198game.com/

COL ol : http://book.2345.com/yanqing.html

COL ol : http://buy.2345.com/

COL ol : http://cf.qq.com/

COL ol : http://china.nba.com/

COL ol : http://cn.wsj.com/

COL ol : http://danji.2345.com/

COL ol : http://day.2345.com/joke.htm

COL ol : http://dianying.2345.com/

COL ol : http://dianying.2345.com/detail/53555.html

COL ol : http://ditu.google.cn/

COL ol : http://email.163.com/

COL ol : http://esf.soufun.com/

COL ol : http://fcwr.jstv.com/

COL ol : http://finance.sina.com.cn/

COL ol : http://game.2345.com/

COL ol : http://im.qq.com/qq/all.shtml

COL ol : http://image.baidu.com/

COL ol : http://lux.2345.com/

COL ol : http://lvyou.2345.com/

COL ol : http://m.2345.com/pages/

COL ol : http://mail.10086.cn/

COL ol : http://mail.cn.yahoo.com/

COL ol : http://mail.qq.com/

COL ol : http://mail.sina.com.cn/

COL ol : http://military.china.com/

COL ol : http://mobile.pconline.com.cn/

COL ol : http://mobile.pcpop.com/

COL ol : http://mobile.zol.com.cn/

COL ol : http://mp3.baidu.com/

COL ol : http://mp3.sogou.com/

COL ol : http://my.2345.com/member/edit_info.php

COL ol : http://my.ku6.com/

COL ol : http://news.163.com/12/0815/04/88U0D1K60001124J.html

COL ol : http://news.7k7k.com/roco/

COL ol : http://news.baidu.com/

COL ol : http://news.ifeng.com/

COL ol : http://news.ifeng.com/mainland/special/cqqiangjiqiangjie/content-3/detail_2012_08/15/16829686_0.shtml

COL ol : http://news.ifeng.com/mainland/special/diaoyudaozhengduan/content-3/detail_2012_08/15/16832117_0.shtml

COL ol : http://news.ifeng.com/mainland/special/diaoyudaozhengduan/content-3/detail_2012_08/16/16844487_0.shtml

COL ol : http://news.ifeng.com/mil/

COL ol : http://news.qq.com/

COL ol : http://news.sina.com.cn/

COL ol : http://news.sohu.com/

COL ol : http://pic.2345.com/

COL ol : http://r.dangdang.com/2345kuzhan

COL ol : http://sg.198game.com/

COL ol : http://shouji.2345.com/

COL ol : http://sports.163.com/

COL ol : http://sports.sina.com.cn/

COL ol : http://sports.sohu.com/

COL ol : http://t.qq.com/

COL ol : http://tech.sina.com.cn/mobile/

COL ol : http://tianqi.2345.com/

COL ol : http://tianqi.2345.com/indexs.htm

COL ol : http://tieba.baidu.com/

COL ol : http://tieba.baidu.com/index.html

COL ol : http://tools.2345.com/

COL ol : http://tools.2345.com/weizhang.htm

COL ol : http://tools.2345.com/xingzuoyc.htm

COL ol : http://tools.2345.com/zhgjm.htm

COL ol : http://tuan.27.cn/

COL ol : http://tv.2345.com/detail/11406.html

COL ol : http://tv.2345.com/detail/11453.html

COL ol : http://v.2345.com/

COL ol : http://v.2345.com/dv/

COL ol : http://v.2345.com/live/

COL ol : http://v.2345.com/zongyi/

COL ol : http://v.2345.com/zt/40

COL ol : http://v.ifeng.com/

COL ol : http://video.baidu.com/

COL ol : http://wan.2345.com/sxd_a/

COL ol : http://wangyou.2345.com/

COL ol : http://web.7k7k.com/games/dpcq/2345dhz/2345dhz-wzl.html

COL ol : http://web.7k7k.com/games/frxz2/2345dhz/2345dhz-syyxztsan.html

COL ol : http://web.7k7k.com/games/lb/2345dhz/2345dhz-syyxzt6.html

COL ol : http://web.7k7k.com/games/lb/2345dhz/2345dhz-wzl6.html

COL ol : http://web.qq.com/

COL ol : http://ww.xitek.com/

COL ol : http://www.10010.com/

COL ol : http://www.10086.cn/

COL ol : http://www.12306.cn/mormhweb/

COL ol : http://www.126.com/

COL ol : http://www.163.com/

COL ol : http://www.17173.com/

COL ol : http://www.1ting.com/

COL ol : http://www.2345.com/about/about.htm

COL ol : http://www.2345.com/about/gyhd.htm

COL ol : http://www.2345.com/aihao.htm

COL ol : http://www.2345.com/art.htm

COL ol : http://www.2345.com/bank.htm

COL ol : http://www.2345.com/baoxian.htm

COL ol : http://www.2345.com/bbs.htm

COL ol : http://www.2345.com/blog.htm

COL ol : http://www.2345.com/caipiao.htm

COL ol : http://www.2345.com/car.htm

COL ol : http://www.2345.com/chat.htm

COL ol : http://www.2345.com/child.htm

COL ol : http://www.2345.com/city/

COL ol : http://www.2345.com/corp.htm

COL ol : http://www.2345.com/daxue.htm

COL ol : http://www.2345.com/desk.htm

COL ol : http://www.2345.com/edu.htm

COL ol : http://www.2345.com/eng.htm

COL ol : http://www.2345.com/fashion.htm

COL ol : http://www.2345.com/flower.htm

COL ol : http://www.2345.com/football.htm

COL ol : http://www.2345.com/gov.htm

COL ol : http://www.2345.com/gupiao.htm

COL ol : http://www.2345.com/haob.htm

COL ol : http://www.2345.com/hardware.htm

COL ol : http://www.2345.com/health.htm

COL ol : http://www.2345.com/help/szsy.htm

COL ol : http://www.2345.com/homepage.htm

COL ol : http://www.2345.com/house.htm

COL ol : http://www.2345.com/indexold.htm

COL ol : http://www.2345.com/indexz.htm

COL ol : http://www.2345.com/jianfei.htm

COL ol : http://www.2345.com/jijin.htm

COL ol : http://www.2345.com/job.htm

COL ol : http://www.2345.com/junshi.htm

COL ol : http://www.2345.com/kaoshi.htm

COL ol : http://www.2345.com/kv.htm

COL ol : http://www.2345.com/laonian.htm

COL ol : http://www.2345.com/law.htm

COL ol : http://www.2345.com/love.htm

COL ol : http://www.2345.com/mail.htm

COL ol : http://www.2345.com/manwoman.htm

COL ol : http://www.2345.com/map.htm

COL ol : http://www.2345.com/meishi.htm

COL ol : http://www.2345.com/mm.htm

COL ol : http://www.2345.com/navhtm/city.htm

COL ol : http://www.2345.com/nba.htm

COL ol : http://www.2345.com/newsweek.htm

COL ol : http://www.2345.com/pic.htm

COL ol : http://www.2345.com/qtwz.htm

COL ol : http://www.2345.com/quyi.htm

COL ol : http://www.2345.com/shaoer.htm

COL ol : http://www.2345.com/sheji.htm

COL ol : http://www.2345.com/shenghuo.htm

COL ol : http://www.2345.com/sheying.htm

COL ol : http://www.2345.com/shouji.htm

COL ol : http://www.2345.com/shuma.htm

COL ol : http://www.2345.com/shwz.htm

COL ol : http://www.2345.com/soft/

COL ol : http://www.2345.com/software.htm

COL ol : http://www.2345.com/sport.htm

COL ol : http://www.2345.com/tongxun.htm

COL ol : http://www.2345.com/weibo.htm

COL ol : http://www.2345.com/world.htm

COL ol : http://www.2345.com/xszn/xshelp.htm

COL ol : http://www.2345.com/ylwz.htm

COL ol : http://www.2345.com/yule.htm

COL ol : http://www.2345.com/zt/girl.htm

COL ol : http://www.37wan.com/tg/2345dao/

COL ol : http://www.37wan.com/tg/2345dao_jxqy

COL ol : http://www.39.net/

COL ol : http://www.4399.com/

COL ol : http://www.4738.com/

COL ol : http://www.5200tv.com/

COL ol : http://www.6.cn/

COL ol : http://www.7k7k.com/

COL ol : http://www.9ku.com/

COL ol : http://www.abchina.com/

COL ol : http://www.aili.com/

COL ol : http://www.alipay.com/

COL ol : http://www.baixing.com/

COL ol : http://www.bankcomm.com/

COL ol : http://www.boc.cn/

COL ol : http://www.ccb.com/

COL ol : http://www.cheshi.com/

COL ol : http://www.chinabug.net/

COL ol : http://www.chinaiiss.com/

COL ol : http://www.chinanews.com.cn/

COL ol : http://www.cmbchina.com/

COL ol : http://www.cnfol.com/

COL ol : http://www.cntv.cn/index.shtml

COL ol : http://www.digu.com/

COL ol : http://www.douban.com/

COL ol : http://www.duote.com/

COL ol : http://www.duote.com/soft/2054.html

COL ol : http://www.duote.com/soft/21205.html

COL ol : http://www.duote.com/soft/21413.html

COL ol : http://www.duote.com/soft/54347.html

COL ol : http://www.duote.com/soft/5841.html

COL ol : http://www.fortunechina.com/

COL ol : http://www.hexun.com/

COL ol : http://www.homeinns.com/

COL ol : http://www.hongxiu.com/

COL ol : http://www.hotmail.com/

COL ol : http://www.hupu.com/

COL ol : http://www.ifeng.com/

COL ol : http://www.imobile.com.cn/

COL ol : http://www.iqiyi.com/

COL ol : http://www.junshitx.com/

COL ol : http://www.jxdyf.com/

COL ol : http://www.jxedt.com/

COL ol : http://www.kaixin001.com/

COL ol : http://www.kanshu.com/

COL ol : http://www.ku6.cn/

COL ol : http://www.kugou.com/

COL ol : http://www.m1905.com/vod/

COL ol : http://www.miercn.com/

COL ol : http://www.miitbeian.gov.cn/

COL ol : http://www.mop.com/

COL ol : http://www.onlinedown.net/

COL ol : http://www.onlylady.com/

COL ol : http://www.pcpop.com/

COL ol : http://www.people.com.cn/

COL ol : http://www.qianyan001.com/

COL ol : http://www.qidian.com/

COL ol : http://www.qiushibaike.com/

COL ol : http://www.qq.com/

COL ol : http://www.readnovel.com/

COL ol : http://www.renren.com/

COL ol : http://www.ruichuang.net/join_us/

COL ol : http://www.sina.com.cn/

COL ol : http://www.skycn.com/

COL ol : http://www.sohu.com/

COL ol : http://www.stockstar.com/

COL ol : http://www.taobao.com/

COL ol : http://www.tianya.cn/

COL ol : http://www.tudou.com/

COL ol : http://www.vdolady.com/

COL ol : http://www.vogue.com.cn/

COL ol : http://www.xinhuanet.com/

COL ol : http://www.xunlei.com/

COL ol : http://www.xxsy.net/

COL ol : http://www.youku.com/

COL ol : http://www.zaobao.com/

COL ol : http://www.zb8.com/

COL ol : http://www.zhcw.com/

COL ol : http://www.zhenai.com/901002.html

COL ol : http://www.zhiji.com/

COL ol : http://www.zhulang.com/

COL ol : http://www.zol.com.cn/

COL ol : http://xiaoyouxi.2345.com/zhuanti/110.htm

COL ol : http://xiaoyouxi.2345.com/zhuanti/174.htm

COL ol : http://xiaoyouxi.2345.com/zhuanti/61.htm

COL ol : http://yule.iqiyi.com/zghsy.html

COL ol : http://zhidao.baidu.com/

< p >

==============================================>

ROW com.2345.www:http/

COL p : c //text

COL p : sig //signature

COL p : st //parseStatus

COL p : t //title

< s >

==============================================>

ROW com.2345.www:http/

COL s : s

加入了parse标记，加入了ol(outlink)外部链接，也即从刚才fetch的页面http://www.2345.com/内分析之后得到的链接，还加入了parse的信息，参见注释。

五、updatedb

命令：./nutch updatedb

执行后webpage的变化：

< f >

==============================================>

ROW cn.10086.mail:http/

COL f : fi

COL f : st //status

COL f : ts

ROW cn.10086.www:http/

COL f : fi

COL f : st

COL f : ts

ROW cn.12306.www:http/mormhweb/

COL f : fi

COL f : st

COL f : ts

ROW cn.27.tuan:http/

COL f : fi

COL f : st

COL f : ts

ROW cn.6.www:http/

COL f : fi

COL f : st

COL f : ts

ROW cn.boc.www:http/

< h >

==============================================>

ROW com.2345.www:http/

COL h : Accept-Ranges

COL h : Cache-Control

COL h : Connection

COL h : Content-Encoding

COL h : Content-Length

COL h : Content-Type

COL h : Date

COL h : ETag

COL h : Expires

COL h : Last-Modified

COL h : Server

COL h : Vary

COL h : _ip

< il >

==============================================>

ROW cn.10086.mail:http/

COL il : http://www.2345.com/

ROW cn.10086.www:http/

COL il : http://www.2345.com/

ROW cn.12306.www:http/mormhweb/

COL il : http://www.2345.com/

ROW cn.27.tuan:http/

COL il : http://www.2345.com/

ROW cn.6.www:http/

COL il : http://www.2345.com/

ROW cn.boc.www:http/

< mk >

==============================================>

ROW com.2345.www:http/

COL mk : _injmrk_

< mtdt >

==============================================>

ROW cn.10086.mail:http/

COL mtdt : _csh_

ROW cn.10086.www:http/

COL mtdt : _csh_

ROW cn.12306.www:http/mormhweb/

COL mtdt : _csh_

ROW cn.27.tuan:http/

COL mtdt : _csh_

ROW cn.6.www:http/

COL mtdt : _csh_

ROW cn.boc.www:http/

< ol >

==============================================>

ROW com.2345.www:http/

COL ol : http://182.xdwan.com/gg442a/

COL ol : http://2012.cntv.cn/

COL ol : http://2345.qq.okooo.com/shuangseqiu/

COL ol : http://25.198game.com/

COL ol : http://book.2345.com/yanqing.html

COL ol : http://buy.2345.com/

COL ol : http://cf.qq.com/

COL ol : http://china.nba.com/

COL ol : http://cn.wsj.com/

COL ol : http://danji.2345.com/

COL ol : http://day.2345.com/joke.htm

COL ol : http://dianying.2345.com/

COL ol : http://dianying.2345.com/detail/53555.html

COL ol : http://ditu.google.cn/

COL ol : http://email.163.com/

COL ol : http://esf.soufun.com/

COL ol : http://fcwr.jstv.com/

COL ol : http://finance.sina.com.cn/

COL ol : http://game.2345.com/

COL ol : http://im.qq.com/qq/all.shtml

COL ol : http://image.baidu.com/

COL ol : http://lux.2345.com/

COL ol : http://lvyou.2345.com/

COL ol : http://m.2345.com/pages/

COL ol : http://mail.10086.cn/

COL ol : http://mail.cn.yahoo.com/

COL ol : http://mail.qq.com/

COL ol : http://mail.sina.com.cn/

COL ol : http://military.china.com/

COL ol : http://mobile.pconline.com.cn/

COL ol : http://mobile.pcpop.com/

COL ol : http://mobile.zol.com.cn/

COL ol : http://mp3.baidu.com/

COL ol : http://mp3.sogou.com/

COL ol : http://my.2345.com/member/edit_info.php

COL ol : http://my.ku6.com/

COL ol : http://news.163.com/12/0815/04/88U0D1K60001124J.html

COL ol : http://news.7k7k.com/roco/

COL ol : http://news.baidu.com/

COL ol : http://news.ifeng.com/

COL ol : http://news.ifeng.com/mainland/special/cqqiangjiqiangjie/content-3/detail_2012_08/15/16829686_0.shtml

COL ol : http://news.ifeng.com/mainland/special/diaoyudaozhengduan/content-3/detail_2012_08/15/16832117_0.shtml

COL ol : http://news.ifeng.com/mainland/special/diaoyudaozhengduan/content-3/detail_2012_08/16/16844487_0.shtml

COL ol : http://news.ifeng.com/mil/

COL ol : http://news.qq.com/

COL ol : http://news.sina.com.cn/

COL ol : http://news.sohu.com/

COL ol : http://pic.2345.com/

COL ol : http://r.dangdang.com/2345kuzhan

COL ol : http://sg.198game.com/

COL ol : http://shouji.2345.com/

COL ol : http://sports.163.com/

COL ol : http://sports.sina.com.cn/

COL ol : http://sports.sohu.com/

COL ol : http://t.qq.com/

COL ol : http://tech.sina.com.cn/mobile/

COL ol : http://tianqi.2345.com/

COL ol : http://tianqi.2345.com/indexs.htm

COL ol : http://tieba.baidu.com/

COL ol : http://tieba.baidu.com/index.html

COL ol : http://tools.2345.com/

COL ol : http://tools.2345.com/weizhang.htm

COL ol : http://tools.2345.com/xingzuoyc.htm

COL ol : http://tools.2345.com/zhgjm.htm

COL ol : http://tuan.27.cn/

COL ol : http://tv.2345.com/detail/11406.html

COL ol : http://tv.2345.com/detail/11453.html

COL ol : http://v.2345.com/

COL ol : http://v.2345.com/dv/

COL ol : http://v.2345.com/live/

COL ol : http://v.2345.com/zongyi/

COL ol : http://v.2345.com/zt/40

COL ol : http://v.ifeng.com/

COL ol : http://video.baidu.com/

COL ol : http://wan.2345.com/sxd_a/

COL ol : http://wangyou.2345.com/

COL ol : http://web.7k7k.com/games/dpcq/2345dhz/2345dhz-wzl.html

COL ol : http://web.7k7k.com/games/frxz2/2345dhz/2345dhz-syyxztsan.html

COL ol : http://web.7k7k.com/games/lb/2345dhz/2345dhz-syyxzt6.html

COL ol : http://web.7k7k.com/games/lb/2345dhz/2345dhz-wzl6.html

COL ol : http://web.qq.com/

COL ol : http://ww.xitek.com/

COL ol : http://www.10010.com/

COL ol : http://www.10086.cn/

COL ol : http://www.12306.cn/mormhweb/

COL ol : http://www.126.com/

COL ol : http://www.163.com/

COL ol : http://www.17173.com/

COL ol : http://www.1ting.com/

COL ol : http://www.2345.com/about/about.htm

COL ol : http://www.2345.com/about/gyhd.htm

COL ol : http://www.2345.com/aihao.htm

COL ol : http://www.2345.com/art.htm

COL ol : http://www.2345.com/bank.htm

COL ol : http://www.2345.com/baoxian.htm

COL ol : http://www.2345.com/bbs.htm

COL ol : http://www.2345.com/blog.htm

COL ol : http://www.2345.com/caipiao.htm

COL ol : http://www.2345.com/car.htm

COL ol : http://www.2345.com/chat.htm

COL ol : http://www.2345.com/child.htm

COL ol : http://www.2345.com/city/

COL ol : http://www.2345.com/corp.htm

COL ol : http://www.2345.com/daxue.htm

COL ol : http://www.2345.com/desk.htm

COL ol : http://www.2345.com/edu.htm

COL ol : http://www.2345.com/eng.htm

COL ol : http://www.2345.com/fashion.htm

COL ol : http://www.2345.com/flower.htm

COL ol : http://www.2345.com/football.htm

COL ol : http://www.2345.com/gov.htm

COL ol : http://www.2345.com/gupiao.htm

COL ol : http://www.2345.com/haob.htm

COL ol : http://www.2345.com/hardware.htm

COL ol : http://www.2345.com/health.htm

COL ol : http://www.2345.com/help/szsy.htm

COL ol : http://www.2345.com/homepage.htm

COL ol : http://www.2345.com/house.htm

COL ol : http://www.2345.com/indexold.htm

COL ol : http://www.2345.com/indexz.htm

COL ol : http://www.2345.com/jianfei.htm

COL ol : http://www.2345.com/jijin.htm

COL ol : http://www.2345.com/job.htm

COL ol : http://www.2345.com/junshi.htm

COL ol : http://www.2345.com/kaoshi.htm

COL ol : http://www.2345.com/kv.htm

COL ol : http://www.2345.com/laonian.htm

COL ol : http://www.2345.com/law.htm

COL ol : http://www.2345.com/love.htm

COL ol : http://www.2345.com/mail.htm

COL ol : http://www.2345.com/manwoman.htm

COL ol : http://www.2345.com/map.htm

COL ol : http://www.2345.com/meishi.htm

COL ol : http://www.2345.com/mm.htm

COL ol : http://www.2345.com/navhtm/city.htm

COL ol : http://www.2345.com/nba.htm

COL ol : http://www.2345.com/newsweek.htm

COL ol : http://www.2345.com/pic.htm

COL ol : http://www.2345.com/qtwz.htm

COL ol : http://www.2345.com/quyi.htm

COL ol : http://www.2345.com/shaoer.htm

COL ol : http://www.2345.com/sheji.htm

COL ol : http://www.2345.com/shenghuo.htm

COL ol : http://www.2345.com/sheying.htm

COL ol : http://www.2345.com/shouji.htm

COL ol : http://www.2345.com/shuma.htm

COL ol : http://www.2345.com/shwz.htm

COL ol : http://www.2345.com/soft/

COL ol : http://www.2345.com/software.htm

COL ol : http://www.2345.com/sport.htm

COL ol : http://www.2345.com/tongxun.htm

COL ol : http://www.2345.com/weibo.htm

COL ol : http://www.2345.com/world.htm

COL ol : http://www.2345.com/xszn/xshelp.htm

COL ol : http://www.2345.com/ylwz.htm

COL ol : http://www.2345.com/yule.htm

COL ol : http://www.2345.com/zt/girl.htm

COL ol : http://www.37wan.com/tg/2345dao/

COL ol : http://www.37wan.com/tg/2345dao_jxqy

COL ol : http://www.39.net/

COL ol : http://www.4399.com/

COL ol : http://www.4738.com/

COL ol : http://www.5200tv.com/

COL ol : http://www.6.cn/

COL ol : http://www.7k7k.com/

COL ol : http://www.9ku.com/

COL ol : http://www.abchina.com/

COL ol : http://www.aili.com/

COL ol : http://www.alipay.com/

COL ol : http://www.baixing.com/

COL ol : http://www.bankcomm.com/

COL ol : http://www.boc.cn/

COL ol : http://www.ccb.com/

COL ol : http://www.cheshi.com/

COL ol : http://www.chinabug.net/

COL ol : http://www.chinaiiss.com/

COL ol : http://www.chinanews.com.cn/

COL ol : http://www.cmbchina.com/

COL ol : http://www.cnfol.com/

COL ol : http://www.cntv.cn/index.shtml

COL ol : http://www.digu.com/

COL ol : http://www.douban.com/

COL ol : http://www.duote.com/

COL ol : http://www.duote.com/soft/2054.html

COL ol : http://www.duote.com/soft/21205.html

COL ol : http://www.duote.com/soft/21413.html

COL ol : http://www.duote.com/soft/54347.html

COL ol : http://www.duote.com/soft/5841.html

COL ol : http://www.fortunechina.com/

COL ol : http://www.hexun.com/

COL ol : http://www.homeinns.com/

COL ol : http://www.hongxiu.com/

COL ol : http://www.hotmail.com/

COL ol : http://www.hupu.com/

COL ol : http://www.ifeng.com/

COL ol : http://www.imobile.com.cn/

COL ol : http://www.iqiyi.com/

COL ol : http://www.junshitx.com/

COL ol : http://www.jxdyf.com/

COL ol : http://www.jxedt.com/

COL ol : http://www.kaixin001.com/

COL ol : http://www.kanshu.com/

COL ol : http://www.ku6.cn/

COL ol : http://www.kugou.com/

COL ol : http://www.m1905.com/vod/

COL ol : http://www.miercn.com/

COL ol : http://www.miitbeian.gov.cn/

COL ol : http://www.mop.com/

COL ol : http://www.onlinedown.net/

COL ol : http://www.onlylady.com/

COL ol : http://www.pcpop.com/

COL ol : http://www.people.com.cn/

COL ol : http://www.qianyan001.com/

COL ol : http://www.qidian.com/

COL ol : http://www.qiushibaike.com/

COL ol : http://www.qq.com/

COL ol : http://www.readnovel.com/

COL ol : http://www.renren.com/

COL ol : http://www.ruichuang.net/join_us/

COL ol : http://www.sina.com.cn/

COL ol : http://www.skycn.com/

COL ol : http://www.sohu.com/

COL ol : http://www.stockstar.com/

COL ol : http://www.taobao.com/

COL ol : http://www.tianya.cn/

COL ol : http://www.tudou.com/

COL ol : http://www.vdolady.com/

COL ol : http://www.vogue.com.cn/

COL ol : http://www.xinhuanet.com/

COL ol : http://www.xunlei.com/

COL ol : http://www.xxsy.net/

COL ol : http://www.youku.com/

COL ol : http://www.zaobao.com/

COL ol : http://www.zb8.com/

COL ol : http://www.zhcw.com/

COL ol : http://www.zhenai.com/901002.html

COL ol : http://www.zhiji.com/

COL ol : http://www.zhulang.com/

COL ol : http://www.zol.com.cn/

COL ol : http://xiaoyouxi.2345.com/zhuanti/110.htm

COL ol : http://xiaoyouxi.2345.com/zhuanti/174.htm

COL ol : http://xiaoyouxi.2345.com/zhuanti/61.htm

COL ol : http://yule.iqiyi.com/zghsy.html

COL ol : http://zhidao.baidu.com/

< p >

==============================================>

ROW com.2345.www:http/

COL p : c

COL p : sig

COL p : st

COL p : t

< s >

==============================================>

ROW cn.10086.mail:http/

COL s : s //score

ROW cn.10086.www:http/

COL s : s

ROW cn.12306.www:http/mormhweb/

COL s : s

ROW cn.27.tuan:http/

COL s : s

ROW cn.6.www:http/

COL s : s

ROW cn.boc.www:http/

更新db之后，把之前parse得出的urls全部采用倒排索引的方式生成行键，与刚开始注入时相同，唯有一处是f:st，已经用蓝色标出，指fetch的状态。

整个过程简单的就是这样了，当然这里测试的只是初始的过程，稳定抓取时还会有复杂的情况出现。
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 点击这里给我发消息
标签： 
相关文章推荐
新的分享
章节导航