Nutch基本命令

1：nutch读取hbase数据导出文本文件：./nutch readdb -dump /data/nutch_db/1108 -crawlId TestCrawl -content
会执行一个mr程序，/data/nutch_db/1108是mr的输出路径TestCrawl是hbase表名的前半部分。
2：
 inject         inject new urls into the database
 hostinject     creates or updates an existing host table from a text file
 generate       generate new batches to fetch from crawl db
 fetch          fetch URLs marked during generate
 parse          parse URLs marked during fetch
 updatedb       update web table after parsing
 updatehostdb   update host table after parsing
 readdb         read/dump records from page database
 readhostdb     display entries from the hostDB
 elasticindex   run the elasticsearch indexer
 solrindex      run the solr indexer on parsed batches
 solrdedup      remove duplicates from solr
 parsechecker   check the parser for a given url
 indexchecker   check the indexing filters for a given url
 plugin         load a plugin and run one of its classes main()
 nutchserver    run a (local) Nutch server on a user defined port
 junit          runs the given JUnit test
 or
 CLASSNAME      run the class named CLASSNAME
Most commands print help when invoked w/o parameters.
3：./crawl <seedDir> <crawlID> <solrURL> <numberOfRounds>
4，
    ./nutch inject /nutch/urls/seed.txt -crawlId 7day把url放入对应的hbase以bbs开头的bbs_webpage表中
    ./nutch generate -topN 5 -crawlId 7day
    ./nutch fetch -all -crawlId 7day -threads 20
    ./nutch parse -all -crawlId 7day
    ./nutch updatedb -all -crawlId 7day
    ./nutch index -D solr.server.url=http://192.168.4.129:8983/solr/ -all -crawlId 7day
    ./nutch readdb -crawlId 7day_beijing -dump /home/nutch_output/beijing_1/
5，gora-hbase-mapping.xml该文件定义了列族及列的含义