Ubuntu下配置和运行Heritrix

1. 因为决定从源码运行Heritrix, 所以首先安装Eclipse:

$sudo apt-get install eclipse

2. 下载Heritrix源码:
http://sourceforge.net/projects/archive-crawler/files/
3. 解压:

$tar xzvf /home/xxx/workspace/heritrix-1.14.4-src.tar.gz

4. 将文件夹改名为heritrix(不是必须的, 这里是便于导入到eclipse):

$mv heritrix-1.14.4 heritrix

5. 打开eclipse, New->Java Project->Create Project from exsiting source

注意保证输入的Project Name和项目文件夹名称一致.

6. 进入Run Configurations, 指定Main class为

org.archive.crawler.Heritrix
添加VM arguments:

-Dheritrix.development -Xmx512m

添加Classpath: Advanced->Add External Folder, 将src/conf和src/resources添加到Classpath中. 否则会报两个错误:

Exception in thread "main" java.io.IOException: Failed to load properties file from filesystem or from classpath.

at org.archive.crawler.Heritrix.getPropertiesInputStream(Heritrix.java:924)

at org.archive.crawler.Heritrix.loadProperties(Heritrix.java:863)

at org.archive.crawler.Heritrix.containerInitialization(Heritrix.java:492)

at org.archive.crawler.Heritrix.main(Heritrix.java:555)
和

2010-07-10 10:03:01.250 严重 thread-10 org.archive.util.ArchiveUtils.() TLD list unavailable

java.lang.NullPointerException

at java.io.Reader.(Unknown Source)

at java.io.InputStreamReader.(Unknown Source)

at org.archive.util.ArchiveUtils.(ArchiveUtils.java:759)

........
更严重的是

, 会使Heritrix的Modules界面不能改变选择项

.

7. 其他诸如设置用户名密码, 添加和运行job之类的, 这里就不详细说明了, 以下是一些参考链接:

Heritrix User Manual

Heritrix developer documentation

基于lucence+heritrix的开源搜索引擎构建（一）

Heritrix使用的初步总结