Nutch2.3 + Mysql（错误解决方案）

试了几次，最后没有成功 导致错误的原因是一些包的版本问题 （如果你还想用Nutch2.3+Mysql 那么再搜一下有没有对应的教程）

最后我用了Nutch2.2.1+Mysql （参考了 http://blog.csdn.net/david_snjly/article/details/42965017） 其中有些步骤会出现问题 在下面进行说明

sonar-ant-task-2.2.jar下载地址 http://download.csdn.net/detail/u010167215/9186933
jdk 1.7 tools.jar 下载地址 http://download.csdn.net/detail/u010167215/9186897
一、先说2.3的问题2.2.3一共会遇到两个问题 Google了一圈没有找到方案

问题1：执行 crawl 出现了下面的问题， ant自动下载的是1.7.6版本的

看了1.7.6的API org.apache.avro.ipc 确实没有ByteBufferOutputStream这个方法（可以导入到Eclipse中 换一下低版本的avro试一试，我没有试成功 ）

我把ivy/ivy.xml中的存储部分 Hbase给注释掉 只剩下Mysql ，问题解决了，但是出现了问题2

问题1：

at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:264)at org.apache.nutch.storage.StorageUtils.getDataStoreClass(StorageUtils.java:93)at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:77)at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:218)at
org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)Caused
by: java.lang.ClassNotFoundException: org.apache.avro.ipc.ByteBufferOutputStreamat java.net.URLClassLoader.findClass(URLClassLoader.java:381)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)at
java.lang.ClassLoader.loadClass(ClassLoader.java:357)... 9 moreError running:

问题2： 重新ant runtime 再crawl的过程 （出现了下面的问题）我看了bulid中的lib 发现有多个版本的avro包 有1.3 和1.7.6 （所以我就放弃了）

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.avro.Schema.access$1400()Ljava/lang/ThreadLocal;
at org.apache.avro.Schema$Parser.parse(Schema.java:950)
at org.apache.avro.Schema$Parser.parse(Schema.java:943)
at org.apache.nutch.storage.WebPage.<clinit>(WebPage.java:30)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.gora.util.ReflectionUtils.newInstance(ReflectionUtils.java:76)
at org.apache.gora.persistency.impl.BeanFactoryImpl.<init>(BeanFactoryImpl.java:53)
at org.apache.gora.store.impl.DataStoreBase.initialize(DataStoreBase.java:80)
at org.apache.gora.sql.store.SqlStore.initialize(SqlStore.java:146)
at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:135)
at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:78)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:218)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)

二、2.2.1的问题（参考 http://blog.csdn.net/david_snjly/article/details/42965017） 1.Mysql建标过程中，把长度位767的改为255

2.在最后执行Crawl过程中 如果出现以下问题（在/runtime/local/ 下建立logs/hadoop.log，并确定权限 最简单的就是 chmod 777 logs/hadooplog）

Injecting seed URLs
/usr/pro/apache-nutch-2.3/runtime/local/bin/nutch inject urls -crawlId -depth
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/pro/apache-nutch-2.3/runtime/local/logs/hadoop.log (没有那个文件或目录)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288)
at org.apache.nutch.crawl.InjectorJob.<clinit>(InjectorJob.java:64)

3.在执行ant runtime过程中 可能出现以下问题 （原因是tools版本太高了 我试了jdk8的tools发现不行 后来改为jdk7的tools就可以了 前文有下载地址）

/usr/pro/apache-nutch-2.3/build.xml:101: java.lang.UnsupportedClassVersionError: com/sun/tools/javac/Main : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at org.apache.tools.ant.taskdefs.compilers.CompilerAdapterFactory.doesModernCompilerExist(CompilerAdapterFactory.java:167)
at org.apache.tools.ant.taskdefs.compilers.CompilerAdapterFactory.getCompiler(CompilerAdapterFactory.java:126)
at org.apache.tools.ant.taskdefs.Javac.findSupportedFileExtensions(Javac.java:979)
at org.apache.tools.ant.taskdefs.Javac.scanDir(Javac.java:956)
at org.apache.tools.ant.taskdefs.Javac.execute(Javac.java:927)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:435)
at org.apache.tools.ant.Target.performTasks(Target.java:456)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393)
at org.apache.tools.ant.Project.executeTarget(Project.java:1364)
at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1248)
at org.apache.tools.ant.Main.runBuild(Main.java:851)
at org.apache.tools.ant.Main.startAnt(Main.java:235)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)

3.在执行ant runtime过程中 可能出现以下问题 （原因是tools版本太高了 我试了jdk8的tools发现不行 后来改为jdk7的tools就可以了 前文有下载地址）

4.在执行ant runtime过程中 可能出现以下问题 （原因是nucth/lib下缺少了Sonar.jar 前文有下载地址 把包下载后放到lib文件爱你夹中就可以了）

<!-- Define the Sonar task if this hasn't been done in a common script -->
<taskdef uri="antlib:org.sonar.ant" resource="org/sonar/ant/antlib.xml">
<classpath path="${ant.library.dir}" />
<classpath path="${mysql.library.dir}" />
<classpath><fileset dir="lib/" includes="sonar*.jar" /></classpath>
</taskdef>

5.在执行ant runtime过程中 可能出现以下问题（原因是ant执行过程中需要tools 但是系统默认的jre没有tools 下载完放进去就可以了 前文有下载地址）

BUILD FAILED
/usr/pro/apache-nutch-2.3/build.xml:101: Unable to find a javac compiler;
com.sun.tools.javac.Main is not on the classpath.
Perhaps JAVA_HOME does not point to the JDK.
It is currently set to "/usr/lib/jvm/java-7-openjdk-amd64/jre"