spark sql 访问hive数据时找不mysql的解决方法
2014-08-14 18:07
477 查看
1现象:
错误如下:org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table SOGOUQ1
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:958)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:924)
at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:54)
at org.apache.spark.sql.hive.HiveContext$anon$1.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$super$lookupRelation(HiveContext.scala:143)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$anonfun$lookupRelation$3.apply(Catalog.scala:96)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$anonfun$lookupRelation$3.apply(Catalog.scala:96)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:96)
at org.apache.spark.sql.hive.HiveContext$anon$1.lookupRelation(HiveContext.scala:149)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$anonfun$apply$2.applyOrElse(Analyzer.scala:83)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$anonfun$apply$2.applyOrElse(Analyzer.scala:81)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:165)
at org.apache.spark.sql.catalyst.trees.TreeNode$anonfun$4.apply(TreeNode.scala:183)
at scala.collection.Iterator$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:212)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:168)
at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:156)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:81)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:80)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1$anonfun$apply$2.apply(RuleExecutor.scala:62)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1$anonfun$apply$2.apply(RuleExecutor.scala:60)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1.apply(RuleExecutor.scala:60)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1.apply(RuleExecutor.scala:52)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:52)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:313)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:313)
at org.apache.spark.sql.hive.HiveContext$QueryExecution.optimizedPlan$lzycompute(HiveContext.scala:248)
at org.apache.spark.sql.hive.HiveContext$QueryExecution.optimizedPlan(HiveContext.scala:247)
at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:316)
at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:316)
at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:319)
at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:319)
at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:405)
at $iwC$iwC$iwC$iwC.<init>(<console>:15)
at $iwC$iwC$iwC.<init>(<console>:20)
at $iwC$iwC.<init>(<console>:22)
at $iwC.<init>(<console>:24)
at <init>(<console>:26)
at .<init>(<console>:30)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
at org.apache.spark.repl.SparkILoop$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
at org.apache.spark.repl.SparkILoop$anonfun$process$1.apply(SparkILoop.scala:884)
at org.apache.spark.repl.SparkILoop$anonfun$process$1.apply(SparkILoop.scala:884)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1212)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:950)
... 86 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
... 91 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:781)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:275)
at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:304)
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:234)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:209)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:64)
at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
... 96 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:281)
at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:239)
at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:292)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1069)
at org.datanucleus.NucleusContext.initialise(NucleusContext.java:359)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:768)
... 125 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BoneCP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:237)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:110)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
... 143 more
Caused by: org.datanucleus.store.rdbms.datasource.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.datasource.AbstractDataSourceFactory.loadDriver(AbstractDataSourceFactory.java:58)
at org.datanucleus.store.rdbms.datasource.BoneCPDataSourceFactory.makePooledDataSource(BoneCPDataSourceFactory.java:61)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:217)
... 145 more
我尝试着在classpath中加n入mysql的驱动仍不行
解决方法:在启动的时候加入参数--driver-class中加入mysql 驱动
[hadoop@master spark-1.0.1-bin-hadoop2]$ bin/spark-shell --driver-class-path lib/mysql-connector-java-5.1.30-bin.jar Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/14 16:07:14 INFO spark.SecurityManager: Changing view acls to: hadoop 14/08/14 16:07:14 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop) 14/08/14 16:07:15 INFO spark.HttpServer: Starting HTTP Server 14/08/14 16:07:15 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/08/14 16:07:15 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:43007 Welcome to ____ __ / __/__ ___ _____/ /__ _ / _ / _ `/ __/ '_/ /___/ .__/_,_/_/ /_/_ version 1.0.1 /_/ Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_65) Type in expressions to have them evaluated. Type :help for more information. 14/08/14 16:07:25 INFO spark.SecurityManager: Changing view acls to: hadoop 14/08/14 16:07:25 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop) 14/08/14 16:07:26 INFO slf4j.Slf4jLogger: Slf4jLogger started 14/08/14 16:07:26 INFO Remoting: Starting remoting 14/08/14 16:07:27 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@master:60674] 14/08/14 16:07:27 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@master:60674] 14/08/14 16:07:27 INFO spark.SparkEnv: Registering MapOutputTracker 14/08/14 16:07:27 INFO spark.SparkEnv: Registering BlockManagerMaster 14/08/14 16:07:27 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140814160727-d7c2 14/08/14 16:07:27 INFO storage.MemoryStore: MemoryStore started with capacity 297.0 MB. 14/08/14 16:07:27 INFO network.ConnectionManager: Bound socket to port 57661 with id = ConnectionManagerId(master,57661) 14/08/14 16:07:27 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/08/14 16:07:27 INFO storage.BlockManagerInfo: Registering block manager master:57661 with 297.0 MB RAM 14/08/14 16:07:27 INFO storage.BlockManagerMaster: Registered BlockManager 14/08/14 16:07:27 INFO spark.HttpServer: Starting HTTP Server 14/08/14 16:07:27 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/08/14 16:07:27 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:41277 14/08/14 16:07:27 INFO broadcast.HttpBroadcast: Broadcast server started at http://192.168.200.182:41277 14/08/14 16:07:27 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-b5c432fe-e4fb-45d0-8514-ba50fea348c9 14/08/14 16:07:27 INFO spark.HttpServer: Starting HTTP Server 14/08/14 16:07:27 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/08/14 16:07:27 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:46317 14/08/14 16:07:38 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/08/14 16:07:38 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 14/08/14 16:07:38 INFO ui.SparkUI: Started SparkUI at http://master:4040 14/08/14 16:07:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/14 16:07:39 INFO executor.Executor: Using REPL class URI: http://192.168.200.182:43007 14/08/14 16:07:40 INFO repl.SparkILoop: Created spark context.. Spark context available as sc. scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/08/14 16:07:53 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@30667a81 scala> scala> hiveContext.hql("Select count(*) from SOGOUQ1").collect().foreach(println) 14/08/14 16:08:15 INFO parse.ParseDriver: Parsing command: Select count(*) from SOGOUQ1 14/08/14 16:08:15 INFO parse.ParseDriver: Parse Completed 14/08/14 16:08:15 INFO analysis.Analyzer: Max iterations (2) reached for batch MultiInstanceRelations 14/08/14 16:08:15 INFO analysis.Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences 14/08/14 16:08:16 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 14/08/14 16:08:16 INFO metastore.ObjectStore: ObjectStore, initialize called 14/08/14 16:08:16 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 14/08/14 16:08:16 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 14/08/14 16:08:17 WARN bonecp.BoneCPConfig: Max Connections < 1. Setting to 20 14/08/14 16:08:28 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 14/08/14 16:08:28 INFO metastore.ObjectStore: Initialized ObjectStore 14/08/14 16:08:30 WARN bonecp.BoneCPConfig: Max Connections < 1. Setting to 20 14/08/14 16:08:31 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=SOGOUQ1 14/08/14 16:08:31 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_table : db=default tbl=SOGOUQ1 14/08/14 16:08:31 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 14/08/14 16:08:31 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 14/08/14 16:08:32 INFO analysis.Analyzer: Max iterations (2) reached for batch Check Analysis 14/08/14 16:08:32 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/08/14 16:08:32 INFO storage.MemoryStore: ensureFreeSpace(391062) called with curMem=0, maxMem=311387750 14/08/14 16:08:32 INFO storage.MemoryStore: Block broadcast_0 stored as values to memory (estimated size 381.9 KB, free 296.6 MB) 14/08/14 16:08:32 INFO sql.SQLContext$anon$1: Max iterations (2) reached for batch Add exchange 14/08/14 16:08:32 INFO sql.SQLContext$anon$1: Max iterations (2) reached for batch Prepare Expressions 14/08/14 16:08:32 INFO spark.SparkContext: Starting job: collect at SparkPlan.scala:52 14/08/14 16:08:33 INFO mapred.FileInputFormat: Total input paths to process : 1 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Registering RDD 6 (mapPartitions at Exchange.scala:69) 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Got job 0 (collect at SparkPlan.scala:52) with 1 output partitions (allowLocal=false) 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Final stage: Stage 0(collect at SparkPlan.scala:52) 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 1) 14/08/14 16:08:33 INFO scheduler.DAGScheduler: Missing parents: List(Stage 1) 14/08/14 16:08:34 INFO scheduler.DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[6] at mapPartitions at Exchange.scala:69), which has no missing parents 14/08/14 16:08:34 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from Stage 1 (MapPartitionsRDD[6] at mapPartitions at Exchange.scala:69) 14/08/14 16:08:34 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 2 tasks 14/08/14 16:08:34 INFO scheduler.TaskSetManager: Starting task 1.0:0 as TID 0 on executor localhost: localhost (PROCESS_LOCAL) 14/08/14 16:08:34 INFO scheduler.TaskSetManager: Serialized task 1.0:0 as 4567 bytes in 12 ms 14/08/14 16:08:34 INFO executor.Executor: Running task ID 0 14/08/14 16:08:34 INFO storage.BlockManager: Found block broadcast_0 locally 14/08/14 16:08:34 INFO rdd.HadoopRDD: Input split: hdfs://master:9000/sougouq1/SogouQ1.txt:0+54375287 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 14/08/14 16:08:34 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id 14/08/14 16:08:36 INFO executor.Executor: Serialized size of result for 0 is 776 14/08/14 16:08:36 INFO executor.Executor: Sending result for 0 directly to driver 14/08/14 16:08:36 INFO executor.Executor: Finished task ID 0 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Starting task 1.0:1 as TID 1 on executor localhost: localhost (PROCESS_LOCAL) 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Serialized task 1.0:1 as 4567 bytes in 2 ms 14/08/14 16:08:36 INFO executor.Executor: Running task ID 1 14/08/14 16:08:36 INFO storage.BlockManager: Found block broadcast_0 locally 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Completed ShuffleMapTask(1, 0) 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Finished TID 0 in 1812 ms on localhost (progress: 1/2) 14/08/14 16:08:36 INFO rdd.HadoopRDD: Input split: hdfs://master:9000/sougouq1/SogouQ1.txt:54375287+54375287 14/08/14 16:08:36 INFO executor.Executor: Serialized size of result for 1 is 776 14/08/14 16:08:36 INFO executor.Executor: Sending result for 1 directly to driver 14/08/14 16:08:36 INFO executor.Executor: Finished task ID 1 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Finished TID 1 in 682 ms on localhost (progress: 2/2) 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Completed ShuffleMapTask(1, 1) 14/08/14 16:08:36 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Stage 1 (mapPartitions at Exchange.scala:69) finished in 2.492 s 14/08/14 16:08:36 INFO scheduler.DAGScheduler: looking for newly runnable stages 14/08/14 16:08:36 INFO scheduler.DAGScheduler: running: Set() 14/08/14 16:08:36 INFO scheduler.DAGScheduler: waiting: Set(Stage 0) 14/08/14 16:08:36 INFO scheduler.DAGScheduler: failed: Set() 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Missing parents for Stage 0: List() 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Submitting Stage 0 (MappedRDD[10] at map at SparkPlan.scala:52), which is now runnable 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 0 (MappedRDD[10] at map at SparkPlan.scala:52) 14/08/14 16:08:36 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 2 on executor localhost: localhost (PROCESS_LOCAL) 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 4766 bytes in 0 ms 14/08/14 16:08:36 INFO executor.Executor: Running task ID 2 14/08/14 16:08:36 INFO storage.BlockManager: Found block broadcast_0 locally 14/08/14 16:08:36 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329 14/08/14 16:08:36 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks 14/08/14 16:08:36 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 10 ms 14/08/14 16:08:36 INFO executor.Executor: Serialized size of result for 2 is 1075 14/08/14 16:08:36 INFO executor.Executor: Sending result for 2 directly to driver 14/08/14 16:08:36 INFO executor.Executor: Finished task ID 2 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Completed ResultTask(0, 0) 14/08/14 16:08:36 INFO scheduler.DAGScheduler: Stage 0 (collect at SparkPlan.scala:52) finished in 0.163 s 14/08/14 16:08:36 INFO scheduler.TaskSetManager: Finished TID 2 in 171 ms on localhost (progress: 1/1) 14/08/14 16:08:36 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 14/08/14 16:08:36 INFO spark.SparkContext: Job finished: collect at SparkPlan.scala:52, took 4.498212701 s [1000000]然后执行成功,
总结:
1.spark的版本必须编译的时候加上了hive 1.0.0预编译版没有加入hive 1.0.1是含有hive的2.需要将hive-site.xml这个配置文件放到spark conf下面
3.需要将mysql驱动加载到classpath中
相关文章推荐
- spark sql 访问hive数据时找不mysql的解决方法
- spark sql 访问hive数据时找不mysql的解决方法
- MySQL 5.7.x 数据库名中文访问出错或表内中文显示?? 解决方法
- spark sql读取hive数据时报找不到mysql驱动的解决办法
- 关于mysql版本升级后,原有的数据无法访问的解决方法,mysql错误代码:1558
- SQOOP中从mysql导入数据到hive中报错解决方法
- jsp存取mysql中文数据结果为乱码的解决方法(未经验证)
- 解决MySQL不允许从远程访问的方法 (收藏)
- 解决MySQL不允许从远程访问的方法
- VS2005+ACCESS WEB程序出错数据访问权限错误的解决方法
- MySql “找不到请求的 .Net Framework 数据提供程序。可能没有安装。”解决方法
- mysql jsp 解决乱码 创建数据库 数据表 方法
- 由于mysql运行权限导致无法找到 Discuz! 论坛数据表! 的解决方法
- MySQL 不允许从远程访问的解决方法
- php向MySQL插入数据不全 解决方法
- 解决MySQL不允许从远程访问的方法
- 解决MySQL不允许从远程访问的方法
- [转]解决MySQL不允许从远程访问的方法
- 解决MySQL不允许从远程访问的方法
- Windows2000 在保存IP安全数据时出现拒绝访问错误80070005的解决方法