启动 ./spark-shell 命令报错
2017-07-04 00:37
543 查看
当使用./spark-shell 命令报错
抛出了很多错误,甚至我退出spark-shell,再次进入抛出的错误还不一样,最后一个错误是Name node is in safe mode. 先解决这个
网上说这是hdfs的分布式文件系统的安全模式,当安全模式时文件系统中的内容不允许修改和删除,直到安全模式结束,安全模式就是系统在检查各个dataNode数据块的有效性
用户可以通过dfsadmin -safemode value 来操作安全模式,参数value的说明如下:
//enter - 进入安全模式
//leave - 强制NameNode离开安全模式
//get - 返回安全模式是否开启的信息
//wait - 等待,一直到安全模式结束。
当离开安全模式再次spark-shell,抛出异常
可以看到,这个异常是说在/usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/ 中已经有另一个数据库(Derby)实例了,我一直都是使用mysql数据库,没用过Derby数据库,最后想想可能是我在安装spark的时候,连接了hive,那个时候我的hive还
配置mysql数据库,hive默认使用Derby数据库,所以当我启动spark连接hive的时候,就自动生成了Derby的实例。我看了'metastore_db' 文件夹的内容是Derby数据库的配置
解决方法:
删除/usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/ metastore_db 文件夹。
重新启动
./spark-shell
Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@476fde05, see the next exception for details. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 153 more Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/metastore_db. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source) -------------------- Caused by: java.sql.SQLException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------ java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@476fde05, see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) ------------------- Caused by: org.apache.hadoop.ipc.RemoteException: Cannot create directory /tmp/hive/root/5436b1aa-85e3-4512-b505-b0bdc7444e46. Name node is in safe mode. The reported blocks 0 needs additional 9 blocks to reach the threshold 0.9990 of total blocks 9. The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3895) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenode
抛出了很多错误,甚至我退出spark-shell,再次进入抛出的错误还不一样,最后一个错误是Name node is in safe mode. 先解决这个
网上说这是hdfs的分布式文件系统的安全模式,当安全模式时文件系统中的内容不允许修改和删除,直到安全模式结束,安全模式就是系统在检查各个dataNode数据块的有效性
bin/hadoop dfsadmin -safemode leave //离开安全模式
用户可以通过dfsadmin -safemode value 来操作安全模式,参数value的说明如下:
//enter - 进入安全模式
//leave - 强制NameNode离开安全模式
[root@node1 sbin]# hdfs dfsadmin -safemode leave Safe mode is OFF
//get - 返回安全模式是否开启的信息
[root@node1 sbin]# hdfs dfsadmin -safemode get Safe mode is ON
//wait - 等待,一直到安全模式结束。
当离开安全模式再次spark-shell,抛出异常
Caused by: org.apache.derby.iapi.error.StandardException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@476fde05, see the next exception for details. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 153 more Caused by: org.apache.derby.iapi.error.StandardException: Another instance of Derby may have already booted the database /usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/metastore_db. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
可以看到,这个异常是说在/usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/ 中已经有另一个数据库(Derby)实例了,我一直都是使用mysql数据库,没用过Derby数据库,最后想想可能是我在安装spark的时候,连接了hive,那个时候我的hive还
配置mysql数据库,hive默认使用Derby数据库,所以当我启动spark连接hive的时候,就自动生成了Derby的实例。我看了'metastore_db' 文件夹的内容是Derby数据库的配置
解决方法:
删除/usr/local/development/spark-2.1.1-bin-hadoop2.7/bin/ metastore_db 文件夹。
重新启动
./spark-shell
[root@node1 bin]# ./spark-shell Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/07/05 00:13:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/07/05 00:13:29 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 17/07/05 00:13:29 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException 17/07/05 00:13:40 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException Spark context Web UI available at http://192.168.177.120:4040 Spark context available as 'sc' (master = local[*], app id = local-1499184787668). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.1.1 /_/ Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131) Type in expressions to have them evaluated. Type :help for more information. scala> scala> sc res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@178c4480
相关文章推荐
- spark-shell on yarn 出错解决【启动命令bin/spark-shell --master yarn-client出现错误,类ExecutorLauncher 找不到】
- Shell脚本与vi编辑器:vi启动与退出、工作模式、命令大全
- 第20讲 常用命令补讲 shell补讲 linux启动过程分析
- 本地启动spark-shell
- Spark1.51 本地启动了Master和一个Slave后,在Spark-Shell里跑,看出sc.textFile找数据的stacktrace
- mongodb操作命令------shell脚本启动-插入
- 【Android】通过adb shell命令查看内存,CPU,启动时间,电量等信息
- spark:spark启动、spark-shell启动及测试--7
- 一句话Shell命令关闭不需要的随机启动服务
- 启动spark-shell
- adb启动程序命令:adb shell am start
- Shell脚本与vi编辑器:vi启动与退出、工作模式、命令大全
- linux ulimit命令参数及用法详解--linux限定shell启动进程资源
- 启动python解释器的命令(python manage.py shell和python的区别)
- spark定制之一:shell启动
- 在linux如何在程序中调用shell命令启动另一个程序
- linux ulimit命令参数及用法详解--linux限定shell启动进程资源
- linux ulimit命令参数及用法详解--linux限定shell启动进程资源
- adb shell am 命令启动不知类名情况下的 main activity
- execlp启动android进程命令窗口通过adb shell 进入android 的Linux命令界面,输入am -help看到如下信息: 我们可以通过命令启动android中的