impala大数据量查询/tmp/impala-scratch创建异常
2015-08-17 10:28
671 查看
使用impala做大数据量查询distinct的时候报如下错误
By default, intermediate files used during large sort, join, aggregation, or analytic function operations are stored in the directory /tmp/impala-scratch. These files are removed when the operation finishes. (Multiple concurrent queries can perform operations that use the "spill to disk" technique, without any name conflicts for these temporary files.) You can specify a different location by starting the impalad daemon with the --scratch_dirs="path_to_directory" configuration option or the equivalent configuration option in the Cloudera Manager user interface. You can specify a single directory, or a comma-separated list of directories. The scratch directories must be on the local filesystem, not in HDFS. You might specify different directory paths for different hosts, depending on the capacity and speed of the available storage devices. Impala will not start if it cannot create or read and write files in the "scratch" directory. If there is less than 1 GB free on the filesystem where that directory resides, Impala still runs, but writes a warning message to its log.
直接到各个impalad节点上创建/tmp/impala-scratch目录,并赋予读写权限
WARNINGS: Create file /tmp/impala-scratch/24414dab2c19caca:e54b206c5ab149d_24414dab2c19caca:e54b206c5ab149f_91001337-9d70-4c93-84ce-e7916c1ae804 failed with errno=2 description=Error(2): No such file or directory Backend 4:Create file /tmp/impala-scratch/24414dab2c19caca:e54b206c5ab149d_24414dab2c19caca:e54b206c5ab149f_91001337-9d70-4c93-84ce-e7916c1ae804 failed with errno=2 description=Error(2): No such file or directory查阅文档知道impala在大数据量处理时会用到磁盘保存数据
By default, intermediate files used during large sort, join, aggregation, or analytic function operations are stored in the directory /tmp/impala-scratch. These files are removed when the operation finishes. (Multiple concurrent queries can perform operations that use the "spill to disk" technique, without any name conflicts for these temporary files.) You can specify a different location by starting the impalad daemon with the --scratch_dirs="path_to_directory" configuration option or the equivalent configuration option in the Cloudera Manager user interface. You can specify a single directory, or a comma-separated list of directories. The scratch directories must be on the local filesystem, not in HDFS. You might specify different directory paths for different hosts, depending on the capacity and speed of the available storage devices. Impala will not start if it cannot create or read and write files in the "scratch" directory. If there is less than 1 GB free on the filesystem where that directory resides, Impala still runs, but writes a warning message to its log.
直接到各个impalad节点上创建/tmp/impala-scratch目录,并赋予读写权限
mkdir /tmp/impala-scratch chmod 777 /tmp/impala-scratch
相关文章推荐
- Impala安装杂记
- 今天开始使用impala
- Impala 安装笔记1一Cloudera CDH4.3.0安装
- 安装和使用impala出现的问题及解决方法
- Impala简明调优手册
- Big Data's New Use Cases: Transformation, Active Archive, and Exploration
- Cloudera Impala: Real-Time Queries in Apache Hadoop, For Real
- Impala原理及其调优
- Impala1.2.4安装手册
- Cloudera Manager 用parcel安装impala
- Impala 1.4.0 新特性
- Impala学习--Impala概述
- Impala学习--Impala系统架构
- Impala学习--Impala前端代码分析
- Impala学习--代码生成(Code Generation)
- cloudera impala 交集、差集实际测试数据
- 使用cloudera impala sql 进行交集、并集、差集查询
- [Hadoop] Cloudera Impala:基于Hadoop的实时查询开源项目
- hive (2) partion 学习 相关文章
- 安装implala报错找不到jdbc