impala大数据量查询/tmp/impala-scratch创建异常
2017-02-10 17:37
169 查看
使用impala对大数据量查询distinct的时候出现如下错误
查阅文档发现impala在大数据量处理时会用到磁盘保存中间数据
By default, intermediate files used during large sort, join, aggregation, or analytic function operations are stored in the directory /tmp/impala-scratch.
These files are removed when the operation finishes. (Multiple concurrent queries can perform operations that use the "spill
to disk" technique, without any name conflicts for these temporary files.) You can specify a different location by starting the impalad daemon
with the --scratch_dirs="path_to_directory" configuration
option or the equivalent configuration option in the Cloudera Manager user interface. You can specify a single directory, or a comma-separated list of directories. The scratch directories must be on the local filesystem, not in HDFS. You might specify different
directory paths for different hosts, depending on the capacity and speed of the available storage devices. Impala will not start if it cannot create or read and write files in the "scratch" directory.
If there is less than 1 GB free on the filesystem where that directory resides, Impala still runs, but writes a warning message to its log.
直接到各个impalad节点上创建/tmp/impala-scratch中间目录,并赋予读写权限:
5ab149d_24414dab2c19caca:e54b206c5ab149f_91001337-9d70-4c93-84ce-e7916c1ae804 failed with errno=2 description=Error(2): No such file or directory Backend 4:Create file /tmp/impala-scratch/24414dab2c19caca:e54b206c5ab149d_24414dab2c19caca:e54b206c5ab149f_91001337-9d70-4c93-84ce-e7916c1ae804 failed with errno=2 description=Error(2): No such file or directory
查阅文档发现impala在大数据量处理时会用到磁盘保存中间数据
By default, intermediate files used during large sort, join, aggregation, or analytic function operations are stored in the directory /tmp/impala-scratch.
These files are removed when the operation finishes. (Multiple concurrent queries can perform operations that use the "spill
to disk" technique, without any name conflicts for these temporary files.) You can specify a different location by starting the impalad daemon
with the --scratch_dirs="path_to_directory" configuration
option or the equivalent configuration option in the Cloudera Manager user interface. You can specify a single directory, or a comma-separated list of directories. The scratch directories must be on the local filesystem, not in HDFS. You might specify different
directory paths for different hosts, depending on the capacity and speed of the available storage devices. Impala will not start if it cannot create or read and write files in the "scratch" directory.
If there is less than 1 GB free on the filesystem where that directory resides, Impala still runs, but writes a warning message to its log.
直接到各个impalad节点上创建/tmp/impala-scratch中间目录,并赋予读写权限:
mkdir /tmp/impala-scratch chmod 777 /tmp/impala-scratch
相关文章推荐
- IO之DataInputStream
- 大数据时代必不可少的大数据分析和制作工具大全
- 微信技术总监讲大数据高并发系统架构
- ABAP MESSAGE xxxx RAISING xxxx解析
- 寒假学习之stm32(15)----DMA(direct memory access)
- RocketMq Exception "connect to <:10909> failed"问题
- Http Cookie 测试 Domain & Path
- 关于大数据测试方面的收集
- pcntl_fock & pcntl_waitpid 初学
- 【数据分析知识点】detailed table of contents
- 指针数组 和 数组指针分析 main函数的参数
- 桥基站设备通信协议-AIS
- 阐述游戏AI设计的两个禁忌及解决方法
- AIS中STM32程序,Sprintf函数,函数字符串形参
- canvas和paint
- 2017.2.10【初中部 GDKOI】模拟赛B组 粉刷(paint) 题解
- KAFKA 知识点
- 机智云、中国电信、韦德电子结盟助推物联网产业创新转型
- Cassandra 入门介绍
- Your project path contains non-ASCII characters