查看Hadoop HDFS 中的一个文件对应block信息
2017-10-05 21:57
573 查看
本文地址:http://blog.csdn.net/chengyuqiang/article/details/78163091
如果需要查看Hadoop HDFS 中的一个文件对应block信息,比如block数、block大小、block所在位置等,可以使用
hdfs fsck input/ncdc.txt -files -blocks -locations
如果需要查看Hadoop HDFS 中的一个文件对应block信息,比如block数、block大小、block所在位置等,可以使用
hdfs fsck命令。
1. HDFS示例文件
hdfs dfs -ls /user/root/input[root@node1 data]# hdfs dfs -ls /user/root/input Found 7 items -rw-r--r-- 3 root supergroup 281498 2017-09-20 10:11 /user/root/input/Hamlet.txt -rw-r--r-- 3 root supergroup 9789248 2017-09-22 10:26 /user/root/input/age.txt -rw-r--r-- 3 root supergroup 71 2017-08-27 09:18 /user/root/input/books.txt -rw-r--r-- 3 root supergroup 264075431 2017-10-05 09:37 /user/root/input/cite75_99.txt drwxr-xr-x - root supergroup 0 2017-08-13 09:33 /user/root/input/emp.bak drwxr-xr-x - root supergroup 0 2017-09-24 04:08 /user/root/input/ml-1m -rw-r--r-- 3 root supergroup 871353053 2017-10-05 09:40 /user/root/input/ncdc.txt [root@node1 data]#
2. hdfs fsck用法
[root@node1 data]# hdfs fsck Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]] [-includeSnapshots] [-storagepolicies] [-blockId <blk_Id>] <path> start checking from this path -move move corrupted files to /lost+found -delete delete corrupted files -files print out files being checked -openforwrite print out files opened for write -includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it -list-corruptfileblocks print out list of missing blocks and files they belong to -blocks print out block report -locations print out locations for every block -racks print out network topology for data-node locations -storagepolicies print out storage policy summary for the blocks -blockId print out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc) Please Note: 1. By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually tagged CORRUPT or HEALTHY depending on their block allocation status 2. Option -includeSnapshots should not be used for comparing stats, should be used only for HEALTH check, as this may contain duplicates if the same file present in both original fs tree and inside snapshots. Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] [root@node1 data]#
3. 查看block基本信息
hdfs fsck input/cite75_99.txt[root@node1 data]# hdfs fsck input/cite75_99.txt Connecting to namenode via http://node1:50070/fsck?ugi=root&path=%2Fuser%2Froot%2Finput%2Fcite75_99.txt FSCK started by root (auth:SIMPLE) from /192.168.80.131 for path /user/root/input/cite75_99.txt at Thu Oct 05 09:41:58 EDT 2017 .Status: HEALTHY Total size: 264075431 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 2 (avg. block size 132037715 B) Minimally replicated blocks: 2 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 3 Number of racks: 1 FSCK ended at Thu Oct 05 09:41:58 EDT 2017 in 3 milliseconds The filesystem under path '/user/root/input/cite75_99.txt' is HEALTHY [root@node1 data]# hdfs fsck input/ncdc.txt Connecting to namenode via http://node1:50070/fsck?ugi=root&path=%2Fuser%2Froot%2Finput%2Fncdc.txt FSCK started by root (auth:SIMPLE) from /192.168.80.131 for path /user/root/input/ncdc.txt at Thu Oct 05 09:42:22 EDT 2017 .Status: HEALTHY Total size: 871353053 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 7 (avg. block size 124479007 B) Minimally replicated blocks: 7 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 3 Number of racks: 1 FSCK ended at Thu Oct 05 09:42:22 EDT 2017 in 2 milliseconds The filesystem under path '/user/root/input/ncdc.txt' is HEALTHY [root@node1 data]#
4. -files -blocks -locations选项
hdfs fsck input/ncdc.txt -files -blockshdfs fsck input/ncdc.txt -files -blocks -locations
[root@node1 data]# hdfs fsck input/ncdc.txt -files -blocks Connecting to namenode via http://node1:50070/fsck?ugi=root&files=1&blocks=1&path=%2Fuser%2Froot%2Finput%2Fncdc.txt FSCK started by root (auth:SIMPLE) from /192.168.80.131 for path /user/root/input/ncdc.txt at Thu Oct 05 09:47:14 EDT 2017 /user/root/input/ncdc.txt 871353053 bytes, 7 block(s): OK 0. BP-766589174-192.168.80.131-1500731607717:blk_1073742821_2026 len=134217728 repl=3 1. BP-766589174-192.168.80.131-1500731607717:blk_1073742822_2027 len=134217728 repl=3 2. BP-766589174-192.168.80.131-1500731607717:blk_1073742823_2028 len=134217728 repl=3 3. BP-766589174-192.168.80.131-1500731607717:blk_1073742824_2029 len=134217728 repl=3 4. BP-766589174-192.168.80.131-1500731607717:blk_1073742825_2030 len=134217728 repl=3 5. BP-766589174-192.168.80.131-1500731607717:blk_1073742826_2031 len=134217728 repl=3 6. BP-766589174-192.168.80.131-1500731607717:blk_1073742827_2032 len=66046685 repl=3 Status: HEALTHY Total size: 871353053 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 7 (avg. block size 124479007 B) Minimally replicated blocks: 7 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 3 Number of racks: 1 FSCK ended at Thu Oct 05 09:47:14 EDT 2017 in 2 milliseconds The filesystem under path '/user/root/input/ncdc.txt' is HEALTHY [root@node1 data]# hdfs fsck input/ncdc.txt -files -blocks -locations Connecting to namenode via http://node1:50070/fsck?ugi=root&files=1&blocks=1&locations=1&path=%2Fuser%2Froot%2Finput%2Fncdc.txt FSCK started by root (auth:SIMPLE) from /192.168.80.131 for path /user/root/input/ncdc.txt at Thu Oct 05 09:47:45 EDT 2017 /user/root/input/ncdc.txt 871353053 bytes, 7 block(s): OK 0. BP-766589174-192.168.80.131-1500731607717:blk_1073742821_2026 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK]] 1. BP-766589174-192.168.80.131-1500731607717:blk_1073742822_2027 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK]] 2. BP-766589174-192.168.80.131-1500731607717:blk_1073742823_2028 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK], DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK]] 3. BP-766589174-192.168.80.131-1500731607717:blk_1073742824_2029 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK]] 4. BP-766589174-192.168.80.131-1500731607717:blk_1073742825_2030 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK]] 5. BP-766589174-192.168.80.131-1500731607717:blk_1073742826_2031 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK]] 6. BP-766589174-192.168.80.131-1500731607717:blk_1073742827_2032 len=66046685 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK]] Status: HEALTHY Total size: 871353053 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 7 (avg. block size 124479007 B) Minimally replicated blocks: 7 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 3 Number of racks: 1 FSCK ended at Thu Oct 05 09:47:45 EDT 2017 in 2 milliseconds The filesystem under path '/user/root/input/ncdc.txt' is HEALTHY [root@node1 data]#
相关文章推荐
- hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息
- hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息
- hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息
- HDFS设计思路,HDFS使用,查看集群状态,HDFS,HDFS上传文件,HDFS下载文件,yarn web管理界面信息查看,运行一个mapreduce程序,mapreduce的demo
- Hadoop中查看一个文件位置信息
- Hadoop 用FileStatus类来查看HDFS中文件或目录的元信息
- HDFS设计思路,HDFS使用,查看集群状态,HDFS,HDFS上传文件,HDFS下载文件,yarn web管理界面信息查看,运行一个mapreduce程序,mapreduce的demo
- Hadoop 用FileStatus类来查看HDFS中文件或目录的元信息
- HDFS文件系统如何查看文件对应的block
- hadoop入门例子(一)-输出hdfs上面的某一个目录下面的文件
- hadoop学习;大数据集在HDFS中存为单个文件;安装linux下eclipse出错解决;查看.class文件插件
- Git如何查看一个文件的修改历史信息
- 查看HADOOP中一个文件有多少块组成及所在机器ip
- Hadoop(八)Java程序访问HDFS集群中数据块与查看文件系统
- 转载: 查看HADOOP中一个文件有多少块组成及所在机器ip
- 分享一个查看Hdfs目录下文件大小的脚本,支持排序
- stat 查看一个文件的修改时间,创建时间等信息的命令
- 查看HADOOP中一个文件有多少块组成
- (转)查看栈信息中 .so文件里地址对应的方法名称
- Hadoop编程学习(四):使用FileSystem类进行文件读写及查看文件信息