hadoop hdfs命令汇总

1.job相关的命令

hadoop job <command> <args>

Usage: CLI <command> <args>

[-submit <job-file>]

[-status <job-id>]

[-counter <job-id> <group-name> <counter-name>]

[-kill <job-id>]

[-set-priority <job-id> <priority>]. Valid values for priorities are: VERY_HIGH HIGH NORMAL LOW VERY_LOW

[-events <job-id> <from-event-#> <#-of-events>]

[-history <jobHistoryFile>]

[-list [all]]

[-list-active-trackers]

[-list-blacklisted-trackers]

[-list-attempt-ids <job-id> <task-type> <task-state>]. Valid values for <task-type> are MAP REDUCE. Valid values for <task-state> are running, completed

[-kill-task <task-attempt-id>]

[-fail-task <task-attempt-id>]

[-logs <job-id> <task-attempt-id>]

hadoop version 查看版本

2.文件系统及文件检查相关的命令

hadoop fs -help 所有的命令及用法

Usage: hadoop fs [generic options]

[-appendToFile <localsrc> ... <dst>]

[-cat [-ignoreCrc] <src> ...]

[-checksum <src> ...]

[-chgrp [-R] GROUP PATH...] 例如：hadoop fs -chgrp -R supergroup /user 其中supergroup为组名，-R表示递归

[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]

[-chown [-R] [OWNER][:[GROUP]] PATH...]

[-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]

[-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]

[-count [-q] [-h] <path> ...]

[-cp [-f] [-p | -p[topax]] <src> ... <dst>]

[-createSnapshot <snapshotDir> [<snapshotName>]]

[-deleteSnapshot <snapshotDir> <snapshotName>]

[-df [-h] [<path> ...]]

[-du [-s] [-h] <path> ...]

[-expunge]

[-find <path> ... <expression> ...]

[-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]

[-getfacl [-R] <path>]

[-getfattr [-R] {-n name | -d} [-e en] <path>]

[-getmerge [-nl] <src> <localdst>]

[-help [cmd ...]]

[-ls [-d] [-h] [-R] [<path> ...]]

[-mkdir [-p] <path> ...]

[-moveFromLocal <localsrc> ... <dst>]

[-moveToLocal <src> <localdst>]

[-mv <src> ... <dst>]

[-put [-f] [-p] [-l] <localsrc> ... <dst>]

[-renameSnapshot <snapshotDir> <oldName> <newName>]

[-rm [-f] [-r|-R] [-skipTrash] <src> ...]

[-rmdir [--ignore-fail-on-non-empty] <dir> ...]

[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]

[-setfattr {-n name [-v value] | -x name} <path>]

[-setrep [-R] [-w] <rep> <path> ...]

[-stat [format] <path> ...]

[-tail [-f] <file>]

[-test -[defsz] <path>]

[-text [-ignoreCrc] <src> ...]

[-touchz <path> ...]

[-truncate [-w] <length> <path> ...]

[-usage [cmd ...]]

hadoop fs -help <command>,查看某个命令的帮助文档，如hadoop fs -help mkdir

hadoop fs -du -s /user 统计/user目录总大小，单位为字节

hadoop fs -du /user 统计/user目录下的子目录大小，单位为字节

hadoop fs -du -h /user 统计/user目录下的子目录大小，单位为M

hadoop fs -dus -h /user <==>hadoop fs -du -s -h /user 统计/user目录总大小，单位为M

Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]

<path> start checking from this path 检查指定路径的文件完整性

-move move corrupted files to /lost+found 移除损坏文件至/lost+found

-delete delete corrupted files

-files print out files being checked

-openforwrite print out files opened for write

-includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it

-list-corruptfileblocks print out list of missing blocks and files they belong to

-blocks print out block report

-locations print out locations for every block

-racks print out network topology for data-node locations

-storagepolicies print out storage policy summary for the blocks

-blockId print out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc)

用法示例：hadoop fsck /user/a.txt -files -blocks -racks

3.管理命令

hdfs dfsadmin

[-report [-live] [-dead] [-decommissioning]] 文件系统信息

[-safemode <enter | leave | get | wait>] 安全模式相关操作

[-saveNamespace]

[-rollEdits]

[-restoreFailedStorage true|false|check]

[-refreshNodes]

[-setQuota <quota> <dirname>...<dirname>]

[-clrQuota <dirname>...<dirname>]

[-setSpaceQuota <quota> [-storageType <storagetype>] <dirname>...<dirname>]

[-clrSpaceQuota [-storageType <storagetype>] <dirname>...<dirname>]

[-finalizeUpgrade]

[-rollingUpgrade [<query|prepare|finalize>]]

[-refreshServiceAcl]

[-refreshUserToGroupsMappings]

[-refreshSuperUserGroupsConfiguration]

[-refreshCallQueue]

[-refresh <host:ipc_port> <key> [arg1..argn]

[-reconfig <datanode|...> <host:ipc_port> <start|status>]

[-printTopology]

[-refreshNamenodes datanode_host:ipc_port]

[-deleteBlockPool datanode_host:ipc_port blockpoolId [force]]

[-setBalancerBandwidth <bandwidth in bytes per second>]

[-fetchImage <local directory>]

[-allowSnapshot <snapshotDir>]

[-disallowSnapshot <snapshotDir>]

[-shutdownDatanode <datanode_host:ipc_port> [upgrade]]

[-getDatanodeInfo <datanode_host:ipc_port>]

[-metasave filename]

[-triggerBlockReport [-incremental] <datanode_host:ipc_port>]

[-help [cmd]]

具体用法参见英文说明。