HDFS常用命令整理

簡介：

• HDFS 文件系統提供了相當多的shell 操作命令，大大方便了程序員和系統管理人員查看、修改HDFS 上的文件。進一步，HDFS 的操作命令和Unix/Linux 的命令名稱和格式相當一致，因而學習HDFS 命令的成本也大為縮小。

1.命令幫助

[hadoop@hadoop000 ~]$ hdfs dfs
Usage: hadoop fs [generic options]
        [-appendToFile <localsrc> ... <dst>]
        [-cat [-ignoreCrc] <src> ...]
        [-checksum <src> ...]
        [-chgrp [-R] GROUP PATH...]
        [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-copyFromLocal [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
        [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] <path> ...]
        [-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>]
        [-createSnapshot <snapshotDir> [<snapshotName>]]
        [-deleteSnapshot <snapshotDir> <snapshotName>]
        [-df [-h] [<path> ...]]
        [-du [-s] [-h] [-x] <path> ...]
        [-expunge]
        [-find <path> ... <expression> ...]
        [-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-getfacl [-R] <path>]
        [-getfattr [-R] {-n name | -d} [-e en] <path>]
        [-getmerge [-nl] [-skip-empty-file] <src> <localdst>]
        [-help [cmd ...]]
        [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]]
        [-mkdir [-p] <path> ...]
        [-moveFromLocal <localsrc> ... <dst>]
        [-moveToLocal <src> <localdst>]
        [-mv <src> ... <dst>]
        [-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
        [-renameSnapshot <snapshotDir> <oldName> <newName>]
        [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
        [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
        [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
        [-setfattr {-n name [-v value] | -x name} <path>]
        [-setrep [-R] [-w] <rep> <path> ...]
        [-stat [format] <path> ...]
        [-tail [-f] <file>]
        [-test -[defsz] <path>]
        [-text [-ignoreCrc] <src> ...]
        [-touchz <path> ...]
        [-truncate [-w] <length> <path> ...]
        [-usage [cmd ...]]
# 注：hdfs dfs等同于hadoop fs

2.文件操作命令案例

hdfs dfs -copyFromLocal /local/data /hdfs/data：將本地文件上傳到 hdfs 上（原路徑只能是一個文件）
hdfs dfs -put /tmp/ /hdfs/ ：和 copyFromLocal 區別是，put 原路徑可以是文件夾等
hadoop fs -ls / ：查看根目錄文件
hadoop fs -ls /tmp/data：查看/tmp/data目錄
hadoop fs -cat /tmp/a.txt ：查看 a.txt，與 -text 一樣
hadoop fs -mkdir dir：創建目錄dir
hadoop fs -get /aaa/jdk.tar.gz :等同于copyToLocal，就是從hdfs下載文件到本地
hadoop fs -df -h / :統計文件系統的可用空間信息
hadoop fs -cp /aaa/jdk.tar.gz /bbb/jdk.tar.gz.2 :從hdfs的一個路徑拷貝hdfs的另一個路徑
hadoop fs -mv /aaa/jdk.tar.gz / :在hdfs目錄中移動文件
hadoop fs -du -s -h /aaa/* :統計文件夾的大小信息
hadoop fs -rm -r /aaa/bbb/ :刪除文件或文件夾
hadoop fs -rmdir /aaa/bbb/ccc :刪除目錄
hadoop fs -tail /weblog/access_log.1 :顯示一個文件的末尾

3.管理命令

# 命令幫助
[hadoop@hadoop000 ~]$ hdfs dfsadmin
Usage: hdfs dfsadmin
Note: Administrative commands can only be run as the HDFS superuser.
        [-report [-live] [-dead] [-decommissioning]]
        [-safemode <enter | leave | get | wait>]
        [-saveNamespace]
        [-rollEdits]
        [-restoreFailedStorage true|false|check]
        [-refreshNodes]
        [-setQuota <quota> <dirname>...<dirname>]
        [-clrQuota <dirname>...<dirname>]
        [-setSpaceQuota <quota> [-storageType <storagetype>] <dirname>...<dirname>]
        [-clrSpaceQuota [-storageType <storagetype>] <dirname>...<dirname>]
        [-finalizeUpgrade]
        [-rollingUpgrade [<query|prepare|finalize>]]
        [-refreshServiceAcl]
        [-refreshUserToGroupsMappings]
        [-refreshSuperUserGroupsConfiguration]
        [-refreshCallQueue]
        [-refresh <host:ipc_port> <key> [arg1..argn]
        [-reconfig <datanode|...> <host:ipc_port> <start|status|properties>]
        [-printTopology]
        [-refreshNamenodes datanode_host:ipc_port]
        [-deleteBlockPool datanode_host:ipc_port blockpoolId [force]]
        [-setBalancerBandwidth <bandwidth in bytes per second>]
        [-getBalancerBandwidth <datanode_host:ipc_port>]
        [-fetchImage <local directory>]
        [-allowSnapshot <snapshotDir>]
        [-disallowSnapshot <snapshotDir>]
        [-shutdownDatanode <datanode_host:ipc_port> [upgrade]]
        [-evictWriters <datanode_host:ipc_port>]
        [-getDatanodeInfo <datanode_host:ipc_port>]
        [-metasave filename]
        [-triggerBlockReport [-incremental] <datanode_host:ipc_port>]
        [-help [cmd]]

hdfs dfsadmin -report :顯示所有dataNode
hdfs dfsadmin -safemode leave :離開安全模式