hadoop snippet

hadoop

Command

1. Create a directory in HDFS at given path(s).

1
Usage:
1
hdfs dfs -mkdir <paths>
1
Example:
1
hadoop dfs -mkdir /user/saurzcode/dir1 /user/saurzcode/dir2

2. List the contents of a directory.

1
Usage :
1
hdfs dfs -ls <args>
1
Example:
1
hdfs dfs -ls /user/saurzcode

3. Upload and download a file in HDFS.

Upload:

hdfs fs -put:

Copy single src file, or multiple src files from local file system to the hdfs data file system

1
Usage:
1
hdfs dfs -put <localsrc> ... <HDFS_dest_Path>
1
Example:
1
hdfs dfs -put /home/saurzcode/Samplefile.txt  /user/saurzcode/dir3/

*Download:*

hdfs fs -get:

Copies/Downloads files to the local file system

1
Usage:
1
hdfs dfs -get <hdfs_src> <localdst>
1
Example:
1
hdfs dfs -get /user/saurzcode/dir3/Samplefile.txt /home/

4. See contents of a file

Same as unix cat command:

1
Usage:
1
hdfs dfs -cat <path[filename]>
1
Example:
1
hdfs dfs -cat /user/saurzcode/dir1/abc.txt

5. Copy a file from source to destination

This command allows multiple sources as well in which case the destination must be a directory.

1
Usage:
1
hdfs dfs -cp <source> <dest>
1
Example:
1
hdfs dfs -cp /user/saurzcode/dir1/abc.txt /user/saurzcode/dir2

6. Copy a file from/To Local file system to HDFS

copyFromLocal

1
Usage:
1
hdfs dfs -copyFromLocal <localsrc> URI
1
Example:
1
hdfs dfs -copyFromLocal /home/saurzcode/abc.txt  /user/saurzcode/abc.txt

Similar to put command, except that the source is restricted to a local file reference.

copyToLocal

1
Usage:
1
hdfs dfs -copyToLocal [-ignorecrc] [-crc] URI <localdst>

Similar to get command, except that the destination is restricted to a local file reference.

7. Move file from source to destination.

Note:- Moving files across filesystem is not permitted.

1
Usage :
1
hdfs dfs -mv <src> <dest>
1
Example:
1
hdfs dfs -mv /user/saurzcode/dir1/abc.txt /user/saurzcode/dir2

8. Remove a file or directory in HDFS.

Remove files specified as argument. Deletes directory only when it is empty

1
Usage :
1
hdfs dfs -rm <arg>
1
Example:
1
hdfs dfs -rm /user/saurzcode/dir1/abc.txt

*Recursive version of delete.*

1
Usage :
1
hdfs dfs -rmr <arg>
1
Example:
1
hdfs dfs -rmr /user/saurzcode/

9. Display last few lines of a file.

Similar to tail command in Unix.

1
Usage :
1
hdfs dfs -tail <path[filename]>
1
Example:
1
hdfs dfs -tail /user/saurzcode/dir1/abc.txt

10. Display the aggregate length of a file.

1
Usage :
1
hdfs dfs -du <path>
1
2
Example:
hdfs dfs -du /user/saurzcode/dir1/abc.txt

Please comment which of these commands you found most useful while dealing with Hadoop /HDFS.

11. 查看指定目录下的所有文件

1
hadoop fs -lsr /your/hdfs/directory

Solution

Hadoop Operation category READ is not supported in state standby

  • 报错原因
    hadoop集群中的两个NameNode均为standby状态。

    1
    2
    hdfs haadmin -getServiceState nn1
    hdfs haadmin -getServiceState nn2

​ 如果两个NameNode均为standby状态，那么很有可能是DFSZKFailoverController出现了问题。

  • 报错解决

    • 手动改变其中一个NameNode的状态

      1
      hdfs haadmin -transitionToActive -forcemanual nn1

    • 如果手动改变状态不能解决问题，那就重启dfs

      1
      2
      stop-dfs.sh
      start-dfs.sh

  • 再检查NameNode状态

    此时正常状态是nn1为active，nn2为standby