HDFS How to List Files

You can list files in HDFS with either of the following commands:

hdfs dfs -ls hdfs://my-host1:500
hadoop fs -ls hdfs://my-host1:500

This is a very common Hadoop/HDFS command. It is very similar to the common Unix/Linux “ls” command. It is a bit different but should be pretty easy to use no matter what you are familiar with.

Here is the command usage

Usage: hadoop fs -ls [-d] [-h] [-R] [-t] [-S] [-r] [-u] <args>
Options:
-d: Directories are listed as plain files.
-h: Format file sizes in a human-readable fashion (eg 64.0m instead of 67108864).
-R: Recursively list subdirectories encountered.
-t: Sort output by modification time (most recent first).
-S: Sort output by file size.
-r: Reverse the sort order.
-u: Use access time rather than modification time for display and sorting.

If you wanted to list all files recursively and sorted by timestamp, you would use the following:

hdfs dfs -ls hdfs://my-host1:500

You can sort files by output size like this:

hdfs dfs -ls -S hdfs://my-host1:500

This is useful if you are searching for files that are taking up a large amount of space.

The “-h” option is really handy option. It produces user readable sizes. For example, file sizes will be displayed in M or G rather than just the total number of bytes.

hadoop fs -ls -h hdfs://my-host1:500