Check Hadoop space usage

/root>sudo -u hdfs hdfs dfs -df -h
Filesystem                  Size       Used   Available Use%
hdfs://xyz.com:8020 957.6 G 74.4 G 770.2 G     8%

/root>sudo -u hdfs hdfs dfs -du -h /
0 0                    /system
4.2 G 12.7 G    /tmp
20.4 G 62.6 G /user

To set the disk space quota on a directory:

/root>sudo -u hdfs hdfs dfsadmin -setSpaceQuota 150g /user/dirname

To check the disk space quota:
/root>sudo -u hdfs hdfs dfs -count -q /user/dirname

/root> sudo -u hdfs hdfs dfs -count -q /user/dirname | awk ‘{print $8 ” Quota is ” $3/1024/1024 “Mb and Real size is ” $7/1024/1024 “Mb”}’

/user/dirname Quota is 153600Mb and Real size is 45255.9Mb

/root>sudo -u hdfs hdfs dfsadmin -report

Configured Capacity: 1028213293056 (957.60 GB)
Present Capacity: 906962639594 (844.67 GB)
DFS Remaining: 827034483501 (770.24 GB)
DFS Used: 79928156093 (74.44 GB)
DFS Used%: 8.81%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

————————————————-
Live datanodes (4):

Name:
Hostname:
Rack: /default
Decommission Status : Normal
Configured Capacity: 257053323264 (239.40 GB)
DFS Used: 21261950976 (19.80 GB)
Non DFS Used: 17912508416 (16.68 GB)
DFS Remaining: 214761225601 (200.01 GB)
DFS Used%: 8.27%
DFS Remaining%: 83.55%
Configured Cache Capacity: 3613392896 (3.37 GB)
Cache Used: 0 (0 B)
Cache Remaining: 3613392896 (3.37 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 8
Last contact: Wed Nov 08 16:40:09 EST 2017

Name:
Hostname:
Rack: /default
Decommission Status : Normal
Configured Capacity: 257053323264 (239.40 GB)
DFS Used: 21306994688 (19.84 GB)
Non DFS Used: 64821764096 (60.37 GB)
DFS Remaining: 167673145615 (156.16 GB)
DFS Used%: 8.29%
DFS Remaining%: 65.23%
Configured Cache Capacity: 1600126976 (1.49 GB)
Cache Used: 0 (0 B)
Cache Remaining: 1600126976 (1.49 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 11
Last contact: Wed Nov 08 16:40:08 EST 2017

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.