HDFS balancer怎么用

小編給大家分享一下HDFS balancer怎么用，相信大部分人都還不怎么了解，因此分享這篇文章給大家參考一下，希望大家閱讀完這篇文章后大有收獲，下面讓我們一起去了解一下吧！

HDFS balancer

在hadoop的hdfs集群中新添加節點后，在新節點上運行./start-balancer.sh。

運行的日志信息如下，重復判斷、遷移，速度相當慢。

為了降低集群負荷、避免干擾其他用戶，均衡器被設計為在后臺運行。在不同節點之間復制數據的帶寬也是受限的，默認值是1MB/s。

2 over-utilized: [Source[192.168.70.94:50010, utilization=61.784966362172845], Source[192.168.70.92:50010, utilization=64.37189641437116]]

3 underutilized: [BalancerDatanode[192.168.60.153:50010, utilization=37.48304150996222], BalancerDatanode[192.168.60.151:50010, utilization=38.02279713694489], BalancerDatanode[192.168.60.154:50010, utilization=8.601460127855951]]

Need to move 1.71 TB to make the cluster balanced.

Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.153:50010

Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.151:50010

Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.154:50010

Will move 30 GB in this iteration

2015-10-13 09:21:29,585 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: namenodes = [hdfs://hmaster91/]
2015-10-13 09:21:29,598 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
2015-10-13 09:21:30,350 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-10-13 09:21:32,371 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.93:50010
2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.153:50010
2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.151:50010
2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.94:50010
2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.152:50010
2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.92:50010
2015-10-13 09:21:32,374 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.154:50010
2015-10-13 09:21:32,375 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 over-utilized: [Source[192.168.70.94:50010, utilization=61.784966362172845], Source[192.168.70.92:50010, utilization=64.37189641437116]]
2015-10-13 09:21:32,375 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 underutilized: [BalancerDatanode[192.168.60.153:50010, utilization=37.48304150996222], BalancerDatanode[192.168.60.151:50010, utilization=38.02279713694489], BalancerDatanode[192.168.60.154:50010, utilization=8.601460127855951]]
2015-10-13 09:21:32,380 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 1.71 TB to make the cluster balanced.
2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.153:50010
2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.151:50010
2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.154:50010
2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GB in this iteration
2015-10-13 09:31:41,945 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515941_8775312 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:50010
2015-10-13 09:31:47,378 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610315_8869686 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.153:50010
2015-10-13 09:31:59,425 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082639980_8899357 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.94:50010
2015-10-13 09:32:02,929 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515919_8775290 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:50010
2015-10-13 09:32:03,938 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082510026_8769397 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:50010
2015-10-13 09:32:05,370 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1078753887_5013250 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.70.92:50010
2015-10-13 09:32:06,871 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610299_8869670 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.153:50010
2015-10-13 09:32:10,923 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610274_8869645 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.70.94:50010
2015-10-13 09:32:12,930 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515963_8775334 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:50010
2015-10-13 09:32:13,424 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082509872_8769243 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:50010
2015-10-13 09:32:13,429 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610182_8869553 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.151:50010
2015-10-13 09:32:17,423 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082509962_8769333 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:50010
2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.151:50010
2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.94:50010
2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.154:50010
2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.93:50010
2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.92:50010
2015-10-13 09:32:43,570 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.153:50010
2015-10-13 09:32:43,570 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.152:50010
2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 over-utilized: [Source[192.168.70.94:50010, utilization=61.598206824722595], Source[192.168.70.92:50010, utilization=64.2691664228416]]
2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 underutilized: [BalancerDatanode[192.168.60.151:50010, utilization=38.248797525595634], BalancerDatanode[192.168.60.154:50010, utilization=8.900929142083783], BalancerDatanode[192.168.60.153:50010, utilization=37.771715581234325]]
2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 1.68 TB to make the cluster balanced.
2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.151:50010
2015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.154:50010
2015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.153:50010
2015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GB in this iteration
2015-10-13 09:42:49,653 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082424981_8684352 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.94:50010
2015-10-13 09:42:51,028 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082269380_8528751 with size=125931484 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.93:50010
2015-10-13 09:43:12,136 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082558990_8818361 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.93:50010
2015-10-13 09:43:17,073 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082547829_8807200 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.92:50010
2015-10-13 09:43:17,135 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082558979_8818350 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.93:50010
2015-10-13 09:43:18,541 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082544853_8804224 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.93:50010
2015-10-13 09:43:20,635 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082685915_8945292 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.94:50010
2015-10-13 09:43:24,574 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082685889_8945266 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.92:50010
2015-10-13 09:43:25,574 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082423441_8682812 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.92:50010
2015-10-13 09:43:27,540 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082675683_8935060 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.94:50010
2015-10-13 09:43:27,563 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082675703_8935080 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.94:50010
2015-10-13 09:43:28,065 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082424970_8684341 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.94:50010

以上是“HDFS balancer怎么用”這篇文章的所有內容，感謝各位的閱讀！相信大家都有了一定的了解，希望分享的內容對大家有所幫助，如果還想學習更多知識，歡迎關注億速云行業資訊頻道！