2. ThresholdShedder + LeastResourceUsageWithWeight

2.1 Environment Configuration

A pulsar cluster was built with 5 brokers and 30 bookies.

The load balancing-related configurations are as follows:

loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.LeastResourceUsageWithWeight

Using the combination of ThresholdShedder and LeastResourceUsageWithWeight, most of the configuration is set to default values, and the bundle split and even distribution features are also disabled.

loadBalancerDistributeBundlesEvenlyEnabled=false
loadBalancerAutoBundleSplitEnabled=false
loadBalancerAutoUnloadSplitBundlesEnabled=false

2.2 Over Placement Problem

Three pressure testing tasks were launched:

After starting the pressure testing tasks, it took 22 minutes for the cluster to reach a stable state and triggered 8 rounds of bundle unload.

To facilitate debugging, some logs were added. The log regarding the first bundle unload is presented as follows:

2024-06-11T15:33:42,642+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder - brokerAvgResourceUsageWithoutHistory: {XXX.83:8081=0.6200548553466797, XXX.32:8081=0.6142524337768555, XXX.87:8081=0.34531837463378906, XXX.206:8081=0.6850704193115235, XXX.161:8081=0.4193758010864258}

This log line prints the intermediate scores of all brokers (i.e., the maximum resource utilization of each broker currently, before using the historical weight scoring algorithm). Evidently, brokers at addresses XXX.83:8081, XXX.32:8081, and XXX.206:8081 are under high load, whereas the remaining two brokers are under low load.

2024-06-11T15:33:42,642+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder - brokerAvgResourceUsage: {XXX.83:8081=0.146445592841173, XXX.32:8081=0.1780747564543283, XXX.87:8081=0.13442747117624326, XXX.206:8081=0.28951184996156754, XXX.161:8081=0.11923764428233738}, avgUsage: 0.1735394629431299, threshold: 0.1, minThroughputThreshold: 10.0MB

This log line prints the final scores (i.e., the result of the historical scoring algorithm), average score, and threshold for all brokers.

From the first two log lines, it can be seen that before starting the pressure testing tasks, the load on each broker was low, so the scores of all brokers at this time were significantly different from the actual load. Only the score of XXX.206:8081 exceeded the threshold: 28.95% > 17.35% + 10.0%. Consequently, a bundle unload operation was performed on it, giving rise to the following log:

2024-06-11T15:33:42,642+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder - Attempting to shed load on XXX.206:8081, which has max resource usage above avgUsage  and threshold 28.951184996156755% > 17.35394629431299% + 10.0% -- Offloading at least 14.70925448705765 MByte/s of traffic, left throughput 208.25151973726722 MByte/s

Unload a bundle and immediately execute the placement policy LeastResourceUsageWithWeight:

2024-06-11T15:33:42,663+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.LeastResourceUsageWithWeight - brokerAvgResourceUsageWithWeight:{XXX.83:8081=0.08251018381118773, XXX.32:8081=0.11141611766815185, XXX.87:8081=0.0459994751214981, XXX.206:8081=0.23925241661071778, XXX.161:8081=0.06012571454048156}, avgUsage:0.10786078155040742, diffThreshold:0.1, candidates:[XXX.83:8081, XXX.32:8081, XXX.87:8081, XXX.206:8081, XXX.161:8081]
2024-06-11T15:33:42,663+0800 [pulsar-load-manager-1-1] WARN  org.apache.pulsar.broker.loadbalance.impl.LeastResourceUsageWithWeight - Assign randomly as all 5 brokers are overloaded.

Given that the sum of any broker's score and 10 is greater than the average score of 10.7%, the candidate broker list remains empty, thereby triggering random allocation. This is the issue we described with LeastResourceUsageWithWeight: the candidate broker list can easily be empty, leading to random allocation.

2024-06-11T15:33:42,663+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.LeastResourceUsageWithWeight - Selected 5 best brokers: [XXX.83:8081, XXX.32:8081, XXX.87:8081, XXX.206:8081, XXX.161:8081] from candidate brokers: [XXX.83:8081, XXX.32:8081, XXX.87:8081, XXX.206:8081, XXX.161:8081], assign bundle public/default/0x70000000_0x80000000 to broker XXX.83:8081
 
2024-06-11T15:33:42,663+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - [ThresholdShedder] Unloading bundle: public/default/0x70000000_0x80000000 from broker XXX.206:8081 to dest broker XXX.83:8081

As can be seen, the unloaded bundle was assigned to the high-load broker XXX.83:8081! This is a wrong load balancing decision. In this experiment, the probability of triggering this issue is extremely high, nearly inevitable. As depicted in the figure below, the problem has been triggered four times consecutively.

The historical weight scoring algorithm also contributes to the high probability of occurrence. This is due to the fact that the criterion for choosing candidate brokers is that a broker's score must be 10 points lower than the average score. This implies that there needs to be a considerable gap among the scores of different brokers.

However, owing to the historical - weight scoring algorithm, the scores of all brokers can only gradually approximate their real load starting from around 20, making it difficult for the scores of different brokers to widen the gap. As a result, the LeastResourceUsageWithWeight algorithm can only perform random allocation.

2.3 Over Unloading Problem

To increase the load on a single broker, two brokers were shut down, and an abnormal load balancing was observed.

It can be seen that the yellow machine frequently switches between high-load and low-load states. Its CPU utilization descends from 80% to 40%, and then ascends back to 80%. In this process, the over unloading problem was first triggered, followed by the over placement problem.

Three rounds of load balancing were performed:

• In the first round, bundles were unloaded from the highest-load yellow machine XXX.206:8081 to the green machine XXX.83:8081. However, four bundle unload operations were performed in this round, causing the load on the yellow machine XXX.206:8081 to drop significantly and quickly become the broker with the lowest load, thus encountering the over unloading problem.

• In the second round, 11 bundles were unloaded from the highest-load blue machine XXX.32:8081 and assigned to the high-load green machine XXX.83:8081and the low-load machine XXX.206:8081. In this process, the over unloading problem occurred again, and the blue machine XXX.32:8081 became the broker with the lowest load. At the same time, there was also an over placement problem, as the bundle was mistakenly assigned to the high-load green machine XXX.83:8081.

• In the third round, the bundle was unloaded from the highest-load green machine XXX.83:8081 and reloaded to the blue machine XXX.32:8081. The cluster then entered a balanced state, with the entire process taking a total of 30 minutes.

With the help of broker logs, we can gain a deeper insight into the above process:

Within the first 10 minutes of the first round of bundle unload, it can be observed that the system continously identified XXX.206:8081 as a high-load broker and unloaded bundles from this node, eventually leading to the over unloading problem. This is due to the characteristics of the historical weight scoring algorithm, which causes the score of XXX.206:8081 to change very slowly. Even though the node has already unloaded the bundle and its real load has changed correspondingly, the score remains high, so it is still identified as a high-load broker and continues to be unloaded.

In the second round of bundle unload, bundles were unloaded from the highest-load XXX.32:8081. However, an over placement problem was encountered as these bundles were unloaded to another high - load server, XXX.83:8081.

2024-06-12T10:24:02,245+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder - Attempting to shed load on XXX.32:8081, which has max resource usage above avgUsage  and threshold 78.09468007403726% > 65.79112414711298% + 10.0% -- Offloading at least 14.886936767013715 MByte/s of traffic, left throughput 188.94441491927364 MByte/s
 
2024-06-12T10:24:02,246+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.LeastResourceUsageWithWeight - brokerAvgResourceUsageWithWeight:{XXX.83:8081=0.6968493632164602, XXX.32:8081=0.6564280053774565, XXX.206:8081=0.5447576150322107}, avgUsage:0.6326783278753757, diffThreshold:0.1, candidates:[XXX.83:8081, XXX.32:8081, XXX.206:8081]
2024-06-12T10:24:02,246+0800 [pulsar-load-manager-1-1] WARN  org.apache.pulsar.broker.loadbalance.impl.LeastResourceUsageWithWeight - Assign randomly as all 3 brokers are overloaded.
 
2024-06-12T10:24:02,247+0800 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - [ThresholdShedder] Unloading bundle: public/default/0x30000000_0x40000000 from broker XXX.32:8081 to dest broker XXX.83:8081

Up to this point, we have experimentally verified the two core defects of ThresholdShedder + LeastResourceUsageWithWeight:

• Over placement problem

• Over unloading problem

Both issues are exacerbated by the historical weight scoring algorithm. It is also evident that the load-balancing speed of ThresholdShedder + LeastResourceUsageWithWeight is slow. Due to incorrect load balancing decisions, the system often needs to perform load-balancing repeatedly to eventually reach a stable state.