4. Production Effect

The algorithm is configured as follows:

loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder
loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder
loadBalancerAvgShedderHitCountHighThreshold = 2
loadBalancerAvgShedderHitCountLowThreshold = 8
loadBalancerAvgShedderLowThreshold = 15
loadBalancerAvgShedderHighThreshold = 40
maxUnloadPercentage = 0.5
minUnloadMessageThroughput = 10 * 1024 * 1024
minUnloadMessage = 10000

Since we have disabled the consideration of memory usage and direct memory usage, only network card usage and CPU usage remain as factors to monitor. Given that our network card performance is quite good and the performance bottleneck lies in the CPU, the basis for scoring brokers is CPU usage.

Therefore, we added the following panel: CPU usage range, to monitor the load gap between the highest-load and lowest-load brokers in the cluster.

max(pulsar_lb_cpu_usage) - min(pulsar_lb_cpu_usage)

the load gap between the highest-load and lowest-load brokers

According to the monitor data, although the CPU usage occasionally exceeds the 15% threshold, within a month, only one bundle unload operation was triggered as a result of the limit on the number of triggers. The stability of the system fully meets our expected requirements. At the same time, the load balance is also up to expectation, with the CPU usage range always controlled within 15%.

The threshold can be configured in line with your specific circumstances. If you want the resource usage between the machines in the cluster to be more balanced, you can further reduce the loadBalancerAvgShedderHitCountLowThreshold to 10. At the same time, to mitigate the potential increase in bundle unload frequency caused by this threshold reduction, you can correspondingly increase the loadBalancerAvgShedderHitCountHighThreshold.

Here, we also show the ratio of maximum to minimum throughput and the ratio of maximum to minimum message rate.

the ratio of maximum to minimum throughput

the ratio of maximum to minimum message rate

The ratios are well-controlled within a good range.