1. Configuration

1.1 Enable AvgShedder

First and foremost, without a doubt, I would recommend applying AvgShedder as the shedding and placement strategy. It represents the optimal solution among all available options. To configure AvgShedder as your load - balancing algorithm, set up the following configuration:

loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder
loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder

It should be noted that AvgShedder was introduced in versions 3.0.6, 3.2.4, and 3.3.1. Therefore, if you are using an earlier version, you need to upgrade your Pulsar version to 3.0.6 or later.

One more thing to note is that there is a collision in the configuration name of AvgShedder in versions 3.0.6-3.0.7, 3.2.4 and 3.3.1-3.3.2, which is fixed by PR: Fix AvgShedder strategy check. You need to apply the following configuration additionally to avoid the collision:

loadBalancerPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder

To check if the configuration is successful, you can scan your leader broker logs for the following log:

log.info("brokerHitCountForHigh:{}, brokerHitCountForLow:{}", brokerHitCountForHigh, brokerHitCountForLow);

This log will be printed every time AvgShedder is triggered. If you see this log, it indicates that AvgShedder is successfully configured and running.

For the sake of simplicity, you may simply add the following configuration to your broker.conf file to circumvent the compatibility issue.

loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder
loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder
loadBalancerPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder

1.2 Fine tune AvgShedder

AvgShedder reuse some configurations from UniformLoadShedder, while the default values of these configurations are not suitable for AvgShedder. You need to fine tune these configurations to make AvgShedder work better. The following is the recommended configuration:

# In the UniformLoadShedder and AvgShedder strategy, the maximum unload ratio.
# For AvgShedder, recommend to set to 0.5, so that it will distribute the load evenly
#  between the highest and lowest brokers.
maxUnloadPercentage = 0.5

Setting maxUnloadPercentage to 0.5 implies that AvgShedder will identify the brokers with the highest and lowest loads. Subsequently, it will evenly distribute the traffic between these two brokers, which represents a core concept of AvgShedder. When you set it to 0.2, AvgShedder will only allocate 20% of the traffic between the highest-loaded and lowest-loaded brokers. As a result, this would not yield the optimal outcome.

There are four configurations introduced by AvgShedder, which are used to control the shedding and placement strategy. The default values of these configurations are good for most scenarios. However, you can adjust them according to your needs.

# configuration for AvgShedder, a new shedding and placement strategy
# The low threshold for the difference between the highest and lowest loaded brokers.
loadBalancerAvgShedderLowThreshold = 15

# The high threshold for the difference between the highest and lowest loaded brokers.
loadBalancerAvgShedderHighThreshold = 40

# The number of times the low threshold is triggered before the bundle is unloaded.
loadBalancerAvgShedderHitCountLowThreshold = 8

# The number of times the high threshold is triggered before the bundle is unloaded.
loadBalancerAvgShedderHitCountHighThreshold = 2

AvgShedder handle load fluctuation with multiple hits algorithm, which means that the threshold is triggered multiple times before the bundle unload is finally triggered. For example, when the difference between a pair of brokers exceeds the threshold two times, load balancing is triggered.

In situations of cluster rolling restart or expansion, there is often a significant load difference between different brokers, thereby we hope to commence the load balancing more rapidly.

Consequently, we introduce two thresholds:

• loadBalancerAvgShedderLowThreshold, default value is 15

• loadBalancerAvgShedderHighThreshold, default value is 40

Two thresholds respectively correspond to two hit-count requirements:

• loadBalancerAvgShedderHitCountLowThreshold, default value is 8

• loadBalancerAvgShedderHitCountHighThreshold, default value of 2

When the difference in scores between two paired brokers exceeds the loadBalancerAvgShedderLowThreshold by loadBalancerAvgShedderHitCountLowThreshold times, or exceeds the loadBalancerAvgShedderHighThreshold by loadBalancerAvgShedderHitCountHighThreshold times, a bundle unload is triggered.

For example, with the default value, if the score difference exceeds 15, it needs to be triggered 8 times continuously, and if the score difference exceeds 40, it needs to be triggered 2 times continuously.

The default value of loadBalancerAvgShedderLowThreshold is 15. This implies that the range of the load difference between the brokers with the highest and lowest loads will be restricted to 15%. Should you desire to attain a more balanced state, you can reduce this value. For instance, you can set it to 10. Nevertheless, if you set it to an overly small value, it might cause the load balancing mechanism to be triggered too frequently, potentially resulting in performance degradation.

The default value of loadBalancerAvgShedderHitCountLowThreshold is 8, which implies that the load balancing mechanism will be triggered when the load difference between the brokers with the highest and lowest loads exceeds 15% for 8 times. Increasing this value will lead to a more stable load balancing mechanism with the trade-off of slower response speed.

1.3 Other configurations

1.3.1 Resource weight

AvgShedder reuse the resource weight configuration from ThresholdShedder, which is used to calculate the score of the broker. The default value of these configurations are good for most scenarios. However, you can adjust them according to your needs.

Setting these values to 0 means that the corresponding resource will not be used to calculate the score of the broker. Values between 0-1 are valid. The default value of these configurations are:

# The BandWidthIn usage weight when calculating new resource usage.
loadBalancerBandwidthInResourceWeight=1.0

# The BandWidthOut usage weight when calculating new resource usage.
loadBalancerBandwidthOutResourceWeight=1.0

# The CPU usage weight when calculating new resource usage.
loadBalancerCPUResourceWeight=1.0

# The direct memory usage weight when calculating new resource usage.
# Direct memory usage cannot accurately reflect the machine's load,
# and it is not recommended to use it to score the machine's load.
loadBalancerDirectMemoryResourceWeight=0

# The heap memory usage weight when calculating new resource usage.
# Deprecated: Memory is no longer used as a load balancing item
loadBalancerMemoryResourceWeight=0

The configuration name of loadBalancerBandwidthInResourceWeight was misspelled as loadBalancerBandwithInResourceWeight prior to version 4.0, which is resolved by PIP-357: Correct the conf name in load balance module. If you are using an earlier version and want to modify it, you should use loadBalancerBandwithInResourceWeight instead of loadBalancerBandwidthInResourceWeight. Same issue exists for loadBalancerBandwidthOutResourceWeight.

But we still support the old configuration name for backward compatibility.

/**
     * Get the load balancer bandwidth in resource weight.
     * To be compatible with the old configuration, we still support the old configuration.
     * If a configuration is not the default configuration, use that configuration.
     * If both the new and the old are configured different from the default value, use the new one.
     */
    public double getLoadBalancerBandwidthInResourceWeight() {
        if (loadBalancerBandwidthInResourceWeight != 1.0) {
            return loadBalancerBandwidthInResourceWeight;
        }
        if (loadBalancerBandwithInResourceWeight != 1.0) {
            return loadBalancerBandwithInResourceWeight;
        }
        return 1.0;
    }

The default value of loadBalancerDirectMemoryResourceWeight is changed to 0 in version 3.1.1, which is fixed by disable balancing based on DirectMemory. If you are using an earlier version, It is recommended to set it to 0 as direct memory usage cannot accurately reflect the machine's load.

The default value of loadBalancerMemoryResourceWeight is changed to 0 in disable balancing based on DirectMemory.

For simplicity, i suggest you add the following configuration to your broker.conf file so that you don't need to worry about version compatibility:

loadBalancerMemoryResourceWeight=0
loadBalancerDirectMemoryResourceWeight=0

1.3.2 do not distribute bundles evenly

There is a hidden mechanism in load balancer that it will forcefully distribute bundles evenly, which will conflicts with the load balancer algorithm severely.

    @FieldContext(
    dynamic = true,
    category = CATEGORY_LOAD_BALANCER,
    doc = "enable/disable distribute bundles evenly"
    )
    private boolean loadBalancerDistributeBundlesEvenlyEnabled = true;

I strongly recommend setting it to false, otherwise the load balancing algorithm will not work as expected. It is rather astonishing that this configuration is neither documented in the official documentation nor in the broker.conf file, and yet the default value is set to true.