2. OverloadShedder

Next, let's introduce OverloadShedder. When understanding different Shedders, focus on two key points:

• What criteria are used to determine if a broker is overloaded.

• Which bundles are selected for unloading once a broker is deemed overloaded.

OverloadShedder differs from ThresholdShedder in determining broker overload, but after calculating the amount of throughput to be unloaded, the algorithm for selecting bundles is the same: it selects bundles from largest to smallest.

Below is a detailed explanation of how OverloadShedder determines if a broker is overloaded. First, each broker is scored using the following method:

org.apache.pulsar.policies.data.loadbalancer.LocalBrokerData#getMaxResourceUsage

    public double getMaxResourceUsage() {
        // does not consider memory because it is noisy by gc.
        return max(cpu.percentUsage(), directMemory.percentUsage(), bandwidthIn.percentUsage(),
                bandwidthOut.percentUsage()) / 100;
    }

However, note that this does not use configurations like loadBalancerCPUResourceWeight for weighted calculations. Instead, it simply uses resource usage. As previously described, direct memory usage does not have a significant correlation with actual load, so it is likely to be affected by direct memory usage, leading to misjudgments of the broker's true load. Therefore, it would be better to make configurations like loadBalancerCPUResourceWeight effective for OverloadShedder as well, providing a means to avoid interference from direct memory usage. I submitted the following PR, which has been merged into the main branch:

PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl.

After obtaining the load score for each broker, it is compared with loadBalancerBrokerOverloadedThresholdPercentage. If the score exceeds this threshold, the broker is deemed overloaded. The default value is 85%.

# Usage threshold to determine a broker as over-loaded
loadBalancerBrokerOverloadedThresholdPercentage=85

Once a broker is determined to be overloaded, the amount of traffic to be unloaded is calculated as follows:

double percentOfTrafficToOffload = currentUsage - overloadThreshold + ADDITIONAL_THRESHOLD_PERCENT_MARGIN;
double brokerCurrentThroughput = localData.getMsgThroughputIn() + localData.getMsgThroughputOut();

double minimumThroughputToOffload = brokerCurrentThroughput * percentOfTrafficToOffload;

private static final double ADDITIONAL_THRESHOLD_PERCENT_MARGIN = 0.05;

For example, if a broker's maximum resource usage is 90%, the threshold is the default value of 85, and the broker's throughput is 10GB, then the amount of throughput to be unloaded is (0.9 - 0.85 + 0.05) * 10 GB = 1GB.

The subsequent bundle selection algorithm is the same as that employed by ThresholdShedder. Thus, it will not be elaborated on herein.

This algorithm is relatively simple but has many serious corner cases and is likely not widely used. Here are two examples:

• When every broker in the cluster reaches the threshold, bundle unloading will continuously occur. However, it will only switch traffic from one overloaded broker to another overloaded broker, which is meaningless and only disrupts normal user traffic.

• If no broker in the cluster reaches the threshold, adding a new idle broker will not balance traffic to the new broker.

Both of these issues have significant impacts, so it is not recommended to use OverloadShedder.