Group-batching Algorithm for High-throughput Network Requests

Abstract

It is common in distributed cloud systems to enhance the throughput of the system by grouping multiple API requests into a single batch API request to complete tasks in bulk. It is easy to batch them when the tasks are already managed by a single synchronous thread. But it is hard to group independent asynchronous tasks into several batches, each having its own unique attributes. This paper designed a leader election algorithm to effectively group high-throughput requests into several batches based on the requests’ attributes. So that the upstream service is less likely to be throttled by the downstream service, and the downstream service has lower workload.