Workers in a ring add up their gradients chunk by chunk, each talking only to its neighbour: every worker gets the sum while sending less than two copies of its vector, however many workers there are.
This interactive visualization needs JavaScript.