Skip to content

Instantly share code, notes, and snippets.

@vanbasten23

vanbasten23/gist:42e056748dabb38e81d34bb3d3cc19b3

Created November 19, 2025 18:31
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save vanbasten23/42e056748dabb38e81d34bb3d3cc19b3 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save vanbasten23/42e056748dabb38e81d34bb3d3cc19b3 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
Let's trace the values for my_id = 1 with num_devices = 4:
outer_step phase Accumulation Source left_copy_device right_copy_device Device providing the data
0 LEFT x_ref[left_copy_device, ...] (1+0+1)%4 = 2 (1-0-1)%4 = 0 Device 2
0 RIGHT x_ref[right_copy_device, ...] (1+0+1)%4 = 2 (1-0-1)%4 = 0 Device 0
1 LEFT x_ref[left_copy_device, ...] (1+1+1)%4 = 3 (1-1-1)%4 = 3 Device 3
1 RIGHT x_ref[right_copy_device, ...] (1+1+1)%4 = 3 (1-1-1)%4 = 3 Device 3
2 LEFT x_ref[left_copy_device, ...] (1+2+1)%4 = 0 (1-2-1)%4 = 2 Device 0
2 RIGHT x_ref[right_copy_device, ...] (1+2+1)%4 = 0 (1-2-1)%4 = 2 Device 2
As you can see, with each outer_step, the *_copy_device variables change, ensuring that the reduction operation fetches data from a new, distinct device. This systematic progression guarantees that by the end of all steps, each device has accumulated its required portion of the total sum from all other devices.
In summary, left_copy_device and right_copy_device are crucial for orchestrating the "reduce" part of the reduce-scatter by indexing into the correct source device's data at each step of the algorithm.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment