GPU work groups: why addresses determine memory traffic

By How things work · myAudience

Original myAudience educational illustration. Image: illustration, not a documentary photograph. · myAudience educational illustration; no vendor affiliation or measured hardware performance is implied.

GPU work groups: why addresses determine memory traffic

A GPU is useful when a task contains enough independent or cooperative work to keep many execution resources busy. Imagine applying the same calculation to a long row of sensor readings. The arithmetic may be simple, yet delivering the readings can take more time than using them. Capacity describes how many bytes fit, bandwidth the rate at which bytes cross a particular boundary, and latency the delay of a particular operation. These three quantities must not be exchanged in an explanation. A CPU can coordinate requests and handle irregular control while a GPU performs suitable parallel work. Neither role establishes that every problem benefits from an accelerator; the complete task includes preparation, transfer, waiting and checking the answer.

CUDA supplies one vendor's programming model for expressing this work. A kernel is device code executed by many threads; threads belong to blocks, and a block's threads are grouped into warps of 32. A lane here means a thread's position in that execution group, not one permanent physical processor core. The distinction matters: scheduling and instruction implementation depend on the hardware, and software should follow documented synchronization rules rather than assume a convenient physical lockstep. Different threads may select different branches, which can leave only some lanes doing useful work during a particular operation. More declared threads therefore do not by themselves prove more useful throughput.

Data live in several spaces with different access and lifetime rules. Registers hold thread-local values; block-shared workspace allows a block's threads to cooperate; caches may satisfy some accesses without fetching another copy from attached DRAM. Device global memory is accessible across the device's execution resources, not necessarily cheaply accessible from everywhere in the computer. A unified virtual address space organizes addresses; it does not assert that every byte occupies one physical location or that migration is free. Capacity must still accommodate simultaneously live allocations. Keeping intermediate data near the computation can avoid repeated transfers, but which boundary benefits depends on the actual access sequence and cache behavior.

The illustration isolates one request step to make its byte accounting testable. It stipulates 32 lanes requesting distinct four-byte words, aligned 32-byte service segments and no useful cache reuse. Such segment accounting corresponds to a documented CUDA coalescing example for compute capability 6.0 and above; it is not a rule for every accelerator or every protocol layer. Each lane's starting address identifies its segment through floor(address divided by 32). Because our four-byte words are properly aligned, no individual word crosses a segment boundary. Count distinct segments, then multiply by 32 to obtain the transferred bytes at this declared boundary. Useful bytes remain 32 times four, or 128, regardless of where the values are located.

With addresses 0, 4 and so on through 124, the requests occupy segments starting at 0, 32, 64 and 96. All 128 fetched bytes are useful to this step. Shift every address by four: the requests now run from 4 through 128, touching a fifth segment even though there are still only 128 useful bytes. Moving 160 bytes gives a useful fraction of 128 divided by 160, or 80%. A stride of 32 makes the addresses 0, 32 and so on through 992. Each word occupies a separate segment, so the service account is 1024 bytes and the useful fraction is 12.5%. This comparison counts traffic, not elapsed-time speedup. Request concurrency, caching and other work could change the time consequence.

Matrix layouts provide a reason these patterns occur. Adjacent values in a stored row can be close together, while following a column may jump by an entire row's length. A program can sometimes load nearby values cooperatively into shared workspace and then use a reordered view. The transformation is useful only if its saved global traffic exceeds its own extra work and storage costs. Consumers must not read values before the relevant producers have finished writing them; any reuse also needs a synchronization rule appropriate to its scope. Shared memory has banks, so removing one poor global pattern can introduce another conflict. The lesson's diagram does not certify a production kernel or claim that reordering makes every access inexpensive.

An engineering evaluation tests both meaning and performance. Compare outputs against a suitable reference, then profile a representative workload at explicitly named boundaries. A kernel can move fewer bytes and still become slower because added indexing, shared storage or register pressure limits other resources. Latency hiding schedules other available work during waits; it cannot create additional bandwidth or remove a true dependency. An apparent benefit from one small case may disappear with a different matrix shape or a working set that no longer fits in cache. Measure the complete useful task as well as the interesting kernel, and state whether initialization, data preparation and host-device copies are included.

The unfamiliar-transfer question asks what happens when the same 32 requested values already reside in a valid cache. Their addresses and useful-byte count have not changed, but the cold DRAM service account may no longer describe what crosses that boundary. A careful answer therefore retains the logical access pattern while changing the physical traffic claim. The same discipline prevents an unearned environmental conclusion: fewer modeled bytes alone do not supply a complete energy reading or justify a hardware purchase. Access, cost and privacy belong in the design brief alongside speed. Image: illustration, not a documentary photograph. It shows a bounded access model, with representative requests and explicit continuation marks rather than a literal layout of processing cores.

Worked example: A stipulated cold memory step uses 32 lanes, each requesting one distinct 4-byte value. Service units are aligned 32-byte segments, without cache reuse or extra protocol traffic. Compare aligned addresses 4i, shifted 4 + 4i and strided 32i for i = 0…31.

First define the boundary. Every lane requests a distinct, aligned four-byte word, and a service segment covers 32 consecutive bytes beginning at a multiple of 32. There are 32 × 4 = 128 useful bytes. This is a cold-request account without cached reuse. It counts memory service bytes, not every packet or joule in a computer, and assumes none of our individual words straddles a segment boundary.

For the aligned series, address 4i has segment index floor(4i/32) = floor(i/8), where i runs from 0 through 31. The indices are 0, 1, 2 and 3, each reached by eight lanes. Therefore four segments supply 4 × 32 = 128 bytes. Useful fraction 128/128 = 1, or 100%. No useful bytes have disappeared merely because one request supplies several lanes.

Shift the starting address by one word. Now the segment index is floor((4 + 4i)/32). Indices 0 through 4 occur: the first segment contributes only seven requested words, the middle three contribute eight each, and the last contributes one. These total 7 + 24 + 1 = 32 words. Five segments move 160 bytes, so useful fraction is 128/160 = 0.80, or 80%.

For stride 32, the address is 32i and segment index i. All 32 lanes touch different segments, although each wants only four bytes. Moved bytes are 32 × 32 = 1024; useful fraction 128/1024 = 0.125, or 12.5%. Seven of each segment’s eight words are not requested by this step. Later reuse could make them useful, but that would be another explicitly stated observation window.

The ratios alone do not establish a kernel speed ratio of eight or any fixed slowdown. A cache may supply some values; requests can overlap; another instruction or dependency may dominate. Our comparison keeps useful data and arithmetic unchanged and isolates one boundary. An actual performance experiment must state the cache condition, timing window and representative access sequence rather than silently convert an accounting result into a measurement.

A candidate layout conversion may put neighboring desired values together. It must preserve which value belongs to each output and complete the required writes before consumers read them. It also requires reads, writes, workspace and synchronization that cost time. Thus the justified recommendation is to test whether reuse over the actual task repays those costs, not to assert that every rearrangement or every larger processor is an improvement.

Answer: The same 128 useful bytes require 128, 160 and 1024 service bytes in the aligned, shifted and strided cold-request models. Their useful fractions are 100%, 80% and 12.5%. The independently enumerated segment counts are four, five and 32. These results support a conditional traffic diagnosis, not an observed speedup or energy saving: a changed cache state, observation window or conversion cost can change the engineering conclusion.

Check your understanding: If the 32 values are already in a valid cache, why cannot the same physical DRAM-byte or elapsed-time conclusion be carried across unchanged? Explain one layout-conversion cost and one correctness obligation.

Answer: The address grouping model counts transactions at a declared uncached service boundary; a hit can supply data without those DRAM transfers. Conversion adds reading/writing/workspace and must preserve index-to-value correspondence, so savings are conditional. A race-free completion rule is needed before reordered data are consumed.

Why: A valid comparison retains the declared task, precision and boundary; a theoretical limiting-resource account is not an observed user-time or energy guarantee.