Where AI computation moves its data
By How things work · myAudience
myAudience original rendered teaching illustration. Image: illustration, not a documentary photograph. · Operator-authored generated illustration; scientific concept model reviewed separately.
Where AI computation moves its data
A processor performs operations only when the required instructions and data are available. Capacity asks how much a memory can hold; bandwidth asks how much data can move in a given time; latency asks how long a particular access takes. These are different limits.
CPUs and GPUs support different patterns of control and parallel work. In NVIDIA’s CUDA model, a CPU host coordinates work on GPU devices, which contain groups of execution resources and memory. That is one vendor’s documented model, not a definition of every accelerator or a promise that a GPU makes every task faster.
Registers and caches keep some data near execution units; working memory holds larger active data. Storage retains datasets and saved models beyond a particular computation. Moving data across these boundaries takes time. A processor can wait even while its arithmetic units have unused capacity.
High-bandwidth memory is an established specialised DRAM packaging family, distinct from a bigger disk. Its exact operation and variants deserve a later checked lesson. Here the essential idea is the data path: capacity, transfer rate and access pattern together determine useful performance.
Commercial CPUs, GPUs and dedicated accelerators are established. New architectures remain workload-dependent. Compare complete tasks with declared measurement boundaries; peak arithmetic rates alone omit data movement, software, communication and power costs.
Discuss: a team can buy faster accelerators or make its service usable on inexpensive devices. Which users gain or lose under each choice? Consider access, operating cost and measurable benefit rather than assuming faster hardware is always the fairer outcome.
Image: illustration, not a documentary photograph.
Worked example: An invented transfer moves 8 GB across a link sustaining 2 GB/s. Find an ideal lower-bound transfer time, ignoring startup, protocol overhead and contention.
Time = amount of data ÷ sustained transfer rate.
8 GB ÷ 2 GB/s = 4 s.
Other overheads can lengthen elapsed time; memory capacity alone does not change this ratio.
Answer: 4 seconds under the stated simplified assumptions.
Check your understanding: A model needs 12 GB working memory but a device provides 8 GB. Does doubling link bandwidth guarantee that the unchanged model fits?
Answer: No. Capacity is still insufficient for the unchanged 12 GB allocation; bandwidth measures transfer rate.
Why: Capacity and bandwidth answer different questions. Partitioning or representation changes would be additional design decisions.