High-bandwidth memory

Memory built from DRAM chips stacked vertically beside a processor, delivering far more data per second than ordinary memory.

• Eight to sixteen thinned memory dies are stacked and linked by thousands of vertical connections that pass straight through the silicon.
• Each stack has a 1,024-bit interface (2,048 in HBM4) versus 32 or 64 bits for a normal memory chip, giving over 1 terabyte per second per stack.
• Generating each AI output token means reading the model's weights, so memory bandwidth often sets AI speed more than compute does.
• Only SK hynix, Samsung and Micron make it at scale; stacking yields are hard, and supply has often been sold out a year ahead.