Muse models

Current obstacles

Physics limits

• Every word re-reads the modelTo write each token the chip must stream all active weights from memory. Memory bandwidth (about 8 TB/s on top GPUs), not arithmetic, caps speed and cost per token.
• Returns shrink as a power lawError falls only as a small power of training compute: each fixed step of improvement needs roughly 10x more compute, energy and money than the one before.
• Trained for likely text, not true textThe model learns to produce plausible words. Facts seen only once or twice in training can't be recalled reliably, so some rate of confident errors is built into the method.