Broadcasting: how a [4,256] table adds to a whole batch
The positional-embedding sum quietly relies on broadcasting. Here is the whole rule — align from the right, stretch the 1s — and why PyTorch never copies a byte.
Tag
The positional-embedding sum quietly relies on broadcasting. Here is the whole rule — align from the right, stretch the 1s — and why PyTorch never copies a byte.