Almost-scaled dot-product attention

3 minute read

TL;DR: Scaled dot product attention isn’t properly scaled, and that’s a good thing!