https://prismix.dev/news/da85b988b44f
Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer...
arXiv:2609.19363v1 Announce Type: new Abstract: The query and key projections $\WQ,\WK$ in attention are almost always trained by Euclidean optimizers with no...
the geometryprojection matricesstiefelattentiontransformer