The Loss Does Not See the Basis, But Adam Does [R]
In a factored model W = UV^T, the loss is invariant to rotations (U,V) → (UQ, VQ). GD respects that. Adam's per-coordinate second moment doesn't, because it depends on which basis you happen to write the factors in. The
![The Loss Does Not See the Basis, But Adam Does [R]](https://preview.redd.it/cldvfu1oyyih1.png?width=140&height=54&auto=webp&s=61d65e0f3ac17cc14df5849e09b0d2fb4da4f34f)
📄
This source provides headlines only. Use the button below to read the complete article on the original site.
📰 Read the original article on r/MachineLearning
Originally published by r/MachineLearning. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.