SOFO Project
- Developed SOFO, a second-order subspace Gauss-Newton optimiser that uses parallelised forward-mode differentiation instead of backpropagation, achieving constant memory complexity with respect to sequence length.
- SOFO enables efficient training of recurrent neural networks on long-horizon tasks, outperforming Adam on challenging benchmarks while maintaining per-iteration runtimes comparable to first-order optimisers.
- Check out our Neurips Spotlight paper here.