Linformer

Linformer is a neural network architecture designed to reduce the computational complexity of the Transformer model’s self-attention mechanism from quadratic to linear with respect to the input sequence length. It achieves this by projecting the attention computation into a lower-dimensional space, enabling efficient processing of long sequences.