An Explanation of In-Context Learning
Published:
This post surveys theoretical explanations of in-context learning based on Transformers implementing optimization algorithms in their forward pass. It reviews how linear attention can realize preconditioned gradient descent for linear regression and how nonlinear, kernel-based attention generalizes this mechanism to functional gradient descent, together with the underlying assumptions and limitations.