Blog posts

2024

An Explanation of In-Context Learning

22 minute read

Published:

This post surveys theoretical explanations of in-context learning based on Transformers implementing optimization algorithms in their forward pass. It reviews how linear attention can realize preconditioned gradient descent for linear regression and how nonlinear, kernel-based attention generalizes this mechanism to functional gradient descent, together with the underlying assumptions and limitations.