Transformers and attention

The transformer architecture uses an attention mechanism that lets the model focus on relevant parts of input. This powers GPT, BERT, and similar models.

• Attention lets models focus on relevant input.
• Transformers power modern language models like GPT.