Foundation models and generative AI

This week, we discuss how to build multimodal foundation models. Building on concepts from Weeks 2, 5, and 6, we will introduce new learning principles, building blocks, and architectures that extend single-modality models to multimodal ones. We will focus in particular on vision-language models, before turning in the coming weeks to examples in biology, chemistry, and the earth sciences.

After discussing learning principles, building blocks, architectures, and examples across language, vision, biology, and physics, we will now turn to the phenomena that arise when foundation models scale—most notably emergent behavior and in-context learning. These capabilities are not explicitly programmed but appear as a consequence of training at scale, and they are central to understanding both the promise and limitations of foundation models.

This Week's TA: Lukas Klein