In neuroscience, episodic memory is depicted as a process of activating “engrams” in the hippocampus that provide a static and faithful record of the past. In reality, behavioral research has established that human memory is dynamic and constructive, such that we do not replay the past, but rather, we rely on prior knowledge about events, along with a small amount of retrieved information to imagine how the past could have been. Drawing from this work, I propose a radical alternative to the dominant view in systems neuroscience: Rather than recording every moment of experience, the brain might reconstruct past events from prior knowledge and a small amount of event-specific information encoded at moments of high uncertainty or prediction error called “event boundaries”. Our data are consistent with the view that the hippocampus and neocortex serve as complementary learning systems, with the former playing a role in recording snapshots of cortical activity at event boundaries, and the latter involved in using prior knowledge to understand and reconstruct past events. Beyond episodic memory, this division of labor might be computationally optimal for spatial navigation, prediction, and decision-making.