Studying how model scale, pretraining exposure, and retrieval shape language model performance.
Overview
Our paper studies the interaction between pretraining and retrieval across model sizes, training budgets, and external datastores. The results show that retrieval complements learned knowledge, with benefits that vary by task and evaluation metric.