This article explores the potential of cross‐national achievement studies through a critical reflection upon the experience of South Africa’s participation in the Trends in International Mathematics and Science Study (TIMSS). There are a growing number of multi‐country and international achievement studies. While recognizing the limitations of these studies, this article considers how far these studies can be an effective tool for the generation of in‐country information that can usefully inform policy and practice. It is argued that improvements can be made by using a simpler methodology for the analysis and presentation of information; better ways of analysing the influence of domestic dimensions of poverty, inequality and social stratification; and improved strategies for assessing progress over time against national baselines. The article concludes with a set of questions for systems to consider when thinking about participating in international or multi‐country achievement studies.