This study investigated the relationship between readability of 73 text-only multiple-choice questions from Trends in International Mathematics and Science Study (TIMSS) 2003 and performance of two groups of South African learners: those with limited English-language proficiency (learners attending African schools) and those with better English-language proficiency (learners attending non-African schools). Both groups were exposed to the same intended curriculum, but differed with respect to the quality of teaching they received, the availability of resources, and the level of functionality of their schools. Learners from non-African schools performed significantly better than learners from African schools. Three readability factors (sentence complexity, unfamiliar words, and long words) were analyzed. High sentence complexity resulted in random guessing in non-African schools, and favoring an incorrect answer in African schools. Some TIMSS items have complex wording, with numerous prepositional phrases and clauses, and unclear questions. Recommendations for maximum readability and comprehensibility were not met, and these items are therefore invalid for learners with limited English-language proficiency. Learners employ a range of strategies in attempting to answer questions that they do not understand. Overall, though, poor readability of TIMSS items does not fully account for South African learners' poor performance in TIMSS.