Background: Cognitive and physical impairment frequently co-occur in older people. The aim of this study was to assess the temporal order of these age-related changes in cognitive and physical performance and to assess whether a relationship was different across specific cognitive and physical domains and age groups. Methods: Cognitive domains included global, executive, and memory function; physical domains included gait speed and handgrip strength. These domains were assessed in two population-based longitudinal cohorts covering the age ranges of 55-64, 65-74, 75-85, and 85-90 years with a follow-up of 5-12 years. Cross-lagged panel models were applied to assess the temporal relationships between the different cognitive and physical domains adjusting for age, sex, education, comorbidity, depressive symptoms, and physical activity. Results: Over all age groups, poorer executive function was associated with a steeper decline in gait speed (p < .05). From the age of 85 years, this relationship was found across all cognitive and physical domains (p < .02). From the age of 65 years, slower gait speed and/or weaker handgrip strength were associated with steeper declines in global cognitive function (p < .02), with statistically significant results across all cognitive domains in the age group of 75-85 years (p < .04). Conclusions: The temporal relationship between cognitive and physical performance differs across domains and age, suggesting a specific rather than a general relationship. This emphasizes the importance of repeated measurements on different domains and encourages future research to the development of domain-and age-specific interventions.