Based on data from the China Health and Retirement Longitudinal Study (CHARLS) from 2011 to 2020, this study followed 5,776 non-frail middle-aged and older adults aged 45 and above with a baseline frailty index < 0.25, using Cox mixed-effects models to examine the determinants of frailty onset. The results indicated that: (1) a total of 1,863 individuals (32.3%) developed frailty during the 9-year follow-up, and participants who progressed to frailty were more likely to reside in single-story houses, lack running water and bathing facilities, rely on solid fuels, maintain lower indoor cleanliness, and have access to fewer community activity venues; (2) higher physical fitness facility scores exhibited a robust negative association with frailty risk. Interaction and subgroup analyses revealed significant population heterogeneities where high-quality housing conditions exerted a protective effect primarily among partnered individuals and the 1951–1966 birth cohort, the protective benefit of fitness facilities demonstrated broad consistency while showing a significant interaction effect with smoking status, and socio-cultural facilities displayed marked urban-rural disparities by predominantly benefiting rural populations. In conclusion, these findings suggest that future age-friendly environmental renovations should prioritize the expansion of community fitness facilities while implementing targeted interventions for vulnerable populations.