Recent studies suggest climate-related delays in the timing of leaf coloration and abscission in maple trees but lack baseline data prior to the late 20th century. To better understand how autumn foliar phenology and late-season damage risks have changed for this genus over the past century, we evaluated 2,972 digitized herbaria specimens of red and sugar maple collected between 1826 and 2016 for the presence of leaves, autumn leaf coloration, and pathogen or herbivore damage. We found that the onset (first appearance) of colored leaves has shifted 0.26 days later each year, leading to a delay of more than a month in autumn phenology since 1880. We find that these shifts are related to precipitation regimes in both the fall and summer seasons and that more severe droughts are associated with higher probabilities of colored leaves. Moreover, we found that the probability of both herbivory and pathogen damage has increased significantly over the study period. In particular, we find a strong association between increasing summer drought conditions and increased probability of herbivory. Furthermore, the presence of foliar damage increased the probability of leaf coloration on herbaria specimens. However, the end-of-season abscission date (last appearance of leaves) was strongly associated with herbivory and climate in a contrary direction: Increasing yearly drought, higher fall temperatures, and the presence of herbivory were associated with earlier abscission. In fact, the last leaf dates for specimens with herbivory were nearly 2 weeks earlier than specimens without herbivore damage. Our study documents significant changes in maple senescence over the last 150 years and suggests that incorporating herbivory into models may improve our ability to predict forest responses to climate shifts.