Asphalt mixture is a kind of granular material, which consists of aggregates with different particle sizes. Extracting the particle distribution around the failure location of asphalt mixture is important for revealing the failure mechanism. In this paper, the digital image correlation (DIC) method is used to obtain the full-field strain of asphalt mixture, and then the high strain zone of asphalt mortar is extracted by digital image processing technology. For getting the aggregate distribution around the failure position, aggregate extraction method is proposed based on the image dilation algorithm. For AC-16, SMA-16 and OGFC-16 asphalt mixtures, under different temperature and loading speed conditions, according to the analysis of the aggregates distribution around the high strain area, it can be seen that the proportion of the main skeleton aggregates and the disruption aggregates around the high strain area is constantly changing with the loading process. It shows that the proportion of the main skeleton aggregates around the high strain area is decreasing, while the proportion of the disruption aggregates is increasing. This phenomenon shows that the damage evolution of asphalt mixture is expanding from the surrounding area of main skeleton aggregates to that of disruption aggregates.