Determining fatigue and stiffness in polymer matrix composites helps with assessing their structural stability and longevity. We propose a model based on deep learning that uses attention aggregation networks to detect fatigue damage in composite materials, combining feature extraction, ensemble learning, and time–frequency analysis. First, sensor data are converted to scalogram images through continuous wavelet transforms to capture the complex, non-stationary features of Lamb wave data. These scalograms are then analyzed using AlexNet—a deep convolutional neural network (CNN) used is a transfer-learning approach with a pre-trained architecture —to obtain high-level spatial information with a low likelihood of overfitting due to data augmentation and dropout procedures. A minimum redundancy–maximum relevance (MRMR) algorithm is then used to clarify the relationships between the extracted features and both the fatigue states and optimal feature space. Finally, an ensemble learning technique is used to make the classification generalizable. Thus, we combine time–frequency feature extraction, CNN-based deep feature learning, MRMR feature optimization, and ensemble classification into a single pipeline to predict fatigue and stiffness in polymer matrix composites, achieving accuracy in excess of 99.77% on controlled laboratory datasets using CFRP specimens under Lamb wave interrogation.