Gear transmissions, due to demanding operating conditions, are particularly susceptible to wear. Among the most difficult to predict and, at the same time, the most critical forms of damage to the working surfaces of cylindrical gears—alongside scuffing—is surface fatigue wear (pitting). Pitting is classified as an unacceptable tribological degradation process leading to catastrophic wear, the propagation of which ultimately results in tooth fracture and consequently the failure of the entire gear system. For this reason, detecting them is an important task. This paper presents algorithms for classifying pitting defects using deep learning networks. Classification is based on recorded images of defects. Three types of feature extractors (convolutional network, vision transformer, and hybrid model) used in the classifier were tested. A neural network and a vision language model were used as the classifier. The best accuracy of 86% was achieved for the vision transformer together with the neural network. The novel elements are a comparison of three types of feature extractors with respect to their application to the detection (classification) of gear tooth pitting damage and the use of a multimodal language model for the detection of gear tooth pitting damage.