Farmers‍‌‍‍‌‍‌‍‍‌ deep learning models used in precision agriculture are usually limited by the few computational resources of edge devices. Although many Convolutional Neural Network (CNN) architectures are available, it is not clear through rigorous benchmarking which model would provide the best trade-off between accuracy and latency for a specific crop like Chilli (Capsicum annuum) that has complicated pathological features like leaf curling. In this paper, We‍‌‍‍‌‍‌‍‍‌ have supported in this work different benchmark tests that have been run with cutting-edge lightweight architectures—MobileNetV3, ShuffleNetV2, and EfficientNet-B0—versus the heavy baseline ResNet-50 and a new attention-enhanced framework (Effi-AttnNet) in our paper. In order to compare the performance of different models, we conducted training and testing of all of them under similar hyperparameters on a dataset comprising six classes of chilli diseases. The comparative study results highlight the large differences in the models' performance: For example, ShuffleNetV2 was able to produce an accuracy as high as 98.66% at a very fast inference speed (122 FPS), whereas the widely used MobileNetV3 was hardly able to generalize and therefore its accuracy was only 81.23%. Effi-AttnNet, which is our‍‌‍‍‌‍‌‍‍‌ custom-built framework, performs better compared to the above-mentioned models with an accuracy of 99.73%. Such a high precision result combined with the model's capability to be efficiently deployed at the edge and essentially achieves an inference speed of 60.3 FPS. This benchmarking work has been instrumental in providing essential information to the decision-makers in choosing the ideal architecture for on-the-go, mobile-based plant disease diagnosis ‍‌‍‍‌‍‌‍‍‌systems.