Agribusiness is the main source of income for roughly 70% of people who reside in rural areas. India is the world's second-largest producer of pulses, textile raw materials, spices, coconuts, and other agricultural products. India's gross domestic product (GDP) is significantly impacted by the agriculture industry. Technology advancements help the agricultural industry to forecast various elements, such as soil quality, crop quality, and, disease detection to boost crop yield. Disease detection is one of the essential tasks that have to be carried out in agriculture. The early identification of the leaf disease helps to prevent further spread to other leaves in the plant by which the yield can be improved. In this work, plant leaf disease detection and stage classification are performed based on the severity of leaf infection. A deep learning model, you only look once version5 (YOLOv5) is used to detect plant leaf disease then background of the diseased leaf is removed using U2-Net architecture followed by stage classification performed using vision transformer (ViT) for classifying it as different stages such as low, moderate, and high. A recommendation solution has been provided to mitigate the leaf disease. YOLOv5 was trained using different open-source datasets namely 1) PlantDoc and 2) Plantvillage. This work mainly concentrates on the apple leaf for performing stage classification. The YOLO v5 gives a maximum f1-score of 0.57 at a confidence score of 0.2 and the vision transformer with a background image gives an f1-score of 0.758 and without a background image, 0.908 f1-score is achieved.