Apple ‘Foliar’ Disease Detection Analysis 🍎🌳
21 min readNov 16, 2022
Analyze the Plant Pathology 2020 dataset to build a CNN-based multi-class classification deep learning model that can predict the most common diseases in apple tree leaves.
Table of Contents:
- General facts about Apple Trees
- The life cycle of an Apple Tree
- How does an Apple Tree propagate?
- What are some common Apple Diseases?
- World Apple Production by country
- Apple Production in Indian States
- Economic loss caused by Apple Scab in India
- Economic loss caused by Apple Rust in India
- Problem Statement
- Machine Learning Formulation
- Challenges in building an AI System
- Dataset Description
- Performance metric
- Exploratory Data Analysis
- Data Pre-processing
- Modeling
- Models Performance
- Improving Best Model
- Explaining Model Predictions
- Project related documents
- Kaggle submission, LinkedIn profile, GitHub Repository, and WebApp
- What didn’t work?
- Future work
- References
General facts about Apple Trees 🌳
- Apple (biological name: Malus Domestica), is a deciduous tree in the family Rosaceae which is grown for its fruits, known as apples.
- Apple fruits are one of the most widely cultivated fruits in the world, are round (pome) in shape and range in color from green to red.
- When planted from a seed, an apple tree can take six to ten years to mature and produce fruit of its own.
- Apple trees are small to medium-sized trees reaching heights of 5–10 m (16.4–32.8 ft), with a central trunk that divides into several branches. The leaves of the tree are oval in shape and can reach up to 13 cm (5.1 in) in length and 7 cm (2.8 in) in width.
Apple Tree Life Cycle ♻️
It all starts with the plantation of seeds. Then, they change into a seedling which grows into adult apple tree. An adult apple tree grows flowers. And, flowers make fruits with seeds.
How does an Apple Tree propagate? 🌱🍏
- Apple trees grow best in the tropics and at higher latitudes, they require a mild growing season and a cold winter to break their dormancy.
- At these latitudes, the tree will flower in spring, and the fruit will ripen in the fall.
- In the tropics, the leaves will remain on the tree longer so that it is essentially evergreen and flowering and fruiting will happen sporadically throughout the year.
- Apple trees can also be propagated by grafting and mound layering.
What are some common Apple Diseases? 🦠
- Many different types of fungi and some varieties of bacteria can cause infections in an apple garden. Below are some of the common diseases as per their categories:
Bacterial Infection 🦠 — Crown Gall
- This disease is caused by the bacteria Rhizobium rhizogenes (old name Agrobacterium tumefaciens).
- If the tree looks like it has warts, it is probably crown gall. The galls start out light in color but turn dark over time.
- The growths vary in size — typically pea-sized to several inches — but they can grow to as much as a foot in diameter.
- Most of the bacteria will be at or just below the soil line, but some could be on the upper stems.
Bacterial Infection 🦠 — Fire Blight
- This disease is caused by the bacteria Erwinia amylovora.
- It is the most severe on new shoots but can rapidly spread to all parts of the tree above ground.
- If the plant appears as if it has been scorched by fire; watery exudate may be present on infected areas then it is probably a Fire Blight.
- The weather greatly affects the spread of this disease. Rain and high humidity cause it to spread rapidly.
Fungal Infection — Apple Scab🍃
- This serious fungal disease caused by Venturia inaequalis first manifests as small lesions on the bottoms of the leaves in the spring.
- From there, it spreads to the tops of the leaves and then to the fruit where it exhibits little brown scabs.
- It can result in total foliage loss by midsummer, which then makes the tree susceptible to other diseases.
- Symptoms are: Yellow or chlorotic spots on leaves; dark olive green spots on leaves and fruit; maybe a velvety growth on spots on the undersides of leaves; twisting of leaves; distorted leaves; severely infected leaves turn yellow and drop from a tree.
Fungal Infection — Rust🍃
- The most common type of rust on apples is cedar apple rust caused by the fungus Gymnosporangium juniperi-virginianae.
- The diseases known as rust are caused by complicated fungi. They live on another kind of plant before they spread to their final host — apples in this case.
- Their alternate host is a kind of juniper known commonly as eastern red cedar.
- Symptoms are bright orange or yellow patches on the top side of leaves surrounded by a red band and small black spots in the center; by mid-summer, cup-like structures called aecia form on the leaf undersides; these become covered in tubular structures from which spores are released.
World Apple Production by country 🌎
- Worldwide 87,236,221 tonnes of apple is produced per year. China is the largest apple producer in the world with 42,426,578 tonnes of production per year.
- China and the United States of America produce together more than 50% of the world’s total apples.
- India, with 2,316,000 tonnes of production per year is ranked at 5.
Apple Production in Indian States
- Jammu and Kashmir is the state to contribute around 77.71% of the total apple production of India. The total apple production of this state is around 1808.33 tonnes.
- And, Himachal Pradesh covers 19.19% of apple fruit production globally.
💰💰 Economic loss caused by Apple Scab
- Farmers spend a lot of money on disease control and inadequate technical support, but the results always lead to poor disease control.
- Foliar diseases spread rapidly and can destroy large portions of the yield in a very short time. In some cases, these diseases destroy the entire crop if the disease is not controlled quickly and accurately.
- These diseases reduce crop yields, fruit quality, and nutritional value, resulting in lower returns for the farmers.
“Journal of Plant Pathology,” says that In India, the total loss caused by apple scab may exceed that caused by all other pests and diseases combined. In epidemic years, apple losses have been as high as 70%. [1]
- The surveys from 2015 to 2018 in the 40 apple orchards in different blocks of Shimla district revealed that 17.80 quintals less fruit production was recorded in the affected orchards with net loss up to Rs. 3,91,102/- per hectare (72.68% loss in net return per hectare).
As per “Horticultural Research & Training Station” of Himachal Pradesh, the apple scab has caused damage to 50% of some of the orchards (in Shimla district) in Baghi, Ratnari, Kharapathar, parts of Jubbal areas. Also, in June 2019, apple scab disease caused a Rs 4,000–4,500 crore loss to the apple economy. [2]
💰💰 Economic loss caused by Apple Rust
- An apple grower from Rohru in India, Shimla district tells The Wire that “he never saw the hills of Himachal as warm and in need of rainfall as now.”
“Till June 2022, approximately 30% of the crop is already damaged in my orchard,” he says.
- Cracked and rusted apples, categorized as ‘Grade B’ and ‘Grade C’, are not sold in the main market.
- A healthy crop fetches its farmers Rs 60–70 a kilogram. But these apples are sold at no more than Rs 5–10 per kilogram.
In South Kashmir, a widespread outbreak of fungal diseases called “rust” in 2019’s apple crop was being reported. Scientists posted at the Krishi Vigyan Kendra (KVK) in Balpora Shopian said that the continuous rains are the main reason why rust diseases have afflicted apples.
- However, they said, rust doesn’t affect the life of an apple if it doesn’t develop cracks. They suggest that wetness from heavy dew and heavy rainfall for a long period of time, amid cold temperatures, leads to rust diseases. It also says that if frost occurs in the early fruit development stage then the chances of rust are high.
Problem Statement
Misdiagnosis of diseases impacts agricultural crops and can lead to:
- misuse of chemicals leads to the emergence of resistant pathogen strains,
- increased input costs, and
- more outbreaks with significant economic loss and environmental impacts
Current disease diagnosis based on human scouting is time-consuming and expensive. So here we are challenged to train a model which will classify a given image into one among —
- Healthy,
- Multiple Disease,
- Rust, and
- Scab
Machine Learning Formulation
In this problem, we are supposed to classify a given leaf image into one of the 4 above-mentioned categories. Hence this problem can be formulated as a multi-class classification problem.
- We will use the Macro-averaged F1 Score and AUC (Area under the ROC Curve) score to judge the performance of our model.
- And, while training, the model will focus to reduce the categorical cross-entropy loss.
Challenges in building an AI System
- Major crops today are plagued by a variety of diseases. Diseases in crops can occur in various parts of the plant, such as the roots, leaves, and stems, although the leaves are the most typical site for disease detection.
It is difficult to detect and diagnose diseases because leaves have a variety of sizes, shapes, and colors.
- Current disease diagnosis based on human scouting is time-consuming and expensive, and although computer-vision-based models have the promise to increase efficiency,
the great variance in symptoms due to the age of infected tissues, genetic variations, and light conditions within trees decreases the accuracy of detection.
- In recent years, with the development of machine learning technology and the enhancement of computer data processing capabilities, researchers use image processing, machine learning, and other methods to automatically identify crop diseases, such as those in rice, corn, wheat, cotton, tomato, and cucumber.
Thus, with the help of computer-vision-based techniques, we want to train a model using images of the training dataset to:
- accurately classify an unseen image into a different diseased category or a healthy leaf;
- accurately distinguish between many diseases, sometimes more than one on a single leaf;
- deal with rare classes and novel symptoms;
- address depth perception — angle, light, shade, physiological age of the leaf; and
- incorporate expert knowledge in identification, annotation, quantification, and guiding computer vision to search for relevant features during learning.
Dataset Description
The dataset that has been for this case study is taken from Plant Pathology 2020 Competition.
- It consists of high-quality, real-life RGB images of multiple apple foliar disease symptoms during the 2019 growing season from commercially grown cultivars in an unsprayed apple orchard at Cornell AgriTech (Geneva, New York, USA).
Source of Data: The dataset is available on Kaggle’s website. Please refer to the below-mentioned link:
Performance metric
- One of the biggest challenges in this problem is the number or percentage of multiple disease leaves is very less as compared to other classes which introduces the problem of severe class imbalance. Thus, ‘accuracy’ won’t be the correct metric to measure the performance of deep learning models.
- As an important initial step, we will need to plot the confusion matrix. Then we need to check the misclassification i.e. FP and FN. FN means the leaves predicted by the model are healthy but actually they were suffering from some diseases. FP means the leaves detected by the model are suffering from some diseases but actually were healthy.
So, we need to look at the below-mentioned metrics for measuring the model performance:
- Confusion Matrix: It is the table where TP, FP, TN, and FN counts will be plotted. From this table, we can visualize and track the number of mistakes made by the model.
- F1 Score: It is the harmonic mean of precision and recall.
As the Macro F1 score consists of both Precision and Recall. It is insensitive to the imbalance of the classes and treats them all as equal. Therefore, it will be a correct metric in our case of imbalanced data.
- AUC Score: AUC stands for Area Under ROC(Receiver Operating Characteristics) Curve. ROC plots TPR concerning FPR for different thresholds. The area under the curve depends on the ranking of the predicted probability score, not on absolute values.
- FPR and FNR: As the cost of misclassification is very high, therefore we need to monitor the FPR and FNR separately. We can afford to have a few False Positives because that will only raise a red flag for further manual investigation, however, we need to have the False Negatives as low as possible.
Exploratory Data Analysis
⭐ Target Label Data Distributions
Q1. What is the distribution of plant diseases in the TRAIN images of Plant Pathology Dataset 2020❓
OBSERVATION
From the above plot, we came to know that:
- 71.7% of TRAIN images are unhealthy. And, only 5% of TRAIN images have multiple diseases.
- Only 28% of the entire samples are non-infected or totally healthy.
- And, in the dataset, we have approximately the same number of infected and non-infected samples for
RustandScab.
OBSERVATION
- The above pie chart shows us the % share of records in a much better way.
- 34.2% of TRAIN images are of RUST. Quite similarly, 32.5% of TRAIN images belong to SCAB.
Q2. Does multiple diseases target label in the dataset means infected from both RUST & SCAB in Plant Pathology Dataset 2020❓
OBSERVATION
- The above graph confirms that the samples that are labeled as
Multiple Diseasesare also marked asun-healthy. - And, the multiple disease samples have no intersection with
RustandScabsamples.
Q3. Does un-healthy target label in the dataset means infected from either one or multiples diseases❓
OBSERVATION
- The above table and graph confirm that the samples that are
unhealthyare distributed among the other 3 classes. - And, there are no common records in this case.
OBSERVATION
- The above plot shows us the relationship between all 4 target categories.
- For a healthy leaf (healthy == 1) we don’t have any scabs, rust, or multiple diseases.
- Every unhealthy leaf has one scab, rust, or multiple diseases.
⭐ Displaying some images of each target disease category
Q4. Let’s visualize some of the images of HEALTHY class❓
OBSERVATION
From the above plot, we came to know that:
- Many images contain the background area or we can say that the main leaf is not in focus. Thus, we need to rescale or resize the images.
- Also, the light intensity is different in the images. As images have been taken in bright sunlight and in shadow.
- In some of the images, we can see a group of leaves in the main focus so either the entire group is labeled as
HEALTHYor we may need to bring the main leaf into focus.
OBSERVATION
From the above images, we can say that:
- The highlighted area within the red box can be the potential portion of the image for the CNN model to make the correct decision.
- The images have varying light intensities and overlapping leaves, thus we want the model to capture the feature of the focussed leaf.
Q5. Let’s visualize some of the images of MULTIPLE DISEASES class❓
OBSERVATION
From the above plot, we came to know that:
- The infected images contain some small-small patches. We need to check the change in the RGB intensity of these patches in the images.
- Many images contain the background area or we can say that the main leaf is not in focus. Thus, we need to rescale or resize the images.
- Also, the light intensity is different in the images. As images have been taken in bright sunlight and in shadow.
- In some of the images, we can see a group of leaves in the main focus area so either the entire group is labeled as
MULTIPLE DISEASESor we may need to bring the main leaf into focus.
OBSERVATION
In the above image, we found that:
- The highlighted area contains varying light intensity and small-small blackish spots which will be useful for correct identification.
- In the grayscale image, we can actually see these small spots along with the varying light intensity.
Q6. Let’s visualize some of the images of RUST class❓
OBSERVATION
From the above plot, we came to know that:
- The infected images contain some small-small patches. We need to check the change in the RGB intensity of these patches in the images.
- Many images contain the background area or we can say that the main leaf is not in focus. Thus, we need to rescale or resize the images.
- Also, the light intensity is different in the images. As images have been taken in bright sunlight and in shadow.
- In some of the images, we can see a group of leaves in the main focus so either the entire group is labeled as
RUSTor we may need to bring the main leaf into focus.
OBSERVATION
In the above image, we found that:
- The highlighted area contains varying light intensity and small-small yellowish spots which will be useful for correct identification.
- In the grayscale image, we can actually see these small spots along with the varying light intensity.
Q7. Let’s visualize some of the images of SCAB class❓
OBSERVATION
From the above plot, we came to know that:
- The infected images contain some small-small patches. Also, we can see the big black/blueish patches. We need to check the change in the RGB intensity of these patches in the images.
- Many images contain the background area or we can say that the main leaf is not in focus. Thus, we need to rescale or resize the images.
- Also, the light intensity is different in the images. As images have been taken in bright sunlight and in shadow.
- In some of the images, we can see a group of leaves in the main focus area so either the entire group is labeled as
SCABor we may need to bring the main leaf into focus.
OBSERVATION
In the above image, we found that:
- The highlighted area contains varying light intensity and small-small blackish spots which will be useful for correct identification.
- In the grayscale image, we can actually see these small spots along with the varying light intensity.
⭐ View all images channels distribution
Q8. What are the distributions of the RGB channels in the TRAIN & TEST images❓
📊 ALL (TRAIN + TEST) Images
OBSERVATION
- The above graph shows us the distribution of RGB channel values for all the images.
- This initial look indicates the RED channel values are very much like the Normal Distribution. Whereas, for the BLUE channel we can see the heavy right tail.
- And, for the GREEN channel, we have a long left tail with overall values greater than the other two channels.
OBSERVATION
- The above plot is showing us how RGB channel values exist in the images.
- This again tells us that the peak of the BLUE channel is flat and wide. And, there are very few images with extreme values.
- And, majority of the images for the RED and GREEN channels have values closer to the mean.
Data Pre-processing 🧮
- If the dataset we are dealing with contains images of different sizes then it becomes mandatory to resize it to a fixed size.
- Large-sized images require more trainable parameters to train a model, resulting in the need for more computation power. Hence if we have limited computing power, size reduction of images can be the savior.
- The size to which we resize our image should be chosen carefully. We must make sure that we are not throwing away much of the information by reducing the size, at the same time not overloading our model with too many parameters.
- The second case would result in very slow training and Resource Exhaust errors.
- In this case study, I resized the images to (224, 224), and it gave decently good results.
Along with this, it is also a good idea to use Image Augmentation.
- Image augmentation is a process of creating new training examples from existing ones.
- It is done by applying operations like Scaling, Cropping, Flipping, Padding, and many more on Images.
This idea is useful especially in the case when we have a limited number of images in the provided dataset.
In my analysis, Image Pre-processing and Augmentation helped me achieve a better Macro F1 & AUC Score.
Modeling
- Convolutional Neural Network (CNN) based deep learning models have been the choice for Image Classification related tasks over Multilayer Perceptron (MLP).
- In every SOTA model architecture, most probably you will find CNN being used in it.
- Here are a few advantages of using CNN over MLP for image Classification tasks -
- Image datasets are usually of huge sizes. Using Convolutional layers shrinks the size of images and hence reduces the computation.
- CNN do weight sharing, also known as parameter sharing, because of the filters (or kernels) it uses.
- It makes use of pooling which helps in location invariance.
- It uses filters to identify patterns in image data without flattening, unlike MLPs, which flatten the input images. Hence CNN makes effective use of spatial information.
In my analysis and experimentations, I used Transfer Learning.
- In simple words, Transfer Learning is making use of pre-trained models.
- The advantage of using such a model is that these models are built after some extensive research and are trained using high-end processors, hence in most cases, such models give very good results.
- I tried various models, and DenseNet — 121 models initialized with ‘image-net’ weights having custom top and global tuning gave the best results.
Below is the code for my best model-
Models Performance
Below are the model scores and Confusion Matrix showing its performance on the validation dataset.
OBSERVATION
- Using image augmentation has really helped in building an impressive model for better identification of diseases in Apple Leaves.
- With the help of augmentation techniques, we are able to create a larger dataset with a wider variety of images.
- Using class weights has significantly improved the performances of the above set of models.
- In the performed experimentations, I found that
DenseNet-121&MobileNet V3 Smallhas performed well as compared to ResNet-50 & other models.
Improving Best Model
Below improvements are done on the above best-performing model to achieve better results:
Hyper-parameterized best performing model (DenseNet-121 -- Adam -- 0.0005)
- How the model is behaving on unseen images and explaining its predictions?
Improvement - 1
- Using
Focal Lossin place of Cross Entropy to deal with the class imbalance problem. - Using
Label Smoothingto reduce the chances of overfitting the model.
To gain an understanding of Focal Loss & Label Smoothing check out some of the good resources mentioned in the references.
Below are the results after Improvement 1:
Improvement - 2
- Using the above model as a
CNN feature extractorthentraining the ML modelson DL-based features to yield better results.
Explaining Model Predictions
After completing the training, hyper-parameter tuning, and improving the best-trained model, I went on to investigate on which basis the model is giving the predictions and which area of the image it is more focussing to generate the results.
For generating the explanations regarding the predictions from the trained model I used the popular approach Grad-CAM i.e. Gradients — Class Activation Mapping. It is a form of post-hoc attention and an extension of the earlier approach CAM. To gain an understanding of these approaches kindly check out the links mentioned in the references.
And, I made the comparison for incorrect predictions of the best-trained model that was corrected after introducing the HP-Tuning and Improvement-1.
Example — 1
OBSERVATIONS
- For the above image, the model has given the correct prediction with higher confidence.
- And, as we can see it has also focussed on the correct infected area of the image.
Example — 2
OBSERVATIONS
- For the above image, the model has given the correct prediction with higher confidence.
- And, as we can see it has also focussed on the correct infected area of the image.
Mistakes of BestModel corrected after Improvement
Example — 3
OBSERVATIONS
- For the above image, the model has gone completely wrong, although it has focussed on the correct area but is unable to capture the features representing the disease.
OBSERVATIONS
- For the above image, the model has given the correct prediction with higher confidence.
- And, as we can see it has also focussed on the correct infected area of the image.
Project related documents
Business Deck — This Document is explaining the problem of apple foliar disease detection from a global perspective.
Models Performance & Results — This spreadsheet is containing the results of various experimentations performed with the intent to get the best-performing model.
Project GitHub Repository
You can find the code components related to this project here:
Kaggle Submission
Model Deployment — WebApp
You can access the best-trained model via the web application:
LinkedIn profile
You can find me here: https://www.linkedin.com/in/raj-shr
What didn’t work?
- I experimented with various sizes of input images and found that reducing the size further from 224x224 didn’t help in improving the model's performance. Although, reducing the size does help in faster computation but it deteriorates the overall performance results.
- I experimented with various optimizers and learning rates for the global tuning of the model and found that the models were underachievers in terms of performance with non-momentum optimizers. And, using a higher learning rate(like 0.05, 0.01, 0.005) resulted in a model similar to a baseline model.
- I also experimented with fine-tuning custom top layers of different models and found that such models were slightly better in terms of performance as compared to the Simple CNN & Modified AlexNet models. But, they were only good at capturing the patterns for the
SCABYimages. For the minority class, their predictions were completely haywire.
Future Work
There are various interesting experiments that we can do -
- We can try increasing the Image size and Batch size while training the model.
- We can try some different weights initialization for now I have only experimented with ‘ImageNet’ weights but we can look for ‘NoisyStudent’ pre-trained weights.
- We can also experiment with other Image Augmentation libraries to bring more differentiations in the augmented images.
- We can also try to ensemble the different best-performing models to achieve a better final score.
- We can also experiment with GANs to solve the problem of insufficient images to achieve dataset expansion based on traditional image augmentation methods.
References
- [1] : https://croplife.org/case-study/fungicides-are-key-for-apple-production-in-india/
- [2] : https://en.gaonconnection.com/after-three-decades-apple-scab-disease-intensifies-in-himachals-orchards/
- Apple Leaf Disease Identification with a Small and Imbalanced Dataset Based on Lightweight Convolutional Networks
Access Link: https://www.mdpi.com/1424-8220/22/1/173
- AFD‑Net: Apple Foliar Disease multi-classification using deep learning on plant pathology dataset
Access Link: https://link.springer.com/content/pdf/10.1007/s11104-022-05407-3
- Diagnosis of Typical Apple Diseases: A Deep Learning Method Based on Multi-Scale Dense Classification Network
Access Link: https://www.frontiersin.org/articles/10.3389/fpls.2021.698474/full
- Understand Focal Loss & Label Smoothing
- Understand CAM & Grad-CAM
