IEEE ICMLA 2024 Publication
Multi-Modal AI Approach for Multi-Class Skin Disease Classification
- python
- ml
- ai
- bio
Last modified:
Click to view the paper
This study explores the application of deep learning models for skin disease classification using the PAD-UFES-20 dataset, which contains smartphone-captured images and clinical patient information, to provide a scalable, non-invasive screening tool that can complement conventional methods.
The proposed approach implements intermediate fusion to combine the dataset’s modalities using vision transformers (ViT or DinoV2) to extract image features, and employs classical machine learning (XGBoost) to classify the fused data. Data preprocessing and augmentation techniques were applied to handle class imbalance and low-quality images. Results show that the proposed approach achieves high accuracy, outperforming benchmarks from existing studies.
Grad-CAM visualizations confirmed that the models primarily focus on relevant skin lesion features, but also identified areas of potential improvement, such as reducing sensitivity to irrelevant regions.
This research demonstrates the promise of deep learning models in dermatology for remote skin disease screening, potentially improving access to dermatological care in underserved populations and healthcare deserts. Future work will focus on refining the proposed approach to further increase its performance and generalization capabilities, as well as to refine the explanation provided with the model’s classification.
View poster below
