Multi-Modal Deep Learning Approach for Waste Management: Integrating Image Classification and Text Mining for Environmental Awareness

Santi Prayudani; Ainul Hizriadi; Yuyun Yusnida Lase

doi:10.56741/IISTR.esl.002036

Authors

Santi Prayudani
santiprayudani@polmed.ac.id
Politeknik Negeri Medan https://orcid.org/0009-0006-7367-4761
Ainul Hizriadi Universitas Sumatera Utara https://orcid.org/0009-0004-4867-6453
Yuyun Yusnida Lase Politeknik Negeri Medan https://orcid.org/0009-0004-6828-8693

Keywords:

Convolutional Neural Networks, Deep Learning, Environmental Awareness, Multi-Modal Learning, Text Mining, Waste Classification

Abstract

Environmental degradation caused by inefficient waste management remains a major global challenge, largely due to the limitations of conventional systems that rely on manual waste sorting and limited utilization of heterogeneous data sources. This study proposes a novel multi-modal deep learning framework that integrates visual and textual information to enhance waste classification performance while simultaneously providing insights into environmental awareness. The proposed framework combines convolutional neural networks (CNNs) for waste image classification and a recurrent neural network with long short-term memory (LSTM) architecture for text analysis. Visual and textual feature representations are integrated through a feature-level fusion strategy using vector concatenation before final classification. The image dataset consists of six waste categories, cardboard, glass, metal, paper, plastic, and trash, while the textual dataset contains waste management descriptions, community feedback, and environmental discourse collected from public and field sources. Environmental awareness was assessed through text mining by identifying dominant themes related to recycling practices, waste sorting behavior, environmental responsibility, and public concern regarding pollution and sustainability issues. Experimental results demonstrate that the proposed multimodal framework achieves an accuracy of 88.9% and an F1-score of 0.89, outperforming image-only and text-only models with accuracies of 78.4% and 81.2%, respectively. This corresponds to absolute performance improvements of 10.5% over the image-based model and 7.7% over the text-based model, while reducing the classification error rate by 40.96%. Furthermore, the multimodal model exhibits superior robustness under degraded data conditions, with only a 4.7% reduction in accuracy compared to larger performance declines observed in unimodal approaches. The main contribution of this study lies in the integration of waste image recognition and environmental-awareness extraction within a unified multimodal learning framework, enabling not only accurate waste categorization but also the generation of behavioral and sustainability-related insights that support more intelligent and sustainable waste management systems.

Downloads

Download data is not yet available.

Author Biographies

Santi Prayudani, Politeknik Negeri Medan

Santi Prayudani is a lecturer in the Software Engineering Technology program at Politeknik Negeri Medan, Indonesia. Her academic interests include software engineering, information systems development, and applied computing. She is actively involved in teaching and academic activities related to software design, system development, and the implementation of technology-based solutions to support organizational and societal needs. She can be contacted at santiprayudani@polmed.ac.id.

Ainul Hizriadi, Universitas Sumatera Utara

Ainul Hizriadi is a lecturer in Information Technology at Universitas Sumatera Utara, Medan, Indonesia. His research interests include artificial intelligence, data science, and intelligent information systems. He can be contacted at ainul.hizriadi@usu.ac.id.

Yuyun Yusnida Lase, Politeknik Negeri Medan

Yuyun Yusnida Lase is a lecturer in Software Engineering Technology at Politeknik Negeri Medan, Indonesia. Her interests include software engineering and applied computing. Contact: yuyunlase@polmed.ac.id.

References

T. Kumari and A. S. Raghubanshi, “Waste management practices in the developing nations: challenges and opportunities,” Waste Manag. Resour. Recycl. Dev. world, pp. 773–797, 2023. DOI: https://doi.org/10.1016/B978-0-323-90463-6.00017-8

A. Lakhouit, “Revolutionizing urban solid waste management with AI and IoT: a review of smart solutions for waste collection, sorting, and recycling,” Results Eng., p. 104018, 2025. DOI: https://doi.org/10.1016/j.rineng.2025.104018

A. K. Rathoure, “Revolutionizing waste management with advancements in sorting and processing technologies,” in Municipal Solid Waste Management and Recycling Technologies, IGI Global, 2024, pp. 387–406. DOI: https://doi.org/10.4018/979-8-3693-4054-7.ch015

P. K. Gupta, V. Shree, L. Hiremath, and S. Rajendran, “The use of modern technology in smart waste management and recycling: artificial intelligence and machine learning,” in Recent advances in computational intelligence, Springer, 2019, pp. 173–188. DOI: https://doi.org/10.1007/978-3-030-12500-4_11

C. Ren, H. Jung, S. Lee, and D. Jeong, “Coastal waste detection based on deep convolutional neural networks,” Sensors, vol. 21, no. 21, p. 7269, 2021. DOI: https://doi.org/10.3390/s21217269

T. H. Yan, S. N. M. Azam, Z. M. Sani, and A. Azizan, “Accuracy study of image classification for reverse vending machine waste segregation using convolutional neural network.,” Int. J. Electr. Comput. Eng., vol. 14, no. 1, 2024. DOI: https://doi.org/10.11591/ijece.v14i1.pp366-374

W.-L. Mao, W.-C. Chen, C.-T. Wang, and Y.-H. Lin, “Recycling waste classification using optimized convolutional neural network,” Resour. Conserv. Recycl., vol. 164, p. 105132, 2021. DOI: https://doi.org/10.1016/j.resconrec.2020.105132

S. Alim, Komunikasi Lingkungan: Konsep Kunci dan Studi Kasus Terkini di Asia dan Indonesia. Universitas Brawijaya Press, 2024.

R. Egger and E. Gokce, “Natural language processing (NLP): An introduction: making sense of textual data,” in Applied data science in tourism: Interdisciplinary approaches, methodologies, and applications, Springer, 2022, pp. 307–334. DOI: https://doi.org/10.1007/978-3-030-88389-8_15

T. Sadiq and C. W. Omlin, “Sensing in Smart Cities: A Multimodal Machine Learning Perspective,” Smart Cities, vol. 9, no. 1, p. 3, 2025. DOI: https://doi.org/10.3390/smartcities9010003

T. Diwan and J. V Tembhurne, “Sentiment analysis: a convolutional neural networks perspective,” Multimed. Tools Appl., vol. 81, no. 30, pp. 44405–44429, 2022. DOI: https://doi.org/10.1007/s11042-021-11759-2

A. Rosewelt and A. Renjit, “Semantic analysis-based relevant data retrieval model using feature selection, summarization and CNN,” Soft Comput., vol. 24, no. 22, pp. 16983–17000, 2020. DOI: https://doi.org/10.1007/s00500-020-04990-w

Z. Ahmad, R. Jindal, A. Ekbal, and P. Bhattachharyya, “Multi-modality helps in crisis management: An attention-based deep learning approach of leveraging text for image classification,” Expert Syst. Appl., vol. 195, p. 116626, 2022. DOI: https://doi.org/10.1016/j.eswa.2022.116626

I. K. S. Al-Tameemi, M.-R. Feizi-Derakhshi, S. Pashazadeh, and M. Asadpour, “Multi-model fusion framework using deep learning for visual-textual sentiment classification,” Comput. Mater. Contin., vol. 76, no. 2, pp. 2145–2177, 2023. DOI: https://doi.org/10.32604/cmc.2023.040997

J. Yang and Q. Long, “A modification of adaptive moment estimation (adam) for machine learning,” J. Ind. Manag. Optim., vol. 20, no. 7, pp. 2516–2540, 2024. DOI: https://doi.org/10.3934/jimo.2024014

D. V. Yevle and P. S. Mann, “Artificial Intelligence‐Based Waste Management: A Review of Classification, Techniques, Issues, and Challenges,” Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 15, no. 2, p. e70025, 2025. DOI: https://doi.org/10.1002/widm.70025

Y. Huang, C. Du, Z. Xue, X. Chen, H. Zhao, and L. Huang, “What makes multi-modal learning better than single (provably),” Adv. Neural Inf. Process. Syst., vol. 34, pp. 10944–10956, 2021.

S. Li, Z. Chen, Q. Liu, W. Shi, and K. Li, “Modeling and analysis of performance degradation data for reliability assessment: A review,” IEEE Access, vol. 8, pp. 74648–74678, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2987332

U. Ullah et al., “A review of multi-modal learning from the text-guided visual processing viewpoint,” Sensors, vol. 22, no. 18, p. 6816, 2022. DOI: https://doi.org/10.3390/s22186816

Z. Lin, S. Yu, Z. Kuang, D. Pathak, and D. Ramanan, “Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 19325–19337. DOI: https://doi.org/10.1109/CVPR52729.2023.01852

Z. Han, F. Yang, J. Huang, C. Zhang, and J. Yao, “Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 20707–20717. DOI: https://doi.org/10.1109/CVPR52688.2022.02005

C. Shorten, T. M. Khoshgoftaar, and B. Furht, “Text data augmentation for deep learning,” J. big Data, vol. 8, no. 1, p. 101, 2021. DOI: https://doi.org/10.1186/s40537-021-00492-0

S. Zhu et al., “Multilingual Large Language Models: A Systematic Survey,” arXiv Prepr. arXiv2411.11072, 2024.