Comparative evaluation of attention-based early fusion techniques for fine-grained multimodal fake news detection
Idza Aisara Norabid1, Masita Jalil2, Rozniza Ali3 and Noor Hafhizah Abd Rahim3
Faculty of Computer Science and Mathematics,Universiti Malaysia Terengganu,Kuala Terengganu,Malaysia1
Software Engineering Group,Faculty of Computer Science and Mathematics, Universiti Malaysia Terengganu,Kuala Terengganu,Malaysia2
Artificial Intelligence Group,Faculty of Computer Science and Mathematics, Universiti Malaysia Terengganu,Kuala Terengganu,Malaysia3
Software Engineering Group,Faculty of Computer Science and Mathematics, Universiti Malaysia Terengganu,Kuala Terengganu,Malaysia2
Artificial Intelligence Group,Faculty of Computer Science and Mathematics, Universiti Malaysia Terengganu,Kuala Terengganu,Malaysia3
Corresponding Author : Noor Hafhizah Abd Rahim
Recieved : 31-December-2025; Revised : 14-June-2026; Accepted : 15-June-2026
Abstract
The rapid proliferation of misinformation across online platforms has intensified the need for reliable automated fake news detection systems, particularly those capable of processing multimodal content that combines textual and visual information. This study investigates the effectiveness of attention-based fusion mechanisms, namely self-attention (SA), multi-head attention (MHA), and co-attention (CoAtt), for enhancing multimodal fake news detection through the integration of textual features extracted using bidirectional encoder representations from transformers (BERT) and visual features derived from residual network (ResNet) architectures. Two model configurations, BERT+ResNet18 and BERT+ResNet50, were evaluated using a fine-grained multimodal dataset comprising six news categories: true, satire, misleading, manipulated, false connection, and imposter content. Feature fusion was performed at the representation level, and attention mechanisms were employed to improve cross-modal feature interactions and representation learning. Model performance was evaluated using accuracy, precision, recall, and F1-score metrics. The experimental results demonstrate that MHA consistently outperforms the other attention-based fusion methods and remains competitive with conventional fusion approaches across both model configurations, achieving accuracies of 73% and 74% for the BERT+ResNet18 and BERT+ResNet50 models, respectively. Furthermore, per-class analysis reveals that MHA provides a more balanced classification performance across both majority and minority classes, whereas SA and CoAtt exhibit reduced effectiveness when handling underrepresented categories. Overall, the findings suggest that attention-based fusion strategies, particularly MHA, provide a robust and effective framework for fine-grained multimodal fake news detection.
Keywords
Fake news detection, Multimodal learning, BERT, ResNet, multi-head attention (MHA), Fusion technique.
Cite this article
Norabid IA, Jalil M, Ali R, Rahim NHA. Comparative evaluation of attention-based early fusion techniques for fine-grained multimodal fake news detection. International Journal of Advanced Technology and Engineering Exploration. 2026;13(139):955-967. DOI : 10.19101/IJATEE.2025.121221676
References
[1]
Singh VK, Ghosh I, Sonagara D. Detecting fake news stories via multimodal analysis. Journal of the Association for Information Science and Technology. 2021; 72(1):3-17.
[2]
Shen X, Huang M, Hu Z, Cai S, Zhou T. Multimodal fake news detection with contrastive learning and optimal transport. Frontiers in Computer Science. 2024; 6:1-10.
[3]
Mostafa M, Almogren AS, Al-qurishi M, Alrubaian M. Modality deep-learning frameworks for fake news detection on social networks: a systematic literature review. ACM Computing Surveys. 2024; 57(3):1-50.
[4]
Chen H, Yu Y, Guo H, Hu B, Hu S, Hu J, et al. A self-learning multimodal approach for fake news detection. Frontiers in Artificial Intelligence. 2025; 8:1-10.
[5]
Bhattarai B, Granmo OC, Jiao L. Explainable tsetlin machine framework for fake news detection with credibility score assessment. In proceedings of the thirteenth language resources and evaluation conference 2022 (pp. 4894-903). ELRA.
[6]
Davoudi M, Moosavi MR, Sadreddini MH. DSS: a hybrid deep model for fake news detection using propagation tree and stance network. Expert Systems with Applications. 2022; 198:116635.
[7]
Nadeem MI, Mohsan SA, Ahmed K, Li D, Zheng Z, Shafiq M, et al. HyproBert: a fake news detection model based on deep hypercontext. Symmetry. 2023; 15(2):1-21.
[8]
Comito C, Caroprese L, Zumpano E. Multimodal fake news detection on social media: a survey of deep learning techniques. Social Network Analysis and Mining. 2023; 13(1):1-22.
[9]
Shahi GK. Multimodal misinformation detection using early fusion of linguistic, visual, and social features. In companion publication of the 17th web science conference 2025 (pp. 11-8). ACM.
[10]
Zeng F, Yao J, Xu Y, Liu Y. A multimodal knowledge representation method for fake news detection. In 4th international conference on computer, control and robotics (ICCCR) 2024 (pp. 360-4). IEEE.
[11]
Zou T, Qian Z, Li P, Zhu Q. Pvcg: prompt-based vision-aware classification and generation for multi-modal rumor detection. In international conference on acoustics, speech and signal processing (ICASSP) 2024 (pp. 11036-40). IEEE.
[12]
Lin SY, Chen YC, Chang YH, Lo SH, Chao KM. Text–image multimodal fusion model for enhanced fake news detection. Science Progress. 2024; 107(4):1-31.
[13]
Xu P, Zhu X, Clifton DA. Multimodal learning with transformers: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023; 45(10):12113-32.
[14]
Uppada SK, Patel P, B S. An image and text-based multimodal model for detecting fake news in OSN’s. Journal of Intelligent Information Systems. 2023; 61(2):367-93.
[15]
Ashish V. Attention is all you need. Advances in Neural Information Processing Systems. 2017; 30.
[16]
Lu J, Batra D, Parikh D, Lee S. Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Proceedings of the 33rd international conference on neural information processing systems 2019 (pp. 13-23). ACM.
[17]
Yadav A, Gaba S, Khan H, Budhiraja I, Singh A, Singh KK. ETMA: efficient transformer-based multilevel attention framework for multimodal fake news detection. IEEE Transactions on Computational Social Systems. 2023; 11(4):5015-27.
[18]
Luvembe AM, Li W, Li S, Liu F, Wu X. CAF-ODNN: complementary attention fusion with optimized deep neural network for multimodal fake news detection. Information Processing & Management. 2024; 61(3):1-26.
[19]
Tuan NM, Minh PQ. Multimodal fusion with BERT and attention mechanism for fake news detection. In RIVF international conference on computing and communication technologies (RIVF) 2021 (pp. 1-6). IEEE.
[20]
Liu S, Yue X, Wu F, Sun J, Feng Y, Ji Y. Semantic distillation and structural alignment network for fake news detection. In international conference on acoustics, speech and signal processing (ICASSP) 2024 (pp. 6620-4). IEEE.
[21]
Sachan T, Pinnaparaju N, Gupta M, Varma V. SCATE: shared cross attention transformer encoders for multimodal fake news detection. In proceedings of the international conference on advances in social networks analysis and mining 2021 (pp. 399-406). IEEE.
[22]
Yan F, Zhang M, Wei B, Ren K, Jiang W. FMC: multimodal fake news detection based on multi-granularity feature fusion and contrastive learning. Alexandria Engineering Journal. 2024; 109:376-93.
[23]
Yang Y, Liu J, Yang Y, Cen L. Dual-stream fusion network with multi-head self-attention for multi-modal fake news detection. Applied Soft Computing. 2024; 167:112358.
[24]
Qian S, Wang J, Hu J, Fang Q, Xu C. Hierarchical multi-modal contextual attention network for fake news detection. In proceedings of the 44th international SIGIR conference on research and development in information retrieval 2021 (pp. 153-62). ACM.
[25]
Lu Y, Yao N. A fake news detection model using the integration of multimodal attention mechanism and residual convolutional network. Scientific Reports. 2025; 15(1):1-15.
[26]
Gan C, Fu X, Feng Q, Zhu Q, Cao Y, Zhu Y. A multimodal fusion network with attention mechanisms for visual–textual sentiment analysis. Expert Systems with Applications. 2024; 242:1-12.
[27]
Wu Y, Zhan P, Zhang Y, Wang L, Xu Z. Multimodal fusion with co-attention networks for fake news detection. In findings of the association for computational linguistics: ACL-IJCNLP 2021 (pp. 2560-9). ACL.
[28]
Cao J, Wu J, Shang W, Wang C, Kang S, Tong Y, et al. Fake news detection based on cross-modal ambiguity computation and multi-scale feature fusion. Computers, Materials, & Continua. 2025; 83(2):2659-75.
[29]
Zheng J, Zhang X, Guo S, Wang Q, Zang W, Zhang Y. MFAN: multi-modal feature-enhanced attention networks for rumor detection. In proceedings of the thirty-first international joint conference on artificial intelligence 2022 (pp. 2413-9).
[30]
Qian L, Xu R, Zhou Z. MRDCA: a multimodal approach for fine-grained fake news detection through integration of RoBERTa and DenseNet based upon fusion mechanism of co-attention. Annals of Operations Research. 2025; 348(1):257-78.
[31]
Hu L, Zhao Z, Qi W, Song X, Nie L. Multimodal matching-aware co-attention networks with mutual knowledge distillation for fake news detection. Information Sciences. 2024; 664:120310.
[32]
Visweswaran M, Mohan J, Kumar SS, Soman KP. Synergistic detection of multimodal fake news leveraging TextGCN and vision transformer. Procedia Computer Science. 2024; 235:142-51.
[33]
Nakamura K, Levy S, Wang WY. Fakeddit: a new multimodal benchmark dataset for fine-grained fake news detection. In proceedings of the twelfth language resources and evaluation conference 2020 (pp. 6149-57). ELRA.
[34]
Abdureyim D, Ma B, Yang Y, Dong R, Chen Y, Anwar A, et al. Bidirectional feature fusion and adaptive decision network for multimodal fake news detection. In international conference on multimedia and expo (ICME) 2025 (pp. 1-6). IEEE.
[35]
Devlin J, Chang MW, Lee K, Toutanova K. Bert: pre-training of deep bidirectional transformers for language understanding. In proceedings of the conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) 2019 (pp. 4171-86). ACL.
[36]
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In proceedings of the conference on computer vision and pattern recognition 2016 (pp. 770-8). IEEE.
[37]
Chen Y, Li D, Zhang P, Sui J, Lv Q, Tun L, et al. Cross-modal ambiguity learning for multimodal fake news detection. In proceedings of the web conference 2022 (pp. 2897-905). ACM.
[38]
Almarashy AH, Feizi-derakhshi MR, Salehpour P. Elevating fake news detection through deep neural networks, encoding fused multi-modal features. IEEE Access. 2024; 12:82146-55.
[39]
Hua J, Cui X, Li X, Tang K, Zhu P. Multimodal fake news detection through data augmentation-based contrastive learning. Applied Soft Computing. 2023; 136:110125.
[40]
Zhang B, Xiong D, Su J, Duan H, Zhang M. Variational neural machine translation. In proceedings of the conference on empirical methods in natural language processing 2016 (pp. 521-30). ACL.
[41]
Huang Z, Hu Y, Zeng Z, Li X, Sha Y. Multimodal stacked cross attention network for fine-grained fake news detection. In international conference on multimedia and expo (ICME) 2023 (pp. 2837-42). IEEE.
[42]
Kumar V, Singh P, Ahmad W. Enhancing fake news detection: a multi-modal approach integrating reinforcement learning, random forest, and LSTM. Turkish Journal of Engineering. 2025; 9(4):831-47.
[43]
Zhang C, Bengio S, Hardt M, Recht B, Vinyals O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM. 2021; 64(3):107-15.
[44]
Nakkiran P, Kaplun G, Bansal Y, Yang T, Barak B, Sutskever I. Deep double descent: where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment. 2021; 2021(12):1-33.
[45]
Chen J, Pan T, Zhu Z, Liu L, Zhao N, Feng X, et al. A deep learning-based multimodal medical imaging model for breast cancer screening. Scientific Reports. 2025; 15(1):1-13.
[46]
Hamed SK, Ab AMJ, Yaakub MR. A review of fake news detection approaches: a critical analysis of relevant studies and highlighting key challenges associated with the dataset, feature representation, and data fusion. Heliyon. 2023; 9(10):1-21.