The Convergence of Vision and Language: A Deep Learning Approach to Enhanced Visual Question Answering via Question-Guided Attention
DOI:
https://doi.org/10.66395/globeis.3Keywords:
Artificial Intelligence, Computer Vision, Natural Language Processing, Vision-Language Models (VLMs), Visual Question Answering (VQA), Deep Learning, Transformer, Multimodal AIAbstract
Computer Vision (CV) and Natural Language Processing (NLP) in combination, which is represented by Vision-Language Models (VLMs), is a critical field in the development of cognitive Artificial Intelligence (AI). The paper discusses one such limitation of the current VLMs, which is a major weakness of the current VLMs in Visual Question Answering (VQA) in that they are prone to linguistic bias and poor visual grounding. We propose an in-depth discussion and experimental evidence of a new framework of VLM that adds a Question-Guided Attention (QGA) module under the cross-modal fusion phase. QGA module filters the raw visual features dynamically and puts more emphasis to those that are most relevant to the question vector which is a semantic feature thus giving a cleaner visual signal to be used later in the reasoning process. The QGA-VLM model is experimentally tested on VQA v2.0 benchmark. Findings indicate that VQA accuracy has significantly improved in general as compared to baseline VLM, and that the most significant improvements in this case were achieved in more complicated types of reasoning, including counting and relational questions, which improved by an average of $\mathbf{+4.75 percent). The effectiveness of the QGA module in improving visual grounding and minimizing reliance on dataset priors is confirmed by this strong performance, and more trustworthy and human-interpretable multimodal AI systems are on the horizon.
Downloads
References
1. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539 DOI: https://doi.org/10.1038/nature14539
2. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778. https://doi.org/10.1109/CVPR.2016.90 DOI: https://doi.org/10.1109/CVPR.2016.90
3. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Houlsby, N., & Uszkoreit, J. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations (ICLR).
4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NIPS), 30.
5. Khan, A., Sohail, A., Zafar, N., & Ahmad, S. (2022). Vision Transformer (ViT): A systematic review of current advancements and future directions. Engineering Applications of Artificial Intelligence, 114, 105151. https://doi.org/10.1016/j.engappai.2022.105151 DOI: https://doi.org/10.1016/j.engappai.2022.105151
6. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. https://doi.org/10.18653/v1/N19-1423 DOI: https://doi.org/10.18653/v1/N19-1423
7. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems (NIPS), 33, 1877-1901.
8. Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning for natural language processing. IEEE Computational Intelligence Magazine, 13(3), 55–74. https://doi.org/10.1109/MCI.2018.2840738 DOI: https://doi.org/10.1109/MCI.2018.2840738
9. Baltrušaitis, T., Ahuja, C., & Morency, L. P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423-443. https://doi.org/10.1109/TPAMI.2018.2849184 DOI: https://doi.org/10.1109/TPAMI.2018.2798607
10. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Chatkovsky, G., McGoohan, C., ... & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. International Conference on Machine Learning (ICML), 8748-8763.
11. Tsimpoukelli, A., Menick, J., Cabi, S., Schmitt, S., Saatchi, K., Haines, T., ... & Hadsell, R. (2022). Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems (NIPS), 35, 12690-12702.
12. Lu, J., Batra, D., Parikh, D., & Lee, S. (2019). ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in Neural Information Processing Systems (NIPS), 32.
13. Tan, H., & Bansal, M. (2019). LXMERT: Learning cross-modality encoder representations from transformers. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5094-5101. https://doi.org/10.18653/v1/D19-1514 DOI: https://doi.org/10.18653/v1/D19-1514
14. Yu, H., Li, J., Cui, Z., Wu, X., & Lin, Y. (2019). Deep Modular Co-Attention Networks for Visual Question Answering. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 9037-9044. https://doi.org/10.1609/aaai.v33i01.33019037 DOI: https://doi.org/10.1609/aaai.v33i01.33018997
15. Agrawal, A., Lu, J., Antol, S., Mitchell, M., Parikh, D., & Batra, D. (2018). VQA: The language prior problem and the role of image. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 645-654.
16. Goyal, A., Agrawal, A., Lu, J., Parikh, D., & Batra, D. (2017). Making the V in VQA Matter: Elevating the Role of Image in Visual Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6904-6913. https://doi.org/10.1109/CVPR.2017.670 DOI: https://doi.org/10.1109/CVPR.2017.670
17. Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019). From recognition to cognition: Visual Commonsense Reasoning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6720-6731. https://doi.org/10.1109/CVPR.2019.00688 DOI: https://doi.org/10.1109/CVPR.2019.00688
18. Cadene, R., Dancette, C., Cord, M., & Perez, P. (2019). Rubi: Reducing unimodal biases in visual question answering. Advances in Neural Information Processing Systems (NIPS), 32.
19. Teney, D., Liu, L., & van den Hengel, A. (2020). The power of explicit knowledge in visual question answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8047-8056.
20. Ramakrishnan, M., Maillard, J., Tufvesson, M., & Lazaridou, A. (2021). VQA-E: Explaining the answer of visual question answering for robust grounding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12518-12528.
21. Le, T. H., Niu, Z., & Chen, Z. (2023). Guiding Visual Question Answering With Attention Priors. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2800-2809. https://doi.org/10.1109/WACV56688.2023.00282 DOI: https://doi.org/10.1109/WACV56688.2023.00436
22. Nguyen, K., & Okatani, T. (2018). Improved fusion of visual and language representations by dense symmetric attention for visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7064-7073. https://doi.org/10.1109/CVPR.2018.00741 DOI: https://doi.org/10.1109/CVPR.2018.00637
23. Chen, Y., Fan, H., Ma, T., & Liu, S. (2021). Revisiting Attention in VQA: a Question-Guided Approach. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 1079-1088.
24. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015). VQA: Visual Question Answering. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2901-2911. https://doi.org/10.1109/ICCV.2015.279 DOI: https://doi.org/10.1109/ICCV.2015.279
25. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NIPS), 25(2), 1097-1105.
26. Perez, E., Fischer, A., & Schiele, B. (2018). Multi-modal compact bilinear pooling for visual question answering and visual grounding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2807-2816. https://doi.org/10.1109/CVPR.2018.00296 DOI: https://doi.org/10.1109/CVPR.2018.00296
27. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR).
28. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., ... & Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NIPS), 32.
29. Li, Y., Wu, Z., Chen, Q., & Wang, Y. (2020). Multi-step reasoning visual question answering with refined relational attention. Neurocomputing, 416, 122-132. https://doi.org/10.1016/j.neucom.2020.08.006 DOI: https://doi.org/10.1016/j.neucom.2020.08.006
30. Zhang, H., Lu, H., & Zhang, C. (2021). A VQA approach with attention mechanism for spatial reasoning in urban scenes. Sensors, 21(2), 405. https://doi.org/10.3390/s21020405 DOI: https://doi.org/10.3390/s21020405
31. Wu, Y., Sun, W., Zhu, Y., & Wang, M. (2023). Multi-level feature fusion with self-attention for complex visual question answering. Applied Intelligence, 53(16), 20560-20573. https://doi.org/10.1007/s10489-023-04561-x
32. Alikhani, M., & Pereira, M. G. (2024). A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges. arXiv preprint arXiv:2405.09918.
33. Dinh, T. T. A., Nguyen, M. T., & Luong, M. T. (2023). Interpretable visual question answering with multi-modal attention and semantic grounding. Expert Systems with Applications, 218, 119561. https://doi.org/10.1016/j.eswa.2023.119561 DOI: https://doi.org/10.1016/j.eswa.2023.119561
34. Gunning, D., Stefik, M., Maynor, T., & Miller, T. (2019). XAI—Explainable Artificial Intelligence. Defense Advanced Research Projects Agency (DARPA) Report. DOI: https://doi.org/10.1126/scirobotics.aay7120
35. Liu, H., Li, C., Wu, Y., & Li, Y. (2023). Visual Instruction Tuning. Advances in Neural Information Processing Systems (NIPS), 36. DOI: https://doi.org/10.52202/075280-1516
36. He, J., Li, L., Li, H., & Li, X. (2023). A systematic review of deep learning for natural language processing in medical applications. Journal of Biomedical Informatics, 138, 104273. https://doi.org/10.1016/j.jbi.2023.104273
37. Wan, Y., Li, S., Wang, H., & Zhang, J. (2023). Vision-Language Models for Medical Image Analysis: A Survey. IEEE Transactions on Medical Imaging, 42(9), 2465-2481. https://doi.org/10.1109/TMI.2023.3287102
38. Alayrac, J. B., Donahue, J., Luc, P., Miech, K., Zhai, X., Kolesnikov, A., ... & Hadsell, R. (2022). Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NIPS), 35, 23716–23736. DOI: https://doi.org/10.52202/068431-1723
39. Russell, S., & Norvig, P. (2020). Artificial Intelligence: A Modern Approach (4th ed.). Pearson Education.
40. Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR).
41. Vinyals, O., Toshev, A., Bengio, S., & Lu, J. (2015). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3156-3164. https://doi.org/10.1109/CVPR.2015.7298935 DOI: https://doi.org/10.1109/CVPR.2015.7298935
42. Clark, R., Kembhavi, A., & Lee, J. (2023). On the limits of visual question answering: Investigating and rectifying model priors. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2577-2587.
43. Shekhar, S., Lu, J., & Batra, D. (2022). Towards conversational visual question answering. International Conference on Learning Representations (ICLR).
44. Li, W., Li, X., Liu, C., & Zhang, H. (2024). A Survey of State-of-the-Art Vision-Language Models. Journal of Artificial Intelligence Research, 70, 1-64.
45. He, J., Li, L., Li, H., & Li, X. (2023). A systematic review of deep learning for natural language processing in medical applications. Journal of Biomedical Informatics, 138, 104273. https://doi.org/10.1016/j.jbi.2023.104273
46. Bajwa, J., Munir, K., Mehmood, S. A., & Bano, N. (2021). Artificial Intelligence in healthcare: Transforming diagnosis, treatment, and medical research. Informatics in Medicine Unlocked, 23, 100521. https://doi.org/10.1016/j.imu.2021.100521 DOI: https://doi.org/10.1016/j.imu.2021.100521
47. Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 248–255. https://doi.org/10.1109/CVPR.2009.5206848 DOI: https://doi.org/10.1109/CVPR.2009.5206848
48. Wagh, K., & Alampallam, P. (2024). A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question Answering. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2901-2911.
49. Varma, P., & Verma, N. (2024). Physically Grounded Vision-Language Models for Robotic Manipulation. 2024 IEEE International Conference on Robotics and Automation (ICRA), 12462-12469. https://doi.org/10.1109/ICRA57147.2024.10610090 DOI: https://doi.org/10.1109/ICRA57147.2024.10610090
50. Gao, K., Zhu, D., Liu, Y., & Chen, J. (2024). A Visual Question Answering Model Based on Image Captioning. IOP Conference Series: Earth and Environmental Science, 1113(1), 012017. https://doi.org/10.1088/1755-1315/1113/1/012017 DOI: https://doi.org/10.1088/1755-1315/1113/1/012017
Downloads
Published
Issue
Section
License
Copyright (c) 2025 GlobeIS International Journal of Global Information Systems
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
