A Hybrid Machine Learning Framework for Sentiment-Based Early Crisis Detection in News Streams
DOI:
https://doi.org/10.66395/globeis.21Keywords:
Crisis detection, sentiment analysis, news headlines, TF-IDF, MiniLM, Linear SVM, machine learningAbstract
The rapid growth of digital news sources has made the automatic analysis of news content increasingly important for the early identification of potential societal crises. In this study, an integrated approach is proposed to classify the sentiment of news headlines and to support the identification of potential crisis periods by monitoring temporal changes in the intensity of negative sentiment. News headlines from the News Category Dataset V3 were categorized into three sentiment classes—positive, neutral, and negative—using the VADER sentiment analysis method. For text representation, TF-IDF features, which capture the statistical importance of words, and MiniLM embeddings, which model semantic and contextual relationships, were employed. A hybrid representation was constructed by combining these two feature groups, and several machine learning models were comparatively evaluated on this representation. The best-performing model, a hybrid Linear SVM combining TF-IDF and MiniLM features, achieved 91.22% accuracy and a Macro-F1 score of 0.9099 in predicting the VADER-generated sentiment labels. Furthermore, when comparing VADER labels with independent human evaluations, the model achieved 86.67% accuracy, a Macro-F1 score of 0.8659, and a Cohen’s Kappa value of 0.7975. The resulting hybrid representation was then integrated into the proposed temporal crisis assessment framework, enabling sentiment classification outputs to monitor unusual changes in negative sentiment over time. The proposed approach not only classifies news headlines by sentiment but also enables analysis of temporal changes in the proportion of negative sentiment. By integrating the statistical strength of TF-IDF with the contextual representation capability of MiniLM, the proposed framework provides a high-performing, lightweight, and practical solution. It is anticipated that this study will contribute to decision support systems for monitoring news streams and facilitating the early assessment of potential crisis trends.
Downloads
References
1. Hamza, A., Muhammad, A., Abbas, M. S., & Jan, S. U. (2026). Contextual deep learning for accurate news article categorization with pre-trained embeddings. Scientific Reports, 16(1). https://doi.org/10.1038/s41598-026-38998-3 DOI: https://doi.org/10.1038/s41598-026-38998-3
2. Zare, M. (2025). Attention-guided feature fusion (AGFF) model for integrating statistical and semantic features in news text classification (arXiv:2511.17184). arXiv. https://doi.org/10.48550/arXiv.2511.17184
3. Aprilio, P., Felix, M., Nugraha, P. S., & Fahmi, H. (2025). Hybrid feature combination of TF-IDF and BERT for enhanced information retrieval accuracy. JISA (Jurnal Informatika dan Sains), 8(1), 8–15. https://doi.org/10.31326/jisa.v8i1.2179 DOI: https://doi.org/10.31326/jisa.v8i1.2179
4. Zhang, G., & Hu, J. (2024). Enhanced industrial text classification via hyper variational graph-guided global context integration. PeerJ Computer Science, 10, Article e1788. https://doi.org/10.7717/peerj-cs.1788 DOI: https://doi.org/10.7717/peerj-cs.1788
5. Chhetri, G., Das, S., & Chowdhury, T. I. (2025). WISE: Web information satire and fakeness evaluation (arXiv:2512.24000). arXiv. https://doi.org/10.48550/arXiv.2512.24000 DOI: https://doi.org/10.1145/3779211.3793174
6. Zhang, W., Deng, Y., Liu, B., Pan, S. J., & Bing, L. (2024). Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 3881–3906). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-naacl.246 DOI: https://doi.org/10.18653/v1/2024.findings-naacl.246
7. Kuila, A., & Sarkar, S. (2024). Deciphering political entity sentiment in news with large language models: Zero-shot and few-shot strategies. In Proceedings of the 1st Workshop on Natural Language Processing for Political Sciences (PoliticalNLP 2024). DOI: https://doi.org/10.63317/3mb6cdyeboum
8. Dharrao, D., MR, A., Mital, R., Vengali, A., Pangavhane, M., Rajput, S., & Bongale, A. M. (2024). An efficient method for disaster tweets classification using gradient-based optimized convolutional neural networks with BERT embeddings. MethodsX, 13, Article 102843. https://doi.org/10.1016/j.mex.2024.102843 DOI: https://doi.org/10.1016/j.mex.2024.102843
9. Yicheng, S., Louw, S. A., Thant, Y. M., Arya, A., Konduru, R. T., & Avtar, R. (2026). BART-Disaster-X: An effective model to extract information from disaster-related tweets on X. Remote Sensing Applications: Society and Environment, Article 102174. https://doi.org/10.1016/j.rsase.2026.102174 DOI: https://doi.org/10.1016/j.rsase.2026.102174
10. Jahan, I., Islam, M. N., Hasan, M. M., & Siddiky, M. R. (2024). Comparative analysis of machine learning algorithms for sentiment classification in social media text. World Journal of Advanced Research and Reviews, 23(3), 2842–2852. https://doi.org/10.30574/wjarr.2024.23.3.2983 DOI: https://doi.org/10.30574/wjarr.2024.23.3.2983
11. Peng, L.-J., Shao, X.-G., & Huang, W.-M. (2021). Research on the early-warning model of network public opinion of major emergencies. IEEE Access, 9, 44162–44172. https://doi.org/10.1109/ACCESS.2021.3066242 DOI: https://doi.org/10.1109/ACCESS.2021.3066242
12. Dui, Y., & Hu, H. (2024). Social media public opinion detection using multimodal natural language processing and attention mechanisms. IET Information Security, 2024(1). https://doi.org/10.1049/2024/8880804 DOI: https://doi.org/10.1049/2024/8880804
13. Kahraman, İ., & Önder, E. (2026). A crisis that began with a tweet: Temporal and thematic analysis of social media sentiment dynamics in financial collapse processes. Yalova Sosyal Bilimler Dergisi, 16(1), 88–110. https://doi.org/10.17828/yalovasosbil.1785839
14. Kırcı, P., & Gülbak, E. (2020). Instagram verileri ile duygu analizi [Sentiment analysis with Instagram data]. Avrupa Bilim ve Teknoloji Dergisi (EJOSAT), 360–364. https://doi.org/10.31590/ejosat.780129 DOI: https://doi.org/10.31590/ejosat.780129
15. Çılgın, C., Baş, M., Bilgehan, H., & Ünal, C. (2022). Twitter sentiment analysis during Covid-19 outbreak with VADER. AJIT-e: Academic Journal of Information Technology, 13(49), 72–89. https://doi.org/10.5824/ajite.2022.02.001.x DOI: https://doi.org/10.5824/ajite.2022.02.001.x
16. Kınık, D., & Güran, A. (2021). TF-IDF ve Doc2Vec tabanlı Türkçe metin sınıflandırma sisteminin başarım değerinin ardışık kelime grubu tespiti ile arttırılması [Improving the performance of a TF-IDF and Doc2Vec-based Turkish text classification system through consecutive word group detection]. Avrupa Bilim ve Teknoloji Dergisi (EJOSAT), (21), 323–332. https://doi.org/10.31590/ejosat.774144 DOI: https://doi.org/10.31590/ejosat.774144
17. Çelik, Ö., & Koç, B. C. (2021). TF-IDF, Word2vec ve Fasttext vektör model yöntemleri ile Türkçe haber metinlerinin sınıflandırılması [Classification of Turkish news texts using TF-IDF, Word2vec and FastText vector model methods]. Dokuz Eylül Üniversitesi Mühendislik Fakültesi Fen ve Mühendislik Dergisi, 23(67), 121–127. https://doi.org/10.21205/deufmd.2021236710 DOI: https://doi.org/10.21205/deufmd.2021236710
18. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 4656–4667). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.372 DOI: https://doi.org/10.18653/v1/2020.findings-emnlp.372
19. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982–3992). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1410 DOI: https://doi.org/10.18653/v1/D19-1410
20. Hassan, S. U., Ahamed, J., & Ahmad, K. (2022). Analytics of machine learning-based algorithms for text classification. Sustainable Operations and Computers, 3, 238–248. https://doi.org/10.1016/j.susoc.2022.03.001 DOI: https://doi.org/10.1016/j.susoc.2022.03.001
21. Yang, J. (2026). Methods of improving sentiment analysis based on sentiment dictionary. Franklin Open, 15, Article 100616. https://doi.org/10.1016/j.fraope.2026.100616 DOI: https://doi.org/10.1016/j.fraope.2026.100616
22. Moraes, R., Valiati, J. F., & Gavião Neto, W. P. (2013). Document-level sentiment classification: An empirical comparison between SVM and ANN. Expert Systems with Applications, 40(2), 621–633. https://doi.org/10.1016/j.eswa.2012.07.059 DOI: https://doi.org/10.1016/j.eswa.2012.07.059
23. Aburomman, A. A., & Reaz, M. B. I. (2016). A novel SVM-kNN-PSO ensemble method for intrusion detection system. Applied Soft Computing, 38, 360–372. https://doi.org/10.1016/j.asoc.2015.10.011 DOI: https://doi.org/10.1016/j.asoc.2015.10.011
24. Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427–437. https://doi.org/10.1016/j.ipm.2009.03.002 DOI: https://doi.org/10.1016/j.ipm.2009.03.002
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Sena Nur ÇULCU, Canan KOÇ (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
