Energy-based Predictive Root Cause Analysis for Real-Time Anomaly Detection in Big Data Systems

Demirbaga, Umit; Singh Aujla, Gagangeet; Sun, Hongjian

Abstract

As the scale of data continues to grow exponentially, managing resource allocation and energy consumption in big data systems becomes increasingly complex and critical. Moreover, with big data systems, energy efficiency is more important daily. In cloud environments, it can be the determining factor between reduced costs and lowered environmental damage. This paper presents a deep learning-based framework for accurately predicting instant energy consumption in real-time and detecting anomalies of different sizes in big data clusters. We use SmartMonit to gather task execution and real-time infrastructure data. A Feedforward Neural Network (FNN) predicts energy consumption from CPU utilisation, memory usage, and task profiling research. The system will track any deviation from predicted consumption with root cause analysis (RCA) if there are significant anomalies. We also integrate an Autoencoder to identify straggler tasks and inefficient resource utilisation. User-defined functions are next applied to examine these anomalies and try to detect the underlying reasons, like distributed data processing, locality of computation exploitation, or resource waste. Given the scale and heterogeneity of big data workloads, the system's ability to dynamically adjust and optimise resource usage is essential for handling complex processing tasks. The experimental results prove that the proposed system effectively enhances resource allocation and decreases wasted energy.

Citation

Demirbaga, U., Singh Aujla, G., & Sun, H. (2025, June). Energy-based Predictive Root Cause Analysis for Real-Time Anomaly Detection in Big Data Systems. Presented at 2025 IEEE International Conference on Communications (ICC), Montreal, Canada

• Presentation Conference Type: Edited Proceedings
• Conference Name: 2025 IEEE International Conference on Communications (ICC)
• Start Date: Jun 8, 2025
• End Date: Jun 12, 2025
• Acceptance Date: Jan 17, 2025
• Deposit Date: Jan 18, 2025
• Peer Reviewed: Peer Reviewed
• Keywords: Index Terms-Energy efficiency; Cloud computing; Predictive analytics; Big Data; Anomaly detection
• Public URL: https://durham-repository.worktribe.com/output/3342090
• Publisher URL: https://icc2025.ieee-icc.org/