RootPath: Root Cause and Critical Path Analysis to Ensure Sustainable and Resilient Consumer-Centric Big Data Processing under Fault Scenarios

Demirbaga, Umit; Aujla, Gagangeet Singh

Abstract

The exponential growth of consumer-centric big data has led to increased concerns regarding the sustainability and resilience of data processing systems, particularly in the face of fault scenarios. This paper presents an innovative approach integrating Root Cause Analysis (RCA) and Critical Path Analysis (CPA) to address these challenges and ensure sustainable, resilient consumer-centric big data processing. The proposed methodology enables the identification of root causes behind system faults probabilistically, implementing Bayesian networks. Furthermore, an Artificial Neural Network (ANN)-based critical path method is employed to identify the critical path that causes high makespan in MapReduce workflows to enhance fault tolerance and optimize resource allocation. To evaluate the effectiveness of the proposed methodology, we conduct a series of fault injection experiments, simulating various real-world fault scenarios commonly encountered in operational environments. The experiment results show that both models perform very well with high accuracies, 95%, and 98%, respectively, enabling the development of more robust and reliable consumer-centric systems.

• Journal Article Type: Article
• Acceptance Date: Oct 28, 2023
• Online Publication Date: Nov 2, 2023
• Publication Date: 2023
• Deposit Date: Nov 5, 2023
• Publicly Available Date: Nov 7, 2023
• Journal: IEEE Transactions on Consumer Electronics
• Print ISSN: 0098-3063
• Electronic ISSN: 1558-4127
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• DOI: https://doi.org/10.1109/tce.2023.3329545
• Public URL: https://durham-repository.worktribe.com/output/1897791