Federated-ANN-Based Critical Path Analysis and Health Recommendations for MapReduce Workflows in Consumer Electronics Applications

Demirbaga, Umit; Aujla, Gagangeet Singh

Abstract

Although much research has been done to improve the performance of big data systems, predicting the performance degradation of these systems quickly and efficiently remains a significant challenge. Unfortunately, the complexity of big data systems is so vast that predicting performance degradation ahead of time is quite tricky. Long execution time is often discussed in the context of performance degradation of big data systems. This paper proposes MrPath, a Federated AI-based critical path analysis approach for holistic performance prediction of MapReduce workflows for consumer electronics applications while enabling root-cause analysis of various types of faults. We have implemented a federated artificial neural network (FANN) to predict the critical path in a MapReduce workflow. After the critical path components (e.g., mapper1, reducer2) are predicted/detected, root cause analysis uses user-defined functions (UDF) to pinpoint the most likely reasons for the observed performance problems. Finally, health node classification is performed using an ANN-based Self-Organising Map (SOM). The results show that the AI-based critical path analysis method can significantly illuminate the reasons behind the long execution time in big data systems.

Citation

Demirbaga, U., & Aujla, G. S. (2024). Federated-ANN-Based Critical Path Analysis and Health Recommendations for MapReduce Workflows in Consumer Electronics Applications. IEEE Transactions on Consumer Electronics, 70(1), 2639-2647. https://doi.org/10.1109/tce.2023.3318813

• Journal Article Type: Article
• Acceptance Date: Sep 21, 2023
• Online Publication Date: Sep 25, 2023
• Publication Date: 2024-02
• Deposit Date: Nov 2, 2023
• Publicly Available Date: Nov 3, 2023
• Journal: IEEE Transactions on Consumer Electronics
• Print ISSN: 0098-3063
• Electronic ISSN: 1558-4127
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Volume: 70
• Issue: 1
• Pages: 2639-2647
• DOI: https://doi.org/10.1109/tce.2023.3318813
• Keywords: Electrical and Electronic Engineering; Media Technology
• Public URL: https://durham-repository.worktribe.com/output/1875624