Umit Demirbaga
Federated-ANN based Critical Path Analysis and Health Recommendations for MapReduce Workflows in Consumer Electronics Applications
Demirbaga, Umit; Aujla, Gagangeet Singh
Abstract
Although much research has been done to improve the performance of big data systems, predicting the performance degradation of these systems quickly and efficiently remains a significant challenge. Unfortunately, the complexity of big data systems is so vast that predicting performance degradation ahead of time is quite tricky. Long execution time is often discussed in the context of performance degradation of big data systems. This paper proposes MrPath, a Federated AI-based critical path analysis approach for holistic performance prediction of MapReduce workflows for consumer electronics applications while enabling root-cause analysis of various types of faults. We have implemented a federated artificial neural network (FANN) to predict the critical path in a MapReduce workflow. After the critical path components (e.g., mapper1, reducer2) are predicted/detected, root cause analysis uses user-defined functions (UDF) to pinpoint the most likely reasons for the observed performance problems. Finally, health node classification is performed using an ANN-based Self-Organising Map (SOM). The results show that the AI-based critical path analysis method can significantly illuminate the reasons behind the long execution time in big data systems.
Citation
Demirbaga, U., & Aujla, G. S. (2023). Federated-ANN based Critical Path Analysis and Health Recommendations for MapReduce Workflows in Consumer Electronics Applications. IEEE Transactions on Consumer Electronics, 1-1. https://doi.org/10.1109/tce.2023.3318813
Journal Article Type | Article |
---|---|
Acceptance Date | Sep 21, 2023 |
Online Publication Date | Sep 25, 2023 |
Publication Date | 2023 |
Deposit Date | Nov 2, 2023 |
Publicly Available Date | Nov 3, 2023 |
Journal | IEEE Transactions on Consumer Electronics |
Print ISSN | 0098-3063 |
Electronic ISSN | 1558-4127 |
Publisher | Institute of Electrical and Electronics Engineers |
Peer Reviewed | Peer Reviewed |
Pages | 1-1 |
DOI | https://doi.org/10.1109/tce.2023.3318813 |
Keywords | Electrical and Electronic Engineering; Media Technology |
Public URL | https://durham-repository.worktribe.com/output/1875624 |
Files
Author Accepted Manuscript
(1 Mb)
PDF
Licence
http://creativecommons.org/licenses/by/4.0/
Copyright Statement
This accepted manuscript is licensed under the Creative Commons Attribution licence.
You might also like
Software Defined Internet of Everything
(2022)
Book
Edge Computing Based Resource Supplementation for Software Defined Vehicular Networks
(2022)
Presentation / Conference Contribution
A Federated Leaning Perspective for Intelligent Data Communication Framework in IoT Ecosystem
(2022)
Presentation / Conference Contribution
Downloadable Citations
About Durham Research Online (DRO)
Administrator e-mail: dro.admin@durham.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search