Juan Marcelo Parra-Ullauri
Event-driven Temporal Models for Explanations - ETeMoX: Explaining Reinforcement Learning
Parra-Ullauri, Juan Marcelo; Garcıa-Domınguez, Antonio; Bencomo, Nelly; Zheng, Changgang; Zhen, Chen; Boubeta-Puig, Juan; Ortiz, Guadalupe; Yang, Shufan
Authors
Antonio Garcıa-Domınguez
Dr Nelly Bencomo nelly.bencomo@durham.ac.uk
Associate Professor
Changgang Zheng
Chen Zhen
Juan Boubeta-Puig
Guadalupe Ortiz
Shufan Yang
Abstract
Modern software systems are increasingly expected to show higher degrees of autonomy and self-management to cope with uncertain and diverse situations. As a consequence, autonomous systems can exhibit unexpected and surprising behaviours. This is exacerbated due to the ubiquity and complexity of Artificial Intelligence (AI)-based systems. This is the case of Reinforcement Learning (RL), where autonomous agents learn through trial-and-error how to find good solutions to a problem. Thus, the underlying decision-making criteria may become opaque to users that interact with the system and who may require explanations about the system’s reasoning. Available work for eXplainable Reinforcement Learning (XRL) offers different trade-offs: e.g. for runtime explanations, the approaches are model-specific or can only analyse results after-the-fact. Different from these approaches, this paper aims to provide an online model-agnostic approach for XRL towards trustworthy and understandable AI. We present ETeMoX, an architecture based on temporal models to keep track of the decision-making processes of RL systems. In cases where the resources are limited (e.g. storage capacity or time to response), the architecture also integrates complex event processing, an event-driven approach, for detecting matches to event patterns that need to be stored, instead of keeping the entire history. The approach is applied to a mobile communications case study that uses RL for its decision-making. In order to test the generalisability of our approach, three variants of the underlying RL algorithms are used: Q-Learning, SARSA and DQN. The encouraging results show that using the proposed configurable architecture, RL developers are able to obtain explanations about the evolution of a metric, relationships between metrics, and were able to track situations of interest happening over time windows.
Citation
Parra-Ullauri, J. M., Garcıa-Domınguez, A., Bencomo, N., Zheng, C., Zhen, C., Boubeta-Puig, J., …Yang, S. (2022). Event-driven Temporal Models for Explanations - ETeMoX: Explaining Reinforcement Learning. Software and Systems Modeling, 21(3), 1091-1113. https://doi.org/10.1007/s10270-021-00952-4
Journal Article Type | Article |
---|---|
Acceptance Date | Nov 4, 2021 |
Online Publication Date | Dec 18, 2021 |
Publication Date | 2022-06 |
Deposit Date | Dec 3, 2021 |
Publicly Available Date | Jan 17, 2022 |
Journal | Software and Systems Modeling |
Print ISSN | 1619-1366 |
Electronic ISSN | 1619-1374 |
Publisher | Springer |
Peer Reviewed | Peer Reviewed |
Volume | 21 |
Issue | 3 |
Pages | 1091-1113 |
DOI | https://doi.org/10.1007/s10270-021-00952-4 |
Public URL | https://durham-repository.worktribe.com/output/1220141 |
Files
Published Journal Article
(2.7 Mb)
PDF
Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/
Copyright Statement
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
You might also like
Software Engineering for Self-Adaptive Systems: Research Challenges in the Provision of Assurances
(2013)
Presentation / Conference Contribution
Towards History-Aware Self-Adaptation with Explanation Capabilities
(2019)
Presentation / Conference Contribution
RaM: Causally-Connected and Requirements-Aware Runtime Models using Bayesian Learning
(2019)
Presentation / Conference Contribution
Temporal Models for History-Aware Explainability
(2020)
Presentation / Conference Contribution
RDMSim: An Exemplar for Evaluation and Comparison of Decision-Making Techniques for Self-Adaptation
(2021)
Presentation / Conference Contribution
Downloadable Citations
About Durham Research Online (DRO)
Administrator e-mail: dro.admin@durham.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search