Lifelong Learning for Deep Neural Networks with Bayesian Principles

Nguyen, Cuong V.; Swaroop, Siddharth; Bui, Thang D.; Li, Yingzhen; Turner, Richard E.

Abstract

This chapter describes a general Bayesian framework for the lifelong learning of artificial neural networks that can handle catastrophic forgetting in a principled way. The framework can be applied to both discriminative and generative models as well as task-aware and task-agnostic settings. We introduce the variational continual learning algorithm, a realization of this framework that uses online variational inference with a small amount of memory or coreset for effective lifelong learning. We examine various practical considerations when using this algorithm and show that it performs competitively against other lifelong learning approaches on different benchmarks. We also discuss several improvements to the algorithm and outline some future research directions for Bayesian lifelong learning.

Citation

Nguyen, C. V., Swaroop, S., Bui, T. D., Li, Y., & Turner, R. E. (2024). Lifelong Learning for Deep Neural Networks with Bayesian Principles. In X. Li, S. Ramasamy, A. Ambikapathi, S. Sundaram, & H. M. Fayek (Eds.), Towards Human Brain Inspired Lifelong Learning (51-72). World Scientific Publishing. https://doi.org/10.1142/9789811286711_0004

• Online Publication Date: Apr 24, 2024
• Publication Date: 2024-05
• Deposit Date: Jun 3, 2024
• Publicly Available Date: Apr 25, 2025
• Publisher: World Scientific Publishing
• Pages: 51-72
• Book Title: Towards Human Brain Inspired Lifelong Learning
• Chapter Number: 4
• ISBN: 9789811286704
• DOI: https://doi.org/10.1142/9789811286711_0004
• Public URL: https://durham-repository.worktribe.com/output/2472032