Addressing Performance Inconsistency in Domain Generalization for Image Classification

Stirling, Jamie; Moubayed, Noura Al

Abstract

Domain Generalization (DG) in computer vision aims to replicate the human ability to generalize well under a shift of data distribution, or domain. In recent years, the field of domain generalization has seen a steady increase in average left-out test accuracy, measured as the average test accuracy achieved when each domain (in turn) is left out of training and used only for testing. To date, average left-out test accuracy is the only metric used for evaluating and comparing different techniques in DG. We observe that despite the steady increase in average left-out test accuracy, there remains a vast inconsistency between the left-out test accuracy scores measured for individual domains. To the best of our knowledge, this domain inconsistency persists across all published DG methods to date. In this work, we argue that domain generalization cannot be said to be successful without substantially reducing this performance inconsistency between domains. We propose a formal metric for measuring domain inconsistency and apply it to results in the literature. We run experiments to explore how alternative choices of pretraining affects domain inconsistency, finding that, in some settings, careful choice of pretraining can improve consistency with minimal negative (and sometimes positive) impact on average left-out test accuracy. Finally we discuss other potential sources of domain inconsistency and limitations of our work.

Citation

Stirling, J., & Moubayed, N. A. (2023). Addressing Performance Inconsistency in Domain Generalization for Image Classification. In 2023 International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/ijcnn54540.2023.10191685

• Conference Name: 2023 International Joint Conference on Neural Networks (IJCNN)
• Conference Location: Gold Coast, Australia
• Start Date: Jun 18, 2023
• End Date: Jun 23, 2023
• Acceptance Date: Jun 1, 2023
• Publication Date: Aug 2, 2023
• Deposit Date: Aug 29, 2023
• Publisher: Institute of Electrical and Electronics Engineers
• Series ISSN: 2161-4393
• Book Title: 2023 International Joint Conference on Neural Networks (IJCNN)
• ISBN: 9781665488686
• DOI: https://doi.org/10.1109/ijcnn54540.2023.10191685
• Public URL: https://durham-repository.worktribe.com/output/1726324