Few-Shot Image and Sentence Matching via Gated Visual-Semantic Embedding

Huang, Yan; Long, Yang; Wang, Liang

Abstract

Word similarity and word relatedness are fundamental to natural language processing and more generally, understanding how humans relate concepts in semantic memory. A growing number of datasets are being proposed as evaluation benchmarks,however, the heterogeneity and focus of each respective dataset makes it difficult to draw plausible conclusions as to how a unified semantic model would perform. Additionally, we want to identify the transferability of knowledge obtained from one task to another, within the same domain and across domains. Hence, this paper first presents an evaluation and comparison of eight chosen datasets tested using the best performing regression models. As a baseline, we present regression models that incorporate both lexical featuresand word embeddings to produce consistent and competitive results compared to the state of the art.We present our main contribution, the best performing model across seven of the eight datasets - a Gated Recurrent Siamese Networkthat learns relationships between lexical word definitions.A parameter transfer learning strategy is employed for theSiamese Network. Subsequently, we present a secondary contribution which is the best performing non-sequential model:an Inductive and Transductive Transfer Learning strategy fortransferring decision trees within a Random Forest to a target task that is learned from only few instances. The method involves measuring semantic distance between hidden factored matrix representations of decision tree traversal matrices.

Citation

Huang, Y., Long, Y., & Wang, L. (2019, December). Few-Shot Image and Sentence Matching via Gated Visual-Semantic Embedding. Presented at Thirty-Second AAAI Conference on Artificial Intelligence

• Presentation Conference Type: Conference Paper (published)
• Conference Name: Thirty-Second AAAI Conference on Artificial Intelligence
• Acceptance Date: Oct 31, 2018
• Online Publication Date: Apr 27, 2019
• Publication Date: 2019
• Deposit Date: Sep 1, 2019
• Pages: 5342-5349
• Series ISSN: 2374-3468
• Book Title: Thirty-Second AAAI Conference on Artificial Intelligence ; proceedings.
• Public URL: https://durham-repository.worktribe.com/output/1142112
• Publisher URL: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16981