Language as a latent sequence: Deep latent variable models for semi-supervised paraphrase generation

Yu, Jialin; Cristea, Alexandra I.; Harit, Anoushka; Sun, Zhongtian; Aduragba, Olanrewaju Tahir; Shi, Lei; Al Moubayed, Noura

Abstract

This paper explores deep latent variable models for semi-supervised paraphrase generation, where the missing target pair for unlabelled data is modelled as a latent paraphrase sequence. We present a novel unsupervised model named variational sequence auto-encoding reconstruction (VSAR), which performs latent sequence inference given an observed text. To leverage information from text pairs, we additionally introduce a novel supervised model we call dual directional learning (DDL), which is designed to integrate with our proposed VSAR model. Combining VSAR with DDL (DDL+VSAR) enables us to conduct semi-supervised learning. Still, the combined model suffers from a cold-start problem. To further combat this issue, we propose an improved weight initialisation solution, leading to a novel two-stage training scheme we call knowledge-reinforced-learning (KRL). Our empirical evaluations suggest that the combined model yields competitive performance against the state-ofthe-art supervised baselines on complete data. Furthermore, in scenarios where only a fraction of the labelled pairs are available, our combined model consistently outperforms the strong supervised model baseline (DDL) by a significant margin (𝑝 < .05; Wilcoxon test).

Citation

Yu, J., Cristea, A. I., Harit, A., Sun, Z., Aduragba, O. T., Shi, L., & Al Moubayed, N. (2023). Language as a latent sequence: Deep latent variable models for semi-supervised paraphrase generation. AI open, 4, 19-32. https://doi.org/10.1016/j.aiopen.2023.05.001

• Journal Article Type: Article
• Acceptance Date: May 18, 2023
• Online Publication Date: May 26, 2023
• Publication Date: 2023
• Deposit Date: May 30, 2023
• Publicly Available Date: May 30, 2023
• Journal: AI Open
• Electronic ISSN: 2666-6510
• Peer Reviewed: Peer Reviewed
• Volume: 4
• Pages: 19-32
• DOI: https://doi.org/10.1016/j.aiopen.2023.05.001
• Public URL: https://durham-repository.worktribe.com/output/1172628