Dynamic Unary Convolution in Transformers

Duan, Haoran; Long, Yang; Wang, Shidong; Zhang, Haofeng; Willcocks, Chris G.; Shao, Ling

Abstract

It is uncertain whether the power of transformer architectures can complement existing convolutional neural networks. A few recent attempts have combined convolution with transformer design through a range of structures in series, where the main contribution of this paper is to explore a parallel design approach. While previous transformed-based approaches need to segment the image into patch-wise tokens, we observe that the multi-head self-attention conducted on convolutional features is mainly sensitive to global correlations and that the performance degrades when these correlations are not exhibited. We propose two parallel modules along with multi-head self-attention to enhance the transformer. For local information, a dynamic local enhancement module leverages convolution to dynamically and explicitly enhance positive local patches and suppress the response to less informative ones. For mid-level structure, a novel unary co-occurrence excitation module utilizes convolution to actively search the local co-occurrence between patches. The parallel-designed Dynamic Unary Convolution in Transformer (DUCT) blocks are aggregated into a deep architecture, which is comprehensively evaluated across essential computer vision tasks in image-based classification, segmentation, retrieval and density estimation. Both qualitative and quantitative results show our parallel convolutional-transformer approach with dynamic and unary convolution outperforms existing series-designed structures.

Citation

Duan, H., Long, Y., Wang, S., Zhang, H., Willcocks, C. G., & Shao, L. (2023). Dynamic Unary Convolution in Transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 12747 - 12759. https://doi.org/10.1109/tpami.2022.3233482

• Journal Article Type: Article
• Acceptance Date: Jan 1, 2023
• Online Publication Date: Jan 2, 2023
• Publication Date: Nov 1, 2023
• Deposit Date: Jan 16, 2023
• Publicly Available Date: Jan 16, 2023
• Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence
• Print ISSN: 0162-8828
• Electronic ISSN: 1939-3539
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Volume: 45
• Issue: 11
• Pages: 12747 - 12759
• DOI: https://doi.org/10.1109/tpami.2022.3233482
• Public URL: https://durham-repository.worktribe.com/output/1185317