Skip to main content

Research Repository

Advanced Search

Dynamic Unary Convolution in Transformers

Duan, Haoran; Long, Yang; Wang, Shidong; Zhang, Haofeng; Willcocks, Chris G.; Shao, Ling

Dynamic Unary Convolution in Transformers Thumbnail


Haoran Duan
PGR Student Doctor of Philosophy

Shidong Wang

Haofeng Zhang

Ling Shao


It is uncertain whether the power of transformer architectures can complement existing convolutional neural networks. A few recent attempts have combined convolution with transformer design through a range of structures in series, where the main contribution of this paper is to explore a parallel design approach. While previous transformed-based approaches need to segment the image into patch-wise tokens, we observe that the multi-head self-attention conducted on convolutional features is mainly sensitive to global correlations and that the performance degrades when these correlations are not exhibited. We propose two parallel modules along with multi-head self-attention to enhance the transformer. For local information, a dynamic local enhancement module leverages convolution to dynamically and explicitly enhance positive local patches and suppress the response to less informative ones. For mid-level structure, a novel unary co-occurrence excitation module utilizes convolution to actively search the local co-occurrence between patches. The parallel-designed Dynamic Unary Convolution in Transformer (DUCT) blocks are aggregated into a deep architecture, which is comprehensively evaluated across essential computer vision tasks in image-based classification, segmentation, retrieval and density estimation. Both qualitative and quantitative results show our parallel convolutional-transformer approach with dynamic and unary convolution outperforms existing series-designed structures.


Duan, H., Long, Y., Wang, S., Zhang, H., Willcocks, C. G., & Shao, L. (2023). Dynamic Unary Convolution in Transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 12747 - 12759.

Journal Article Type Article
Acceptance Date Jan 1, 2023
Online Publication Date Jan 2, 2023
Publication Date Nov 1, 2023
Deposit Date Jan 16, 2023
Publicly Available Date Jan 16, 2023
Journal IEEE Transactions on Pattern Analysis and Machine Intelligence
Print ISSN 0162-8828
Publisher Institute of Electrical and Electronics Engineers
Peer Reviewed Peer Reviewed
Volume 45
Issue 11
Pages 12747 - 12759
Public URL


Accepted Journal Article (14.6 Mb)

Copyright Statement
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

You might also like

Downloadable Citations