\section{Introduction}
\label{sec:intro}

Video Object Detection~(VOD) aims to localize and classify objects in a series of subsequent video frames. Recent efforts in video object detection demonstrate that exploiting feature aggregation of temporal information~\cite{zhu2017deep, zhu2017flow, feichtenhofer2017detect, wang2018fully, wu2019sequence, han2020mining, gong2021temporal, han2021class, cui2021tf, hua2021temporal, han2020exploiting} produce superior performance than leveraging temporal information at the post-processing stage~\cite{han2016seq, kang2016object, kang2017object, sabater2020robust}. The former approaches mainly enhance the target frame feature representation through aggregating features from neighbouring frames or an entire video clip by designing a specific module, thereby boosting detection results.~The majority of these works~\cite{wu2019sequence, hua2021temporal, jiang2020learning, han2020mining, shvets2019leveraging} employ two-stage detectors such as Faster R-CNN~\cite{ren2015faster} or R-FCN~\cite{dai2016r} to design their VOD pipelines.

\begin{figure}
\includegraphics[width=12.5cm]{images/problem_def_new.pdf}
\caption{Comparison between previous region-based and the proposed video object detection methods. Subplot (a): Despite the temporal feature aggregation from support frames \textit{t-s} and \textit{t+s} in existing proposal-based approaches, the detector overlooks \textit{Turtle} in \textcolor{violet}{pink} on a low-quality target frame \textit{t} due to unreliable region proposals. Subplot (b): To tackle such limitations, we propose a novel SparseVOD framework that iteratively refines region proposals through leveraging spatio-temporal feature aggregation prior to final detection. The Temporal Dynamic Head is explained in Fig.~\ref{fig:pipeline}.}
\label{fig:problem_definition}
\vspace{-15pt}
\end{figure}

Albeit the enormous success, it is important to highlight that the temporal feature aggregation scheme of all prior region-based VOD methods~\cite{wu2019sequence, han2021class, han2020mining, hua2021temporal, jiang2020learning, cui2021tf} heavily relies on object proposals from RPN~\cite{ren2015faster} trained without any temporal information. Consequently, these methods suffer from several underlying restrictions. First, they require an additional step of NMS~\cite{neubeck2006efficient} at the beginning to perform dense-to-sparse matching of hand-crafted anchors, making the overall VOD pipeline not end-to-end optimizable. Second, on low-quality frames (appearance deterioration), the generated object proposals are unreliable, leading to ineffective temporal feature aggregation.
Fig.~\ref{fig:problem_definition}\textcolor{red}{(a)} illustrates that although the detector leverages spatio-temporal information from support frames \textit{t-s} and \textit{t+s}, it fails to detect \textit{Turtle} at the target frame \textit{t}. The main reason for such missed prediction is that generated proposals from single-frame RPN overlooks \textit{Turtle} in the target frame \textit{t}. This corrupts object features during instance-level feature aggregation.
The third restriction is that these methods require several support frames to calibrate proposal feature representation for the target frame, decreasing the run time performance. Fourth, since the RPN in these methods is optimized on a single IoU level (generally IoU=0.5), they struggle to provide high-quality detections (IoU > 0.5) despite producing impressive performance on a lower IoU threshold. Moreover, these methods necessitate complex post-processing methods~\cite{chen2018optimizing, kang2017t} or additional proposal classifier networks~\cite{han2021class, han2020exploiting} to accomplish state-of-the-art performance.

To tackle the aforementioned challenges, we propose SparseVOD, an end-to-end trainable framework that exploits temporal information to learn sparse (merely 100) object proposals for VOD.
The SparseVOD employs the recently introduced Sparse R-CNN~\cite{sun2021sparse} that has shown impressive performance by eliminating the need for dense priors enumerating over frames and alleviating the interaction between object queries and dense frame features. In particular, motivated from~\cite{wu2019sequence, gong2021temporal}, we incorporate a Temporal (Region of Interest) RoI Feature Extraction (TFE) head that replaces a single image RoI extractor in the dynamic instance interactive head in~\cite{sun2021sparse}.~Furthermore, inspired by~\cite{wu2019sequence}, we fuse attention guided Semantic Proposal Feature Aggregation (SPFA) module that enhances the feature representation of object proposals in target frames through semantic level sequence aggregation from support frames.

Contrary to most prior VOD works, our SparseVOD operates on a sparse-in sparse-out matching scheme.~This not only eliminates components like post-processing and NMS but also enables faster training network convergence without pre-training the detector as done in~\cite{he2021end}.~Furthermore, thanks to the spatio-temporal learning, iterative refinement of object proposals lead to successful predictions even on a low-quality target frame \textit{t} as shown in Fig.~\ref{fig:problem_definition}\textcolor{red}{(b)}.~Moreover, our spatio-temporal proposal learning alleviates the need for several support frames in a video and brings significant performance gains on increasing IoU thresholds (see Fig.~\ref{fig:high_quality_AP}).

Herein, our main contributions are as follows.
(1) We propose SparseVOD, a novel end-to-end trainable video object detection method. To our knowledge, \textit{this is the first work that exploits temporal information to learn object proposals for video object detection}. (2) We extend the design of Sparse R-CNN~\cite{sun2021sparse} by introducing a Temporal Feature Extraction (TFE) module that leverages temporal information to extract RoI proposal features.~Furthermore, we fuse Semantic Proposal Feature Aggregation (SPFA) in~\cite{sun2021sparse} to enhance object feature representation before final detection inspired by~\cite{wu2019sequence, gong2021temporal}.
(3) By introducing the proposed TFE and SPFA modules in~\cite{sun2021sparse}, our SparseVOD improves the baseline by far (5-6\% mAP). Without bells and whistles, our SparseVOD achieves the \textit{new best mAP of 80.3\% on the ImageNet VID benchmark using ResNet-50 as the backbone}.~Moreover, it surpasses prior state-of-the-art methods by far in terms of high-quality detections (higher IoU thresholds, see Fig.~\ref{fig:high_quality_AP}) and achieves optimal speed-accuracy tradeoff (Fig.~\ref{fig:tradeoff}).

\section{Related Work}
\label{sec:related_work}

\noindent \textbf{Proposal Learning for Image Object Detection.}
\hspace{0.8pt}
\label{subsec:image_object_detection}
Ren \emph{et al}\bmvaOneDot~\cite{ren2015faster} introduce the Region Proposal Network (RPN) in Faster R-CNN to predict object proposals. The RPN consists of a small fully convolutional network~\cite{long2015fully} that receives an anchor as an input, classifies it as an object or background, and performs box regression. This design is widely incorporated in later two-stage approaches~\cite{dai2016r, he2017mask, lin2017feature, cai2018cascade}.~MetaAnchor~\cite{yang2018metaanchor} proposes to exploit meta-learning to generate anchors dynamically. Cascade RPN~\cite{vu2019cascade} improves the object proposal quality of the conventional RPN through multi-stage refinement and adaptive convolution. Recently, Sparse R-CNN~\cite{sun2021sparse} introduces a sparse-in sparse-out paradigm that simplifies the sophisticated two-stage object detection pipeline by alleviating complex components such as Non-Maximum Suppression (NMS)~\cite{neubeck2006efficient} and dense priors.~Following a similar line of work, this paper proposes spatio-temporal learnable proposals to simplify the video object detection pipeline.

\vspace{3pt}
\noindent \textbf{Exploiting Temporal Information in Video Object Detection.}
\hspace{0.8pt}
\label{subsec:vide_object_detection}
Exploiting temporal information from other frames in a video is a natural choice to tackle the challenges of video object detection, and our work derives from the same idea. Existing approaches leveraging temporal information mainly follow one of the two directions.~The first line of works~\cite{han2016seq, kang2016object, kang2017object, sabater2020robust} mainly employ temporal information to make still-image detection results more coherent and stable. The performance of such VOD methods is sub-optimal because they are not end-to-end trainable and heavily rely on the capabilities of the initial still-image detector. On the contrary, the other direction of methods~\cite{zhu2017deep, zhu2017flow, feichtenhofer2017detect, wang2018fully, jiang2019video, wu2019sequence, han2020mining, gong2021temporal, han2021class, cui2021tf, hua2021temporal, han2020exploiting} utilizes temporal information during the course of training.~Earlier works in this category adopt FlowNet~\cite{dosovitskiy2015flownet} to propagate warp features across frames~\cite{zhu2017flow, wang2018fully, zhu2017deep, kang2017t}. However, temporal exploitation of optical flow-based works is limited to neighbouring frames, yielding inferior performance in occlusions. PSLA~\cite{guo2019progressive} proposes to learn the spatial correspondence between neighbouring frames by employing the progressive sparser stride. All these methods can only capitalize temporal information from a small number of nearby frames to refine the target frame features.~Alternatively, global feature aggregation methods~\cite{wu2019sequence, deng2019object, shvets2019leveraging} have been proposed to utilize long-term semantic information.~Recent VOD methods adopt this aggregation scheme and propose blending of temporal features~\cite{cui2021tf}, class-aware feature aggregation~\cite{han2021class, han2020exploiting}, temporal RoIAlign~\cite{gong2021temporal}, and temporal meta-adaptor~\cite{hua2021temporal} to achieve state-of-the-art results. Although these methods produce superior performance from prior efforts, their feature aggregation rely on object proposals generated without temporal information.

\vspace{3pt}
\noindent \textbf{Refining Object Proposals in Video Object Detection.}
\label{subsec:object_proposals}
\hspace{0.8pt}
Recent efforts have shown that enhancing object proposal features can alleviate the obstacles of object confusion in videos~\cite{shvets2019leveraging, han2020mining, jiang2020learning}. Shvets~\emph{et al}\bmvaOneDot~\cite{shvets2019leveraging} refine the proposal for the target frame by learning similarities between proposals from different frames. LSTS~\cite{jiang2020learning} models the spatio-temporal correspondence to alleviate misalignment before aggregating features from different frames.~Han~\emph{et al}\bmvaOneDot~\cite{han2020mining} propose integrating inter-video and intra-video proposals to boost target proposal features. \textit{Despite the promising improvements, the effectiveness of all these methods heavily relies on the initial quality of object proposals retrieved from single-frame RPN}~\cite{ren2015faster}.~Alternatively, this paper exploits temporal information to generate object proposals for video object detection.

Recently, MAMBA~\cite{sun2021mamba} proposes to extract region proposals from the enhanced feature maps through a pixel-level memory bank.~TransVOD~\cite{he2021end} proposes the transformer-based VOD pipeline by extending Deformable DETR~\cite{zhu2020deformable} to exploit temporal information in videos.~Despite the simple and end-to-end trainable framework, the temporal transformer in TransVOD depends on object queries generated by the spatial transformer optimized without temporal information.~Furthermore, owing to the interaction between each object query and the dense features of an entire frame,~\cite{he2021end} is not a pure sparse method~\cite{sun2021sparse}. As a result of this dense interaction, TransVOD requires pre-training the detector on a similar dataset~\cite{lin2014microsoft}.~On the contrary, our method leverages temporal information to generate object proposals.~Moreover, following~\cite{sun2021sparse}, our proposed SparseVOD operates on a pure sparse paradigm and does not require pre-training the detector.

\vspace{-10pt}
\section{Method}
\label{sec:method}
\textbf{Overview.}~This section explains our proposed SparseVOD, which consists of two main components: Temporal RoI extraction and Semantic Proposal Feature Aggregation incorporated in the Temporal Dynamic Head. Finally, we discuss network optimization.~Note that due to space constraints, the detailed explanation of the still-image object detector Sparse R-CNN~\cite{sun2021sparse} is omitted here and can be found in the supplementary material.

\begin{figure}
\vspace{-3pt}
\includegraphics[width=12.5cm]{images/pipeline.pdf}
\caption{Our SparseVOD Framework.~The input for the first stage of the temporal dynamic head consists of initial proposal boxes, proposal features, and spatial features for the target and support frames.~The TFE extracts RoI proposal features for the target frame by exploiting RoI features from support frames.~Then, SPFA receives object features from each IIH (identical as in~\cite{sun2021sparse}) and aggregates object features with guided attention heads.~This enhanced object feature representation from SPFA and box predictions after FFN serve as input proposal features and proposal boxes for the next iterative stage.}
\label{fig:pipeline}
\vspace{-16pt}
\end{figure}

\subsection{SparseVOD Architecture}
\label{subsec:sparseVOD}
The SparseVOD is a simple, end-to-end trainable framework, as shown in Fig.~\ref{fig:pipeline}. It receives a target frame and multiple support frames from the same video as input and outputs the class and location of objects in the target frame.~For each target and support frame, the extracted feature maps from the backbone, proposal boxes and corresponding proposal features are fed into the iterative Temporal Dynamic head consisting of multiple stages. We create Temporal Dynamic Head by incorporating two main components into the dynamic head used in~\cite{sun2021sparse} to effectively exploit the temporal information in videos.~First, inspired by~\cite{gong2021temporal}, we leverage support frame RoI features to extract temporal RoI features for the target frame.~Then, the corresponding Instance Interactive Head~(IIH) produces object feature representation for each video frame.~Subsequently, the Semantic Proposal Feature Aggregation module enhances the representation of the target frame by intelligently aggregating object features from support frames. The optimal object features are fed to the corresponding feed-forward network for classification and regression. Similar to~\cite{sun2021sparse}, we follow the iterative architecture in our SparseVOD. In the next iteration, the newly refined object features and the bounding box predictions serve as the target frame's proposal features and proposal boxes.

\noindent \textbf{Temporal RoI Feature Extraction.}
\label{subsec:troi}
The Sparse R-CNN~\cite{sun2021sparse} applies the conventional RoIAlign~\cite{he2017mask} pooling operation to extract proposal features, and it is widely adopted in existing VOD methods~\cite{wu2019sequence, zhu2017flow, han2020mining}. However, the naive RoIAlign operation only restricts the proposal feature extraction to exploit intra-frame features.~Therefore, motivated by~\cite{gong2021temporal}, we incorporate the Temporal Feature Extraction (TFE) module in our Temporal Dynamic Head, as illustrated in Fig.~\ref{fig:pipeline}.~The TFE leverages temporal information for the same object instance across support frames in a video. Since the features of the same object instance have high semantic resemblance across video frames, we calculate cosine similarities between target proposal features and support frame feature maps. Given target proposal features $P_{t}$ and feature map from support frame $F_{t+s}$, the cosine similarity $Sim_{t+s}(m)$ is computed as $P_{t}(m) \otimes F_{t+s}$, where $m$ represents the spatial location of $P_{t}$ and $\otimes$ denotes dot product. Note that the target frame proposal is mapped on the most similar feature maps from support frames to extract the most similar RoI features. Following the temporal attentional feature aggregation in~\cite{gong2021temporal}, we adopt the self-attention mechanism~\cite{vaswani2017attention} to aggregate the RoI features of target and support frames.

\vspace{3pt}
\noindent \textbf{Semantic Proposal Feature Aggregation.}
\label{subsec:tfa}
The Semantic Proposal Feature Aggregation (SPFA) head aims to learn the enhanced feature representation from target RoI features (containing temporal information of the same instance) and support frame RoI features to perform final classification and regression. Since we already have temporal RoI features, we follow the spirits of~\cite{wu2019sequence} and adopt semantic similarity as the metric to aggregate features from support frame proposals. We compute semantic similarity between target and support proposal features in the same way as in TFE.
For effective feature aggregation, the SPFA applies multi-head attention~\cite{vaswani2017attention} on temporal proposal features of a target frame $\overline{P_{t}}$ and support proposal features $P_{s}$ as  follows:
\vspace{-6pt}
\begin{equation}
\vspace{-6pt}
    W_t = softmax( \frac{ \phi(P_{s}) \cdot (\theta(\overline{P_{t}}))^T }{\sqrt{d_{\theta(\overline{P_{t}})}}})\cdot \sigma(P_{s})
\end{equation}
\noindent where $\phi(.)$, $\theta(.)$, and $\sigma(.)$ are some linear transformations. The symbol $T$ represents transposition, $d$ denotes the size of transformed $\overline{P_{t}}$, and $W_t$ is the enhanced proposal representation of a target frame. This rich instance representation improves robustness against inherent challenges of VOD, such as appearance deterioration.

\vspace{3pt}
\noindent \textbf{Loss Function.}
\label{subsec:loss_function}
Since our SparseVOD operates on a one-to-one label matching, our method's loss function and training process are similar to the original Sparse R-CNN. We adopt set predictions loss~\cite{carion2020end, zhu2020deformable, he2021end}, which aims to optimize the bipartite matching among the ground truth and predictions. Following~\cite{he2021end, sun2021sparse, carion2020end}, the cost function is defined as $\mathcal{L} = \lambda_{cls} \cdot \mathcal{L}_{cls}+ \lambda_{L1} \cdot \mathcal{L}_{L1} + \lambda_{giou} \cdot \mathcal{L}_{giou}$, where $\mathcal{L}_{cls}$ is the focal loss~\cite{lin2017focal} for classification. $\mathcal{L}_{L1}$ and $\mathcal{L}_{giou}$ are L1 loss and  generalized IoU loss~\cite{rezatofighi2019generalized} for regression, respectively. $\lambda_{cls}$, $\lambda_{L1}$, and $\lambda_{giou}$ are coefficients to balance the loss. We employ an identical setting to balance these losses as in~\cite{sun2021sparse}.
\vspace{-15pt}

\section{Experiments and Results}
\label{sec:experiments}
\vspace{-5pt}
\subsection{Experimental Settings}
\label{subsec:experimental setup}
We perform experiments on the ImageNet VID dataset~\cite{russakovsky2015imagenet}, which comprises 3862 training videos and 555 validation videos. Following prior works~\cite{cui2021tf, wu2019sequence, zhu2017flow}, we train our model on a combination of ImageNet VID and DET datasets and evaluate the results on the validation set. We adopt ImageNet pre-trained~\cite{deng2009imagenet} ResNet-50~\cite{he2016deep}, ResNet-101, and ResNeXt-101~\cite{xie2017aggregated} backbones to compare performance with recent state-of-the-art methods. We train our network for 12 epochs with a batch size of 8 on 8 GPUs. Analogous to~\cite{sun2021sparse}, we use AdamW~\cite{loshchilov2017decoupled} optimizer with a weight decay of $10^{-4}$.  Initially, the learning rate is set to $2.5\times10^{-5}$ and divided by 10 at the 8-th and 11-th epochs. Following~\cite{sun2021sparse, carion2020end, zhu2020deformable}, we set $\lambda$\textsubscript{cls}=2, $\lambda$\textsubscript{L1}=5, and $\lambda$\textsubscript{giou}=2. We follow the basic settings of~\cite{sun2021sparse} and set the number of iterative stages, proposal boxes, and the corresponding proposal features to 6, 100, and 100, respectively. We refer readers to supplementary materials for more details.

\subsection{Main Results}
\label{subsec:qantity}
We compare the performance of the proposed SparseVOD with prior state-of-the-art VOD methods on ImageNet VID dataset in Table~\ref{table:sota_comparison}.~Besides the conventional mAPs @IoU=0.5, we compute mAPs @IoU=0.75 and @IOU=0.5:95 as in~\cite{lin2014microsoft} to analyze the precision of detections. Owing to the unavailability of code at the time of experiments, apart from~\cite{he2021end, sun2021mamba}, we reproduce the results of existing methods from the code provided by the original papers for direct comparison. It is important to mention that all the results shown in Table~\ref{table:sota_comparison} are without any post-processing. By looking at results under the backbone of ResNet-50, it is evident that our SparseVOD outperforms recent methods~\cite{zhu2017flow, wu2019sequence, chen2020memory, gong2021temporal}, mainly relying on feature aggregation of region proposals.~Furthermore, it surpasses the previous best score of 79.9\% by~\cite{he2021end} and achieves a new best score of 80.3\% on ResNet-50.~Note that alongside the improvement on mAP @IoU=0.5, our SparseVOD demonstrates a significant increase (7.3 and 6 points) in the precise localization on mAPs @IoU=0.75 and @IOU=0.5:95 from the previous best method~\cite{gong2021temporal}, reflecting the superiority of spatio-temporal learnable proposals.

\begin{table}
\begin{center}
\scriptsize
\begin{tabular}{ccccccc}
\toprule
Methods & Venue & Backbone & Detector & mAP$_{50:95}$($\%$) & mAP$_{50}$($\%$)  &  mAP$_{75}$($\%$)\\
\noalign{\smallskip}
\hline
FGFA*~\cite{zhu2017flow}  & ICCV'17 & ResNet-50 & Faster R-CNN & 47.1 & 74.7 & 52.0 \\
SELSA*~\cite{wu2019sequence}  & ICCV'19 & ResNet-50 & Faster R-CNN & 48.6 & 78.4 & 52.5 \\
MEGA*~\cite{chen2020memory}  & CVPR'20 & ResNet-50 & Faster R-CNN & 48.1 & 77.3 & 52.2 \\
TROI*~\cite{gong2021temporal}  & AAAI'21 & ResNet-50 & Faster R-CNN & \textbf{\color{blue} 48.8}  & 78.9 & \textbf{\color{blue}52.8} \\
TransVOD~\cite{he2021end}  & ACM MM'21 & ResNet-50 & Deformable DETR& -  & \textbf{\color{blue}79.9} & - \\
\hline
{Frame Baseline~\cite{sun2021sparse}}& CVPR'21 & ResNet-50 & Sparse R-CNN &48.7 & 71.1 & 52.4 \\
{\textbf{SparseVOD} }& BMVC'22 & ResNet-50 & Sparse R-CNN & \textbf{\color{red}54.7} & \textbf{\color{red}80.3} & \textbf{\color{red}60.1} \\
\hline
FGFA*~\cite{zhu2017flow}  & ICCV'17 & ResNet-101 & Faster R-CNN & 50.4 & 78.1 & 56.7 \\
SELSA*~\cite{wu2019sequence}  & ICCV'19 & ResNet-101 & Faster R-CNN & 52.4 & 81.5 &  57.9 \\
MEGA*~\cite{chen2020memory}  & CVPR'20 & ResNet-101 & Faster R-CNN & \textbf{\color{blue}53.1}  & \textbf{\color{blue} 82.9} & \textbf{\color{blue}59.1}\\
TROI*~\cite{gong2021temporal}  & AAAI'21 & ResNet-101 & Faster R-CNN& 51.6  &   82.6 & 56.3 \\
MAMBA~\cite{sun2021mamba}  & AAAI'21 & ResNet-101 & Faster R-CNN& -  &  \textbf{\color{red} 84.6} & - \\
TransVOD~\cite{he2021end}  & ACM MM'21 & ResNet-101 & Deformable DETR & - &81.9 & - \\
\hline
{Frame Baseline~\cite{sun2021sparse}}& CVPR'21 & ResNet-101 & Sparse R-CNN & 51.7 & 74.6 & 53.9 \\
{\textbf{SparseVOD} }& BMVC'22 & ResNet-101 & Sparse R-CNN  & \textbf{\color{red}56.9} & 81.9 & \textbf{\color{red}63.1}\\
\hline
FGFA*~\cite{zhu2017flow}  & ICCV'17 & ResNeXt-101 & Faster R-CNN& 52.5  & 79.6 & 59.8 \\
SELSA*~\cite{wu2019sequence}  & ICCV'19 & ResNeXt-101 & Faster R-CNN& \textbf{\color{blue}  54.2} & 83.1 &  \textbf{\color{blue} 61.3}  \\
MEGA~\cite{chen2020memory}  & CVPR'20 & ResNeXt-101 & Faster R-CNN  & - & \textbf{\color{blue}84.1}& -\\
TROI*~\cite{gong2021temporal}  & AAAI'21 & ResNeXt-101 & Faster R-CNN  & 54.4 & \textbf{\color{red} 84.3} & 60.9\\
\hline
{Frame Baseline~\cite{sun2021sparse}}& CVPR'21 & ResNeXt-101 & Sparse R-CNN & 53.3 & 76.6 & 57.9 \\
{\textbf{SparseVOD} }& BMVC'22 & ResNeXt-101 & Sparse R-CNN  & \textbf{\color{red}58.0} & 83.1 & \textbf{\color{red}64.3}\\
\bottomrule
\end{tabular}
\end{center}
\caption{Comparison with other state-of-the-art methods on the ImageNet VID dataset. Results with * are reproduced. The two best results are highlighted in red and blue.}
\label{table:sota_comparison}
\vspace{-20pt}
\end{table}

When stronger backbones of ResNet-101 and ResNeXt-101 are incorporated into SparseVOD, the performance (mAP$_{50}$) further increases to 81.9\% and 83.1\%, respectively. Note that although MAMBA~\cite{sun2021mamba}, MEGA~\cite{chen2020memory}, and TROI~\cite{gong2021temporal} demonstrate better results at mAP$_{50}$, our SparseVOD supersedes them by far (4$\sim$5 points in mAP) on higher IoU thresholds. These results correspond to our argument that while prior VOD methods operating on dense to sparse detection pipelines show impressive results on mAP$_{50}$, they fail to produce confident and precise predictions. Furthermore, our SparseVOD boosts the single-frame baseline (Sparse R-CNN~\cite{sun2021sparse}) by a strong margin (8\%$\sim$9\% mAP$_{50}$) with all backbone networks. This noticeable increase in mAP highlights the importance of leveraging temporal information to generate object proposals in VOD.
\vspace{-5pt}
\subsection{Iterative proposal Visualization Analysis}
\label{subsec:qualitative}
\vspace{-5pt}
We visualize the behaviour of learned proposals boxes of a trained model on a video clip from the validation set in Fig.~\ref{fig:iterative_results}.~Note that these proposals cover almost all potential regions in the target frames.~This ensures high recall performance even with sparse proposals.~Moreover, each stage in the cascaded architecture refines the bounding box offset and removes duplication.~This makes our pipeline independent of any post-processing techniques to produce precise predictions.~Fig.~\ref{fig:iterative_results}\textcolor{red}{(d)} further exhibits the robustness of our SparseVOD by producing high-quality predictions in challenging scenarios with camera defocus and part-occlusions. Please see supplementary materials for more qualitative analysis. \vspace{-5pt}

\subsection{Ablation Studies}
\label{subsec:ablation}
\vspace{-5pt}
This section discusses the effect of key components and validates the design choices in our proposed method. Following~\cite{he2021end}, we perform all experiments on ImageNet VID dataset with ResNet-50 as the backbone. The run time (FPS) is tested on a single DGX A100 GPU. More ablation studies can be found in supplementary materials.

\vspace{3pt}
\noindent \textbf{Effectiveness of each component in SparseVOD.}
\hspace{1pt}
Table~\ref{table:ablation_each_module} summarizes the impact of adding each component to build our proposed method. Beginning with the single-frame baseline, we incorporate the TFE module (explained in Section~\ref{subsec:troi}) and boost the AP$_{50}$ from 71.1\% to 76.9\%.~This demonstrates the benefit of exploiting inter-frame information to extract RoI proposal features. On the other hand, by separately plugging the SPFA module (explained in Section~\ref{subsec:tfa}) into a single-frame baseline, we achieve substantial gains in AP$_{50}$ from 71.1\% to 79.1\%. Finally, by combining both TFE and SPFA to build the proposed SparseVOD, we gain a further boost of 1.2\% in AP$_{50}$, accomplishing 80.3\%. These results establish the superiority of introducing spatio-temporal feature aggregation for learnable proposals.
\begin{figure}[H]
\centering
\vspace{-10pt}
\includegraphics[width=11.5cm]{images/iterative_stage_results.pdf}
\vspace{-5pt}
\caption{Illustration of the learned proposals and bounding box predictions at different iterative stages from a converged model. For brevity, we only visualize predictions with a confidence score greater than 0.3.~Note that learned proposal boxes in (a) cover possible regions in all three video frames while the cascading heads in each stage enhance detections.}
\label{fig:iterative_results}
\vspace{-20pt}
\end{figure}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\begin{table}[H]
\small
\begin{center}
\begin{tabular}{ccc|cccc}
\toprule
Single Frame Baseline & TFE & SPFA  & AP$_{50}$(\%) &  AP$_{75}$($\%$)  &  AP$_{50:95}$($\%$)& FPS \\
\hline\noalign{\smallskip}
\textcolor{Green}{\cmark} & \color{red}\xmark & \color{red}\xmark  & 71.1 & 52.4 & 48.7 &  24.3 \\
 \color{Green}\cmark &  \color{Green}\cmark&  \color{red}\xmark & 76.9$_{\uparrow}$\textsubscript{5.8} & 57.5$_{\uparrow}$\textsubscript{5.1} &52.1$_{\uparrow}$\textsubscript{3.4} & 14.9 \\
 \color{Green}\cmark & \color{red}\xmark & \color{Green}\cmark& 79.1$_{\uparrow}$\textsubscript{8.0} & 59.0$_{\uparrow}$\textsubscript{6.6} & 54.1$_{\uparrow}$\textsubscript{5.4}& 17.7  \\
 \color{Green}\cmark &  \color{Green}\cmark& \color{Green}\cmark& 80.3$_{\uparrow}$\textsubscript{9.2} & 60.1$_{\uparrow}$\textsubscript{7.7} & 54.7$_{\uparrow}$\textsubscript{6.0} &  14.4 \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-5pt}
\caption{Ablation on effectiveness of each module in SparseVOD.}
\label{table:ablation_each_module}
\vspace{-15pt}
\end{table}
\begin{minipage}{\linewidth}
  \begin{minipage}{0.35\linewidth}
    \vspace{-10pt}
    \hspace{-10pt}
  \begin{table}[H]
    \small
    \begin{tabular}{c|ccc}
        \toprule
        Stages & AP$_{50}$(\%) & AP$_{75}$(\%) & FPS\\
        \hline
        1 & 60.0 & 27.2 & 42.2\\
        2 & 74.5 & 52.8 & 29.4\\
        3 & 78.0 & 58.3 & 23.7\\
        4 & 78.5 & 58.6 & 19.8\\
        5 & 79.1 & 59.8 & 16.1\\
        {6} & \textbf{80.3} & \textbf{60.1} & \textbf{14.4}\\
        12 & 77.1 & 55.2 & 5.7\\
        \bottomrule
  \end{tabular}\\
  \vspace{-5pt}
  \caption{Ablation on the number of stages.}
  \label{subtab:stages}
  \end{table}
  \end{minipage}
  \hspace{0.05\linewidth}
  \begin{minipage}{0.5\linewidth}
  \vspace{-10pt}

      \begin{figure}[H]
        \includegraphics[width=7cm]{images/mAP_comparison_SOTA.jpeg}
        \vspace{-20pt}
        \caption{The detection performance of recent VOD methods and our SparseVOD on increasing IoU threshold.}
        \label{fig:high_quality_AP}
    \end{figure}
  \end{minipage}
\end{minipage}

\begin{table}[H]
\footnotesize
\begin{center}
\begin{tabular}{cccc|cccc}
\toprule\noalign{\smallskip}
TFE (Avg.) & TFE (MSA) & SPFA (Avg.) & SPFA (MSA)  & AP$_{50}$(\%) &  AP$_{75}$($\%$)  &  AP$_{50:95}$($\%$)& FPS \\
\hline \noalign{\smallskip}
     \color{red}\xmark & \color{red}\xmark & \color{red}\xmark
     & \color{red}\xmark  &71.1 & 52.4 & 48.7 &  24.3 \\
     \hline
     \color{Green}\cmark & \color{red}\xmark & \color{Green}\cmark
     & \color{red}\xmark  &75.8 & 52.9 & 49.1 &  18.5 \\

     \color{Green}\cmark & \color{red}\xmark & \color{red}\xmark  & \color{Green}\cmark &79.4 & 58.9 & 54.1 &  16.3 \\

    \color{red}\xmark  &\color{Green}\cmark &  \color{Green}\cmark & \color{red}\xmark  &78.5 & 58.1 & 52.1 &  14.9 \\

    \color{red}\xmark  &\color{Green}\cmark & \color{red}\xmark  & \color{Green}\cmark & \textbf{80.3} & \textbf{60.1} & \textbf{54.7} & 14.4\\
\bottomrule
\end{tabular}
\end{center}
\caption{Ablation on the effectiveness of multi-head attention in TFE (Temporal Feature Extraction) and SPFA (Semantic Proposal Feature Aggregation) modules.~The first row represents results from a single-frame baseline. The terms Avg.~and MSA denote simple averaging and multi-head self-attention.}
\label{table:ablation_attention}
\vspace{-20pt}
\end{table}

\begin{minipage}{\linewidth}

  \begin{minipage}{0.35\linewidth}
  \begin{table}[H]
  \hspace{-10pt}
    \small
        \begin{tabular}{c|ccc}
            \toprule
            N\textsubscript{ref} & AP$_{50}$(\%) &  AP$_{75}$($\%$)  & FPS \\
            \hline
            1 & 71.1 & 52.4 & 24.3\\
            2 & 79.0 & 58.2 & 17.7\\
            4 & 79.9 & 59.5 & 15.9\\
            {6} & \textbf{80.3} & \textbf{60.1} & \textbf{14.4}\\
            10 & 80.2 & 60.2 & 11.7\\
            14 & 80.3 & 60.2 & 9.5\\
            \bottomrule
        \end{tabular}\\
        \vspace{-5pt}
          \caption{Ablation on number of support frames.}
          \label{tab:ablation_support_frames}
      \end{table}
  \end{minipage}
  \hspace{0.03\linewidth}
  \begin{minipage}{0.55\linewidth}
        \begin{figure}[H]
        \includegraphics[width=7cm]{images/tradeoff.jpeg}
        \vspace{-10pt}
        \caption{Speed-accuracy tradeoff between SparseVOD and previous best competitor (TROI+SELSA~\cite{gong2021temporal}).}
        \label{fig:tradeoff}
      \end{figure}
  \end{minipage}
\end{minipage}

\vspace{5pt}
\noindent \textbf{Number of Stages.}
The impact of increasing stages in an iterative architecture is summarized in Table~\ref{subtab:stages}. Note that without iterative architecture, even though the performance AP$_{50}$(\%) reaches 60.0, the AP$_{75}$ is merely 27.2\%. Since the input proposal boxes at the first stage are just random distribution of possible object locations, this result (AP$_{50}$=60.0\%) indicates that computing AP on a single IoU threshold of 0.5 is not a reliable evaluation metric.~By increasing iterative stages to 3, the performance already reaches a comparable AP$_{50}$ of 78\% and surpasses prior methods on AP$_{75}$ with 58.3\%. Finally, similar to~\cite{sun2021sparse}, the performance saturates at 6 stages. Hence, we adopt 6 stages in our experiments.

\vspace{2pt}
\noindent \textbf{Comparing High-Quality Detection.}
Fig.~\ref{fig:high_quality_AP} shows the AP curves of recent state-of-the-art VOD methods and our SparseVOD under increasing IoU thresholds. It is evident that the proposed method consistently outperforms prior works with a significant margin on all the evaluation metrics.~Note that although the difference is mild with the previous best competitor TROI~\cite{gong2021temporal}, on low IoU threshold (0.5), it rises on higher IoU thresholds. These results reflect the superiority of sparse spatio-temporal learnable proposals over hand-crafted dense priors optimized on a single IoU level in existing VOD methods.
\vspace{2pt}

\noindent \textbf{Effectiveness of Multi-head Attention in TFE and SPFA.}
We demonstrate the effectiveness of multi-head attentional blocks~\cite{vaswani2017attention} in Temporal Feature Extraction~(TFE) and Semantic Proposal Feature Aggregation~(SPFA) modules in Table~\ref{table:ablation_attention}.~For direct comparison, we conduct baseline experiments where attentional weights in TFE and SPFA are replaced by simple averaging to aggregate features from target and support frames. As shown in Table~\ref{table:ablation_attention}, with averaging in both modules, the AP\textsubscript{50} reaches 75.8\%, reflecting the benefit of leveraging temporal information to refine object proposals. Furthermore, we conduct experiments by switching attention to averaging in one of the two modules. With TFE~(Avg) and SPFA~(MSA), we observe an impressive speed-accuracy tradeoff of 79.4\% AP\textsubscript{50} and 16.3 FPS. Since the averaging in TFE is performed on the most similar RoI features computed with cosine similarities, when combined with SPFA~(MSA), it already provides an acceptable object feature representation. However, these results are still inferior (-0.9 points in AP\textsubscript{50}) to the performance achieved when multi-head attention is plugged in both TFE and SPFA. These results (80.3\% AP\textsubscript{50}) indicate the superiority of multi-head attentional aggregation in our method.

\noindent \textbf{Number of Support Frames.}
Table~\ref{tab:ablation_support_frames} presents the ablations on the number of support frames. We follow the identical frame sampling strategy as in~\cite{wu2019sequence, gong2021temporal}, where support frames are uniformly sampled from the entire video. We can see that the AP\textsubscript{50} already reaches 79.0\% with 2 support frames. Upon increasing support frames, the performance keeps increasing and tends to stabilize after reaching the AP\textsubscript{50} of 80.3\% with 6 support frames.

\noindent \textbf{Speed-accuracy Tradeoff.}
\label{subsec:comp_analysis}
Table~\ref{table:ablation_each_module} shows that the computational load in our SparseVOD stems from Temporal Feature Extraction~(TFE) and Semantic Proposal Feature Aggregation~(SPFA). Since the results of~\cite{he2021end} are not reproducible, for direct comparison, we analyse the speed-accuracy tradeoff of the second best method TROI~\cite{gong2021temporal} and our SparseVOD in Fig.~\ref{fig:tradeoff}.~Note that TROI~\cite{gong2021temporal} is built upon SELSA~\cite{wu2019sequence} to enhance performance.~With only 6 support frames sampled from the entire video, our SparseVOD achieves a new best AP\textsubscript{50} of 80.3\% with a run time of 14.4 FPS. In contrast, TROI manages to reach its best performance (AP\textsubscript{50} of 78.8\%) with a run time of 7.5 FPS after utilizing 14 support frames. The results in Fig.~\ref{fig:tradeoff} demonstrate that the temporal feature aggregation from several support frames in prior works~\cite{wu2019sequence, gong2021temporal} lead to a major increase in run time, producing a sub-optimal speed-accuracy tradeoff. Contrarily, thanks to the spatio-temporal learnable proposals, our SparseVOD yields an optimal speed-accuracy tradeoff~(79.0\% AP\textsubscript{50} and 17.7 FPS) with merely 2 support frames.

\vspace{-15pt}
\section{Conclusion}
\label{sec:conclusion}
\vspace{-5pt}
This paper proposes SparseVOD, a novel video object detection pipeline which introduces spatio-temporal feature aggregation to refine object proposals.~The SparseVOD effectively eliminates hand-crafted dense priors and provides reliable proposal features even with deteriorated input frames.~Particularly, the SparseVOD incorporates attention-guided Temporal Feature Extraction and Semantic Proposal Feature Aggregation modules in Sparse R-CNN~\cite{sun2021sparse}. Extensive experiments validate that SparseVOD significantly improves the baseline performance by 8\%-9\% in mAP and achieves the state-of-the-art 80.3\% mAP\textsubscript{50} on the ImageNet VID dataset with ResNet-50 backbone.~Besides, our SparseVOD beats existing methods in terms of high-quality predictions and optimal speed-accuracy tradeoff.~To our knowledge, our work is the first one that exploits temporal information in directly generating a sparse set of object proposals for video object detection.~We hope similar work can be applied to other video analysis tasks like object tracking and video instance segmentation.


\section{Introduction}
\label{seq:intro}

In a continuous quest for better pre-training strategies, models based on image and language supervision have set impressive milestones, with CLIP~\citep{clip_icml21}, ALIGN~\citep{align_icml21} and Flamingo~\citep{flamingo_arxiv22} being leading examples.
Contrastively trained vision-language models consist of image and text encoders that align semantically-related concepts in a joint embedding space.
Such models offer impressive zero-shot image classification by using the text encoder to generate classifier weights from arbitrarily newly defined category classes without relying on any visual data. In particular, the class name is used within a handcrafted \textit{prompt} template and then tokenized and encoded into the shared embedding space to generate new classifier weights. Rather than manually defining prompts,
\citet{zhou2022learning} and \citet{lester2021power} proposed that prompts can be instead optimized in a data-driven manner through back-propagation by maximizing a cross-entropy loss on the downstream task.
However, despite the performance improvement on downstream tasks, prompt learning negatively affects the generalization capability of the vision-language model.

While subsequent works have focused on how to bridge the generalization gap, e.g. \citet{zhou2022conditional,zhu2022prompt}, in practice the generalization power of the foundation model is significantly degraded. Our work tackles this same problem, as it seeks to improve downstream performance but without degrading the generalization capability of the original model.

To do so, we propose a data-driven method for directly learning the underlying distribution within the prompt space associated to the target concept.
In particular, we frame prompt tuning as a variational inference problem, where a base learned prompt is combined with a residual vector sampled from the instance-specific underlying distribution. This formulation provides two advantages. First, it investigates the prompt space more thoroughly and results in more informative use of the language space, leading to better generalization. Second, it enables us to boost performance by capturing the uncertainty information in fine-grained classification problems.
The resulting approach is orthogonal to standard prompt learning approaches, being effective when combined with both standard~\citep{zhou2022learning} and conditional~\citep{zhou2022conditional} approaches. In fact, when combined with the conditional approach, our method maintains the gains on the seen classes provided by the conditional method while simultaneously matching or even surpassing the generalization capability on unseen classes of the original vision-language model.

In summary, our contributions in this paper are as follows:
\begin{enumerate}
\item We propose a variational framework that is capable of capturing the general or instance specific distribution within the prompt space. Since generalization is obtained through transfer from the language space, we obtain better generalization capability.
\item We show that the proposed approach is orthogonal to recent developments, and can be successfully combined with both standard and conditional prompt learning variants.
\item We empirically show that our proposed method improves performance and provides better generalization, leading to state-of-the-art accuracy in 24 out of 28 standard benchmarks set forth by prior work, surpassing \cocoop by 1.6\% average Top-1 accuracy.
\end{enumerate}

\section{Related works}
\label{sec:related}

\textbf{Prompt learning in NLP.} Prompt learning was originally proposed within the NLP domain, following the appearance of foundation models such as GPT-3~\citep{brown2020language}.
Early prompt learning methods constructed prompts by combining words in the language space such that the model would perform better on downstream evaluation \citep{shin2020autoprompt,jiang2020can}.
Subsequent methods, e.g. \cite{li2021prefix,lester2021power}, prepend a set of learnable prompts to the input of a frozen model and optimize through back-propagation, which allows better flexibility than using existing words, at the cost of leading to prompts that do not correspond to an actual phrase.
Instead, \citet{he2022hyperprompt} focus on a multi-task scenario and use a HyperNetwork to conditionally generate task-specific and layer-specific prompts that are pre-pended to the values and keys inside the self-attention layers of a the frozen model. Within the NLP domain, prompt learning has also been shown to work better than in-context learning~\citep{liu2022few}.

\textbf{Prompting in Vision and Language models.} Research on prompt learning for vision-language models have been largely inspired by prior work within NLP. Similar to e.g. \citet{li2021prefix}, \coop \citep{zhou2022learning} proposes a prompt learning method that optimizes unified or class specific prompts in the continuous space through back-propagation. While \coop obtains good accuracy on downstream tasks, it negatively affects the generalization ability to new unseen classes. \cocoop \citep{zhou2022conditional} extends \coop and partially bridges the generalization gap by generating instance-specific prompt residuals through a conditioning mechanism dependent on the visual data. ProGrad~\citep{zhu2022prompt} shares the same goal as \cocoop of bridging the generalization gap, but instead proposes to match the gradient of the prompt to the general knowledge of the CLIP model to prevent prompt tuning from forgetting the general knowledge learned from the foundation model. Alternative directions consist of test-time prompt tuning~\citep{testtimept_neurips22}, where consistency across multiple views is used as the supervisory signal, and unsupervised prompt learning~\citep{unsup_prompt22}, where a pseudo-labelling strategy is proposed instead to obtain the labels needed to drive the prompt learning.
Perhaps the most similar work to ours is~\citet{lu2022prompt}.
In this work,
the authors use an ensemble of prompts and model their distribution within the language embedding space,
with optimization seeking to minimize the negative log-likelihood with respect to the corresponding visual embedding.
Unlike ours,
their method relies on hand-crafted rules
to define the prompt ensemble, thus still relying on the effectiveness of hand-crafted designs. The number of learnable prompts is also pre-defined, potentially offering sub-optimal coverage of an NLP concept.
Finally,
it is not clear how to apply their strategy within the context of conditional prompt learning.
We believe that modelling the input prompt space rather than relying on a fixed number of templates is a more powerful and flexible approach.
We provide empirical evidence of the superiority of our approach in the experiments.

While beyond our current scope, it is worth noting that prompt learning has been applied to a wider range of problems and scenarios, which highlights its power and flexibility.
Among them are important topics such as unsupervised domain adaptation~\citep{ge2022domain}, multi-label classification~\citep{sun2022dualcoop}, video classification~\citep{ju2021prompting}, object detection~\citep{detpro_cvpr22,promptdet_eccv22} or pixel-level labelling~\citep{rao2022denseclip}. Finally, prompt learning as a means to adapt pre-trained models has also been applied to purely vision models~\citep{jia2022visual,sandler2022fine} providing similar performance to fine tuning the whole model but with great parameter efficiency.

\section{Method}

\begin{figure*}[t!]
\centering
\includegraphics[width=.9\textwidth,clip,trim=0cm 0cm 0cm 0.4cm]{images/variational_prompt.pdf}
\caption{\textbf{Overview of variational prompt tuning}. For each input image $\mathbf{x}$, we use image features $f(\mathbf{x})$ to infer the mean $\mu(\mathbf{x})$ and standard deviation $\Sigma(\mathbf{x})$ of the residual distribution using metanet $\pi_\phi$. The prompts to generate the classifier weights are constructed by summing up learnable prompts $\textbf{p}$ and residual samples from the residual distribution. The obtained prompts are fed through a text encoder $g(\textbf{t})$, and the classifier weights are estimated. Finally, the cosine similarity scores are computed between the image features $f(\mathbf{x})$ and the classifier weights.}
\vspace{-3mm}
\label{fig:overall-idea}
\end{figure*}

\subsection{Background}

\textbf{Contrastive Language-Image Pretraining (CLIP)}~\citep{clip_icml21} consists of an \textit{image encoder} $f(\mathbf{x})$ and \textit{text encoder} $g(\mathbf{t})$, each producing a $d$-dimensional ($L_2$ normalized) embedding from an arbitrary image $\mathbf{x} \in \mathbb{R}^{3 \times H \times W}$, and word embeddings $\mathbf{t} \in \mathbb{R}^{L \times e}$, with $L$ representing the text length and $e$ the embedding dimension\footnote{In CLIP the word embedding is learned together with the text encoder. A \textit{tokenizer} is used to convert the text into one-hot vectors, or tokens, that can be directly mapped into the word embeddings. For the sake of clarity we refer indistinctly to words and word embeddings.}. Both encoders are trained together using a contrastive loss from a large-scale dataset composed of paired images and captions. Once trained, CLIP can be used for zero-shot $C$-class image classification by generating each of the $c$ classifier weights $\mathbf{w}_c$ as the $d$-dimensional text encoding $g(\mathbf{t}_c)$. Here $\mathbf{t}_c$ results from adding the class-specific word embedding $\mathbf{e}_c$ to a pre-defined prompt $\mathbf{p} \in \mathbb{R}^{L-1 \times e}$, i.e., $\mathbf{w}_c {=} g(\mathbf{t}_c)$ with $\mathbf{t}_c {=} \{\mathbf{p},\mathbf{e}_c\}$. The prompt $\mathbf{p}$ is manually crafted to capture the semantic meaning of the downstream task, e.g., $\mathbf{t}_c = ``\texttt{An image of a \{class\}}"$. The probability of image $\mathbf{x}$ being classified as $y \in \{1...C\}$ is thus defined as
$p(y|\mathbf{x}) {=} \frac{e^{f(\mathbf{x})^T\mathbf{w}_y}}{\sum_{c}^C e^{f(\mathbf{x})^T \mathbf{w}_{c}}}$.

\textbf{Context Optimization (CoOp)}~\citep{zhou2022learning} provides a learned alternative to manually defining prompts. CoOp learns a fixed prompt from a few annotated samples. The prompt is designed as a learnable embedding matrix $\mathbf{p} \in \mathbb{R}^{L \times e}$ which is updated via back-propagating the classification error through the frozen CLIP model. Specifically, for a set of $N$ annotated meta-training samples $\{\mathbf{x}_i, y_i\}_{i=1}^N$, the prompt $\mathbf{p}$ is obtained by minimizing the cross-entropy loss, as:
\begin{equation}
\label{eq:coop}
   \mathbf{p}^* = \arg \min_\mathbf{p} \mathbb{E}_{\mathbf{x}_i,y_i} [-\log p(y_i | \mathbf{x}_i, \mathbf{p})].
\end{equation}
Note that this approach, while resembling that of common meta-learning approaches, can still be deployed in a zero-shot scenario provided that for new classes the classification weights will be given by the text encoder. Although this approach allows to generalize to new tasks with few training iterations, learning a fixed prompt is sensitive to domain shifts between the annotated samples and the test set.

\textbf{Conditional Prompt Learning (CoCoOp)}~\citep{zhou2022conditional} attempts to overcome domain shifts by learning an instance-specific continuous prompt that is conditioned on the input image. To ease the training of a conditional prompt generator, \cocoop defines each conditional token in a residual way, with a task-specific, learnable set of tokens $\mathbf{p}$ and a residual vector that is conditioned on the input image. Assuming $\mathbf{p}$ to be composed of $L$ learnable tokens $\mathbf{p} {=} [\mathbf{p}_1, \mathbf{p}_2, ... , \mathbf{p}_L]$, the residual vector ${\mathbf{r}}(\mathbf{x}) {=} \pi_\phi(f(\mathbf{x})) \in \mathbb{R}^e$ is produced by a small neural network $\pi_\phi$ with as input the image features $f(\mathbf{x})$. The new prompt is then computed as $\mathbf{p}(\mathbf{x}) {=} [\mathbf{p}_1 + {\mathbf{r}}(\mathbf{x}), \mathbf{p}_2 + {\mathbf{r}}(\mathbf{x}) , ... , \mathbf{p}_L + {\mathbf{r}}(\mathbf{x})]$. The training now comprises learning the task-specific prompt $\mathbf{p}$ and the parameters $\phi$ of the neural network $\pi_\phi$. Defining the context-specific text embedding $\mathbf{t}_c(\mathbf{x}) {=} \{\mathbf{p}(\mathbf{x}), \mathbf{e}_c \}$, and $p(y | \mathbf{x})$ as :
\begin{equation}
\label{eq:prob_y}
    p(y | \mathbf{x}) = \frac{ e^{ f(\mathbf{x})^T g( \mathbf{t}_c(\mathbf{x}) )} }{ \sum_{c}^C e^{ f(\mathbf{x})^T g( \mathbf{t}_{c}(\mathbf{x}) ) } },
\end{equation}
the learning is formulated as:
\begin{equation}
    \label{eq:cocoop}
   \mathbf{p}^*, \phi^* = \arg \min_{\mathbf{p}, \phi} \mathbb{E}_{\mathbf{x}_i,y_i} [-\log p(y_i | \mathbf{x}_i, \mathbf{p}, \phi)].
\end{equation}
While \cocoop achieves state-of-the-art results in a large variety of downstream tasks, it is still prone to the domain shift problem, considering that $\pi_\phi$ provides a deterministic residual vector from the image features $f(\mathbf{x})$ which are expected to be domain-specific.

\textbf{Prompt Distribution Learning (ProDA)}~\citep{lu2022prompt} is work concurrent to \cocoop that focuses learning a distribution of prompts that generalize to a broader set of tasks. ProDA proposes to learn a collection of prompts $\mathbf{P}{=}\{\mathbf{p}_k\}_{k=1}^\mathcal{K}$ that can be used to subsequently generate an \textit{a posteriori} distribution of the classifier weights for each of the target classes. For a given mini-batch of $K$ sampled prompts $\mathbf{p}_k \sim \mathbf{P}$, the classifier weights $\mathbf{w}_c$ are sampled from the posterior distribution $\mathcal{N}(\mu_{\mathbf{w}_{1:C}}, \Sigma_{\mathbf{w}_{1:C}})$, with mean $\mu_{\mathbf{w}_{1:C}}$ and covariance $\Sigma_{\mathbf{w}_{1:C}}$ computed from the collection $\{\mathbf{w}_{k,c} = g(\mathbf{t}_{k,c})\}_{c=1:C, k=1:K}$, with $\mathbf{t}_{k,c} = \{\mathbf{p}_k, \mathbf{e}_c\}$. The objective is now formulated as:
\begin{equation}
\label{eq:variational_coop}
   \mathbf{P}^* = \arg \min_{\mathbf{P}} \mathbb{E}_{\mathbf{x}_i,y_i} [-\log \mathbb{E}_{\mathbf{w}_l \sim \mathcal{N}(\mu_{\mathbf{w}_{1:C}},\Sigma_{\mathbf{w}_{1:C}})}\, p(y_i | \mathbf{x}_i, \mathbf{w}_l)].
\end{equation}
Computing $\mathbb{E}_{\mathbf{w}_l} \, p(y_i | \mathbf{x}_i, \mathbf{w}_l)]$ is intractable and an upper bound to Eq.~\ref{eq:variational_coop} is derived. During inference, the classifier weights are set to those given by the predictive mean $\mathbf{w}_c = \mu_{\mathbf{w}_{1:C}}$, computed across the collection of learned prompts $\mathbf{P}$. While showing promising results compared to CoOp, how to combine it with the conditional prompt learning framework of CoCoOp is unclear.

\subsection{Variational Prompt Tuning}
\label{ssec:method}

In this paper, we propose to model the input prompt space in a probabilistic manner, as an \textit{a priori, instance-specific} distribution. In particular, we define a distribution $p_\gamma$ over the prompts $\mathbf{p}$ that is instance-specific, i.e. $\mathbf{p} \sim p_\gamma(\mathbf{x})$. To this end, we assume that $\mathbf{p}$ can be split into a fixed set of prompts $\mathbf{p}_i$ and an instance specific residual vector ${\mathbf{r}}$ that act as a latent variable over $\mathbf{p}$. The instance-specific prompt is then defined as:
 \begin{equation}
\label{eq:new_prompt}
\mathbf{p}_\gamma(\mathbf{x}) = [\mathbf{p}_1 + {\mathbf{r}}_\gamma, \mathbf{p}_2 + {\mathbf{r}}_\gamma, \cdots, \mathbf{p}_L + {\mathbf{r}}_\gamma], \, \, {\mathbf{r}}_\gamma \sim p_\gamma(\mathbf{x}),
\end{equation}
 where $p_\gamma(\mathbf{x})$ refers to the real posterior distribution over ${\mathbf{r}}$ conditioned on the observed features $\mathbf{x}$. Denoting the class-specific input as $\mathbf{t}_{c,\gamma}(\mathbf{x}) {=} \{\mathbf{p}_\gamma(\mathbf{x}), \mathbf{e}_c\}$, the marginal likelihood $p(y | \mathbf{x})$ is now defined as:
\begin{equation}
     \label{eq:probabilistic}
 p(y | \mathbf{x}) = \int_\gamma \frac{ e^{ f(\mathbf{x})^T g( \mathbf{t}_{c,\gamma}(\mathbf{x}) )} }{ \sum_{c'} e^{ f(\mathbf{x})^T g( \mathbf{t}_{c',\gamma}(\mathbf{x}) ) } } p(\mathbf{p}_\gamma(\mathbf{x})) d \gamma.
\end{equation}
Solving Eq.~\ref{eq:cocoop} with the marginal likelihood defined as in Eq.~\ref{eq:probabilistic} is intractable, as it requires computing $p_\gamma({\mathbf{r}} | \mathbf{x}) p_\gamma(\mathbf{x})$. Instead, we resort to deriving a lower bound, by introducing a variational posterior distribution $\pi_\phi(\mathbf{x})$ from which the residual ${\mathbf{r}}_\gamma$ can be sampled. The variational bound is defined as:
\begin{equation}
\label{eq:elbo}
\log p(y | \mathbf{x}) \geq \mathbb{E}_{\pi_\phi({\mathbf{r}} | \mathbf{x})} [\log p(y | \mathbf{x}, {\mathbf{r}})] - D_{\mathrm{KL}}\big[\pi_{\phi}(\mathbf{r} | \mathbf{z}) \| p_{\gamma}(\mathbf{r})\big],
\end{equation}
with $p(y | \mathbf{x}, {\mathbf{r}}) \propto e^{ f(\mathbf{x})^T g( \mathbf{t}_{c,\gamma}(\mathbf{x}) )}$, where the dependency on ${\mathbf{r}}$ comes through the definition of $\mathbf{t}_{c,\gamma}$. The variational posterior distribution $\pi_\phi$ plays a role akin to the metanet in CoCoOp. We thus refer to it as metanet in the following to align terminology. Following standard variational optimization practices~\citep{kingma2013auto,gordon2018metalearning}, we define $\pi_\phi$ as a Gaussian distribution conditioned on the input image features $\mathbf{x}$, as ${\mathbf{r}}(\mathbf{x}) \sim \mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$, with $\mu$ and $\Sigma$ parameterized by two linear layers placed on top of the metanet $\pi_\phi$ (see Figure~\ref{fig:overall-idea}). The prior $p_\gamma({\mathbf{r}})$ is defined as $\mathcal{N}(\mathbf{0},\mathbf{I})$, and we make use of the reparameterization trick to generate Monte-Carlo samples from $\pi_\phi$ to maximize the right side of Eq.~\ref{eq:elbo}.
The optimization of Eq.~\ref{eq:elbo} comprises learning the prompt embeddings $\{\mathbf{p}_i\}_{i=1}^L$ as well as the parameters of the metanet $\pi_\phi$ and the linear layers parameterizing $\mu$ and $\Sigma$. Note that this adds little complexity as it requires learning $\mathbf{p}$ and $\pi_\phi$, given that $\mu$ and $\Sigma$ are defined as two linear layers on top of $\pi_\phi$.

\textbf{Inference.} At test time, $K$ residuals are sampled from the conditional distribution $\pi_\phi(\mathbf{x})$, which are used to generate $K$ different prompts per class $\mathbf{p}_{k} {=} [\mathbf{p}_1 + {\mathbf{r}}_k, \mathbf{p}_2 + {\mathbf{r}}_k, \cdots, \mathbf{p}_L + {\mathbf{r}}_k]$. Each prompt is prepended to the class-specific embedding to generate a series of $K$ separate classifier weights $\mathbf{w}_{k,c}$. We then compute $p(y {=} c \,| \, \mathbf{x}) {=} (1/K) \sum_{k=1}^K p(y {=} c \, | \, \mathbf{x}, \mathbf{w}_{k,c})$ and select $\hat{c} {=} \arg \max_{c} p(y {=} c \, | \, \mathbf{x})$ as the predicted class. It is worth noting that because the posterior distribution is generated by the text encoder, it is not expected that for $K \rightarrow \infty$, $(1/K) \sum_{k} p( y {=} c \, | \, \mathbf{x}, \mathbf{w}_{c,k}) \rightarrow p( y {=} c | \mathbf{x}, g(\{\mu(\mathbf{x}), \mathbf{e}_c\})$, meaning that sampling at inference time remains relevant. We study the dependency on the number of samples in the ablations.

Notably, our framework can be also used with the unconditional setting of CoOp, by simply removing the dependency of the input image from the latent distribution. In such scenario, we keep a fixed set of prompt embeddings and learn a global latent distribution $p_\gamma$ over the residual vectors ${\mathbf{r}}$. In this case, $p_\gamma$ is a general distribution learned during training with no dependency on the input sample $\mathbf{x}$. We show that CoOp with our proposed approach improves generalization on new classes.

\section{Experiments and Results}
\label{sec:experiments}

\subsection{Experimental setup}

We follow the exact experimental setup of CoCoOp~\citep{zhou2022conditional}, which is currently the state-of-the-art approach for prompt tuning for vision-language models. We describe the setup in the following for completeness.

\textbf{Three tasks and fifteen datasets.} We evaluate variational prompt tuning for three different tasks: \textit{base-to-new generalization}, \textit{cross-dataset transfer}, and \textit{domain generalization}. For base-to-new generalization and cross-dataset transfer tasks,
we rely on the same 11 image recognition datasets as \citet{zhou2022learning,zhou2022conditional}. These include generic image classification datasets (ImageNet by~\citep{deng2009imagenet} and Caltech101 by~\citep{fei2004learning}), fine-grained classification datasets (OxfordPets by~\citep{parkhi2012cats}, StanfordCars by~\citep{krause20133d}, Flowers102 by~\citep{nilsback2008automated}, Food101 by~\citep{bossard2014food} and FGVCAircraft by~\citep{maji2013fine}), scene recognition (SUN397 by~\citep{xiao2010sun}), action recognition (UCF101 by~\citep{soomro2012dataset}), texture classification (DTD by~\citep{cimpoi2014describing}), and satellite imagery recognition (EuroSAT by~\citep{helber2019eurosat}). For the domain generalization task, we first train our model on  ImageNet and report our evaluation metric on ImageNetV2~\citep{recht2019imagenet}, ImageNet-Sketch~\citep{wang2019learning}, ImageNet-A~\citep{hendrycks2021natural}, and ImageNet-R~\citep{hendrycks2021many}.

\textbf{Evaluation metrics.} We report average accuracy and harmonic mean $H{=}2\times(\text{base}\times\text{new})/(\text{base}+\text{new})$ \citep{xian2017zero} for the base-to-new generalization tasks. For cross-dataset transfer learning and domain adaptation, we provide average accuracy results.

\textbf{Baselines.} We compare against zero-shot CLIP~\citep{clip_icml21}, \coop \citep{zhou2022learning}, \cocoop \citep{zhou2022conditional}, and ProDA~\citep{lu2022prompt}.
For zero-shot CLIP, CoOp, CoCoOp, all results are adopted from~\citep{zhou2022conditional}, and we reproduce all results for ProDA.

\textbf{Implementation details.}
Our variational prompt tuning contains three sub-networks: an image encoder $f(\mathbf{x})$, a text encoder $g(\mathbf{t})$, and a metanet $\pi_{\phi}$. The image encoder $f(\mathbf{x})$ and text encoder $g(\mathbf{t})$ are a ViT-B/16 \citep{dosovitskiy2020image} and transformer \citep{vaswani2017attention}, which are initialized with CLIP's pre-trained weights and kept frozen during training, as in \citet{zhou2022learning, zhou2022conditional}.
The metanet $\pi_{\phi}$ consists of two linear layers followed by ELU activation function as trunk and two linear heads on top to estimate the $\mu$ and $\Sigma$ of the residual distribution. For each task and dataset,
we optimize the number of samples $K$ and epochs. Other hyper-parameters as well as the training pipeline in terms of few-shot task definitions are identical to \citet{zhou2022learning, zhou2022conditional} (see table~\ref{tab:shared_hparams} and~\ref{tab:dataset_hparams} in the appendix). Implementation code will be released.

\subsection{Base-to-new generalization}
\textbf{Setup.} We report the few-shot generalization of our method on $11$ datasets for three different random seeds. Each dataset is divided into two disjoint subsets: base classes and new classes. We train our method on base classes and evaluate it on both base and new classes. For a fair comparison, we follow \citet{zhou2022conditional, zhou2022conditional} in terms of dataset split and number of shots (see Table~\ref{tab:shared_hparams} appendix.)

\textbf{Results.} From the results in Table~\ref{tab:base-to-new},
the lack of generalization capability of the \coop approach is evidenced by the considerable discrepancy between the base and new classes' accuracy. This is expected as \coop only observes a small number of training samples to adapt the CLIP model for downstream tasks, resulting in overfitting. Adding variational prompt tuning to \coop reduces overfitting to base classes, improving the performance of new classes by $11.54\%$ and the harmonic mean by $1.7\%$ across all datasets, at the expense of a base classes accuracy drop by $10.7\%$. Figure~\ref{fig:delta-harmonic-mean}(a) depicts the relative improvement of our proposed strategy when compared to \coop in terms of the harmonic mean, where we observe an improvement in 10 out of 11 datasets. The limited generalization capability of \coop is mitigated by \cocoop by exploiting instance-conditional prompts, which improves the accuracy on new classes from $63.22\%$ to $72.23\%$. Nonetheless, augmenting \cocoop with variational prompts still improves performance on the new classes and the harmonic mean by $3.25\%$ and $1.6\%$, respectively, with a mere decrease of $0.37\%$ in base accuracy. Figure~\ref{fig:delta-harmonic-mean}(b) shows per-dataset relative harmonic mean improvement. We observe an improvement in 10 out of 11 datasets. Note that the drop in the average base accuracy for \cocoopvpt is negligible compared with \coopvpt. Moreover, our best performing model \cocoopvpt performs better than ProDA~\citep{lu2022prompt} in terms of new classes accuracy and harmonic mean by $\%2.64$ and $\%0.78$. It is also worth noting that \coopvpt and \cocoopvpt excel at CLIP zero-shot performance in base, new and harmonic accuracies.

\begin{table*}[t!]
\centering
\caption{\textbf{Base-to-new generalization} comparison between the state-of-the-art and variational prompt tuning. We average our accuracy over three random seeds. Our proposed model is trained on a few-shot training set (base) and then evaluated on held-out classes (new). As shown, \coop and \cocoop overfit on base classes and do not provide good generalization on new classes. However, our model provides better generalization performance on new classes as well as harmonic mean.}
\vspace{-2mm}
\resizebox{\linewidth}{!}{%
\begin{tabular}{lc|c|ccc|ccc|c}
\toprule
\textbf{Dataset} &
 &
\textbf{CLIP} &
\textbf{\coop} &
\textbf{+VPT} &
\textbf{$\Delta$} &
\textbf{\cocoop} &
\textbf{+VPT} &
\textbf{$\Delta$} &
\textbf{ProDA} \\ \midrule
\multirow{3}{*}{Average}       & Base      & 69.34  & 82.66   & 71.98  &  \color{red}{$-$10.71}            & 80.47  & 80.10  &  \color{red}{$-$00.37} & 81.56\\
                               & New       & 74.22  & 63.22   & 74.76  &  \color{blue}{$+$11.54}           & 71.69  & 74.94  &  \color{blue}{$+$03.25} & 72.30\\
                               & H         & 71.69  & 71.65   & 73.34  &  \textbf{\color{blue}{$+$01.69}}  & 75.83  & 77.43  &  \textbf{\color{blue}{$+$01.60}} & 76.65\\
\midrule
\multirow{3}{*}{ImageNet}      & Base      & 72.43  & 76.14   &  74.73 &  \color{red}{$-$01.41}            & 75.98  & 76.00 &  \color{blue}{$+$00.02} & 75.40\\
                               & New       & 68.14  & 67.88   &  70.60 &  \color{blue}{$+$02.72}           & 70.43  & 70.93 &  \color{blue}{$+$00.50} & 70.23\\
                               & H         & 70.21  & 71.77   &  72.60 &  \textbf{\color{blue}{$+$00.83}}  & 73.10  & 73.37 &  \textbf{\color{blue}{$+$00.27}} & 72.72\\
\midrule
\multirow{3}{*}{Caltech101}    & Base      & 96.84  & 98.00   &  95.47 &  \color{red}{$-$02.53}            &  97.96 & 98.00 &  \color{blue}{$+$00.04} & 98.27\\
                               & New       & 94.00  & 89.81   &  93.80 &  \color{blue}{$+$03.99}           &  93.81 & 94.93 &  \color{blue}{$+$01.12} & 93.23\\
                               & H         & 95.39  & 93.72   &  94.62 &  \textbf{\color{blue}{$+$00.90}}  &  95.84 & 96.44 &  \textbf{\color{blue}{$+$00.60}} & 95.68\\
\midrule
\multirow{3}{*}{OxfordPets}    & Base      & 91.17  & 93.67   & 90.77  &  \color{red}{$-$02.90}            & 95.20  & 95.67 &  \color{blue}{$+$00.47} & 95.43\\
                               & New       & 97.26  & 95.29   & 97.83  &  \color{blue}{$+$02.54}           & 97.69  & 98.00 &  \color{blue}{$+$00.31} & 97.83\\
                               & H         & 94.11  & 94.47   & 94.16  &  \color{red}{$-$00.31}            & 96.43  & 96.82 &  \textbf{\color{blue}{$+$00.39}} & 96.62\\
\midrule
\multirow{3}{*}{StanfordCars}  & Base      & 63.37 & 78.12    & 65.27  &  \color{red}{$-$12.85}            & 70.49  & 72.93 &  \color{blue}{$+$02.44} & 74.70\\
                               & New       & 74.89 & 60.40    & 75.97  &  \color{blue}{$+$15.57}           & 73.59  & 73.23 &  \color{red}{$-$00.36} & 71.20\\
                               & H         & 68.65 & 68.12    & 70.21  &  \textbf{\color{blue}{$+$02.09}}  & 72.01  & 73.07 &  \textbf{\color{blue}{$+$01.06}} & 72.91\\
\midrule
\multirow{3}{*}{Flowers102}    & Base      & 72.08 & 97.60    & 72.97  &  \color{red}{$-$24.63}            & 94.87  & 95.70 &  \color{blue}{$+$00.83} & 97.70\\
                               & New       & 77.80 & 59.67    & 75.90  &  \color{blue}{$+$16.23}           & 71.75  & 70.40 &  \color{red}{$-$01.35} & 68.68\\
                               & H         & 74.83 & 74.06    & 74.40  &  \textbf{\color{blue}{$+$00.34}}  & 81.71  & 81.12 &  \textbf{\color{red}{$-$00.59}} & 80.66\\
\midrule
\multirow{3}{*}{Food101}       & Base      & 90.10 & 88.33    & 90.37  &  \color{blue}{$+$02.04}           & 90.70  & 91.03 &  \color{blue}{$+$00.33} & 90.30\\
                               & New       & 91.22 & 82.26    & 91.67  &  \color{blue}{$+$09.41}           & 91.29  & 92.13 &  \color{blue}{$+$00.84} & 88.57\\
                               & H         & 90.65 & 85.18    & 91.01  &  \textbf{\color{blue}{$+$05.83}}  & 90.99  & 91.57 &  \textbf{\color{blue}{$+$00.58}} & 89.43\\
\midrule
\multirow{3}{*}{FGVCAircraft}  & Base      & 27.19 & 40.44    & 29.57  &  \color{red}{$-$10.87}            & 33.41  & 34.40 &  \color{red}{$-$00.99} & 36.90\\
                               & New       & 36.29 & 22.30    & 33.80  &  \color{blue}{$+$11.50}           & 23.71  & 35.00 &  \color{blue}{$+$11.29} & 34.13\\
                               & H         & 31.08 & 28.74    & 31.54  &  \textbf{\color{blue}{$+$02.80}}  & 27.74  & 34.69 &  \textbf{\color{blue}{$+$06.95}} & 35.46\\
\midrule
\multirow{3}{*}{SUN397}        & Base      & 69.36 & 80.60    &  73.77  & \color{red}{$-$06.83}            & 79.74  & 79.17 &  \color{red}{$-$00.57} & 78.67\\
                               & New       & 75.35 & 65.89    &  77.90  & \color{blue}{$+$12.01}           & 76.86  & 77.87 &  \color{blue}{$+$01.01} & 76.93\\
                               & H         & 72.23 & 72.50    &  75.77  & \textbf{\color{blue}{$+$03.27}}  & 78.27  & 78.51 &  \textbf{\color{blue}{$+$00.24}} & 77.79\\
\midrule
\multirow{3}{*}{DTD}           & Base      & 53.24 & 79.44    & 57.67   & \color{red}{$-$21.77}            & 77.01  & 75.30 &  \color{red}{$-$01.71} & 80.67\\
                               & New       & 59.90 & 41.18    & 58.70   & \color{blue}{$+$17.52}           & 56.00  & 60.80 &  \color{blue}{$+$04.80} & 56.48\\
                               & H         & 56.37 & 54.24    & 58.18   & \textbf{\color{blue}{$+$03.94}}  & 64.85  & 67.27 &  \textbf{\color{blue}{$+$02.42}} & 66.44\\
\midrule
\multirow{3}{*}{EuroSAT}       & Base      & 56.48 & 92.19    & 67.97   & \color{red}{$-$24.22}            & 87.49  & 80.30 &  \color{red}{$-$07.19} & 83.90\\
                               & New       & 64.05 & 54.74    & 71.63   & \color{blue}{$+$16.89}           & 60.04  & 75.30 &  \color{blue}{$+$15.26} & 66.00\\
                               & H         & 60.02 & 68.69    & 69.75   & \textbf{\color{blue}{$+$01.06}}  & 71.21  & 77.71 &  \textbf{\color{blue}{$+$06.50}} & 73.88\\
\midrule
\multirow{3}{*}{UCF101}        & Base      & 70.53  & 84.69   & 73.23   & \color{red}{$-$11.46}            & 82.33  & 82.53 &  \color{blue}{$+$00.20} & 85.23\\
                               & New       & 77.50  & 56.05   & 74.63   & \color{blue}{$+$18.58}           & 73.45  & 75.77 &  \color{blue}{$+$02.32} & 71.97\\
                               & H         & 73.85  & 67.45   & 73.92   & \textbf{\color{blue}{$+$06.47}}  & 77.64  & 79.00 &  \textbf{\color{blue}{$+$01.36}} & 78.04\\
\bottomrule
\end{tabular}%
}
\label{tab:base-to-new}
\vspace{-5mm}
\end{table*}

\begin{figure}[t!]
\centering
\begin{subfigure}{.5\linewidth}
  \centering
  \includegraphics[width=1\linewidth]{images/coop_delta.pdf}
  \caption{$\Delta(\text{H})=\text{H}(\text{CoOp+VPT}) - \text{H}(\text{CoOp})$}
  \label{fig:delta-harmonic-mean-coop}
\end{subfigure}%
\begin{subfigure}{.5\linewidth}
  \centering
  \includegraphics[width=1\linewidth]{images/cocoop_delta.pdf}
  \caption{$\Delta(\text{H})=\text{H}(\text{CoCoOp+VPT}) - \text{H}(\text{CoCoOp})$}
  \label{fig:delta-harmonic-mean-cocoop}
\end{subfigure}
\caption{\textbf{Relative enhancement of variational prompt tuning} over \coop and \cocoop in terms of harmonic mean over 11 datasets for 3 distinct random seeds. Variational prompt tuning improves the harmonic mean for all baselines other than OxfordPets for \coop and Flowers102 for CoCoOp.}
\vspace{-3mm}
\label{fig:delta-harmonic-mean}
\end{figure}

\subsection{Cross-dataset transfer learning}

\textbf{Setup.} For cross-dataset transfer learning, the model is trained on a source dataset (ImageNet) and then assessed on 10 distinct target datasets. This experiment tries to determine how effectively our methods generalizes transfer beyond the scope of a single dataset.

\textbf{Results.} As reported in Table~\ref{tab:cross-dataset-transfer}, \coopvpt has a drop in performance in ImageNet by $1.76\%$ while outperforming the target dataset on average by $1.63\%$. Moreover, our proposed method leads to an increase on 9 out of 10 target datasets, with a small drop in accuracy of $0.03\%$ in Caltech101. Note that on target datasets such as FGVCAircraft and DTD, our proposed method achieves an improvement of more than $3\%$.
Similarly to CoOp, augmenting \cocoop with our method still leads to an overall performance enhancement of $0.16\%$ on 7 out of 10 target datasets, showing its effectiveness for cross-dataset transfer learning. In addition, unlike \cocoop, which has better performance in ImageNet-like datasets such as Caltech101 and OxfordPets, our proposed method exhibits improvement on dissimilar datasets (e.g. FGVCAircraft, DTD, and EuroSAT), demonstrating its capacity to capture the unique characteristics of each dataset.

\begin{table*}[htbp]
\centering
\caption{\textbf{Cross-dataset transfer learning} comparison between the state-of-the-art and our variational prompt tuning in terms of average accuracy on three different random seeds. Following~\citep{zhou2022conditional}, our proposed model is trained on a source dataset and evaluated on target datasets. As shown, variational prompt tuning performs better than other baselines 16 out of 20 datasets, although it loses performance on the source dataset.}
\vspace{-2mm}
\resizebox{1\linewidth}{!}{%
\begin{tabular}{lcccccccccccc}
\toprule
 &
\multicolumn{1}{c}{Source} &
\multicolumn{10}{c}{Target}\\  \cmidrule(lr){2-2} \cmidrule(lr){3-13}
\textbf{Methods} & \textbf{\rot{ImageNet}} & \textbf{\rot{Caltech101}} & \textbf{\rot{OxfordPets}} & \textbf{\rot{StanfordCars}} & \textbf{\rot{Flowers102}} & \textbf{\rot{Food101}} & \textbf{\rot{FGVCAircraft}} & \textbf{\rot{SUN397}} & \textbf{\rot{DTD}} & \textbf{\rot{EuroSAT}} & \textbf{\rot{UCF101}} & \textbf{\rot{Average}} \\ \midrule
\coop       & 71.51  & 93.70 & 89.14  & 64.51   & 68.71  & 85.30  & 18.47  & 64.15  & 41.92  & 46.39  & 66.55  & 63.88\\
+VPT        & 69.73  & 93.67 & 89.27  & 65.50   & 70.20  & 86.27  & 22.13  & 66.57  & 46.93  & 47.43  & 67.2  & 65.51  \\
\rowcolor{Gray}
$\Delta$   &\color{red}{$-$1.78} & \color{red}{$-$0.03} & \color{blue}{$+$0.13} & \color{blue}{$+$0.99} & \color{blue}{$+$1.49} & \color{blue}{$+$0.97} & \color{blue}{$+$3.66} & \color{blue}{$+$2.42} & \color{blue}{$+$5.01} & \color{blue}{$+$1.04} & \color{blue}{$+$0.65} & \textbf{\color{blue}{$+$1.63}}
\\\midrule
\cocoop     & 71.02  & 94.43 & 90.14  & 65.32   & 71.88  & 86.06  & 22.94  & 67.36  & 45.73  & 45.37  & 68.21  & 65.74\\
+VPT        & 70.70   & 93.67 & 90.63  & 65.00   & 70.90  & 86.30  & 24.93  & 67.47  & 46.10  & 45.87  & 68.67  & 65.95\\
\rowcolor{Gray}
$\Delta$    & \color{red}{$-$0.32} & \color{red}{$-$0.76} & \color{blue}{$+$0.49} & \color{red}{$-$0.32} & \color{red}{$-$0.98} & \color{blue}{$+$0.24} & \color{blue}{$+$1.99} & \color{blue}{$+$0.11} & \color{blue}{$+$0.37} & \color{blue}{$+$0.50} & \color{blue}{$+$0.46} & \textbf{\color{blue}{$+$0.16}}\\
\bottomrule
\end{tabular}%
}
\label{tab:cross-dataset-transfer}
\vspace{-3mm}
\end{table*}

\subsection{Domain generalization}
\textbf{Setup.} Lastly, We examine variational prompt tuning through the lens of distribution shift and robustness. We train our proposed model on the source dataset (ImageNet) for three different random seeds, and assess it on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R. Prior work such as \coop \citep{zhou2022learning} and \cocoop \citep{zhou2022conditional} demonstrate empirically that learning a soft-prompt improves the model's resilience against distribution shift and adversarial attack. Following their experiments, we are also interested in determining if treating prompts in a variational manner maintain or improve the performance.

\textbf{Results.} As reported in Table~\ref{tab:generalization} our method enhances the accuracy of \coop on ImageNet-Sketch, ImageNet-A, and ImageNet-R by $0.88\%$, $0.70\%$, and $2.19\%$ while degrading the performance on ImageNet and ImageNetV2 by $1.78\%$, by $1.03\%$. However, on CoCoOp, adding VPT, while losing the performance on source dataset similar to CoOp, consistently improves the accuracy on all target datasets which highlight the effectiveness of our proposed method.

\begin{table*}[htbp]
\centering
\caption{\textbf{Domain generalization} comparison between the state-of-the-art and variational prompt tuning in terms of average accuracy on three different random seeds. Our proposed model is trained on a source dataset and evaluated on target classes. Variational prompt tuning outperforms alternative baselines on the target datasets while losing performance on the source dataset.}
\vspace{-2mm}
\resizebox{1\linewidth}{!}{%
\begin{tabular}{lcccccc}
\toprule
 &
 &
\multicolumn{1}{c}{Source} &
\multicolumn{4}{c}{Target}\\  \cmidrule(lr){3-3} \cmidrule(lr){4-7}
 \textbf{Methods} & \textbf{Learnable} & \textbf{ImageNet} & \textbf{ImageNetV2} & \textbf{ImageNet-Sketch} & \textbf{ImageNet-A} & \textbf{ImageNet-R} \\ \midrule
CLIP       & \xmark & 66.73 & 60.83 & 46.15 & 47.77 & 73.96 \\\midrule
\coop       & \cmark & 71.51 & 64.20	& 47.99	& 49.71	& 75.21 \\
\rowcolor{Gray}
+VPT       & \cmark & 69.73 & 63.17 & 48.87 & 50.77 & 77.40 \\
$\Delta$   &    -   & \color{red}{$-$1.78} & \color{red}{$-$1.03} & \color{blue}{$+$0.88}  & \color{blue}{$+$0.70}  & \color{blue}{$+$2.19}  \\\midrule
\cocoop       & \cmark & 71.02 & 64.07 & 48.75 & 50.63 & 76.18 \\
\rowcolor{Gray}
+VPT       & \cmark & 70.70	& 64.23	& 49.20	& 51.33	& 77.00 \\
$\Delta$   &    -   & \color{red}{$-$0.32} & \color{blue}{$+$0.16}  & \color{blue}{$+$0.45}  & \color{blue}{$+$0.70}  & \color{blue}{$+$0.82}  \\
\bottomrule
\end{tabular}%
}

\label{tab:generalization}
\vspace{-3mm}
\end{table*}

\subsection{Ablations}
\label{ablation}

\textbf{Effectiveness of the posterior distribution $q_\phi$.}
We first ablate the effectiveness of the variational posterior distribution. To do this, we consider sampling one residual vector from the uniform distribution
$\mathcal{U}(\text{0},\text{1})$, normal distribution $\mathcal{N}(\text{0},\text{I})$, normal distribution $\mathcal{N}( \mu(\mathbf{x}), \text{0})$, normal distribution $\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$, and report the new class accuracy for \cocoopvpt for one random seed in Table~\ref{tab:ablation-posterior}.
Except for the EuroSAT dataset, a sample from the normal distribution $\mathcal{N}( \mu(\mathbf{x}), \text{0})$ obtains the best performance in comparison with alternatives, showing that the mean of the normal distribution $\mu(\mathbf{x})$ is the most effective sample.
In addition, we find that drawing one sample from $\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$ yields superior results compared to drawing one sample from uniform distribution $\mathcal{U}(\text{0},\text{1})$ and normal distribution $\mathcal{N}(\text{0},\text{I})$, further demonstrating the efficacy of our proposed method in capturing the underlying distribution of the prompt space.
We also ablate increasing the number of samples from the normal distribution $\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$ to understand the informativeness of the learned variational distribution. It is shown that enlarging the number of samples further improves the model performance as they capture the prompt space appropriately.

\begin{table*}[htbp]
\centering
\caption{\textbf{Effectiveness of the posterior distribution}.  The informative posterior distribution $\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$ outperforms the two uninformative distributions  $\mathcal{U}(\text{0},\text{1})$ and $\mathcal{N}(\text{0},\text{I})$ by a large margin for all datasets. Increasing the number of samples further improves results.
}
\vspace{-2mm}
\resizebox{0.95\linewidth}{!}{%
\begin{tabular}{lcccccc}
\toprule
 \textbf{Methods} & \textbf{Samples} & \textbf{DTD} & \textbf{Flowers102} & \textbf{EuroSAT} & \textbf{FGVCAircraft} & \textbf{UCF101} \\ \midrule
$\mathcal{U}(\text{0},\text{1})$                       & 1   & 33.20   & 45.30    & 54.20 & 10.50 & 55.70 \\
$\mathcal{N}(\text{0},\text{I})$                        & 1   & 26.60   & 36.10	& 50.00	&  07.70	& 48.80 \\
$\mathcal{N}( \mu(\mathbf{x}), 0)$               & 1   & 59.80   & 73.50    & 59.90 & 34.10 & 76.50 \\
$\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$     & 1   & 56.40   & 72.30	& 64.50	& 33.00	& 75.60 \\
\midrule
$\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$     & 2   & 60.00   & 73.90	& 67.40	& 33.90	& 76.20 \\
$\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$     & 5   & 62.20   & 74.00	& 71.00	& 34.20	& 76.60 \\
$\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$     & 10  & 61.60   & 73.50	& 73.60	& 34.40	& 77.00 \\

\bottomrule
\end{tabular}%
}
\label{tab:ablation-posterior}
\vspace{-2mm}
\end{table*}

\textbf{Prompt initialization.}
We investigate the effectiveness of the prompt initialization on the new class accuracy for \cocoopvpt for one random seed. We consider two variants. In the first one, we initialize the context tokens randomly using a normal distribution, whereas in second one we initialize the context tokens with ``An image of a $\{\texttt{class}\}$''. Table~\ref{tab:ablation-prompt-init} summarizes this ablation. Comparing the two variants demonstrates that an appropriately initialized prompt consistently outperforms a randomly initialized prompt, highlighting the necessity for further research of the prompt space.

\begin{table*}[htbp]
\centering
\caption{\textbf{Prompt initialization.} Initializing the context tokens with an appropriate prompt ``An image of a $\{\texttt{class}\}$'' improves the performance compared to random tokens.}
\vspace{-2mm}
\resizebox{0.9\linewidth}{!}{%
\begin{tabular}{lccccc}
\toprule
\textbf{Method} & \textbf{DTD} & \textbf{Flowers102} & \textbf{EuroSAT} & \textbf{FGVCAircraft} & \textbf{UCF101} \\
\midrule
Random                             &  74.10& 67.23& 75.10 & 34.30 & 81.30 \\
``An image of a $\{\texttt{class}\}$''     &  74.80& 70.05& 77.90 & 35.50 & 82.50 \\
\bottomrule
\end{tabular}%
}
\label{tab:ablation-prompt-init}
\end{table*}

\begin{wrapfigure}{r}{0.4\textwidth}

  \begin{center}
    \vspace{-0.7cm}
    \includegraphics[width=0.4\textwidth]{images/iclr-context-MC-new.pdf}
    \vspace{-0.7cm}
  \end{center}
  \caption{\textbf{Effectiveness of Monte Carlo samples} on new accuracy. As demonstrated, increasing the number of Monte Carlo samples boosts performance initially but reaches a plateau after 10 samples for all datasets.\vspace{-1cm}}
  \label{fig:MC-new}

\end{wrapfigure}
\textbf{Number of Monte-Carlo samples.}
When approximating the log-likelihood of input data, the number of Monte Carlo samples is an important hyperparameter. Generally, a large number of samples should lead to a better approximation and better classification accuracy. We ablate this hyperparameter on new accuracy for \cocoopvpt by varying the number of Monte Carlo samples at inference time. We show results for a varying number of samples in Figure~\ref{fig:MC-new} for DTD, Flowers102, EuroSAT, FGVCAircraft, and UCF101. Increasing the Monte Carlo samples from $1$ to $10$  consistently improves the new accuracy,  afterwards the accuracy saturates. Hence, we recommend evaluating variational prompt tuning on a larger number of Monte Carlo samples for better model accuracy.

For more ablations, see Section \ref{more_ablation} in the appendix.

\section{Conclusion}
In this paper, we introduce variational prompt tuning, a probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochastic sampling.
By doing so, we are able to generate adaptive context tokens for each data point.
This formulation leads to better generalization capabilities in terms of the new accuracy and harmonic mean for downstream tasks.
We show that it can be seamlessly integrated into both standard and conditional prompt learning frameworks, considerably improving the performance in both cases.
We conduct extensive experiments on 15 datasets and demonstrate the benefits of a variational formulation in learning data-driven prompts. Our method provides the current state-of-the-art for prompt learning, and constitutes, to the best of our knowledge, the first method for CLIP adaptation that fully maintains the generalization capability to new classes of the original model.

\section*{Ethics Statement}
Our method has the potential to affect applications that often need rapid adaptation, such as medical imaging, astronomical imaging, and autonomous driving.
Because of this, there may be negative societal consequences associated with the adoption of our technology. For example, a lack of fairness with models trained on insufficient data, regulatory compliance, patient privacy in medical imaging, and more general biases encoded into large vision-language models such as CLIP.

\section*{Reproducibility Statement}
Details on benchmarks, metrics, and the implementation of variational prompt tuning in terms of architecture and training are contained in Section \ref{sec:experiments}.
We also list of all hyperparameters in Table~\ref{tab:shared_hparams} and~\ref{tab:dataset_hparams}.
Lastly, we will release the source code, scripts to reproduce the results and evaluating the performance of the model at: \url{https://github.com/<redacted>/<redacted>}.