\section{Introduction}\label{sec:int}
\setcounter{equation}{0}

The {\em Yangian} $\Y(\osp_{M|2n})$ associated with the orthosymplectic Lie superalgebra $\osp_{M|2n}$
is a deformation of the universal enveloping algebra $\U(\osp_{M|2n}[u])$ in the class of
Hopf algebras. The original definition is due to Arnaudon {\it et al.\/}~\cite{aacfr:rp},
where some basic properties of the Yangian were also described. Its
$\gl_{m|n}$ counterpart was introduced earlier by Nazarov~\cite{n:qb}, and
the finite-dimensional irreducible representations of the Yangian $\Y(\gl_{m|n})$
were classified by Zhang~\cite{zh:sy} in a way similar to the
Drinfeld Yangians \cite{d:nr}. Apart from the case of the Yangian $\Y(\osp_{1|2n})$
considered in \cite{m:ry}, the general
classification problem for the orthosymplectic Yangians remains open.
We show in this paper that the
representations of the Yangian $\Y(\osp_{2|2n})$ are
described in essentially the same way as those of the Yangian $\Y(\gl_{m|n})$,
thus extending the similarity between the modules over the Lie superalgebras
of types $A$ and $C$ as given by Kac~\cite{k:rc}.

In more detail, we will regard the Yangian
$\Y(\osp_{2|2n})$
as a quotient
of the {\em extended Yangian} $\X(\osp_{2|2n})$ by an ideal generated by central elements.
The highest weight representation $L(\la(u))$ of $\X(\osp_{2|2n})$ is defined as
the irreducible quotient of the Verma module $M(\la(u))$
associated with an $(n+2)$-tuple $\la(u)=(\la_1(u),\dots,\la_{n+2}(u))$ of formal series
in $u^{-1}$. The tuple is called the {\em highest weight}
of the representation.

A standard argument shows that every finite-dimensional irreducible representation
of the extended Yangian is of the form $L(\la(u))$ for a certain highest weight $\la(u)$.
The key step in the classification is to determine
the necessary and sufficient conditions on the highest weight for
the representation $L(\la(u))$
to be finite-dimensional. We will rely
on reduction properties of the representations
to establish necessary conditions, and
their sufficiency will be verified by constructing a family of
{\em fundamental representations} of the Yangian $\X(\osp_{2|2n})$.
This will prove the following theorem.

\begin{mthm}\label{thm:yclassi}
Every finite-dimensional irreducible representation of the algebra $\X(\osp_{2|2n})$
is isomorphic to $L(\la(u))$ for a certain highest weight $\la(u)$.
The representation $L(\la(u))$
is finite-dimensional if and only if
there exist monic polynomials
$\ol Q(u),Q(u),P_2(u),\dots,P_{n+1}(u)$ in
$u$ such that
\begin{align}
\frac{\la_1(u)}{\la_{2}(u)}&=\frac{\ol Q(u)}{Q(u)},
\label{gloo}\\[0.4em]
\frac{\la_{i+1}(u)}{\la_i(u)}&=\frac{P_i(u+1)}{P_i(u)}\qquad\text{for}\quad i=2,\dots,n,
\label{spao}\\
\intertext{and}
\frac{\la_{n+2}(u)}{\la_{n+1}(u)}&=\frac{P_{n+1}(u+2)}{P_{n+1}(u)}.
\label{spala}
\end{align}
The finite-dimensional irreducible
representations of the Yangian $\Y(\osp_{2|2n})$ are in a one-to-one correspondence with
the tuples $\big(\ol Q(u),Q(u),P_2(u),\dots,P_{n+1}(u)\big)$ of monic
polynomials in $u$, where the polynomials
$\ol Q(u)$ and $Q(u)$ are of the same degree and have no common roots.
\end{mthm}

We will refer to the monic
polynomials occurring in these conditions as
the {\em Drinfeld polynomials} of the representation; cf. \cite{d:nr}.

The Appendix is devoted to the polynomial evaluation modules over the Yangian $\Y(\gl_{m|n})$.
Such modules over the Yangian $\Y(\gl_n)$
proved to be useful in the analysis of the representations of the orthosymplectic Yangians;
see Proposition~\ref{prop:xikdp} and also \cite{amr:rp}, \cite{m:ry}.
Many applications of the evaluation homomorphism rely on its $R$-matrix interpretation
going back to Cherednik~\cite{c:ni}. In this interpretation, the polynomial evaluation modules
arise from the Schur--Weyl duality and are constructed as submodules of
the tensor product of the vector representation of the general linear
Lie algebra. As another ingredient, the construction uses
some key steps of the {\em fusion procedure} for the symmetric group
originated in the work of Jucys~\cite{j:yo}. It was
rediscovered in \cite{c:ni}, with detailed arguments given in \cite{n:yc}; see also \cite[Sec.~6.4]{m:yc}.

A super-version of the Schur--Weyl duality involving the general linear Lie superalgebra and
the symmetric group is due to Sergeev~\cite{s:ta} and Berele and Regev~\cite{br:hy}.
It is called the {\em Schur--Sergeev duality} in \cite[Sec.~3.2]{cw:dr}.
We will use this duality to
give an $R$-matrix construction of the polynomial evaluation modules over the Yangian
associated with the Lie superalgebra $\gl_{m|n}$. More precisely,
for any Young diagram $\lambda$ contained in the $(m,n)$-hook we consider
a standard $\la$-tableau $\Uc$ and the associated primitive idempotent $e_{\Uc}\in \CC\Sym_d$,
where $d$ is the number of boxes in $\la$.
By using the action of the symmetric group $\Sym_d$ on the tensor product of the $\ZZ_2$-graded
vector spaces $\CC^{m|n}$, we show that the
subspace $e_{\Uc}(\CC^{m|n})^{\ot d}$ is a representation of
the Yangian $\Y(\gl_{m|n})$. We derive that this representation
and its twisted version are isomorphic to
evaluation modules over $\gl_{m|n}$ and identify them by calculating
the highest weights.

\section{Representations of the orthosymplectic Yangian}
\label{sec:ns}

\subsection{Definition and basic properties of the Yangian}
\label{subsec:db}

Fix an integer $n\geqslant 1$ and introduce the
involution $i\mapsto i\pr=2n-i+3$ on
the set $\{1,2,\dots,2n+2\}$.
Consider the $\ZZ_2$-graded vector space $\CC^{2|2n}$ over $\CC$ with the
canonical basis
$e_1,e_2,\dots,e_{2n+2}$, where
the vector $e_i$ has the parity
$\bi\mod 2$ and
\ben
\bi=\begin{cases} 0\qquad\text{for}\quad i=1,1',\\
1\qquad\text{for}\quad i=2,3,\dots,2\pr.
\end{cases}
\een
The endomorphism algebra $\End\CC^{2|2n}$ gets a $\ZZ_2$-gradation with
the parity of the matrix unit $e_{ij}$ found by
$\bi+\bj\mod 2$. We will identify
the algebra of
even matrices over a superalgebra $\Ac$ with the tensor product algebra
$\End\CC^{2|2n}\ot\Ac$, so that a matrix $A=[a_{ij}]$ is regarded as the element
\ben
A=\sum_{i,j=1}^{2n+2}e_{ij}\ot a_{ij}(-1)^{\bi\tss\bj+\bj}\in \End\CC^{2|2n}\ot\Ac.
\een
The involutive matrix {\em super-transposition} $t$ is defined by
$(A^t)_{ij}=A_{j'i'}(-1)^{\bi\bj+\bj}\tss\ta_i\ta_j$,
where we set
\ben
\ta_i=\begin{cases} \phantom{-}1\qquad\text{for}\quad i=1,\dots,n+1,1',\\
-1\qquad\text{for}\quad i=n+2,\dots,2\pr.
\end{cases}
\een
This super-transposition is associated with the bilinear form on the space $\CC^{2|2n}$
defined by the anti-diagonal matrix $G=[\de_{ij'}\tss\ta_i]$.
We will also regard $t$ as the linear map
\beql{suptra}
t:\End\CC^{2|2n}\to \End\CC^{2|2n}, \qquad
e_{ij}\mapsto e_{j'i'}(-1)^{\bi\bj+\bi}\tss\ta_i\ta_j.
\eeq

A standard basis of the general linear Lie superalgebra $\gl_{2|2n}$ is formed by elements $E_{ij}$
of the parity $\bi+\bj\mod 2$ for $1\leqslant i,j\leqslant 2n+2$ with the commutation relations
\ben
[E_{ij},E_{kl}]
=\de_{kj}\ts E_{i\tss l}-\de_{i\tss l}\ts E_{kj}(-1)^{(\bi+\bj)(\bk+\bl)}.
\een
We will regard the orthosymplectic Lie superalgebra $\osp_{2|2n}$
associated with the bilinear from defined by $G$ as the subalgebra
of $\gl_{2|2n}$ spanned by the elements
\ben
F_{ij}=E_{ij}-E_{j'i'}(-1)^{\bi\tss\bj+\bi}\ts\ta_i\ta_j.
\een
The symplectic Lie algebra $\osp_{0|2n}\cong \spa_{2n}$ will be considered as the subalgebra
of $\osp_{2|2n}$ spanned by the elements $F_{ij}$ with $2\leqslant i,j\leqslant 2'$.

Introduce the permutation operator $P$ by
\ben
P=\sum_{i,j=1}^{2n+2} e_{ij}\ot e_{ji}(-1)^{\bj}\in \End\CC^{2|2n}\ot\End\CC^{2|2n}
\een
and set
\ben
Q=\sum_{i,j=1}^{2n+2} e_{ij}\ot e_{i'j'}(-1)^{\bi\bj}\ts\ta_i\ta_j
\in \End\CC^{2|2n}\ot\End\CC^{2|2n}.
\een
The $R$-{\em matrix} associated with $\osp_{2|2n}$ is the
rational function in $u$ given by
\beql{ru}
R(u)=1-\frac{P}{u}+\frac{Q}{u-\ka},\qquad \ka=-n.
\eeq
This is a super-version of the $R$-matrix
originally found in \cite{zz:rf}.
Following \cite{aacfr:rp}, we
define the {\it extended Yangian\/}
$\X(\osp_{2|2n})$
as a $\ZZ_2$-graded algebra with generators
$t_{ij}^{(r)}$ of parity $\bi+\bj\mod 2$, where $1\leqslant i,j\leqslant 2n+2$ and $r=1,2,\dots$,
satisfying certain quadratic relations. To write them down,
introduce the formal series
\beql{tiju}
t_{ij}(u)=\de_{ij}+\sum_{r=1}^{\infty}t_{ij}^{(r)}\ts u^{-r}
\in\X(\osp_{2|2n})[[u^{-1}]]
\eeq
and combine them into the matrix $T(u)=[t_{ij}(u)]$.
Consider the elements of the tensor product algebra
$\End\CC^{2|2n}\ot\End\CC^{2|2n}\ot \X(\osp_{2|2n})[[u^{-1}]]$ given by
\ben
T_1(u)=\sum_{i,j=1}^{2n+2} e_{ij}\ot 1\ot t_{ij}(u)(-1)^{\bi\tss\bj+\bj}\Fand
T_2(u)=\sum_{i,j=1}^{2n+2} 1\ot e_{ij}\ot t_{ij}(u)(-1)^{\bi\tss\bj+\bj}.
\een
The defining relations for the algebra $\X(\osp_{2|2n})$ take
the form of the $RTT$-{\em relation}
\beql{RTT}
R(u-v)\ts T_1(u)\ts T_2(v)=T_2(v)\ts T_1(u)\ts R(u-v).
\eeq
As shown in \cite{aacfr:rp}, the product $T(u-\ka)\ts T^{\tss t}(u)$ is a scalar matrix with
\beql{ttra}
T(u-\ka)\ts T^{\tss t}(u)=c(u)\tss 1,
\eeq
where $c(u)$ is a series in $u^{-1}$. As with the Lie algebra case considered in \cite{amr:rp},
all its coefficients belong to
the center $\ZX(\osp_{2|2n})$ of $\X(\osp_{2|2n})$ and generate the center.

We will also use the {\em extended Yangian} $\X(\osp_{0|2n})$ which is defined
in the same way as $\X(\osp_{2|2n})$, by using the subspace $\CC^{0|2n}\subset\CC^{2|2n}$
with odd basis vectors $e_2,e_3,\dots,e_{2'}$, instead of $\CC^{2|2n}$
and the corresponding $RTT$-relation \eqref{RTT}.
The $R$-matrix is defined
by \eqref{ru} with the modified expressions
\ben
P=-\sum_{i,j=2}^{2n+1} e_{ij}\ot e_{ji}
\Fand
Q=-\sum_{i,j=2}^{2n+1} e_{ij}\ot e_{i'j'}\ts\ta_i\ta_j
\een
and with the value $\ka=-n-1$. It will be convenient to use
the index set $\{2,3,\dots,2'\}$ to label the corresponding generating series
which we will denote by $\bar t_{ij}(u)$. We have an isomorphism
$\X(\osp_{0|2n})\cong \X(\spa_{2n})$ so that the series $\bar t_{ij}(-u)$
satisfy the defining relations for the extended Yangian
$\X(\spa_{2n})$; see \cite{aacfr:rp},
\cite{amr:rp}.

The {\em Yangian} $\Y(\osp_{2|2n})$
is defined as the subalgebra of
$\X(\osp_{2|2n})$ which
consists of the elements stable under
the automorphisms
\beql{muf}
t_{ij}(u)\mapsto f(u)\ts t_{ij}(u)
\eeq
for all series
$f(u)\in 1+u^{-1}\CC[[u^{-1}]]$.
We have the tensor product decomposition
\beql{tensordecom}
\X(\osp_{2|2n})=\ZX(\osp_{2|2n})\ot \Y(\osp_{2|2n}).
\eeq
The Yangian $\Y(\osp_{2|2n})$ is isomorphic to the quotient
of $\X(\osp_{2|2n})$
by the relation $c(u)=1$.

An explicit form of the defining relations \eqref{RTT} can be written
with the use of super-commutator in terms of the series \eqref{tiju} as follows:
\begin{align}
\big[\tss t_{ij}(u),t_{kl}(v)\big]&=\frac{1}{u-v}
\big(t_{kj}(u)\ts t_{il}(v)-t_{kj}(v)\ts t_{il}(u)\big)
(-1)^{\bi\tss\bj+\bi\tss\bk+\bj\tss\bk}
\non\\
{}&-\frac{1}{u-v-\ka}
\Big(\de_{k i\pr}\sum_{p=1}^{2n+2}\ts t_{pj}(u)\ts t_{p'l}(v)
(-1)^{\bi+\bi\tss\bj+\bj\tss\bp}\ts\ta_i\ta_p
\label{defrel}\\
&\qquad\qquad\qquad
{}-\de_{l j\pr}\sum_{p=1}^{2n+2}\ts t_{k\tss p'}(v)\ts t_{ip}(u)
(-1)^{\bj+\bp+\bi\tss\bk+\bj\tss\bk+\bi\tss\bp}\ts\ta_j\ta_p\Big).
\non
\end{align}
Note that the mapping
\beql{shift}
t_{ij}(u)\mapsto t_{ij}(u+a),\quad a\in \CC,
\eeq
defines an automorphism of $\X(\osp_{2|2n})$.

The universal enveloping algebra $\U(\osp_{2|2n})$ can be regarded as a subalgebra of
$\X(\osp_{2|2n})$ via the embedding
\beql{emb}
F_{ij}\mapsto \frac12\big(t_{ij}^{(1)}-t_{j'i'}^{(1)}(-1)^{\bj+\bi\bj}\ts\ta_i\ta_j\big)(-1)^{\bi}.
\eeq
This fact relies on the Poincar\'e--Birkhoff--Witt theorem for the orthosymplectic Yangian
which was pointed out in \cite{aacfr:rp}.
It states that the associated graded algebra
for $\Y(\osp_{2|2n})$ is isomorphic to $\U(\osp_{2|2n}[u])$.
This implies that the algebra $\X(\osp_{2|2n})$ is generated by
the coefficients of the series $c(u)$ and $t_{ij}(u)$ with $i+j\leqslant 2n+3$,
excluding $t_{11'}(u)$ and $t_{1'1}(u)$. Moreover, given any total ordering
on the set of the generators, the ordered monomials with the powers of odd generators
not exceeding $1$, form a basis of the algebra.
The theorem follows by using essentially the same
arguments as in the non-super case; see \cite[Sec.~3]{amr:rp}.

The extended Yangian $\X(\osp_{2|2n})$ is a Hopf algebra with the coproduct
defined by
\beql{Delta}
\De: t_{ij}(u)\mapsto \sum_{k=1}^{2n+2} t_{ik}(u)\ot t_{kj}(u).
\eeq
The coproduct on the algebra $\X(\osp_{0|2n})$ is defined by the same formula
for the series $\bar t_{ij}(u)$ instead of $t_{ij}(u)$, with the sum taken
over $k=2,3,\dots,2n+1$.

\subsection{Highest weight representations}
\label{subsec:hw}

The proof of the necessity of the conditions of the Main Theorem will rely on the following
reduction property for representations of the extended Yangians $\X(\osp_{2|2n})$
which is verified in the same way as for the case of $\X(\osp_{1|2n})$ in \cite[Prop.~4.1]{m:ry};
cf. \cite[Lemma~5.13]{amr:rp}.
For an arbitrary $\X(\osp_{2|2n})$-module $V$ set
\beql{vplus}
V^+=\{\eta\in V\ |\ t_{1j}(u)\ts\eta=0\quad\text{for}\quad j>1\fand
t_{i\tss 1'}(u)\ts\eta=0\quad\text{for}\quad i<1'\}.
\eeq

\bpr\label{prop:vplus}
The subspace $V^+$ is stable under the action of the operators $t_{ij}(u)$
subject to $2\leqslant i,j\leqslant 2'$.
Moreover, the assignment $\bar t_{ij}(u)\mapsto t_{ij}(u)$
defines
a representation of the extended Yangian $\X(\osp_{0|2n})$ on $V^+$.
\qed
\epr

A representation $V$ of the algebra $\X(\osp_{2|2n})$
is called a {\em highest weight representation}
if there exists a nonzero vector
$\xi\in V$ such that $V$ is generated by $\xi$,
\begin{alignat}{2}
t_{ij}(u)\ts\xi&=0 \qquad &&\text{for}
\quad 1\leqslant i<j\leqslant 2n+2, \qquad \text{and}\non\\
t_{ii}(u)\ts\xi&=\la_i(u)\ts\xi \qquad &&\text{for}
\quad i=1,\dots,2n+2,
\label{trianb}
\end{alignat}
for some formal series
\beql{laiu}
\la_i(u)\in 1+u^{-1}\CC[[u^{-1}]].
\eeq
The vector $\xi$ is called the {\em highest vector}
of $V$.

\bpr\label{prop:nontrvm}
The series $\la_i(u)$ associated with a highest weight representation $V$
satisfy
the consistency conditions
\beql{nontrvm}
\la_i(u)\tss \la_{i\pr}(u+n-i+2)=\la_{i+1}(u)\tss \la_{(i+1)'}(u+n-i+2)
\eeq
for $i=1,\dots,n$.
Moreover, the coefficients of the series $c(u)$ act in the representation
$V$ as the multiplications by scalars
determined by
$
c(u)\mapsto \la_1(u)\tss \la_{1'}(u+n).
$
\epr

\bpf
Introduce the subspace $V^+$ by \eqref{vplus} and note that
the vector $\xi$ belongs to $V^+$. By applying Proposition~\ref{prop:vplus} we find that
the cyclic span
$\X(\osp_{0|2n})\ts\xi$ is a highest weight submodule with the highest
weight $(\la_2(u),\dots,\la_{2'}(u))$.
Due to the isomorphism
$\X(\osp_{0|2n})\cong \X(\spa_{2n})$, conditions
\eqref{nontrvm} for $i=2,\dots,n$ follow from the consistency conditions
for the highest weight modules over $\X(\spa_{2n})$; see \cite[Prop.~5.14]{amr:rp}.
Furthermore, using the defining relations \eqref{defrel}, we get
\ben
t_{12}(u)\ts t_{1'2'}(v)\ts \xi=
-\frac{1}{u-v-\kappa}\ts\Big(t_{12}(u)\ts t_{1'2'}(v)+
\la_{1}(u)\ts \la_{1'}(v)-\la_{2}(u)\ts \la_{2'}(v)\Big)\ts \xi
\een
and so
\ben
(u-v-\kappa+1)\ts t_{12}(u)\ts t_{1'2'}(v)\ts \xi
=\big({-}\la_{1}(u)\ts \la_{1'}(v)+\la_{2}(u)\ts \la_{2'}(v)\big)\ts \xi.
\een
Setting $v=u-\kappa+1=u+n+1$, we obtain
\eqref{nontrvm} for $i=1$. Finally, the last part of the proposition
is obtained by using the expression for $c(u)$ implied by taking the $(1',1')$ entry in
the matrix relation \eqref{ttra}.
\epf

By Proposition~\ref{prop:nontrvm}, the series $\la_i(u)$ in \eqref{trianb}
with $i>n+2$ are uniquely
determined by the first $n+2$ series. The corresponding $(n+2)$-tuple
$\la(u)=(\la_{1}(u),\dots,\la_{n+2}(u))$
will be called the {\em highest weight\/} of $V$.

For an arbitrary $(n+2)$-tuple $\la(u)=(\la_{1}(u),\dots,\la_{n+2}(u))$
of formal series of the form \eqref{laiu}
define
the {\em Verma module} $M(\la(u))$ as the quotient of the algebra $\X(\osp_{2|2n})$ by
the left ideal generated by all coefficients of the series $t_{ij}(u)$
with $1\leqslant i<j\leqslant 2n+2$, and $t_{ii}(u)-\la_i(u)$ for
$i=1,\dots,2n+2$, assuming that the series $\la_i(u)$
with $i=n+3,\dots,2n+2$ are defined to satisfy the consistency conditions \eqref{nontrvm}.

The Poincar\'e--Birkhoff--Witt theorem for the algebra $\X(\osp_{2|2n})$
implies that the Verma module $M(\la(u))$
is nonzero, and we denote by $L(\la(u))$ its irreducible quotient.
It is clear that the isomorphism
class of $L(\la(u))$ is determined by $\la(u)$.

The first part of the Main Theorem is implied by the following proposition
whose proof is the same as in the non-super case; see \cite[Thm~5.1]{amr:rp}.

\bpr\label{prop:fdhw}
Every finite-dimensional irreducible representation of the algebra $\X(\osp_{2|2n})$
is a highest weight representation.
Moreover, it contains a unique, up to a constant factor, highest vector.
\qed
\epr

We will now suppose that the representation $L(\la(u))$ of $\X(\osp_{2|2n})$
is finite-dimensional and derive the conditions on the highest weight $\la(u)$ given
in the Main Theorem. First consider
the subalgebra $\Y_0$ of $\X(\osp_{2|2n})$ generated by the coefficients
of the series $t_{11}(u)$, $t_{12}(u)$, $t_{21}(u)$ and $t_{22}(u)$.
This subalgebra is isomorphic to the Yangian $\Y(\gl_{1|1})$, and the cyclic span
$\Y_0\tss \xi$ of the highest vector $\xi$ is a finite-dimensional
module with the highest weight $(\la_1(u),\la_2(u))$.
This implies that condition \eqref{gloo} must hold by \cite[Thm.~4]{zh:rs};
see also \cite[Prop.~5.1]{m:ry}.

Furthermore, Proposition~\ref{prop:vplus} implies that the subspace
$L(\la(u))^+$ is a module over the extended Yangian
$\X(\osp_{0|2n})\cong \X(\spa_{2n})$. The vector $\xi$ generates a highest weight
$\X(\spa_{2n})$-module
with the highest weight $(\la_{2}(-u),\dots,\la_{n+2}(-u))$.
Since this module is finite-dimensional, conditions
\eqref{spao} and \eqref{spala} now follow by
\cite[Thm.~5.16]{amr:rp}.

\subsection{Fundamental representations}
\label{sec:fm}

Our next step in the proof of the Main Theorem is to show that the conditions
\eqref{gloo}, \eqref{spao} and \eqref{spala} are sufficient for the representation
$L(\la(u))$ of $\X(\osp_{2|2n})$ to be finite-dimensional.
The tuple
\beql{tdp}
\big(\ol Q(u),Q(u),P_2(u),\dots,P_{n+1}(u)\big)
\eeq
of Drinfeld
polynomials determines the highest weight
$\la(u)$ of the representation up to a simultaneous multiplication of all components
$\la_i(u)$ by a formal series $f(u)\in 1+u^{-1}\CC[[u^{-1}]]$.
This operation corresponds to twisting the action of
the algebra $\X(\osp_{2|2n})$ on $L(\la(u))$ by the automorphism \eqref{muf}.
Hence, it suffices to prove that
a particular module $L(\la(u))$ corresponding to a given tuple \eqref{tdp}
is finite-dimensional.

Suppose that $L(\nu(u))$ and $L(\mu(u))$ are the irreducible
highest weight modules with the highest weights
\beql{munu}
\nu(u)=\big(\nu_{1}(u),\dots,\nu_{n+2}(u)\big)\Fand
\mu(u)=\big(\mu_{1}(u),\dots,\mu_{n+2}(u)\big).
\eeq
By the coproduct rule
\eqref{Delta},
the cyclic span $\X(\osp_{2|2n})(\xi\ot\xi')$ of the tensor product
of the respective highest vectors $\xi\in L(\nu(u))$ and $\xi'\in L(\mu(u))$
is a highest weight
module
with the highest weight
\beql{munup}
\big(\nu_{1}(u)\tss\mu_{1}(u),\dots,\nu_{n+2}(u)\tss\mu_{n+2}(u)\big).
\eeq
This observation implies the corresponding transition rule for the associated tuples
of Drinfeld polynomials. Namely, if
the highest weights in \eqref{munu}
are associated with
the respective tuples
\ben
\big(\ol Q(u),Q(u),P_2(u),\dots,P_{n+1}(u)\big)\Fand
\big(\ol Q^{\circ}(u),Q^{\circ}(u),P^{\circ}_2(u),\dots,P^{\circ}_{n+1}(u)\big),
\een
then
the highest weight \eqref{munup} is associated with the tuple
\beql{traru}
\Big(\frac{\ol Q(u)\tss\ol Q^{\circ}(u)}{d(u)},\frac{Q(u)\tss Q^{\circ}(u)}{d(u)},
P^{}_2(u)P^{\circ}_2(u),\dots,P^{}_{n+1}(u)P^{\circ}_{n+1}(u)\Big),
\eeq
where $d(u)$ is the monic polynomial in $u$ defined as the greatest common divisor of
the polynomials $\ol Q(u)\tss\ol Q^{\circ}(u)$ and $Q(u)\tss Q^{\circ}(u)$.
Therefore, it will be enough
to show that the {\em fundamental representations} of $\X(\osp_{2|2n})$
are finite-dimensional. They correspond to the tuples of Drinfeld polynomials
of the form
\beql{fundam}
\big(u+\al,u+\be,1,\dots,1\big)\Fand
\big(1,\dots,1,u+\ga,1,\dots,1\big),
\eeq
for $\al\ne\be$, where $u+\ga$ represents the polynomial $P_d(u)$ and $d$ runs
over the values $2,\dots,n+1$.
Note that by applying the shift automorphism \eqref{shift},
we can assume that $\be=0$ in the first tuple and use a particular value of $\ga$
in the second tuple.

We will begin with the first tuple in \eqref{fundam}.
The required property is implied by the following.

\bpr\label{prop:laon}
The representation $L(\la(u))$ of the extended Yangian $\X(\osp_{2|2n})$
with the components of the
highest weight
given by $\la_1(u)=1+\al\tss u^{-1}$
and $\la_i(u)=1$ for $i=2,3,\dots,n+2$, is finite-dimensional.
\epr

\bpf
Let $\xi$ denote the highest vector of $L(\la(u))$.
We will show first that all vectors $t_{kl}^{(r)}\xi$ with $k,l\in\{2,3,\dots,2'\}$
and $r\geqslant 1$ are zero. This holds by definition for $k\leqslant l$, while
the defining relations \eqref{defrel} imply that for $k>l$
we have the properties $t_{1j}(u)\tss t_{kl}^{(r)}\xi=0$ for $j>1$ and
$t_{i1'}(u)\tss t_{kl}^{(r)}\xi=0$ for $i<1'$. This means that the vector
$t_{kl}^{(r)}\xi$ belongs to the subspace $L(\la(u))^+$ defined in \eqref{vplus}.
By Proposition~\ref{prop:vplus}, the subspace $L(\la(u))^+$
is a representation of the algebra $\X(\osp_{0|2n})$, and $\xi$ generates
a highest weight module over this algebra with the highest weight $(1,\dots,1)$.
The irreducible quotient of this module is one-dimensional, which implies that
the vector $t_{kl}^{(r)}\xi\in L(\la(u))$ is annihilated
by all coefficients of the series $t_{ij}(u)$ with $1\leqslant i<j\leqslant 1'$
and therefore must be zero.

We now prove two lemmas providing
the formulas for the action of some other elements
of the extended Yangian on the highest vector $\xi$.

\ble\label{lem:vani}
We have $t_{k1}^{(r)}\xi=0$
for all $k=2,3,\dots,2'$ and $r\geqslant 2$.
\ele

\bpf
As in the above argument, we will check
that the vectors $t_{k1}^{(r)}\xi$ are annihilated by all
coefficients of the series $t_{ij}(u)$ with $1\leqslant i<j\leqslant 1'$.
The Poincar\'e--Birkhoff--Witt theorem and the defining relations
imply that it is sufficient to check this property
for the coefficients of the series $t_{i,i+1}(u)$ with $i=1,\dots,n+1$.
Suppose first that $2\leqslant k\leqslant n+1$. Then \eqref{defrel} gives
\ben
\big[\tss t_{k1}(v),t_{i,i+1}(u)\big]=\frac{1}{v-u}
\big(t_{i1}(v)\ts t_{k,i+1}(u)-t_{i1}(u)\ts t_{k,i+1}(v)\big)(-1)^{\bi}.
\een
Since $t_{k,i+1}(u)\tss\xi=\de_{k,i+1}\tss\xi$, we derive
\beql{tiipo}
t_{i,i+1}(u)\tss t_{k1}(v)\tss\xi=\frac{1}{v-u}\ts \de_{k,i+1}\tss
\big(t_{i1}(v)-t_{i1}(u)\big)\tss\xi.
\eeq
Now proceed by induction on $k$.
For $k=2$, by \eqref{tiipo} we have
\ben
t_{12}(u)\tss t_{21}(v)\tss\xi=\frac{1}{v-u}\ts \big(t_{11}(v)-t_{11}(u)\big)\tss\xi
=-\al\tss u^{-1}\tss v^{-1}\tss\xi
\een
which implies
that $t_{21}(v)\tss\xi=v^{-1}\tss t_{21}^{(1)}\tss\xi$. Using \eqref{tiipo}
for the induction step, we can conclude that $t_{k1}(v)\tss\xi=v^{-1}\tss t_{k1}^{(1)}\tss\xi$
for all $2\leqslant k\leqslant n+1$.

Now let $(n+1)'\leqslant k\leqslant 2'$ and use induction on $k$ starting from $k=(n+1)'$.
Note that \eqref{tiipo} holds for these values of $k$ unless $i=k'$. By applying \eqref{defrel}
to the super-commutator $\big[t_{(n+1)',1}(v),t_{n+1,(n+1)'}(u)\big]$ and using the assumptions
$\la_i(u)=1$ for $i=2,3,\dots,n+2$,
we get
\ben
\bal
t_{n+1,(n+1)'}(u)\ts t_{(n+1)',1}(v)\ts \xi
&=\frac{1}{v-u}\ts \big(t_{n+1,1}(v)-t_{n+1,1}(u)\big)\tss\xi\\[0.4em]
{}&+\frac{1}{v-u+n}\ts t_{11}(v)\ts t_{1',(n+1)'}(u)\tss\xi
+\frac{1}{v-u+n}\ts t_{n+1,1}(v)\tss\xi.
\eal
\een
By another application of \eqref{defrel} we derive
\ben
\frac{v-u+n+1}{v-u+n}\ts t_{11}(v)\ts t_{1',(n+1)'}(u)\tss\xi=
\frac{v+\al}{v}\ts t_{1',(n+1)'}(u)\tss\xi-\frac{1}{v-u+n}\ts t_{n+1,1}(v)\tss\xi.
\een
Therefore, the previous expression can be written in the form
\begin{align}
t_{n+1,(n+1)'}(u)\ts t_{(n+1)',1}(v)\ts \xi
&=\frac{1}{v-u}\ts \big(t_{n+1,1}(v)-t_{n+1,1}(u)\big)\tss\xi
\label{tnpn}\\[0.4em]
{}&+\frac{v+\al}{v(v-u+n+1)}\ts t_{1',(n+1)'}(u)\tss\xi
+\frac{1}{v-u+n+1}\ts t_{n+1,1}(v)\tss\xi.
\non
\end{align}
On the other hand, by calculating the super-commutator $[t_{n+1,1}(v),t_{1'1'}(u)]$
we obtain
\ben
t_{1'1'}(u)\ts t_{n+1,1}(v)\ts\xi=\frac{v-u+n}{v-u+n+1}\ts t_{n+1,1}(v)\ts t_{1'1'}(u)\ts\xi
-\frac{1}{v-u+n+1}\ts t_{1',(n+1)'}(u)\tss\xi.
\een
By the consistency conditions of Proposition~\ref{prop:nontrvm}, we find that
\ben
t_{1'1'}(u)\ts\xi=\frac{u-n-1}{u+\al-n-1}\ts\xi,
\een
and so the residue at $v=u-n-1$ gives
\ben
t_{1',(n+1)'}(u)\tss\xi=-\frac{u-n-1}{u+\al-n-1}\ts t_{n+1,1}(u-n-1)\ts\xi.
\een
As we showed above, $t_{n+1,1}(v)\tss\xi=v^{-1}\tss t_{n+1,1}^{(1)}\tss\xi$,
so that \eqref{tnpn} simplifies to
\ben
t_{n+1,(n+1)'}(u)\ts t_{(n+1)',1}(v)\ts \xi=-\Big(\frac{1}{u\tss v}+\frac{1}{(u+\al-n-1)\tss v}\Big)
\ts t_{n+1,1}^{(1)}\tss\xi.
\een
Hence we may conclude that the vectors
$t_{(n+1)',1}^{(r)}\xi$ with $r\geqslant 2$ are annihilated by all
coefficients of the series $t_{i,i+1}(u)$ with $i=1,\dots,n+1$
and so $t_{(n+1)',1}^{(r)}\xi=0$.

Now suppose that $k=l'$ with $2\leqslant l\leqslant n$. Calculating as above, we find that
\beql{tlpo}
t_{l,l+1}(u)\tss t_{l',1}(v)\ts \xi=\frac{1}{v-u+n}\ts\big(t_{11}(v)\ts t_{1',l+1}(u)\ts\xi
-t_{(l+1)',1}(v)\ts\xi\big).
\eeq
Furthermore, using the defining relations again, we derive the relation
\ben
t_{11}(v)\ts t_{1',l+1}(u)\ts\xi=\frac{1}{v-u+n+1}\ts t_{(l+1)',1}(v)\ts\xi
+\frac{(v+\al)(v-u+n)}{v\tss(v-u+n+1)}\ts t_{1',l+1}(u)\ts\xi,
\een
which by taking the residue at $v=u-n-1$ also implies
\ben
t_{(l+1)',1}(u-n-1)\ts\xi=\frac{u+\al-n-1}{u-n-1}\ts t_{1',l+1}(u)\ts\xi.
\een
Since
$
t_{(l+1)',1}(v)\ts\xi=v^{-1}\ts t_{(l+1)',1}^{(1)}\ts\xi
$
by the induction hypothesis,
relation \eqref{tlpo} simplifies to
\ben
t_{l,l+1}(u)\tss t_{l',1}(v)\ts \xi=
\frac{1}{(u+\al-n-1)\tss v}
\ts t_{(l+1)',1}^{(1)}\ts\xi,
\een
thus proving that the vectors
$t_{l',1}^{(r)}\xi$ with $r\geqslant 2$ are annihilated by the
coefficients of the series $t_{l,l+1}(u)$.
The proof is competed by taking into account \eqref{tiipo}
for the remaining values $i\ne l$.
\epf

\ble\label{lem:for}
We have
\ben
t_{1'1}(v)\ts\xi=\frac{1}{v\ts(v+\al-n-1)}\ts \sum_{k=2}^{n+1}\tss t_{k'1}^{(1)}\tss t_{k\tss 1}^{(1)}\ts\xi.
\een
\ele

\bpf
Calculate the commutator $[t_{11}(v),t_{1'1}(u)]$ by \eqref{defrel}
and rearrange the terms to get
\ben
\bal
\frac{v-u+n+1}{v-u+n}\ts t_{11}(v)\ts t_{1'1}(u)&=
\frac{n}{(v-u)(v-u+n)}\ts t_{1'1}(v)\ts t_{11}(u)\\
{}&+\frac{v-u-1}{v-u}\ts t_{1'1}(u)\ts t_{11}(v)
-\frac{1}{v-u+n}\ts\sum_{k=2}^{2'}t_{k1}(v)\ts t_{k'1}(u)\ts\ta_k.
\eal
\een
By setting $u=v+n+1$ we come to the relation
\ben
\frac{n}{n+1}\ts t_{1'1}(v)\ts t_{11}(v+n+1)
+\frac{n+2}{n+1}\ts t_{1'1}(v+n+1)\ts t_{11}(v)
=-\sum_{k=2}^{2'}t_{k1}(v)\ts t_{k'1}(v+n+1)\ts\ta_k.
\een
Apply both sides of the relation to the highest vector $\xi$.
By Lemma~\ref{prop:laon} we have
\ben
t_{k'1}(v+n+1)\ts\xi=(v+n+1)^{-1}\ts t_{k'1}^{(1)}\ts\xi.
\een
Hence, using the super-commutator
$
[t_{k1}(v),t_{k'1}^{(1)}]=-t_{1'1}(v)\ts\ta_k
$
we obtain
\begin{multline}
\frac{n(v+\al-n-1)}{(n+1)(v+n+1)}\ts t_{1'1}(v)\ts \xi
+\frac{(n+2)(v+\al)}{(n+1)v}\ts t_{1'1}(v+n+1)\ts \xi\\
{}=\frac{1}{v\ts(v+n+1)}\ts \sum_{k=2}^{2'}\tss \ta_k\ts t_{k'1}^{(1)}
\tss t_{k\tss 1}^{(1)}\ts\xi.
\non
\end{multline}
This relation uniquely determines the series $t_{1'1}(v)\ts \xi$. It is easily seen to be
satisfied by
\ben
t_{1'1}(v)\ts \xi=\frac{1}{2\tss v\ts(v+\al-n-1)}\ts
\sum_{k=2}^{2'}\tss \ta_k\ts t_{k'1}^{(1)}\tss t_{k\tss 1}^{(1)}\ts\xi.
\een
In particular, $t_{1'1}^{(1)}\ts \xi=0$ and so $[t_{k'1}^{(1)}, t_{k\tss 1}^{(1)}]\ts\xi=0$
yielding the required relation.
\epf

Returning to the proof of the proposition, we can now derive that
the representation
$L(\la(u))$ is spanned by the vectors of the form
\beql{basve}
t_{k_1 1}^{(1)}\dots t_{k_s 1}^{(1)}\ts\xi,\qquad 2\leqslant k_1<\dots<k_s\leqslant 2',
\eeq
with $s\geqslant 0$.
Indeed, by the first part of the proof,
and the Poincar\'e--Birkhoff--Witt theorem applied to a suitable ordering of the generators,
it is enough to verify that
the span of these vectors is stable under the action of all generators of the form
$t_{k\tss 1}^{(r)}$ with $k=2,\dots,2'$. Note that by taking the $(1',1)$ entry
in the matrix relation \eqref{ttra},
we find that $t_{1'1}^{(1)}=0$ in the algebra $\X(\osp_{2|2n})$. This implies that
the elements $t_{k\tss 1}^{(1)}$ with $k=2,\dots,2'$ pairwise anticommute.
They also have the property $[t^{}_{1'1}(v),t_{k\tss 1}^{(1)}]=0$.
Hence, applying the series $t_{k\tss 1}(v)$ to a basis vector \eqref{basve}, we get
\ben
t^{}_{k\tss 1}(v)\ts t_{k_1 1}^{(1)}\dots t_{k_s 1}^{(1)}\ts\xi
=\sum_{i=1}^s(-1)^{i-1}\ts t_{k_1 1}^{(1)}\dots [t^{}_{k 1}(v),t_{k_i\tss 1}^{(1)}]
\dots t_{k_s 1}^{(1)}\ts\xi+(-1)^s\ts t_{k_1 1}^{(1)}\dots t_{k_s 1}^{(1)}\ts
t^{}_{k\tss 1}(v)\ts\xi.
\een
Since $[t^{}_{k\tss 1}(v),t_{k_i 1}^{(1)}]=-\de_{k'k_i}t_{1'1}(v)\tss\ta_k$,
the sum on the right hand side equals
\ben
\ta_k\ts\sum_{i=1}^s(-1)^{i}\ts\de_{k'k_i}\ts t_{k_1 1}^{(1)}\dots \wh t_{k_i 1}^{\ts(1)}
\dots t_{k_s 1}^{(1)}\ts t_{1'1}(v)\ts\xi.
\een
By Lemma~\ref{lem:vani} we have $t^{}_{k\tss 1}(v)\ts\xi=v^{-1}\ts t_{k\tss 1}^{(1)}\ts\xi$,
and together with the relation of Lemma~\ref{lem:for} this proves that
the span of the vectors \eqref{basve} is stable under the action of all
operators $t_{k1}^{(r)}$ with $k=2,\dots,2'$. Since the representation
$L(\la(u))$ is spanned by these vectors, its dimension does not exceed $4^n$.
\epf

Let $V(\mu)$ denote the irreducible highest weight representation of the Lie superalgebra
$\osp_{2|2n}$ with the highest weight $\mu=(\mu_1,\dots,\mu_{n+1})$ with respect to the
upper-triangular Borel subalgebra. This means that $F_{ii}\tss\ze=\mu_i\ts\ze$ for $i=1,\dots,n+1$
and $F_{ij}\ts\ze =0$ for $i<j$, where $\ze$ is the highest vector of $V(\mu)$.
The modules of the form
$V(\al,0,\dots,0)$ are known to be {\em typical}, if and only if $\al$ does not
belong to the set $\{0,1,\dots,n-1\}\cup\{n+1,\dots,2n\}$; see \cite{k:rc}.

\bco\label{cor:exls}
The typical representations $V(\al,0,\dots,0)$ of the Lie superalgebra $\osp_{2|2n}$ extend
to modules over the Yangian $\X(\osp_{2|2n})$.
\eco

\bpf
Equip the representation $L(1+\al\ts u^{-1},1,\dots,1)$ of $\X(\osp_{2|2n})$ with the action
of $\osp_{2|2n}$ via the embedding \eqref{emb}. By taking the $(i,j)$
entry in the matrix relation \eqref{ttra} for $i\ne j$ we find
\ben
t_{ij}^{(1)}+t_{j'i'}^{(1)}(-1)^{\bj+\bi\bj}\ts\ta_i\ta_j=0.
\een
Therefore, under the embedding we have $F_{ij}\mapsto t_{ij}^{(1)}(-1)^{\bi}$.

On the other hand, any typical representation $V(\al,0,\dots,0)$
is isomorphic to the corresponding Kac module
and so is equipped with a basis of the form
\ben
F_{k_1 1}\dots F_{k_s 1}\ts\ze,\qquad 2\leqslant k_1<\dots<k_s\leqslant 2',
\een
with $s\geqslant 0$. The proof of Proposition~\ref{prop:laon} shows that the mapping
$\ze\to\xi$ extends to an $\osp_{2|2n}$-module isomorphism
$V(\al,0,\dots,0)\to L(1+\al\ts u^{-1},1,\dots,1)$.
\epf

We will now turn to the second family of
fundamental representations in \eqref{fundam}.
We will show that they can be constructed as subquotients of tensor products
of the vector representations of $\X(\osp_{2|2n})$ and representations
of the first family in \eqref{fundam}.

The {\em vector representation} of $\X(\osp_{2|2n})$ on $\CC^{2|2n}$ is defined by
\beql{vectre}
t_{ij}(u)\mapsto \de_{ij}+u^{-1}\tss
e_{ij}(-1)^{\bi}-(u-n)^{-1}\tss e_{j'i'}(-1)^{\bi\bj}\ts\ta_i\ta_j.
\eeq
The homomorphism property
follows from the $RTT$-relation \eqref{RTT} and
the Yang--Baxter equation satisfied by $R(u)$ as in \eqref{ybe}; cf. \cite{aacfr:rp}.
The mapping $T(u)\mapsto R(u)$ defines an algebra homomorphism
$\X(\osp_{2|2n})\to \End\CC^{2|2n}$, and to get \eqref{vectre} we take
its composition with the
automorphism $t_{ij}(u)\mapsto t_{i'j'}(u+n)\ts\ta_i\ta_j(-1)^{\bi+\bj}$.

Now use the coproduct \eqref{Delta} and the shift automorphism \eqref{shift}
to equip
the tensor square $V=(\CC^{2|2n})^{\ot 2}$
with the action of $\X(\osp_{2|2n})$ by setting
\beql{tijtprei}
t_{ij}(u)\mapsto
\sum_{l=1}^{2n+2} t_{i\tss l}(u-1)\ot t_{lj}(u),
\eeq
where the generators act in the respective copies of the vector space
$\CC^{2|2n}$ via the rule \eqref{vectre}.
Introduce the vectors $w_k\in V$ by $w_k=e_1\ot e_k-e_k\ot e_1$
for $k=2,3,\dots,2'$ and denote by $W$ the subspace of $V$
spanned by these vectors.

The vector representation $\CC^{0|2n}$ of the extended Yangian
$\X(\osp_{0|2n})$ is defined by
the rule similar to \eqref{vectre}; see \cite{amr:rp}:
\beql{vectreo}
\bar t_{ij}(u)\mapsto \de_{ij}-u^{-1}\tss
e_{ij}+(u-n-1)^{-1}\tss e_{j'i'}\ts\ta_i\ta_j,\qquad 2\leqslant i,j\leqslant 2'.
\eeq

\bpr\label{prop:vectv}\quad {\rm (i)}\ \
The subspace $W$ is invariant with respect to
the action of the operators $t_{ij}(u)$
with $2\leqslant i,j\leqslant 2'$.
Moreover, the assignment $\bar t_{ij}(u)\mapsto t_{ij}(u)$
defines
a representation of the algebra $\X(\osp_{0|2n})$ on $W$
isomorphic to the vector representation.
\par
{\rm (ii)}\ \  The subspace $W$ is annihilated by the operators $t_{1j}(u)$ with $j>1$ and
$t_{i\tss 1'}(u)$ with $i<1'$. Moreover, the coefficients of the series $t_{11}(u)$
act on $W$ as multiplications by scalars determined by $t_{11}(u)\mapsto 1+u^{-1}$.
\epr

\bpf
Assuming that $2\leqslant i,j\leqslant 2'$, for
the action of the operator $t_{ij}(u)$
we have
\ben
\bal
t_{ij}(u)\ts w_k&=t_{ij}(u)\ts(e_1\ot e_k-e_k\ot e_1)\\
&{}=\sum_{l=1}^{2n+2} t_{i\tss l}(u-1)\ts e_1\ot t_{lj}(u)\ts e_k
-\sum_{l=1}^{2n+2} (-1)^{l+j}\ts t_{i\tss l}(u-1)\ts e_k\ot t_{lj}(u)\ts e_1.
\eal
\een
Due to \eqref{vectre},
a nonzero contribution can only come from the terms
\begin{multline}
t_{i1}(u-1)\ts e_1\ot t_{1j}(u)\ts e_k+t_{ii}(u-1)\ts e_1\ot t_{ij}(u)\ts e_k\\[0.3em]
-t_{ij}(u-1)\ts e_k\ot t_{jj}(u)\ts e_1+t_{i1'}(u-1)\ts e_k\ot t_{1'j}(u)\ts e_1
\non
\end{multline}
and the calculation yields the formula
\ben
t_{ij}(u)\ts w_k=\de_{ij}\ts w_k-\de_{kj}\ts u^{-1}\ts w_i+\de_{ki'}\ts\ta_i\tss\ta_j\ts
 (u-n-1)^{-1}\ts w_{j'}.
\een
Taking onto account \eqref{vectreo}, we thus prove
the first part of the proposition. The second part is verified by
a similar calculation.
\epf

For any $a\in\CC$ we will denote by $V_a$ the representation
of the algebra $\X(\osp_{2|2n})$ on the space $V=(\CC^{2|2n})^{\ot 2}$
obtained by twisting the action \eqref{tijtprei}
with the automorphism \eqref{shift}. Similarly, we let $W_a$ denote the
representation of $\X(\osp_{0|2n})$ on $W$ twisted by the same shift automorphism
on $\X(\osp_{0|2n})$.
For $d=1,\dots,n$ introduce the vectors
\ben
\xi_d=\sum_{\si\in\Sym'_d} \sgn\si\cdot
w_{\si(2)}\ot w_{\si(3)}\ot\dots\ot w_{\si(d+1)} \in V_0\ot V_{-1}\ot\dots\ot V_{-d+1},
\een
where $\Sym'_d$ denotes the group of permutations of the set $\{2,\dots,d+1\}$.

\bpr\label{prop:xikdp}
The cyclic span $\X(\osp_{2|2n})\ts\xi_d$ is a highest weight module
whose tuple of Drinfeld polynomials \eqref{tdp} has the form
\beql{tudp}
(u+1,u-d,1,\dots,1,u-d,1,\dots,1)\quad \text{with}\quad P_{d+1}(u)=u-d,
\eeq
for $d=1,\dots,n-1$, while for $d=n$
it has the form
\beql{tudpn}
(u+1,u-n,1,\dots,1,u-n-1).
\eeq
\epr

\bpf
It follows from Proposition~\ref{prop:vectv} and the coproduct rule \eqref{Delta}
that for all $2\leqslant i,j\leqslant 2'$ the image $t_{ij}(u)\ts\xi_d$
coincides with $\bar t_{ij}(u)\ts\xi_d$ in the representation
$W_0\ot W_{-1}\ot\dots\ot W_{-d+1}$
of the algebra $\X(\osp_{0|2n})$. Since the series $\bar t_{ij}(-u)$ satisfy
the defining relations of the extended Yangian $\X(\spa_{2n})$, we derive from
the proof of \cite[Thm.~5.16]{amr:rp} (see also Corollary~\ref{cor:fungl} below), that
the vector $\xi_d$ has the properties
\ben
t_{ij}(u)\ts \xi_d=0\qquad\text{for}\quad 2\leqslant i<j\leqslant 2'
\een
and
\beql{tiixi}
t_{ii}(u)\ts \xi_d=\begin{cases}
\dfrac{u-d}{u-d+1}\ts\xi_d \qquad&\text{for}
\quad i=2,\dots, d+1,\\[0.6em]
\ \xi_d\qquad&\text{for}\quad i=d+2,\dots,n+2,
\end{cases}
\eeq
where $d=1,\dots,n-1$.
Moreover, the same relations hold for $d=n$, except for \eqref{tiixi} with $i=n+2$,
which is replaced by
\ben
t_{n+2,n+2}(u)\ts \xi_n=\frac{u-n}{u-n-1}\ts\xi_n.
\een
Finally, Proposition~\ref{prop:vectv}\tss(ii) and \eqref{Delta} imply that $\xi_d$
is annihilated by the action of $t_{1j}(u)$ with $j>1$, while
\ben
t_{11}(u)\ts\xi_d=\frac{u+1}{u-d+1}\ts\xi_d.
\een
Thus, the vector $\xi_d$ generates a highest weight module over the algebra
$\X(\osp_{2|2n})$, whose highest weight is found from the action of the series $t_{ii}(u)$.
The formulas for the Drinfeld polynomials easily follow.
\epf

We are now in a position to complete the proof of the Main Theorem.
Proposition~\ref{prop:laon} implies that the
irreducible highest weight
representations of $\X(\osp_{2|2n})$, associated with any tuple
of Drinfeld polynomials of
the first type in \eqref{fundam} is finite-dimensional.
Furthermore, Proposition~\ref{prop:xikdp} shows that the irreducible highest weight
representations of $\X(\osp_{2|2n})$ associated
with the tuples of the form
\eqref{tudp}
and \eqref{tudpn} are also finite-dimensional. On the other hand, by applying
the shift automorphism \eqref{shift} and using the transition rule
\eqref{traru}, we can get all tuples of Drinfeld polynomials
of the second type in \eqref{fundam}
from certain tuples of the first type and those appearing in \eqref{tudp}
and \eqref{tudpn}. This proves the sufficiency of the conditions of the Main Theorem
for the representation $L(\la(u))$ to be finite-dimensional.
The last part of the theorem is an easy consequence of the decomposition
\eqref{tensordecom}; cf. \cite[Cor.~5.19]{amr:rp}.


\section{Introduction}
\label{section:sec1}
Deep learning based computer vision algorithms have achieved impressive performance in various fields. In most of the applications, image sensors provide almost all the information consumed by vision algorithms. Modern consumer cameras usually employ mono-sensor technique to generate three-channel color images, where a color filter array (CFA) such as Bayer pattern array\cite{RN4} is covered on the image sensor to restrict each pixel captures only one color component among the red, green and blue. A step-by-step chain of image processing operations, usually referred to as image signal processing (ISP) pipeline, is then applied to generate a finely-rendered color image.

As a matter of course, most of the modern computer vision algorithms, take color images generated by ISP pipelines as input. The suitability of processed color images (instead of RAW images from image sensors) for computer vision applications is seldom questioned, though it is well-known that ISP pipelines may undermine the original information captured by the image sensors \cite{RN14}. The most critical reason is that all the well labeled large-scale datasets are in the format of processed, three-channel color images. In many applications, it is  assumed that images are directly related to the actual scene radiance, with cameras being accurate light measuring devices. However, it is well-known that ISP pipelines used in digital cameras are usually non-linear. They are designed for photography with a goal of generating high-quality images for human consumption, rather than capturing accurate physical descriptions of the scene. Therefore, the output pixel intensity values are non-linearly related to spectral scene radiance for most of the consumer cameras\cite{RN61,RN62,RN63}. On the other hand, the experimental results in \cite{RN61} show that the output RAW values of image sensors are linearly related to image irradiance and many visual tasks can benefit from these linearity properties. However, compensating the nonlinearities of processed color images from consumer cameras is difficult, since ISP pipelines of different camera manufactures are confidential and will not release to users. Moreover, it is intuitive that the optimal ISP pipeline configuration, i.e., necessary stages and their orders, for a specific vision task, may be different. Therefore, instead of using a standard ISP pipeline, it is necessary to determine the configuration of the ISP pipeline according to the vision tasks. In other words, co-design of ISP pipeline and vision algorithm is the key for further improvement of computer vision performance.

To take advantage of the linearity of RAW images and explore ISP configuration for specific tasks, the commonly used stages in ISP pipelines have been studied for both traditional and deep learning based computer vision algorithms \cite{RN59,RN23,RN56,RN57,RN55,RN64}. Due to the high complexity of modern ISP pipelines, sufficiently large number of RAW Bayer image data is needed for the study of configuration of ISP pipelines for different computer vision tasks. However, almost all the large-scale datasets are provided in the format of processed color image and there is few RAW Bayer image datasets. Although there are options to save RAW Bayer images for both digital single lens reflex (DSLR) cameras and mobile phones\cite{RN65}, it takes a huge amount of efforts to produce a large-scale RAW Bayer image dataset like ImageNet \cite{RN75}. To the best knowledge of the authors, there are only some small-scale RAW Bayer image datasets available\cite{RN55,RN64}. But they are produced for traditional computer vision algorithms like support vector machine (SVM), such that the volume of these datasets are too small for deep learning-based algorithms. Therefore, it is crucial to generate a RAW Bayer image dataset to explore the co-design of ISP pipelines and vision algorithms.

Another interesting application of RAW Bayer image dataset is in smart image sensor, where the image sensor and back-end computing engine are integrated together in one chip. This paradigm is always referred to as in-sensor computing\cite{RN78}. For in-sensor machine vision applications, if the ISP pipeline is completely bypassed, the performance of the algorithms may degrade by as much as 60\% according to \cite{RN70}. This is because the models are trained on finely rendered color image datasets, while the input images for inference are RAW Bayer pattern images. However, if the models can be trained or fine-tuned on RAW Bayer image datasets, this kind of performance degradation may be avoided, or at least alleviated.

For RAW image dataset generation, apart from capturing and labeling directly, reversible pipeline is another solution to generate RAW Bayer images from their color versions\cite{RN61,RN62,RN63}. One example is the configurable \& reversible imaging pipeline (CRIP) tool proposed in \cite{RN23}, where an ISP pipeline is modeled as serious of reversible transformations, such that finely rendered color images can be converted back to RAW Bayer images by reverse transformations. Motivated by the reversible ISP pipelines as well as the success of generative adversarial networks (GAN) in image-to-image translation \cite{RN66,RN67,RN68,RN69}, we propose in this work to transform a finely rendered color image to its corresponding RAW Bayer image pixel by pixel directly, where the color-to-RAW transformation is modeled as an image-to-image translation problem. Since the RAW Bayer images have their specific CFA patterns, e.g., the Bayer pattern, GAN cannot be directly applied to color-to-RAW image transformation, otherwise the color information will be lost and undesirable effects will be produced. Therefore, we need to maintain the CFA patterns during the transformation to avoid loose of color information.

In this paper, we propose a color-to-RAW conditional-GAN (cGAN)  to convert finely rendered color images to their corresponding RAW Bayer images, where an improved loss function is used to maintain the CFA structure. The contributions of this work are as follows:
\begin{itemize}
    \item Propose a method to add un-formulatable constraints in image generation (e.g., to generate specific patterns), and implement the proposed method on an end-to-end GAN model to convert finely rendered color images to RAW Bayer images in a pixel to pixel manner.

    \item Introduce large-scale RAW Bayer datasets for works which aim to find the optimal ISP configurations for specific computer vision tasks. These datasets are reverted version of standard datasets that are commonly used in the computer vision field.
    \item Provide a new solution to the mismatch between testing set and training set for the in-sensor computing paradigm \cite{RN70}.
\end{itemize}

\section{Related Work and Applications}
\subsection{Radiometric Calibration \& Reversible Camera Model}
In the field of computer vision, radiometric calibration refers to the process of converting the irradiance received by an image sensor to pixel intensity. Conventional ``grayscale'' radiometric calibration\cite{RN71,RN72,RN73} can be expressed as
\begin{equation}
    \label{gray_scale_calibration}
    I\left ( x,y \right )=f\left ( \kappa \left ( x,y \right ) \right ).
\end{equation}
Here, $\kappa (x,y)$ is the camera's RAW output at pixel location $(x,y)$, which is linearly related to the image irradiance, and $I(x,y)$ is the corresponding finely rendered pixel intensity. The function $f(\cdot)$ is the radiometric response function, which is assumed to be a fixed property of a camera. To deal with color images, we can apply the model in \eqref{gray_scale_calibration} to each channel independently, such that $I$ can be extended as
\begin{equation}
    \begin{split}
    \label{color_calibration1}
    I(x,y)&=\begin{bmatrix}
        I_r(x,y)\\
        I_g(x,y)\\
        I_b(x,y)
        \end{bmatrix}=\begin{bmatrix}
        f_r(\kappa _r(x,y))\\
        f_g(\kappa _g(x,y))\\
        f_b(\kappa _b(x,y))
        \end{bmatrix}.\\
    \end{split}
\end{equation}

Note that rendering linear data $\kappa$ into $I$ at an output color space (e.g., sRGB) takes a series of processing steps, which is always referred to as the ISP pipeline. It is therefore deficient to model such a process with the change of pixel intensity. To explain color preferably, the authors of \cite{RN61} propose a 24-parameter model with color transformation, which further extends the model in \eqref{gray_scale_calibration} to

\begin{equation}
    \label{color_calibration2}
    I\left ( x,y \right )=f\left (T \cdot \kappa \left ( x,y \right ) \right )=f\left (T \cdot\begin{bmatrix}
                 \kappa _{r}(x,y)\\
                 \kappa _{g}(x,y)\\
                 \kappa _{b}(x,y)
                    \end{bmatrix}
                    \right ).
\end{equation}
Here $T$ is a $3\times3$ matrix corresponding to the white balance step in ISP pipelines, $f(\cdot)$ is modeled with a $5$-th order polynomial for each channel and explained as color rendering function.  Inspired by \cite{RN61}, Kim et al. further found that the transformation in \eqref{color_calibration2} will fail for highly saturated colors. This is because the color mapping component including gamut mapping is missing in \eqref{color_calibration2}, and some supersaturated values outside the sRGB gamut cannot be converted to RAW accurately by this model\cite{RN62,RN63}. Based on these observations, the authors in \cite{RN62} and \cite{RN63} propose a model as
\begin{equation}
    \label{color_calibration3}
    \begin{split}
        I\left ( x,y \right )=f\left (T_s\cdot T_w \cdot\begin{bmatrix}
            \kappa _{r}(x,y)\\
            \kappa _{g}(x,y)\\
            \kappa _{b}(x,y)
               \end{bmatrix}
               \right ).
    \end{split}
\end{equation}
Here, $T_w$ is a $3\times3$ white balance diagonal matrix, and $T_s$ is another $3\times3$ matrix corresponding to color space transformation.

With the aforementioned reversible models, the rendered color image pixel intensities can be inverted back to their corresponding linear RAW values directly. However, there are some noticeable limitations in the aforementioned works:
\begin{itemize}
    \item As the authors suggested in \cite{RN61}, the color rendering function $f(\cdot)$ is mostly a scene-dependent process instead of a fixed property of a camera. Moreover, a fixed nonlinearity per channel/camera model as used in traditional radiometric calibration are often inadequate to model $f(\cdot)$.
    \item Work \cite{RN63} is an improved version of \cite{RN61}, but their experiments are carried out under stringent conditions, e.g., cameras should work on photographic reproduction mode to disable all the auto optimizers and use a fixed ISP pipeline, i.e., images for inverse transformation should be shot and processed under similar conditions to ensure the accuracy of the reverse conversion. Unfortunately, not all cameras support this mode, and it is hard to ensure the conditions.
    \item Models in \cite{RN61,RN62,RN63}  are established under the premise of a lot of simplifications. For example, a commonly used commercial ISP pipeline usually consist of many steps, but the pipeline in\cite{RN61,RN62,RN63} includes only a small number of them (e.g., white balance, gamut mapping, etc.), and others are modeled as noise processes.
\end{itemize}

\subsection{ISP Configuration \& In-Sensor Computing}
It has been mentioned in the Section~\ref{section:sec1} that conventional ISP pipeline is design for photography. For computer vision tasks, excessive processing may introduce cumulative errors and undermine the original information captured by image sensors \cite{RN14}. For example, demosaicing usually smooths the images, gamma compression corrects the inconsistency between the ways cameras capture scenes, monitors displays images and human visual system (HSV) processes light \cite{RN74}.

The optimal configuration for computer vision oriented ISP pipeline remains an open problem, where RAW dataset is crucial to solve this problem, since modern computer vision algorithms are mostly data-driven. In \cite{RN64}, the authors show both theoretically and experimentally that gradient-based features can be directly extracted from RAW Bayer images, such that the entire ISP pipeline can be bypassed completely. For deep learning based computer vision tasks, eight existing vision algorithms are extensively experimented in \cite{RN23} to find the minimal but sufficient ISP pipeline. The authors use the CRIP tool\cite{RN23} to generate RAW Bayer datasets by converting commonly used standard datasets such as MNIST\cite{RN76}, ImageNet\cite{RN75} and CIFAR-10\cite{RN77} back into RAW. The CRIP tool is implemented based on the model proposed in \cite{RN63}, but images in these datasets are from the Internet and captured using different cameras. Therefore, there may be a mismatch in camera models. Moreover, in CRIP, the noise needs to be explicitly added to the image signal in the reverse ISP pipeline. All these limitations may result in inaccurate estimation of RAW Bayer data, which may further affect the evaluation of stages in the ISP pipeline.

As mentioned in introduction, the gap between training set and testing set will decline the performance of in-sensor machine vision applications. \cite{RN70} proposes to solve this problem by finding a minimalistic ISP pipeline, consisting of gamma compression and pixel binning, to approximate the data distribution of images processed by ISP pipelines. Although the model's performance can be improved after these operations, several modifications need to be done to image sensors.

\begin{figure*}[htbp]
    \centering
    \includegraphics[width=7.3in]{./figures/gan/raw_gan3.png}
    \caption{The training process of proposed method. The real and synthesized data are transformed by $T$ before fed into the discriminator.}
    \label{raw_gan_structure}
\end{figure*}

\subsection{GAN for Image-to-image Translation}
Image-to-image translation aims to convert an image from the source domain to a specified target domain, while preserving the main presentations of the input images. With the application of adversarial training, cGAN\cite{RN80} are increasingly used in image-to-image translation works with promising results \cite{RN66,RN67,RN68,RN69}.
Inspired by the success of GAN, we model the color-to-Bayer image conversion as sRGB domain to RAW domain translation, which can be effectively solved by GAN. The details of how to use GAN to generate high quality Bayer pattern images from their color versions will be discussed in the remaining of this paper.

\section{Proposed Method}
\subsection{Theoretical Analysis}

The basic idea is to model the color image to RAW Bayer pattern image transformation as a task of image-to-image translation, where the RAW Bayer pattern images and the color images are regarded as two different image styles.
The image-to-image transformation model used in this work is cGAN \cite{RN66}, whose objective can be expressed as
    \begin{equation}
        \label{valigan_loss_function}
        \begin{split}
            \underset{G}{min}\,\underset{D}max\,V(G,D)=&\mathbb{E}_{x\sim p_{data}}[\log(D(x|c))]\\+ &\mathbb{E}_{y\sim p_g}[\log(1-D(y))].
        \end{split}
    \end{equation}
Here, $p_{data}$ is the distribution of the real data, $p_g$ is the distribution of generated data which is learned by the generator $G$ through mapping prior noise $z \sim p_z$ to the data sample $G(z|c) \sim p_g$. Meanwhile, $x$ and $y=G(z|c)$ are the data sampled from $p_{data}$ and $p_g$, respectively, $c$ is the auxiliary information or the condition sampled from the target domain.

\begin{figure}
    \centering
    \includegraphics[width=3.5in]{./figures/demosaic/demosaic.png}
    \caption{Illustration of demosaicing process.}
    \label{demosaicing}
\end{figure}

Suppose $T = \{T_1, T_2,...,T_m\}$ is a set of image transformation techniques, the real and generated data are transformed by the transformation $T(\cdot)$ before feeding into the discriminator $D_T$, then the min-max objective of cGAN would be
\begin{equation}
    \label{mygan_loss_function}
    \begin{split}
        \underset{G}{min}\,\underset{D_T}max\,V(G,D_T)=&\mathbb{E}_{x\sim p^T_{data}}[\log(D_T(x|c))]\\+ &\mathbb{E}_{y\sim p^T_{g}}[\log(1-D_T(y)],
    \end{split}
\end{equation}
where $p^T(\cdot)$ is the distribution of the corresponding transformed data. We will deduce the feasibility of using the transformed generated data $p^T_{g}$ to improve the learning of original data $p_{g}$ theoretically in the next part.

\begin{figure*}
    \centering
    \includegraphics[width=7.3in]{./figures/architecture/pix2pix_interpolation.png}
    \caption{The overall architecture of the proposed method.}
    \label{architecture}
\end{figure*}

\subsubsection*{GAN Training with Transformed Data}
It has been shown in \cite{RN93} that suppose $T : \mathbb{X}  \rightarrow \mathbb{X}^T $ is a differentiable and invertible transformation which transforms $x$ to $x^T$, with fixed $G$, the optimal discriminator $D_{T}^{*}$ trained with transformed data $x^T$ can be expressed as
\begin{equation}
    \label{dt_and_d}
    \begin{split}

    D_{T}^{*}(x^T)&=\frac{p^T_{data}(x^T)}{p^T_g(x^T)+p^T_{data}(x^T)}\\ &=\frac{p_{data}(x)\left | \mathcal{J}^T(x) \right |^{-1}}{p_g(x) \left | \mathcal{J}^T(x) \right |^{-1}+p_{data}(x)\left | \mathcal{J}^T(x) \right |^{-1}}\\ &=\frac{p_{data}(x)}{p_g(x)+p_{data}(x)}\\&=D^{*}(x),
    \end{split}
\end{equation}
meaning that the optimal solution for $D^T$ and $D$ are exactly the same. Here, $\left | \mathcal{J}^T(x) \right|$ is the determinant of Jacobian matrix of transformation $T$. Therefore, the virtual training criterion\cite{RN80} of \eqref{mygan_loss_function} can be expressed as
\begin{equation}
    \label{gan_vitural}
    V(G,D_{T}^{*})=-\log(4)+2\cdot JS(p^T_{data}||p^{T}_{g}).
\end{equation}
Here, $JS(\cdot)$ is the Jensen–Shannon (JS) divergence between two distributions $p_{data}$ and $p_g$, which is non-negative. Therefore, the global minimal of \eqref{gan_vitural} is achieved when $JS(p^T_{data}||p^T_{g})=0$, and the only solution is $p^T_{data}=p^T_g$. According to $theorem 1$ in \cite{RN93}, under a differentiable and invertible transformation $T$, the JS divergence between
two distributions is invariant, which can be expressed as
\begin{equation}
    \label{JS_theorem}
    JS(p_{data}||p_{g})=JS(p^T_{data}||p^T_{g}).
\end{equation}
Combining \eqref{gan_vitural} and \eqref{JS_theorem}, we have
\begin{equation}
    \label{gan_vitural2}
    V(G,D^{*}_{T})=-log(4)+2\cdot JS(p_{data}||p_{g}).
\end{equation}
Equation \eqref{gan_vitural2} means that given an optimal discriminator $D^{*}_{T}$ trained with transformed samples, the objective of $V(G,D^{*}_{T})$ is to minimize the JS divergence between the distribution of generated data and original data, which guarantees $G$ to learn to generate original samples instead of transformed samples.

Now, suppose the set $T$ contains only the demosaicing transformation, then the model would be as in Fig. \ref{raw_gan_structure}. Both the generated images and real RAW Bayer pattern images are demosaiced to color images before feeding into the discriminator $D_T$, which is equivalent to adding a constraint to the generated images, ensuring they can be demosaiced. That is, the generated images would maintain the Bayer pattern.

\subsubsection*{Demosaicing}
Fundamentally, demosaicing can be regarded as an interpolation operation. A typical demosaicing process is illustrated in Fig. \ref{demosaicing}. A Bayer pattern image is first split into three channels, i.e., R, G, and B, and the missing components are interpolated by using the information of existing components. There are different ways to interpolate these missing pixel components, which can be divided into two broad categories. The first one is to interpolate using only the pixels from the same channel. Typical methods in this category includes nearest neighbor interpolation, bilinear interpolation, cubic interpolation, etc. Another category is based on spectral correlation rules such as color ratio constancy or color difference constancy rules, during demosacing process. These methods take the strong spectral correlation between color components into account and generally obtain better performance\cite{RN1}.

For a specific Bayer pattern,  e.g., the $RGGB$ pattern as in Fig. \ref{demosaicing}, the demosaicing process can be described as
\begin{equation}
    \label{demosaic_eq}
    I_{k}=func[I_{bayer}\odot(\left \{P_{R},P_{G},P_{B}\right \}\otimes J)].
\end{equation}
Here, $\odot$ representes the Hadamard product operation, $\otimes$ is the Kronecker product operation, $func$ is a specific interpolation method for demosaicing and $J$ is an all-one matrix. For an $m\times n$ Bayer pattern image $I$, $J$ is a $\frac{m}{2} \times \frac{n}{2}$ matrix, $P_{k}$ is the $2\times 2$ Bayer pattern template, e.g., for $RGGB$ pattern,
$P_{R}=\begin{bmatrix}
        1 & 0\\
        0 & 0
        \end{bmatrix}$,
$P_{G}=\begin{bmatrix}
        0 & 1\\
        1 & 0
        \end{bmatrix}$,
$P_{B}=\begin{bmatrix}
        0 & 0\\
        0 & 1
        \end{bmatrix}$, $k\in \left \{ R,G,B \right \}$.
\vspace{0.1cm}

Correspondingly, for a demosaiced (without any further processing) three-channel color image, since the original captured pixels are not affected by interpolation, we can obtain them directly to restore the Bayer pattern image as
\begin{equation}
    \label{reverse_demosaicing}
    I_{bayer}=\sum (I_{k}\odot(P_{k}\otimes J)).
\end{equation}
This means that the demosaicing process is invertible.

Moreover, the Hadamard product $\odot$ and the Kronecker product $\otimes$ operations are differentiable. For interpolation operations, they only include simple four-arithmetic operations, which is also differentiable in the training process.

\subsubsection*{Generate Bayer Pattern Images with Arbitrary Size}
Note that the RAW Bayer pattern images always have a large size. For example, images in the PASCALRAW dataset have a resolution of $4012\times 6034$. Training networks using images with such a high resolution will consume a lot of storage and computing resources, making them even not applicable for training. For many computer vision tasks, the size of input images are much smaller than that in the PASCALRAW dataset. For example, classification tasks usually take images with a resolution of $224\times 224$ \cite{RN81,RN82,RN83,RN84}. As for objection detection tasks, Faster R-CNN typically ingests $1000 \times 600$ images \cite{RN85}, single shot multibox detector (SSD) consumes $300\times 300$ or $512\times512$ images\cite{RN86}, RetinaNet takes $500\times500$ or $800\times 800$ input images \cite{RN90}, and YOLOv3 typically runs on either $320\times 320$ or $416\times 416$ or $608\times608$ images\cite{RN89}. However, down-sampling the Bayer pattern images directly will destroy the `mosaic' structure and result in the loss of color information\cite{RN64}. Although down-sampling with superpixel structure as proposed in \cite{RN64} can maintain the mosaic structure, the image quality may decrease obviously when the down-sampling factor is large.

One way to solve this problem is to split the images into small patches during the training process\cite{RN91}. However, since only local information can be learned, this solution is not friendly to model learning. The trained model may perform well on small patches, but there will be obvious stitches when stitching the small patches back into a complete image\cite{RN91}.

In this work, to generate images with arbitrary size, another transformation $T_2$ is added to the set $T$ to resize the demosacied image to a desired size. Since the resize operation is performed on demosaiced images, the previously mentioned problems on image quality and `mosaic' structure can be completely avoided.

\subsubsection*{Other Transformations}
Expect for above-mentioned transformations, other transformations or steps in ISP can also be added to set $T$. For example, the white balance stage, which multiple three parameters to $R$, $G$ and $B$ channel respectively, and the gamma compression is to compute the $a$ power of each pixel, where $a\in \left [ 0,1 \right ]$, both of them are invertible and differentiable transformations. Moreover, the proposed method is deduced from the GAN loss, which has the potential to be extended to other GAN models. In this paper, the proposed method is implemented on an improved pix2pixHD model to demonstrate the effectiveness of the proposed method. Detailed experimental results and corresponding discussions are presented in Section \ref{sec:image_quality}.

\subsection{Proposed Model Architecture}
In this work, cGAN is used as an example to demonstrate the proposed method. The overall architecture of the proposed model is illustrated in Fig. \ref{architecture}. Multi-scale generator and discriminator architectures are employed in the proposed model. The generator, consisting of two sub-networks, i.e., $G_1$ and $G_2$, generates images in a coarse-to-fine manner. The discriminators, consisting of $D_1$ and $D_2$, are  trained to discriminate real and synthesized images at 2 different scales. The inputs to $G$ are the finely rendered images, and the inputs to $G_1$ are $2\times$ down-sampled version of their $G_2$ counterparts. The inputs to $D$ are the concatenation of demosaiced version of $G$'s outputs and the corresponding ground truth. Both $G_1$ and $G_2$ consist of a convolution front-end, $n$ residual blocks, and a transposed convolution back-end.

The following modifications have been made to improve the quality of the synthesized images.
\begin{itemize}
    \item Water droplets artifacts elimination. In our initial design, we observed water droplets artifacts in the synthesized images as described in \cite{RN94}. Analysis results show that they are caused by instance normalization. Following the suggestion in \cite{RN94}, we replace the instance normalization with a weight demodulation operation as
    \begin{equation}
        \label{demodulate}
        w'_{ijk_hk_w}=\frac{s_i\cdot w_{ijk_hk_w}}{\sqrt{\sum _{i,k_h,k_w}(s_i\cdot w_{ijk_hk_w})^2+\varepsilon}},
    \end{equation}
    which is baked to a single convolution layer named demodulated convolution (DemodConv in Fig. \ref{architecture}). Here, $w$ is the original weight while $w'$ is the demodulated weights, $s_i$ is the scale factor corresponding to the $i$-th input feature map, $j$ is the $j$-th output feature map, $k_h$ and $k_w$ are the width and height of the kernel, respectively, and $\varepsilon$ is a small constant to avoid numerical issues.
    \item Checkboard artifacts alleviation. The checkboard artifacts can be observed when generating images by neural networks. This kind of artifacts is usually caused by deconvolution operation\cite{RN95}. To alleviate the checkboard artifacts, we replace the deconvolution layers with ``upsample$+$demodulated convolution'' operations, that is, the feature maps are first upsampled by nearest-neighbor, followed by a demodulate convolution operation \cite{RN110}.

    \item To match the Bayer pattern. It has been mentioned in Section~\ref{section:sec1} that the Bayer pattern images consist of alternating $RGGB$ pixels. In order to matain the Bayer pattern, the last module of $G_2$ contains a "demodulated convolution $+$ periodic shuffing" operation. More specially, suppose there is a $3\times H\times W$ input color image and we want to generate an $H\times W$ Bayer pattern image. The output of $G_2$'s last convolution layer is a $4\times \frac{H}{2}\times\frac{W}{2}$ feature
    map, where each channel represents $RGGB$. Then a periodic shuffing operation \cite{RN96} is applied to generate an $H\times W$ image. Similarly, the input of $D_1$ is first average pooled to $4\times \frac{H}{4}\times\frac{W}{4}$, then a $\frac{H}{2}\times\frac{W}{2}$ image is generated by periodic shuffing.

\end{itemize}

\subsection{The Overall Loss Function}
The overall loss function of the proposed model can be expressed as
\begin{equation}
\label{overall loss function}
    \mathcal{L}=\mathcal{L}_{GAN}+\alpha_{1}\mathcal{L}_{FM}+\alpha_{2}\mathcal{L}_{VGG},
\end{equation}
where, $\mathcal{L}_{GAN}$ is the adversarial loss, $\mathcal{L}_{FM}$ is the feature matching loss, $\mathcal{L}_{VGG}$ is the perceptual loss and $\alpha_{1}$, $\alpha_{2}$ control the importance of $\mathcal{L}_{FM}$ and $\mathcal{L}_{VGG}$.
\begin{itemize}
    \item The adversarial loss is used to enable the model to generate data as close to the real data distributions as possible. With the two-scale discriminators $D_1$ and $D_2$ as shown in Fig. \ref{architecture}, the adversarial loss can be formulated as

        \begin{equation}
            \mathcal{L}_{GAN}=\underset{G}{min}\,\underset{D^1_T,D^2_T}{max}\,\sum_{k=1,2} V(G,D^k_T),
        \end{equation}
    where $V(G,D^k_T)$ is the single adversarial loss of the $k$-th discriminator $D_{k}$, as formulated in \eqref{mygan_loss_function}.

    \item The feature matching loss is used for learning information from multi-scale features by matching the intermediate feature maps of discriminator $D^k_T$ between the real and the synthesized images, which can be formulated as
    \begin{equation}
        \mathcal{L}_{FM}=\underset{G}{min}\,\sum_{k=1,2} l_{FM}(G,D^k_T).
    \end{equation}
    Here, $l_{FM}(G,D^k_T)$ is the feature matching loss from the $k$-th discriminator, given by
    \begin{equation}
        l_{FM}(G,D^k_T)=\mathbb{E}_{y\sim p^T_{g}}\sum_{i=1}^{M}\frac{1}{C_iW_iH_i}\left [ \left \| F_i(x)-F_i(y) \right \|_1 \right ],
    \end{equation}
    where $F_i(\cdot)$ is the output of $i$-th intermediate layer of discriminator $D_T^k$, $\|\cdot \|_1$ is the L1 distance, $M$ is the total number of layers and $H$, $W$, $C$ represents the height, width and channels of corresponding output feature maps, respectively.
    \item The perceptual loss is used for keeping the perceptual and semantic
    fidelity. Based on a pre-trained VGGNet \cite{RN109}, the perceptual loss is defined as
    \begin{equation}
        \mathcal{L}_{VGG}(x,y)=\underset{G}{min}\sum_{i=1}^{5}\frac{\lambda_i}{C_iW_iH_i}\left [ \left \| B_i(x)-B_i(y) \right \|_1 \right ].
    \end{equation}
    Here, $B_i$ is the $i$-th block of VGGNet and $\lambda_i$ is the corresponding weight coefficient of the $i$-th block.
\end{itemize}

\section{Experiments}
In this section, we implement the proposed method on an improved pix2pixHD model to convert the finely rendered color images to the corresponding Bayer pattern images. Experimental results on image quality as well as performance for computer vision tasks are presented to demonstrate the effectiveness of the proposed method. Datasets are first introduced in this section, followed by the details of the experiments setup and experimental results.
\subsection{Datasets}
There are two datasets used in our experiments, which are the Zurich RAW2RGB (ZRR) dataset \cite{RN100} and the PASCALRAW dataset \cite{RN55}. The ZRR dataset consists of RAW photos captured by a Huawei P20 mobile phone and a Canon 5D DSLR. These images are cropped into small patches. The images taken by Huawei P20 are used in our experiments because there are paired color and RAW images. The cropping procedure results in 90 thousand training patches and 2.4 thousand testing patches.
The PASCALRAW dataset, containing 4259 annotated RAW images and 3 annotated object classes (car, person and bicycle), is modeled following the manner of the PASCAL VOC dataset \cite{RN99}. All these images are captured by a Nikon D3200 DSLR camera, where both RAW images ($RGGB$ pattern) and the corresponding finely rendered color images are saved.

\subsection{Model Configuration}

In our training process, the initial learning rate is fixed to $2\times10^{-4}$ for discriminator and $2\times10^{-5}$ for generator. For optimization, an Adam optimizer with $\beta_1=0.5$, $\beta_2=0.999$ is used and the batch size is set to 4. The proposed model is implemented with PyTorch on four NVIDIA GeForce RTX 2080 TI GPUs and trained for more than $460$k steps.

\subsection{Image Quality Assessment Methods}
To evaluate the quality of synthesized Bayer pattern images, different image quality assessment methods are used in our experiments.
The Fréchet inception distance (FID) \cite{RN105}, which calculates the squared Wasserstein metric between two multidimensional Gaussian distributions, is a commonly used metric to evaluate the quality of images generated by GAN. Lower FID scores indicates higher similarity between two groups of images. The FID implementation in \cite{RN106} is adopted in our evaluation.
\begin{figure}[t
    \centering

    \subfigure{
    \includegraphics[width=1.1in]{./figures/synthesized/gt/7.png}}
    \setcounter{subfigure}{0}
    \hspace*{-4.2mm}
    \subfigure[]{
    \label{fig:ZRDD_gt}
    \includegraphics[width=1.1in]{./figures/synthesized/gt/17.png}}
    \hspace*{-3mm}
    \subfigure{
    \includegraphics[width=1.1in]{./figures/synthesized/gt/1152.png}}

    \subfigure{
    \includegraphics[width=1.1in]{./figures/synthesized/pix2pixhd/zoom_image/7.png}}
    \setcounter{subfigure}{1}
    \hspace*{-4.2mm}
    \subfigure[]{
    \label{fig:ZRDD_pix2pix}
    \includegraphics[width=1.1in]{./figures/synthesized/pix2pixhd/zoom_image/17.png}}
    \hspace*{-3mm}
    \subfigure{
    \includegraphics[width=1.1in]{./figures/synthesized/pix2pixhd/zoom_image/1152.png}}

    \subfigure{
    \includegraphics[width=1.1in]{./figures/synthesized/proposed/7.png}}
    \hspace*{-4.2mm}
    \setcounter{subfigure}{2}
    \subfigure[]{
    \label{fig:ZRDD_propose}
    \includegraphics[width=1.1in]{./figures/synthesized/proposed/17.png}}
    \hspace*{-3mm}
    \subfigure{
    \includegraphics[width=1.1in]{./figures/synthesized/proposed/1152.png}}

    \caption{Comparison between (a) original RAW Bayer images from ZRR dataset, (b) images generated by original pix2pixHD and (c) images generated by the proposed model. All these images are demosaiced using hybrid interpolation for display purpose.}
    \label{fig:compare_ZRDD}
\end{figure}

\begin{figure*}[t
    \centering

    \subfigure[]{
    \label{fig:ZRDD_Examples_color}
    \includegraphics[width=2.3in]{./figures/mosaic_imgs/ZRDD/input.pdf}}
    \hspace{-3mm}
    \subfigure[]{
    \label{fig:ZRDD_Examples_bayer}
    \includegraphics[width=2.3in]{./figures/mosaic_imgs/ZRDD/gt.pdf}}
    \hspace{-3mm}
    \subfigure[]{
    \label{fig:ZRDD_Examples_syn}
    \includegraphics[width=2.3in]{./figures/mosaic_imgs/ZRDD/syn.pdf}}
    \hspace{-3mm}

    \caption{Examples of image patches form the ZRR dataset. (a) The original color images. (b) The demosaiced original Bayer pattern images. (c) The demosaiced synthesized Bayer pattern images.}
    \label{fig:ZRDD_Examples}
    \end{figure*}

\begin{figure*}[t
    \centering

    \subfigure[]{
    \label{fig:PASCAL_original_color}
    \includegraphics[width=3.5in]{./figures/2014_001596/2014_001596.jpg}}
    \hspace{-3.5mm}
    \subfigure[]{
    \label{fig:PASCAL_gt_bayer}
    \includegraphics[width=1.75in]{./figures/2014_001596/2in1bayer.pdf}}
    \hspace{-3.5mm}
    \subfigure[]{
    \label{fig:PASCAL_gt_color}
    \includegraphics[width=1.75in]{./figures/2014_001596/2in1color.pdf}}

    \caption{Visual comparison of images processed by different pipelines. (a) An original color image from the PASCALRAW dataset. (b) The demosaiced version of the corresponding original (top) and synthesized (bottom) Bayer pattern images. (c) The fully processed original (top) and synthesized (bottom) Bayer pattern images by a simple ISP pipeline.}
    \label{fig:compare_PASCAL}
    \end{figure*}

    \begin{figure*}[t
        \centering

        \subfigure[]{
        \label{fig:pascal_Examples:color}
        \includegraphics[width=2.3in]{./figures/mosaic_imgs/pascal/input.pdf}}
        \hspace{-3mm}
        \subfigure[]{
        \label{fig:pascal_Examples:bayer}
        \includegraphics[width=2.3in]{./figures/mosaic_imgs/pascal/gt.pdf}}
        \hspace{-3mm}
        \subfigure[]{
        \label{fig:pascal_Examples:syn}
        \includegraphics[width=2.3in]{./figures/mosaic_imgs/pascal/syn.pdf}}
        \hspace{-3mm}

        \caption{Examples of original and synthesized images form the PASCALRAW dataset. (a) The original color images. (b) The fully processed (using a simple ISP pipeline) original Bayer pattern images. (c) The fully processed synthesized Bayer pattern images.}
        \label{fig:pascal_Examples}
        \end{figure*}

The peak signal to noise ratio (PSNR) and mean structural similarity (MSSIM) \cite{RN16} are two commonly used full-reference image quality metrics. For two given images, the PSNR can be calculated as
\begin{equation}
    \label{psnr}
    PSNR=10\log_{10}\left(\frac{(2^b-1)^2}{MSE}\right).
\end{equation}
Here, $b$ is the bit depth of images, and mean squared error (MSE) is defined as
\begin{equation}
    \label{mse}
    MSE=\frac{1}{H\times W}\sum ^H_{h=1} \sum ^W_{w=1}(I_1(h,w)-I_2(h,w)),
\end{equation}
where $H$ and $W$ are the height and width of images, respectively.
To maintain the consistency with MSE metric, in our experiments, the average PSNR among a specific dataset is calculated from average MSE as
\begin{equation}
    \label{ave_psnr}
    AVE\_PSNR=10\log_{10}\left(\frac{(2^b-1)^2}{\frac{1}{n}\sum_{i=1}^{n}MSE_i}\right),
\end{equation}
where $n$ is the number of images in the corresponding dataset.

MSSIM compares the luminance, contrast and structure of two images. To compute MSSIM, the images are splited into blocks and the average SSIM among all block pairs is calculated. Note that MSSIM ranges from 0 to 1, where 1 means two images are identical.

\begin{table}[]
    \centering
    \caption{Comparison Results between The Demosaiced Real Bayer Pattern Images and Demosaiced Synthesized Bayer Pattern Images}
    \label{tab:compare_eresults_ZRDD}
    \setlength{\tabcolsep}{10pt}{
    \begin{tabular}{lccc}
    \toprule
                        &\textbf{FID}    &\textbf{PSNR}   &\textbf{MSSIM}\\
    \midrule
    Original pix2pixHD    &22.87         &18.45  &0.86\\
    Proposed              &13.38         &20.17   &0.88\\
    \bottomrule
    \end{tabular}
    }
\end{table}

\subsection{Image Quality Comparison}
\label{sec:image_quality}
We implement the proposed model and  train it on the ZRR dataset. To better demonstrate the effectiveness of the proposed architecture, we also train an original pix2pixHD framework \cite{RN69} with same settings. The trained models are tested on ZRR testing set (2462 pairs) for image quality evaluation. Note that we use hybrid interpolation \cite{RN108} for all the transformation $T$ in our model.

Table \ref{tab:compare_eresults_ZRDD} presents the comparison results between the original Bayer pattern images and synthesized Bayer pattern images. Both images are demosaiced using hybrid interpolation, and the FID, PSNR and MSSIM metrics are calculated with demosaiced ground truth, Bayer pattern images (the original RAW Bayer images from the cameras). It can be found that for all these metrics, the proposed model outperforms the original pix2pixHD model. To have a visual comparison, Fig. \ref{fig:compare_ZRDD} shows the demosaicd versions of ground truth, images generated by the original pix2pixHD and images generated by the proposed method. It can be found from Fig. \ref{fig:ZRDD_pix2pix} that the original pix2pixHD model may generate random artifacts (highlighted by boxes) in the training process and hard to be get rid of. Moreover, there will be obvious differences in the color saturation and overall brightness between the images generated by the original pix2pixHD model and the ground truth, while images generated by the proposed model is closer to the ground truth visually. Fig. \ref{fig:ZRDD_Examples} gives more examples from the ZRR dataset.

As mentioned before, the proposed model is able to generate Bayer pattern images with arbitrary size. To achieve this, another transformation $T_2$ needs to be added after the demosaicing step to resize the image to the desired size. As we have discussed in the Introduction, directly change the size of Bayer pattern images will cause image quality degradation and loss of color information. The proposed method avoids these problems. As a visual comparison, Fig. \ref{fig:compare_PASCAL} shows (a) the original color image, (b) the demosaiced (using hybrid interpolation) original (real) and synthesized Bayer pattern images, (c) the fully processed original and syhthesized Bayer pattern images. Note that the images in Fig. \ref{fig:PASCAL_gt_color} are processed by a simple ISP pipeline consisting of linearization, denoise, demosaicing, white balance and gamma compression. To compute the indices, the real and synthesized Bayer pattern images are demosaiced using hybrid interpolation. The average FID, PSNR and MSSIM calculated between demosaiced real Bayer pattern images and demosaiced synthesized Bayer pattern images are 6.956, 29.061 and 0.887, respectively, indicating the synthesized images have a good quality and very close to real images visually.

It can also be found that there exists difference between the Bayer pattern images processed by different ISP pipelines. Take white balance as an example, Fig. \ref{fig:PASCAL_original_color} is processed by the camera's white balance algorithm, while images in Fig. \ref{fig:PASCAL_gt_color} are processed by the gray world algorithm \cite{RN107}. Fig. \ref{fig:pascal_Examples} shows more examples of images from the PASCALRAW dataset. As can be seen,  mapping a rendered color images to their corresponding Bayer domain can also be used for re-rendering images in photography.

\begin{figure}[t
    \centering

    \subfigure[]{
    \label{fig:boxes:gt}
    \includegraphics[width=1.7in]{./figures/2014_001203/gt.png}}
    \hspace{-3mm}
    \subfigure[]{
    \label{fig:boxes:color}
    \includegraphics[width=1.7in]{./figures/2014_001203/color.png}}
    \hspace{-3mm}
    \subfigure[]{
    \label{fig:boxes:bayer}
    \includegraphics[width=1.7in]{./figures/2014_001203/bayer.png}}
    \hspace{-3mm}
    \subfigure[]{
    \label{fig:boxes:syn}
    \includegraphics[width=1.7in]{./figures/2014_001203/syn.png}}
    \hspace{-3mm}

    \caption{Detected bounding boxes when apply different models on original Bayer pattern images. (a) The ground truth. (b) The model trained on the color images. (c) The model trained on the original Bayer pattern images. (d) The model trained on the synthesized Bayer pattern images.}
    \label{fig:boxes}
    \end{figure}

\subsection{Bayer Pattern Images for End-to-end Computer Vision}
As mentioned before, there are gaps between training data and testing data for end-to-end (from RAW Bayer image to computer vision results) computer vision architectures. Nowadays, almost all the computer vision models are trained on finely rendered color images, such that directly apply these models to Bayer pattern images will cause significant performance degradation. Since the synthesized RAW Bayer pattern images and finely rendered color images correspond to the same senses, we propose to use the synthesized Bayer pattern images generated by our method to train or fine-tune the computer vision models to improve their performance on Bayer pattern images, enabling end-to-end (from Raw Bayer images to vision tasks) computer vision system design. Note that in end-to-end vision systems, since the entire ISP pipeline is skipped, the overall system complexity will be reduced significantly \cite{RN64}, leading to reduced power consumption and more compact system design. Moreover, by skipping the ISP pipeline, the  latency of the entire vision system, which is crucial in applications such as autonomous vehicles, can also be significantly reduced.

In this experiment, the object detection task is taken as an example, where Faster R-CNN, SSD and YOLO-v3 frameworks are used as detection backbones. Since the PASCALRAW dataset provides both RAW Bayer pattern images and color images, we convert the rendered color images in the PASCALRAW dataset to their Bayer pattern counterparts, and train models on color images, original Bayer pattern images and synthesized Bayer pattern images to evaluate their performance.

Table \ref{tab:object_detection_corresponding} presents the object detection results, where training and evaluation are performed on same version of datasets, e.g., both training and testing are performed on the color image dataset. It can be found that for all these frameworks, on the strict testing standards, i.e., AP (the Intersection-over-Union (IoU) from 0.5:0.95) and AP\textsubscript{75} (IoU=0.75), the detection performance on finely rendered color images is slightly better (about $2\%\sim5\%$) than that on original Bayer and synthesized Bayer datasets. But these difference will be further reduced for AP\textsubscript{50} (IoU=0.5). This indicates that some stages in the ISP pipeline may be beneficial for object detection tasks, or more likely the models are designed for finely rendered images without taking the properties of Bayer pattern images, such as alternating RGB pattern, into account. Moreover, the detection rate on original Bayer dataset is very close to that on synthesized Bayer image dataset for all these testing standards.

Table \ref{tab:object_detection_onbayer} presents the object detection results where the models are trained on color, original Bayer and synthesized Bayer datasets, and evaluations are performed on original Bayer pattern images. As can be seen from row 2, 5 and 8 of Table \ref{tab:object_detection_corresponding} and Table \ref{tab:object_detection_onbayer}, the average precision of object detection decreases significantly when apply the model trained on color dataset to original Bayer pattern images directly. The performance degradation can be as high as 10\% for the strict testing standards. But if we train the model on synthesized Bayer images, and apply it on original Bayer pattern images, the object detection performance can be improved significantly, which is very close the model trained on original Bayer pattern images. Moreover, it can be found from Table \ref{tab:object_detection_corresponding} and Table \ref{tab:object_detection_onbayer} (rows 4, 7 and 10) that if the model is trained on synthesized Bayer pattern image dataset, there is little difference on performance when evaluated on synthesized or original Bayer pattern image dataset.

Fig. \ref{fig:boxes} gives the ground truth (Fig. \ref{fig:boxes:gt}) and detection performance when apply the models trained on different datasets to original Bayer pattern images (Fig. \ref{fig:boxes:color}-\ref{fig:boxes:syn}). Note that detected objects (the bounding boxes) are drawn on the finely rendered color images for the convenience of visuality. It can be found from Fig. \ref{fig:boxes:color} that there are detection failures when apply the model trained on the color images to the original Bayer pattern images directly, which are also the main cause of the performance degradation presented in Table \ref{tab:object_detection_onbayer}. One of the problem is the miss detection, especially for small objects or objects in the dark scenes. For example, the person who ride a bicycle on the left upper corner in Fig. \ref{fig:boxes:color}. Another problem is that the detector can only recognize part of the objects, leading to the bounding boxes cannot cover the objects completely (the car in Fig. \ref{fig:boxes:color}). In this case, although the objects are detected, the bounding boxes will be regarded as false positive since the IoU score between the bounding boxes and the ground truth does not meet the threshold. Moreover, for the models trained on the original Bayer pattern images and synthesized Bayer pattern images in Fig. \ref{fig:boxes:bayer} and \ref{fig:boxes:syn}, the detection results are close to each other, except for a slight difference in confidence score and the boundary bounding boxes, which can be judged as true positive.

\begin{table}
    \caption{Object Detection Results When Training and Evaluation are on the Same Version of Dataset}
    \label{tab:object_detection_corresponding}
    \centering
    \setlength\arrayrulewidth{0.3pt}
    \renewcommand{\arraystretch}{1.2}
    \setlength{\tabcolsep}{10pt}{
    \begin{tabular}{ccccc}
    \toprule
    \multicolumn{1}{l}{}         & Training Set & AP & AP\textsubscript{50} & AP\textsubscript{75}  \\
    \cline{2-5}
    \multirow{3}{*}{Faster R-CNN} & Color        & 71.94                 & 96.54                & 87.17                 \\
                                 & Bayer        & 69.56                 & 95.46                & 84.21                 \\
                                 & Synthesized  & 69.19                 & 95.13                & 83.52                 \\
    \cline{2-5}
    \multirow{3}{*}{SSD300}      & Color        & 66.48                 & 94.59                & 79.69                 \\
                                 & Bayer        & 63.91                 & 92.90                & 75.49                 \\
                                 & Synthesized  & 63.33                 & 92.33                & 74.74                 \\
    \cline{2-5}
    \multirow{3}{*}{Yolo-v3}     & Color        & 73.32                 & 96.21                & 88.25                 \\
                                 & Bayer        & 71.29                 & 95.37                & 85.31                 \\
                                 & Synthesized  & 70.86                 & 95.41                & 85.20                 \\
    \bottomrule
    \end{tabular}}
    \end{table}

\begin{table}
    \caption{Object Detection Results When Train Models on Different Datasets and Perform Evaluation on Real Bayer Dataset}
    \label{tab:object_detection_onbayer}
    \centering
    \setlength\arrayrulewidth{0.3pt}
    \renewcommand{\arraystretch}{1.2}
    \setlength{\tabcolsep}{10pt}{
    \begin{tabular}{ccccc}
    \toprule
    \multicolumn{1}{l}{}         & Training Set & AP & AP\textsubscript{50} & AP\textsubscript{75}  \\
    \cline{2-5}
    \multirow{3}{*}{Faster R-CNN} & Color        & 62.09                   & 91.26                  & 74.67                   \\
                                 & Bayer        & 69.56                   & 95.46                  & 84.21                   \\
                                 & Synthesized  & 68.81                   & 94.83                  & 82.95                   \\
    \cline{2-5}
    \multirow{3}{*}{SSD300}      & Color        & 57.45                   & 86.85                  & 67.14                   \\
                                 & Bayer        & 63.91                   & 92.90                  & 75.49                   \\
                                 & Synthesized  & 63.21                   & 92.58                  & 73.94                   \\
    \cline{2-5}
    \multirow{3}{*}{Yolo-v3}     & Color        & 64.54                   & 90.93                  & 75.76                   \\
                                 & Bayer        & 71.29                   & 95.37                  & 85.31                   \\
                                 & Synthesized  & 70.82                   & 95.31                  & 85.20                   \\
    \bottomrule
    \end{tabular}}
    \end{table}

\section{Conclusion}
 In this paper, we propose a method to add constraints that are un-formulatable in GAN training. By using the proposed method, we implemented a model to synthesize RAW Bayer pattern images with arbitrary size from their color image counterparts. Based on the invariant of JS divergence between two distributions, we first theoretically deduce the feasibility of using transformed generated data to improve the learning of the generator on original data. Then the proposed architecture is implemented, along with some well-designed modules, which match the structure of Bayer pattern to improve the quality of synthesized images. Experimental results show that under the same training condition, the proposed method is able to generate RAW Bayer pattern images with better FID score, PSNR and MSSIM than existing methods, and the training process is more stable. The proposed method can be used to generate image dataset for applications which need RAW Bayer pattern images. For example, works aiming to explore the optimal configuration of ISP pipeline for CV performance (instead of for photography), works in the field of in-sensor computing/smart image sensor design, or even photography. It is also shown in the experiments that by training object detection frameworks using the synthesized RAW Bayer images, they can work in an end-to-end manner with negligible performance degradation, i.e., RAW Bayer images (from the image sensor) can be directly fed to object detection algorithms without image processing. By skipping the ISP pipeline, both computational complexity and power consumption of the overall vision system can be significantly reduced. Moreover, without an image processor in the middle, image sensors can be directly integrated with back-end vision processors, making in-sensor computing a practical approach.

\bibliographystyle{ieeetr}