\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the ICCV 2021 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for ICCV 2021.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\iccvfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
tech reports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a tech report for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the tech report as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the ICCV70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.

Page numbers should be included for review submissions but not for the
final paper. Review submissions papers should have page numbers in the
footer with numbers centered and .75 inches (1.905 cm) from the bottom
of the page and start on the first page with the number 1.

Page numbers will be added by the publisher to all camera-ready papers
prior to including them in the proceedings and before submitting the
papers to IEEE Xplore. As such, your camera-ready submission should
not include any page numbers. Page numbers should automatically be
removed by uncommenting (if it's not already) the line
\begin{verbatim}
\end{verbatim}
near the beginning of the .tex file.

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centered.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the ICCV 2021 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

{\small
\bibliographystyle{ieee_fullname}

\section{Discussion and future works}
Our approach is the first to provide robust 3D human pose and shape outputs of each individual in in-the-wild crowd scenes.
However, it is still demanding to recover accurate 3D poses from images with extremely close interactions, such as tussles in sports.
The extreme cases often involve challenging poses and severe inter-person occlusion, both of which are rare in existing train data.
Although~\cite{li2019crowdpose,zhang2019pose2seg,qiu2020peeking} recently proposed crowd datasets, they only provide 2D annotations and a limited number of images.
As a result, the effect of additionally training on such datasets is marginal, as shown in the fourth and fifth rows of Table~\ref{table:sota_3dpw_crowd}.
Future works should suggest a 3D train dataset that covers diverse poses and occlusion in crowd scenes.

\section{Conclusion}
We present 3DCrowdNet, the first single image-based 3D human pose and shape estimation system that targets in-the-wild crowd scenes.
Previous methods suffer from noisy human bounding boxes and inter-person occlusion in crowd scenes.
We resolve the issue by leveraging crowd-scene robust 2D pose outputs.
The proposed 3DCrowdNet is capable of focusing on a target person in crowd scenes, and effectively utilizes the articulation information in the 2D poses.
We provide extensive ablation studies for the crowd-scene robust system, which would give meaningful insight for future works.
Also, we show that 3DCrowdNet provides far more robust outputs on in-the-wild crowd scenes than previous methods numerically and qualitatively.

\section{More qualitative results.}

\begin{figure}
\begin{center}
\includegraphics[width=1.0\linewidth]{fig/3dview_suppl.pdf}
\end{center}
  \caption{
  Visualization of 3D shapes from different viewpoints, given inaccurate input 2D poses. Our 3DCrowdNet can recover a 3D shape that best describes the target person in an image, even when provided with inaccurate 2D poses, using both the 2D pose and image feature.
  }
\label{fig:pose_correction}
\end{figure}

\noindent\textbf{Pose correction.}
Figure~\ref{fig:pose_correction} shows that our 3DCrowdNet can estimate robust 3D pose and shape given inaccurate 2D poses in crowd scenes.
Due to inter-person occlusion and overlapping bounding boxes between people, 2D pose estimators~\cite{cao2017realtime,cheng2020higherhrnet} may produce inaccurate joint predictions as shown in the first, second, and third rows.
In this regard, 3DCrowdNet assigns don't-care values to the joint predictions with low confidence (\textit{e.g.} lower than $0.1$ for outputs from Cheng~\etal~\cite{cheng2020higherhrnet}) at first.
Then 3DPoseNet of 3DCrowdNet refines the inaccurate input 2D pose using image feature that contains the context information in images.
Last, 3DShapeNet of 3DCrowdNet estimates human model parameters, SMPL~\cite{loper2015smpl} parameters.
The whole process makes our system predict a 3D human shape that best describes a target person in images and is plausible, even when provided with inaccurate 2D joint predictions.
The fourth and fifth rows prove that our approach is also effective on estimating robust 3D shapes for truncated images.

\begin{figure*}
\vspace*{2mm}
\begin{center}
\includegraphics[width=1.0\linewidth]{fig/i2l_p2m_comp.pdf}
\end{center}
\vspace*{-3mm}
  \caption{
  Qualitative comparison on the CrowdPose~\cite{li2019crowdpose} test set. From left, an input image, input 2D poses estimated by Cheng~\etal~\cite{cheng2020higherhrnet}, 3DCrowdNet, I2L-MeshNet~\cite{moon2020i2l}, and Pose2Mesh~\cite{choi2020p2m} outputs. Our 3DCrowdNet successfully disentangles a target person from other people in a bounding box compared to I2L-MeshNet. Also, 3DCrowdNet produces a 3D shape that best describes a target person in images, while Pose2Mesh estimates a plausible 3D shape for given 2D poses, which could not correspond to images. 3DCrowdNet and I2L-MeshNet use the same bounding boxes to crop an image for each person. 3DCrowdNet and Pose2Mesh use the same 2D poses from Cheng~\etal~\cite{cheng2020higherhrnet}.
  }
\vspace*{-3mm}
\label{fig:i2l_p2m_quality_comparison}
\end{figure*}

\begin{figure*}
\begin{center}
\includegraphics[width=1.0\linewidth]{fig/failures.pdf}
\end{center}
\vspace*{-3mm}
  \caption{
  Failure cases of 3DCrowdNet.
  }
\label{fig:failure_quality}
\end{figure*}

\noindent\textbf{Comparison with I2L-MeshNet~\cite{moon2020i2l} and Pose2Mesh~\cite{choi2020p2m}.}
Figure~\ref{fig:i2l_p2m_quality_comparison} shows the qualitative comparison between 3DCrowdNet, I2L-MeshNet~\cite{moon2020i2l}, and Pose2Mesh~\cite{choi2020p2m}.
Among the three methods, 3DCrowdNet apparently produces much more robust 3D shapes on the crowd scenes.
I2L-MeshNet tends to provide noisy body shape estimation in crowd scenes.
It misses a person in images with overlapping bounding boxes as depicted in the fifth and sixth rows.
Limb outputs in the second, fourth, and sixth rows are inaccurately estimated or swapped with other people.
Overall, I2L-MeshNet is hard to disentangle people in overlapping bounding boxes and provides inaccurate outputs when a body is occluded in crowd scenes.

Pose2Mesh estimates more stable 3D shapes than I2L-MeshNet in crowd scenes, but the 3D poses of Pose2Mesh are different from the poses depicted in the input images.
The legs of people in the second, third, sixth rows are inaccurately estimated.
Especially, Pose2Mesh wrongly predicts the legs of the most right person (apricot color) in the third row; although the input 2D pose is correct.
Also, in the first row, the most front person's global orientation prediction (pink color) is inaccurate.
The evidence proves that solely relying on 2D geometry information and not leveraging image feature has limitations for crowd-scene robust 3D shape estimation, since the approach produces plausible 3D shapes for given 2D geometry, not 3D shapes that best describe a person in images.

To sum up, our 3DCrowdNet estimates the most accurate 3D pose and shape from in-the-wild crowd scenes compared to the competitors~\cite{moon2020i2l,choi2020p2m} by leveraging image feature, the guidance of crowd-scene robust 2D pose outputs, and the articulation information in 2D poses.

\noindent\textbf{Failure cases of 3DCrowdNet.}
As Figure~\ref{fig:failure_quality} shows, 3DCrowdNet falls short in images with extreme interactions, such as tussles in sports.
Input 2D poses, which are the outputs of Cheng~\etal~\cite{cheng2020higherhrnet}, are highly inaccurate in such crowd scenes, and 3DCrowdNet does not effectively estimate the correct 3D pose and shape from them.
Severe inter-person occlusion, challenging poses, and similar appearance of clothes are the main causes of the failures.
For example, in the left bottom image, 3DCrowdNet wrongly predicts a green-colored person's right leg, who is tackling in a soccer game.
Data with the above situations is scarce in existing train datasets, and a large-scale train dataset that primarily covers such crowd scenes should be introduced.

\begin{figure}
\begin{center}
\includegraphics[width=1.0\linewidth]{fig/3dpw_crowd.pdf}
\end{center}
  \caption{
 3DCrowdNet outputs on 3DPW-Crowd.
  }
\label{fig:3dpw_crowd_result}
\end{figure}

\noindent\textbf{Results on 3DPW-Crowd.}
Figure~\ref{fig:3dpw_crowd_result} illustrates the 3DCrowdNet results on 3DPW-Crowd.
3DCrowdNet estimates robust 3D pose and shape on images that show people having highly close interaction.
Different people in overlapping bounding boxes are disentangled, and occluded body parts are reasonably reconstructed.

\section{Datasets.}

\noindent\textbf{3DPW-Crowd.}
The sequence names of 3DPW-Crowd are \textit{courtyard\_hug\_00} and \textit{courtyard\_dancing\_00}, a subset of the 3DPW~\cite{von20183dpw} validation set.
3DPW-Crowd contains 1073 images and 1923 persons with GT 3D pose and shape annotations.
The average bounding box IoU is 37.5\%, and the CrowdIndex~\cite{li2019crowdpose} is 49.3\%.
We used 14 joints defined by Human3.6M~\cite{ionescu2014human3} for evaluating PA-MPJPE and MPJPE following the previous works~\cite{kolotouros2019learning,choi2020p2m,moon2020i2l}.

Figure~\ref{fig:3dpw_crowd_samples} presents sampled frames of 3DPW-Crowd sequences in order.
The sequences reveal multiple challenges of in-the-wild crowd scenes, such as heavy inter-person occlusion and overlapping bounding boxes.

\noindent\textbf{MuPoTS.}
MuPoTS~\cite{mehta2018single} contains 20 sequences, 8370 images, and 20899 persons with GT 3D pose annotations.
The sequences are captured indoors and outdoors, and GT 3D poses are obtained by a multi-view marker-less motion capture system.
The average bounding box IoU is 3.8\%, and the CrowdIndex~\cite{li2019crowdpose} is 13.2\%.
We used the official MATLAB code for evaluation.

\noindent\textbf{CMU-Panoptic.}
We selected four sequences that show people doing social activities, namely \textit{Haggling}, \textit{Mafia}, \textit{Ultimatum}, and \textit{Pizza} following~\cite{zanfir2018deep,zanfir2018monocular,jiang2020coherent}.
Sequences captured by the 16th and 30th cameras are selected.
The sequences contain 9600 frames and 21,404 persons with GT 3D pose annotations.
The average bounding box IoU is 2.0\%, and the CrowdIndex~\cite{li2019crowdpose} is 11.1\%.
We used pre-processed GT annotations and followed the evaluation protocol of Jiang~\etal~\cite{jiang2020coherent} in their official code repository.

\twocolumn[{
\maketitle
\begin{center}\centering
\setlength{\tabcolsep}{0.1cm}
\setlength{\itemwidth}{5.5cm}
\hspace*{-\tabcolsep}
\begin{tabular}{c}

\includegraphics[width=\linewidth]{fig/hug.pdf}
\\
\textit{courtyard\_hug\_00}
\vspace{2mm}
\\
\includegraphics[width=\linewidth]{fig/dance.pdf}
\\
\textit{courtyard\_dancing\_00}

\end{tabular}

\captionof{figure}
{
Sampled frames of 3DPW-Crowd sequences in order.
}
\label{fig:3dpw_crowd_samples}
\end{center}
\vspace{0.5em}
}]

\noindent\textbf{3DPW.}
We used the test set of 3DPW~\cite{von20183dpw} following the official split protocol.
The test set contains 26240 images and 35515 persons with GT 3D pose and shape annotations.
The average bounding box IoU is 3.7\%, and the CrowdIndex~\cite{li2019crowdpose} is 4.9\%.
Sequences starring one actor are excluded in computing the bounding box IoU and the CrowdIndex.
We used 14 joints defined by Human3.6M~\cite{ionescu2014human3} for evaluating PA-MPJPE and MPJPE following the previous works~\cite{kolotouros2019learning,choi2020p2m,moon2020i2l}.

\section{2D pose estimators.}
In this work, we used 2D pose outputs from Cao~\etal~\cite{cao2017realtime} and Cheng~\etal~\cite{cheng2020higherhrnet}.
The outputs from Cao~\etal~\cite{cao2017realtime} used in 3DPW-Crowd and 3DPW~\cite{von20183dpw} are included in the annotations of 3DPW~\cite{von20183dpw}.
The outputs from Cao~\etal~\cite{cao2017realtime} used in MuPoTS~\cite{mehta2018single} are obtained by running the third-party PyTorch~\cite{paszke2017automatic} code implementation.
The model is trained on \textbf{COCO2017 train}~\cite{lin2014mscoco} dataset and achieves 0.653 mAP (mean Average Precision) on \textbf{COCO2017 val} dataset.
The outputs from Cheng~\etal~\cite{cheng2020higherhrnet} are all obtained by running the official code implementation.
The model is trained on \textbf{COCO2017 train} dataset and achieves 0.671 mAP on \textbf{COCO2017 val} dataset.
\section{Related works}

\noindent\textbf{Crowd scenes 2D human pose estimation.}
Early works of multi-person 2D human pose estimation did not explicitly target crowd scenes. However, their methods are related to diverse challenges from crowd scenes, such as overlapping human bounding boxes, human detection error, and inter-person occlusion.
Typically, there are two approaches, namely bottom-up and top-down approaches.
Bottom-up methods~\cite{pishchulin2016deepcut,insafutdinov2016deepercut,cao2017realtime,newell2016associative} first detect  all joints of the people, and group them to each person.
Top-down methods~\cite{he2017mask,papandreou2017towards,fang2017rmpe,chen2018cascaded} first detect all human bounding boxes, and apply a single-person 2D pose estimation method to each person.
Top-down methods generally achieve lower error on traditional 2D pose benchmarks such as MSCOCO~\cite{lin2014mscoco}, but underperform on crowd scene benchmarks~\cite{li2019crowdpose,zhang2019pose2seg,qiu2020peeking} than bottom-up methods due to overlapping human bounding boxes and human detection error.

Recently, many works explicitly address crowd scenes 2D human pose estimation or report accuracy on crowd scene benchmarks.
Li~\etal~\cite{li2019crowdpose} combined top-down and bottom-up approaches using joint-candidate single person pose estimation and global maximum joints association.
Cheng~\etal~\cite{cheng2020higherhrnet} proposed to learn scale-aware representations using high-resolution feature pyramids.
It is the current state-of-the-art method among bottom-up methods on MSCOCO~\cite{lin2014mscoco}, and achieves the best accuracy on CrowdPose~\cite{li2019crowdpose}, the popular 2D crowd scene benchmark.
Jin~\etal~\cite{jin2020differentiable} made a grouping process of the bottom-up approach differentiable using a graph neural network.
Qui~\etal~\cite{qiu2020peeking} suggested refining invisible joints' prediction using an image-guided progressive graph convolutional network.

\noindent\textbf{Multi-person 3D human pose estimation.}
Few multi-person 3D human pose estimation works have focused on crowd scenes 3D human pose estimation.
Rogez~\etal~\cite{rogez2017lcr,rogez2019lcr} introduced a top-down method called LCR-Net, which consists of localization, classification, and regression components.
Mehta~\etal~\cite{mehta2018single} introduced a bottom-up approach using an occlusion-robust pose-map (ORPM) formulation.
Moon~\etal~\cite{moon2019camera} and Wang~\etal~\cite{wang2020hmor} proposed fully learning-based multi-person 3D human pose estimation methods, which focus on recovering absolute depth of each person.
Zhen~\etal~\cite{zhen2020smap} exploited the method of Cao~\etal~\cite{cao2017realtime}, and incorporated absolute depth prediction for grouping joints to each person.
Recently, Chen~\etal~\cite{chen2020multi} explicitly targeted crowd scenes based on multi-view geometry.
They proposed a graph model for fast cross-view matching and MAP optimization for robust 3D crowd reconstruction.
XNect~\etal~\cite{mehta2020xnect} extended Mehta~\etal~\cite{mehta2018single} by proposing a novel backbone SelecSLS Net and a 3D pose encoding map that improved ORPM.

Although XNect~\cite{mehta2020xnect} is an occlusion robust method, it requires multiple pre-training steps on 2D and 3D datasets~\cite{andriluka2014mpii,lin2014mscoco,mehta2018single}, and a certain joint (\textit{i.e.} neck) must be visible for human detection.
On the contrary, our system can be end-to-end trained without complicated pre-training steps, and reconstructs full 3D human pose and shape from diverse partially invisible people in crowd scenes.

\noindent\textbf{3D human pose and shape estimation in crowd scenes.}
Most of the previous 3D human pose and shape estimation methods~\cite{kanazawa2018end,pavlakos2018learning,omran2018neural,varol2018bodynet,kolotouros2019learning,guler2019holopose,jiang2020coherent,kolotouros2019convolutional,moon2020i2l,zanfir2020weakly,georgakis2020hierarchical} take images as input and regress a 3D human shape.
To estimate accurate depth information, they necessarily rely on 3D human pose datasets~\cite{ionescu2014human3,mehta2017monocular}, which are captured from a studio environment far from in-the-wild crowd scenes.
Also, they are generally based on a top-down approach, which inevitably suffers from overlapping bounding boxes and human detection error.
As a result, their prediction often misses an occluded person or reveals swapped joints between people, due to overlapping bounding boxes and inter-person occlusion in in-the-wild crowd scenes.

To address such issues, a pioneering paper by Zanfir~\etal~\cite{zanfir2018deep} proposed a bottom-up system for multi-person 3D human pose and shape estimation from a single image.
It simultaneously estimates 2D and 3D poses, and resolves the joint grouping problem via a binary integer programming based on 2D pose prediction scores.
Then, SMPL parameters are regressed from the predicted 3D pose based on an auto-encoding scheme.
Different from the method of Zanfir~\etal~\cite{zanfir2018deep}, 3DCrowdNet can easily leverage off-the-shelf state-of-the-art 2D pose estimators.
In addition, our system infers a 3D shape from not only a 3D pose but also image feature, which are crucial for recovering accurate shape.

Guler~\etal~\cite{guler2019holopose} presented HoloPose that utilizes joint-level image feature based on its' 2D joint predictions.
In the case of a 2D joint prediction with low confidence, the method estimates the corresponding joint angle as an average of neighbor joint angles or fills it with the resting pose if neighbor joints also have low confidence.
However, crowd scenes frequently contain heavy occlusion that brings many 2D joint predictions with low confidence, and these heuristics are likely to malfunction.
On the contrary, our 3DShapeNet samples every joint-level feature from image feature and can benefit from a data-driven learning strategy.

\noindent\textbf{3D human pose and shape estimation from 2D geometry.}
Recently,~\cite{choi2020p2m,zhang2020learning,song2020human,sengupta2020synthetic} proposed systems that only take 2D geometry information, such as 2D joint locations, for SMPL parameter regression.
While the systems can benefit from in-the-wild robust 2D estimators, discarding image feature is abandoning rich 3D depth and shape cues in images.
In addition, relying solely on 2D geometry drives a network to produce the most plausible outputs for the given input, not the 3D pose and shape that best describes the person in images.
On the contrary, our 3DCrowdNet leverages appearance and shadow information related to shape and depth in images, and reconstructs 3D pose and shape that corresponds to the target person.

\section{Introduction}

Various methods have been proposed to recover 3D human pose and shape from a single image.
However, the 3D pose and shape estimation from in-the-wild crowd scenes has been barely studied, despite their common presence.

Most of the previous 3D pose and shape estimation methods~\cite{kanazawa2018end,pavlakos2018learning,kolotouros2019learning,georgakis2020hierarchical} first crop an image using a bounding box of a target person detected from off-the-shelf human detectors~\cite{he2017mask,redmon2018yolov3}.
Then they take the target person's cropped image and regress the human model parameters, such as SMPL~\cite{loper2015smpl}.
While the previous methods achieve high accuracy when provided with accurate human bounding boxes and little inter-person occlusion, they tend to output poor results on in-the-wild crowd scenes as depicted in Figure~\ref{fig:front_figure}.
In-the-wild crowd scenes involve overlapping and inaccurate human bounding boxes, which often include other people.
The noisy bounding boxes confuse the network on which to focus.
Inter-person occlusion in the crowd scenes hinders accurate 3D pose prediction and causes swapped joint prediction between people.

\begin{figure*}
\begin{center}
\includegraphics[width=1.0\linewidth]{fig/architecture.pdf}
\end{center}
\vspace*{-5mm}
   \caption{
   The overall architecture of 3DCrowdNet, which consists of 3DPoseNet and 3DShapeNet. 3DPoseNet produces 2D pose-guided feature that has high activation on a target person. Then it estimates 3D joint coordinates from it. 3DShapeNet samples joint-level feature from the 2D pose-guided feature based on ($x$,$y$) positions of the predicted 3D joint coordinates. Then it outputs SMPL parameters, which are decoded to a 3D shape.
   }
\vspace*{-3mm}
\label{fig:architecture}
\end{figure*}

On the other hand, 2D human pose estimation methods tend to suffer less from the inaccurate human bounding boxes and inter-person occlusion.
We argue that the difference is attributed to the composition of train data.
3D human pose and shape estimation methods typically leverage 3D human pose train datasets~\cite{ionescu2014human3,mehta2017monocular} to learn depth information from images.
However, to attain accurate ground truth (GT) 3D annotations via multi-view geometry algorithm, many 3D datasets are captured from a multi-view studio starring a single person without a crowd.
The perfect human bounding boxes and no inter-person occlusion are guaranteed for accurate 3D annotations, but the images are far from in-the-wild crowd scenes.
On the contrary, 2D human pose estimation methods are trained on diverse in-the-wild data~\cite{lin2014mscoco,andriluka2014mpii,wu2017ai}.
The 2D datasets contain images with heavily overlapping bounding boxes and inter-person occlusion, where human manually annotates the data.
While the 3D methods can use the in-the-wild 2D data in a weakly-supervised manner, their strong reliance on 3D data leads to low pose accuracy on crowd scenes.

In this regard, we propose 3DCrowdNet, a 2D human pose-guided system for 3D human pose and shape estimation from in-the-wild crowd scenes.
This study is the first to explicitly tackle the 3D crowd human pose and shape estimation from a single image to the best of our knowledge.
3DCrowdNet consists of 3DPoseNet and 3DShapeNet, and the inference process proceeds in three steps.
First, we leverage off-the-shelf bottom-up 2D pose estimators, which first detect all human joints and group them to each person, to obtain crowd-scene robust 2D pose outputs and bounding boxes of each person.
The bounding boxes are computed by enlarging the 2D pose boundaries and used to acquire each person's image feature.
The bottom-up 2D pose estimators allow us to obtain the 2D poses and bounding boxes simultaneously.
Second, 3DPoseNet concatenates the 2D pose output and the image feature from a cropped person to guide a network to focus on the person who is the target.
The 2D pose output activates the target person's image feature area and encourages the network to disentangle the target person from other people included in a bounding box.
In addition, it provides essential human articulation information to the 3DCrowdNet.
Last, 3DPoseNet refines the possibly noisy 2D pose output, and 3DShapeNet estimates SMPL parameters.
3DShapeNet is a joint-based regressor, which extracts joint-level feature from the 2D pose-guided image feature based on the output from 3DPoseNet.
Figure~\ref{fig:architecture} describes the overall architecture.

The experimental results show that the proposed 3DCrowdNet significantly outperforms the previous 3D human pose and shape estimation methods on in-the-wild crowd scenes.
Also, it achieves state-of-the-art accuracy on multi-person 3D benchmarks~\cite{mehta2018single,joo2017panoptic} and in-the-wild benchmark~\cite{von20183dpw}.
Extensive ablation experiments are carried out to validate the system, and would give insight for future works of 3D human pose and shape estimation from in-the-wild crowd scenes.

Our contributions can be summarized as follows.
\begin{itemize}
\item We present 3DCrowdNet, which firstly targets in-the-wild crowd scenes for 3D human pose and shape estimation from a single image.
The proposed system leverages the crowd-scene robust 2D pose outputs to guide a network to focus on a target person.
\item We provide extensive ablation experiments to justify the design choices of our system.
3DCrowdNet outperforms previous methods on both in-the-wild crowd scenes and conventional 3D benchmarks.

\end{itemize}

\section{Implementation detail}

PyTorch~\cite{paszke2017automatic} is used for implementation.
We initialize the weights of ResNet~\cite{he2016deep} with the pre-trained weights from Xiao~\etal~\cite{xiao2018simple}.
It proved to be effective on faster convergence during training.
The weights of the whole model are updated by the Adam optimizer~\cite{kingma2014adam} with a mini-batch size of 64.
The initial learning rate is $10^{-4}$.
The model is trained for six epochs, and the learning rate is reduced by a factor of 10 after the $3$th and $5$th epochs.
The cropped image is resized to 256$\times$256.
We use four NVIDIA RTX 2080 Ti GPUs for training, and it takes about 9 hours on average.
We will release the codes for more implementation details.
\section{Experiment}

\subsection{Datasets}

\noindent\textbf{Training sets.}
We use Human3.6M~\cite{ionescu2014human3}, MuCo-3DHP~\cite{mehta2018single}, MSCOCO~\cite{lin2014mscoco}, MPII~\cite{andriluka2014mpii}, and CrowdPose~\cite{li2019crowdpose} for training.
Only the training sets of the datasets are used, following the standard split protocols.
To generate pseudo-GT of SMPL parameters, we use NeuralAnnot~\cite{moon2020neuralannot}.

\noindent\textbf{Testing sets.}
We report accuracy performance on MuPoTS~\cite{mehta2018single}, CMU-Panoptic~\cite{joo2017panoptic}, and 3DPW~\cite{von20183dpw}.
MuPoTS is a multi-person test benchmark captured from indoor and outdoor environments, starring 3 to 4 people.
CMU-Panoptic is a large-scale multi-person dataset captured from the Panoptic studio.
Following~\cite{zanfir2018deep,jiang2020coherent}, we pick four sequences presenting 3 to 7 people socializing each other for the evaluation.
3DPW is a widely-used 3D benchmark captured from an in-the-wild environment, and we use the test set of 3DPW following the official split protocols.
More details are in the supplementary material.

\begin{figure}
\begin{center}
\includegraphics[width=0.9\linewidth]{fig/data_stats.pdf}
\end{center}
\vspace*{-5mm}
   \caption{
   3DPW-Crowd, a subset of 3DPW~\cite{von20183dpw} validation set, has much higher bounding box IoU and CrowdIndex~\cite{li2019crowdpose} than other datasets. CrowdIndex measures other people's joints' ratio over each person's joints in a bounding box. The statistics support that 3DPW-Crowd is a suitable benchmark for the evaluation on in-the-wild crowd scenes.
   }
\vspace*{-3mm}
\label{fig:data_statistics}
\end{figure}

\subsection{Evaluation protocols}

\noindent\textbf{Evaluation on crowd scenes.}
As Zhang~\etal~\cite{li2019crowdpose} addressed, the principal obstacle of crowd scenes pose estimation is not the number of people, but the inter-person occlusion due to diverse interaction between people.
Thus, MuPoTS~\cite{mehta2018single} and CMU-Panoptic~\cite{joo2017panoptic} have limitations for the evaluation on in-the-wild crowd scenes, not only because they are not in-the-wild data, but also because they show limited interaction between people.

In this regard, we provide a numerical evaluation on a subset of 3DPW~\cite{von20183dpw} validation set that shows substantial interaction, to investigate the method's robustness on in-the-wild crowd scenes thoroughly.
The subset contains hugging and dancing sequences that have considerably higher average intersection over union (IoU) of bounding boxes and CrowdIndex~\cite{li2019crowdpose} than other datasets as shown in Figure~\ref{fig:data_statistics}.
We name the subset as 3DPW-Crowd, since it reveals the challenges of in-the-wild crowd scenes, such as overlapping bounding boxes and severe inter-person occlusion.
More details about 3DPW-Crowd are in the supplementary material.
We also show extensive qualitative results on the test set of CrowdPose~\cite{li2019crowdpose}.

\noindent\textbf{Evaluation metrics.}
We report 3D pose and 3D shape evaluation metrics.
For the 3D pose evaluation, we use mean per joint position error (MPJPE), Procrustes-aligned mean per joint position error (PA-MPJPE), and 3DPCK proposed in Mehta~\etal~\cite{mehta2017monocular}.
Following~\cite{kanazawa2018end,kolotouros2019learning,moon2020i2l,choi2020p2m}, we use the 3D joint coordinates regressed from the final 3D shape as predictions.
For the 3D shape evaluation, we use mean per vertex position error (MPVPE).

\subsection{Ablation study}

We carry out the ablation study on 3DPW-Crowd to validate our method's robustness on crowd scenes.

\begin{table}
\small
\centering
\setlength\tabcolsep{1.0pt}
\def1.1{1.1}
\caption{Comparison between networks taking different input feature on 3DPW-Crowd. For our 2D pose-guided system, we provide results using 2D pose outputs from~\cite{cao2017realtime,cheng2020higherhrnet}. The results show that the 2D pose-guided feature from the bottom-up 2D pose estimators~\cite{cao2017realtime,cheng2020higherhrnet} is crucial for robust estimation on crowd scenes.}
\begin{tabular}{C{5.1cm}|C{1.2cm}C{1.8cm}}
\specialrule{.1em}{.05em}{.05em}
input feature & MPJPE & PA-MPJPE \\ \hline
image feature wo. guide & 109.6 & 63.3 \\
\cellcolor{Gray}2D pose~\cite{cao2017realtime}-guided feature & \cellcolor{Gray}87.6 & \cellcolor{Gray}58.5 \\
\textbf{2D pose~\cite{cheng2020higherhrnet}-guided feature} & \textbf{85.6} & \textbf{55.4} \\
\specialrule{.1em}{.05em}{.05em}
\end{tabular}
\label{table:input_ablation_3dpw_crowd}
\end{table}

\begin{figure}
\begin{center}
\includegraphics[width=1.0\linewidth]{fig/activation.pdf}
\end{center}
\vspace*{-5mm}
   \caption{
   Qualitative comparison between 3DCrowdNet and the baseline that takes only image feature as input. A 2D pose activates the target person's area in the image feature in our system and encourages the network to disentangle the target person from other people in a bounding box.
   }
\vspace*{-3mm}
\label{fig:activation_ablation}
\end{figure}

\noindent\textbf{Effect of 2D human pose-guided feature.}
Table~\ref{table:input_ablation_3dpw_crowd} shows that the 2D human pose-guided image feature's effectiveness is definite on crowd scenes.
The first-row network, the baseline, crops an image using a GT bounding box and extracts image feature without a 2D pose guide.
MPJPE and PA-MPJPE drop by 24.0$mm$ and 7.9$mm$ respectively, when 3DCrowdNet guides the image feature with 2D pose outputs from Cheng~\etal~\cite{cheng2020higherhrnet}, the state-of-the-art bottom-up 2D pose estimator.
The result proves that the crowd-scene robust 2D pose outputs guide a network to focus on a target person.
Figure~\ref{fig:activation_ablation} illustrates how the image feature is affected by the 2D pose in our system.
Unlike the baseline network, our 3DCrowdNet activates the target person's region and successfully disentangles different people who have heavily overlapping bounding boxes.

\begin{table}
\small
\centering
\setlength\tabcolsep{1.0pt}
\def1.1{1.1}
\caption{Comparison between the joint-based and the HMR~\cite{kanazawa2018end}-style regressors on 3DPW-Crowd. We used 2D pose outputs from Cheng~\etal~\cite{cheng2020higherhrnet}. The joint-based regressor outperforms the HMR-style regressor on crowd scenes.
}
\begin{tabular}{C{4.0cm}|C{1.3cm}C{1.8cm}}
\specialrule{.1em}{.05em}{.05em}
parameter regressor type & MPJPE &  PA-MPJPE \\ \hline

HMR~\cite{kanazawa2018end}-style regressor & 89.0 &  59.5 \\
\cellcolor{Gray}\textbf{joint-based regressor (Ours)}  &  \cellcolor{Gray}\textbf{85.6} &  \cellcolor{Gray}\textbf{55.4} \\
\specialrule{.1em}{.05em}{.05em}
\end{tabular}
\label{table:regressor_ablation}
\end{table}

\noindent\textbf{Joint-based regressor vs. HMR-style regressor.}
Table~\ref{table:regressor_ablation} shows that the joint-based regressor outperforms the HMR~\cite{kanazawa2018end}-style regressor on 3DPW-Crowd.
The results prove that the joint-based regressor better preserves the 2D pose cues for the target person in crowd scenes.
Note that the joint-based regressor utilizes joint-level feature sampled from ($x$,$y$) positions of the predicted 3D pose. In contrast, the HMR-style regressor regresses SMPL parameters from the global-average-pooled feature.

\begin{table}
\small
\centering
\setlength\tabcolsep{1.0pt}
\def1.1{1.1}
\caption{Ablation on 3DPW-Crowd about 3DPoseNet that recovers a 3D pose from the 2D human pose-guided image feature.
We used 2D pose outputs from Cheng~\etal~\cite{cheng2020higherhrnet}.
The results validate that 3DPoseNet increases the robustness of our system on crowd scenes.
}
\begin{tabular}{C{4.2cm}|C{1.3cm}C{1.8cm}}
\specialrule{.1em}{.05em}{.05em}
network & MPJPE &  PA-MPJPE \\ \hline
without 3DPoseNet & 96.7 & 60.1 \\
\cellcolor{Gray}2DPoseNet & \cellcolor{Gray}88.3 &  \cellcolor{Gray}56.4 \\
\textbf{3DPoseNet (Ours)} & \textbf{85.6} & \textbf{55.4} \\
\specialrule{.1em}{.05em}{.05em}
\end{tabular}
\label{table:3dposenet_ablation}
\end{table}

\noindent\textbf{Validity of 3DPoseNet.}
Table~\ref{table:3dposenet_ablation} validates 3DPoseNet by comparing it with its variations.
The first-row network samples the joint-level feature based on 2D joint predictions from off-the-shelf 2D pose estimators.
The second-row network, 2DPoseNet, estimates a 2D pose instead of a 3D pose.
Both networks concatenate the joint-level feature and 2D joint coordinates, and feed it to 3DShapeNet.

The accuracy significantly drops without 3DPoseNet on 3DPW-Crowd.
The result proves that the refining process of the 2D pose input is essential for the joint-based regressor of 3DCrowdNet.
We can also verify the effectiveness of estimating a 3D pose in Table~\ref{table:3dposenet_ablation}.
The clear accuracy improvement proves that the depth information can be reliably estimated from a 2D pose and image feature, and it is beneficial for the accuracy of the final output.

\subsection{Comparison with state-of-the-art methods}
Unless indicated, our 3DCrowdNet is not trained on CrowdPose~\cite{li2019crowdpose} train set in this section.
\begin{table}
\small
\centering
\setlength\tabcolsep{1.0pt}
\def1.1{1.1}
\caption{Comparison between 3DCrowdNet and previous methods on 3DPW-Crowd. $*$ means that we used CrowdPose~\cite{li2019crowdpose} train data.}
\begin{tabular}{C{3.0cm}|C{1.3cm}C{2.0cm}C{1.3cm}}
\specialrule{.1em}{.05em}{.05em}
method & MPJPE & PA-MPJPE & MPVPE \\ \hline
SPIN~\cite{kolotouros2019learning} & 121.2 & 69.9 & 144.1\\
\cellcolor{Gray}Pose2Mesh~\cite{choi2020p2m} & \cellcolor{Gray}124.8 & \cellcolor{Gray}79.8 & \cellcolor{Gray}149.5 \\
I2L-MeshNet~\cite{moon2020i2l} & 115.7 & 73.5 & 162.0 \\
\cellcolor{Gray}3DCrowdNet (Ours) & \cellcolor{Gray}86.9 & \cellcolor{Gray}56.8 & \cellcolor{Gray}110.3 \\
\textbf{3DCrowdNet (Ours)$*$} & \textbf{85.6} & \textbf{55.4} & \textbf{109.0} \\

\specialrule{.1em}{.05em}{.05em}
\end{tabular}
\label{table:sota_3dpw_crowd}
\end{table}

\noindent\textbf{3DPW-Crowd.}
We compare our 3DCrowdNet with~\cite{kolotouros2019learning,choi2020p2m,moon2020i2l} on 3DPW-Crowd in Table~\ref{table:sota_3dpw_crowd}.
They are recent state-of-the-art 3D human pose and shape estimation methods on 3DPW~\cite{von20183dpw}, and publicly released the codes for an evaluation.
Our approach outperforms SPIN~\cite{zhang2020learning}, which takes only the image feature as input and uses a HMR~\cite{kanazawa2018end}-style regressor.
The result is coherent with the results in Table~\ref{table:input_ablation_3dpw_crowd} and~\ref{table:regressor_ablation} of our ablation studies.
3DCrowdNet also defeats Pose2Mesh, a method that can benefit from crowd-scene robust 2D pose outputs.
We used the same 2D pose outputs from Cheng~\etal~\cite{cheng2020higherhrnet} for Pose2Mesh and 3DCrowdNet.
While Pose2Mesh heavily depends on the accuracy of the 2D pose outputs, our system improves the accuracy of pose and shape using rich depth and shape cues in the image feature.
Figure~\ref{fig:3d_quality} supports our statement.
Leveraging the image feature, 3DCrowdNet reconstructs a 3D shape that best describes the target person in images, even when the 2D pose is inaccurate.

\begin{table}
\small
\centering
\setlength\tabcolsep{1.0pt}
\def1.1{1.1}
\caption{Comparison between 3DCrowdNet and previous methods on MuPoTS~\cite{mehta2018single}. The numbers are 3DPCK for all annotations (All) and annotations matched to a prediction (Matched). The tiny scripts indicate the source of bounding boxes or input 2D poses of our system.}
\begin{tabular}{C{4.3cm}|C{1.3cm}C{1.5cm}}
\specialrule{.1em}{.05em}{.05em}
method & All & Matched \\ \hline
SMPLify-X~\cite{pavlakos2019expressive} \tiny(Cao~\etal~\cite{cao2017realtime}) & 62.8 & 68.0 \\
\cellcolor{Gray}HMR~\cite{kanazawa2018end} \tiny(Cao~\etal~\cite{cao2017realtime}) & \cellcolor{Gray}66.0 & \cellcolor{Gray}70.9 \\
HMR~\cite{kanazawa2018end} \tiny(He~\etal~\cite{he2017mask}) & 65.6 & 68.6 \\
\cellcolor{Gray}Jiang~\etal~\cite{jiang2020coherent} & \cellcolor{Gray}69.1 & \cellcolor{Gray}72.2 \\
3DCrowdNet (Ours) \tiny(Cao~\etal~\cite{cao2017realtime}) & 70.2 & 70.9 \\
\cellcolor{Gray}\textbf{3DCrowdNet (Ours)} \tiny(Cheng~\etal~\cite{cheng2020higherhrnet}) & \cellcolor{Gray}\textbf{72.7} & \cellcolor{Gray}\textbf{73.3} \\

\specialrule{.1em}{.05em}{.05em}
\end{tabular}
\label{table:sota_mupots}
\end{table}

\noindent\textbf{MuPoTS.}
Table~\ref{table:sota_mupots} compares our 3DCrowdNet with~\cite{jiang2020coherent,kanazawa2018end,pavlakos2019expressive}.
As the second and fifth rows show, 3DCrowdNet outperforms HMR~\cite{cao2017realtime} when both methods are based on the 2D poses from Cao~\etal~\cite{cao2017realtime}.
While HMR~\cite{cao2017realtime} utilizes the 2D pose only to get a bounding box, 3DCrowdNet additionally uses the 2D pose to guide a network to focus on a target person from crowd scenes.
Leveraging more information in given outputs is natural, and leads to better accuracy.
Moreover, our 3DCrowdNet benefits from improved 2D pose outputs from Cheng~\etal~\cite{cheng2020higherhrnet}.
In summary,  Table~\ref{table:sota_mupots} proves that 3DCrowdNet provides accurate 3D human pose and shape in images with multi-person, and has the potential to improve with better 2D pose outputs.
The numbers of other methods in Table~\ref{table:sota_mupots} are from Jiang~\etal~\cite{jiang2020coherent}.

\begin{table}
\small
\centering
\setlength\tabcolsep{1.0pt}
\def1.1{1.1}
\caption{Comparison between 3DCrowdNet and previous methods on CMU-Panoptic~\cite{joo2017panoptic}. We follow the publicly released evaluation protocol of Jiang~\etal~\cite{jiang2020coherent}. }
\begin{tabular}{C{2.9cm}|C{0.9cm}C{0.9cm}C{0.9cm}C{0.9cm}|C{0.9cm}}
\specialrule{.1em}{.05em}{.05em}
method & Haggl. & Mafia & Ultim. & Pizza & Mean \\ \hline
Zanfir~\etal~\cite{zanfir2018monocular} & 140.0 & 165.9 & 150.7 & 156.0 & 153.4 \\
\cellcolor{Gray}Zanfir~\etal~\cite{zanfir2018deep} & \cellcolor{Gray}141.4 & \cellcolor{Gray}152.3 & \cellcolor{Gray}145.0 & \cellcolor{Gray}162.5 & \cellcolor{Gray}150.3  \\
Jiang~\etal~\cite{jiang2020coherent} & 129.6 & \textbf{133.5} & 153.0 & 156.7 & 143.2 \\
\cellcolor{Gray}\textbf{3DCrowdNet (Ours)} & \cellcolor{Gray}\textbf{109.60} & \cellcolor{Gray}135.9 & \cellcolor{Gray}\textbf{129.8} & \cellcolor{Gray}\textbf{135.6} & \cellcolor{Gray}\textbf{127.6}\\
\specialrule{.1em}{.05em}{.05em}
\end{tabular}
\label{table:sota_cmu}
\end{table}

\noindent\textbf{CMU-Panoptic.}
Table~\ref{table:sota_cmu} shows that our 3DCrowdNet significantly outperforms previous 3D human pose and shape estimation methods on CMU-Panoptic.
The result demonstrates that the proposed 3DCrowdNet can perform competitively on crowd scenes with daily social activities.
Note that no data from CMU-Panoptic are used for training.

\begin{table}
\small
\centering
\setlength\tabcolsep{1.0pt}
\def1.1{1.1}
\caption{Comparison on 3DPW~\cite{von20183dpw} between 3DCrowdNet and previous methods of 3D human pose and shape estimation from a single image. 3DCrowdNet achieves the best accuracy in all metrics. All methods except Zanfir~\etal~\cite{zanfir2020weakly} did not use 3DPW train data.}
\begin{tabular}{C{3.0cm}|C{1.3cm}C{2.0cm}C{1.3cm}}
\specialrule{.1em}{.05em}{.05em}
method & MPJPE & PA-MPJPE & MPVPE \\ \hline
HMR~\cite{kanazawa2018end} & 130 & 76.7 & -\\
\cellcolor{Gray}GraphCMR~\cite{kolotouros2019convolutional} & \cellcolor{Gray}- & \cellcolor{Gray}70.2 & \cellcolor{Gray}- \\
SPIN~\cite{kolotouros2019learning} & 96.9 & 59.2 & 116.4\\
\cellcolor{Gray}STRAPS~\cite{sengupta2020synthetic} & \cellcolor{Gray}- & \cellcolor{Gray}66.8 & \cellcolor{Gray}- \\
Zanfir~\etal~\cite{zanfir2020weakly} & 90.0 & 57.1 & - \\
\cellcolor{Gray}Pose2Mesh~\cite{choi2020p2m} & \cellcolor{Gray}88.9 & \cellcolor{Gray}58.3 & \cellcolor{Gray}106.3 \\
I2L-MeshNet & 93.2 & 57.7 & 110.1 \\
\cellcolor{Gray}Song~\etal~\cite{song2020human} & \cellcolor{Gray}- & \cellcolor{Gray}55.9 & \cellcolor{Gray}- \\
\textbf{3DCrowdNet (Ours)} & \textbf{82.8} & \textbf{52.2} & \textbf{100.2} \\

\specialrule{.1em}{.05em}{.05em}
\end{tabular}
\label{table:sota_3dpw}
\vspace*{-3mm}
\end{table}

\begin{figure*}
\begin{center}
\includegraphics[width=1.0\linewidth]{fig/quality_comparison.pdf}
\end{center}
\vspace*{-4mm}
  \caption{
  Qualitative comparison on the CrowdPose~\cite{li2019crowdpose} test set. From left, an input image, 3DCrowdNet outputs, SPIN~\cite{kolotouros2019learning} outputs. Both methods used the same bounding boxes computed from 2D pose outputs of Cheng~\etal~\cite{cheng2020higherhrnet}.
  }
\vspace*{-4mm}
\label{fig:quality_comparison}
\end{figure*}

\begin{figure*}
\begin{center}
\includegraphics[width=1.0\linewidth]{fig/3d_vis.pdf}
\end{center}
\vspace*{-4mm}
   \caption{
   Visualization of 3D shapes from different viewpoints. The proposed 3DCrowdNet can recover 3D pose and shape that best describes the target person in an image, even when provided with incorrect 2D pose outputs, using both the 2D pose output and image feature.
   }
\vspace*{-4mm}
\label{fig:3d_quality}
\end{figure*}

\noindent\textbf{3DPW.}
Table~\ref{table:sota_3dpw} shows that 3DCrowdNet achieves the state-of-the-art accuracy on general in-the-wild scenes.
The result validates that 3DCrowdNet is robust to diverse challenges in in-the-wild scenes, although our method is designed to target crowd scenes.
The left samples in the second and third rows of Figure~\ref{fig:quality_comparison} depicts the robustness of 3DCrowdNet to truncation and object occlusion in images, which are common in general in-the-wild scenes.
The proposed system reconstructs plausible full 3D pose and shape from image feature and partial 2D poses of truncated and occluded bodies.

We provide the qualitative comparison between 3DCrowdNet and SPIN~\cite{kolotouros2019learning} in Figure~\ref{fig:quality_comparison}.
SPIN~\cite{kolotouros2019learning} is the most relevant competitor among current state-of-the-art methods, since it leverages image feature and estimates SMPL parameters as 3DCrowdNet.
Apparently, 3DCrowdNet produces much more robust 3D shapes on crowd scenes than SPIN.
3DCrowdNet disentangles different people from a target person in a bounding box, and estimates reasonable 3D pose and shape under diverse occlusion, including inter-person occlusion.
We provide more qualitative comparison with other methods~\cite{choi2020p2m,moon2020i2l} and failure cases in the supplementary material.

\section{3DCrowdNet}
\subsection{Preparing 2D human pose input}
\label{section:prepare_2dpose}
Our proposed system uses 2D joint coordinates $\mathbf{P}^{\text{2D}} \in \mathbb{R}^{J \times 2}$ predicted by bottom-up off-the-shelf 2D human pose estimators~\cite{cao2017realtime,cheng2020higherhrnet}.
$J$ denotes the number of human joints, and it varies among different 2D human pose estimators.
The capability to exploit off-the-shelf 2D human pose estimators is advantageous, since our system can benefit from the improving 2D methods without re-training our system.
During training, we mimic the 2D human pose outputs $\mathbf{P}^{\text{2D}}$ by adding realistic errors on the GT (Ground Truth) 2D pose following~\cite{moon2019posefix,chang2020abs,choi2020p2m}.

We generate a 2D pose heatmap $\mathbf{H}^{\text{2D}} \in \mathbb{R}^{J_s \times H \times W}$ from $\mathbf{P}^{\text{2D}}$ by making a Gaussian blob on the 2D coordinates.
$H$ and $W$ denote the height and width of the image feature to be concatenated.
$J_s$ indicates the number of joints in a superset of joints sets defined by multiple datasets.
We assign don't-care values to the undefined joints and joint predictions with low confidence in inference time, multiplying zero to the corresponding joint heatmap.

\subsection{3DPoseNet}
3DPoseNet estimates 3D joint coordinates from the 2D joint heatmaps and an image.

\noindent\textbf{2D human pose-guided image feature.}
3DPoseNet uses a modified version of ResNet~\cite{he2016deep} to extract 2D human pose-guided image feature.
First, it obtains early-stage image feature $\mathbf{F} \in \mathbb{R}^{C \times H \times W}$ from a cropped image $\mathbf{I} \in \mathbb{R}^{3 \times 4H \times 4W}$ using the first convolution and max-pooling layers of ResNet.
$C$ is the feature channel dimension, and $H$ and $W$ are a quarter of the height and width of $\mathbf{I}$.
The cropped image $\mathbf{I}$ is acquired using a bounding box derived from the 2D pose $\mathbf{P}^{\text{2D}}$.
Then, it concatenates $\mathbf{F}$ and the 2D pose heatmap $\mathbf{H}^{\text{2D}}$ along the channel dimension.
The concatenated feature is processed by a 3-by-3 convolution block, which keeps the feature's height and width but changes the channel dimension to $C$.
Finally, the feature with $C$ channels is fed back to the remaining part of ResNet, where the output is
2D human pose-guided image feature $\mathbf{F^\prime} \in \mathbb{R}^{C^\prime \times H/4 \times W/4}$.
Different from the original ResNet output feature, $\mathbf{F^\prime}$ has more activation on the corresponding 2D joint locations of $\mathbf{P}^{\text{2D}}$.
The 2D human pose-guided image feature $\mathbf{F^\prime}$ is essential for crowd scenes 3D pose and shape estimation, since it disentangles the target person's image feature from others.

\noindent\textbf{3D joint coordinates.} 3DPoseNet recovers 3D joint coordinates $\mathbf{P}^{\text{3D}} \in \mathbb{R}^{J_c \times 3}$ from the 2D human pose-guided feature $\mathbf{F^\prime}$.
$J_c$ denotes the number of joints that are most common among multiple datasets.
($x$,$y$) values of $\mathbf{P}^{\text{3D}}$ are defined in a 2D image coordinate space, and $z$ value of $\mathbf{P}^{\text{3D}}$ represents root joint-relative depth.
3DPoseNet processes $\mathbf{F^\prime}$ by a 1-by-1 convolution block, and outputs a 3D heatmap $\mathbf{H}^{\text{3D}} \in \mathbb{R}^{J_c \times H/4 \times W/4}$ after reshaping the tensor.
$\mathbf{P}^{\text{3D}}$ is computed from $\mathbf{H}^{\text{3D}}$ using the soft-argmax operation~\cite{sun2018integral}.

\subsection{3DShapeNet}
\label{section:3dshapenet}
3DShapeNet estimates global rotation of a person $\theta^g \in \mathbb{R}^{3}$, SMPL parameters $\theta \in \mathbb{R}^{21 \times 3}$, $\beta \in \mathbb{R}^{10}$, and camera parameters $k \in \mathbb{R}^{3}$ for projection.
$\theta$ and $\beta$ are pose and shape parameters, respectively.
$\beta$ and $k$ are regressed from $\mathbf{F^\prime}$ after spatially averaging it, followed by a separate fully connected layer.

\noindent\textbf{Joint-based regressor.}
$\theta^g$ and $\theta$ are estimated from $\mathbf{F^\prime}$, $\mathbf{P}^{\text{3D}}$, and 3D joint prediction confidence of $\mathbf{P}^{\text{3D}}$ from $\mathbf{H}^{\text{3D}}$.
We design 3DShapeNet as a joint-based model that preserves the 2D pose cues for the target person in crowd scenes.
The joint-based model~\cite{guler2019holopose,zhang2020learning} utilizes joint-level image feature to estimate joint angles such as SMPL pose parameters.
First, 3DShapeNet samples image feature per joint from $\mathbf{F^\prime}$ using the ($x$,$y$) position of $\mathbf{P}^{\text{3D}}$.
Second, it concatenates the sampled joint-level image feature, $\mathbf{P}^{\text{3D}}$, and the 3D joint prediction confidence, to attain $\mathbf{F}^{S} \in \mathbb{R}^{J_c \times (C^\prime+3+1)}$.
Last, 3DShapeNet processes $\mathbf{F}^{S}$ using a graph convolution network, and predicts $\theta^g$ and $\theta$ by a separate fully connected layer.

\noindent\textbf{Graph convolution.}
For the graph convolution network, we use joint-specific graph convolution that learns separate weights for each graph vertex.
We define learnable weight matrices $\{W_j \in \mathbb{R}^{C_{\text {out}} \times C_{\text {in}}}\}_{j=1}^{J_c}$ for all joints of each graph convolution layer, where $C_{\text {in}}$ and $C_{\text {out}}$ denotes input and output channel dimensions, respectively.
Then, the output graph feature of joint $j$ is derived as $\mathbf{F}_j^{\text {out}} = \sigma_\text{ReLU}(\sum_{i \in \hat{\mathcal{N}}_j} \tilde{a}_{ji} \sigma_\text{BN}( W_i \mathbf{F}_i^{\text {in}}))$, where $\mathbf{F}_i^{\text {in}}$ is the input graph feature of joint $i$.
$\sigma_\text{ReLU}$ and $\sigma_\text{BN}$ denotes ReLU activation function and 1D batch normalization~\cite{ioffe2015batch}, respectively.
$\hat{\mathcal{N}}_j$ is defined as $\mathcal{N}_j \cup \{j\}$, where $\mathcal{N}_j$ denotes neighbors of a vertex $j$.
$\tilde{a}_{ji}$ is an entry of the normalized adjacency matrix $\tilde {\mathbf{A}}$ at $(j,i)$, where $\tilde {\mathbf{A}}=\mathbf{D}^{-\frac{1}{2}}(\mathbf{A}+\mathbf{I}) \mathbf{D}^{-\frac{1}{2}}$.
$\mathbf{A} \in \{0,1\}^{J_c \times J_c}$ is the adjacency matrix constructed based on the human skeleton hierarchy and fixed during the training and testing stages.
$\mathbf{D}$ is a diagonal matrix of $\mathbf{A}+\mathbf{I}$.
3DShapeNet adopts the network architecture of Liu~\etal~\cite{liu2020comprehensive}, which consists of one graph convolution block and four graph residual blocks.
We empirically observed that the joint-specific graph convolution provides faster convergence on test accuracy than fully connected layers.

\subsection{Network Training}
3DPoseNet and 3DShapeNet are integrated and trained in an end-to-end manner.
We use pseudo-GT SMPL fits obtained from a fitting framework, in addition to GT annotations from train datasets following~\cite{kolotouros2019learning,moon2020i2l,choi2020p2m,zanfir2020weakly}.
The loss function for 3DPoseNet is defined as follows:

\begin{equation}
L_{\text{pose}} = L_{\text{coord}}^{\text{3DPoseNet}},
\end{equation}
where $L_{\text{coord}}^{\text{3DPoseNet}}$ denotes the L1 distance between the predicted $\mathbf{P}^{\text{3D}}$ and the GT.
Owing to the soft-argmax operation~\cite{sun2018integral}, $L_{\text{coord}}^{\text{3DPoseNet}}$ allows the 2D and 3D mixed data supervision.

To supervise 3DShapeNet, following~\cite{kolotouros2019learning,kanazawa2018end,moon2020i2l,choi2020p2m} we regress 3D pose $\mathbf{\hat{P}}^{\text{3D}} \in \mathbb{R}^{J_s \times 3}$ from the 3D shape decoded from $\theta^g$, $\theta$, and $\beta$.
Camera parameters $k$ is used to project $\mathbf{\hat{P}}^{\text{3D}}$ to image space, to produce $\mathbf{\hat{P}}^{\text{2D}} \in \mathbb{R}^{J_s \times 2}$.
The loss function for 3DShapeNet is defined as follows:

\begin{equation}
L_{\text{shape}} = L_{\text{param}} + L_{\text{coord}}^{\text{3DShapeNet}},
\end{equation}
where $L_{\text{param}}$ denotes the L1 distance between the predicted $\theta^g$, $\theta$, and $\beta$, and the pseudo-GT parameters;
$L_{\text{coord}}^{\text{3DShapeNet}}$ includes the L1 distance loss of $\mathbf{\hat{P}}^{\text{3D}}$ and $\mathbf{\hat{P}}^{\text{2D}}$.