\section{\uppercase{Introduction}}

Most comets are observed to show activity when they reach heliocentric distances $r_\mathrm{H} \lesssim 5$--6 AU, where the most abundant cometary volatile, water ice, begins to sublimate as a result of increasing insolation. However, a few comets have been observed to exhibit activity at greater heliocentric distances. This distant activity cannot be explained by sublimation of water ice, but other mechanisms including crystallization of amorphous ice (Prialnik \& Bar-Nun 1992) and sublimation of supervolatile species (A'Hearn et al.~2012) may be responsible. The reason why only a few distant comets have been observed to be active is twofold. Firstly, when comets are far away from the Sun, they are intrinsically less active because of their lower temperatures. Secondly, distant comets tend to be extremely faint, introducing an observational bias.

Fortunately, thanks to ever-advancing technology and better sky coverage by ongoing sky surveys, recent years have witnessed an increasing number of discoveries of distant comets (e.g., C/2006 S3 (LONEOS) discovered at $r_{\mathrm{H}} = 14.3$ AU, C/2010 U3 (Boattini) at $r_{\mathrm{H}} = 18.4$ AU, both pre-perihelion), making a better understanding of activity in distant comets possible.

Comet C/2017 K2 (hereafter ``K2")  was detected by Pan-STARRS at Haleakala, Hawai`i on UT 2017 May 21 (Wainscoat et al.~2017), when it was $r_\mathrm{H} \approx 16$ AU from the Sun. The current orbital solution by the JPL Horizons ephemeris service identifies it as a long-period comet, with perihelion distance $q = 1.811$ AU, eccentricity $e = 1.00034$, inclination $i = 87\degr.6$, and a perihelion passage on UT 2022 Dec 21.\footnote{The elements are at epoch TT 2017 Jun 10.0, retrieved on 2017 Oct 04.}

In earlier work (Jewitt et al.~2017), we used the Hubble Space Telescope (\textit{HST}) to set a limit to the size of the nucleus ($R_{\mathrm{N}} \lesssim 9$ km) and established that the coma of K2 consists of large ($\gtrsim 0.1$ mm) sized dust grains released over a period of years. We identified a prediscovery detection from 2013 at heliocentric distance $r_{\mathrm{H}} = 23.7$ AU. Meech et al.~(2017) argued instead that the coma grains are small ($\sim$1 $\mu$m) and, using a sublimation model, inferred a nucleus radius $14 \le R_{\mathrm{N}} \le 80$ km. Both papers conclude that the activity is likely driven by the sublimation of a supervolatile ice (CO, CO$_2$, N$_2$, or O$_2$ according to Jewitt et al.~(2017) and CO according to Meech et al.~(2017)). Other, non-equilibrium processes are also possible.

In this paper, we present archival, serendipitous prediscovery observations of K2, and we explore the orbit of K2 using Monte Carlo simulations.

\section{\uppercase{Observations}}

We used the Solar System Object Image Search (SSOIS; Gwyn et al.~2012) of the Canadian Astronomy Data Centre (CADC), to find K2 in archival data. Ten $U$-band images were found, taken at the 3.6-m Canada-France-Hawaii Telescope (CFHT) on UT 2013 May 10, 12 and 13, using the MegaCam prime focus imager. The offsets from ephemerides by JPL Horizons and the Minor Planet Center (MPC) at the time of our prediscovery were enormous, $\sim$$+3\arcmin$ in RA and $-11\arcmin$ in declination for the former, and $\sim$$+1\arcmin$ in RA and $-3\arcmin$ in declination for the latter, but the rate of angular motion  was fully consistent with both sources. Detailed descriptions of the observations and the image of the comet are given in Jewitt et al.~(2017). Judging from its non-stellar appearance ($\mathrm{FWHM} = 1\arcsec.5 \pm 0\arcsec.1$, compared to $\mathrm{FWHM} \approx 0\arcsec.9$ for nearby background stars), the comet was active in 2013, although it is slightly trailed in the data (0\arcsec.98 in length) because of the long exposure and its non-sidereal motion.

Unfortunately none of the available star catalogs provide $U$-band magnitude data for stars in the field of view (FOV) of the CFHT data. Therefore, we calibrated $U$-band magnitudes of field stars then adjacent to the comet using the Keck-I 10-m telescope.  Observations on UT 2017 September 20 were taken using a $u'$-broadband filter (central wavelength 3404 \AA, full width at half maximum transmission 3750 \AA) under photometric conditions with the Low Resolution Imaging Spectrometer (LRIS; Oke et al.~1995).  The image scale was 0\arcsec.135 pixel$^{-1}$.  We calibrated field stars in the 2013 CFHT field using observations of photometric standard stars PG1648+536E and PG1633+099A  from the catalog by Landolt (1992).

Using image search software documented in Clark (2014), we also identified K2 in archival images from the Catalina Sky Survey (CSS) taken as early as 2015 November (see Table \ref{tab_geo} for details, and Figure \ref{fig_K2_CSS} for images), although with barely detectable motion with respect to the background sources, because of low angular resolution. The detections were consolidated by checking red plates from the Second Digitized Sky Survey (DSS2-red) having a similar limiting magnitude but a much better resolution. The CSS images from 2015 and 2016 were taken through a 0.7-m f/1.8 Schmidt telescope equipped with an unfiltered 4K $\times$ 4K CCD having an image scale of 2\arcsec.50 pixel$^{-1}$ and a FOV of 8.1 deg$^{2}$. Those from 2017 were obtained through the same telescope but equipped with an unfiltered 11K $\times$ 11K CCD having a scale of 1\arcsec.50 pixel$^{-1}$ and a FOV of 19.4 deg$^{2}$. All the images use an individual exposure time of $t_\mathrm{exp} = 30$ s, except that the set from 2015 has $t_\mathrm{exp} = 45$ s. The point-source $V$-band limiting magnitude of the images is $\sim$20.

We also searched for the comet in images taken between UT 2014 May 12 and 15 by the Palomar Transient Factory (PTF) 1.2-m diameter telescope (Law et al.~2009). The images, with an exposure time of 60 s and a FOV of $0\degr.6 \times 1\degr.1$, have a point-source $V$-band limiting magnitude of $\sim$20.5 (Waszczak et al.~2017), and a scale of 1\arcsec.01 pixel$^{-1}$. We had no success in detecting the comet in the individual images. Neither could we see anything above the noise level of the background in a stacked image coadded from the consecutive four-days of data with registration on the calculated motion of the comet. More recent PTF archival data from 2015 and 2016 cover the region of the comet, however, they are still proprietary, and we have no access to them. Given what we found with the CSS archival data, the comet should be detectable therein.

Prediscovery observations by the Pan-STARRS survey were reported as early as UT 2014 Mar 20 (Meech et al.~2017). We made no attempts to identify K2 in archival images prior to the CFHT observations in 2013, because the SSOIS inquiry shows no serendipitous observations from sufficiently large telescopes or with sufficiently long  exposure times.

\section{\uppercase{Results}}
\label{sec_rslt}

We performed aperture photometry on all the available images. Using an aperture 2\arcsec.3 in radius, we obtained the apparent magnitude of K2 in a stacked image, coadded from all the CFHT individual frames from UT 2013 May 10, 12 and 13 with registration on the apparent motion of the comet, as $m_U = 22.83 \pm 0.08$. This value benefits from the Keck calibration data described above and supercedes the coarse estimate ($m_U = 23.7 \pm 0.3$) by Jewitt et al.~(2017).

We also performed aperture photometry on the CSS images. Since they were obtained without a photometric-standard filter, we had to first determine the zero-points of the images ($ZP$) by introducing $k_\mathrm{c}$, which is the color term satisfying the following equation

\begin{equation}
m_{\ast,V} - ZP_\lambda + 2.5 \log \left( \frac{\mathcal{F}_{\ast}}{t_\mathrm{exp}} \right) - k_\mathrm{c} \left( m_{\ast,B} - m_{\ast,V} \right) = 0
\label{eq_zp},
\end{equation}

\noindent where $m_{\ast, \lambda}$ is the star magnitude in some bandpass, and $\mathcal{F}_{\ast}$ is the signal in ADU of the star measured within photometric apertures of 15\arcsec.0 and 12\arcsec.0 in radius, respectively for the images taken by the old and new CCDs, which are approximately twice the FWHM of the field stars. The sky background was computed in annuli having inner and outer radii of $\sim$3$\times$ and $5 \times$ FWHM for the data, respectively. We utilised the AAVSO Photometric All-Sky Survey Data Release 9 (APASS-DR9; Henden et al.~2016) and reduced the zero-points of all the CSS images in the $V$ band from the least-squares fit (see Figure \ref{fig_CSS_ZP} as an example). The color of the comet ($m_U - m_V = 1.11 \pm 0.03$, $m_B - m_V = 0.74 \pm 0.02$, $m_V - m_R = 0.45 \pm 0.02$, from our Keck observation), is assumed to be unchanged. Then the measured flux of comet K2, which was obtained by applying the same photometric aperture and sky annulus that we used for stars, is converted to the apparent $V$-band magnitude. The errors stem mainly from the uncertainty in the determination of the zero-points ($\sim$0.2 mag), as well as the low signal-to-noise ratio of the comet.

Our results are summarised in Table \ref{tab_mag}. The temporal evolution of the apparent magnitude of K2 is shown in Figure \ref{fig_lc_2017K2}(a), in which we have included photometry from Meech et al.~(2017).\footnote{Magnitude data by Meech et al.~(2017) from the prediscovery Pan-STARRS observations were converted from Sloan-$r'$ to $V$-band magnitude using a transformation equation from Jordi et al.~(2006).} Note that different sizes of photometric apertures have been employed (see Table \ref{tab_mag}). Meech et al.~(2017) scaled their measurements by means of curves of growth to a set that would have been obtained using an aperture of 5\arcsec.0 in radius, which is much smaller than aperture sizes we applied to the CSS images. As the comet is a diffuse source, $\sim$9\arcsec~in radius in June 2017 (Jewitt et al.~2017), we expect that their measurements  systematically underestimate the brightness (as is evident in Figure \ref{fig_lc_2017K2}). The closer the comet, the more significant is the difference. For example, for a steady-state coma with sufficient signal-to-noise ratio, the aperture correction in the prediscovery Pan-STARRS observations from $\mathit{\Delta} = 22.2$ to 16.3 AU results in a difference of $\sim$0.3 mag, with the more recent photometry being too faint. Indeed, this phenomenon can be readily seen in Figure \ref{fig_lc_2017K2}(a).  However, our attempts to correct the Pan-STARRS photometry to an aperture of fixed linear (as opposed to angular) radius  based on the surface brightness profile obtained in Jewitt et al.~(2017) failed to give satisfactory results.   We simply decided not to perform aperture corrections.

In order to investigate the intrinsic brightness of the comet, the effect of the varying viewing geometry ought to be eliminated, so we compute the absolute magnitude from

\begin{equation}
m_V \left(1,1,0 \right) = m_V \left( r_\mathrm{H}, \mathit{\Delta}, \alpha \right) - 5 \log \left(r_\mathrm{H} \mathit{\Delta} \right) + 2.5 \log \phi \left( \alpha \right)
\label{eq_H},
\end{equation}

\noindent where $\alpha$ is the phase angle, and $\phi (\alpha)$ is the phase function of the coma.  For the latter, we use the empirical phase function of dust (Marcus 2007; Schleicher \& Bair 2011; \url{http://asteroid.lowell.edu/comet/dustphase.html}), and normalised at $\alpha=0$\degr. The result is shown in Figure \ref{fig_lc_2017K2}(b). We then estimate the effective cross-section, $C_\mathrm{e}$, using

\begin{equation}
C_\mathrm{e} = \frac{\pi r_{\oplus}^2}{p_{V}} 10^{-0.4 \left[m_{V} \left(1,1,0 \right) - m_{\odot,V}\right]}
\label{eq_Xsect},
\end{equation}

\noindent where $r_\oplus \approx 1.5 \times 10^{8}$ km is the mean Sun-Earth distance, $p_{V}$ is the $V$-band geometric albedo, assumed to be $p_V = 0.04 \pm 0.02$ (see Lamy et al.~2004), and $m_{\odot,V} = -26.74$ is the $V$-band apparent magnitude of the Sun. We plot the temporal variation of the effective cross-section of K2 in Figure \ref{fig_xs_2017K2}, and summarise our photometry in Table \ref{tab_mag}.

\section{\uppercase{Discussion}}

\subsection{Dust Dynamics}
\label{sec_dd}

We attempted to constrain properties of dust grains of comet K2 in the high-resolution post-discovery \textit{HST} observation by Jewitt et al.~(2017) using our Monte Carlo model, in which dust grains are released from the nucleus with a range of non-zero initial velocities, and are subsequently subject to the solar radiation pressure force and the gravitational force from the Sun, whose ratio is denoted as $\beta$. The grain size is related to $\beta$ by $\beta \propto \left(\rho_{\mathrm{d}} \mathfrak{a} \right)^{-1}$, where $\mathfrak{a}$ is dust grain size, and $\rho_{\mathrm{d}}$ is bulk density. Together with the release time of the dust particles from the observed epoch $\tau$, their trajectories can be uniquely determined. We assumed that the dust particles follow a simple power-law size distribution, i.e., $\mathrm{d} n \left(\mathfrak{a} \right) \propto \mathfrak{a}^{-3.5} \mathrm{d}\mathfrak{a}$, where $\mathrm{d} n $ is the number of dust grains having radii between $\mathfrak{a}$ and $\mathfrak{a} + \mathrm{d} \mathfrak{a}$, and that their ejection speeds are described by an empirical relationship $v_{\mathrm{ej}} \propto \mathfrak{a}^{-0.5}$. Similar Monte Carlo models have been widely applied elsewhere (e.g., Fulle 1989; Ishiguro 2008; Moreno 2009; Ye \& Hui 2014; Hui et al.~2017).

We tested our simulation with different combinations of minimum, $\mathfrak{a}_{\min}$, and maximum, $\mathfrak{a}_{\max}$, particle radius,   earliest dust release time from the observed epoch, $\tau_{0}$, and $v_{\mathrm{ej}}$. A successful model should be able to match the observed morphology of the coma. The model is insensitive to the size of the largest dust grains, $\mathfrak{a}_{\max}$, because such particles are rare and carry a negligible fraction of the total scattering cross-section for the power-law index assumed. In Figure (\ref{fig_mdl_17K2}) we show two models computed using  $\mathfrak{a}_{\max} = 2$ mm, $v_{\mathrm{ej}}$ = 1.9 m s$^{-1}$ and $\tau_{0} = 1500$ days, consistent with the CFHT observation that K2 has been active at least since 2013 May. The remaining dust parameters can then be obtained without much ambiguity. By inspection, we determine that large dust grain (radii of $0.5 \lesssim \mathfrak{a} \lesssim 2$ mm) models  closely simulate the morphology of comet K2 in the \textit{HST} data (c.f.~the middle- and right-hand panels of Figure \ref{fig_mdl_17K2}). On the other hand, models including smaller particles (radii of $0.01 \lesssim \mathfrak{a} \lesssim 2$ mm) show a clear radiation-pressure-swept tail which is not present in the data (c.f.~left and right-hand panels of the Figure).  The Monte Carlo models thus support the inference made by Jewitt et al.~(2017) to the effect that the coma is dominated by submillimeter particles, but contradicts the one by Meech et al.~(2017), who assumed 2 $\mu$m-sized dust particles in their sublimation model. We found that this conclusion cannot be mitigated even if  more recent ejection times, $\tau_{0}$, and higher ejection speeds are adopted. To conclude, the observed dust particles of comet K2 must be large, at least submillimeter sized, to avoid the formation of an observable, radiation pressure swept tail.

\subsection{Mass Loss}
\label{sec_ml}

There is a large scatter in the absolute magnitudes (see Figure \ref{fig_lc_2017K2}(b)) in part due to the aperture issue discussed in Section (\ref{sec_rslt}).  Nevertheless, the comet appears to brighten in data from 2017 compared to 2013. The brightening corresponds to a maximum possible increase in the scattering cross-section $\Delta C_\mathrm{e} \approx \left(4.5 \pm 2.1 \right) \times 10^{4}$ km$^{2}$.  The mean mass-loss rate of the comet, denoted as $\overline{\dot{M}}$, can be estimated from

\begin{equation}
\overline{\dot{M}} = \frac{4\rho_{\mathrm{d}} \Delta C_\mathrm{e} \bar{\mathfrak{a}}}{3\Delta t}
\label{eq_mloss},
\end{equation}

\noindent where $\rho_{\mathrm{d}}$ and $\bar{\mathfrak{a}}$ are the bulk density and the mean radius of the dust grains, respectively. With a nominal $\rho_{\mathrm{d}} = 0.5$ g cm$^{-3}$,  $\bar{\mathfrak{a}} \approx 1$ mm, and $\Delta t \approx 1.2 \times 10^{8}$ s, Equation (\ref{eq_mloss}) yields $\overline{\dot{M}} \approx \left(2.4 \pm 1.1 \right) \times 10^{2} $ kg s$^{-1}$. The uncertainty only incorporates the error from photometry together with the error in albedo. Our estimate for the mass-loss rate is larger than $\overline{\dot{M}} \sim 60$ kg s$^{-1}$ by Jewitt et al.~(2017), mainly because we adopted a larger mean grain size based on our Monte Carlo simulations. Given the fact that the maximum dust dimension cannot be confidently constrained and that there are many other unknowns, such a difference is not significant.

A lower limit to the size of the nucleus of comet K2 can be estimated by assuming that the activity is supported by equilibrium sublimation of exposed ices. We solve the energy equilibrium equation between insolation, thermal emission and sublimation,

\begin{equation}
\left(1 - A_{\mathrm{B}} \right) S_{\odot}\left( \frac{r_{\oplus}}{r_{\mathrm{H}}} \right)^{2} \cos \zeta = \epsilon \sigma T^{4} + L\left( T \right) f_{\mathrm{s}} \left( T \right)
\label{eq_sub},
\end{equation}

\noindent in which $A_{\mathrm{B}}$ is the Bond albedo, $S_{\odot} = 1361$ W m$^{-2}$ is the solar constant, $\cos \zeta$ is the effective projection factor for the surface ($\cos \zeta = 1$ for a subsolar scenario, and $\cos \zeta = 1/4$ corresponding to an isothermal nucleus), $\epsilon$ is the emissivity, $\sigma = 5.67 \times 10^{-8}$ W m$^{-2}$ K$^{-4}$ is the Stefan-Boltzmann constant, $T$ is the surface temperature in K, and $L$ (in J kg$^{-1}$) and $f_{\mathrm{s}}$ (in kg m$^{-2}$ s$^{-1}$) are the latent heat of the sublimating material, and the mass flux of the sublimated ice, respectively, both as functions of temperature. The heat conduction towards the nucleus interior is ignored. For simplicity, we assign $\epsilon = 0.9$ and $A_{\mathrm{B}} = 0.01$ (e.g., Buratti et al.~2004) and analyse the sublimation of CO (whose volatility is representative of other potential supervolatiles like N$_2$ and O$_2$) and CO$_{2}$ at the time-averaged heliocentric distance $r_{\mathrm{H}} = 17.1$ AU during the period 2013--2017.

We adopted empirical thermodynamic parameters of CO and CO$_2$ respectively listed in Prialnik et al.~(2004) and Cowan \& A'Hearn (1979), solved Equation (\ref{eq_sub}) and obtained $4.9 \times 10^{-6} \le f_{\mathrm{s}} \le 2.0 \times 10^{-5}$ kg m$^{-2}$ s$^{-1}$ for CO, and $8.6 \times 10^{-11} \le f_{\mathrm{s}} \le 2.2 \times 10^{-6}$ kg m$^{-2}$ s$^{-1}$ for CO$_{2}$, where the lower limits correspond to isothermal sublimation and the upper ones are from subsolar sublimation. In order to supply the mass-loss rate inferred from photometry, the minimum surface area, $A_{\mathrm{s}} = \overline{\dot{M}} / f_{\mathrm{s}}$, has to be in the range $12 \lesssim A_{\mathrm{s}} \lesssim 48$ km$^2$ for sublimation of CO, and $1.1 \times 10^{2} \lesssim A_{\mathrm{s}} \lesssim 2.8 \times 10^{6}$ km$^2$ for CO$_{2}$. These are equivalent to equal-area circles of radii $R_{\mathrm{N}} \sim \sqrt{A_{\mathrm{s}} / \pi} \gtrsim 2$ km and $R_{\mathrm{N}} \gtrsim 6$ km, respectively.  Given that the upper limit to the nucleus radius from the HST measurement is $R_{\mathrm{N}} <$ 9 km, we see that sublimation of CO (and N$_2$, O$_2$) is  easily capable of supplying the coma even if only a small fraction of the nucleus surface is active, while CO$_2$ must be sublimating from near the subsolar point, if it is present.

We then proceed to estimate the critical grain size, $\mathfrak{a}_{\mathrm{c}}$, of dust particles, which can be lifted off from the surface by the gas-drag force $F_{\mathrm{D}} = C_{\mathrm{D}} \pi \mathfrak{a}^2 \mu \mathfrak{m}_{\mathrm{H}} N v_{\mathrm{th}}^{2}$, where $C_{\mathrm{D}}$ is the dimensionless drag coefficient, $\mu$ is the molecular weight ($\mu = 28$ for CO, and $\mu = 44$ for CO$_2$), $\mathfrak{m}_{\mathrm{H}} = 1.67 \times 10^{-27}$ kg is the mass of the hydrogen atom, $N$ is the number density of the molecule, and $v_{\mathrm{th}}$ is the thermal speed of the gas. By equating the gas-drag force and the gravitational force at the surface and ignoring spinning of the body, with simple algebra we derive the critical grain dimension as

\begin{equation}
\mathfrak{a}_{\mathrm{c}} = \frac{9C_{\mathrm{D}} \overline{\dot{M}}}{32 \pi^2 G \rho \rho_{\mathrm{d}} R^3} \sqrt{\frac{2 k_{\mathrm{B}} T}{\pi \mu \mathfrak{m}_{\mathrm{H}}}}
\label{eq_ac},
\end{equation}

\noindent where $G = 6.67 \times 10^{-11}$ m$^{3}$ kg$^{-1}$ s$^{-2}$ is the gravitational constant, $\rho$ is the density of the nucleus, and $k_{\mathrm{B}} = 1.38 \times 10^{-23}$ J K$^{-1}$ is the Boltzmann constant. Assuming a unity gas-to-dust production ratio, along with $C_{\mathrm{D}} = 1$ and $\rho = \rho_{\mathrm{d}}$, Equation (\ref{eq_ac}) yields $\mathfrak{a}_{\mathrm{c}} \lesssim 4$ mm for sublimation of CO, and $\mathfrak{a}_{\mathrm{c}} \lesssim 0.2$ mm for CO$_2$, which is in line with Jewitt et al.~(2017). Note that our dust model suggests the size of the dust grains $\mathfrak{a} \gtrsim 0.5$ mm. We thus prefer CO (and materials of similar volatility) sublimation as the cause of the activity, but since there are many approximations in our model (e.g.~the neglect of rotation and the neglect of contact forces at the nucleus surface, see Gundlach et al.~2015), we feel that it would be premature to rule out CO$_2$ as the activity driver.

\subsection{Orbital Evolution}
\label{sec_orb}

We next examine the dynamical evolution of comet K2 in an attempt to understand its recent history. We downloaded the astrometric measurements of the comet from the MPC, which include our own astrometry from the prediscovery archival images. The measurements were debiased following Farnocchia et al.~(2015). The code \textit{EXORB9}, a part of the \textit{SOLEX12} package developed by A. Vitagliano, was exploited for orbit determination. Weights on each set of observations were adjusted to approximately accommodate ad hoc astrometric residuals whenever they were found aggressive. Twenty-four observations with residuals greater than 1\arcsec.5 either in RA or declination were discarded, leaving 336 observations (93\% of the total number) to be fitted by orbit determination. An optimised solution was thereby obtained, having a weighted rms of 0\arcsec.472. Our derived orbital elements are generally similar to those in the solution by JPL Horizons (Table \ref{tab_orb}), despite different choices of the weighting scheme and the threshold for filtering bad-residuals astrometry (D. Farnocchia, private communication). We then generated 10$^4$ clones of the nominal orbit according to the associated covariance matrix of the orbital elements, and performed backward N-body integration in \textit{MERCURY6} (Chambers 1999) using the 15th-order RADAU integrator (Everhart 1985) for the past 1 kyr, and the hybrid symplectic algorithm for the past 1 Myr to investigate the dynamical evolution of K2.\footnote{Switching to the hybrid integration scheme is a measure to reduce the computation time, at the cost of losing some accuracy. Since we have no interest in examining orbits of individual clones, but are looking at the total statistics, the result is not influenced.} Gravitational perturbations from the eight major planets and Pluto, post-Newtonian corrections (Arminjon 2002), and the influence of the galactic tide, which is a major perturber of the Oort cloud (e.g., Heisler \& Tremaine 1986; Fouchard et al.~2005), were included in the simulation. The possible distant giant planet claimed by Trujillo \& Sheppard (2015) and Batygin \& Brown (2016) was not considered, since the evidence for this body remains equivocal (Shankman et al.~2017). Neither have we included stellar perturbations, although the frequency of encounters with stellar systems passing within 1 pc of the Sun is estimated to be as many as $11.7 \pm 1.3$ Myr$^{-1}$ (Garc\'{i}a-S\'{a}nchez et al.~2001), yet 73\% of the encounters are with M dwarfs having low masses ($\lesssim 0.4 M_{\odot}$, where $M_{\odot}$ is the solar mass).

We did not incorporate possible non-gravitational acceleration of K2 in the orbital solution. To test the impact of this neglect, we employed \textit{EXORB9} to repeat the aforementioned procedures to solve for non-gravitational parameters $A_{j}$ ($j=1,2,3$) as defined in Marsden et al.~(1973) but obeying an empirical momentum-transfer law from sublimation of CO and CO$_2$ in a hemispherical scenario,  following the method in Hui \& Jewitt (2017). No detection of non-gravitational acceleration was made above the formal uncertainty levels (well below $1\sigma$). Solving for the non-gravitational parameters barely helps reduce the rms of the fit (to 0\arcsec.471),  justifying our omission of the non-gravitational effect.

The orbital evolution of K2 in terms of the reciprocal of the semimajor axis ($a^{-1}$), perihelion distance $q$, eccentricity $e$ and inclination $i$ in the past 1 kyr is shown in Figure \ref{fig_oscele_1ky_17K2}. Note that the orbital elements are still referred to the heliocentric reference system. We can see that the ranges of $q$ and $i$ sway increasingly with time in the past 1 kyr, whereas $a^{-1}$ gradually approaches $\sim$10$^{-5}$ AU$^{-1}$ and $e$ tends to creep $< 1$. The examined orbital elements exhibit zigzagging oscillations with a dominant period of $\sim$11.9 yr, close enough to the orbital period of Jupiter to indicate non-negligible gravitational perturbations from the gas giant.

Now we move on to results from the backward integration for the past 1 Myr. Starting from now, we change the reference origin to the barycenter of the solar system. We obtain median values $a_{\mathrm{med}}^{-1} = \left(3.61 \pm 1.71 \right) \times 10^{-5}$ AU$^{-1}$ and $e_{\mathrm{med}} < 1$ from the clones (See Figure \ref{fig_baryorb_1Myr_17K2}, the assigned uncertainty is the standard deviation). Only 173 ($\sim$1.7\%) of the total clones have originally hyperbolic orbits and so we conclude that the comet is very unlikely to be of interstellar origin.  Instead, K2 is probably from the Oort spike, which consists of a mix of dynamically new and old comets (Kr{\'o}likowska \& Dybczy{\'n}ski 2010; Fouchard et al.~2013).  We cannot determine whether the comet is dynamically new or old from our backward integration, because the integration time (1 Myr) is shorter than the orbital period of the comet, $P = a^{3/2} \gtrsim 2$ Myr. Only from the region with $a^{-1} < 2.5 \times 10^{-6}$ AU$^{-1}$ in the Oort spike, are dynamically old comets completely absent (Kr{\'o}likowska \& Dybczy{\'n}ski 2017). As a result, whether K2 penetrated into the planetary region during the previous perihelion passage must be regarded as unsettled. Analysis of the forward integration of the orbit of K2 is not performed, because we have concern that intensified  sublimation activity as the comet approaches the Sun will intensify   non-gravitational effects.

Although detected as early as 2013, K2 managed to repeatedly escape detection by the major sky surveys. Why was comet K2 not discovered much earlier? The two important reasons, we suspect, are the high inclination and the low rate of its angular motion. The majority of sky surveys are optimised for making discoveries of small bodies that move at much higher speeds, such as near-Earth and main-belt asteroids. Furthermore, angualr resolution has been generally sacrificed for wider-FOV coverage, making discovery of slow-moving objects even more difficult. Although surveys like the Outer Solar System Origin Survey\footnote{\url{http://www.ossos-survey.org/about.html}} are dedicated to transneptunian objects, and should have had capability to detect objects moving as slowly as K2, they mainly search along the ecliptic plane.

\section{\uppercase{Summary}}

Key conclusions of our study about comet C/2017 K2 (PANSTARRS) are summarised as follows.

\begin{enumerate}

\item The comet was recorded serendipitously by the CFHT and the CSS on many occasions since 2013. At $r_{\mathrm{H}} = 23.7$ AU, K2 is the most distant comet ever observed on the way to perihelion.

\item The combined archival photometry suggests that the activity of the comet has been slowly increasing since 2013, as it approaches the Sun.

\item By means of our Monte Carlo simulation of the dust motion, we confirm that dust properties estimated by Jewitt et al.~(2017) during the \textit{HST} observation are valid, i.e., predominant dust grains of the comet are $\gtrsim 0.5$ mm in radius, with ejection speeds of $\sim$1--3 m s$^{-1}$, and have been released in a continuous manner since 2013 May.

\item By assuming a cometary albedo $p_V = 0.04 \pm 0.02$, the mass-loss rate of comet K2 during the period of 2013--2017 was estimated to be $\overline{\dot{M}} \approx \left(2.4 \pm 1.1 \right) \times 10^{2}$ kg s$^{-1}$, which requires a minimum active surface area of $\sim$12 km$^{2}$ if the activity is driven by sublimation of CO, and $\sim$110 km$^{2}$ for CO$_2$. The nucleus must be at least of kilometer-size to sustain the observed activity by sublimation.

\item Monte Carlo simulations of the pre-entry orbit of the comet give original (1 Myr ago) reciprocal semimajor axis  $a_{\mathrm{med}}^{-1} = \left(3.61 \pm 1.71 \right) \times 10^{-5}$ AU$^{-1}$ (referred to the barycenter of the solar system).   We find  that some 98\% of orbital clones originate from within the Oort spike.

\end{enumerate}

\acknowledgements
{
We thank Eric Christensen, Davide Farnocchia, Aldo Vitagliano, and Quan-Zhi Ye for their generous help, and the anonymous referee for a speedy review. This research is in part based on observations obtained with MegaPrime/MegaCam, a joint project of CFHT and CEA/DAPNIA, at the Canada-France-Hawaii Telescope (CFHT) which is operated by the National Research Council (NRC) of Canada, the Institut National des Sciences de l'Univers of the Centre National de la Recherche Scientifique of France, and the University of Hawaii. The facilities of the Canadian Astronomy Data Centre operated by the National Research Council of Canada with the support of the Canadian Space Agency, and the AAVSO Photometric All-Sky Survey (APASS), funded by the Robert Martin Ayers Sciences Fund, were used. Some of our data presented herein were obtained at the W. M. Keck Observatory, which is operated as a scientific partnership among the California Institute of Technology, the University of California and the National Aeronautics and Space Administration. The Observatory was made possible by the generous financial support of the W. M. Keck Foundation. The products of the Digitized Sky Surveys, which were produced at the Space Telescope Science Institute under U.S. Government grant NAG W-2166, were employed in this research. The NASA/ IPAC Infrared Science Archive, which is operated by the Jet Propulsion Laboratory, California Institute of Technology, under contract with the NASA, has been made use of. This work is funded by a grant from NASA to D.J.
}

\clearpage


\section{Introduction}

Self-supervised learning has opened an intriguing new avenue into unsupervised learning. It is intellectually satisfying due to the way it resembles gestalt-like mechanisms of learning in visual cortical formation. It is also appealing for that fact that it can be implemented with standard off-the-shelf neural networks and toolkits.

Self-supervised learning methods create a protocol whereby the computer can learn to teach itself a supervised task. For instance, in \cite{Doersch15} a convolutional neural network (CNN) was taught to learn the arrangement of patches in an image. By learning the relative position of these patches, the network would be forced to also learn the features and semantics that underlie the image. Although the network is trained to learn patch positions, the final goal was to generalize the learned representation to solve other tasks. For instance, the self-supervised network was trained on a transfer task (fine-tuned) to classify objects in the PASCAL VOC dataset \cite{Everingham10}, and compared with a CNN trained on a supervised task, such as learning to classify the ImageNet dataset \cite{Imagenet09}. If the self-supervised network learned good generalizations of image features and semantics, it should perform as well as a supervised network on transfer learning.

Over the last few years, several methods of self-supervised learning have been introduced. For instance, \cite{Dosovitskiy15} trained a CNN to recognize which transformation had been performed on an image. Since then, methods have been introduced that use context arrangement of image patches \cite{Doersch15,Noroozi16a}, image completion \cite{Pathak16}, image colorization \cite{Zhang17,Zhang16,Larsson17}, motion segmentation \cite{Pathak17}, motion frame ordering \cite{Wang15b,Wang17,Lee17}, object counting \cite{Noroozi17} and a multi-task ensemble of many models \cite{Doersch17,Kim18}.

Each method has relative strengths and weaknesses. For instance, \cite{Zhang17} uses a customized ``split-brain'' architecture that makes it less off-the-shelf than other solutions which frequently use a standard network such as AlexNet \cite{AlexNet}. \cite{Wang15b,Wang17,Lee17,Pathak17} all use motion cues, but have the downside of being constrained to video data.  Many patch based methods use a Siamese network architecture which incurs extra memory demands \cite{Doersch15,Noroozi16a,Noroozi17,Lee17}. However, every self-supervised method suffers the drawback of still being very short of supervised methods in terms of transfer learning performance.

Our intent in this work is to improve the performance of self-supervised learning. We present a variety of techniques we hope are applicable to other approaches as well. We will demonstrate generalizability of these techniques by running them on several different neural network architectures and many kinds of datasets and tasks.  For instance, as we will discuss, one dataset we wish to add to the corpus of standard self-supervised tests, the Caltech-UCSD Birds 200 set (CUB) \cite{Welinder10} is excellent for finding potential short comings of techniques designed to address the well-known \emph{chromatic aberration} problem \cite{Doersch15}. We will suggest this is due to the importance of color patterns in bird classification \cite{Cornell09}.

\section{Issues and Related Work}
\begin{figure*}
\centering
\includegraphics[height=3.75 cm]{Chroma-Blur_v2_sm.jpg}
\caption{These are examples of patches taken from ImageNet that are used during self-supervised training. Below each original is an example with chroma blurring. It is frequently difficult to distinguish the blurred from original images, because humans are not very spatially sensitive to variation of color. Chroma blurring can sometimes result in a loss of color saturation and color bleeding of very saturated regions (such as the red ship bow, third from right). Notice the original gold fish image has signs of strong chromatic aberration (top-left of head). This is blended out effectively by chroma blurring which switches the green aberration to the fish's own red color. \emph{See supplementary material appendix} figure ~\ref{fig:chroma_blur_layer} for conv1 layer filter comparisons. }
\label{fig:chroma_blur}
\end{figure*}
We use a patch/context approach for the issue of self-supervised learning \cite{Doersch15,Noroozi16a}. This is a popular method, but is by no means the only active path of inquiry. Patch/context approaches work by creating an arrangement of image patches in either space or time. Each distinct arrangement is assigned a class label, and the network then predicts the correct arrangement of these patches by solving a supervised classification problem. The network can be a typical supervised network such as AlexNet \cite{AlexNet}, VGG \cite{VGGNet}, GoogLeNet \cite{GoogLeNet} or ResNet \cite{ResNet}. In order to view multiple patches at the same time, a Siamese network is frequently used where each patch is fed into an independent network path, and each path shares weights. \cite{Doersch15} used a system of two patches in a finite set of eight possible spatial configurations. \cite{Noroozi16a} created an extension using as many as nine patches in a \emph{puzzle} configuration. Temporal ordering of patches can also be used. For instance, \cite{Lee17} shuffled four consecutive video frames to create 12 classes for prediction. Another temporal method \cite{Wang17} queried the networks ability to determine if a patch came from the same object later in time or a similar but different object. This can be considered a meta self-learner since it leverages \cite{Doersch15} as a pre-self-supervised learner to determine object similarity.

Patch based methods have the advantage of being easy to understand, network architecture agnostic, and frequently straightforward to implement. They also tend to perform well on standard measures of transfer learning. For instance, \cite{Doersch15} is a top performer on PASCAL VOC 2007 detection \cite{Girshick15}, even among a large number of new arrivals. \cite{Noroozi16b} is almost tied for the top score on PASCAL VOC 2007 classification \cite{Krahenbuhl16} and has the top score for PASCAL VOC 2007 detection and the second highest score for PASCAL VOC 2012 segmentation \cite{Long15,Pathak16}.

However, patch/context-based networks typically suffer from an issue of being able to ``cheat'' and bypass learning the desired semantic structure by finding incidental clues that reveal the location of a patch. An example of this is \emph{chromatic aberration}, which occurs naturally as a result of camera lensing where different frequencies of light exit the lens at slightly different angles. This radially offsets colors such as magenta and green modestly from the center of the image. By looking at the offset relation of different color channels, a network can determine the angular location of a patch in an image. Aberration varies between lenses, so it's an imperfect cue, but one that none-the-less exists. A common remedy is to withhold color data from one or more channels by using channel-dropping \cite{Doersch15,Doersch17}, channel replication \cite{Lee17}, or conversion to gray scale \cite{Noroozi17}.  The primary difficulty with these approaches is that color becomes decorrelated (or absent) since colors are not observed together. This makes it difficult to learn color opponents for patterns that emerge in supervised training sets, such as ImageNet. Another approach is to jitter color channels \cite{Noroozi16a}, but this has a similar effect to blurring an image, and it might affect the sharpness of learned wavelet features.

An often-cited worry in all patch/context works relates to trivial low-level boundary pattern completion \cite{Doersch15,Noroozi16a,Doersch17,Noroozi17,Lee17}. The neural network may learn the alignment of patches not based on (for instance) desirable semantic information, but instead by matching the top or bottom part of simple line segments. Two common approaches are to provide a large enough gap between patches and to randomly jitter the patches. This last technique may be dubious since a convolutional neural network can align simple patterns at arbitrary offsets. This issue may also be implicitly addressed by having non-4-connected adjacent patches. Half the patches in \cite{Doersch15} are arranged diagonally which should make them resistant to trivial low-level boundary pattern completion. Also, we should note that while we would not want a self-supervised learner to use this cheat all the time, it could be used as a cue to help form low level features. So, it is somewhat unclear how much of a problem this might be.

In another \emph{possible} problem for self-supervised networks in general, mid-layers in the network may not train as well as the early and later layers.  For instance, \cite{Doersch17} created a self-supervised network using an ensemble of different methods. They then created an automated lasso to grab layers in the network most useful for their task. The lasso tended to grab layers very early or very late in the network. This suggests that for many self-supervised tasks, the information in the middle network layers is not very essential for training. Another piece of evidence comes from the CSAIL Places linear test \cite{Zhang17,Zhang16}, which shows how well each layer in the network performs on transfer learning. Many self-supervised networks perform as well or better than a supervised ImageNet trained network at the first and second convolutional layers in AlexNet, but struggle at deeper layers.

\section{Approach}

Our approach is comprised of three parts. The first is a collection of tools and enhancements which for the most part should be transferable to other self-supervised methods. In our second part, we utilize two new datasets to make our experiments more diverse and general. The third part is a demonstration on several different neural networks to verify generalization and demonstrate portability.

For our general approach, we start with the baseline of \cite{Doersch15} using a two-patch spatial configuration paradigm. This approach gives good results, and is easy to both implement and understand. We then augment this approach using various techniques. For each technique, we vigorously test effects empirically to justify their usage.

\subsection{Our Toolbox of Methods}

\subsubsection{Chroma Blurring (CB)}

We address the problem of chromatic aberration by removing cues about image aberration while allowing color patterns and opponents to be at least partially preserved. We note that the human visual system is not very sensitive to spatial frequency of chroma, but is much better at discerning detail about shifts in intensity \cite{Livingstone02}. However, even with this lack of spatial acuity for color, we can still discern meaningful color patterns. As such, we balance the tradeoff between decreasing color spatial information and removing chromatic aberration cues.

To preserve intensity information, but reduce aberration, we start by converting images into \emph{Lab} color space. We then apply a blurring operation to the chroma \emph{a} and \emph{b} channels. In this case, we use a 13x13 box filter. It is two times the size of the 7x7 convolution filter of our original GoogLeNet target network. The luminance channel is left untouched, and we convert the image back to RGB. Figure~\ref{fig:chroma_blur} shows several patches which we have \emph{chroma blurred} for comparison.

\subsubsection{Yoked Jitter of Patches (YJ)}

Most patch/context methods apply a random jitter between patches \cite{Doersch15,Noroozi16a,Doersch17,Lee17}. The different patches are jittered in different amounts and different directions.  One issue with applying a random jitter is that it might distort or skew the spatial understanding in the network. As an example, if the head of an animal is observed in one patch and the feet in the other, the true spatial extent between these items would be difficult to discern given a random jitter: the patches might be 30 pixels apart, or they might be 60. If on the other hand, the patches maintained a fixed spacing, reasoning about the extent of an object between patches would be easier. Thus, the network might make better inferences about the larger shape of an object beyond each patch itself.

We do this by yoking the patch jitter. Each patch is randomly jittered to create a random crop effect, but they are jittered by the same amount in the same direction. This might make us prone to trivial low-level boundary pattern completion, but as mentioned, we suspect this can be partially addressed by having non-4-connect patches. Also, it is unclear how well a random jitter will address such a problem since features do not need to be aligned in order to be recognized in a CNN. Additionally, a certain amount of low-level boundary pattern completion may not be a problem since it may enhance learning of simple features.
\begin{figure*}
\centering
\includegraphics[height=8.4 cm]{Patches_84h_Examples_3_sm.jpg}
\caption{The left column shows the location patches are extracted from in the image. The next column over shows some example configurations obtained from those patches. In the middle are all 20 patches extracted from this (and every) image. The order is labeled for each patch in a set as \emph{P1},\emph{P2} and \emph{P3}. The right column shows how these patches are fed into the process to then create the final patches fed into the neural network.}
\label{fig:patch_example}
\end{figure*}
\subsubsection{Additional Patches and Configurations}

We use the same 96x96 sized patches as \cite{Doersch15}, since it fits the receptive field of a 3x3 convolution at the end of many CNNs. That is, most of the popular networks have a five-layer topology of layer groups with each layer group being half the dimension of the preceding one \cite{GoogLeNet,ResNet,DenseNet,AlexNet,Mundhenk2016} (AlexNet technically has four group layers, but the first layer has half-scale of most other networks). There is some dimensional variation caused by the omission of padding in some layers, but one pixel on the end layer maps to an extent of about 32 pixels in the input image. Since most networks tend to only use 3x3 convolutions at the last layers, a patch size of 96x96 is justified to cover its full field.

We use three patches {\bf (TP)} in each set. In the two-patch configuration of \cite{Doersch15}, one of the two patches may not cover an area with useful information. For instance, half of the image may be covered by ocean. We can address this problem by using more patches. \cite{Noroozi16a} uses a ``puzzle'' system of nine 80x80 patches. However, if we want to use 96x96 sized patches and be able to train larger, more contemporary networks, a 9x9 puzzle may not be feasible to train. If, on the other hand, we just add one more patch, we create a triple Siamese network which is smaller than a single network over the traditional 224x224 image size. This makes it easy to move to a larger network. We did not test four patches.

We add extra patches configurations {\bf (EPC)}. That is, we added several new configurations of patches seen in Figure~\ref{fig:patch_example}. Adding new patterns (1) creates more orthogonal and unique patterns (2) covers more of the image at once (3) mixes scales to prevent simple pattern completion (4) creates a natural way to cover the image, but use multiple scales for training.

We draw three different patterns of patches. We start by extracting all patches at a 110x110 resolution. \emph{3x3} patches are taken from a 384x384 image. This is taken from the center of an aspect preserved image by reducing the smallest dimension to that size. The patches are evenly spaced and aligned with the image corners to cover the image (there is no edge margin). \emph{2x2} patches are taken from a 256x256 image and \emph{overlap} patches are taken from a 196x196 image. We then use eight 3x3 patterns similar to \cite{Doersch15}, 2x2 patterns are L-shaped and we use four of them. We combine patches from 3x3 and overlap patches to create \emph{hybrid} patch sets. These are hybrid scale patches which allow more semantic reasoning by preventing easy matching of simple features between patches since they are at different scales. The processed patches are fed into the network in batches. The final patches fed into the network are sized 96x96. Note that Figure~\ref{fig:patch_example} shows all 20 typical combinations from a sample image. These are the exact same patterns extracted from all images in the ImageNet training set. In all, we obtain 25,623,340 training patch sets from ImageNet.

\subsubsection{Random Aperture of Patches (RA)}
\begin{figure}
\centering
\includegraphics[height=3.0 cm]{aperture_2.jpg}
\caption{The grayed-out area has been apertured on a 96x96 size patch. The aperture is 64x64, the smallest size we use. The left image shows the pixel arrangement on group layer four. At least one 3x3 region is not directly interfered with by the aperture. However, on the right, layer 5, only one pixel is fully uncovered. All spatial interactions at this layer will involve at least one occluded region. Ideally, this would create some inhibition to layer five forming meaningful spatial associations and perhaps bias towards layer 4 which can. Note that this description is simplified and somewhat imprecise since image information can propagate laterally through consecutive layers.}
\label{fig:aperture}
\end{figure}
We mentioned some (minor) evidence that middle layers in a self-supervised network are being neglected. One approach would be to try and create a bias towards these neurons. In the general five-layer topology, the fourth group layer has a receptive field of 48x48 given a 3x3 filter. In AlexNet, this would include layers conv3, conv4 and conv5. In GoogLeNet, this would include all 4th layers (4a, 4b etc). If we create an aperture, we could create a patch that doesn't cover the extent of the 5th group layers, but does cover the extent of the 4th group layers filters. This could bias against the 5th layers from learning since it cannot see the whole patch. Ideally, this would put emphasis on learning in the 4th layers. See Figure~\ref{fig:aperture} for an example of this.

A random aperture on two of the three patches in a set is created. The idea behind leaving one patch un-apertured is so that we don't completely bias against group layer 5, we still want it to learn. The aperture is square and for each sample is randomly sized between 64x64 and 96x96. The minimum size is 64 since this is the smallest size we can use and guarantee that at least one 3x3 convolution is unobstructed in the fourth layer. The position of the aperture is also randomized but must fit inside the patch so we can never have a viewable area less than 64x64.  The area outside the aperture is filled with ImageNet mean RGB. The size and position of the aperture in two patches is yoked. Which two patches are apertured is randomized for each sample.

\subsubsection{Rotation with Classification (RWC)}

\begin{figure}
\centering
\includegraphics[height=3.1 cm]{RWC_aberation.jpg}
\caption{On the left is an example of the famous thatcher illusion \cite{Thompson80,MTeffect06}. It demonstrates conditional sensitivity to upside-down features in an image against the background. We used this mostly as inspiration. On the left house image \cite{ChromaHouse}, the network can tell that the blue bordered area comes from the upper left corner based on chromatic aberration alone. However, on the right image, rotation with classification makes it tell us if the patch is inverted and comes from the lower right corner. If it uses chromatic aberration as the only cue, it would be wrong 50\% of the time. (Figure is enlarged in appendix: see figure ~\ref{fig:rwc_big})}
\label{fig:rwc}
\end{figure}
Each patch in a patch/context model may simply contain a part of a much larger object. In general, this is the intent of the patch/context approach. Might it help if parts can be understood at different orientations? For instance, if one has seen an upside-down roof top, one may better understand a triangular yield sign or a funnel. Additionally, humans have the ability to \emph{conditionally} recognize upside-down parts embedded in a whole image. This is illustrated by the famous Thatcher illusion \cite{Thompson80} (see Figure~\ref{fig:rwc}). We reason that self-supervised learning might benefit from exposure to upside-down patches, and it would help to make the network identify if patches are right-side-up or upside-down. We do this by flipping the whole image so that all patches are flipped. Then, we double the number of classes by giving each upside-down image its own class. For instance, if we have 20 classes of patch arrangements, when we add upside-down images, we have 40 classes. We also explore 90 and 270 degree rotations. This yields a total of 80 classes.

Forcing the network to classify patches as upside-down also reduces the strength of clues generated by chromatic aberration. Aberration radiates from the center of the image. Without rotating the image, a downward sloping arch of green/magenta to the left indicates the patch comes from the upper left-hand corner. However, in a flipped image, the same pattern indicates the lower right corner instead. By just trying to guess upper left-hand corner from the chromatic aberration pattern, it will be wrong 50\% of the time. With four rotations, it will be wrong 75\% of the time.

\subsubsection{Miscellany}

 We present experiments with a few other tricks which we found helpful to varying degrees. One method is a typical mixture of label preserving transformations \cite{Simard03,Ciresan11,Ciresan12} we are calling the \emph{usual bag of tricks} {\bf (UBT)}. This involves augmentation by randomly mirroring, zooming, and cropping images. The mirroring is simple horizontal flipping and has no special classification, like with RWC, since this would most likely prove confusing to the network. For random zooming, we randomly scaled each input 110x110 patch to between 96x96 to 128x128, and then extract a random 96x96 patch from this. The zoom and crop location is random for each sample, but is yoked between the three input patches in a set.

Borrowed from \cite{Noroozi17}, we take the idea of mixing the method of rescaling during UBT. Each of the three patches in a set is rescaled by one of four randomly chosen rescale techniques (Bilinear, Area, Bicubic or Lanczos). The random selection is \emph{not} yoked between patches. The idea is yet again to make it harder to match low level statistics between patches (trivial solutions). We call this randomization of rescaling methods {\bf (RRM)}.

We also tried varying the learning rate and decay rate of network layers to increase learning of middle layer weights. For instance, one can adjust the first layer to have 70\% the learning rate as the center most layer. We try to linearly increase the learning rate towards a middle layer and then reduce the rate back down. Given the nine layers of a Siamese AlexNet, we would have learning rates \{0.7, 0.8, 0.9, 1.0, 0.9, 0.8, 0.7, 0.6, 0.5\}. Here, conv4 layer has learning rate multiplier 1.0 and the last fully connected layer has 0.5. We call this weight varying {\bf (WV)}.

\subsection{Verification Datasets}

The development and testing of new techniques generally requires fishing for results. As such, one should avoid using the target dataset for testing each new idea. Fishing leads to solutions specialized towards the specific dataset rather than a general solution. For self-supervised learning, test metrics based on PASCAL VOC \cite{Everingham10}, ImageNet \cite{Imagenet09} and CSAIL Places \cite{Places14} are commonly used. Therefore, we test techniques on a few new datasets with a certain amount of overlap, but which possess differences so that we can be more confident in generalization.

For validation, we use a combination of CUB birds \cite{Welinder10} (a fine-grained bird species dataset) and CompCars \cite{Yang15} (a fine-grained car model dataset). We call this combination CUB/CCars (Examples from these sets are in the appendix as figures~\ref{fig:birds} and ~\ref{fig:compcars}). We use these sets by training a network in a self-supervised manner and then apply transfer learning by fine-tuning them for classification. Both data sets are fine grained for their respective class (birds and cars). However, there are major differences between the kinds of features cars and birds have. Additionally, the CUB birds dataset provides an ideal test set for dealing with chromatic aberration. The four keys to identifying birds are size/shape, habitat, behavior and color pattern \cite{Cornell09}. When trying to control chromatic aberration, one may alter the image in a way that negatively affects color processing and thus classification for birds.

\subsection{Alternative Networks}

\begin{figure}
\centering
\includegraphics[height=5.6 cm]{AlexNet_crop.png}
\caption{This is our custom batch normalized triple Siamese AlexNet. It is very similar to \cite{Doersch15}. Each layer has a batch norm layer after it. Notice we have removed LRN \cite{AlexNet} layers.}
\label{fig:alexnet}
\end{figure}
We are interested in generalization of our solutions and portability to other architectures. If we constrain self-supervised learning to mostly being a training protocol, it's easier to train on different networks. As with using many datasets, using many networks also helps to assure that a technique is not network specific, but works well on other designs. We demonstrate results on four different networks. These are (a) standard CaffeNet type AlexNet \cite{AlexNet,Caffe} (b) AlexNet with \emph{batch normalization} {\bf (BN)} \cite{BatchNorm} (figure ~\ref{fig:alexnet}) (c) a ResCeption network \cite{Mundhenk2016} (d) an Inception network with BN \cite{Inceptionv4,BatchNorm}.

\begin{table*}
\begin{center}
\footnotesize
\begin{tabular}{l|lllll|lllll}
\hline\noalign{\smallskip}
\multicolumn{1}{c}{} & \multicolumn{5}{c}{{\bf Accuracy}} & \multicolumn{5}{c}{{\bf Improvement}} \\
\hline\noalign{\smallskip}
{\bf Method} &	{\bf CUB} &	{\bf CCars} &	{\bf Mean} &	{\bf VOC} &	{\bf All} &	{\bf CUB} &	{\bf CCars} &	{\bf Mean} &	{\bf VOC} &	{\bf All}\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
No Pretrain & 	56.20 &	59.13 &	57.67 &	55.12 &	56.82	& -- & -- & -- & -- & --\\
ImageNet Supervised &	74.44 &	85.40 &	79.92 &	72.67\textcolor{myblue}{\textsuperscript{\S}} &	77.50 & -- & -- & -- & -- & --\\\noalign{\smallskip}
\hline
\noalign{\smallskip}
Baseline 2 Patch Protocol &	62.33 &	79.86 &	71.09 &	63.61 &	68.60	& -- & -- & -- & -- & --\\			\noalign{\smallskip}
\hline
\noalign{\smallskip}
add Chroma Blurring {\bf (CB)} &	64.29 &	80.80 &	72.55 &	64.98 &	70.02 &	1.97 &	0.94 &	1.45 &	1.37 &	1.42\\
add Yoked Jitter {\bf (YJ)} &	65.17 &	80.95 &	73.06 &	65.15 &	70.42 &	0.87 &	0.15 &	0.51 &	0.16 &	0.40\\
add 3rd Patch {\bf (TP)} &	65.19 &	81.54 &	73.36 &	65.27 &	70.66 &	0.02 &	0.59 &	0.30 &	0.12 &	0.24\\
add Extra Patch Cfgs. {\bf (EPC)} &	67.07 &	80.50 &	73.79 &	65.67 &	71.08 &	1.89 &	\textcolor{myred}{-1.04} &	0.43 &	0.41 &	0.42\\
add Usual Tricks {\bf (UBT)} &	67.91 &	80.83 &	74.37 &	65.58 &	71.44 &	0.84 &	0.33 &	0.58 &	\textcolor{myred}{-0.10} &	0.35\\
add Rand. Aperture {\bf (RA)} &	68.01 &	82.07 &	75.04 &	66.79 &	72.29 &	0.10 &	1.24 &	0.67 &	1.21 &	0.85\\
add Rotation 180 {\bf (RWC)} &	68.89 &	84.23 &	76.56 &	68.39 &	73.83 &	0.88 &	2.16 &	1.52 &	1.60 &	1.55\\
add Rotation 90, 270 and WV &	69.39 &	84.25 &	76.82 &	68.31 &	73.99 &	0.50 &	0.02 &	0.26 &	\textcolor{myred}{-0.07} &	0.15\\
\hline
\end{tabular}
\end{center}
\caption{
This is a basic ablation showing the effect of adding each method one at a time. The scores for CUB birds (CUB) and CompCars (CCars) are the single class classification accuracy. PASCAL VOC uses mean average precision (mAP). The mean column is for CUB and CCars, but we show a mean of CUB, CCars and VOC as ``All''. VOC is the \emph{Post Hoc} classification results run after the fact to see how well our CUB/CCars surrogate set matches a core self-supervised benchmark test. The baseline two patch protocol uses color dropping and matches the protocol of \cite{Doersch15}. Gains in CUB/CCars appear to correlate with gains in VOC (but not perfectly). The largest gains for both CUB/CCars and VOC are from rotation with classification, chroma blurring and random aperture.  Also notice that the results for CompCars is only one percentage point less than the ImageNet pretrained network. \textcolor{myblue}{\textsuperscript{\S}}The ImageNet pretrain for VOC uses conv1 through conv5. All fully connected (fc) layers are initialized new.}
\label{table:ablation}
\end{table*}
\begin{table}
\begin{center}
\scriptsize
\begin{tabular}{llllll}
\hline\noalign{\smallskip}
{\bf Method} &	{\bf Class.} &	{\bf Det.} &	{\bf C + D} &	{\bf Seg.} &	{\bf All}\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
ImageNet Labels \cite{Imagenet09,AlexNet}	& 79.9	& 56.8	& 68.4	& 48.0	& 61.6\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Jayaraman \cite{Jayaraman15} & --	& 41.7	& --	& --	& --\\
Li \cite{Li16}	& 56.6	& --	& --	& --	& --\\
Misra \cite{Misra16}	& --	& 42.4	& --	& --	& --\\
Owens \cite{Owens16}\textcolor{myblue}{\textsuperscript{\textdagger}}	& 61.3	& --	& --	& --	& --\\
Larsson	\cite{Larsson17} & 65.9	& --	& --	& 38.4	& --\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Agrawal \cite{Agrawal15}\textcolor{myblue}{\textsuperscript{\textdagger}}	& 54.2	& 43.9	& 49.1	& --	& --\\
Gomez \cite{Gomez17}	& 55.7	& 43.0	& 49.4	& --	& --\\
Pathak (Inpainting) \cite{Pathak16}	& 56.5	& 44.5	& 50.5	& 29.7	& 43.6\\
Donahue \cite{Donahue17}\textcolor{myblue}{\textsuperscript{\textdagger}} & 60.1	& 46.9	& 53.5	& 35.2	& 47.4\\
Wang (Video) \cite{Wang15b}\textcolor{myblue}{\textsuperscript{\textdagger}} & 63.1	& 47.4	& 55.3	& --	& --\\
Lee \cite{Lee17}	& 63.8	& 46.9	& 55.4	& --	& --\\
Zhang (Colorizing) \cite{Zhang16}\textcolor{myblue}{\textsuperscript{\textdagger}}	& 65.9	& 46.9	& 56.4	& 35.6	& 49.5\\
Pathak (Move) \cite{Pathak17}	& 61.0	& 52.2	& 56.6	& --	& --\\
Zhang (Split-Brain) \cite{Zhang17}	& 67.1	& 46.7	& 56.9	& 36.0	& 49.9\\
Bojanowski \cite{Bojanowski17}	& 65.3	& 49.4	& 57.4	& --	& --\\
Doersch (Patches)  \cite{Doersch15}\textcolor{myblue}{\textsuperscript{\textdagger}}	& 65.3	& 51.1	& 58.2	& --	& --\\
Noroozi (Counting) \cite{Noroozi17}	& 67.7	& 51.4	& 59.6	& 36.6	& 51.9\\
Noroozi (Puzzle) \cite{Noroozi16b,Noroozi16a}\textcolor{myblue}{\textsuperscript{\textdaggerdbl}}	& 67.6	& \underline{53.2} 		& 60.4	& 37.6	& 52.8\\
Kim \cite{Kim18}\textcolor{myblue}{\textsuperscript{\textbullet}}  & \underline{69.2} & 52.4 & \underline{60.8} & \underline{39.3} & \underline{53.6}\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Doersch (Multi-Task)\textcolor{myblue}{\text{*}} \cite{Doersch17}	& --	& \underline{54.9}	& --	& --	& --\\
Wang (Invariance)\textcolor{myblue}{\text{*}} \cite{Wang17}	& --	& 53.1	& --	& --	& --\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
RWC 180	& 69.0	& 54.9	& 61.9	& 40.4	& 54.8\\
add rotations 90, 270	& 69.5	& 55.5	& 62.5	& {\bf 41.4}	& 55.5\\
add RRM	& {\bf 69.6}	& {\bf 55.8}	& {\bf 62.7}	& 41.2	& {\bf 55.6}\\
\hline
\end{tabular}
\end{center}
\caption{These are classification mAP \cite{Krahenbuhl16}, detection mAP \cite{Girshick15} and segmentation mIU \cite{Long15} test results over PASCAL VOC \cite{Everingham10}. Mean scores are shown for classification + detection \emph{(C + D)} as well as for all three if the segmentation score is available \emph{(All)}. The bottom three results are ours and include all methods except for WV. These are: CB, YJ, TP, EPC, UBT, RA and RWC. RWC (four rotations) gives the best results, but adding in RRM yields only slightly better results. \textcolor{myblue}{\textsuperscript{\textdagger}}To conserve space, we have taken the largest of two scores when network weights have been rescaled \cite{Krahenbuhl16}.  \textcolor{myblue}{\text{*}}Denotes that this is an estimate for the score based on a very recent result with a different network other than AlexNet. The estimate is computed by adding the gain reported in the work to a mutual baseline method that has an AlexNet result and also appears in our table (namely \cite{Doersch15}).  \textcolor{myblue}{\textsuperscript{\textbullet}}Results were published while this paper was under review. \textcolor{myblue}{\textsuperscript{\textdaggerdbl}}Using corrected results from ArXiv paper, not ECCV.}
\label{table:pascal_voc_class}
\end{table}

\begin{table}
\begin{center}
\scriptsize
\begin{tabular}{lllllll}
\hline\noalign{\smallskip}
{\bf Method} &	{\bf C1} &	{\bf C2} &	{\bf C3} &	{\bf C4} &	{\bf C5} &	{\bf Best}\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
ImageNet \cite{Imagenet09,AlexNet,Zhang16} &	19.3 &	36.3 &	44.2 &	48.3 &	50.5 &	50.5\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Random	\cite{Noroozi17} & 11.6 &	17.1 &	16.9 &	16.3 &	14.1 &	17.1\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Pathak	(Inpainting) \cite{Pathak16} & 14.1 &	20.7 &	21.0 &	19.8 &	15.5 &	21.0\\
Donahue	\cite{Donahue17} & 17.7 &	24.5 &	31.0 &	29.9 &	28.0 &	31.0\\
Doersch	(Patches) \cite{Doersch15} & 16.2 &	23.3 &	30.2 &	31.7 &	29.6 &	31.7\\
Zhang  (Colorizing) \cite{Zhang16} &	13.1 &	24.8 &	31.0 &	32.6 &	32.6 &	32.6\\
Noroozi  (Puzzle) \cite{Noroozi16b,Noroozi16a} &	\underline{18.2} &	28.8 &	34.0 &	33.9 &	27.1 &	34.0\\
Noroozi  (Counting) \cite{Noroozi17} &	18.0 &	\underline{30.6} &	34.3 &	32.5 &	25.7 &	34.3\\
Kim \cite{Kim18} & 14.5 & 27.2 & 32.8 & 34.3 & \underline{32.9} & 34.3\\
Zhang (Split-Brain) \cite{Zhang17} &	17.7 &	29.3 &	\underline{35.4} &	\underline{35.2} &	32.8 &	\underline{35.4}\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
RWC 180 &	19.4 &	31.2 &	36.7 &	37.1 &	32.8 &	37.1\\
add rotations 90, 270 &	19.5 &	31.6 &	37.1 &	37.7 &	{\bf 33.7} &	37.7\\
add RRM &	{\bf 19.6} &	{\bf 31.8} &	{\bf 37.6} &	{\bf 37.8} &	33.4 &	{\bf 37.8}\\
\hline
\end{tabular}
\end{center}
\caption{This is the linear test for ImageNet data \cite{Imagenet09}. The network is fine-tuned up to the convolution layer shown. Our results are the bottom three rows. These are the same three self-supervised conditions used in table~\ref{table:pascal_voc_class}. These use all the methods we have presented except for WV.  The maximum score is shown in bold with the previous best result underlined. Rotation with Classification using 90, 180 and 270 degree rotations is generally the best performer.  Here, RRM edges out the other two by a small margin.}
\label{table:linear_imagenet}
\end{table}

\begin{table}
\begin{center}
\scriptsize
\begin{tabular}{lllllll}
\hline\noalign{\smallskip}
{\bf Method} &	{\bf C1} &	{\bf C2} &	{\bf C3} &	{\bf C4} &	{\bf C5} &	{\bf Best}\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Places \cite{Places14,Zhang16} &	22.1 &	35.1 &	40.2 &	43.3 &	44.6 &	44.6\\
ImageNet \cite{Imagenet09,AlexNet,Zhang16} & 22.7 &	34.8 &	38.4 &	39.4 &	38.7 &	39.4\\
Random \cite{Noroozi17} & 15.7 &	20.3 &	19.8 &	19.1 &	17.5 &	20.3\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Pathak (Inpainting) \cite{Pathak16} 	& 18.2 &	23.2 &	23.4 &	21.9 &	18.4 &	23.4\\
Wang (Video) \cite{Wang15b} &	20.1 &	28.5 &	29.9 &	29.7 &	27.9 &	29.9\\
Zhang (Colorizing) \cite{Zhang16} &	16.0 &	25.7 &	29.6 &	30.3 &	29.7 &	30.3\\
Donahue	\cite{Donahue17} & 22.0 &	28.7 &	31.8 &	31.3 &	29.7 &	31.8\\
Owens	\cite{Owens16}  & 19.9 &	29.3 &	32.1 &	28.8 &	29.8 &	32.1\\
Doersch	(Patches) \cite{Doersch15}  & 19.7 &	26.7 &	31.9 &	32.7 &	30.9 &	32.7\\
Zhang (Split-Brain) \cite{Zhang17} &	21.3 &	30.7 &	34.0 &	34.1 &	\underline{32.5} &	34.1\\
Noroozi (Puzzle) \cite{Noroozi16b,Noroozi16a} & 	23.0 &	31.9 &	35.0 &	34.2 &	29.3 &	35.0\\
Noroozi (Counting) \cite{Noroozi17} &	\underline{23.3} &	\underline{33.9} &	\underline{36.3} &	\underline{34.7} &	29.6 &	\underline{36.3}\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
RWC 180	 & {\bf 23.7} &	33.9 &	37.1 &	{\bf 37.2} &	34.1 &	{\bf 37.2}\\
add rotations 90, 270	 & 23.5 &	34.0 &	{\bf 37.2} &	{\bf 37.2} &	{\bf 34.9} &	{\bf 37.2}\\
add RRM	 & 23.5 &	{\bf 34.2} &	{\bf 37.2} &	37.0 &	34.4 &	{\bf 37.2}\\
\hline
\end{tabular}
\end{center}
\caption{This is the linear test for CSAIL Places \cite{Places14} data. The network is fine-tuned up to the convolution layer shown. Our results are the bottom three rows. These are the same three self-supervised conditions used in table~\ref{table:pascal_voc_class}. These all use the methods we have presented except for WV.  The maximum score is shown in bold with the previous best underlined. RWC (four rotations) is slightly better, but all three variations obtain the same max score.}
\label{table:linear_places}
\end{table}

\section{Experiments}

We perform a variety of experiments. We show the ablation gain of each tool on our CUB/CCars dataset combination, and also \emph{post hoc} on VOC classification.

\subsection{Self-supervised Training}

We use triple Siamese networks which share weights between branches, and are then concatenated together and run through a few fully connected layers. The input is a set of three 96x96 RGB image patches processed from 110x110 patches, taken from the ImageNet training dataset. Recall that we apply the chroma blur operation offline before we train to remove the expense of repeated Lab conversion and blurring. The output is a softmax classification for the patch arrangement pattern class with 20, 40 or 80 classes. All networks load in a list of shuffled training and testing patches. The list is reshuffled after each epoch.

We use a slightly different protocol for training the batch normalized networks than for training the non-normalized CaffeNet. The batch normalized networks train with stochastic gradient descent (SGD) for 750k iterations with a batch size of 128 and an initial learning rate of 0.01. A step protocol is used with a size of 300k and gamma 0.1. Momentum is 0.9 and weight decay is 0.0002. For our CaffeNet, we use a Google exponential style training \cite{Inceptionv4}. We train for 1.5 million iterations with a batch size of 128 and initial learning rate of 0.00666 (the fastest rate seemingly stable). We train SGD with a step size of 10k and a gamma of 0.96806. Momentum is 0.9 and weight decay is 0.0002.

\subsection{Validation and Ablation on CUB Birds and CompCars}

The bulk of our testing and validation was carried out by fine-tuning a self-supervise trained network to the CUB/CCars datasets. Both sets were split \emph{a priori} into training and testing sets by the authors. We use provided bounding boxes from both sets to pre-crop the images. Some further details can be seen in the appendix \ref{compcars_appendix}.

We perform most ablation and validation experiments on our custom batch normalized AlexNet which can be seen in figure ~\ref{fig:alexnet}. The target network is similar to \cite{Doersch15} in that we use the same conv6 and conv6b layers, but we do not try to transfer these layers from the self-supervise trained network. We kept these layers mostly for diversity, so that our batch normalized AlexNet is somewhat different from the very standard CaffeNet/AlexNet we perform benchmark tests on. Again, generalization is important to us. We self-supervise train, then transfer the weights of the five convolution and batch norm layers to the non-Siamese network and initialize new fully connected layers. Both CUB and CCars are trained the same way. The methods for training both had been established \emph{a priori} to avoid over-tuning of hyperparameters. For fine-tuning, we use a polynomial learning policy with an initial learning rate of 0.01 with SGD for 100k iterations with a batch size of 64. Polynomial power is 0.5. Momentum is 0.9 and weight decay is 0.0002. For each condition we wished to test, we trained three times and took the average testing accuracy to reduce minor variation within condition results.

Ablation results for each method can be seen in Table~\ref{table:ablation}. We show \emph{post hoc} results from PASCAL VOC 2007 classification on the same network and condition to see how well our validation set results map to one of our target data sets. VOC was trained by the standard classification method described in \cite{Krahenbuhl16} and results are in mean average precision (mAP). Finer details on ablation and \underline{more experiments} can be seen in the \emph{appendix} \ref{ablation_appendix}.

\subsection{Standard Transfer Learning Testing Battery}

We demonstrate how the results we have obtained compared with self-supervised methods using a suite of standard benchmark tests. These include classification \cite{Krahenbuhl16}  and detection \cite{Girshick15} on PASCAL VOC 2007, and segmentation \cite{Long15,Pathak16} on PASCAL VOC 2012. They also include the ``linear classifier'' tests on CSAIL places and ImageNet \cite{Zhang16}. We note two possible differences from the standard benchmark methodology here. For detection, we use multiscale training and testing. This is common and used by \cite{Pathak17,Noroozi16a,Noroozi16b,Doersch15,Doersch17}, but not all authors use it. For segmentation, most authors use surgery to map trained fully connected layers to convolution layers six and seven. Our trained network does not have the correct number of weights in the fully connected layers to do this. So we only copy convolution layers one through five and initialize layers six and seven randomly.

For these tests, we self-supervise train a triple Siamese CaffeNet type AlexNet using the non-batch normalized protocol previously described. Inputs are padded by 5 pixels, but \emph{only} during the self-supervised triple network training. No batch normalization is used at any stage of training. After pooling layer 5, we use the same Siamese structure as our custom batch normalized AlexNet. We leave out batch normalization in these layers, but insert dropout layers after joined\_fc1 and joined\_fc2 with a dropout ratio of 0.5. Convolution weights from layers one through five are transferred to a completely off the shelf CaffeNet.  Training and testing are performed in the standard way defined by the authors of each test (with the two noted differences). Results can be seen in tables~\ref{table:pascal_voc_class}, ~\ref{table:linear_imagenet} and ~\ref{table:linear_places}. Our improvements yield results that out-perform all other methods on all of the standard benchmark tests.

\subsection{Portability to Other Networks}
\begin{table}
\begin{center}
\scriptsize
\begin{tabular}{llll}
\hline\noalign{\smallskip}
{\bf Method} &	{\bf AlexNet BN} &	{\bf ResCeption} &	{\bf Inception 21k}\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
ImageNet Pretrain &	79.92 &	88.62 &	89.01\\
add Rotations 180 (RWC) &	76.56 &	86.37 &	85.20\\
add rotations 90, 270 (RWC) &	76.82 &	86.52 &	85.81\\
Diff from ImageNet &	3.10 &	2.10 &	3.20\\
\hline
\end{tabular}
\end{center}
\caption{These are the mean results for CUB and CompCars on the different networks.}
\label{table:networks}
\end{table}
We trained on two more networks to demonstrate portability and generalization. The first new network, ResCeption \cite{Mundhenk2016} is a GoogLeNet \cite{GoogLeNet} like network with batch normalization (BN) \cite{BatchNorm} and residual short-cutting \cite{ResNet}. It has 5x5 convolutions in group layer 5 which extend beyond the self-supervised receptive layer. So we self-supervise trained by replacing these with 1x1 surrogate filters that cannot train. Then we put freshly initialized 5x5 convolutions back in place for this layer when we fine-tuned.

We also used a standard inception network with BN. The ImageNet pre-train was performed by \cite{Imagenet21k} on the full set of 21k ImageNet labels. The network required no augmentation. All weights are copied from self-supervised training except for the very top fully connected layer which would be discarded anyway.  Table ~\ref{table:networks} shows the results from these new networks with our BN AlexNet. CompCars results tend to be within about one to two percentage point of ImageNet supervised training. However, CUB runs from three to six.  The results from self-supervision seem enticingly close to ImageNet supervised, but are not yet there.

\section*{Acknowledgments}
The authors would like to thank several people who contributed ideas and conversation for this work: Carmen Carrano, Albert Chu, Alexei Efros, Gerald Friedland, Will Grathwohl, Brenda Ng, Doug Poland, Richard Zhang, Miles the Cat and the CVPR reviewers. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 and was supported by the LLNL-LDRD Program under Project No. 17-SI-003. Support was also provided by the LLNL DSSI Summer Institute.

{\small
\bibliographystyle{ieee}