cs.CL
Pre training Text Representations as Meta Learning. Pre training text representations has recently been shown to significantly improve the state of the art in many natural language processing tasks. The central goal of pre training is to learn text representations that are useful for subsequent tasks. However, existing approaches are optimized by minimizing a proxy objective, such as the negative log likelihood of language modeling. In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi task pre training and model agnostic meta learning with a sequence of meta train steps. The standard multi task learning objective adopted in BERT is a special case of our learning algorithm where the depth of meta train is zero. We study the problem in two settings unsupervised pre training and supervised pre training with different pre training objects to verify the generality of our approach.Experimental results show that our algorithm brings improvements and learns better initializations for a variety of downstream tasks.