cs.CL cs.LG cs.NE
Memory Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages. We introduce three memory augmented Recurrent Neural Networks (MARNNs) and explore their capabilities on a series of simple language modeling tasks whose solutions require stack based mechanisms. We provide the first demonstration of neural networks recognizing the generalized Dyck languages, which express the core of what it means to be a language with hierarchical structure. Our memory augmented architectures are easy to train in an end to end fashion and can learn the Dyck languages over as many as six parenthesis pairs, in addition to two deterministic palindrome languages and the string reversal transduction task, by emulating pushdown automata. Our experiments highlight the increased modeling capacity of memory augmented models over simple RNNs, while inflecting our understanding of the limitations of these models.