cs.CL cs.LG
Train, Sort, Explain Learning to Diagnose Translation Models. Evaluating translation models is a trade off between effort and detail. On the one end of the spectrum there are automatic count based methods such as BLEU, on the other end linguistic evaluations by humans, which arguably are more informative but also require a disproportionately high effort. To narrow the spectrum, we propose a general approach on how to automatically expose systematic differences between human and machine translations to human experts. Inspired by adversarial settings, we train a neural text classifier to distinguish human from machine translations. A classifier that performs and generalizes well after training should recognize systematic differences between the two classes, which we uncover with neural explainability methods. Our proof of concept implementation, DiaMaT, is open source. Applied to a dataset translated by a state of the art neural Transformer model, DiaMaT achieves a classification accuracy of 75 and exposes meaningful differences between humans and the Transformer, amidst the current discussion about human parity.