Wav2vec2 xlsr nan train loss

Models
1

Hi,
I’m running into nan training_loss when training wav2vec2 xlsr with my custom dataset.
Weird thing is that even though training_loss goes to nan, eval_loss still goes down, and error_rate (cer and wer) also goes down.
I’ve experimented with lower learning_rate, but still getting similar behavior. I’m logging with wandb.

My graphs look like the following:

lr
lr2400×1200 118 KB

error_rate
error_rate2400×1200 139 KB

loss
loss2400×1200 122 KB

There’s no value for train/loss after ~60 steps since it is nan, but eval/loss is still decreasing.

Has anyone experienced similar behavior?

2

I’ve let it train over the weekend, still NAN train loss, but eval loss and both WER and CER continue to decrease