Can run_clm.py do early stopping?

Intermediate
1

I am using run_clm.py to fine tune a pretrained model at Huggingface. I want fine tuning to stop when eval loss stops decreasing. Is it possible to get early stop behavior with run_clm? Thanks.

2

Hey, you can tweak run_clm a bit , first import these

from transformers import Trainer, TrainingArguments, EarlyStoppingCallback, IntervalStrategy

and add these in TrainingArguments

training_args = TrainingArguments(...)
  1. Use load_best_model_at_end = True (EarlyStoppingCallback() requires this to be True).
  2. evaluation_strategy = 'steps'
  3. eval_steps = N (evaluate the loss after N steps).

After use

trainer = Trainer(
    model,
    ...
    callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]
)

You can change the patience here, hope it helps

3

Thank you very much Muhtasham Oblokulov!

1 Like