I would like to pretrain and then fine tune on longT5 on a custom dataset. Ideally, I would train a tokenizer on my data, then merge tokenizers with longT5 tokenizer and then pretrain from the published longT5 checkpoint. has anyone tried this or is aware of any good resources?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Any reference on how to train (unsupervised learning) longformer model from scratch (not fine-tuning) using Tensorflow? | 0 | 984 | February 25, 2021 | |
| Fine tune LongT5 mdoel | 4 | 946 | December 15, 2022 | |
| Example of how to pretrain T5? | 15 | 16241 | March 16, 2023 | |
| Use LongT5 model with T5 checkpoint | 0 | 365 | January 16, 2023 | |
| Longt5 summarization using huggingface sample code | 1 | 870 | July 8, 2022 |