bf16=True in TrainingArgument

gptzerozero · July 2, 2023, 4:15pm

Is the following code in the official Qlora repo

artidoro/qlora/blob/3da535abdfaa29a2d0757eab0971664ed2cd97e8/qlora.py#L334-L343


      
          for name, module in model.named_modules():
              if isinstance(module, LoraLayer):
                  if args.bf16:
                      module = module.to(torch.bfloat16)
              if 'norm' in name:
                  module = module.to(torch.float32)
              if 'lm_head' in name or 'embed_tokens' in name:
                  if hasattr(module, 'weight'):
                      if args.bf16 and module.weight.dtype == torch.float32:
                          module = module.to(torch.bfloat16)

similar to passing bf16=True into transformers.TrainingArgument when using transformers.TrainingArgument with transformers.Trainer?

Topic		Replies	Views
Using Quantization with fp16/bf16 Trainer flag 🤗Transformers	0	754	February 14, 2024
Bitsandbytes quantization and QLORA fine-tuning 🤗Transformers	1	346	November 5, 2024
GPTQ model to bfloat16 🤗Transformers	0	454	January 10, 2024
The role of the bf16 arguments in SFTConfig 🤗Transformers	0	557	July 25, 2024
Llama2 torch_dtype Models	0	315	November 20, 2023

bf16=True in TrainingArgument

Related topics