Can I use LoRA with jhu-clsp/ettin-encoder-1b?

Beginners
1

It looks like jhu-clsp/ettin-encoder-1b does not have any proj layers. Is it possible to use LoRA with this model:

from transformers import AutoModelForSequenceClassification
model_name = ‘jhu-clsp/ettin-encoder-1b’
model = AutoModelForSequenceClassification.from_pretrained(model_name)
for parent_name, module in model.named_modules():
    for child_name, child in module.named_children():
        if ‘proj’ in child_name:
            print(child_name)
            print(“_________”)

This code returned nothing.

1 Like
2

It seems that for ModernBERT-based models, the target_modules names aren’t proj*. You can apparently also automatically select the target_modules using ="all-linear".

  "target_modules": [
    "Wqkv",
    "Wi",
    "Wo"
  ],
1 Like
3

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.