Error when training Llama-3.2-3B-Instruct-Q5_K_L.gguf with Lora

Beginners
1

when i load Llama-3.2-3B-Instruct-Q5_K_L.gguf with llama.cpp, it succeeded, but when i try to train the model with data of mine, it failed, log as following:

18:35:21-574250 WARNING LoRA training has only currently been validated for

                     LLaMA, OPT, GPT-J, and GPT-NeoX models. **(**Found model   

                     type: LlamaServer**)**

18:35:26-595624 INFO Loading JSON datasets

Map: 0%| | 0/3325 [00:00<?, ? examples/s]

Traceback (most recent call last):

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/gradio/queueing.py”, line 580, in process_events

response = await route_utils.call_process_api(

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/gradio/route_utils.py”, line 276, in call_process_api

output = await app.get_blocks().process_api(

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/gradio/blocks.py”, line 1928, in process_api

result = await self.call_function(

         ^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/gradio/blocks.py”, line 1526, in call_function

prediction = await utils.async_iteration(iterator)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/gradio/utils.py”, line 657, in async_iteration

return await iterator.\__anext_\_()

       ^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/gradio/utils.py”, line 650, in _anext_

return await anyio.to_thread.run_sync(

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/anyio/to_thread.py”, line 56, in run_sync

return await get_async_backend().run_sync_in_worker_thread(

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/anyio/_backends/_asyncio.py”, line 2476, in run_sync_in_worker_thread

return await future

       ^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/anyio/_backends/_asyncio.py”, line 967, in run

result = context.run(func, \*args)

         ^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/gradio/utils.py”, line 633, in run_sync_iterator_async

return next(iterator)

       ^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/gradio/utils.py”, line 816, in gen_wrapper

response = next(iterator)

           ^^^^^^^^^^^^^^

File “/Users/Peter/text-generation-webui/modules/training.py”, line 486, in do_train

train_data = data\['train'\].map(generate_and_tokenize_prompt, new_fingerprint='%030x' % random.randrange(16\*\*30))

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/datasets/arrow_dataset.py”, line 560, in wrapper

out: Union\["Dataset", "DatasetDict"\] = func(self, \*args, \*\*kwargs)

                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/datasets/arrow_dataset.py”, line 3318, in map

for rank, done, content in Dataset.\_map_single(\*\*unprocessed_kwargs):

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/datasets/arrow_dataset.py”, line 3650, in _map_single

for i, example in iter_outputs(shard_iterable):

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/datasets/arrow_dataset.py”, line 3624, in iter_outputs

yield i, apply_function(example, i, offset=offset)

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/opt/anaconda3/envs/textgen/lib/python3.11/site-packages/datasets/arrow_dataset.py”, line 3547, in apply_function

processed_inputs = function(\*fn_args, \*additional_args, \*\*fn_kwargs)

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/text-generation-webui/modules/training.py”, line 482, in generate_and_tokenize_prompt

return tokenize(prompt, add_eos_token)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/text-generation-webui/modules/training.py”, line 367, in tokenize

input_ids = encode(prompt, True)

            ^^^^^^^^^^^^^^^^^^^^

File “/Users/Peter/text-generation-webui/modules/training.py”, line 357, in encode

if len(result) >= 2 and result\[:2\] == \[shared.tokenizer.bos_token_id, shared.tokenizer.bos_token_id\]:

                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

AttributeError: ‘LlamaServer’ object has no attribute ‘bos_token_id’

1 Like
2

GGUF is not usually well suited for fine tuning. There seems to be a way to decode GGUF and train directly, but I think your software doesn’t supports it.

Personally, I think it’s faster to find the Transoformers format weights before converted to GGUF and train using those in general.