File size: 2,872 Bytes
be2a97b bd051c5 21ecfe7 bd051c5 61ba441 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 |
---
datasets:
- yahma/alpaca-cleaned
base_model:
- meta-llama/Llama-3.1-8B
library_name: adapter-transformers
---
---
# Meta Llama 3.1 8B - Alpaca
This model card describes the LLaMA 3 8B Alpaca model fine-tuned for instruction-following and chat tasks. It has been optimized for fast and accurate text generation using LoRA and bf16 precision.
## Model Details
### Model Description
This model is a fine-tuned version of LLaMA 3 8B using Alpaca-style instruction-following data. It is designed for natural language understanding and generation tasks, supporting conversational AI and other NLP applications.
- **Developed by:** Anezatra
- **Shared by:** HuggingFace Community
- **Model type:** Transformer-based Language Model (Instruction-Finetuned)
- **Language(s) (NLP):** English (with potential for multilingual input via translation)
- **License:** Apache 2.0
- **Finetuned from model:** LLaMA 3 8B
### Model Sources
- **Repository:** https://huggingface.co/unsloth/meta-llama-3.1-8b-alpaca
- **Paper:** https://arxiv.org/abs/2302.13971 (LLaMA 3)
- **Demo:** [Example usage with llama-cli]
## Training Details
### Training Data
- Instruction-following datasets (Alpaca-style(Cleaned))
### Training Procedure
- Fine-tuned using LoRA
- **Epochs:** 1
- **Steps:** 120
- **Learning rate:** 2e-4
- **Batch size:** 6 per GPU
- **Gradient accumulation:** 5 steps
- **Precision:** bf16 mixed precision
### Preprocessing
- Text normalization: trimming, punctuation correction, lowercasing
- Tokenization using LLaMA tokenizer
- Merging conversational history for context-aware generation
### Speeds, Sizes, Times
- Model size (8B parameters) ~ 16 GB in bf16
- Q4_K_M quantized GGUF ~ 4.9 GB
- Training completed in under 24 hours on single GPU with LoRA
## Evaluation
### Testing Data, Factors & Metrics
- Evaluated on held-out instruction-following tasks
- Metrics: Perplexity, accuracy on factual Q&A, and BLEU for sequence generation
- Human evaluation for conversational coherence
### Results
- Perplexity: ~1.35 on validation set
- Maintains context across multiple turns in dialogue
- Generates coherent and instruction-following responses
## Environmental Impact
- **Hardware Type:** NVIDIA L4 24GB GPU
- **Hours used:** -
## Technical Specifications
### Model Architecture and Objective
- Transformer decoder-only architecture
- 8B parameters, 32 layers, 32 attention heads
- Optimized for instruction-following tasks and conversational AI
### Compute Infrastructure
#### Hardware
- Single 24GB L4 GPU
#### Software
- PyTorch + Transformers + Unsloth LoRA integration
- llama.cpp for GGUF inference on CPU
## Citation
**BibTeX:**
```
@misc{meta-llama-3.1-8b-alpaca,
title={LLaMA 3 8B Alpaca Fine-Tuned Model},
author={Anezatra},
year={2025},
howpublished={\url{https://huggingface.co/unsloth/meta-llama-3.1-8b-alpaca}}
}
``` |