---
pipeline_tag: text-generation
inference: false
license: apache-2.0
library_name: transformers
tags:
- language
- aquif
- gpt2
- text-generation-inference
- math
- coding
- small
language:
- en
datasets:
- tatsu-lab/alpaca
- databricks/databricks-dolly-15k
- OpenAssistant/oasst1
---

# aquif-neo-2-345m-c1

This is the first checkpoint of the 'aquif-neo-2-345m' model, a next-generation language model developed by aquif AI. This checkpoint is fine-tuned on a diverse dataset including conversational, code, and math data, serving as the initial step in a 5-checkpoint training process designed to create a versatile and capable model.

## Model Details

**Base Model**: gpt2-medium\
**Method**: LoRA (Low-Rank Adaptation)\
**Parameter Count**: 355 million params\

## Training Information

This checkpoint was trained as the first stage of a multi-checkpoint process. The training was performed using a network-resilient script that includes fallback mechanisms for data loading and model initialization.

Checkpoint Number: 1/5\
Hardware: Trained on a Google Colab T4 GPU.\
Training Duration: Approximately 2.5 hours for this checkpoint.\
Training Framework: PyTorch, Hugging Face Transformers, PEFT, bitsandbytes, TRL.\
Quantization: 8-bit.\

## LoRA Configuration:
r=8\
lora_alpha=16\
target_modules: ["q_attn", "c_attn", "c_proj", "c_fc", "attn.c_attn", "attn.c_proj", "mlp.c_fc", "mlp.c_proj"]\
lora_dropout=0.05\
bias="none"\
task_type="CAUSAL_LM"\
Training Arguments:\
per_device_train_batch_size=2\
gradient_accumulation_steps=16\
num_train_epochs=1 (for this checkpoint)\
learning_rate=1e-5\
max_steps=400\
\
*Optimized for 8-bit training.*

## Training Loss Data

The following table shows the training loss recorded during the training of this checkpoint:\

| Step | Training Loss |
|------|---------------|
| 20   | 3.4444        |
| 40   | 3.4754        |
| 60   | 3.4954        |
| 80   | 3.4213        |
| 100  | 3.3338        |
| 120  | 3.1749        |
| 140  | 3.2208        |
| 160  | 3.0503        |
| 180  | 2.9293        |
| 200  | 2.8377        |
| 220  | 2.8094        |
| 240  | 2.7225        |
| 260  | 2.6260        |
| 280  | 2.7452        |
| 300  | 2.6614        |
| 320  | 2.5056        |
| 340  | 2.5391        |
| 360  | 2.5115        |
| 380  | 2.4892        |
| 400  | 2.5117        |

*Note: Training loss is a metric that indicates how well the model is learning. A decreasing loss generally suggests improvement.*\

## Intended Use
This checkpoint is an intermediate model in the development of the full 'aquif-neo-2'. It is not intended for production use but serves as a foundation for subsequent fine-tuning checkpoints focusing on specific domains and tasks.

## How to Load the Model

You can load this model using the Hugging Face 'transformers' library:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "aquiffoo/aquif-neo-2-345m-c1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
```


## Future Checkpoints
This is the first of 5 planned checkpoints. Future checkpoints will continue to fine-tune the model on additional data to improve its capabilities across various domains.
\
**License**: Apache 2.0