---
language:
- en
license: apache-2.0
tags:
- sentence-transformers
- sparse-encoder
- sparse
- splade
- generated_from_trainer
- dataset_size:630000
- loss:SpladeLoss
- loss:SparseMultipleNegativesRankingLoss
- loss:FlopsLoss
base_model: drexalt/NeoBERT-RetroMAE-pretrain
widget:
- text: placeholder
datasets:
- lightonai/ms-marco-en-bge-gemma
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
- dot_accuracy@1
- dot_accuracy@3
- dot_accuracy@5
- dot_accuracy@10
- dot_precision@1
- dot_precision@3
- dot_precision@5
- dot_precision@10
- dot_recall@1
- dot_recall@3
- dot_recall@5
- dot_recall@10
- dot_ndcg@10
- dot_mrr@10
- dot_map@100
- query_active_dims
- query_sparsity_ratio
- corpus_active_dims
- corpus_sparsity_ratio
- avg_flops
model-index:
- name: splade-NeoBERT-RetroMAE-pretrain trained on LightOn MS MARCO (triplets)
results:
- task:
type: sparse-information-retrieval
name: Sparse Information Retrieval
dataset:
name: NanoSciFact
type: NanoSciFact
metrics:
- type: dot_accuracy@1
value: 0.6
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.74
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.82
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.86
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.6
name: Dot Precision@1
- type: dot_precision@3
value: 0.26
name: Dot Precision@3
- type: dot_precision@5
value: 0.176
name: Dot Precision@5
- type: dot_precision@10
value: 0.09399999999999999
name: Dot Precision@10
- type: dot_recall@1
value: 0.575
name: Dot Recall@1
- type: dot_recall@3
value: 0.705
name: Dot Recall@3
- type: dot_recall@5
value: 0.795
name: Dot Recall@5
- type: dot_recall@10
value: 0.84
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.7148834371587105
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.6839999999999999
name: Dot Mrr@10
- type: dot_map@100
value: 0.6735399659066178
name: Dot Map@100
- type: query_active_dims
value: 72.83999633789062
name: Query Active Dims
- type: query_sparsity_ratio
value: 0.997613524790712
name: Query Sparsity Ratio
- type: corpus_active_dims
value: 197.2052001953125
name: Corpus Active Dims
- type: corpus_sparsity_ratio
value: 0.9935389161852004
name: Corpus Sparsity Ratio
- type: avg_flops
value: 8.401775360107422
name: Avg Flops
- task:
type: sparse-information-retrieval
name: Sparse Information Retrieval
dataset:
name: NanoMSMARCO
type: NanoMSMARCO
metrics:
- type: dot_accuracy@1
value: 0.46
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.66
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.74
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.86
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.46
name: Dot Precision@1
- type: dot_precision@3
value: 0.22
name: Dot Precision@3
- type: dot_precision@5
value: 0.14800000000000002
name: Dot Precision@5
- type: dot_precision@10
value: 0.08599999999999998
name: Dot Precision@10
- type: dot_recall@1
value: 0.46
name: Dot Recall@1
- type: dot_recall@3
value: 0.66
name: Dot Recall@3
- type: dot_recall@5
value: 0.74
name: Dot Recall@5
- type: dot_recall@10
value: 0.86
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.6431962437851808
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.5753809523809524
name: Dot Mrr@10
- type: dot_map@100
value: 0.5824329734592893
name: Dot Map@100
- type: query_active_dims
value: 25.31999969482422
name: Query Active Dims
- type: query_sparsity_ratio
value: 0.9991704344507298
name: Query Sparsity Ratio
- type: corpus_active_dims
value: 169.36923217773438
name: Corpus Active Dims
- type: corpus_sparsity_ratio
value: 0.9944509130405039
name: Corpus Sparsity Ratio
- type: avg_flops
value: 1.7089946269989014
name: Avg Flops
- task:
type: sparse-information-retrieval
name: Sparse Information Retrieval
dataset:
name: NanoTouche2020
type: NanoTouche2020
metrics:
- type: dot_accuracy@1
value: 0.6938775510204082
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.8979591836734694
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.9387755102040817
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.9795918367346939
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.6938775510204082
name: Dot Precision@1
- type: dot_precision@3
value: 0.6462585034013605
name: Dot Precision@3
- type: dot_precision@5
value: 0.6000000000000001
name: Dot Precision@5
- type: dot_precision@10
value: 0.5224489795918368
name: Dot Precision@10
- type: dot_recall@1
value: 0.04804845439858598
name: Dot Recall@1
- type: dot_recall@3
value: 0.1277026155093339
name: Dot Recall@3
- type: dot_recall@5
value: 0.20012022654858883
name: Dot Recall@5
- type: dot_recall@10
value: 0.33551454872997677
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.5801862770479804
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.799546485260771
name: Dot Mrr@10
- type: dot_map@100
value: 0.40458590167194947
name: Dot Map@100
- type: query_active_dims
value: 19.836734771728516
name: Query Active Dims
- type: query_sparsity_ratio
value: 0.9993500840452222
name: Query Sparsity Ratio
- type: corpus_active_dims
value: 159.20904541015625
name: Corpus Active Dims
- type: corpus_sparsity_ratio
value: 0.9947837938074124
name: Corpus Sparsity Ratio
- type: avg_flops
value: 2.905405044555664
name: Avg Flops
- task:
type: sparse-information-retrieval
name: Sparse Information Retrieval
dataset:
name: NanoSCIDOCS
type: NanoSCIDOCS
metrics:
- type: dot_accuracy@1
value: 0.4
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.6
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.7
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.8
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.4
name: Dot Precision@1
- type: dot_precision@3
value: 0.29333333333333333
name: Dot Precision@3
- type: dot_precision@5
value: 0.24
name: Dot Precision@5
- type: dot_precision@10
value: 0.172
name: Dot Precision@10
- type: dot_recall@1
value: 0.08366666666666667
name: Dot Recall@1
- type: dot_recall@3
value: 0.18166666666666664
name: Dot Recall@3
- type: dot_recall@5
value: 0.24566666666666667
name: Dot Recall@5
- type: dot_recall@10
value: 0.3526666666666667
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.33612398316083025
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.5252698412698412
name: Dot Mrr@10
- type: dot_map@100
value: 0.2530288256484673
name: Dot Map@100
- type: query_active_dims
value: 28.65999984741211
name: Query Active Dims
- type: query_sparsity_ratio
value: 0.9990610051815932
name: Query Sparsity Ratio
- type: corpus_active_dims
value: 212.24008178710938
name: Corpus Active Dims
- type: corpus_sparsity_ratio
value: 0.9930463245597566
name: Corpus Sparsity Ratio
- type: avg_flops
value: 4.049935817718506
name: Avg Flops
- task:
type: sparse-information-retrieval
name: Sparse Information Retrieval
dataset:
name: NanoNFCorpus
type: NanoNFCorpus
metrics:
- type: dot_accuracy@1
value: 0.46
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.58
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.62
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.68
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.46
name: Dot Precision@1
- type: dot_precision@3
value: 0.38
name: Dot Precision@3
- type: dot_precision@5
value: 0.324
name: Dot Precision@5
- type: dot_precision@10
value: 0.276
name: Dot Precision@10
- type: dot_recall@1
value: 0.043259907874071926
name: Dot Recall@1
- type: dot_recall@3
value: 0.09462904166649215
name: Dot Recall@3
- type: dot_recall@5
value: 0.11342306096418099
name: Dot Recall@5
- type: dot_recall@10
value: 0.1395484399996232
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.34884301397980044
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.5304126984126983
name: Dot Mrr@10
- type: dot_map@100
value: 0.1572236749054705
name: Dot Map@100
- type: query_active_dims
value: 20.65999984741211
name: Query Active Dims
- type: query_sparsity_ratio
value: 0.999323111203479
name: Query Sparsity Ratio
- type: corpus_active_dims
value: 194.85540771484375
name: Corpus Active Dims
- type: corpus_sparsity_ratio
value: 0.9936159030301146
name: Corpus Sparsity Ratio
- type: avg_flops
value: 2.3607852458953857
name: Avg Flops
- task:
type: sparse-nano-beir
name: Sparse Nano BEIR
dataset:
name: NanoBEIR mean
type: NanoBEIR_mean
metrics:
- type: dot_accuracy@1
value: 0.5227755102040816
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.6955918367346939
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.7637551020408164
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 0.8359183673469387
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.5227755102040816
name: Dot Precision@1
- type: dot_precision@3
value: 0.3599183673469388
name: Dot Precision@3
- type: dot_precision@5
value: 0.29760000000000003
name: Dot Precision@5
- type: dot_precision@10
value: 0.23008979591836734
name: Dot Precision@10
- type: dot_recall@1
value: 0.24199500578786487
name: Dot Recall@1
- type: dot_recall@3
value: 0.35379966476849856
name: Dot Recall@3
- type: dot_recall@5
value: 0.4188419908358873
name: Dot Recall@5
- type: dot_recall@10
value: 0.5055459310792532
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.5246465910265005
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.6229219954648526
name: Dot Mrr@10
- type: dot_map@100
value: 0.41416226831835895
name: Dot Map@100
- type: query_active_dims
value: 33.51807144655281
name: Query Active Dims
- type: query_sparsity_ratio
value: 0.9989018389539823
name: Query Sparsity Ratio
- type: corpus_active_dims
value: 178.98493979922588
name: Corpus Active Dims
- type: corpus_sparsity_ratio
value: 0.9941358711814682
name: Corpus Sparsity Ratio
- type: avg_flops
value: 2.6044538021087646
name: Avg Flops
---
# splade-NeoBERT-RetroMAE-pretrain trained on LightOn MS MARCO (triplets)
This is a [SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [drexalt/NeoBERT-RetroMAE-pretrain](https://huggingface.co/drexalt/NeoBERT-RetroMAE-pretrain) on the [ms-marco-en-bge-gemma](https://huggingface.co/datasets/lightonai/ms-marco-en-bge-gemma) dataset using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.
## Model Details
### Model Description
- **Model Type:** SPLADE Sparse Encoder
- **Base model:** [drexalt/NeoBERT-RetroMAE-pretrain](https://huggingface.co/drexalt/NeoBERT-RetroMAE-pretrain)
- **Maximum Sequence Length:** 256 tokens
- **Output Dimensionality:** 30522 dimensions
- **Similarity Function:** Dot Product
- **Training Dataset:**
- [ms-marco-en-bge-gemma](https://huggingface.co/datasets/lightonai/ms-marco-en-bge-gemma)
- **Language:** en
- **License:** apache-2.0
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder)
### Full Model Architecture
```
SparseEncoder(
(0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'NeoBERTLMHead'})
(1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SparseEncoder
# Download from the 🤗 Hub
model = SparseEncoder("drexalt/splade-NeoBERT-msmarco-triplets-muon")
# Run inference
sentences = [
'The weather is lovely today.',
"It's so sunny outside!",
'He drove to the stadium.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 30522]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
## Evaluation
### Metrics
#### Sparse Information Retrieval
* Datasets: `NanoSciFact`, `NanoMSMARCO`, `NanoTouche2020`, `NanoSCIDOCS` and `NanoNFCorpus`
* Evaluated with [SparseInformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseInformationRetrievalEvaluator)
| Metric | NanoSciFact | NanoMSMARCO | NanoTouche2020 | NanoSCIDOCS | NanoNFCorpus |
|:----------------------|:------------|:------------|:---------------|:------------|:-------------|
| dot_accuracy@1 | 0.6 | 0.46 | 0.6939 | 0.4 | 0.46 |
| dot_accuracy@3 | 0.74 | 0.66 | 0.898 | 0.6 | 0.58 |
| dot_accuracy@5 | 0.82 | 0.74 | 0.9388 | 0.7 | 0.62 |
| dot_accuracy@10 | 0.86 | 0.86 | 0.9796 | 0.8 | 0.68 |
| dot_precision@1 | 0.6 | 0.46 | 0.6939 | 0.4 | 0.46 |
| dot_precision@3 | 0.26 | 0.22 | 0.6463 | 0.2933 | 0.38 |
| dot_precision@5 | 0.176 | 0.148 | 0.6 | 0.24 | 0.324 |
| dot_precision@10 | 0.094 | 0.086 | 0.5224 | 0.172 | 0.276 |
| dot_recall@1 | 0.575 | 0.46 | 0.048 | 0.0837 | 0.0433 |
| dot_recall@3 | 0.705 | 0.66 | 0.1277 | 0.1817 | 0.0946 |
| dot_recall@5 | 0.795 | 0.74 | 0.2001 | 0.2457 | 0.1134 |
| dot_recall@10 | 0.84 | 0.86 | 0.3355 | 0.3527 | 0.1395 |
| **dot_ndcg@10** | **0.7149** | **0.6432** | **0.5802** | **0.3361** | **0.3488** |
| dot_mrr@10 | 0.684 | 0.5754 | 0.7995 | 0.5253 | 0.5304 |
| dot_map@100 | 0.6735 | 0.5824 | 0.4046 | 0.253 | 0.1572 |
| query_active_dims | 72.84 | 25.32 | 19.8367 | 28.66 | 20.66 |
| query_sparsity_ratio | 0.9976 | 0.9992 | 0.9994 | 0.9991 | 0.9993 |
| corpus_active_dims | 197.2052 | 169.3692 | 159.209 | 212.2401 | 194.8554 |
| corpus_sparsity_ratio | 0.9935 | 0.9945 | 0.9948 | 0.993 | 0.9936 |
| avg_flops | 8.4018 | 1.709 | 2.9054 | 4.0499 | 2.3608 |
#### Sparse Nano BEIR
* Dataset: `NanoBEIR_mean`
* Evaluated with [SparseNanoBEIREvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseNanoBEIREvaluator) with these parameters:
```json
{
"dataset_names": [
"scifact",
"msmarco",
"touche2020",
"scidocs",
"nfcorpus"
]
}
```
| Metric | Value |
|:----------------------|:-----------|
| dot_accuracy@1 | 0.5228 |
| dot_accuracy@3 | 0.6956 |
| dot_accuracy@5 | 0.7638 |
| dot_accuracy@10 | 0.8359 |
| dot_precision@1 | 0.5228 |
| dot_precision@3 | 0.3599 |
| dot_precision@5 | 0.2976 |
| dot_precision@10 | 0.2301 |
| dot_recall@1 | 0.242 |
| dot_recall@3 | 0.3538 |
| dot_recall@5 | 0.4188 |
| dot_recall@10 | 0.5055 |
| **dot_ndcg@10** | **0.5246** |
| dot_mrr@10 | 0.6229 |
| dot_map@100 | 0.4142 |
| query_active_dims | 33.5181 |
| query_sparsity_ratio | 0.9989 |
| corpus_active_dims | 178.9849 |
| corpus_sparsity_ratio | 0.9941 |
| avg_flops | 2.6045 |
## Training Details
### Training Dataset
#### ms-marco-en-bge-gemma
* Dataset: [ms-marco-en-bge-gemma](https://huggingface.co/datasets/lightonai/ms-marco-en-bge-gemma) at [1a1ffe7](https://huggingface.co/datasets/lightonai/ms-marco-en-bge-gemma/tree/1a1ffe7cde403016be12ae532b249965b2293114)
* Size: 630,000 training samples
* Columns: query_id, document_ids, and scores
* Approximate statistics based on the first 1000 samples:
| | query_id | document_ids | scores |
|:--------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------|:------------------------------------|
| type | int | list | list |
| details |
603900 | [7010623, 6442837, 7350525, 8692778, 2372626, ...] | [29.75, 16.375, 13.5390625, 15.875, 14.4140625, ...] |
| 584347 | [4872208, 725251, 7237781, 6220558, 2359322, ...] | [27.421875, 16.765625, 23.921875, 14.7578125, 16.515625, ...] |
| 745908 | [1682756, 5541499, 7449637, 8635692, 8126024, ...] | [23.5, 21.046875, 15.484375, 14.4375, 14.1875, ...] |
* Loss: [SpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters:
```json
{
"loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)",
"document_regularizer_weight": 0.0003,
"query_regularizer_weight": 0.0003
}
```
### Evaluation Dataset
#### ms-marco-en-bge-gemma
* Dataset: [ms-marco-en-bge-gemma](https://huggingface.co/datasets/lightonai/ms-marco-en-bge-gemma) at [1a1ffe7](https://huggingface.co/datasets/lightonai/ms-marco-en-bge-gemma/tree/1a1ffe7cde403016be12ae532b249965b2293114)
* Size: 10,000 evaluation samples
* Columns: query_id, document_ids, and scores
* Approximate statistics based on the first 1000 samples:
| | query_id | document_ids | scores |
|:--------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------|:------------------------------------|
| type | int | list | list |
| details | 22689 | [3288373, 6422931, 1902940, 1448260, 975132, ...] | [26.03125, 17.9375, 16.375, 16.6875, 15.8046875, ...] |
| 232150 | [6017640, 6833904, 3846509, 2171280, 2048650, ...] | [26.390625, 15.3671875, 13.75, 15.28125, 15.65625, ...] |
| 408655 | [6648930, 6213451, 4063763, 2316914, 8553477, ...] | [24.796875, 11.6640625, 9.4296875, 11.4765625, 10.765625, ...] |
* Loss: [SpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters:
```json
{
"loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)",
"document_regularizer_weight": 0.0003,
"query_regularizer_weight": 0.0003
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: steps
- `per_device_train_batch_size`: 24
- `per_device_eval_batch_size`: 24
- `gradient_accumulation_steps`: 5
- `num_train_epochs`: 8
- `bf16`: True
- `dataloader_drop_last`: True
- `dataloader_num_workers`: 4
- `dataloader_prefetch_factor`: 2
- `load_best_model_at_end`: True
- `batch_sampler`: no_duplicates
#### All Hyperparameters