bigstral-12b-32k-8xMoE

Made using mergekit MoE branch with the following config:

base_model: abacusai/bigstral-12b-32k
gate_mode: random 
dtype: bfloat16
experts_per_token: 2
experts:
  - source_model: abacusai/bigstral-12b-32k
    positive_prompts: []
  - source_model: abacusai/bigstral-12b-32k
    positive_prompts: []
  - source_model: abacusai/bigstral-12b-32k
    positive_prompts: []
  - source_model: abacusai/bigstral-12b-32k
    positive_prompts: []
  - source_model: abacusai/bigstral-12b-32k
    positive_prompts: []
  - source_model: abacusai/bigstral-12b-32k
    positive_prompts: []
  - source_model: abacusai/bigstral-12b-32k
    positive_prompts: []
  - source_model: abacusai/bigstral-12b-32k
    positive_prompts: []