---
language:
- en
base_model:
- openai/clip-vit-large-patch14
tags:
- emotion_prediction
- VEA
- computer_vision
- perceptual_tasks
- CLIP
- EmoSet
---
# Don’t Judge Before You CLIP: Visual Emotion Analysis Model

This model is part of our paper:  
*"Don’t Judge Before You CLIP: A Unified Approach for Perceptual Tasks"*  
It was trained on the *EmoSet dataset* to predict emotion class.

## Model Overview

Visual perceptual tasks, such as visual emotion analysis, aim to estimate how humans perceive and interpret images. Unlike objective tasks (e.g., object recognition), these tasks rely on subjective human judgment, making labeled data scarce.

Our approach leverages *CLIP* as a prior for perceptual tasks, inspired by cognitive research showing that CLIP correlates well with human judgment. This suggests that CLIP implicitly captures human biases, emotions, and preferences. We fine-tune CLIP minimally using LoRA and incorporate an MLP head to adapt it to each specific task.

## Training Details

- *Dataset*: [EmoSet](https://vcc.tech/EmoSet)
- *Architecture*: CLIP Vision Encoder (ViT-L/14) with *LoRA adaptation*
- *Loss Function*: Cross Entropy Loss
- *Optimizer*: AdamW
- *Learning Rate*: 0.0001
- *Batch Size*: 32

## Performance

The model was trained on the *EmoSet dataset* using the common train, val, test splits and exhibits *state-of-the-art performance compared to previous methods.

## Usage

To use the model for inference:

```python
from torchvision import transforms
import torch
from PIL import Image

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load model
model = torch.load("EmoSet_clip_Lora_16.0R_8.0alphaLora_32_batch_0.0001_headmlp.pth").to(device).eval()

# Emotion label mapping
idx2label = {
    0: "amusement",
    1: "awe",
    2: "contentment",
    3: "excitement",
    4: "anger",
    5: "disgust",
    6: "fear",
    7: "sadness"
}

# Preprocessing function
def emo_preprocess():
    transform = transforms.Compose([
        transforms.Resize(224),
        transforms.CenterCrop(size=(224, 224)),
        transforms.ToTensor(),
        # Note: The model normalizes the image inside the forward pass
        # using mean = (0.48145466, 0.4578275, 0.40821073) and 
        # std = (0.26862954, 0.26130258, 0.27577711)
    ])
    return transform

# Load an image
image = Image.open("image_path.jpg").convert("RGB")
image = emo_preprocess()(image).unsqueeze(0).to(device)

# Run inference
with torch.no_grad():
    outputs = model(image)
    _, predicted = outputs.max(1)  # Get the class index

# Get emotion label
predicted_emotion = idx2label[predicted.item()]
print(f"Predicted Emotion: {predicted_emotion}")