LLM Language Quality Metrics: Perplexity, BLEU, ROUGE & BERTScore Deep Dive

LLM Evaluation Metrics — Language Quality

LLM Language Quality
├── Perplexity  → Next-token probability surprise
├── BLEU        → N-gram precision & Brevity penalty
├── ROUGE       → N-gram recall & Longest Common Subsequence
└── BERTScore   → Contextual embedding cosine similarity

#1. Perplexity

Definition: How "surprised" a language model is by a sequence of text.

import torch
import torch.nn.functional as F

def perplexity(logits, target_ids):
    log_probs = F.log_softmax(logits, dim=-1)
    token_log_probs = log_probs.gather(1, target_ids.unsqueeze(1)).squeeze(1)
    return torch.exp(-token_log_probs.mean())

#2. BLEU (Bilingual Evaluation Understudy)

Measures n-gram precision with brevity penalty:

Precision-oriented, penalizes extra words. Weak for open-ended chat where paraphrasing is common.

#3. ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

Counterpart to BLEU, recall-oriented for summarization:

#4. BERTScore

Embeds candidate and reference using pretrained models (e.g. BERT/DeBERTa) and calculates semantic similarity via cosine distance.