Probability Margin#

probability_margin

Probability Margin (PM) computes the average difference between the top two token probabilities for each token in the response.

Definition#

This scorer requires accessing the top-K logprobs per token (K ≥ 2). Let the top-K token probabilities for token \(t_j\) be denoted as \(\{p_{t_{jk}}\}_{k=1}^K\), ordered by decreasing probability.

Probability Margin is defined as:

\[PM(y_i) = \frac{1}{L_i}\sum_{j=1}^{L_i} (p_{t_{j1}} - p_{t_{j2}})\]

where \(p_{t_{j1}}\) is the probability of the selected (top) token and \(p_{t_{j2}}\) is the probability of the second-best token.

Key Properties:

  • Measures how “decisively” the model chose each token

  • Large margins indicate clear preference; small margins indicate ambiguity

  • Score range: \([0, 1]\)

How It Works#

  1. Generate a response with top-K logprobs enabled (K ≥ 2)

  2. For each token position:

    • Get the probabilities of the top two candidate tokens

    • Compute the difference (margin) between them

  3. Average the margins across all token positions

A high probability margin indicates that the model was confident in its token choices throughout the response, while a low margin suggests the model was uncertain between alternatives.

Parameters#

When using WhiteBoxUQ, specify "probability_margin" in the scorers list.

Note

This scorer requires the LLM to support returning top-K logprobs (e.g., OpenAI models with top_logprobs parameter).

Example#

from uqlm import WhiteBoxUQ

# Initialize with probability_margin scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["probability_margin"],
    top_k_logprobs=5  # At least 2 required
)

# Generate responses and compute scores
results = await wbuq.generate_and_score(prompts=prompts)

# Access the probability_margin scores
print(results.to_df()["probability_margin"])

References#

See Also#