Minimum Token Probability#

min_probability

Minimum Token Probability (MTP) uses the minimum among token probabilities for a given response as a confidence score.

Definition#

Minimum token probability is defined as:

\[MTP(y_i) = \min_{t \in y_i} p_t\]

where \(t\) iterates over all tokens in response \(y_i\) and \(p_t\) denotes the token probability.

Key Properties:

  • Identifies the “weakest link” - the least confident token in the response

  • Robust to long responses with mostly high-confidence tokens

  • Score range: \([0, 1]\)

How It Works#

  1. Generate a response with logprobs enabled

  2. Extract the probability for each token in the response

  3. Return the minimum probability across all tokens

This scorer is particularly useful for detecting responses where the model is uncertain about specific parts, even if most of the response is generated with high confidence.

Parameters#

When using WhiteBoxUQ, specify "min_probability" in the scorers list.

Example#

from uqlm import WhiteBoxUQ

# Initialize with min_probability scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["min_probability"]
)

# Generate responses and compute scores
results = await wbuq.generate_and_score(prompts=prompts)

# Access the min_probability scores
print(results.to_df()["min_probability"])

References#

See Also#