Semantic Density#

semantic_density

Semantic Density (SD) approximates a probability density function (PDF) in semantic space for estimating response correctness.

Definition#

Given a prompt \(x\) with candidate response \(y_*\), the objective is to construct a PDF that assigns higher density to regions in semantic space corresponding to correct responses.

Step 1: Sample Reference Responses

Sample \(M\) unique reference responses \(y_i\) (for \(i = 1, 2, \dots, M\)) conditioned on \(x\).

Step 2: Estimate Semantic Distance

For any pair of responses \(y_i, y_j\) with corresponding embeddings \(v_i, v_j\), the semantic distance is estimated as:

\[\mathbb{E}(\|v_i - v_j\|^2) = p_c(y_i, y_j | x) + \frac{1}{2} \cdot p_n(y_i, y_j | x)\]

where \(p_c\) and \(p_n\) denote the contradiction and neutrality scores returned by an NLI model, respectively.

Step 3: Compute Kernel Function

This estimated distance is incorporated in the kernel function \(K\):

\[K(v_*, v_i) = (1 - \mathbb{E}(\|v_* - v_i\|^2)) \cdot \mathbf{1}_{\mathbb{E}(\|v_* - v_i\|) \leq 1}\]

where \(\mathbf{1}\) is the indicator function such that \(\mathbf{1}_{\text{condition}} = 1\) when the condition holds and \(0\) otherwise.

Step 4: Compute Semantic Density

The final semantic density score is:

\[SD(y_* | x) = \frac{\sum_{i=1}^M \sqrt[L_i]{p(y_i|x)} \cdot K(v_*, v_i)}{\sum_{i=1}^M \sqrt[L_i]{p(y_i|x)}}\]

where \(L_i\) denotes the length of \(y_i\) and \(p(y_i|x)\) is the sequence probability.

Key Properties:

Combines semantic similarity with token probability weighting
Uses NLI model to estimate distances in semantic space
Score range: \([0, 1]\)

How It Works#

Generate multiple reference responses with logprobs from the same prompt
For each reference response, compute its length-normalized probability
Use an NLI model to estimate semantic distances between the original and reference responses
Apply a kernel function to convert distances to similarity weights
Compute a probability-weighted average of kernel values

Parameters#

When using WhiteBoxUQ, specify "semantic_density" in the scorers list.

Example#

from uqlm import WhiteBoxUQ

# Initialize with semantic_density scorer
wbuq = WhiteBoxUQ(
    llm=llm,
    scorers=["semantic_density"],
    sampling_temperature=1.0,
    length_normalize=True
)

# Generate responses and compute scores
results = await wbuq.generate_and_score(prompts=prompts, num_responses=5)

# Access the semantic_density scores
print(results.to_df()["semantic_density"])

You can also use the dedicated SemanticDensity class:

from uqlm import SemanticDensity

# Initialize SemanticDensity scorer
sd = SemanticDensity(
    llm=llm,
    nli_model_name="microsoft/deberta-large-mnli",
    length_normalize=True
)

# Generate and score
results = await sd.generate_and_score(prompts=prompts, num_responses=5)

References#

Qiu, L., et al. (2024). Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space. arXiv.