uqlm.black_box.consistency.ConsistencyScorer#

class uqlm.black_box.consistency.ConsistencyScorer(nli_model_name='microsoft/deberta-large-mnli', max_length=2000, use_best=False, scorers=['noncontradiction', 'entailment'], device=None, nli=None, nli_batch_size=32)#

Bases: SimilarityScorer

__init__(nli_model_name='microsoft/deberta-large-mnli', max_length=2000, use_best=False, scorers=['noncontradiction', 'entailment'], device=None, nli=None, nli_batch_size=32)#

Initialize the NonContradictionScorer.

Parameters:

  • use_best (bool, default=False) – Specifies whether to swap the original response for the uncertainty-minimized response based on semantic entropy clusters.

  • device (str or torch.device input or torch.device object, default=None) – Specifies the device that the NLI model uses for prediction. If None, detects and returns the best available PyTorch device. Ignored if nli is provided.

  • nli (NLI, default=None) – An existing NLI instance to reuse. If provided, no new NLI model is loaded and nli_model_name, max_length, device, and nli_batch_size are ignored.

  • nli_batch_size (int, default=32) – Number of premise-hypothesis pairs scored per forward pass. Ignored if nli is provided.

Methods

__init__([nli_model_name, max_length, ...])

Initialize the NonContradictionScorer.

evaluate(responses, sampled_responses[, ...])

Evaluate confidence scores on LLM responses.
evaluate(responses, sampled_responses, available_nli_scores={}, progress_bar=None)#

Evaluate confidence scores on LLM responses.

Return type:

Dict[str, Any]

Parameters:

  • responses (list of strings) – Original LLM response

  • sampled_responses (list of list of strings) – Sampled candidate responses to be compared to the original response

  • progress_bar (rich.progress.Progress, default=None) – If provided, displays a progress bar while scoring responses

Returns:

Dictionary containing mean NLI and (optionally) semantic entropy scores. The dictionary will also contain original and multiple responses, updated if use_best is True

Return type:

Dict

References