uqlm.black_box.nli.NLIScorer#

class uqlm.black_box.nli.NLIScorer(device=None, verbose=False, nli_model_name='microsoft/deberta-large-mnli', max_length=2000)#

Bases: SimilarityScorer

__init__(device=None, verbose=False, nli_model_name='microsoft/deberta-large-mnli', max_length=2000)#

A class to computing NLI-based confidence scores. This class offers two types of confidence scores, namely noncontradiction probability [1] and semantic entropy [2].

Parameters:

  • device (torch.device input or torch.device object, default=None) – Specifies the device that classifiers use for prediction. Set to “cuda” for classifiers to be able to leverage the GPU.

  • verbose (bool, default=False) – Specifies whether to print verbose status updates of NLI scoring process

  • nli_model_name (str, default="microsoft/deberta-large-mnli") – Specifies which NLI model to use. Must be acceptable input to AutoTokenizer.from_pretrained() and AutoModelForSequenceClassification.from_pretrained()

  • max_length (int, default=2000) – Specifies the maximum allowed string length. Responses longer than this value will be truncated to avoid OutOfMemoryError

Methods

__init__([device, verbose, nli_model_name, ...])

A class to computing NLI-based confidence scores.

avg_logprob(logprobs)

Compute average logprob

evaluate(responses, sampled_responses[, ...])

Evaluate confidence scores on LLM responses.

predict(response1, response2)

This method compute probability of contradiction on the provide inputs.
static avg_logprob(logprobs)#

Compute average logprob

Return type:

float

evaluate(responses, sampled_responses, responses_logprobs=None, sampled_responses_logprobs=None, use_best=False, compute_entropy=False, best_response_selection='discrete', progress_bar=None)#

Evaluate confidence scores on LLM responses.

Return type:

Dict[str, Any]

Parameters:

  • responses (list of strings) – Original LLM response

  • sampled_responses (list of list of strings) – Sampled candidate responses to be compared to the original response

  • responses_logprobs (list of list of dicts, default=None) – Log probabilities of the original response

  • sampled_responses_logprobs (list of list of list of dicts, default=None) – Log probabilities of the sampled responses

  • use_best (bool) – Specifies whether to swap the original response for the uncertainty-minimized response based on semantic entropy clusters.

  • compute_entropy (bool, default=False) – Specifies whether to include semantic entropy in returned result

  • best_response_selection (str, default="discrete") – Specifies the type of entropy confidence score to compute best response. Must be one of “discrete” or “token-based”.

  • progress_bar (rich.progress.Progress, default=None) – If provided, displays a progress bar while scoring responses

Returns:

Dictionary containing mean NLI and (optionally) semantic entropy scores. The dictionary will also contain original and multiple responses, updated if use_best is True

Return type:

Dict

predict(response1, response2)#

This method compute probability of contradiction on the provide inputs.

Return type:

Any

Parameters:

  • response1 (str) – An input for the sequence classification DeBERTa model.

  • response2 (str) – An input for the sequence classification DeBERTa model.

Returns:

Probabilities computed by NLI model

Return type:

numpy.ndarray

References