uqlm.black_box.nli.NLIScorer#

class uqlm.black_box.nli.NLIScorer(device=None, verbose=False, nli_model_name='microsoft/deberta-large-mnli', max_length=2000)#

Bases: SimilarityScorer

__init__(device=None, verbose=False, nli_model_name='microsoft/deberta-large-mnli', max_length=2000)#

A class to computing NLI-based confidence scores. This class offers two types of confidence scores, namely noncontradiction probability [1] and semantic entropy [2].

Parameters:

  • device (torch.device input or torch.device object, default=None) – Specifies the device that classifiers use for prediction. Set to “cuda” for classifiers to be able to leverage the GPU.

  • verbose (bool, default=False) – Specifies whether to print verbose status updates of NLI scoring process

  • nli_model_name (str, default="microsoft/deberta-large-mnli") – Specifies which NLI model to use. Must be acceptable input to AutoTokenizer.from_pretrained() and AutoModelForSequenceClassification.from_pretrained()

  • max_length (int, default=2000) – Specifies the maximum allowed string length. Responses longer than this value will be truncated to avoid OutOfMemoryError

Methods

__init__([device, verbose, nli_model_name, ...])

A class to computing NLI-based confidence scores.

evaluate(responses, sampled_responses, use_best)

Evaluate confidence scores on LLM responses.

predict(response1, response2)

This method compute probability of contradiction on the provide inputs.
evaluate(responses, sampled_responses, use_best, compute_entropy=False)#

Evaluate confidence scores on LLM responses.

Return type:

Dict[str, Any]

Parameters:

  • responses (list of strings) – Original LLM response

  • sampled_responses (list of list of strings) – Sampled candidate responses to be compared to the original response

  • use_best (bool) – Specifies whether to swap the original response for the uncertainty-minimized response based on semantic entropy clusters.

  • compute_entropy (bool, default=False) – Specifies whether to include semantic entropy in returned result

Returns:

Dictionary containing mean NLI and (optionally) semantic entropy scores. The dictionary will also contain original and multiple responses, updated if use_best is True

Return type:

Dict

predict(response1, response2)#

This method compute probability of contradiction on the provide inputs.

Return type:

Any

Parameters:

  • response1 (str) – An input for the sequence classification DeBERTa model.

  • response2 (str) – An input for the sequence classification DeBERTa model.

Returns:

Probabilities computed by NLI model

Return type:

numpy.ndarray

References