uqlm.black_box.nli.NLIScorer#
- class uqlm.black_box.nli.NLIScorer(device=None, verbose=False, nli_model_name='microsoft/deberta-large-mnli', max_length=2000)#
Bases:
SimilarityScorer
- __init__(device=None, verbose=False, nli_model_name='microsoft/deberta-large-mnli', max_length=2000)#
A class to computing NLI-based confidence scores. This class offers two types of confidence scores, namely noncontradiction probability [1] and semantic entropy [2].
- Parameters:
device (torch.device input or torch.device object, default=None) – Specifies the device that classifiers use for prediction. Set to “cuda” for classifiers to be able to leverage the GPU.
verbose (bool, default=False) – Specifies whether to print verbose status updates of NLI scoring process
nli_model_name (str, default="microsoft/deberta-large-mnli") – Specifies which NLI model to use. Must be acceptable input to AutoTokenizer.from_pretrained() and AutoModelForSequenceClassification.from_pretrained()
max_length (int, default=2000) – Specifies the maximum allowed string length. Responses longer than this value will be truncated to avoid OutOfMemoryError
Methods
__init__
([device, verbose, nli_model_name, ...])A class to computing NLI-based confidence scores.
evaluate
(responses, sampled_responses, use_best)Evaluate confidence scores on LLM responses.
predict
(response1, response2)This method compute probability of contradiction on the provide inputs.
- evaluate(responses, sampled_responses, use_best, compute_entropy=False)#
Evaluate confidence scores on LLM responses.
- Return type:
Dict
[str
,Any
]- Parameters:
responses (list of strings) – Original LLM response
sampled_responses (list of list of strings) – Sampled candidate responses to be compared to the original response
use_best (bool) – Specifies whether to swap the original response for the uncertainty-minimized response based on semantic entropy clusters.
compute_entropy (bool, default=False) – Specifies whether to include semantic entropy in returned result
- Returns:
Dictionary containing mean NLI and (optionally) semantic entropy scores. The dictionary will also contain original and multiple responses, updated if use_best is True
- Return type:
Dict
- predict(response1, response2)#
This method compute probability of contradiction on the provide inputs.
- Return type:
Any
- Parameters:
response1 (str) – An input for the sequence classification DeBERTa model.
response2 (str) – An input for the sequence classification DeBERTa model.
- Returns:
Probabilities computed by NLI model
- Return type:
numpy.ndarray
References