uqlm.nli.nli.NLI#

class uqlm.nli.nli.NLI(device=None, verbose=False, nli_model_name='microsoft/deberta-large-mnli', max_length=2000)#

Bases: object

__init__(device=None, verbose=False, nli_model_name='microsoft/deberta-large-mnli', max_length=2000)#

A class to computing NLI-based confidence scores. This class offers two types of confidence scores, namely noncontradiction probability [1] and semantic entropy [2].

Parameters:

  • device (torch.device input or torch.device object, default=None) – Specifies the device that classifiers use for prediction. Set to “cuda” for classifiers to be able to leverage the GPU.

  • verbose (bool, default=False) – Specifies whether to print verbose status updates of NLI scoring process

  • nli_model_name (str, default="microsoft/deberta-large-mnli") – Specifies which NLI model to use. Must be acceptable input to AutoTokenizer.from_pretrained() and AutoModelForSequenceClassification.from_pretrained()

  • max_length (int, default=2000) – Specifies the maximum allowed string length. Responses longer than this value will be truncated to avoid OutOfMemoryError

Methods

__init__([device, verbose, nli_model_name, ...])

A class to computing NLI-based confidence scores.

get_nli_results(response1, response2)

This method computes mean NLI score and determines whether entailment exists.

predict(premise, hypothesis)

This method compute probability of contradiction on the provide inputs.
get_nli_results(response1, response2)#

This method computes mean NLI score and determines whether entailment exists.

Return type:

Dict[str, Any]

predict(premise, hypothesis)#

This method compute probability of contradiction on the provide inputs.

Return type:

Any

Parameters:

  • premise (str) – An input for the sequence classification DeBERTa model.

  • hypothesis (str) – An input for the sequence classification DeBERTa model.

Returns:

Probabilities computed by NLI model

Return type:

numpy.ndarray

References