uqlm.black_box.bert.BertScorer#

class uqlm.black_box.bert.BertScorer(device=None)#

Bases: SimilarityScorer

__init__(device=None)#

Class for computing BERTScore values between original responses and candidates. For more on BERTScore, refer to Zhang et al.(2020) [1].

Parameters:

device (torch.device input or torch.device object, default=None) – Specifies the device that classifiers use for prediction. Set to “cuda” for classifiers to be able to leverage the GPU.

Methods

__init__([device])

Class for computing BERTScore values between original responses and candidates.

evaluate(responses, sampled_responses[, ...])

This method computes model-based text similarity metrics values for the provided pairs of texts.
evaluate(responses, sampled_responses, progress_bar=None)#

This method computes model-based text similarity metrics values for the provided pairs of texts.

Return type:

List[float]

Parameters:

  • responses (list of strings) – Original LLM response

  • sampled_responses (list of list of strings) – Candidate responses to be compared to the original response

  • progress_bar (rich.progress.Progress, default=None) – If provided, displays a progress bar while scoring responses

Returns:

Mean BertScore values

Return type:

List of float

References