uqlm.black_box.bert.BertScorer

class uqlm.black_box.bert.BertScorer

Bases: SimilarityScorer

__init__()

Class for computing BERTScore values between original responses and candidates. For more on BERTScore, refer to Zhang et al.(2020) [1].

Methods

__init__()	Class for computing BERTScore values between original responses and candidates.
evaluate(responses, sampled_responses)	This method computes model-based text similarity metrics values for the provided pairs of texts.

evaluate(responses, sampled_responses)

This method computes model-based text similarity metrics values for the provided pairs of texts.

Return type:

List[float]

Parameters:

• responses (list of strings) – Original LLM response

• sampled_responses (list of list of strings) – Candidate responses to be compared to the original response

Returns:

Mean BertScore values

Return type:

List of float

References

[1]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: evaluating text generation with bert. 2020. URL: https://arxiv.org/abs/1904.09675, arXiv:1904.09675.