uqlm.black_box.bleurt.BLEURTScorer#

class uqlm.black_box.bleurt.BLEURTScorer#

Bases: SimilarityScorer

__init__()#

Class for computing BLEURT Scores between original responses and candidates. For more on BLEURT, refer to Sellam et al.(2020) [1]. Requires installation of bleurt package. Install using: pip install pip install –user git+https://github.com/google-research/bleurt.git

Raises:

RuntimeError – If there’s an error downloading or initializing the BLEURT checkpoint

Methods

__init__()

Class for computing BLEURT Scores between original responses and candidates.

evaluate(responses, sampled_responses)

This method computes model-based text similarity metrics values for the provided pairs of texts.
evaluate(responses, sampled_responses)#

This method computes model-based text similarity metrics values for the provided pairs of texts.

Return type:

List[float]

Parameters:

  • responses (list of strings) – Original LLM response

  • sampled_responses (list of list of strings) – Candidate responses to be compared to the original response

Returns:

Mean BLEURT scores

Return type:

List of float

References