uqlm.black_box.bleurt.BLEURTScorer#
- class uqlm.black_box.bleurt.BLEURTScorer#
Bases:
SimilarityScorer
- __init__()#
Class for computing BLEURT Scores between original responses and candidates. For more on BLEURT, refer to Sellam et al.(2020) [1]. Requires installation of bleurt package. Install using: pip install pip install –user git+https://github.com/google-research/bleurt.git
- Raises:
RuntimeError – If there’s an error downloading or initializing the BLEURT checkpoint
Methods
__init__
()Class for computing BLEURT Scores between original responses and candidates.
evaluate
(responses, sampled_responses)This method computes model-based text similarity metrics values for the provided pairs of texts.
- evaluate(responses, sampled_responses)#
This method computes model-based text similarity metrics values for the provided pairs of texts.
- Return type:
List
[float
]- Parameters:
responses (list of strings) – Original LLM response
sampled_responses (list of list of strings) – Candidate responses to be compared to the original response
- Returns:
Mean BLEURT scores
- Return type:
List of float
References