uqlm.black_box.cosine.CosineScorer#

class uqlm.black_box.cosine.CosineScorer(transformer='all-MiniLM-L6-v2')#

Bases: SimilarityScorer

__init__(transformer='all-MiniLM-L6-v2')#

Compute cosine similarity betwee original and candidate responses.

Parameters:

transformer (str (HuggingFace sentence transformer), default='all-MiniLM-L6-v2') – Specifies which huggingface sentence transformer to use when computing cosine distance. See https://huggingface.co/sentence-transformers?sort_models=likes#models for more information. The recommended sentence transformer is ‘all-MiniLM-L6-v2’.

Methods

__init__([transformer])

Compute cosine similarity betwee original and candidate responses.

evaluate(responses, sampled_responses)

This method computes model-based text similarity metrics values for the provided pairs of texts.

evaluate(responses, sampled_responses)#

This method computes model-based text similarity metrics values for the provided pairs of texts.

Return type:

List[float]

Parameters:
  • responses (list of strings) – Original LLM response

  • sampled_responses (list of list of strings) – Candidate responses to be compared to the original response

Returns:

Mean cosine similarity values

Return type:

List of float

References