langfair.metrics.counterfactual.metrics.bleu.BleuSimilarity

class langfair.metrics.counterfactual.metrics.bleu.BleuSimilarity(how='mean')

Bases: Metric

__init__(how='mean')

Compute variations of social group substitutions of language models. This class enables calculation of counterfactual BLEU. For more information on this metric, refer to: https://arxiv.org/abs/2407.10853

Parameters:

how ({'mean','pairwise'}) – Specifies whether to return the mean cosine similarity over all counterfactual pairs or a list containing cosine distance for each pair.

Methods

__init__([how])	Compute variations of social group substitutions of language models.
evaluate(texts1, texts2)	Returns mean BLEU score between two lists of generated outputs.

evaluate(texts1, texts2)

Returns mean BLEU score between two lists of generated outputs.

Parameters:

• texts1 (list of strings) – A list of generated outputs from a language model each containing mention of the same protected attribute group.

• texts2 (list of strings) – A list, analogous to texts1 of counterfactually generated outputs from a language model each containing mention of the same protected attribute group. The mentioned protected attribute group must be a different group within the same protected attribute as mentioned in texts1.

Returns:

Mean BLEU score for provided lists of texts.

Return type:

float