langfair.metrics.counterfactual.metrics.rougel.RougelSimilarity#

class langfair.metrics.counterfactual.metrics.rougel.RougelSimilarity(rouge_metric='rougeL', how='mean')#

Bases: Metric

__init__(rouge_metric='rougeL', how='mean')#

Compute variations of social group substitutions of language models. This class enables calculation of counterfactual ROUGE-L. For more information on this metric, refer to: https://arxiv.org/abs/2407.10853

Parameters:

  • rouge_metric ({'rougeL','rougeLsum'}, default='rougeL') – Specifies which ROUGE metric to use. If sentence-wise assessment is preferred, select ‘rougeLsum’.

  • how ({'mean','pairwise'}) – Specifies whether to return the mean cosine similarity over all counterfactual pairs or a list containing cosine distance for each pair.

Methods

__init__([rouge_metric, how])

Compute variations of social group substitutions of language models.

evaluate(texts1, texts2)

Returns mean Rouge-L score between two lists of generated outputs.
evaluate(texts1, texts2)#

Returns mean Rouge-L score between two lists of generated outputs.

Parameters:

  • texts1 (list of strings) – A list of generated outputs from a language model each containing mention of the same protected attribute group.

  • texts2 (list of strings) – A list, analogous to texts1 of counterfactually generated outputs from a language model each containing mention of the same protected attribute group. The mentioned protected attribute group must be a different group within the same protected attribute as mentioned in texts1.

Returns:

Mean ROUGE-L score for provided lists of texts.

Return type:

float