uqlm.scorers.white_box.WhiteBoxUQ#
- class uqlm.scorers.white_box.WhiteBoxUQ(llm=None, system_prompt='You are a helpful assistant.', max_calls_per_min=None, scorers=None)#
Bases:
UncertaintyQuantifier
- __init__(llm=None, system_prompt='You are a helpful assistant.', max_calls_per_min=None, scorers=None)#
Class for computing white-box UQ confidence scores. This class offers two confidence scores, normalized probability [1] and minimum probability [2].
- Parameters:
llm (BaseChatModel) – A langchain llm object to get passed to chain constructor. User is responsible for specifying temperature and other relevant parameters to the constructor of their llm object.
max_calls_per_min (int, default=None) – Used to control rate limiting.
system_prompt (str or None, default="You are a helpful assistant.") – Optional argument for user to provide custom system prompt
scorers (subset of {) – “imperplexity”, “geometric_mean_probability”, “min_probability”, “max_probability”,
} – Specifies which black box (consistency) scorers to include. If None, defaults to all.
default=None – Specifies which black box (consistency) scorers to include. If None, defaults to all.
Methods
__init__
([llm, system_prompt, ...])Class for computing white-box UQ confidence scores.
avg_logprob
(logprobs)Compute average logprob
generate_and_score
(prompts)Generate responses and compute white-box confidence scores based on extracted token probabilities.
generate_candidate_responses
(prompts)This method generates multiple responses for uncertainty estimation.
generate_original_responses
(prompts)This method generates original responses for uncertainty estimation.
get_logprobs
(logprobs)Extract log token probabilities
score
(logprobs_results[, prompts, responses])Compute white-box confidence scores from provided logprobs.
- avg_logprob(logprobs)#
Compute average logprob
- Return type:
float
- async generate_and_score(prompts)#
Generate responses and compute white-box confidence scores based on extracted token probabilities.
- async generate_candidate_responses(prompts)#
This method generates multiple responses for uncertainty estimation. If specified in the child class, all responses are postprocessed using the callable function defined by the user.
- Return type:
List
[List
[str
]]
- async generate_original_responses(prompts)#
This method generates original responses for uncertainty estimation. If specified in the child class, all responses are postprocessed using the callable function defined by the user.
- Return type:
List
[str
]
- static get_logprobs(logprobs)#
Extract log token probabilities
- score(logprobs_results, prompts=None, responses=None)#
Compute white-box confidence scores from provided logprobs.
- Return type:
- Parameters:
logprobs_results (list of logprobs_result) – List of dictionaries, each returned by BaseChatModel.agenerate
prompts (list of str, default=None) – A list of input prompts for the model.
responses (list of str, default=None) – A list of model responses for the prompts.
- Returns:
UQResult containing prompts, responses, logprobs, and white-box UQ scores
- Return type:
References