uqlm.scorers.panel.LLMPanel#
- class uqlm.scorers.panel.LLMPanel(judges, llm=None, system_prompt='You are a helpful assistant.', max_calls_per_min=None)#
Bases:
UncertaintyQuantifier
- __init__(judges, llm=None, system_prompt='You are a helpful assistant.', max_calls_per_min=None)#
Class for aggregating multiple instances of LLMJudge using min, max, or majority voting
- Parameters:
judges (list of LLMJudge or BaseChatModel) – Judges to use. If BaseChatModel, LLMJudge is instantiated using default parameters.
llm (BaseChatModel) – A langchain llm object to get passed to chain constructor. User is responsible for specifying temperature and other relevant parameters to the constructor of their llm object.
max_calls_per_min (int, default=None) – Used to control rate limiting. Will be used for original llm and any judges constructed from instances of BaseChatModel in judges
system_prompt (str or None, default="You are a helpful assistant.") – Optional argument for user to provide custom system prompt
Methods
__init__
(judges[, llm, system_prompt, ...])Class for aggregating multiple instances of LLMJudge using min, max, or majority voting
generate_and_score
(prompts)Generate LLM responses to provided prompts and use panel of judges to score responses for correctness.
generate_candidate_responses
(prompts)This method generates multiple responses for uncertainty estimation.
generate_original_responses
(prompts)This method generates original responses for uncertainty estimation.
score
(prompts[, responses])Use panel to of judges to score provided responses for correctness.
- async generate_and_score(prompts)#
Generate LLM responses to provided prompts and use panel of judges to score responses for correctness.
- async generate_candidate_responses(prompts)#
This method generates multiple responses for uncertainty estimation. If specified in the child class, all responses are postprocessed using the callable function defined by the user.
- Return type:
List
[List
[str
]]
- async generate_original_responses(prompts)#
This method generates original responses for uncertainty estimation. If specified in the child class, all responses are postprocessed using the callable function defined by the user.
- Return type:
List
[str
]
- async score(prompts, responses=None)#
Use panel to of judges to score provided responses for correctness. Use if responses are already generated. Otherwise, use generate_and_score.
- Return type:
- Parameters:
prompts (list of str) – A list of input prompts for the model.
responses (list of str, default = None) – A list of LLM responses for the corresponding to the provided prompts.
- Returns:
UQResult containing prompts, responses, Q/A concatenations, judge responses, and judge scores
- Return type:
References