{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 🎯 BS Detector: Off-the-Shelf Ensemble for LLM Uncertainty\n", "\n", "
\n", " Ensemble UQ methods combine multiple individual scorers to provide a more robust and accurate uncertainty estimate. This demo illustrates the BS Detector method proposed in Chen & Mueller, 2023. It uses three components:\n", "
\n", "exact match rate
and noncontradiction probability
Set up LLM instance and load example data prompts.\n", "
Generate LLM Responses and Confidence Scores
\n", "Generate and score LLM responses to the example questions using the UQEnsemble()
class.\n",
"
Evaluate Hallucination Detection Performance
\n", "Visualize model accuracy at different thresholds of the ensemble score, combining exact match rate, noncontradiction probability, and self-judge. Compute precision, recall, and F1-score of hallucination detection.
\n", "\n", " | question | \n", "answer | \n", "
---|---|---|
0 | \n", "There are 87 oranges and 290 bananas in Philip... | \n", "145 | \n", "
1 | \n", "Marco and his dad went strawberry picking. Mar... | \n", "19 | \n", "
2 | \n", "Edward spent $ 6 to buy 2 books each book cost... | \n", "3 | \n", "
3 | \n", "Frank was reading through his favorite book. T... | \n", "198 | \n", "
4 | \n", "There were 78 dollars in Olivia's wallet. She ... | \n", "63 | \n", "
Parameter | \n", "Type & Default | \n", "Description | \n", "
---|---|---|
llm | \n", "BaseChatModeldefault=None | \n",
" A langchain llm `BaseChatModel`. User is responsible for specifying temperature and other relevant parameters to the constructor of the provided `llm` object. | \n", "
device | \n", "str or torch.devicedefault=\"cpu\" | \n",
" Specifies the device that NLI model use for prediction. Only applies to 'semantic_negentropy', 'noncontradiction' scorers. Pass a torch.device to leverage GPU. | \n", "
use_best | \n", "booldefault=True | \n",
" Specifies whether to swap the original response for the uncertainty-minimized response among all sampled responses based on semantic entropy clusters. Only used if `scorers` includes 'semantic_negentropy' or 'noncontradiction'. | \n", "
system_prompt | \n", "str or Nonedefault=\"You are a helpful assistant.\" | \n",
" Optional argument for user to provide custom system prompt for the LLM. | \n", "
max_calls_per_min | \n", "intdefault=None | \n",
" Specifies how many API calls to make per minute to avoid rate limit errors. By default, no limit is specified. | \n", "
use_n_param | \n", "booldefault=False | \n",
" Specifies whether to use n parameter for BaseChatModel . Not compatible with all BaseChatModel classes. If used, it speeds up the generation process substantially when num_responses is large. | \n",
"
postprocessor | \n", "callabledefault=None | \n",
" A user-defined function that takes a string input and returns a string. Used for postprocessing outputs. | \n", "
sampling_temperature | \n", "floatdefault=1 | \n",
" The 'temperature' parameter for LLM model to generate sampled LLM responses. Must be greater than 0. | \n", "
nli_model_name | \n", "strdefault=\"microsoft/deberta-large-mnli\" | \n",
" Specifies which NLI model to use. Must be acceptable input to AutoTokenizer.from_pretrained() and AutoModelForSequenceClassification.from_pretrained() . | \n",
"
🧠 LLM-Specific
\n", "llm
system_prompt
sampling_temperature
📊 Confidence Scores
\n", "nli_model_name
use_best
postprocessor
🖥️ Hardware
\n", "device
⚡ Performance
\n", "max_calls_per_min
use_n_param
Method | \n", "Description & Parameters | \n", "
---|---|
UQEnsemble.generate_and_score | \n", "\n",
" Generate LLM responses, sampled LLM (candidate) responses, and compute confidence scores for the provided prompts. \n", "Parameters: \n", "
Returns: \n",
" 💡 Best For: Complete end-to-end uncertainty quantification when starting with prompts.\n",
" \n",
" | \n",
"
UQEnsemble.score | \n", "\n",
" Compute confidence scores on provided LLM responses. Should only be used if responses and sampled responses are already generated. \n", "Parameters: \n", "
Returns: \n",
" 💡 Best For: Computing uncertainty scores when responses are already generated elsewhere.\n",
" \n",
" | \n",
"
\n", " | prompt | \n", "response | \n", "sampled_responses | \n", "ensemble_score | \n", "noncontradiction | \n", "exact_match | \n", "judge_1 | \n", "
---|---|---|---|---|---|---|---|
0 | \n", "When you solve this math problem only return t... | \n", "145 | \n", "[145, 145, 145, 145, 145] | \n", "1.000000 | \n", "1.000000 | \n", "1.0 | \n", "1.0 | \n", "
1 | \n", "When you solve this math problem only return t... | \n", "19 | \n", "[19, 19 pounds., 19, 19 pounds, 19 pounds] | \n", "0.763228 | \n", "0.995051 | \n", "0.4 | \n", "0.5 | \n", "
2 | \n", "When you solve this math problem only return t... | \n", "$3 | \n", "[$3, $3.00, This math problem has two equation... | \n", "0.878711 | \n", "0.983412 | \n", "0.2 | \n", "1.0 | \n", "
3 | \n", "When you solve this math problem only return t... | \n", "198 | \n", "[198, 198, 198, 198, 198] | \n", "1.000000 | \n", "1.000000 | \n", "1.0 | \n", "1.0 | \n", "
4 | \n", "When you solve this math problem only return t... | \n", "63 | \n", "[63 dollars., 63 dollars, 63, 63, 63] | \n", "0.941517 | \n", "0.995565 | \n", "0.6 | \n", "1.0 | \n", "
\n", " | prompt | \n", "response | \n", "sampled_responses | \n", "ensemble_score | \n", "noncontradiction | \n", "exact_match | \n", "judge_1 | \n", "answer | \n", "response_correct | \n", "
---|---|---|---|---|---|---|---|---|---|
0 | \n", "When you solve this math problem only return t... | \n", "145 | \n", "[145, 145, 145, 145, 145] | \n", "1.000000 | \n", "1.000000 | \n", "1.0 | \n", "1.0 | \n", "145 | \n", "True | \n", "
1 | \n", "When you solve this math problem only return t... | \n", "19 | \n", "[19, 19 pounds., 19, 19 pounds, 19 pounds] | \n", "0.763228 | \n", "0.995051 | \n", "0.4 | \n", "0.5 | \n", "19 | \n", "True | \n", "
2 | \n", "When you solve this math problem only return t... | \n", "$3 | \n", "[$3, $3.00, This math problem has two equation... | \n", "0.878711 | \n", "0.983412 | \n", "0.2 | \n", "1.0 | \n", "3 | \n", "True | \n", "
3 | \n", "When you solve this math problem only return t... | \n", "198 | \n", "[198, 198, 198, 198, 198] | \n", "1.000000 | \n", "1.000000 | \n", "1.0 | \n", "1.0 | \n", "198 | \n", "True | \n", "
4 | \n", "When you solve this math problem only return t... | \n", "63 | \n", "[63 dollars., 63 dollars, 63, 63, 63] | \n", "0.941517 | \n", "0.995565 | \n", "0.6 | \n", "1.0 | \n", "63 | \n", "True | \n", "