\n",
"
\n",
" The UQLM library offers Uncertainty Quantification (UQ) methods for multimodal inputs, given that outputs are text-based. This demo provides an minimal illustration \n",
" of how to use uqlm scorers for multimodal inputs. The following scorers offer multimodal compatibility:\n",
"
\n",
" \n",
"#### Black-Box (Consistency) Scorers\n",
"* Discrete Semantic Entropy ([Farquhar et al., 2024](https://www.nature.com/articles/s41586-024-07421-0); [Kuhn et al., 2023](https://arxiv.org/pdf/2302.09664))\n",
"* Number of Semantic Sets ([Lin et al., 2024](https://arxiv.org/abs/2305.19187); [Vashurin et al., 2025](https://arxiv.org/abs/2406.15627); [Kuhn et al., 2023](https://arxiv.org/pdf/2302.09664))\n",
"* Non-Contradiction Probability ([Chen & Mueller, 2023](https://arxiv.org/abs/2308.16175); [Lin et al., 2024](https://arxiv.org/abs/2305.19187); [Manakul et al., 2023](https://arxiv.org/abs/2303.08896))\n",
"* Entailment Probability ([Lin et al., 2025](https://arxiv.org/abs/2305.19187); [Chen & Mueller, 2023](https://arxiv.org/abs/2308.16175))\n",
"* Exact Match ([Cole et al., 2023](https://arxiv.org/abs/2305.14613); [Chen & Mueller, 2023](https://arxiv.org/abs/2308.16175))\n",
"* BERTScore ([Manakul et al., 2023](https://arxiv.org/abs/2303.08896); [Zheng et al., 2020](https://arxiv.org/abs/1904.09675))\n",
"* Normalized Cosine Similarity ([Shorinwa et al., 2024](https://arxiv.org/pdf/2412.05563); [HuggingFace](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2))\n",
"\n",
"#### White-Box (Token-Probability-Based) Scorers\n",
"* Minimum token probability ([Manakul et al., 2023](https://arxiv.org/abs/2303.08896))\n",
"* Length-Normalized Sequence Probability ([Malinin & Gales, 2021](https://arxiv.org/pdf/2002.07650))\n",
"* Sequence Probability ([Vashurin et al., 2024](https://arxiv.org/abs/2406.15627))\n",
"* Mean Top-K Token Negentropy ([Scalena et al., 2025](https://arxiv.org/abs/2510.11170); [Manakul et al., 2023](https://arxiv.org/abs/2303.08896))\n",
"* Min Top-K Token Negentropy ([Scalena et al., 2025](https://arxiv.org/abs/2510.11170); [Manakul et al., 2023](https://arxiv.org/abs/2303.08896))\n",
"* Probability Margin ([Farr et al., 2024](https://arxiv.org/abs/2408.08217))\n",
"* Monte carlo sequence probability ([Kuhn et al., 2023](https://arxiv.org/abs/2302.09664))\n",
"* Consistency and Confidence (CoCoA) ([Vashurin et al., 2025](https://arxiv.org/abs/2502.04964))\n",
"* Semantic Negentropy ([Farquhar et al., 2024](https://www.nature.com/articles/s41586-024-07421-0)) \n",
"* Semantic Density ([Qiu et al., 2024](https://arxiv.org/abs/2405.13845))\n",
"\n",
"
/home/jupyter/uqlm/uqlm/utils/response_generator.py:105: UQLMBetaWarning: Use of BaseMessage in prompts argument is\n",
"in beta. Please use it with caution as it may change in future releases.\n",
" beta_warning(\"Use of BaseMessage in prompts argument is in beta. Please use it with caution as it may change in \n",
"future releases.\")\n",
"\n"
],
"text/plain": [
"/home/jupyter/uqlm/uqlm/utils/response_generator.py:105: UQLMBetaWarning: Use of BaseMessage in prompts argument is\n",
"in beta. Please use it with caution as it may change in future releases.\n",
" beta_warning(\"Use of BaseMessage in prompts argument is in beta. Please use it with caution as it may change in \n",
"future releases.\")\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"/home/jupyter/uqlm/uqlm/utils/response_generator.py:105: UQLMBetaWarning: Use of BaseMessage in prompts argument is\n",
"in beta. Please use it with caution as it may change in future releases.\n",
" beta_warning(\"Use of BaseMessage in prompts argument is in beta. Please use it with caution as it may change in \n",
"future releases.\")\n",
"\n"
],
"text/plain": [
"/home/jupyter/uqlm/uqlm/utils/response_generator.py:105: UQLMBetaWarning: Use of BaseMessage in prompts argument is\n",
"in beta. Please use it with caution as it may change in future releases.\n",
" beta_warning(\"Use of BaseMessage in prompts argument is in beta. Please use it with caution as it may change in \n",
"future releases.\")\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from uqlm import BlackBoxUQ\n",
"\n",
"bbuq = BlackBoxUQ(llm=llm, scorers=[\"noncontradiction\"], system_prompt=\"Answer as concisely as possible.\", device=device, use_best=False)\n",
"result = await bbuq.generate_and_score(prompts=prompts, num_responses=5)"
]
},
{
"cell_type": "markdown",
"id": "429af11e-0353-43d8-8131-8cb68438bcd9",
"metadata": {},
"source": [
"\n",
"## 3. Evaluate Hallucination Detection Performance"
]
},
{
"cell_type": "markdown",
"id": "19db1925-f7ad-44a8-aa4c-ee94ad128f2b",
"metadata": {},
"source": [
"Finally, we can check which questions the LLM answered correctly and compare to our UQLM's black-box confidence scores. "
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "eebce78c-cadf-4eca-a1f7-a4916dc2c850",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" question | \n",
" response | \n",
" sampled_responses | \n",
" noncontradiction | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" How many berries appear in the image | \n",
" 3 | \n",
" [3, 3, 3, 3, 3] | \n",
" 1.000000 | \n",
"
\n",
" \n",
" | 1 | \n",
" What color are the berries? | \n",
" The berries are red. | \n",
" [Red., The berries are dark red., Red., The berries are a deep red color., The berries are dark ... | \n",
" 0.998269 | \n",
"
\n",
" \n",
" | 2 | \n",
" How many times does the letter R appear in this image? | \n",
" 3 | \n",
" [3, 3, 3, 3, 3] | \n",
" 1.000000 | \n",
"
\n",
" \n",
" | 3 | \n",
" How many times does the letter P appear in this image | \n",
" 1 | \n",
" [2, 2, 2, 2, 1] | \n",
" 0.221604 | \n",
"
\n",
" \n",
" | 4 | \n",
" How many words appear in this image? | \n",
" 2 | \n",
" [2, 2, 2, 2, 2] | \n",
" 1.000000 | \n",
"
\n",
" \n",
"
\n",
"