LangFair

Overview

LangFair is a Python library for conducting use-case level bias and fairness assessments on large language model outputs. It provides metrics and tests to evaluate demographic parity, equalized odds, and other fairness criteria for specific applications.

Best for

Best for
Developers and researchers who need to evaluate and mitigate bias in LLM-based applications at the use-case level.

Use cases

• Auditing LLM outputs for demographic bias in classification tasks
• Comparing fairness metrics across different model versions or prompts
• Integrating bias checks into LLM evaluation pipelines

Notes

LangFair is a Python library for conducting use-case level bias and fairness assessments on large language model outputs. It provides metrics and tests to evaluate demographic parity, equalized odds, and other fairness criteria for specific applications.

258 stars on GitHub. Last updated 2026-01-09.

Use cases

• Auditing LLM outputs for demographic bias in classification tasks
• Comparing fairness metrics across different model versions or prompts
• Integrating bias checks into LLM evaluation pipelines

Pros

• Open source and free to use with a permissive license
• Focused on use-case level assessment, not just aggregate metrics
• Python-native, easy to integrate into existing ML workflows

Cons

• Small community (258 stars) may mean limited support and documentation
• Requires manual setup and configuration for each use case
• Only supports Python, limiting use in polyglot environments

Indexed from awesome-langchain and enriched against its public facts.