promptfoo

Overview

promptfoo is a testing framework for evaluating prompts, agents, and RAG systems across multiple LLM providers including GPT, Claude, Gemini, and DeepSeek. It runs comparative benchmarks, red team tests, and vulnerability scans using declarative YAML configs with CLI and CI/CD support.

Best for

Best for
Teams building LLM applications who need systematic prompt validation and security testing before deployment

Use cases

• Compare prompt performance across different LLM models before production
• Automate security testing and adversarial input scanning for AI applications
• Integrate prompt evaluation into CI/CD pipelines for continuous quality checks

Notes

promptfoo is a testing framework for evaluating prompts, agents, and RAG systems across multiple LLM providers including GPT, Claude, Gemini, and DeepSeek. It runs comparative benchmarks, red team tests, and vulnerability scans using declarative YAML configs with CLI and CI/CD support.

21,784 stars on GitHub. Last updated 2026-06-01. Licensed MIT.

Use cases

• Compare prompt performance across different LLM models before production
• Automate security testing and adversarial input scanning for AI applications
• Integrate prompt evaluation into CI/CD pipelines for continuous quality checks

Pros

• Multi-model comparison built in, reducing vendor lock-in risk
• Red teaming and vulnerability scanning included, not bolted on
• Declarative config approach makes tests reproducible and version-controllable

Cons

• Requires familiarity with YAML config syntax and CLI tooling
• Testing scope limited to prompt and agent behavior, not full application integration
• Costs scale with API calls to external LLM providers during test runs

Indexed from awesome-llm and enriched against its public facts.