Is Promptfoo still open source after the OpenAI acquisition?

Yes — the core CLI and library remain open source and work with Anthropic, Google, Mistral, and local models. What changes over time is roadmap ownership. For internal regression testing this is a non-issue; for published model comparisons, disclose the harness ownership.

Promptfoo vs DeepEval — which one?

Promptfoo is config-first (YAML matrices, CLI, CI diffs) and excels at comparing prompts and models side by side. DeepEval is code-first (pytest assertions, Python metrics) and excels at unit-testing application logic. Many teams run Promptfoo for prompt/model selection and DeepEval for regression gates.

Can Promptfoo replace a dedicated red-teaming tool?

For application-level probing, largely yes: its red-team mode generates attack suites tuned to your app's config. For model-level adversarial research or multi-turn campaign orchestration, Garak and PyRIT go deeper.

Promptfoo

Name: Promptfoo
Brand: Promptfoo
Availability: InStock

Open Sourcetrending #9Model Evaluation & Benchmarking

CLI and library for systematic LLM testing, red-teaming, and eval-driven development — acquired by OpenAI in 2026.

Last verified 2026-07-18

Visit Site GitHub

editorial take

Promptfoo won the developer eval market by being boring in the best way: a config file, a matrix of prompts times models times assertions, and a diff view. Its March 2026 acquisition by OpenAI validated eval-driven development as a category — and raised the obvious question of neutrality when the harness you use to compare models is owned by one of the vendors being compared.

It remains open source and provider-agnostic in code. Teams running competitive model bake-offs should watch governance of the project; teams using it for regression testing and red-teaming their own apps lose nothing.

overview

Promptfoo is a developer-first tool for testing prompts, models, and agents against quality, safety, and security assertions. It ships built-in red-team plugins mapped to the OWASP LLM Top 10 and NIST AI RMF (prompt injection, PII leakage, harmful content, hallucination), runs locally or in CI, and produces structured reports suitable for evidence collection. Used by teams at more than 25% of the Fortune 500. OpenAI announced its acquisition of Promptfoo on March 9, 2026; the open-source project continues, and the technology is being integrated into OpenAI's Frontier enterprise agent platform. Teams with strict vendor-independence requirements for auditing OpenAI models may want to pair it with a provider-neutral scanner such as Garak or PyRIT.

core features

Promptfoo is a developer-first tool for testing prompts, models, and agents against quality, safety, and security assertions.
It ships built-in red-team plugins mapped to the OWASP LLM Top 10 and NIST AI RMF (prompt injection, PII leakage, harmful content, hallucination), runs locally or in CI, and produces structured reports suitable for evidence collection.
Used by teams at more than 25% of the Fortune 500.
OpenAI announced its acquisition of Promptfoo on March 9, 2026; the open-source project continues, and the technology is being integrated into OpenAI's Frontier enterprise agent platform.
Teams with strict vendor-independence requirements for auditing OpenAI models may want to pair it with a provider-neutral scanner such as Garak or PyRIT.

target frameworks

OWASP LLM Top 10NIST AI RMFMITRE ATLAS

faq

Promptfoo — frequently asked questions

Is Promptfoo still open source after the OpenAI acquisition?: Yes — the core CLI and library remain open source and work with Anthropic, Google, Mistral, and local models. What changes over time is roadmap ownership. For internal regression testing this is a non-issue; for published model comparisons, disclose the harness ownership.
Promptfoo vs DeepEval — which one?: Promptfoo is config-first (YAML matrices, CLI, CI diffs) and excels at comparing prompts and models side by side. DeepEval is code-first (pytest assertions, Python metrics) and excels at unit-testing application logic. Many teams run Promptfoo for prompt/model selection and DeepEval for regression gates.
Can Promptfoo replace a dedicated red-teaming tool?: For application-level probing, largely yes: its red-team mode generates attack suites tuned to your app's config. For model-level adversarial research or multi-turn campaign orchestration, Garak and PyRIT go deeper.