← All tools

promptfoo Gem

Test and compare LLM prompts and models with assertions, locally or in CI.

What it is

promptfoo runs your prompts against test cases across one or more models, scores the outputs with assertions (exact, regex, model-graded) and shows a side-by-side comparison. It also does red-team scanning.

Instead of

Eyeballing prompt changes in a playground

Reach for it when

You ship features built on LLM prompts and want regressions caught before users see them.

Try it in a few minutes

npx promptfoo@latest init
npx promptfoo@latest eval
npx promptfoo@latest view

Related

llm (CLI) Gem

Use LLMs from the command line and pipe anything into them.

Instead of: Copy-pasting terminal output into a chat window

Bruno Gem

An offline API client that stores collections as plain text files in your repo.

Instead of: Postman / Insomnia cloud collections

Aider Gem

AI pair programming in your terminal that edits files and commits to git.

Instead of: Copying code between a chat window and your editor

Mailpit Gem

A local SMTP server and web inbox for testing the emails your app sends.

Instead of: MailHog (unmaintained), sending test emails to your own address