llm (CLI) Gem
Use LLMs from the command line and pipe anything into them.
Instead of: Copy-pasting terminal output into a chat window
Test and compare LLM prompts and models with assertions, locally or in CI.
promptfoo runs your prompts against test cases across one or more models, scores the outputs with assertions (exact, regex, model-graded) and shows a side-by-side comparison. It also does red-team scanning.
Eyeballing prompt changes in a playground
You ship features built on LLM prompts and want regressions caught before users see them.
npx promptfoo@latest init
npx promptfoo@latest eval
npx promptfoo@latest view Use LLMs from the command line and pipe anything into them.
Instead of: Copy-pasting terminal output into a chat window
An offline API client that stores collections as plain text files in your repo.
Instead of: Postman / Insomnia cloud collections
AI pair programming in your terminal that edits files and commits to git.
Instead of: Copying code between a chat window and your editor
A local SMTP server and web inbox for testing the emails your app sends.
Instead of: MailHog (unmaintained), sending test emails to your own address