Skip to main content
Looking for Qualifire Guardrails? Check out the Qualifire Guardrails Integration for real-time content moderation, prompt injection detection, PII checks, and more.

​
Pre-Requisites

  1. Create an account on Qualifire
  2. Get your API key and webhook URL from the Qualifire dashboard
pip install litellm

​
Quick Start

Use just 2 lines of code to instantly log your responses across all providers with Qualifire. 
litellm.callbacks = ["qualifire_eval"]

import litellm
import os

# Set Qualifire credentials
os.environ["QUALIFIRE_API_KEY"] = "your-qualifire-api-key"
os.environ["QUALIFIRE_WEBHOOK_URL"] = "https://your-qualifire-webhook-url"

# LLM API Keys
os.environ['OPENAI_API_KEY'] = "your-openai-api-key"

# Set qualifire_eval as a callback & LiteLLM will send the data to Qualifire
litellm.callbacks = ["qualifire_eval"]

# OpenAI call
response = litellm.completion(
  model="gpt-5",
  messages=[
    {"role": "user", "content": "Hi 👋 - i'm openai"}
  ]
)

​
Using with LiteLLM Proxy

1

Setup config.yaml

Configure the LiteLLM proxy with Qualifire eval callback:
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
callbacks: ["qualifire_eval"]

general_settings:
master_key: "sk-1234"

environment_variables:
QUALIFIRE_API_KEY: "your-qualifire-api-key"
QUALIFIRE_WEBHOOK_URL: "https://app.qualifire.ai/api/v1/webhooks/evaluations"
2

Start the proxy

litellm --config config.yaml
3

Test it!

curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hi 👋 - i'\''m openai"}]}'

​
Environment Variables

Both environment variables are required. Get your API key and webhook URL from the Qualifire dashboard.
VariableDescription
QUALIFIRE_API_KEYYour Qualifire API key for authentication
QUALIFIRE_WEBHOOK_URLThe Qualifire webhook endpoint URL from your dashboard

​
What Gets Logged?

Request messages, parameters, and model configuration sent to the LLM provider.
Response content, metadata, finish reason, and any tool calls returned by the model.
Token usage statistics, latency metrics, cost data, and model information. The full LiteLLM Standard Logging Payload is sent on each successful LLM API call.
Once data is in Qualifire, you can:
  • Run evaluations to detect hallucinations, toxicity, and policy violations
  • Set up guardrails to block or modify responses in real-time
  • View traces across your entire AI pipeline
  • Track performance and quality metrics over time

​
Additional Resources