Skip to main content
Looking for Qualifire Guardrails? Check out the Qualifire Guardrails Integration for real-time content moderation, prompt injection detection, PII checks, and more.

​
Pre-Requisites

  1. Create an account on Qualifire
  2. Get your API key and webhook URL from the Qualifire dashboard
pip install litellm

​
Quick Start

Use just 2 lines of code to instantly log your responses across all providers with Qualifire. 
litellm.callbacks = ["qualifire_eval"]

import litellm
import os

# Set Qualifire credentials
os.environ["QUALIFIRE_API_KEY"] = "your-qualifire-api-key"
os.environ["QUALIFIRE_WEBHOOK_URL"] = "https://your-qualifire-webhook-url"

# LLM API Keys
os.environ['OPENAI_API_KEY'] = "your-openai-api-key"

# Set qualifire_eval as a callback & LiteLLM will send the data to Qualifire
litellm.callbacks = ["qualifire_eval"]

# OpenAI call
response = litellm.completion(
  model="gpt-5",
  messages=[
    {"role": "user", "content": "Hi 👋 - i'm openai"}
  ]
)

​
Using with LiteLLM Proxy

1

Setup config.yaml

Configure the LiteLLM proxy with Qualifire eval callback:
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
callbacks: ["qualifire_eval"]

general_settings:
master_key: "sk-1234"

environment_variables:
QUALIFIRE_API_KEY: "your-qualifire-api-key"
QUALIFIRE_WEBHOOK_URL: "https://app.qualifire.ai/api/v1/webhooks/evaluations"
2

Start the proxy

litellm --config config.yaml
3

Test it!

curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hi 👋 - i'\''m openai"}]}'

​
Environment Variables

VariableDescription
QUALIFIRE_API_KEYYour Qualifire API key for authentication
QUALIFIRE_WEBHOOK_URLThe Qualifire webhook endpoint URL from your dashboard

​
What Gets Logged?

The LiteLLM Standard Logging Payload is sent to your Qualifire endpoint on each successful LLM API call. This includes:
  • Request messages and parameters
  • Response content and metadata
  • Token usage statistics
  • Latency metrics
  • Model information
  • Cost data
Once data is in Qualifire, you can:
  • Run evaluations to detect hallucinations, toxicity, and policy violations
  • Set up guardrails to block or modify responses in real-time
  • View traces across your entire AI pipeline
  • Track performance and quality metrics over time

​
Additional Resources