> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qualifire.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Send LiteLLM logs to Qualifire for real-time evaluations, observability, and tracing

<Tip>
  Looking for Qualifire Guardrails? Check out the [Qualifire Guardrails
  Integration](/integrations/litellm-guardrails) for real-time content
  moderation, prompt injection detection, PII checks, and more.
</Tip>

## Pre-Requisites

1. Create an account on [Qualifire](https://app.qualifire.ai/)
2. Get your API key and webhook URL from the Qualifire dashboard

```bash theme={null}
pip install litellm
```

## Quick Start

Use just 2 lines of code to instantly log your responses **across all providers** with Qualifire.

```python theme={null}
litellm.callbacks = ["qualifire_eval"]
```

```python theme={null}
import litellm
import os

# Set Qualifire credentials
os.environ["QUALIFIRE_API_KEY"] = "your-qualifire-api-key"
os.environ["QUALIFIRE_WEBHOOK_URL"] = "https://your-qualifire-webhook-url"

# LLM API Keys
os.environ['OPENAI_API_KEY'] = "your-openai-api-key"

# Set qualifire_eval as a callback & LiteLLM will send the data to Qualifire
litellm.callbacks = ["qualifire_eval"]

# OpenAI call
response = litellm.completion(
  model="gpt-5",
  messages=[
    {"role": "user", "content": "Hi 👋 - i'm openai"}
  ]
)
```

## Using with LiteLLM Proxy

<Steps>
  <Step title="Setup config.yaml">
    Configure the LiteLLM proxy with Qualifire eval callback:

    ```yaml theme={null}
    model_list:
      - model_name: gpt-4o
        litellm_params:
          model: openai/gpt-4o
          api_key: os.environ/OPENAI_API_KEY

    litellm_settings:
    callbacks: ["qualifire_eval"]

    general_settings:
    master_key: "sk-1234"

    environment_variables:
    QUALIFIRE_API_KEY: "your-qualifire-api-key"
    QUALIFIRE_WEBHOOK_URL: "https://app.qualifire.ai/api/v1/webhooks/evaluations"

    ```
  </Step>

  <Step title="Start the proxy">
    ```bash theme={null}
    litellm --config config.yaml
    ```
  </Step>

  <Step title="Test it!">
    ```bash theme={null}
    curl -X POST 'http://0.0.0.0:4000/chat/completions' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer sk-1234' \
    -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hi 👋 - i'\''m openai"}]}'
    ```
  </Step>
</Steps>

## Environment Variables

<Info>
  Both environment variables are required. Get your API key and webhook URL from the [Qualifire dashboard](https://app.qualifire.ai/settings/api-keys).
</Info>

| Variable                | Description                                            |
| ----------------------- | ------------------------------------------------------ |
| `QUALIFIRE_API_KEY`     | Your Qualifire API key for authentication              |
| `QUALIFIRE_WEBHOOK_URL` | The Qualifire webhook endpoint URL from your dashboard |

## What Gets Logged?

<AccordionGroup>
  <Accordion title="Request Data" icon="arrow-up-right-from-square">
    Request messages, parameters, and model configuration sent to the LLM provider.
  </Accordion>

  <Accordion title="Response Data" icon="arrow-down-to-bracket">
    Response content, metadata, finish reason, and any tool calls returned by the model.
  </Accordion>

  <Accordion title="Usage & Performance" icon="gauge-high">
    Token usage statistics, latency metrics, cost data, and model information. The full [LiteLLM Standard Logging Payload](https://docs.litellm.ai/docs/proxy/logging_spec) is sent on each successful LLM API call.
  </Accordion>
</AccordionGroup>

Once data is in Qualifire, you can:

* Run evaluations to detect hallucinations, toxicity, and policy violations
* Set up guardrails to block or modify responses in real-time
* View traces across your entire AI pipeline
* Track performance and quality metrics over time

## Additional Resources

* [Qualifire Dashboard](https://app.qualifire.ai)
* [Qualifire Guardrails Integration](/integrations/litellm-guardrails)
* [LiteLLM Documentation](https://docs.litellm.ai)