> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qualifire.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# SDK

> Integrate Qualifire evaluations and tracing into your Node.js or Python application

## Installation & Setup

<Steps>
  <Step title="Install the SDK">
    <CodeGroup>
      ```bash Node.js theme={null}
      npm install qualifire
      ```

      ```bash Python theme={null}
      pip install qualifire
      ```
    </CodeGroup>
  </Step>

  <Step title="Initialize the client">
    <CodeGroup>
      ```typescript Node.js theme={null}
      import { Qualifire } from "qualifire";

      const qualifire = new Qualifire({
        apiKey: "YOUR_API_KEY",       // Optional: defaults to QUALIFIRE_API_KEY env var
        baseUrl: "https://api.qualifire.ai", // Optional: custom base URL
      });
      ```

      ```python Python theme={null}
      import qualifire

      client = qualifire.client.Client(
          api_key="YOUR_API_KEY",      # Optional: defaults to QUALIFIRE_API_KEY env var
          base_url="https://...",      # Optional: custom base URL
          version="v1",               # Optional: API version
          debug=False,                 # Optional: enable debug mode
          verify=True,                 # Optional: SSL verification
      )
      ```
    </CodeGroup>

    <Tip>If the `apiKey` / `api_key` argument is not provided, the SDK will look for a value in the environment variable `QUALIFIRE_API_KEY`.</Tip>
  </Step>
</Steps>

## Running Evaluations

### Quick Start

Pass simple input/output strings to run checks:

<CodeGroup>
  ```typescript Node.js theme={null}
  const response = await qualifire.evaluate({
    input: "What is the capital of France?",
    output: "Paris",
    contentModerationCheck: true,
    hallucinationsCheck: true,
  });
  ```

  ```python Python theme={null}
  res = client.evaluate(
      input="What is the capital of France?",
      output="Paris",
      content_moderation_check=True,
      hallucinations_check=True,
  )
  ```
</CodeGroup>

### Messages Mode

Send parsed messages directly for evaluation:

<CodeGroup>
  ```typescript Node.js theme={null}
  const response = await qualifire.evaluate({
    messages: [
      { role: "user", content: "What is the capital of France?" },
      { role: "assistant", content: "Paris" },
    ],
    contentModerationCheck: true,
    hallucinationsCheck: true,
    groundingCheck: true,
    piiCheck: true,
    promptInjections: true,
    assertions: ["don't give medical advice"],
    allowedTopics: ["billing", "account management", "technical support"],
  });
  ```

  ```python Python theme={null}
  res = client.evaluate(
      input="what is the capital of France",
      output="Paris",
      prompt_injections=True,
      pii_check=True,
      hallucinations_check=True,
      grounding_check=True,
      content_moderation_check=True,
      assertions=["don't give medical advice"],
      allowed_topics=["billing", "account management", "technical support"],
  )
  ```
</CodeGroup>

### Request-Response Mode

<Info>**Node.js only.** Supported frameworks: `openai`, `vercelai`, `gemini`, `claude`</Info>

Pass the original request and response objects along with the framework name:

```typescript theme={null}
import { Qualifire } from "qualifire";
import OpenAI from "openai";

const qualifire = new Qualifire({ apiKey: "YOUR_QUALIFIRE_API_KEY" });
const openai = new OpenAI({ apiKey: "YOUR_OPENAI_API_KEY" });

const openAiRequest = {
  model: "gpt-4o",
  messages: [
    {
      role: "system",
      content: "You are a helpful assistant that can answer questions.",
    },
    {
      role: "user",
      content: [{ type: "text", text: "Is the sky blue?" }],
    },
  ],
};

const openAiResponse = await openai.chat.completions.create(openAiRequest);

const qualifireResponse = await qualifire.evaluate({
  framework: "openai",
  request: openAiRequest,
  response: openAiResponse,
  contentModerationCheck: true,
  groundingCheck: true,
  hallucinationsCheck: true,
  instructionsFollowingCheck: true,
  piiCheck: true,
  promptInjections: true,
  toolSelectionQualityCheck: false,
});
```

### Streaming Mode

<Info>**Node.js only.** Collect streaming chunks and pass them as an array.</Info>

```typescript theme={null}
import { Qualifire } from "qualifire";
import OpenAI from "openai";

const qualifire = new Qualifire({ apiKey: "YOUR_QUALIFIRE_API_KEY" });
const openai = new OpenAI({ apiKey: "YOUR_OPENAI_API_KEY" });

const openAiRequest = {
  stream: true,
  model: "gpt-4o",
  messages: [
    {
      role: "system",
      content: "You are a helpful assistant that can answer questions.",
    },
    {
      role: "user",
      content: [{ type: "text", text: "Is the sky blue?" }],
    },
  ],
};

const openAiResponseStream = await openai.chat.completions.create(openAiRequest);

const responseChunks: any[] = [];
for await (const chunk of openAiResponseStream) {
  responseChunks.push(chunk);
}

const qualifireResponse = await qualifire.evaluate({
  framework: "openai",
  request: openAiRequest,
  response: responseChunks,
  groundingCheck: true,
  promptInjections: true,
});
```

### Invoke by ID

Invoke a pre-configured evaluation by its ID:

<CodeGroup>
  ```typescript Node.js theme={null}
  // Simple input/output
  const response = await qualifire.invokeEvaluation({
    input: "What is the capital of France?",
    output: "Paris",
    evaluationId: "g2r8puzojwb8q6yi2f6x162a", // Get this from the evaluations page
  });

  // With messages and tools (for tool use quality evaluation)
  const response = await qualifire.invokeEvaluation({
    evaluationId: "g2r8puzojwb8q6yi2f6x162a",
    messages: [
      { role: "user", content: "What's the weather in NYC?" },
      { role: "assistant", content: "Let me check.", tool_calls: [{ name: "get_weather", arguments: { location: "NYC" } }] },
    ],
    availableTools: [
      { name: "get_weather", description: "Get weather for a location", parameters: { type: "object", properties: { location: { type: "string" } } } },
    ],
  });
  ```

  ```python Python theme={null}
  # Simple input/output
  res = client.invoke_evaluation(
      input="what is the capital of France",
      output="Paris",
      evaluation_id="g2r8puzojwb8q6yi2f6x162a",  # Get this from the evaluations page
  )

  # With messages and tools (for tool use quality evaluation)
  from qualifire.types import LLMMessage, LLMToolCall, LLMToolDefinition

  res = client.invoke_evaluation(
      evaluation_id="g2r8puzojwb8q6yi2f6x162a",
      messages=[
          LLMMessage(role="user", content="What's the weather in NYC?"),
          LLMMessage(role="assistant", content="Let me check.", tool_calls=[
              LLMToolCall(name="get_weather", arguments={"location": "NYC"}),
          ]),
      ],
      available_tools=[
          LLMToolDefinition(name="get_weather", description="Get weather for a location", parameters={"type": "object", "properties": {"location": {"type": "string"}}}),
      ],
  )
  ```
</CodeGroup>

## Evaluation Response

<CodeGroup>
  ```typescript Node.js theme={null}
  console.log(response?.status); // "passed" or "failed"
  console.log(response?.score);  // Overall score (0-100)

  response?.evaluationResults.forEach((item) => {
    console.log(`Type: ${item.type}`);
    item.results.forEach((result) => {
      console.log(`  - ${result.name}: ${result.label} (score: ${result.score})`);
      console.log(`    Reason: ${result.reason}`);
    });
  });
  ```

  ```python Python theme={null}
  print(res.status)  # Status of the evaluation
  print(res.score)   # Overall score

  for item in res.evaluationResults:
      print(f"Type: {item.type}")
      for result in item.results:
          print(f"  - {result.name}: {result.label} (score: {result.score})")
          print(f"    Reason: {result.reason}")
  ```
</CodeGroup>

```json Example Output theme={null}
{
  "status": "failed",
  "score": 75,
  "evaluationResults": [
    {
      "type": "grounding",
      "results": [
        {
          "name": "grounding",
          "score": 75,
          "label": "INFERABLE",
          "confidence_score": 100,
          "reason": "The AI's output provides a detailed explanation...",
          "flagged": true
        }
      ]
    },
    {
      "type": "policy",
      "results": [
        {
          "name": "policy",
          "score": 100,
          "label": "PASS",
          "confidence_score": 100,
          "reason": "The output follows the assertion.",
          "flagged": false,
          "data": "don't give medical advice"
        }
      ]
    }
  ]
}
```

## Advanced Configuration

<AccordionGroup>
  <Accordion title="Mode Configuration">
    Control the quality/speed tradeoff for each check:

    | Mode       | Description              |
    | ---------- | ------------------------ |
    | `speed`    | Fastest, lower accuracy  |
    | `balanced` | Default balance          |
    | `quality`  | Highest accuracy, slower |

    <CodeGroup>
      ```typescript Node.js theme={null}
      const response = await qualifire.evaluate({
        messages: [
          { role: "user", content: "What is the capital of France?" },
          { role: "assistant", content: "Paris" },
        ],
        hallucinationsCheck: true,
        groundingCheck: true,
        assertions: ["don't give medical advice"],
        hallucinationsMode: "quality",
        groundingMode: "balanced",
        assertionsMode: "speed",
        consistencyMode: "balanced",
      });
      ```

      ```python Python theme={null}
      from qualifire.types import ModelMode

      res = client.evaluate(
          input="what is the capital of France",
          output="Paris",
          hallucinations_check=True,
          grounding_check=True,
          assertions=["don't give medical advice"],
          hallucinations_mode=ModelMode.QUALITY,
          grounding_mode=ModelMode.BALANCED,
          assertions_mode=ModelMode.SPEED,
          consistency_mode=ModelMode.BALANCED,
      )
      ```
    </CodeGroup>
  </Accordion>

  <Accordion title="Multi-Turn Evaluation">
    Enable multi-turn context for grounding and policy checks:

    <CodeGroup>
      ```typescript Node.js theme={null}
      const response = await qualifire.evaluate({
        messages: [...],
        groundingCheck: true,
        groundingMultiTurnMode: true,
        policyMultiTurnMode: true,
      });
      ```

      ```python Python theme={null}
      res = client.evaluate(
          messages=[...],
          grounding_check=True,
          grounding_multi_turn_mode=True,
          policy_multi_turn_mode=True,
      )
      ```
    </CodeGroup>
  </Accordion>

  <Accordion title="Topic Scoping">
    Restrict conversations to allowed topics:

    <CodeGroup>
      ```typescript Node.js theme={null}
      const response = await qualifire.evaluate({
        messages: [...],
        topicScopingMode: "balanced",
        topicScopingMultiTurnMode: true,
        topicScopingTarget: "output",
        allowedTopics: ["billing", "account management", "technical support"],
      });
      ```

      ```python Python theme={null}
      res = client.evaluate(
          messages=[...],
          topic_scoping_mode=ModelMode.BALANCED,
          topic_scoping_multi_turn_mode=True,
          topic_scoping_target="output",
          allowed_topics=["billing", "account management", "technical support"],
      )
      ```
    </CodeGroup>
  </Accordion>

  <Accordion title="Tool Calls">
    Evaluate tool selection quality (Python example):

    ```python theme={null}
    from qualifire.types import LLMMessage, LLMToolCall, LLMToolDefinition, ModelMode

    res = client.evaluate(
        messages=[
            LLMMessage(
                role="user",
                content="What is the weather tomorrow in New York?",
            ),
            LLMMessage(
                role="assistant",
                content="please run the following tool",
                tool_calls=[
                    LLMToolCall(
                        id="tool_call_id",
                        name="get_weather_forecast",
                        arguments={
                            "location": "New York, NY",
                            "date": "tomorrow",
                        },
                    ),
                ],
            ),
        ],
        available_tools=[
            LLMToolDefinition(
                name="get_weather_forecast",
                description="Provides the weather forecast for a given location and date.",
                parameters={
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA",
                        },
                        "date": {
                            "type": "string",
                            "description": "The date for the forecast, e.g., tomorrow, or YYYY-MM-DD",
                        },
                    },
                    "required": ["location", "date"],
                },
            ),
        ],
        tool_use_quality_check=True,
        tuq_mode=ModelMode.BALANCED,
    )
    ```
  </Accordion>

  <Accordion title="Metadata">
    Attach custom key-value metadata to any evaluation. Metadata is persisted alongside the invocation and can be used for filtering and grouping in the Qualifire UI.

    All values must be strings. The API returns a 422 error if any value is not a string.

    <CodeGroup>
      ```typescript Node.js theme={null}
      const response = await qualifire.evaluate({
        input: "What is the capital of France?",
        output: "Paris",
        hallucinationsCheck: true,
        metadata: {
          environment: "production",
          userId: "user-123",
          sessionId: "sess-abc",
        },
      });
      ```

      ```python Python theme={null}
      res = client.evaluate(
          input="What is the capital of France?",
          output="Paris",
          hallucinations_check=True,
          metadata={
              "environment": "production",
              "user_id": "user-123",
              "session_id": "sess-abc",
          },
      )
      ```
    </CodeGroup>

    Metadata also works with `invokeEvaluation` / `invoke_evaluation`:

    <CodeGroup>
      ```typescript Node.js theme={null}
      const response = await qualifire.invokeEvaluation({
        input: "What is the capital of France?",
        output: "Paris",
        evaluationId: "g2r8puzojwb8q6yi2f6x162a",
        metadata: { environment: "staging" },
      });
      ```

      ```python Python theme={null}
      res = client.invoke_evaluation(
          input="What is the capital of France?",
          output="Paris",
          evaluation_id="g2r8puzojwb8q6yi2f6x162a",
          metadata={"environment": "staging"},
      )
      ```
    </CodeGroup>
  </Accordion>

  <Accordion title="Policy Target">
    Control whether checks apply to input, output, or both:

    <CodeGroup>
      ```typescript Node.js theme={null}
      const response = await qualifire.evaluate({
        messages: [...],
        policyTarget: "output", // "input" | "output" | "both"
      });
      ```

      ```python Python theme={null}
      from qualifire.types import PolicyTarget

      res = client.evaluate(
          messages=[...],
          policy_target=PolicyTarget.OUTPUT,  # INPUT, OUTPUT, or BOTH
      )
      ```
    </CodeGroup>
  </Accordion>

  <Accordion title="Policy Include Tools">
    Include tool definitions and tool calls in the policy assertion context. When enabled, assertions can reference available tools and tool call arguments — for example, "must use the search tool before answering".

    <CodeGroup>
      ```typescript Node.js theme={null}
      const response = await qualifire.evaluate({
        messages: [
          { role: "user", content: "Find the weather in NYC" },
          { role: "assistant", content: "Let me check.", tool_calls: [{ name: "get_weather", arguments: { location: "NYC" } }] },
        ],
        assertions: ["must use the get_weather tool"],
        policyIncludeTools: true,
      });
      ```

      ```python Python theme={null}
      from qualifire.types import LLMMessage, LLMToolCall, LLMToolDefinition

      res = client.evaluate(
          messages=[
              LLMMessage(role="user", content="Find the weather in NYC"),
              LLMMessage(
                  role="assistant",
                  content="Let me check.",
                  tool_calls=[LLMToolCall(name="get_weather", arguments={"location": "NYC"})],
              ),
          ],
          available_tools=[
              LLMToolDefinition(
                  name="get_weather",
                  description="Get weather for a location",
                  parameters={"type": "object", "properties": {"location": {"type": "string"}}},
              ),
          ],
          assertions=["must use the get_weather tool"],
          policy_include_tools=True,
      )
      ```
    </CodeGroup>
  </Accordion>
</AccordionGroup>

## Types Reference

<AccordionGroup>
  <Accordion title="Node.js Types">
    ```typescript theme={null}
    import type {
      EvaluationProxyAPIRequest,
      EvaluationRequestV2,
      EvaluationResponse,
      Framework,
      LLMMessage,
      ModelMode,
      PolicyTarget,
    } from "qualifire";

    // Framework - supported LLM frameworks
    type Framework = "openai" | "vercelai" | "gemini" | "claude";

    // ModelMode - controls quality/speed tradeoff for checks
    type ModelMode = "speed" | "balanced" | "quality";

    // PolicyTarget - specifies what to check
    type PolicyTarget = "input" | "output" | "both";

    // LLMMessage - message format for evaluations
    interface LLMMessage {
      role: string;
      content?: string;
      tool_calls?: LLMToolCall[];
    }
    ```
  </Accordion>

  <Accordion title="Python Types">
    ```python theme={null}
    from qualifire.types import (
        LLMMessage,
        LLMToolCall,
        LLMToolDefinition,
        ModelMode,
        PolicyTarget,
    )

    # ModelMode - controls quality/speed tradeoff for checks
    ModelMode.SPEED      # Fastest, lower accuracy
    ModelMode.BALANCED   # Default balance
    ModelMode.QUALITY    # Highest accuracy, slower

    # PolicyTarget - specifies what to check
    PolicyTarget.INPUT   # Check only input
    PolicyTarget.OUTPUT  # Check only output
    PolicyTarget.BOTH    # Check both (default)

    # Message types
    message = LLMMessage(
        role="user",
        content="Hello, world!",
        tool_calls=None,  # Optional list of LLMToolCall
    )

    tool_call = LLMToolCall(
        name="get_weather",
        arguments={"location": "New York"},
        id="call_123",  # Optional
    )

    tool_definition = LLMToolDefinition(
        name="get_weather",
        description="Get weather for a location",
        parameters={
            "type": "object",
            "properties": {
                "location": {"type": "string"}
            },
            "required": ["location"]
        },
    )
    ```
  </Accordion>
</AccordionGroup>

## Instrumentation (Tracing)

<Steps>
  <Step title="Initialize tracing">
    <CodeGroup>
      ```typescript Node.js theme={null}
      import { Qualifire } from "qualifire";

      const qualifire = new Qualifire({ apiKey: "YOUR_QUALIFIRE_API_KEY" });
      qualifire.init();
      ```

      ```python Python theme={null}
      import qualifire

      qualifire.init(
          api_key="YOUR_QUALIFIRE_API_KEY",
      )
      ```
    </CodeGroup>
  </Step>

  <Step title="Configure your LLM client to use the Qualifire proxy">
    <CodeGroup>
      ```typescript Node.js theme={null}
      import OpenAI from "openai";

      const openai = new OpenAI({
        apiKey: "YOUR_OPENAI_API_KEY",
        baseUrl: "https://proxy.qualifire.ai/api/providers/openai",
        defaultHeaders: {
          "X-Qualifire-API-Key": "YOUR_QUALIFIRE_API_KEY",
        },
      });
      ```

      ```python Python theme={null}
      from openai import OpenAI

      client = OpenAI(
          api_key="YOUR_OPENAI_API_KEY",
          base_url="https://proxy.qualifire.ai/api/providers/openai",
          default_headers={
              "X-Qualifire-API-Key": "YOUR_QUALIFIRE_API_KEY",
          },
      )
      ```
    </CodeGroup>
  </Step>

  <Step title="Make requests as usual">
    <CodeGroup>
      ```typescript Node.js theme={null}
      const response = await openai.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: "Tell me a joke" }],
      });
      ```

      ```python Python theme={null}
      res = client.chat.completions.create(
          model="gpt-4o",
          messages=[{"role": "user", "content": "Tell me a joke"}],
      )
      ```
    </CodeGroup>

    Evaluations and traces will appear in the Qualifire web UI.
  </Step>
</Steps>

<Accordion title="Agent Tracing (LangGraph)">
  Python example using LangGraph:

  ```python theme={null}
  import qualifire
  from langchain.chat_models import init_chat_model
  from langgraph.prebuilt import create_react_agent

  qualifire.init(api_key="YOUR_QUALIFIRE_API_KEY")

  tools = ...

  llm = init_chat_model(
      "openai:gpt-4.1",
      api_key="YOUR_OPENAI_API_KEY",
      base_url="https://proxy.qualifire.ai/api/providers/openai/",
      default_headers={
          "X-Qualifire-API-Key": "YOUR_QUALIFIRE_API_KEY",
      },
  )
  agent = create_react_agent(llm, tools, prompt="system prompt...")

  question = "Tell me a joke"
  for step in agent.stream(
      {"messages": [{"role": "user", "content": question}]},
      stream_mode="values",
  ):
      step["messages"][-1].pretty_print()
  ```
</Accordion>

## Deprecated Parameters

<Accordion title="Deprecated parameter mappings">
  The following parameters are deprecated and will automatically enable `contentModerationCheck` / `content_moderation_check`:

  | Deprecated                                          | Use Instead                                           |
  | --------------------------------------------------- | ----------------------------------------------------- |
  | `dangerousContentCheck` / `dangerous_content_check` | `contentModerationCheck` / `content_moderation_check` |
  | `harassmentCheck` / `harassment_check`              | `contentModerationCheck` / `content_moderation_check` |
  | `hateSpeechCheck` / `hate_speech_check`             | `contentModerationCheck` / `content_moderation_check` |
  | `sexualContentCheck` / `sexual_content_check`       | `contentModerationCheck` / `content_moderation_check` |

  Snake\_case variants are also deprecated in favor of camelCase (Node.js):

  | Deprecated                     | Use Instead                  |
  | ------------------------------ | ---------------------------- |
  | `grounding_check`              | `groundingCheck`             |
  | `hallucinations_check`         | `hallucinationsCheck`        |
  | `pii_check`                    | `piiCheck`                   |
  | `prompt_injections`            | `promptInjections`           |
  | `tool_selection_quality_check` | `toolSelectionQualityCheck`  |
  | `instructions_following_check` | `instructionsFollowingCheck` |
</Accordion>

<Note>API Reference documentation is [here](/api-reference/introduction).</Note>