> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qualifire.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Using the CLI

> How to run Rogue evaluations from the command line.

## 🔧 CLI Mode

The CLI mode provides a **non-interactive** command-line interface for evaluating AI agents against predefined scenarios. It connects to the Rogue server to perform evaluations and is **ideal for CI/CD pipelines** and automated testing workflows.

## 🚀 Usage

The CLI mode requires the Rogue server to be running. You can either:

1. **Start server separately:**

   ```bash theme={null}
   # Terminal 1: Start the server
   uvx rogue-ai server

   # Terminal 2: Run CLI evaluation
   uvx rogue-ai cli [OPTIONS]
   ```

2. **Use the default mode (starts server + TUI, then use TUI for evaluation)**

For development or if you prefer to install locally:

```bash theme={null}
git clone https://github.com/qualifire-dev/rogue.git
cd rogue
uv sync
uv run -m rogue cli [OPTIONS]
```

Or, if you are using pip:

```bash theme={null}
git clone https://github.com/qualifire-dev/rogue.git
cd rogue
pip install -e .
uv run -m rogue cli [OPTIONS]
```

## 📓 CLI Arguments

> **Note**: CLI mode is **non-interactive** and designed for automated evaluation workflows, making it perfect for CI/CD pipelines.

| Argument | Required | Default Value | Description |
| - | - | - | - |
| --workdir | No | `./.rogue` | Directory to store outputs and defaults. |
| --config-file | No | `<workdir>/user_config.json` | Path to a config file generated by the UI. Values from this file are used unless overridden via CLI. If the file does not exist, only cli will be used. |
| --rogue-server-url | No | `http://localhost:8000` | URL of the Rogue server to connect to. |
| --evaluated-agent-url | Yes | | The URL of the agent to evaluate. |
| --evaluated-agent-auth-type | No | `no_auth` | Auth method. Can be one of: `no_auth`, `api_key`, `bearer_token`, `basic`. |
| --evaluated-agent-credentials | Yes\*<br />if `auth_type` is not `no_auth` | | Credentials for the agent (if required). |
| --input-scenarios-file | Yes | `<workdir>/scenarios.json` | Path to scenarios file. |
| --output-report-file | No | `<workdir>/report.md` | Where to save the markdown report. |
| --judge-llm | Yes | | Model name for LLM evaluation (Litellm format). |
| --judge-llm-api-key | No | | API key for LLM (see environment section). |
| --business-context | Yes\*<br />Unless `--business-context-file` is supplied | | Business context as a string. |
| --business-context-file | Yes\*<br />Unless `--business-context` is supplied | `<workdir>/business_context.md` | OR path to file containing the business context.<br />If both given, `--business-context` has priority |
| --deep-test-mode | No | `False` | Enables extended testing behavior. |
| --debug | No | `False` | Enable verbose logging. |

## 📊 Config file

The config file is automatically generated when running the UI.
We will check for a config file in `<workdir>/user_config.json` and use it if it exists.
The config file is a JSON object that can contain all or a subset of the fields from the CLI arguments, except for `--config-file`.
Other keys in the config file are ignored.
Just remember to use snake\_case keys. (e.g. `--evaluated-agent-url` becomes `evaluated_agent_url`).

### Notes

1. ⚠️ Either `--business-context` or `--business-context-file` must be provided.
2. ⚠️ Fields marked as Required are required unless supplied via the config file.

## Examples

### With only a config file:

with our business context located at `./.rogue/business_context.md`

#### `./.rogue/user_config.json`

```json theme={null}
{
  "evaluated_agent_url": "http://localhost:10001",
  "judge_llm": "openai/o4-mini"
}
```

#### Execution

```bash theme={null}
uvx rogue-ai cli
```

### Same example without a config file:

#### Execution

```bash theme={null}
uvx rogue-ai cli \
    --evaluated-agent-url http://localhost:10001 \
    --judge-llm openai/o4-mini \
    --business-context-file './.rogue/business_context.md'
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.