> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qualifire.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Using the CLI

> How to run Rogue evaluations from the command line.

## 🔧 CLI Mode

The CLI mode provides a **non-interactive** command-line interface for evaluating AI agents against predefined scenarios. It connects to the Rogue server to perform evaluations and is **ideal for CI/CD pipelines** and automated testing workflows.

## 🚀 Usage

The CLI mode requires the Rogue server to be running. You can either:

1. **Start server separately:**

   ```bash theme={null}
   # Terminal 1: Start the server
   uvx rogue-ai server

   # Terminal 2: Run CLI evaluation
   uvx rogue-ai cli [OPTIONS]
   ```

2. **Use the default mode (starts server + TUI, then use TUI for evaluation)**

For development or if you prefer to install locally:

```bash theme={null}
git clone https://github.com/qualifire-dev/rogue.git
cd rogue
uv sync
uv run -m rogue cli [OPTIONS]
```

Or, if you are using pip:

```bash theme={null}
git clone https://github.com/qualifire-dev/rogue.git
cd rogue
pip install -e .
uv run -m rogue cli [OPTIONS]
```

## 📓 CLI Arguments

> **Note**: CLI mode is **non-interactive** and designed for automated evaluation workflows, making it perfect for CI/CD pipelines.

| Argument                      | Required                                                | Default Value                   | Description                                                                                                                                             |
| ----------------------------- | ------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| --workdir                     | No                                                      | `./.rogue`                      | Directory to store outputs and defaults.                                                                                                                |
| --config-file                 | No                                                      | `<workdir>/user_config.json`    | Path to a config file generated by the UI. Values from this file are used unless overridden via CLI. If the file does not exist, only cli will be used. |
| --rogue-server-url            | No                                                      | `http://localhost:8000`         | URL of the Rogue server to connect to.                                                                                                                  |
| --evaluated-agent-url         | Yes                                                     |                                 | The URL of the agent to evaluate.                                                                                                                       |
| --evaluated-agent-auth-type   | No                                                      | `no_auth`                       | Auth method. Can be one of: `no_auth`, `api_key`, `bearer_token`, `basic`.                                                                              |
| --evaluated-agent-credentials | Yes\*<br />if `auth_type` is not `no_auth`              |                                 | Credentials for the agent (if required).                                                                                                                |
| --input-scenarios-file        | Yes                                                     | `<workdir>/scenarios.json`      | Path to scenarios file.                                                                                                                                 |
| --output-report-file          | No                                                      | `<workdir>/report.md`           | Where to save the markdown report.                                                                                                                      |
| --judge-llm                   | Yes                                                     |                                 | Model name for LLM evaluation (Litellm format).                                                                                                         |
| --judge-llm-api-key           | No                                                      |                                 | API key for LLM (see environment section).                                                                                                              |
| --business-context            | Yes\*<br />Unless `--business-context-file` is supplied |                                 | Business context as a string.                                                                                                                           |
| --business-context-file       | Yes\*<br />Unless `--business-context` is supplied      | `<workdir>/business_context.md` | OR path to file containing the business context.<br />If both given, `--business-context` has priority                                                  |
| --deep-test-mode              | No                                                      | `False`                         | Enables extended testing behavior.                                                                                                                      |
| --debug                       | No                                                      | `False`                         | Enable verbose logging.                                                                                                                                 |

## 📊 Config file

The config file is automatically generated when running the UI.
We will check for a config file in `<workdir>/user_config.json` and use it if it exists.
The config file is a JSON object that can contain all or a subset of the fields from the CLI arguments, except for `--config-file`.
Other keys in the config file are ignored.
Just remember to use snake\_case keys. (e.g. `--evaluated-agent-url` becomes `evaluated_agent_url`).

### Notes

1. ⚠️ Either `--business-context` or `--business-context-file` must be provided.
2. ⚠️ Fields marked as Required are required unless supplied via the config file.

## Examples

### With only a config file:

with our business context located at `./.rogue/business_context.md`

#### `./.rogue/user_config.json`

```json theme={null}
{
  "evaluated_agent_url": "http://localhost:10001",
  "judge_llm": "openai/o4-mini"
}
```

#### Execution

```bash theme={null}
uvx rogue-ai cli
```

### Same example without a config file:

#### Execution

```bash theme={null}
uvx rogue-ai cli \
    --evaluated-agent-url http://localhost:10001 \
    --judge-llm openai/o4-mini \
    --business-context-file './.rogue/business_context.md'
```