Using the CLI

🔧 CLI Mode

The CLI mode provides a non-interactive command-line interface for evaluating AI agents against predefined scenarios. It connects to the Rogue server to perform evaluations and is ideal for CI/CD pipelines and automated testing workflows.

🚀 Usage

The CLI mode requires the Rogue server to be running. You can either:

Start server separately:

# Terminal 1: Start the server
uvx rogue-ai server

# Terminal 2: Run CLI evaluation
uvx rogue-ai cli [OPTIONS]

Use the default mode (starts server + TUI, then use TUI for evaluation)

For development or if you prefer to install locally:

git clone https://github.com/qualifire-dev/rogue.git
cd rogue
uv sync
uv run -m rogue cli [OPTIONS]

Or, if you are using pip:

git clone https://github.com/qualifire-dev/rogue.git
cd rogue
pip install -e .
uv run -m rogue cli [OPTIONS]

📓 CLI Arguments

Note: CLI mode is non-interactive and designed for automated evaluation workflows, making it perfect for CI/CD pipelines.

Argument	Required	Default Value	Description
—workdir	No	`./.rogue`	Directory to store outputs and defaults.
—config-file	No	`<workdir>/user_config.json`	Path to a config file generated by the UI. Values from this file are used unless overridden via CLI. If the file does not exist, only cli will be used.
—rogue-server-url	No	`http://localhost:8000`	URL of the Rogue server to connect to.
—evaluated-agent-url	Yes		The URL of the agent to evaluate.
—evaluated-agent-auth-type	No	`no_auth`	Auth method. Can be one of: `no_auth`, `api_key`, `bearer_token`, `basic`.
—evaluated-agent-credentials	Yes* if `auth_type` is not `no_auth`		Credentials for the agent (if required).
—input-scenarios-file	Yes	`<workdir>/scenarios.json`	Path to scenarios file.
—output-report-file	No	`<workdir>/report.md`	Where to save the markdown report.
—judge-llm	Yes		Model name for LLM evaluation (Litellm format).
—judge-llm-api-key	No		API key for LLM (see environment section).
—business-context	Yes* Unless `--business-context-file` is supplied		Business context as a string.
—business-context-file	Yes* Unless `--business-context` is supplied	`<workdir>/business_context.md`	OR path to file containing the business context. If both given, `--business-context` has priority
—deep-test-mode	No	`False`	Enables extended testing behavior.
—debug	No	`False`	Enable verbose logging.

📊 Config file

The config file is automatically generated when running the UI. We will check for a config file in <workdir>/user_config.json and use it if it exists. The config file is a JSON object that can contain all or a subset of the fields from the CLI arguments, except for --config-file. Other keys in the config file are ignored. Just remember to use snake_case keys. (e.g. --evaluated-agent-url becomes evaluated_agent_url).

Notes

⚠️ Either --business-context or --business-context-file must be provided.
⚠️ Fields marked as Required are required unless supplied via the config file.

Examples

With only a config file:

with our business context located at ./.rogue/business_context.md

`./.rogue/user_config.json`

{
  "evaluated_agent_url": "http://localhost:10001",
  "judge_llm": "openai/o4-mini"
}

Execution

uvx rogue-ai cli

Same example without a config file:

Execution

uvx rogue-ai cli \
    --evaluated-agent-url http://localhost:10001 \
    --judge-llm openai/o4-mini \
    --business-context-file './.rogue/business_context.md'

Getting Started

How It Works

Protocols & Transports

Configuration

Policy Evaluation

Red Teaming

Examples

🔧 CLI Mode

🚀 Usage

📓 CLI Arguments

📊 Config file

Notes

Examples

With only a config file:

`./.rogue/user_config.json`

Execution

Same example without a config file:

Execution

​🔧 CLI Mode

​🚀 Usage

​📓 CLI Arguments

​📊 Config file

​Notes

​Examples

​With only a config file:

​./.rogue/user_config.json

​Execution

​Same example without a config file:

​Execution

🔧 CLI Mode

🚀 Usage

📓 CLI Arguments

📊 Config file

Notes

Examples

With only a config file:

`./.rogue/user_config.json`

Execution

Same example without a config file:

Execution