Skip to main content

​
🔧 CLI Mode

The CLI mode provides a non-interactive command-line interface for evaluating AI agents against predefined scenarios. It connects to the Rogue server to perform evaluations and is ideal for CI/CD pipelines and automated testing workflows.

​
🚀 Usage

The CLI mode requires the Rogue server to be running. You can either:
  1. Start server separately: 
    # Terminal 1: Start the server
uvx rogue-ai server

# Terminal 2: Run CLI evaluation
uvx rogue-ai cli [OPTIONS]
  2. Use the default mode (starts server + TUI, then use TUI for evaluation)
For development or if you prefer to install locally: 
git clone https://github.com/qualifire-dev/rogue.git
cd rogue
uv sync
uv run -m rogue cli [OPTIONS]
Or, if you are using pip: 
git clone https://github.com/qualifire-dev/rogue.git
cd rogue
pip install -e .
uv run -m rogue cli [OPTIONS]

​
📓 CLI Arguments

Note: CLI mode is non-interactive and designed for automated evaluation workflows, making it perfect for CI/CD pipelines.
ArgumentRequiredDefault ValueDescription
—workdirNo./.rogueDirectory to store outputs and defaults.
—config-fileNo<workdir>/user_config.jsonPath to a config file generated by the UI. Values from this file are used unless overridden via CLI. If the file does not exist, only cli will be used.
—rogue-server-urlNohttp://localhost:8000URL of the Rogue server to connect to.
—evaluated-agent-urlYesThe URL of the agent to evaluate.
—evaluated-agent-auth-typeNono_authAuth method. Can be one of: no_auth, api_key, bearer_token, basic.
—evaluated-agent-credentialsYes*
if auth_type is not no_auth		Credentials for the agent (if required).
—input-scenarios-fileYes<workdir>/scenarios.jsonPath to scenarios file.
—output-report-fileNo<workdir>/report.mdWhere to save the markdown report.
—judge-llmYesModel name for LLM evaluation (Litellm format).
—judge-llm-api-keyNoAPI key for LLM (see environment section).
—business-contextYes*
Unless --business-context-file is supplied		Business context as a string.
—business-context-fileYes*
Unless --business-context is supplied		<workdir>/business_context.mdOR path to file containing the business context.
If both given, --business-context has priority
—deep-test-modeNoFalseEnables extended testing behavior.
—debugNoFalseEnable verbose logging.

​
📊 Config file

The config file is automatically generated when running the UI. We will check for a config file in <workdir>/user_config.json and use it if it exists. The config file is a JSON object that can contain all or a subset of the fields from the CLI arguments, except for --config-file. Other keys in the config file are ignored. Just remember to use snake_case keys. (e.g. --evaluated-agent-url becomes evaluated_agent_url).

​
Notes

  1. ⚠️ Either --business-context or --business-context-file must be provided.
  2. ⚠️ Fields marked as Required are required unless supplied via the config file.

​
Examples

​
With only a config file:

with our business context located at ./.rogue/business_context.md

​
./.rogue/user_config.json

{
  "evaluated_agent_url": "http://localhost:10001",
  "judge_llm": "openai/o4-mini"
}

​
Execution

uvx rogue-ai cli

​
Same example without a config file:

​
Execution

uvx rogue-ai cli \
    --evaluated-agent-url http://localhost:10001 \
    --judge-llm openai/o4-mini \
    --business-context-file './.rogue/business_context.md'
⌘I