This tool allows you to evaluate AI agents against a set of predefined scenarios via the command line.

​
🚀 Usage

Clone the repo: 
git clone https://github.com/qualifire-dev/rogue.git
cd rogue
Run using uv: 
uv sync
uv run -m rogue cli [OPTIONS]
Or, if you are using pip: 
pip install -e .
python -m rogue cli [OPTIONS]

​
📓 CLI Arguments

ArgumentRequiredDefault ValueDescription
—workdirNo./.rogueDirectory to store outputs and defaults.
—config-fileNo<workdir>/user_config.jsonPath to a config file generated by the UI. Values from this file are used unless overridden via CLI. If the file does not exist, only cli will be used.
—evaluated-agent-urlYesThe URL of the agent to evaluate.
—evaluated-agent-auth-typeNono_authAuth method. Can be one of: no_auth, api_key, bearer_token, basic.
—evaluated-agent-credentialsYes*
if auth_type is not no_auth		Credentials for the agent (if required).
—input-scenarios-fileYes<workdir>/scenarios.jsonPath to scenarios file.
—output-report-fileNo<workdir>/report.mdWhere to save the markdown report.
—judge-llm-modelYesModel name for LLM evaluation (Litellm format).
—judge-llm-api-keyNoAPI key for LLM (see environment section).
—business-contextYes*
Unless --business-context-file is supplied		Business context as a string.
—business-context-fileYes*
Unless --business-context is supplied		<workdir>/business_context.mdOR path to file containing the business context.
If both given, --business-context has priority
—deep-test-modeNoFalseEnables extended testing behavior.
—debugNoFalseEnable verbose logging.

​
📊 Config file

The config file is automatically generated when running the UI. We will check for a config file in <workdir>/user_config.json and use it if it exists. The config file is a JSON object that can contain all or a subset of the fields from the CLI arguments, except for --config-file. Other keys in the config file are ignored. Just remember to use snake_case keys. (e.g. --evaluated-agent-url becomes evaluated_agent_url).

​
Notes

  1. ⚠️ Either --business-context or --business-context-file must be provided.
  2. ⚠️ Fields marked as Required are required unless supplied via the config file.

​
Examples

​
With only a config file:

with our business context located at ./.rogue/business_context.md

​
./.rogue/user_config.json

{
  "evaluated_agent_url": "http://localhost:10001",
  "judge_llm_model": "openai/o4-mini"
}

​
Execution

uv run -m rogue cli

​
Same example without a config file:

​
Execution

uv run -m rogue cli \
    --evaluated-agent-url http://localhost:10001 \
    --judge-llm-model openai/o4-mini \
    --business-context-file './.rogue/business_context.md'