Rogue automates the creation of test scenarios to ensure comprehensive testing of your agent’s capabilities and policies.

​
Business Context

The process starts with you providing a “business context”. This is a high-level description of your agent’s purpose, what it’s supposed to do, and any important guardrails or policies it must follow.

You can provide this context in two ways:

  1. AI-Powered Interview: Rogue can guide you through a short, conversational interview to extract the business context. An AI interviewer will ask you up to 5 questions to understand your agent’s domain and critical risk areas.
  2. Manual Entry: You can directly write or paste the business context into a text box.

​
Scenario Creation

Once the business context is defined, Rogue’s LLM Service uses it to generate a list of test scenarios. A powerful LLM (the “Service LLM” you configure) is prompted with the context and a set of instructions to design effective test cases.

The generation focuses on:

  • Edge Cases: Creating scenarios that test the boundaries of the agent’s policies.
  • Basic Functionality: Including a few scenarios to validate core behavior.
  • Policy Compliance: Ensuring the agent adheres to its defined guardrails.

Rogue also includes a static set of tests for common vulnerabilities, such as Prompt Injection. These are added to the dynamically generated scenarios.

The final list of scenarios is presented to you in JSON format, which you can review and even edit before starting the evaluation run.