Skip to main content

​
Evaluations

Evaluation is the fundamental building block for ensuring the integrity of your content. Think of it as a comprehensive test tailored to your specific needs. Familiarizing yourself with Evaluation is crucial, as many other features rely on or integrate with it.

​
Creating a new Evaluation

The process involves selecting one or more ‘checks’ – criteria you want to examine. The available checks are:
ensures that your content is secure, such as and avoiding sensitive information like PIIs and PHIs, and prompt injection detection.
a set of automated tools to scrutinize your content thoroughly. This includes checks for inconsistencies, missing content, identification of false facts and PII/PHI scanning for detecting sensitive information in your content.
Our “Assertions” feature stands out in our checks suite, offering a powerful tool to define specific policies in your content. Whether ensuring a acertain set of guardrails or avoiding mistakes, assertions allow you to express precise requirements. You can use our automatic assertion generator for effortless communication of expectations or take a hands-on approach by crafting your own assertions.
allows you to specify verification criteria to ensure that your content adheres to particular syntax requirements, such as length or format.
You have the flexibility to create Evaluations manually, choose a template from our library, or simply let our AI assistant handle it for you. The choice is yours!
On the Evaluations page, you can manage all your created Evaluations with ease (edit, duplicate or delete them if necessary).
An Evaluation on its own is just a set of checks, you can harness those checks in many ways like, experiments for regression tests, while using the prompt management as part of the prompt engineering phase and of course as a part of you active protection.

​
Qualifire’s Small Language Models (SLMs) Judges

Qualifire employs a suite of fine-tuned, state-of-the-art Small Language Models (SLMs), each specialized for a specific evaluation task. This provides faster, more accurate, and more targeted analysis of agent behavior. We offer to integrate the Qualifire state-of-the-art evaluations to your existing stack, supercharging your evaluation framework.
Sentinel is specialized in detecting prompt injections, ensuring the integrity and security of your agent’s interactions.
F1 Score: 0.988 | Latency: ~20 ms
Cleric is designed for content safety moderation, diligently filtering out inappropriate or harmful content to maintain a secure environment.
F1 Score: 0.946 | Latency: 35 ms
CategoryDescription
Dangerous ContentPromotes/facilitates harmful activities, self-harm, violence instructions
HarassmentAbusive behavior, bullying, targeting individuals/groups
Sexually Explicit InformationDepicts sexual acts, non-consensual sexual content, adult content
Hate SpeechPromotes violence, incites hatred, discrimination against protected groups
Paladin focuses on context grounding, ensuring that your agent’s responses are accurately rooted in provided information.
Balanced Accuracy: 98.48% | Latency: ~80 ms
Ranger evaluates tool selection quality, optimizing your AI agent’s ability to choose and utilize the most effective tools.
F1 Score: 0.945 | Latency: ~500 ms
Magistrate enforces any guardrail, standard or policy, ensuring agent behavior adheres strictly to predefined guidelines and guardrails.
F1 Score: 0.835 | Latency: ~100ms
Sage excels at hallucination detection, identifying and mitigating instances where the agent generates factually incorrect or unfounded information.
F1 Score: 0.8335 | Latency: ~250ms
Hunter is a PII detector, meticulously identifying and redacting personally identifiable information to enhance privacy.
F1 Score: 0.8335 | Latency: ~40ms