Library of pre-built evaluators
Faithfulness, answer relevance, tool-choice correctness, JSON schema validity, toxicity, PII leakage, tone, format compliance
Three evaluator types
LLM-as-judge, deterministic / statistical, and human-in-the-loop
Custom evaluators in Python
Bring your own logic, wired into the same pipeline
Evaluate at any level
Session, trace, or span — score whole conversations or single tool calls
Side-by-side experiment runs
Compare prompt v1 vs v2 across thousands of cases, see win/loss/tie
Production traces → datasets in one click
Curate real failures into golden test sets, no synthetic data drift
Continuous online evaluations
Score live traffic as it flows, surface regressions in minutes
Eval-of-evals
Every judge run is itself traced, so you can audit and trust the scoring
Evaluation explained
"It worked in testing" is how compliance violations, hallucinated refunds, and 3 a.m. pages happen. Most teams catch issues only after a user complains — because they have no systematic way to grade quality.
Netra makes evaluation a first-class step in your development loop. Define what good looks like for your agent — faithfulness to retrieved context, correct tool selection, valid JSON, on-policy tone, no leaked PII — then grade every change against it. Pick from our evaluator library, or write a custom one in Python and run it on the same infrastructure.
Run evaluations three ways: on datasets for offline regression testing before you ship, on experiment runs to compare two prompt versions head-to-head, and on live production traces to catch quality drops the moment they happen. Convert any cluster of production failures into a dataset in one click — your test set evolves with reality, not against it.
When a judge gives a verdict, you can inspect why: every LLM-as-judge execution is itself traced, so you see the exact prompt the judge received and the reasoning it returned. Trust the scores, debug the bad ones, and audit the whole chain for compliance.
Interested?
Everything You Need to Know About Evaluation
Get clear answers, fast — talk to our team and get tailored insights for your business in just a few minutes.
Yes — Auto-Evaluation runs Coherence, Factual Accuracy, and Toxicity on every incoming trace org-wide, until your configured usage limit.
Yes — via Netra MCP: browse/create evaluators, build datasets, and launch test runs directly from Cursor, Claude Code, Windsurf, etc.
Yes. A curated evaluator library across Quality, Multimodal, Guardrails, Performance, Semantic, and Agentic categories, each pre-configured with a pass threshold (e.g., Answer Relevance ≥0.7, Toxicity ≥0.5). One-click add, no setup.
Use the RAG-specific evaluators in the library — Context Precision, Context Recall, Context Relevance, Answer Correctness, Answer Relevance, Semantic Similarity, SQL Semantic Equivalence — or build a custom evaluator if none fit.
Defining the “expected output”/ source of truth for every possible input isn't practical for multi-turn agents. Use Simulation instead — define a goal, persona, and facts, and Netra runs realistic multi-turn conversations and scores them automatically.
Online eval gives you live monitoring of user interactions and agent behavior as it happens — no synthetic data needed. Pull real production traces directly into a dataset and run evaluators against actual user interactions, closing the loop between what you tested and what's really happening in production.
Yes. Evaluations score quality on a dataset or trigger; Alerts monitor cost, latency, error rate, and anomalies in real time across all production traffic — so you're notified the moment something breaks, not just when you run an eval.
Yes. Two ways: LLM-as-Judge for subjective/semantic criteria (correctness, relevance), or Code Evaluator (JS/Python) for deterministic checks like format validation, regex, and calculations.
