Use Case Advisor & Validation Runs
1. Overview
OpenVals provides a workflow for describing a business use case, receiving model recommendations, and validating selected models. The Use Case Advisor recommends suitable models using online provider models, locally installed Ollama models, or both. Validation Runs execute configured checks and present an evidence pack and report.
End-to-end workflow
2. Use Case Advisor
The page uses a two-column layout: the intake form on the left and a Recommended Decision panel on the right. Previous submissions and recommendations can be restored from history or cached state.
Starting or restoring a decision
On first visit, OpenVals normally restores the most recent history entry unless a run is in progress or cached results are available. Submission and recommendation data are retained in the application query cache while navigating. Select New Decision in the header, or Cancel where available, to reset the form to an empty submission.
Complete the intake form
- Use case title.
- Domain (choose from the available domain options).
- Problem statement.
- Data type.
- Primary priority.
- Business risk.
- Evaluation mode: Online, Offline, or Both.
The form is validated before submission. Run Recommendation remains disabled until the required title, problem statement, and evaluation mode are supplied.
Run a recommendation
Select Run Recommendation. OpenVals creates an asynchronous recommendation job and returns a job ID. The page polls the job status every five seconds, up to a 45-minute deadline. A minimum pending display time helps ensure the progress UI is visible.
Understand progress
| Evaluation track | What it does |
|---|---|
| Offline | Synchronizes installed Ollama models, runs live probe evaluations, and scores results. |
| Online | Resolves models available through connected providers, then scores and ranks them. |
| Both | Runs online and offline tracks in parallel. One track may fail while the other completes. |
Review recommendations
Ranked recommendation cards can show a fit-score ring, fit and risk badges, and five metric bars: accuracy, safety, reliability, latency, and compliance. Expand a card to inspect probe evidence, including grounded, adversarial, and PII checks.
Errors and partial results
Online evaluation failures may appear in a rose banner; partial failures may appear in an amber banner. Provider-specific messages can indicate insufficient balance or credits, or a model that is not served by the selected provider. Follow the displayed guidance, verify the provider account and model availability, and use Retry when offered. In Both mode, review results from the track that completed.
History and versions
The use-case dropdown lists saved use cases deduplicated by title. The version dropdown lists each run/version for the selected title. Selecting a history entry fetches its stored details and restores the submission and recommendations without rerunning the job. The title field becomes read-only while a history item is selected. Rerunning the same title creates a new version.
Select models for validation
Use Select for validation on individual cards or Select all. Starting a new recommendation clears the current selection. Once at least one model is selected, choose Run Validation. OpenVals carries the selected model IDs, use-case title, exact use-case version ID, and evaluation mode into Validation Runs.
3. Validation Runs
The page shows the run configuration form on the left and audit history on the right, with progress and the latest evidence pack below. Validation can be started from the Use Case Advisor or directly.
Choose an entry path
| Entry path | Behavior |
|---|---|
| From Use Case Advisor | The promoted model selection, use case, and evaluation mode are locked to the advisor selection. The use-case version ID is submitted to scope validation to that version's recommendations. |
| Direct visit | The form is editable. Use cases are selected from previously submitted use cases when available, with presets as a fallback. The model list is scoped to the selected use case. |
Configure the run
- Models: select one or more models. Ollama models are associated with offline evaluation; connected-provider models are associated with online evaluation.
- Evaluation mode: Online, Offline, or the mode passed from the advisor.
- Use case: choose the use case being validated.
- Suite profile: select the appropriate validation suite profile.
- Included checks: hallucination, prompt injection, PII, bias drift, latency SLO, and audit trail.
Submit and track progress
Submit the form to queue the validation gate. The interface shows staged progress for provisioning a sandbox, loading the adversarial corpus, executing suites, scoring thresholds and the trust index, and compiling the audit trail. These stages communicate progress; the backend run status determines the actual outcome.
How validation executes
OpenVals sends a validation request with the selected model or models, use-case name and ID, suite IDs, and Standard strictness. The backend fans out a run per model and associates the runs with a shared batch ID. Each returned run is polled until it completes or fails, subject to the configured polling limit. The history list refreshes periodically after submission to reconcile batch runs not echoed in the initial response.
Read the evidence pack
- OpenVals trust score.
- Security status.
- Decision label, such as Pilot ready, Passed with guardrails, Needs monitoring, Failed, or Blocked.
- Summaries of passed checks and checks needing monitoring.
- For multi-model batches, the strongest evidence pack among the completed model runs.
Queued or failed runs may show a limited evidence card. Full evidence is published when the run completes and produces results. Check the run status and explanatory message to determine whether evidence is final.
History, filters, and reports
Validation history lists terminal runs (completed or failed). Use the use-case filter to narrow the history; it also scopes the displayed evidence pack and report. Select Preview Report to open the trust report modal, or Download Report to download a PDF.