OpenVals
4. Use the platform

Use Case Advisor & Validation Runs

A practical guide to model recommendations, validation, evidence, and reporting.

1. Overview

OpenVals provides a workflow for describing a business use case, receiving model recommendations, and validating selected models. The Use Case Advisor recommends suitable models using online provider models, locally installed Ollama models, or both. Validation Runs execute configured checks and present an evidence pack and report.

End-to-end workflow

1Describe the use case and choose an evaluation mode in Use Case Advisor.
2Run the recommendation job and monitor progress.
3Review ranked model recommendations, fit/risk indicators, metrics, and probe evidence.
4Select one or more models and choose Run Validation.
5Review validation status, trust score, decision, and evidence pack.
6Preview or download the trust report.

2. Use Case Advisor

The page uses a two-column layout: the intake form on the left and a Recommended Decision panel on the right. Previous submissions and recommendations can be restored from history or cached state.

Starting or restoring a decision

On first visit, OpenVals normally restores the most recent history entry unless a run is in progress or cached results are available. Submission and recommendation data are retained in the application query cache while navigating. Select New Decision in the header, or Cancel where available, to reset the form to an empty submission.

Complete the intake form

  • Use case title.
  • Domain (choose from the available domain options).
  • Problem statement.
  • Data type.
  • Primary priority.
  • Business risk.
  • Evaluation mode: Online, Offline, or Both.

The form is validated before submission. Run Recommendation remains disabled until the required title, problem statement, and evaluation mode are supplied.

Run a recommendation

Select Run Recommendation. OpenVals creates an asynchronous recommendation job and returns a job ID. The page polls the job status every five seconds, up to a 45-minute deadline. A minimum pending display time helps ensure the progress UI is visible.

Understand progress

Evaluation trackWhat it does
OfflineSynchronizes installed Ollama models, runs live probe evaluations, and scores results.
OnlineResolves models available through connected providers, then scores and ranks them.
BothRuns online and offline tracks in parallel. One track may fail while the other completes.

Review recommendations

Ranked recommendation cards can show a fit-score ring, fit and risk badges, and five metric bars: accuracy, safety, reliability, latency, and compliance. Expand a card to inspect probe evidence, including grounded, adversarial, and PII checks.

Errors and partial results

Online evaluation failures may appear in a rose banner; partial failures may appear in an amber banner. Provider-specific messages can indicate insufficient balance or credits, or a model that is not served by the selected provider. Follow the displayed guidance, verify the provider account and model availability, and use Retry when offered. In Both mode, review results from the track that completed.

History and versions

The use-case dropdown lists saved use cases deduplicated by title. The version dropdown lists each run/version for the selected title. Selecting a history entry fetches its stored details and restores the submission and recommendations without rerunning the job. The title field becomes read-only while a history item is selected. Rerunning the same title creates a new version.

Select models for validation

Use Select for validation on individual cards or Select all. Starting a new recommendation clears the current selection. Once at least one model is selected, choose Run Validation. OpenVals carries the selected model IDs, use-case title, exact use-case version ID, and evaluation mode into Validation Runs.

3. Validation Runs

The page shows the run configuration form on the left and audit history on the right, with progress and the latest evidence pack below. Validation can be started from the Use Case Advisor or directly.

Choose an entry path

Entry pathBehavior
From Use Case AdvisorThe promoted model selection, use case, and evaluation mode are locked to the advisor selection. The use-case version ID is submitted to scope validation to that version's recommendations.
Direct visitThe form is editable. Use cases are selected from previously submitted use cases when available, with presets as a fallback. The model list is scoped to the selected use case.

Configure the run

  • Models: select one or more models. Ollama models are associated with offline evaluation; connected-provider models are associated with online evaluation.
  • Evaluation mode: Online, Offline, or the mode passed from the advisor.
  • Use case: choose the use case being validated.
  • Suite profile: select the appropriate validation suite profile.
  • Included checks: hallucination, prompt injection, PII, bias drift, latency SLO, and audit trail.

Submit and track progress

Submit the form to queue the validation gate. The interface shows staged progress for provisioning a sandbox, loading the adversarial corpus, executing suites, scoring thresholds and the trust index, and compiling the audit trail. These stages communicate progress; the backend run status determines the actual outcome.

How validation executes

OpenVals sends a validation request with the selected model or models, use-case name and ID, suite IDs, and Standard strictness. The backend fans out a run per model and associates the runs with a shared batch ID. Each returned run is polled until it completes or fails, subject to the configured polling limit. The history list refreshes periodically after submission to reconcile batch runs not echoed in the initial response.

Read the evidence pack

  • OpenVals trust score.
  • Security status.
  • Decision label, such as Pilot ready, Passed with guardrails, Needs monitoring, Failed, or Blocked.
  • Summaries of passed checks and checks needing monitoring.
  • For multi-model batches, the strongest evidence pack among the completed model runs.

Queued or failed runs may show a limited evidence card. Full evidence is published when the run completes and produces results. Check the run status and explanatory message to determine whether evidence is final.

History, filters, and reports

Validation history lists terminal runs (completed or failed). Use the use-case filter to narrow the history; it also scopes the displayed evidence pack and report. Select Preview Report to open the trust report modal, or Download Report to download a PDF.

4. Troubleshooting

Recommendation is disabled
Complete the title, problem statement, and evaluation mode.
Online recommendation fails
Check provider credentials, account balance/credits, model serving availability, and the displayed provider error.
Only one track returns results in Both mode
Review the completed track and the error banner for the failed track; retry after resolving the issue.
A model is missing from validation choices
Confirm that the model is available for the selected evaluation mode and associated with the selected use case.
Evidence is not available yet
Check whether the run is queued or running. Evidence appears when the run completes and results are available.
A previous decision or run is needed
Use Use Case Advisor history/version selectors or the Validation Runs history and use-case filter.

5. Key concepts

Evaluation mode
Determines whether online provider models, offline Ollama models, or both are evaluated.
Use-case version
A specific saved recommendation run for a use-case title.
Batch run
A validation submission for multiple models, with one individual run per model and a shared batch ID.
Trust score and evidence pack
A summary of validation results, security status, decision, and check outcomes for review and reporting.