Understanding Jev Confidence & Calibration: Thresholds, Probabilities, and Human Routing

When software makes automated decisions, the critical question is not just what the model picked, but how certain it is, and whether that certainty reflects real-world accuracy. TypeSafe Jev outputs explicit numerical certainty metrics for every evaluated question: a confidence float for Choice and Score, and a calibrated probability float (noul) for Noul.

However, a reported confidence of 0.90 does not magically guarantee that the model has a 90% chance of being right on your specific dataset. Calibration depends on your data distribution, how clearly you separated your option criteria, and how much relevant context was present in the state. Setting an arbitrary cutoff like 0.85 without validating against labeled examples risks either automating costly errors or discarding the efficiency gains of automated decision-making.

What Jev Actually Exposes

Each primitive returns distinct metrics documented by TypeSafe:

1. Choice.confidence

In Choice, confidence measures the peakiness of the probability distribution across options (approaching 1.0 when probability concentrates on a single choice, and decreasing toward 0.0 as probability spreads):

{
  "department": {
    "type": "choice",
    "choice": "billing",
    "confidence": 0.942,
    "probabilities": {
      "billing": 0.942,
      "support": 0.041,
      "sales": 0.017
    }
  }
}

• If there are 3 options and the model is completely uncertain, probabilities distribute evenly ({"billing": 0.333, "support": 0.333, "sales": 0.333}), and confidence approaches 0.0.
• If the model strongly matches the state to "billing", confidence approaches 1.0.
• Important: confidence measures distribution peakedness. If your criteria for two options overlap semantically (e.g., "account_problem" vs "login_issue"), the model will split probability between them, resulting in low confidence even if the model accurately understands the situation.

2. Noul.noul

In Noul, the returned payload does not contain a separate confidence field. It returns a single probability (noul) estimating whether the condition is true:

{
  "is_refund_eligible": {
    "type": "noul",
    "noul": 0.875
  }
}

TypeSafe documents that Jev is trained with RLCD to produce calibrated probabilities. In practice, whether noul: 0.80 corresponds to an 80% empirical success rate depends on your task and dataset, and requires empirical validation.

3. Score.confidence

In Score, score represents the probability-weighted expectation across the rubric's numeric level indices ($0 \dots N-1$). TypeSafe provides a confidence metric measuring how concentrated the probability mass is on a single level. However, TypeSafe's jaggedness documentation explicitly cautions that jev-1.13 score levels are weak in numerical calibration: score uncertainty does not necessarily distribute smoothly to adjacent levels, and scores should not be used to interpolate exact magnitudes between levels.

Why 0.90 Is Not a Magic Universal Threshold

Engineers frequently ask: "What threshold should I use to automate decisions?"

There is no universal threshold. A threshold that works for routing a support ticket will fail catastrophically if applied to processing a financial payout:

• Asymmetric Error Costs: If false positives and false negatives carry different business impacts, your threshold must shift. Incorrectly flagging a ticket as "billing" costs 30 seconds of an agent's time to reassign. Incorrectly auto-approving a fraudulent $5,000 refund costs $5,000.
• Domain Distribution Shift: If your production state contains vocabulary, edge cases, or slang that Jev did not encounter during pre-training, calibration degrades. A 0.92 score on out-of-distribution inputs may have an empirical accuracy of only 75%.
• Criteria Ambiguity: Poorly bounded option descriptions artificially depress confidence scores.

The Coverage vs. Error Trade-Off

Setting an automation threshold involves balancing two opposing forces:

Higher Threshold (e.g., 0.95):
  ▲ Lower Error Rate (very few false actions)
  ▼ Lower Coverage (more decisions sent to human review queues)

Lower Threshold (e.g., 0.70):
  ▲ Higher Coverage (most requests automated instantly)
  ▼ Higher Error Rate (more mistakes slip into production)

Risk Profile	Illustrative Gating Pattern	Error Tolerance	Typical Scenario
Low Stakes / Reversible	Branch on moderate confidence; fallback to standard queue	Tolerant of occasional misclassification	Read-only tagging, UI search filters, lead sorting
Balanced Operational	Moderate-to-high threshold; ambiguous cases prompt confirmation	Requires low error rate; edge cases routed to staff	Ticket routing, notification dispatch, subagent delegation
Strict Guardrail	Conservative cutoff; only act automatically under high certainty	Near-zero tolerance for automated mistakes	Database mutations, automated merges, financial disbursements

Practical Method: How to Select an Automation Threshold

To establish an evidence-based threshold for your production workload, follow this 4-step process using 200 to 500 labeled historical examples:

Step 1: Run Evaluations Across the Labeled Set

Collect 200–500 real operational inputs where human operators previously determined the correct decision. Run each through Jev and log the chosen answer, the confidence score, and whether it matched human ground truth.

Step 2: Bucket Results by Confidence Bands

Group evaluations into confidence bands (e.g., 0.50–0.60, 0.60–0.70, 0.70–0.80, 0.80–0.90, 0.90–0.95, 0.95–1.00). For each band, calculate:

1. Sample count: How many requests fell into this band.
2. Empirical accuracy: Percentage of requests in this band where Jev matched human ground truth.

If Jev is well-calibrated on your data, a higher confidence band will show correspondingly higher empirical accuracy. If accuracy in a high band is unexpectedly low, your option criteria are likely ambiguous, overlapping, or your state is noisy.

Step 3: Calculate the Cost Function

Assign dollar or time costs to your operations:

Total Cost = (Automated Errors × Cost of Error) + (Manual Reviews × Cost of Review)

Sweep through potential thresholds from 0.60 to 0.99 in increments of 0.01. Find the threshold value that minimizes total cost on your validation set.

Step 4: Implement a Three-Way Routing Split in Code

In production, do not treat decisions as a binary pass/fail. Use a three-way triage pattern:

from typesafe_sdk import TypeSafeClient, Choice

client = TypeSafeClient()

response = client.system_one(
    model="jev-latest",
    state=ticket_payload,
    questions={
        "routing": Choice(
            instructions="Route ticket to responsible operational team.",
            criteria={
                "billing": "Invoice, charge, refund, payment issues",
                "tech_support": "API errors, bugs, system crashes",
                "security": "Password resets, compromised accounts, MFA"
            }
        )
    }
)

decision = response.answers["routing"]
conf = decision.confidence

if conf >= 0.92:
    # High certainty: execute automated workflow
    auto_route_ticket(ticket_id, queue=decision.choice)
elif conf >= 0.70:
    # Moderate certainty: route with suggested tag for human confirmation
    route_with_suggested_tag(ticket_id, suggested=decision.choice, confidence=conf)
else:
    # Low certainty / ambiguous state: escalate directly to general triage
    send_to_general_triage_queue(ticket_id, reason="low_jev_confidence")

Related Context

Jev Primitives: Choice vs. Score vs. Noul Decision Guide

comparison↗

Writing Instructions, Criteria, and Answer Spaces for Jev

workflow↗

Documented Limitations and Failure Modes of TypeSafe Jev

concept↗