Jev Confidence, Calibration, and Automation Thresholds

TypeSafe Jev outputs explicit numerical certainty metrics for every decision: a confidence float for Choice and Score (reflecting probability concentration across options), and a calibrated probability float (noul) for boolean predicates.

Because general-purpose LLMs generate text token by token and tend to be overconfident or verbally evasive when asked for probabilities, Jev's promise of calibrated System One probabilities is a primary reason engineers evaluate it. But how reliable are those probabilities when you put them into production?

The Core Finding: High Confidence Is Not a Guarantee

Across both evaluations, the broad trend held: higher Jev confidence generally meant higher observed accuracy.

However, our empirical testing reveals two critical operating truths:

1. A high confidence score does not guarantee correctness. In CLINC150 forced-choice routing, Jev misclassified 102 in-scope requests and misrouted 150 completely unsupported out-of-scope requests at $\ge 0.90$ confidence. Even at maximum confidence (confidence = 1.00), 9 unsupported queries slipped into operational queues.
2. Type safe does not mean semantically correct. Jev guarantees schema compliance by construction—it will always emit a valid enum key, score, or float. But when forced to choose among supported categories, it will return a syntactically perfect, high-confidence assignment for a query that has no valid answer in that taxonomy.

Four Practical Questions About Jev Confidence

When an engineer evaluates confidence scores for automated routing, four separate questions matter:

1. Does higher confidence predict higher accuracy on known categories? Yes. On in-scope queries in CLINC150, predictions with confidence $\ge 0.95$ achieved 97.75% accuracy.
2. Does an 0.80 score actually mean an 80% success rate? It depends on how distinct your categories are. In CLINC150 with 150 diverse intents, calibration was sharp (ECE 0.0179). But in Banking77 with 77 overlapping banking intents, Jev suffered from overconfidence in the 0.70–0.89 band (ECE 0.0957), where actual accuracy sat between 48% and 56%.
3. Does confidence drop when inputs do not belong to the taxonomy? Yes, at the population level. In CLINC150, median confidence dropped from 1.00 on legitimate in-scope requests to 0.53 on unsupported out-of-scope requests (AUROC 0.9008).
4. Can unsupported queries still get high confidence scores? Yes. 150 unsupported requests (15.0%) scored $\ge 0.90$, and 9 scored 1.00. High confidence alone cannot eliminate tail errors.

---

Why Prescriptive Thresholds Fail

Vendor documentation and blog posts often suggest generic rules of thumb: "Use 0.70 for low stakes, 0.88 for operational tasks, and 0.95 for strict actions."

In practice, prescriptive threshold numbers do not transfer between workloads:

• A confidence threshold defines an operating point on a precision-versus-coverage curve, not an absolute accuracy guarantee. That curve changes depending on your taxonomy size, how closely related your categories are, and what kind of garbage traffic hits your ingress.
• If an unsupported input shares vocabulary with a supported route (for instance, "how fast does a cheetah run" matching the distance intent), Jev can assign 0.98 confidence to an entirely wrong decision.
• Gating on confidence alone forces a steep coverage penalty. In CLINC150 forced choice, reaching 95% precision required setting $\tau \ge 0.95$, which meant escalating 36.65% of all traffic to humans. Pairing an explicit fallback route with confidence gating recovered nearly 6 percentage points of automation coverage (+328 requests) at the same 95% precision target.

Review the complete 10-tier precision versus coverage trade-off table in our CLINC150 routing guide

workflow↗

---

What Jev Actually Exposes (The Three Primitives)

TypeSafe Jev returns certainty metrics through three distinct primitives:

1. Choice.confidence

In Choice, confidence measures the peakiness of the probability distribution across declared options:

{
  "department": {
    "type": "choice",
    "choice": "billing",
    "confidence": 0.942,
    "probabilities": {
      "billing": 0.942,
      "support": 0.041,
      "sales": 0.017
    }
  }
}

• If probabilities are evenly distributed across options, confidence approaches 0.0.
• If probability concentrates on one option, confidence approaches 1.0.
• Semantic Overlap Warning: If two options have overlapping criteria (e.g. card_arrival vs. card_delivery), Jev will divide probability mass between them, producing a low confidence score even if the model clearly understands the user's situation.

2. Noul.noul

In Noul, Jev does not return a separate confidence field. It returns a single probability (noul) estimating the likelihood that a boolean statement holds true:

{
  "is_refund_eligible": {
    "type": "noul",
    "noul": 0.875
  }
}

TypeSafe trains Jev with RLCD to calibrate these probabilities. In software, you branch on the float (e.g. if result.noul > 0.90:).

3. Score.confidence

In Score, score represents the probability-weighted expected index across numeric rubric levels ($0 \dots N-1$). TypeSafe exposes a confidence metric measuring how concentrated the probability mass is on a single rubric tier. However, TypeSafe's official documentation cautions that jev-1.13 score level calibration is weak, and scores should not be treated as smooth linear measurements.

---

Two Empirical Calibration Studies

We ran calibration evaluations across two very different operational workloads:

1. Banking77: Fine-Grained In-Scope Intent Calibration (385 Samples, 77 Classes)

We evaluated Jev (jev-1.13.0) against Gemini 3.8 Flash across 77 subtle, overlapping banking categories:

• Expected Calibration Error (ECE): 0.0957 (9.57%)
• Multiclass Brier Score: 0.3264
• What happened: When distinguishing closely related categories (such as order_physical_card versus get_physical_card), Jev was markedly overconfident between 0.70 and 0.89. The model concentrated probability mass on one option even when the boundary was genuinely ambiguous.

2. CLINC150: Large-Taxonomy Calibration & Out-of-Scope Separation (5,500 Samples)

We tested Jev on 150 diverse intents and 1,000 unsupported out-of-scope requests:

• In-Scope ECE: 0.0179 (1.79%) — sharp calibration when intents are well separated.
• Multiclass Brier Score: 0.1289.
• OOS Uncertainty Discrimination: AUROC of 0.9008 and AUPRC of 0.7163.
• The Tail Failure: 150 out-of-scope requests scored $\ge 0.90$, and 9 scored confidence = 1.00.

---

Why Did Unsupported Requests Score High Confidence?

In our CLINC150 benchmark, 150 unsupported queries (15.0%) scored $\ge 0.90$ confidence.

Several of the high-confidence false routes had obvious lexical or semantic overlap with the route Jev selected:

• "how fast does a cheetah run" $\to$ routed to distance (0.98 confidence)
• "can you tell me a fact about space" $\to$ routed to fun fact (0.97 confidence)
• "can you define what a computer is" $\to$ routed to definition (0.97 confidence)
• "let me know where jim is right now" $\to$ routed to current location (0.99 confidence)

The pattern is visible in the errors, but this benchmark cannot tell us what internal mechanism caused it. The part we would worry about in production: 9 unsupported queries scored confidence = 1.00. No confidence threshold, no matter how conservative, eliminates out-of-scope leakage entirely without human review or fallback routing.

---

How to Calibrate Thresholds for Your Workload

Setting production confidence thresholds requires measuring your own data against your team's tolerance for misroutes versus human review:

Step 1: Collect a Representative Sample

Gather 200–500 historical production inputs. Include unambiguous requests across all target routes, borderline queries that share vocabulary between two routes, and real off-topic or spam messages your ingress actually encounters.

Step 2: Bucket Predictions by Confidence

Run your dataset through Jev with an explicit fallback route (out_of_scope: "none of the supported intents"). Bucket outputs into confidence tiers (0.50–0.60, 0.60–0.70, 0.70–0.80, 0.80–0.90, 0.90–0.95, 0.95–1.00) and calculate observed precision and coverage in each tier.

Step 3: Weigh Misroute Costs Against Escalation Costs

Quantify the operational trade-off:

Total Cost(threshold) = (False Positives(threshold) × Cost of Misroute) + (Escalated(threshold) × Cost of Human Review)

Sweep candidate thresholds $\tau \in [0.60 \dots 0.99]$ to find the cutoff that minimizes total cost for your team.

Step 4: Implement a Three-Way Dispatch in Code

Never treat routing as a simple pass/fail flag. Route through a three-way branch:

import os
from typesafe_sdk import TypeSafeClient, Choice

client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"])

# Calibrated from your validation dataset (Step 3)
WORKLOAD_THRESHOLD = float(os.environ["JEV_ROUTING_THRESHOLD"])

def process_ticket(ticket_text: str):
    response = client.system_one(
        model="jev-latest",
        state={"message": ticket_text},
        questions={
            "queue": Choice(
                instructions="Classify user request into operational queues.",
                criteria={
                    "billing": "Invoice disputes, payment failures, refunds",
                    "technical": "API errors, bugs, outages, performance",
                    "account": "Password reset, login issues, MFA",
                    "out_of_scope": "none of the supported intents"
                }
            )
        }
    )

    decision = response.choices["queue"]
    route = decision.choice
    conf = decision.confidence

    # 1. Neutral fallback caught unsupported query
    if route == "out_of_scope":
        return divert_to_general_triage(ticket_text, reason="unsupported_intent")

    # 2. High confidence on supported route -> automated dispatch
    if conf >= WORKLOAD_THRESHOLD:
        return auto_dispatch_to_queue(ticket_text, queue=route)

    # 3. Moderate or low confidence -> human review with model's suggested route
    return route_to_human_queue(
        ticket_text,
        suggested_queue=route,
        confidence=conf,
        probabilities=decision.probabilities
    )

Related Context

Read our CLINC150 routing guide comparing forced choice, explicit fallback, and hybrid gating

workflow↗

Review the complete TypeSafe Jev entity guide for architecture, limits, and pricing

entity↗

Inspect our Banking77 benchmark comparing Jev against Gemini 3.8 Flash structured output

comparison↗

Review AI Model Waddle evidence standards and empirical methodologies

concept↗

Sources & Documentation

1. AI Model Waddle CLINC150 Benchmark Dataset — 5,500 evaluated test samples measuring forced choice, explicit fallback, and confidence gating.
2. AI Model Waddle Banking77 Benchmark Dataset — 385 evaluated test samples comparing jev-1.13.0 and gemini-3.8-flash.
3. TypeSafe AI Documentation: Confidence & Calibration — Official documentation detailing probability derivation and confidence calculation.
4. Niculescu-Mizil & Caruana: Predicting Good Probabilities With Supervised Learning — Foundational research on probability calibration and reliability curves in machine learning.