Intent Routing & Fallback Handling with Jev

When an application or agent workflow routes incoming user messages into backend queues, using a general-purpose LLM adds multi-second latency and high output token bills. TypeSafe Jev evaluates bounded routing targets in a single parallel pass (~130ms client-observed latency) at $0.042 per million input tokens with unmetered ($0.00) output tokens.

The operational catch is that real user traffic inevitably contains requests that match none of your queues. If your router is forced to pick from supported destinations, how does it behave on unsupported queries? Does its reported confidence drop, or does it confidently misroute the user?

The Core Rule: Type Safe Does Not Mean Semantically Correct

TypeSafe calls Jev "typesafe" because outputs conform strictly to your declared schema: it returns an allowed choice from your criteria map, a score within your rubric range, or a probability float. In our 5,500 benchmark calls, Jev achieved 100% schema validity with zero JSON parsing failures.

However, type safety is not semantic correctness:

• When forced to choose among 150 supported intents, Jev returned a structurally valid enum choice for every single request.
• On the 1,000 unsupported out-of-scope queries, 100% of those structurally valid assignments were semantic errors.
• More critically, Jev assigned $\ge 0.90$ confidence to 150 of those unsupported requests, and $\ge 0.95$ confidence to 91 requests.

If your routing architecture relies solely on type-safe output or treats high confidence as an infallible guarantee, unsupported customer requests will silently pollute operational queues.

Three Routing Setups We Tested

We tested three distinct routing approaches on the same 5,500 utterances:

1. Forced Choice (Condition A):
   User Input ──> Jev (150 Supported Intents) ──> Always routes to a supported queue
   (Result: 15% of unsupported traffic leaks with high confidence >= 0.90)

2. Explicit Fallback (Condition B):
   User Input ──> Jev (150 Intents + "out_of_scope") ──> Supported Queue OR Fallback Queue
   (Result: Catches 76.8% of unsupported traffic with 91.76% precision; 1.53% false alarms)

3. Hybrid Fallback + Confidence Gate:
   User Input ──> Jev (150 Intents + "out_of_scope")
                     │
                     ├── If predicted "out_of_scope" ──> Divert to Fallback / Triage
                     │
                     └── If predicted Supported Intent:
                           ├── Confidence >= Threshold ──> Auto-route to Destination
                           └── Confidence < Threshold  ──> Escalate to Human Review
   (Result: Best coverage-precision frontier; expands automation by 6–10 percentage points)

---

Condition A: Why Forced Choice Fails on Unsupported Traffic

In Condition A, we gave Jev the canonical 150 supported CLINC intents with neutral instructions ("Select the best matching intent from the allowed options") and no escape route.

On legitimate requests, Jev handled the task well: 91.71% accuracy across 4,500 in-scope test examples, with a median confidence of 1.00 (mean 0.9537).

When we gave Jev the 1,000 unsupported requests, its confidence did fall at the population level: median confidence dropped to 0.53 (mean 0.5665; AUROC 0.9008), and 75% of unsupported queries scored $\le 0.77$.

The High-Confidence Tail

What surprised us was the tail. Even though confidence fell on average, 150 out-of-scope requests (15.0%) still received a supported route at $\ge 0.90$ confidence, and 91 scored $\ge 0.95$.

When an unsupported query shared superficial vocabulary with a supported route, Jev latched onto the lexical cue with high certainty:

• "let me know where jim is right now" $\to$ routed to current location (0.99 confidence)
• "how fast does a cheetah run" $\to$ routed to distance (0.98 confidence)
• "can you tell me a fact about space" $\to$ routed to fun fact (0.97 confidence)
• "can you define what a computer is" $\to$ routed to definition (0.97 confidence)
• "can you send a text to mom" $\to$ routed to text (0.96 confidence)
• "how many ounces are in a pound of beef" $\to$ routed to measurement conversion (0.95 confidence)

None of these requests belong to the supported assistant domain. The part we would worry about in production: 27 unsupported requests scored exactly 1.00 confidence. In a forced-choice setup, no confidence threshold—not even $\tau = 1.00$—eliminates out-of-scope leakage without escalating a large fraction of legitimate traffic.

Systematic In-Scope Confusions

Forced choice also showed systematic blind spots where intent boundaries shared overlapping terminology:

• reminder update vs. reminder: All 30 test examples of reminder update were misrouted to reminder (100% error rate).
• accept reservations vs. restaurant reservation: 20 out of 30 test examples were misrouted (66.7% error rate).
• distance vs. time: 17 test examples confused.
• insurance change vs. insurance: 10 test examples confused.

---

Condition B: The Impact of an Explicit Fallback Route

To test whether Jev could recognize its own domain boundaries, we added a single neutral option to the criteria map:

{
  "out_of_scope": "none of the supported intents"
}

We gave it no few-shot demonstrations and no prompt coaching.

The explicit fallback intercepted 76.8% of unsupported traffic with 91.76% precision, cutting out-of-scope leakage from 1,000 down to 232.

The trade-off was small: Jev falsely flagged 69 legitimate in-scope requests (1.53%) as out of scope, mostly minimal or colloquial phrasings:

• "can i see my transactions" $\to$ out_of_scope (ground truth: transactions)
• "what's my balance" $\to$ out_of_scope (ground truth: balance)
• "let me know how to make a transfer" $\to$ out_of_scope (ground truth: transfer)

For operational systems, a 1.53% false alarm rate is an easy trade to eliminate three-quarters of unsupported traffic upfront.

---

The Hybrid Policy: Fallback + Confidence Gating

While the explicit fallback caught 768 unsupported requests, 232 still slipped past into supported categories. To catch that remaining leakage, we simulated a Hybrid Routing Policy:

1. Rule 1 (Fallback Intercept): If Jev predicts out_of_scope, divert immediately to fallback handling or an exploratory LLM agent.
2. Rule 2 (Confidence Escalation): If Jev predicts a supported intent but its confidence is below threshold $\tau$, escalate to human review or secondary confirmation.
3. Rule 3 (Automated Dispatch): If Jev predicts a supported intent and confidence $\ge \tau$, dispatch directly to the automated destination.

Measured Outcomes Across Thresholds (5,500 Evaluated Requests)

In-Scope Diverted includes both legitimate queries falsely rejected as OOS (69 queries, 1.53%) and legitimate queries escalated due to low confidence.

What the Numbers Mean in Practice

Comparing gating alone against the hybrid policy reveals where the real operational wins come from:

• At 95% Precision: Gating alone requires setting $\tau \ge 0.95$, automating only 63.35% of traffic. The hybrid policy achieves 95.12% precision at $\tau \ge 0.85$, safely automating 69.31% of traffic (+5.96 pp, or +328 additional automated requests) while cutting out-of-scope leakage from 91 down to 78.
• At 97% Precision: Gating alone requires $\tau = 1.00$, collapsing automation coverage to 44.40%. The hybrid policy reaches 97.07% precision at $\tau \ge 0.98$, automating 54.07% of requests (+9.67 pp, or +532 requests).
• The 99% Ceiling: Neither architecture reached 99% precision. Even at maximum confidence ($\tau = 1.00$), 9 out-of-scope queries still slipped into supported routes. In high-stakes workflows, automated routing alone cannot guarantee zero tail errors.

---

Production Implementation Pattern

The following Python snippet implements the 3-step hybrid routing policy using the official typesafe-sdk:

import os
from typesafe_sdk import TypeSafeClient, Choice

client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"])

# Define your supported operational routes
SUPPORTED_ROUTES = {
    "billing": "Invoice questions, refunds, or payment disputes.",
    "account_access": "Password resets, MFA issues, or locked profiles.",
    "order_tracking": "Shipping updates, tracking numbers, or delivery delays.",
    "technical_support": "Software errors, bug reports, or service outages."
}

# Always append an explicit neutral fallback
ROUTING_CRITERIA = {
    **SUPPORTED_ROUTES,
    "out_of_scope": "none of the supported intents"
}

# CALIBRATION NOTE: Selected from validation data for your production workload
CONFIDENCE_THRESHOLD = float(os.environ["JEV_ROUTING_THRESHOLD"])

def route_incoming_message(user_message: str):
    response = client.system_one(
        model="jev-latest",
        state={"message": user_message},
        questions={
            "destination": Choice(
                instructions="Select the best matching intent from the allowed options.",
                criteria=ROUTING_CRITERIA
            )
        }
    )

    result = response.choices["destination"]
    predicted_route = result.choice
    confidence = result.confidence

    # Rule 1: Explicit fallback interception
    if predicted_route == "out_of_scope":
        return {
            "status": "FALLBACK",
            "destination": None,
            "confidence": confidence,
            "action": "divert_to_exploratory_agent_or_triage"
        }

    # Rule 2: Confidence gating on supported routes
    if confidence < CONFIDENCE_THRESHOLD:
        return {
            "status": "ESCALATED",
            "destination": predicted_route,
            "confidence": confidence,
            "action": "route_to_human_with_suggestion"
        }

    # Rule 3: High-confidence automated dispatch
    return {
        "status": "AUTO_ROUTED",
        "destination": predicted_route,
        "confidence": confidence,
        "action": "dispatch_to_handler"
    }

---

Latency and Cost Grounding

Routing latency and cost matter because the router sits on the critical path of every inbound interaction. In our Banking77 benchmark comparing Jev against Gemini 3.8 Flash across 77 intents:

• Jev ran at 188ms median latency (320ms p95), compared to 1,204ms median (3,245ms p95) for Gemini Flash structured outputs.
• Jev cost $0.0679 per 1,000 decisions, compared to $0.4570 for Gemini Flash (an 85% reduction, driven by Jev's free output tokens).
• Gemini Flash was 4.16 percentage points more accurate (83.64% vs. 79.48%), resolving subtle intent boundaries Jev missed.

In our CLINC150 routing study across 151 options, Jev delivered a median latency of 131.77ms (p90: 197.47ms, p95: 233.64ms) and billed $0.1030 per 1,000 decisions ($0.5667 total for 5,500 evaluated queries).

---

Related Context

Review the complete TypeSafe Jev entity guide for architecture, limits, and primitives

entity↗

Examine detailed calibration curves, Brier scores, and distribution shift metrics

concept↗

Inspect the controlled Banking77 benchmark data comparing Jev against Gemini 3.8 Flash

comparison↗

Review AI Model Waddle evidence standards and empirical methodologies

concept↗

Sources & Documentation

1. AI Model Waddle CLINC150 Benchmark Dataset — 5,500 evaluated test samples measuring forced choice, fallback, and hybrid gating.
2. CLINC Out-of-Scope Intent Classification Benchmark — Stefan Larson et al., EMNLP 2019 benchmark dataset (CC-BY-3.0).
3. AI Model Waddle Banking77 Benchmark Dataset — 385 evaluated test samples comparing jev-1.13.0 and gemini-3.8-flash.
4. TypeSafe AI Documentation: Primitives Overview — Official documentation covering Choice, Score, and Noul schemas.