TypeSafe Jev Rate Limits: Requests, Token Throughput, and Concurrency Ceilings

TypeSafe operates Jev under two concurrent throughput ceilings: 1,200 requests per minute (RPM) and 250,000 tokens per second (TPS) per organization. In addition, each request is bounded by a 64,000-token total request ceiling, where the state payload combined with your longest single question definition cannot exceed 32,000 tokens.

Because Jev evaluates all questions in parallel over a shared state, throughput optimization differs fundamentally from sequential LLM pipelines: batching multiple decision questions into a single request consumes only one HTTP request against your 1,200 RPM quota and shares the same state tokens across all evaluations.

Documented Service Limits

Constraint Dimension	Documented Limit	Unit of Measurement	Enforcement Behavior
Request Rate (RPM)	1,200	Requests per minute	Returns HTTP `429 Too Many Requests`
Token Throughput (TPS)	250,000	Input tokens per second	Returns HTTP `429 Too Many Requests`
Request Token Ceiling	64,000	Tokens per request	Returns HTTP `422 Unprocessable Entity`
State + Longest Question	32,000	Tokens per evaluation	Returns HTTP `422 Unprocessable Entity`
Max Options per `Choice`	255	Discrete options	Returns HTTP `422 Unprocessable Entity`
Max Levels per `Score`	10	Ordered rubric levels	Returns HTTP `422 Unprocessable Entity`
Min Options / Levels	2	Minimum choices	Returns HTTP `422 Unprocessable Entity`

Which Limit Will You Hit First?

Whether an application reaches the request rate ceiling (1,200 RPM, or 20 req/s) or the token throughput ceiling (250,000 TPS) first depends directly on the average token size of your payload.

As a calculated mathematical crossover:

Crossover Payload Size = 250,000 tokens/sec / 20 requests/sec = 12,500 tokens per request

Workloads below ~12,500 tokens per request: The system will reach the 1,200 RPM limit before exhausting token throughput. For instance, evaluating 20 requests per second with a compact 500-token payload uses 1,200 RPM but only 10,000 tokens per second (4% of the TPS ceiling).
Workloads above ~12,500 tokens per request: The system will reach the 250,000 TPS limit before exhausting the request quota. For instance, submitting 25,000-token payloads would hit 250,000 TPS at just 10 requests per second (600 RPM).

Optimization Strategy: Question Batching

The most effective architectural pattern for maximizing Jev throughput is batching questions into a single request.

If you need to evaluate four different aspects of an incoming support interaction—such as department routing, urgency score, churn risk, and policy compliance—you can execute this as four separate requests or one unified request.

Inefficient: 4 Separate Requests

Consumes 4 HTTP requests against your 1,200 RPM quota.
Incurs 4x network round-trips (4 × ~100ms = 400ms sequential latency).
Duplicates input state tokens across 4 requests (4 × 500 tokens = 2,000 tokens against your TPS budget).

Optimal: 1 Batch Request

Consumes 1 HTTP request against your 1,200 RPM quota.
Incurs 1 network round-trip (evaluated concurrently within TypeSafe's 70ms to 500ms service range).
State tokens are processed once across all questions (500 tokens total against TPS).

from typesafe_sdk import TypeSafeClient, Choice, Score, Noul

client = TypeSafeClient()

# Evaluates 4 questions concurrently in 1 request over shared state
response = client.system_one(
    model="jev-latest",
    state=customer_interaction_payload,
    questions={
        "dept": Choice(
            instructions="Route ticket",
            criteria={"billing": "Billing", "tech": "Technical", "sales": "Sales"}
        ),
        "urgency": Score(
            instructions="Grade issue severity",
            levels=["low", "medium", "high", "critical"]
        ),
        "churn_risk": Noul(
            instructions="Is customer threatening cancellation or chargeback?"
        ),
        "needs_manager": Noul(
            instructions="Does customer demand supervisor escalation?"
        )
    }
)

# Access all results via response.answers
print(response.answers["dept"].choice)
print(response.answers["urgency"].score)
print(response.answers["churn_risk"].noul)
print(response.answers["needs_manager"].noul)

Dynamic Early-Access Envelopes

TypeSafe operates Jev in an active early-access phase. According to official documentation:

Dynamic Adjustments: Rate limits may adjust dynamically during early access based on overall cluster load. Teams requiring higher concurrency for production workloads can contact sales for quota increases.
Hosting & Latency: Inference servers are hosted in the AWS us-west region, with typical network round-trips within the US ranging from 70ms to 120ms. Regional hosting does not imply a formal uptime SLA or multi-region deployment.
Gateway Routing: If you access Jev through third-party gateways such as OpenRouter (typesafe/jev-1.13), limits and billing are enforced by the gateway's platform rules rather than TypeSafe's direct organization quota.

Automated Backoff and Jitter

To prevent thundering herd problems when operating near quota limits, implement exponential backoff with randomized jitter:

import time
import random

def robust_jev_call(client, state, questions, max_retries=5):
    for attempt in range(max_retries):
        try:
            return client.system_one(
                model="jev-latest",
                state=state,
                questions=questions
            )
        except Exception as err:
            if "429" in str(err) and attempt < max_retries - 1:
                # Base backoff 100ms, doubling each retry, with 10-50ms jitter
                backoff = (0.1 * (2 ** attempt)) + random.uniform(0.01, 0.05)
                time.sleep(backoff)
                continue
            raise err

Related Context

Jev API Reference & Schema Specification

implementation

Jev Pricing & Workload Cost Calculator

concept

Structuring State for TypeSafe Jev

workflow

Troubleshooting Jev HTTP 401, 422, 429, and 529 Errors

concept