How I Built My Own Local Jev with Qwen

Abstract

We are used to Language Models answering with text, so even one small decision makes the model write a word, an explanation, or an entire JSON object. On September 20, 2026, TypeSafe introduced Jev, which receives application state and immediately returns decisions with probabilities.

Jev itself immediately looked useful to me. I constantly work on language detection, tool selection, document classification, and operator handoff. Today we often run a full Language Model and then extract one decision from its generated text. Jev is built for exactly this class of work and could improve both the speed and reliability of these parts of my applications.

I decided to build a local version for two separate reasons. The first was practical: many of my systems work with 1C, financial documents, and other data inside private environments. I cannot simply connect the Jev API, so I need a way to apply the new technique locally without depending on a cloud provider. The second was research: I wanted to reverse-engineer the publicly understandable part of the mechanism, learn how it works, and test how closely an ordinary local Qwen could reproduce it.

I ended up building my own Jev-style path. When all valid answers are known in advance, it returns the preferred option and the strength of that choice.

Part I. How Jev Works

First, let us put the entire idea into one clear chain. We will begin with an ordinary customer message, watch how a Language Model starts an answer, and read its decision before text generation begins.

1. Why Jev caught my attention

I spend a lot of time putting Language Models inside real business systems. The model often performs one small but important job:

• detect the language of a request;
• choose the right tool;
• classify a document;
• decide whether there is enough information for the next step;
• decide whether automation may answer or a human operator is needed.

These are all classification and selection problems, and they appear in almost every agent and service I build. One decision often triggers the next: detect the language, choose a workflow, select a tool, then decide whether its result needs a person.

A regular LLM can do all of this. Every judgment launches a complete text generator to produce one final option. Jev immediately looked like a strong fit for a class of work I already encounter every day. Faster typed decisions with explicit uncertainty improve the entire application pipeline.

The next question was how to use the idea in my own systems.

The most interesting new models usually appear as external APIs first. Private client environments cannot use them directly: financial documents, 1C data, and internal conversations must remain inside the client’s infrastructure.

That is the first reason I wanted a local version of Jev. Price does not solve the problem: a private environment excludes the external service. I wanted the benefit of the technique inside infrastructure I control.

The second motivation was research: I simply wanted to understand how Jev works.

After Tiny Manas, I already understood how a Language Model learns to predict the next token, and Jev uses that familiar mechanism for direct selection between options. I wanted to reverse-engineer the publicly understandable part of the system and answer two questions: what can be reproduced without TypeSafe’s closed weights, and how far can an ordinary local model go if its first prediction already contains the answer I need?

That was something I wanted to test with my own hands.

2. What is Jev?

TypeSafe introduced Jev on September 20, 2026. The company calls it the first System One Model: a model built for fast decisions inside software.

A normal LLM receives text and returns text, while Jev receives application state and a set of predefined questions, then returns values with probabilities. In the simplest form:

regular LLM:  “write an answer”
Jev:         “choose one of these answers”

The System One name refers to the fast, intuitive side of thinking described by Daniel Kahneman. The model is not being asked for a multi-page chain of reasoning. It receives a narrow question and quickly weighs the available choices.

Jev itself is named after economist William Stanley Jevons. The idea is straightforward: when one unit of intelligence becomes dramatically cheaper, we do not merely save money on old use cases; we begin putting intelligence into places where it previously made no sense at all.

3. Why a regular LLM does unnecessary work

Imagine that you run a system for processing customer-support messages, and it must route every request to one of three teams: billing handles payments, refunds, and duplicate charges, technical support handles product failures, and account access helps customers sign in and regain access.

A new message arrives:

I was charged twice for the same subscription.

The system must choose which team receives it:

billing
technical support
account access

The task ends with one route, yet a regular Language Model still writes an answer. If we request JSON, it produces the opening brace, a field name, a colon, the word billing, quotation marks, and the closing brace one token at a time; the application then waits for the response to finish, reads the JSON, and extracts the team name.

It is like an employee writing a memo before ticking every box in a test: a person might read the explanation, but the routing system only uses the tick.

Jev removes the unnecessary part. It receives the ordinary text and weighs the choices the application already supplied.

4. Where the model hides its first choice

To understand the mechanism, recall how an LLM writes. It picks one token, appends it, picks another, then another.

Before choosing the first token, the model holds an enormous internal contest. Every token in its vocabulary receives a score:

A:       8.2
B:       5.5
C:       4.8
word:    1.1
period: -0.3
... thousands more

These scores are called logits. A larger logit means the model currently prefers that token as the next continuation. The logits form a leaderboard before the model chooses a winner.

Now we assign short labels to the three answers:

A = billing
B = technical support
C = account access

The model reads the meaning of every option and must return one letter. The first contest therefore has three relevant participants: A, B, and C.

We take their scores and turn them into understandable shares:

A = 91%  → billing
B =  6%  → technical support
C =  3%  → account access

The operation is called restricted softmax. The name sounds as if some frightening mathematics is about to begin, but do not worry: the idea itself is simple. Ordinary softmax distributes 100% across every candidate, while restricted softmax does it only across the options allowed by the application.

And this is the important moment: the model has not written a single token, yet the decision already exists.

5. Why we need A, B, and C

There is an obvious question. Why not read the score for billing itself?

Because a visible word and a token are not the same thing. billing may be one token for one model and several for another. technical support will almost certainly contain several tokens. Comparing phrases of different lengths would mean scoring them piece by piece, and our simple decision would begin turning back into generation.

A, B, and C work like coat-check numbers: the prompt describes what is stored under each number, and the model returns one short label for the right item.

Even a letter cannot be trusted blindly. A and A with a leading space may be different things to a tokenizer. The system therefore checks that every label is exactly one token at the actual answer position. That is already an implementation detail, so we will return to it in Part II.

6. What the application receives

A normal classifier might return only billing. Jev-style scoring returns more:

billing            0.91
technical support  0.06
account access     0.03

The application now sees both the winner and the shape of the victory: 0.91 / 0.06 / 0.03 looks decisive, while 0.40 / 0.37 / 0.23 is almost a tie. Billing wins in both cases, but the first request can move automatically and the second should go to an operator.

We can write an ordinary rule:

if the first choice is above 0.80
and leads the second by at least 0.20:
    continue automatically
otherwise:
    send to a person

The threshold lives in code, so it can be tested, changed, and discussed with the process owner: the model weighs the options, while the application decides what to do with the result.

There is one more trap. Imagine that the message belongs to none of our three departments: it is not about billing, technical support, or account access. Restricted softmax still has to distribute 100% among them, so the model will confidently choose the most suitable of three wrong answers.

That is why the list needs an OTHER or ESCALATE option. OTHER says that none of the known categories fits, while ESCALATE sends the request to a person or a separate review flow. The model now has an honest way out when the correct door is simply missing.

Only then should we return to the value 0.91. It measures the strength of the preference inside the list we supplied; the real share of correct answers must be measured separately on labeled examples. If the model reports 0.91 for a hundred similar requests, we still need to check how many of those requests it actually classified correctly.

7. Which decisions Jev can describe

The public TypeSafe API has three main question types.

Noul is a yes-or-no question. Does this invoice match the purchase order? Does a person need to review the case?

Choice selects one item from a list. Which team should receive the message? What kind of document is this?

Score places something on an ordered scale. Is the risk low, medium, high, or critical?

A larger workflow can be built from these small answers. Exact properties—amounts, dates, permissions, and mandatory fields—remain in ordinary code. The model enters only where a simple if is too brittle. TypeSafe’s public workflow examples show this pattern for invoices, security incidents, and customer support.

Each question looks modest on its own. Dozens of such decisions inside one process are where speed begins to matter.

8. Which part of Jev can be reproduced?

A regular Language Model can define the choices in advance and read the first-token scores before generating text. That is the part I reproduced locally.

TypeSafe adds a new architecture, parallel output for many decisions, and a separate training method called RLCD. The company does not publish the weights, model architecture, or complete calibration process.

I call my system a Jev-style model built on Qwen. It reproduces the understandable decision interface with a different model and runtime.

The theory is now in place. We can open the hood and see how I built it on a local machine.

Part II. How I Built Local Jev-style Qwen

1. The starting point: Qwen on a Mac Studio

For this experiment, I had access to an Apple M3 Ultra Mac Studio and could use no more than 60 GB of unified memory for the model. Qwen3.8-27B-8bit fit within that limit, so it became the starting point for the research.

Qwen3.8-27B is a dense 27-billion-parameter model with 64 layers, a hidden dimension of 5,120, a 248,320-token vocabulary, and a native context length of 262,144 tokens. I use a pinned MLX quantization at an exact revision so that a later launch cannot silently load different weights. MLX-LM is designed specifically for inference and fine-tuning on Apple silicon.

Keeping a second copy of a 27B model in memory would have been too expensive, so I placed the scorer inside the same local process. The experiment could reuse the loaded weights instead of spending memory on a duplicate model.

2. How I arrived at the implementation

Avi Chawla’s walkthrough was my starting point. It reproduces the same idea through SGLang’s ready-made /v1/score endpoint. That endpoint is explicitly designed to score selected token IDs, as the official SGLang documentation confirms.

My local runtime worked differently. I first requested top_logprobs through the OpenAI-compatible chat API; the parameter was accepted, but the response contained logprobs: null, so the numbers remained inside the model process.

I then followed the runtime deeper. After prefill, the scheduler already held the full next-token log-probability vector and used it for decoding. The expensive computation already existed; I needed to carry a few selected values from the tensor into an API response.

I added two research routes to the same process:

• /v1/jev/tokenize renders the real chat template and verifies which labels occupy exactly one token;
• /v1/jev/score performs prefill, extracts only the declared positions, normalizes them, and returns a compact response.

The full vector of 248,320 values is never serialized. Only the requested scores leave the model process. This keeps network traffic small and avoids turning the model’s entire internal output into a new public surface.

3. One request, step by step

Step 1. Application code sends state and semantic options. It knows about duplicate_card_payment and forgot_password, not token IDs.

Step 2. The local application builds the complete prompt. The description beside each short label gives Qwen the meaning of every option.

Step 3. The tokenizer endpoint verifies labels in the exact chat-template context. Unsafe candidates are rejected. A valid mapping is cached for the same question and choice set; repeating this setup before every business decision would be wasteful.

Step 4. Qwen processes the prompt exactly as it would before an ordinary answer. Its weights, architecture, and knowledge remain unchanged.

Step 5. The runtime receives the next-position logits and keeps only the token IDs assigned to the choices.

Step 6. Restricted softmax turns those logits into values that sum to one.

Step 7. Technical labels are mapped back to semantic names. The application receives duplicate_card_payment, not the letter B.

Step 8. Ordinary code applies policy. It might continue automatically when the top probability is above 0.80 and leads the runner-up by at least 0.20; otherwise it sends the case to a person. The threshold lives in business logic instead of being hidden inside a prompt.

4. What I measured

The first bounded run used 20 real messages from Banking77 across ten banking intents. Every example went through the same Qwen in three modes:

1. Jev-style scoring;
2. one-label generation;
3. compact JSON generation.

All three achieved 100% on that small slice. Median latency was 1.543 s for scoring, 1.590 s for one label, and 2.082 s for JSON. The one-token lower bound is almost impossible to beat by an order of magnitude because both paths pay for the same prefill. Compact JSON therefore became the main comparison because applications use it in practice.

I then expanded the experiment across the three System One shapes:

• Noul: 40 real banking messages and a binary ATM-withdrawal question;
• Choice: 30 messages with 30 banking intents available at once;
• Score: 48 completed examples from the real Stanford Sentiment Treebank test split with five sentiment levels.

I stopped the final block after 48 of 50 examples because the run already contained enough evidence for the decision. I did not fill the two missing rows with substitutes.

5. The central result: same decisions, less latency

The most important result does not fit in a latency table: scoring and one-token generation selected the same option in 118 out of 118 completed cases. The new inference path preserved Qwen’s decisions and read them before the model formatted them as text.

The speedup over one token was 1.04–1.14×, while the gain over compact JSON was 1.33–1.46×. In absolute terms, scoring saved roughly 0.7–0.9 seconds on the median request, and those savings accumulate across any sequential chain of decisions or a large document stream.

Compact JSON returned the winner, while scoring returned the entire distribution in less time. A functionally equivalent generative response with probabilities for all 30 options would require many more output tokens, so I stopped once the experiment had enough evidence instead of spending more compute time.

6. Where the gap from 40–200× comes from

On this machine, reading the prompt with a 27B model dominates the latency of a short request. That phase is the prefill. Nine JSON tokens occupy a smaller share, so removing the autoregressive tail saves only the generation time after prefill.

One-token generation performs the same prefill and one decode step. The 1.04× result on the 30-way Choice task follows directly from that compute structure.

TypeSafe reaches 40–200× with a new architecture, a parallel sampler, multiple questions answered together, and RLCD training. Its workflow comparisons also ask generative frontier models to produce much larger probabilistic structures. TypeSafe itself says that its published 193.6× speed and 444.6× cost figures are likely at the high end of real-world gains.

7. Quality is a control here, not a model competition

On the expanded sample, scoring accuracy was 85.0% for Noul, 86.7% for 30-way Choice, and 43.8% for the difficult five-level sentiment task. The one-token baseline produced the same values and made the same mistakes. The complete agreement ties quality to Qwen and the task definition; scoring preserved its behavior.

JSON occasionally selected a different class and was slightly more accurate on these samples. Several generated tokens create another decoding trajectory and measure a different response path.

Reversing option order exposed another limit. The winner remained stable in 8 of 10 binary cases, 8 of 10 Choice cases, and 9 of 10 Score cases. Qwen is sensitive to ordering. Raw probabilities therefore cannot be treated as calibrated financial risk and connected directly to consequential actions without another evaluation stage.

The next engineering steps are clear: permutation aggregation, stable rubrics, separate calibration and test splits, temperature scaling, and business thresholds chosen before a production pilot. Together they form a proper decision model.

8. Where this belongs in business systems

Generation remains the right tool for client emails, accounting explanations, and analytical reports. Local Jev-style inference handles small forks inside a business process:

• which language the request uses — Choice;
• which tool an agent should call — Choice;
• whether the conversation should go to an operator — Noul plus a confidence threshold;
• does the document belong to the expected legal entity — Noul;
• which expense category does the invoice represent — Choice;
• does the payment purpose match the contract — Noul;
• which team should receive the request — Choice;
• how severe is the discrepancy — Score;
• may the workflow continue automatically, or should an accountant review it — a code threshold over the distribution.

That is where the business value appears: the model becomes a set of fuzzy if statements between exact system rules. Amounts, dates, permissions, and mandatory fields stay in ordinary code, while the model handles judgments that are difficult to express as rigid formulas.

Probabilities make the difference in reliability visible: a clear case moves automatically, a near tie goes to a person, and the logs preserve the model’s uncertainty as numbers.

In the experiment, the entire path ran locally on the Mac Studio: the request entered the already loaded Qwen process, the scorer returned a distribution, and no external model API participated in the computation.

9. What I would build next

The first stage answered the core question: a strong local Qwen can act as a typed decision scorer without retraining. Several independent directions now become possible.

Calibration. Build a labeled set of real tasks from the intended domain, separate calibration from final testing, and measure whether reported confidence matches empirical error rates.

Robustness. Reduce option-order sensitivity and require ESCALATE wherever the answer space may be incomplete.

Parallelism. The local endpoint currently answers one question per pass. The major systems opportunity is to evaluate several independent heads or batches of questions over shared state. This is where speedups may move from percentages to multiples.

Specialization. If production data reveals a stable family of repeated decisions, compare prompt-only Qwen with fine-tuning, distillation, or a smaller decision model. The verified inference path provides the baseline.

10. What I learned from the experiment

Jev proposes a different interface between AI and software: free-form text works well for communicating with people, while code is more comfortable with bounded types, explicit probabilities, and predictable branches.

I began with a familiar class of work: my applications constantly detect languages, choose tools, classify documents, and decide when an operator should step in. Jev offered a suitable interface for these judgments, while research curiosity took me inside the mechanism and toward a local implementation. The result is a working scorer over Qwen3.8-27B that preserves the base model’s decisions and returns the full probability distribution faster.

The measurements also set the boundary of the current approach: prefill dominates short responses, so the gain over compact JSON was 1.33–1.46×. Speedups measured in tens will require specialized architecture and parallel processing, but we now know which part of the idea works, where the current limit lies, and what must be built next.

We often hide one giant black box behind the word “model.” In this experiment, the improvement came from a simple question: why make the model write a whole page when the decision already appeared in its first vector?