Notes on Jev, and a Demo Built on Its New Box Question

Published:
•15 min read

These are my notes on Jev, TypeSafe AI’s decision model: the questions it answers, the practical lessons and basic hygiene I wrote down while using it, the Box question that came with Jev 1.14, and the demo I built with it.

Jev and its questions

Jev does not write text. You send it a state (a text, a JSON object, and since 1.14 images too) and a set of typed questions, and it returns typed answers with probabilities. TypeSafe calls it a System One model.

A Choice picks one option from a list and returns a probability per option. A Score places the state on an ordered rubric. A Noul returns the probability that a statement is true. A Box, new in Jev 1.14, finds where something is in an image: it returns the four edges of one box around it, as fractions of the image’s width and height, and a confidence.

with TypeSafeClient(model="...") as client:
    r = client.system_one(
        state=ticket,
        questions={
            "answer1": Choice(instructions=..., criteria={...}),
            "answer2": Score(instructions=..., criteria=[...]),
            "is_yes_answer3": Noul(instructions=...),
        },
    )
    a = r.answers

One state goes into Jev once. Four questions of different types are answered against it in parallel, each returning its own typed answer.

Practical notes

Toy example. Without an Other option, the probability has to land on one of the listed options, even when none is right. With Other, it lands there.

A reasoning question split in two calls: the first asks several small questions about the state, the second gets the state plus those answers and asks the big question.

TypeSafe’s docs give the formula: for a Choice with n options and top probability p, confidence = (p - 1/n) / (1 - 1/n). A confidence of 0.6 needs a top probability of 0.8 with 2 options, 0.73 with 3, 0.68 with 5 and 0.64 with 10.

Confidence against the top probability for 2, 3, 5 and 10 options. The same confidence of 0.6 is reached at a top probability of 0.80, 0.73, 0.68 and 0.64.

"get_physical_card": {
    "what": "Finding or getting the PIN for a card",
    "not_for": "Changing an existing PIN (change_pin)"
},
"change_pin": {
    "what": "How or where to change an existing PIN",
    "not_for": "Finding the PIN of a card (get_physical_card)"
}

The field names are your own choice: TypeSafe’s docs say what, not_for and examples are not part of the API, and the model reads the names along with the values.

Toy example. Two options that overlap share the probability between them, so neither looks sure. With what and not_for written for each, the probability goes to one.

Toy example. As the threshold rises, Jev handles fewer cases and the cases it keeps are more often right. The sweet spot is the lowest threshold where accuracy has stopped climbing much.

Architecture

RLCD is TypeSafe’s name for its training method, reinforcement learning for calibrated decisions.

Notes after Jev Jam

Jev Jam was indeed very Joyous. A couple of notes:

Basic hygiene:

Jev 1.14 and the Box question

TypeSafe released Jev 1.14 on 14 October 2026, in a post called “Jev: For the builders AND dreamers”. It added three experimental features, not recommended for production yet: images in the state, instructions or criteria; a new Box question type; and a long-context mode. Images and long context are each switched on by a “danger zone” request header, and TypeSafe’s Box docs ask for the images header, TypeSafe-Danger-Zone-Images: true, on every request that has an image or a Box question.

A Box question takes an image and a description of what to find, and returns one bounding box: its four edges as fractions of the image’s width and height, and a confidence. A Box answer is always a box, even when the thing is not in the image, so TypeSafe’s docs suggest asking a Noul in the same request (“is it there?”) and ignoring the box when the Noul is low. My demo uses that pairing.

A Box answer: four edges given as fractions of the image, top, bottom, left and right, plus a confidence. A Noul asked in the same request says whether the target is there at all.

What I built with Box: Jev Highlight

Talk about a website, and the page scrolls to the part you mean and lights it up. If it is on another page or tab of the site, the app clicks through to it first.

Type a website address, press Start and talk. Without a microphone, use the “Type what you would say” box: it sends the phrase through exactly the same path.

It matches meaning, not words: say “what does it cost” and the price lights up; say “how do I get started” and the sign-up button does. “Does it record sessions for compliance” opens tailscale.com’s Privileged Access tab and lights up “Record full sessions for auditability and compliance”. I put it up at jev.pipelinevideo.com, behind a password for now.

How it works

  1. Load: a headless Chrome takes a full-page screenshot of the site, cuts it into tiles (Jev accepts images of 280 to 1568 px a side), then captures the site’s header pages and in-page tab states in the background.
  2. Talk: the browser’s own speech recognition sends the last few words of the current sentence every 0.7 s.
  3. Find: per tile, one Jev request asks a yes/no (“does this tile show what is being talked about?”) and a Box. The yes/no picks the tile; the Box says where. A second Box on a zoomed crop tightens it. If nothing on the current page matches, a text-only Jev question ranks the other pages and tabs, and the best two are searched.
  4. Show: the page scrolls, and the highlight lands, in under a second after you speak.

The Jev Highlight flow: load and tile the site, send the latest words every 0.7 s, ask a Noul and a Box per tile, draw the box, then refine it on a zoomed crop.

Load. The front page is opened at 1440x900, cookie banners are removed, the page is scrolled through so lazy images load, and a full-page screenshot is taken, cut at 14000 px and into 1440x900 tiles overlapping by 180 px. The front page shows at once.

The rest of the site, in the background. Up to 8 navigation pages: visible same-site links in the header (not the logo, login, sign-up, legal, careers). The link’s label and its box on the page are kept. Three pages are captured at a time. In-page tabs on every captured page: ARIA tablists, and rows of 3 to 8 sibling buttons outside the header. Each tab is clicked; the rows of the page that change, compared with a second shot taken without a click to ignore marquees and carousels, become that tab’s own place. A row whose clicks change nothing is not a tab row. All of it stops at 75 s. The row of tabs above the page shows each page, marked while it captures; the status line says what was captured and what was not, and why.

Find, per phrase:

Each 1440x900 tile of the page, overlapping its neighbour by 180 px, is its own Jev request with a Noul and a Box. The tile with the highest Noul wins, and its box is mapped back to page pixels.

How the app decides where to look: the current page wins at a Noul of 0.8 or more; a text-only Choice ranks the other pages and tabs; another place wins at 0.6 or more and 0.15 over a weak match here; below 0.5 everywhere, nothing moves.

The click, shown. The link or tab lights up on the current screenshot, a pulse marks the click, the other screenshot replaces it, and the answer lights up there. Two clicks (another page, then a tab on it) take roughly twice as long on screen as one.

Refine. A second look at a crop around the box, enlarged 2x; the highlight glides to the tighter box. It costs one more round trip, so the browser draws the first box at once and lets the refined one glide in after.

Speech. Web Speech API, en-US, continuous with interim results. The latest words go out every 0.7 s, inside the sentence being spoken. An answer is dropped only when a newer one has already been drawn. On “not found” the last highlight stays and fades by itself.

Why this shape, measured rather than assumed

Requests sent in parallel against how long they take to come back, both indexed to a batch of 17: 2.4 times the requests take 1.4 times as long, 4.7 times the requests 2.2 times as long.

Images are billed as input tokens at the text price (TypeSafe lists $0.042 per million input tokens, output free), so a phrase costs one tile’s worth of tokens for every tile searched.

The app reaches Jev 1.14 through OpenRouter’s Decisions API, which serves the same model. The one difference is how an image travels: OpenRouter takes it as an image_url part inside state, not as a multipart assets entry. Sent the TypeSafe way, the image is silently dropped and Box answers blind (about a twelfth of the input tokens it should be, and a much lower confidence).

Known limits

Images and Box are preview features at TypeSafe, and answers vary a little between runs. This is a demo, not a product.

The same idea in a Pipeline Video film

Alongside the app I made a companion to a short Tailscale film made with Pipeline Video. It shows Tailscale’s front page. For each narrated phrase, Jev (jev-1.14.0) is asked where on the page that phrase is. The page scrolls there and the box Jev returned is highlighted.

The companion film. For each narrated line, the page scrolls to where Jev’s Box found it, draws that box and shows its confidence.

  1. The phrases are the voiceover lines, with the times taken from the speech-to-text word timings.
  2. The page’s content column is cut into 8 tiles of 1360x1000, with 30% overlap. No tile is downscaled.
  3. The relevance call sends one request per phrase, with all 8 tiles and one noul per tile. The tile with the highest noul wins. If that noul is under 0.5, the phrase gets no highlight.
  4. The box call sends one box question on the winning tile alone. If the box confidence is under 0.4, the phrase gets no highlight.

Box questions asked in the same request as all 8 tiles came back loose; one box came back as a tall strip. Asking about one tile on its own gave tight boxes. The film draws only boxes Jev returned; nothing is placed by hand.

The second version has a new, shorter voiceover, written to match the front page. Every line points at something real on the page, in other words than the page uses. That paraphrase is what the demo shows. Before voicing, each draft line was tried against Jev. A line that missed was rewritten until it landed.

Sources

Newsletter subscriptions are not currently available.

Share this post on:
Share this post on X Share this post on BlueSkyShare this post on LinkedInShare this post via WhatsAppShare this post on FacebookShare this post via TelegramShare this post on PinterestShare this post via email