ArticlesOctober 20, 2026日本語で読む

Probably Yes is a daily word-guessing game. Everyone gets the same secret word each day. You ask up to 20 questions in your own words ("Is it food?", "Can you hold it?"), and each one gets one of five answers: Yes, Probably yes, Neither, Probably not, or No.

Every one of those answers is decided by Jev, called through OpenRouter. The game started in Japanese on September 30, 2026, and English, Spanish, Chinese and Korean followed on October 5. This post explains how the answers are made, how accurate they are, what they cost, and what went wrong.

What Jev decides here

Jev is a decision model. You send it a state and a few questions, and it sends back numbers: a probability for a yes/no question ("Noul"), a pick from a list ("Choice"), or a score. It does not write text.

That is the reason we use it for the game's host. A host that only returns a number can't reveal the word in a sentence, wander off topic, or make up an explanation. The worst it can do is be more or less sure than it should be.

Each player question is one Jev call. The state looks like this:

{
  "secret_word": "…",
  "facts_about_secret_word": "…",
  "player_question": "Is it bigger than a person?"
}

The call asks several Noul questions at once:

• yes: is the true answer to the player's question "yes"?
• neither: would most people find it hard to say a clear yes or no?
• guess: is the player naming the secret word itself?
• meta, not_question, about_game, chitchat: is it about the spelling, not a yes/no question, about the game itself, or small talk? These get a fixed reply instead of an answer.
• other_thing: is the player naming a different specific thing (a cat asked "Is it a dog?")?

How the five answers are made

The yes probability p is split at 0.1 and 0.4 from each end:

The middle band alone did not produce "Neither" when it should. We hand-labeled 40 questions as "Neither" (half and half, it depends, or the question doesn't really fit the word), and the band caught only 4 of them. Making the band wider pulled in clear questions just as fast.

So we ask the second question, neither, directly. If it is 0.4 or more and the yes probability is not already a firm Yes or No, the answer is Neither. With that, 15 of the 40 were caught.

A few things are decided before Jev is called at all:

• If the name in a guess ("Is it a cat?") is the word or one of its listed names, it is correct. This is plain string matching.
• If the player asks whether it is related to the word itself, the answer is Yes.
• Some fact sheets contain an answer written for a specific question, and that answer is used as is.

Accuracy

We measured on October 1 with 20 words and 480 questions, each labeled with one of the five answers, plus 100 guesses and 56 inputs that should not be answered.

• Correct guesses are found by name matching: 40 of 40, with 0 false matches out of 596 (60 near misses such as a broader category, and 536 ordinary questions). Jev's guess is only a backup for names not in the list; at our threshold of 0.8 it found 39 of 40 with 0 false matches.
• Spelling questions and non-questions were each rejected 28 of 28, with 0 of 580 ordinary questions rejected by mistake.
• In the evaluation, one Jev call with 7 questions took 191 ms at the median and 253 ms at the 90th percentile.

These numbers are on the optimistic side. One annotator made all the labels, and the thresholds were tuned on the same data. We have also added rules since then (below) and have not re-run the whole set.

The fact sheets stay in Japanese

Each word has a short fact sheet in Japanese: what it is, its size, what it's made of, where you find it, and so on. The sheet goes into the state, and Jev reads it and answers questions in English, Spanish, Chinese and Korean. We did not translate the sheets. Only the names of the words are translated, so that a guess like "Is it a cat?" can be matched.

Before opening the other languages, we tried 26 questions in those languages on two words (Shinkansen and cat). 25 came back as expected.

The fact sheet matters. In the same evaluation, the yes-probability-only setup matched the label exactly 72.5% of the time with the sheet and 61.5% without it, with thresholds tuned separately for each.

Cost

About $0.00018 per player question, as of October 20. That is one Jev call with all the questions above: about 4,200 input tokens and 139 output tokens. It was $0.00007 on October 1; every rule we added to the instructions made the input longer. An identical question about the same word reuses the stored answer instead of calling Jev again; on one day's 276 real questions, 22.8% were reused.

What went wrong

"Is it a train?" On day 4 the word was Shinkansen (the bullet train). "Is it a train?" correctly got Yes, and players took that as having guessed the word. Of the 20 players we looked at, 5 stopped playing right after that Yes. One posted that they had found the answer and the game didn't react. Now, when a question names a thing and gets Yes or Probably yes, the game adds a small note: "train" is not the answer. This note is decided by plain text matching; Jev is not asked.

"Can you touch it?" For an air conditioner, this came back Neither, probably because people don't usually touch one. It's an early question, so every answer after it pointed the player the wrong way. We wrote the rule into the instructions of the yes question: anything with a physical form is touchable, however big, high up, or rarely touched.

"Is it weight?" For a cat, naming a different thing ("Is it weight?", "Is it me?", "Is it time?") came back Neither, when it should simply be No. We added the other_thing question: if it is 0.4 or more, and the yes probability is 0.7 or less, the answer is No.

Size and material. Questions like "Is it bigger than a fridge?" or "Is it made of wood?" got the direction right but also a high neither, so they fell into Neither. For these questions the neither threshold is raised to 0.8, and the yes probability decides.

When players disagree

Since October 14, players can complain about an answer. After finishing the daily puzzle, they pick one of their questions, choose the answer they think is right (one of the other four), and can add a short comment. Only signed-in players of the Japanese version can send one: when anyone could, many complaints missed the point of the question.

Each complaint is reviewed with the word's fact sheet and a fresh answer from the current rules. Claude, the coding assistant that helps build the game, goes through them one by one, and anything that is a matter of interpretation goes to me. Players whose complaints are accepted get +1, and rejected ones −2, and complaints from higher-scoring players are reviewed first.

An accepted complaint sets the answer for that word and question. It is checked before Jev is called, so it holds even when the stored answers are cleared. Often the fact sheet is fixed too, so that the same question asked in other words gets the same answer. In the first week (October 14–19), 28 of 42 complaints were accepted. Some examples:

• Piano, "Is it sold in stores?": Neither → Yes. The fact sheet now says it is sold at music stores.
• Mount Fuji, "Is it related to the weather?": → No. The mountain itself doesn't change with the weather.
• Milk, "Is it organic matter?": → Yes. When we asked this about all 700 words, the answers for drinks and for tools made of wood, paper or cloth were all over the place. So the yes instructions now define it in the chemical sense: living things, food, wood, paper, cloth and plastic are organic; metal, glass, stone, water and salt are not.

Complaints also showed what players misread. "Is it X?" asks whether the thing itself is X, not whether it is related to X: for a slide, "Is it a park?" is No, but "Is it related to parks?" is Yes. The complaint form now says this.

Limits

The answers depend on Jev's probabilities and on what the fact sheet says, so some of them are off. This is most common for questions that people themselves would answer differently, and for words whose fact sheet doesn't cover what players ask. Complaints catch some of these, but only the ones someone bothers to send.

Play today's puzzle. No sign-up is needed.