A simplified functionality reproduction of Jev-style decisions with Gemma 4. Native multi-modal ability, finetuning free, runs locally on your machine.
Disclaimer - GemmaJev is NOT a full reproduction of TypeSafe's original Jev; see Limitations. This project is developed through ~100% vibe coding (powered by Codex w/ GPT-6 Astra). While tested, the code and documentation may contain critical bugs, hallucinations, or inaccuracies. Use at your own risk and verify critical results independently. If you encounter any problem, feel free to open an issue.
Contributing - Any interesting contribution is welcome. Feel free to PR.
GemmaJev turns a decision problem into a multiple-choice question, reads and normalizes the next-token logits for its option labels. No CoT or answer text is generated.
Your request
→ assign single-token labels to options
→ fixed prompt
→ model prefill
→ fetch logits of option labels
→ softmax
→ option-ID probabilities
The model takes a unified prompt template for all decision tasks:
{task}
Your response must begin immediately with one valid option label.
Output only that label without explanation.
Current state:
{state}
Query:
{query}
Options:
A = {option_id}: {option_description}
B = {option_id}: {option_description}
...
Input fields:
task - A fixed description of the decision task.
state - The context or content for the current decision, such as a customer message, supporting evidence, or a proposed answer.
query - The specific question to answer or claim to evaluate using the information in state and the available options.
options - One option is a mapping from a short, meaningful decision term (option_id) to a detailed description (option_description) of the decision. Each option is assigned a single-token label (e.g., A, B, C, ...) for logits decoding.
A typical input example:
{
"task": "Read the evidence and judge the claim using only the evidence provided. Distinguish a contradiction from missing information.",
"state": "Evidence: The museum opens at 10 a.m. on Sundays. Visitors must book a ticket in advance. Admission is free for children under 12.",
"query": "Claim: The museum allows pets.",
"options": {
"supported": "The evidence supports the claim.",
"contradicted": "The evidence contradicts the claim.",
"unknown": "The evidence does not establish whether the claim is true or false."
}
}Notes: The caller owns the self-consistency of its inputs: if
taskrefers to evidence or a claim, those terms should be defined instateandquery, as above. The public API does not require knowledge of internal prompt headings.
The response is only the probability mapping:
{
"supported": 0.01,
"contradicted": 0.01,
"unknown": 0.98
}GemmaJev uses Gemma 4's native image and audio inputs to make decisions about visual content or spoken language. The same task, state, query and options fields define the decision; one image or audio clip is supplied separately. The output remains a probability mapping over the supplied option IDs.
The release preserves 9,176 measured decisions across 13 datasets, with 4,588 examples per model. Both configurations use the same final interface. Detailed protocols, source revisions and metric definitions are in Benchmarks; the compact result archive can be verified without loading a model.
Each task below contains 200 examples. Jev results are published community measurements of the original TypeSafe API, not measurements made by this project.
| Task | Metric | TypeSafe Jev | GemmaJev E4B | GemmaJev 12B |
|---|---|---|---|---|
| AG News | Acc ↑ | 87.00% | 84.50% | 82.50% |
| BoolQ | Acc ↑ | 91.50% | 85.00% | 89.50% |
| MMLU | Acc ↑ | 93.50% | 66.00% | 73.00% |
| LogiQA | Acc ↑ | 76.50% | 34.50% | 49.00% |
| ANLI R3 | Acc ↑ | 66.00% | 45.00% | 52.50% |
| Emotion | Acc ↑ | 50.50% | 49.50% | 47.50% |
| HelpSteer2 | Spearman ↑ | 0.4902 | 0.4686 | 0.5676 |
Jev's historical inputs were not published. We reconstructed the public selection recipe and documented input repairs, so this is a task-level comparison. Source and comparison conditions.
Conclusion - GemmaJev 12B is competitive on reading judgments and helpfulness ranking, while knowledge and complex reasoning remain substantially behind Jev.
TL;DR - GemmaJev is a highly competitive LLM-to-Jev solution without task-specific fine-tuning. Using off-the-shelf models and a unified prompt template, it matches or outperforms several existing open-source alternatives on the evaluated benchmarks, even with smaller model size.
| System and published configuration | Task-specific training | Banking20 F1 ↑ | Newsgroups F1 ↑ | Injection F1 ↑ | Typed accuracy ↑ | Typed MAE ↓ |
|---|---|---|---|---|---|---|
| GemmaJev E4B | No | 83.72% | 64.75% | 82.75% | 67.40% | 0.4236 |
| GemmaJev 12B | No | 85.84% | 68.31% | 92.88% | 72.50% | 0.4530 |
| AnyJev · Qwen2.5-7B-Instruct · raw | No | 73.77% | 66.99% | 67.46% | 62.05% | 0.4375 |
| AnyJev · Qwen2.5-7B-Instruct · L1 | No | 78.12% | 71.35% | 77.79% | 63.20% | 0.4427 |
| AnyJev · Qwen3-32B · L1 | No | — | — | — | 70.05% | 0.4122 |
| Laya · laya-typed-decisions · fine-tuned | Yes | — | — | — | 76.80% | 0.2425 |
| TypeSafe Jev 1.13.0 · reported API | Yes | — | — | — | 72.70% | 0.3910 |
| System | Task-specific training | Authored144 accuracy | WANLI accuracy |
|---|---|---|---|
| GemmaJev E4B | No | 81.94% | 70.14% |
| GemmaJev 12B | No | 95.83% | 75.69% |
| SemIf · Qwen3.5-4B · direct | No | 80.56% | 64.58% |
| SemIf · Qwen3.8-27B-exl3 · direct | No | 95.83% | — |
Task-specific training indicates additional model-weight training for decision tasks, excluding the base checkpoint's original pretraining and instruction tuning.
The repo is tested on a Macbook Pro with MacOS, Apple M2 Pro, 16 GB unified memory, so this release targets Apple Silicon and Metal; other operating systems are outside its supported scope. Contributions on other platform configurations are welcome.
Install uv, CMake and Xcode Command Line Tools (xcode-select --install). Python 3.12 is selected by .python-version; the package requires Python 3.11 or newer.
From a source checkout:
uv sync --locked --extra data
uv run gemmajev setup --model 12b --visionSetup downloads and builds the pinned llama.cpp runtime, then downloads and verifies the selected model. --vision prepares both text and multimodal workers plus the matching projector, as required by the Playground. Model downloads are approximately 4.80 GiB for E4B and 6.50 GiB for 12B, plus a 946 MiB / 167 MiB projector respectively and runtime source/build space. Use --model e4b for E4B, or --model all to prepare both. Omit --vision for text-only API use.
| Local directory | Contents |
|---|---|
.models/ |
Verified model weights |
.runtime/ |
Downloaded source, compiled runtime, logs and fresh benchmark runs |
.datasets/ |
Downloaded benchmark data and preparation caches |
These directories are ignored by Git. The repository includes code, download manifests and measured-result metadata, not model weights, runtime binaries or dataset passages.
uv run gemmajev demo --model 12bThis launches a GemmaJev Playground with examples in your browser. You can also edit the fields, upload images and audio to test your own cases.
from gemmajev import GemmaJev
request = {
"task": "Read the evidence and judge the claim using only that evidence.",
"state": "Evidence: The meeting has been cancelled.",
"query": "Claim: The meeting is cancelled.",
"options": {
"supported": "The evidence supports the claim.",
"contradicted": "The evidence contradicts the claim.",
"unknown": "The evidence does not establish either conclusion."
}
}
with GemmaJev(model="12b") as model:
probabilities = model.decide(request)
print(probabilities)The same request can be supplied as a JSON file or via stdin:
uv run gemmajev decide src/gemmajev/examples/evidence_check.json --model 12b
uv run gemmajev decide - --model 12b < request.jsonFor image input, prepare the model with setup --model 12b --vision as above, then enable vision and pass the image separately:
from gemmajev import GemmaJev
request = {
"task": "Assign the most specific style tag to the attached image. Judge visual appearance, regardless of how the image was created.",
"state": "The attached image is the item to tag.",
"query": "Which style tag best describes this image?",
"options": {
"photography": "Camera-like photography or a photorealistic image.",
"artwork": "Painting, sketch, or illustration without anime/cartoon styling.",
"anime": "Anime, manga, or cel-shaded 2D cartoon illustration.",
"other": "A diagram, chart, document, screenshot, or another style."
}
}
with GemmaJev(model="12b", vision=True) as model:
probabilities = model.decide(
request, image_path="src/gemmajev/examples/images/sample_4.png"
)
print(probabilities)The same image-style request is available as a bundled JSON file. --image automatically selects the vision runtime; the image path is not a field in the request JSON:
uv run gemmajev decide src/gemmajev/examples/image_style.json --model 12b \
--image src/gemmajev/examples/images/sample_4.pngAudio uses the same projector prepared by setup --vision (or setup --audio). The Python and CLI interfaces accept a PCM16, mono, 16 kHz WAV clip between 40 milliseconds and 30 seconds:
import json
from pathlib import Path
from gemmajev import GemmaJev
request = json.loads(Path("src/gemmajev/examples/audio_language.json").read_text())
with GemmaJev(model="12b", audio=True) as model:
probabilities = model.decide(request, audio_path="speech.wav")
print(probabilities)uv run gemmajev decide src/gemmajev/examples/audio_language.json --model 12b \
--audio speech.wavPass either an image or audio attachment per decision. The Playground converts supported WAV recordings to the required audio format before upload.
Recompute all published local scores offline:
uv run gemmajev benchmark verify
uv run gemmajev benchmark reportDownload the frozen data selections, run a short check, then run a complete model panel:
uv run --extra data gemmajev benchmark prepare --dataset all
uv run --extra data gemmajev benchmark run --model 12b --dataset boolq --limit 3
uv run --extra data gemmajev benchmark run --model 12b --dataset allUse --model e4b for the other configuration after preparing its weights. --limit is a smoke check, not a replacement for the published full panel.
See Benchmarks for complete commands and reproduction conditions.
For development, run
uv run --extra data pytest
uv run ruff check .GemmaJev is inspired by TypeSafe Jev. We credit the TypeSafe team for the original Jev system that motivated this exploration.
- Simplified decision interface - GemmaJev does not reproduce Jev's complete
Noul,Choice, andScoreI/O interfaces. It approximates one aspect of Jev-style decision-making by returning a probability distribution over user-defined options. - Research toy - This project is a research prototype. Production infrastructure and comprehensive inference optimization are outside its scope. Its benchmark results should not be interpreted as equivalence to Jev's complete service.
- No calibration - The output probability distributions are not calibrated estimates of real-world correctness as promised in original Jev.
- Performance gap - There are clear weaknesses relative to the published Jev reference, especially in the area of Knowledge and complex reasoning.
GemmaJev is released under the MIT License. Model weights, datasets and third-party components retain their respective licenses.
- TypeSafe Jev.
- Google Gemma and its model terms; checkpoint versions and hashes are pinned in models.toml.
- llama.cpp; see the runtime notes for the pinned revision and required patches.
- AnyJev, SemIf, and Laya Typed Decisions.
- Benchmark data and result provenance. Models and datasets retain their upstream licenses.