jevtok-ts

Offline Jev token counting and request accounting for Node.js and Next.js. A TypeScript port of LabGuy94/jevtok, matching its reconstructed encoder for jev-1.13-20260917.

No Python, network calls, API keys, native extensions, or runtime dependencies. Vocabulary and lookup tables ship inside the package. Ordinary o200k_base tokenization is not interchangeable with Jev: evidence is three tokens and 2024 is four.

Install

npm install jevtok-ts

Requires Node.js 22 or later; CI covers Node 22 and 24. ESM and CommonJS are supported with TypeScript declarations. Browser and Edge runtimes are not supported.

Quick start

import { getEncoding, estimateRequestTokens } from 'jevtok-ts';

const enc = getEncoding('jev-1.13');
enc.count('evidence'); // 3
enc.encode('evidence'); // [68, 1429, 1082]
enc.decode([68, 1429, 1082]); // 'evidence'
enc.encodePieces('evidence'); // Uint8Array[]: e | vid | ence

const questions = {
  relevant: { type: 'noul' as const, instructions: 'Is this about software?' },
};
const inputTokens = estimateRequestTokens({ title: 'Managed Postgres' }, questions);

CommonJS: const { getEncoding } = require('jevtok-ts');.

API

Export	Result
getEncoding(name?) / encodingForModel(name)	Cached Encoding; default jev-1.13
listEncodingNames()	['jev-1.13']
enc.count(text)	Token count after NFC normalization
enc.encode(text)	number[] in the reconstructed o200k rank ID space
enc.encodePieces(text)	Exact token bytes as Uint8Array[]; individual pieces may not be valid UTF-8
enc.decode(ids)	Decoded string; invalid UTF-8 is replaced like upstream
enc.decodeBytes(ids)	Concatenated bytes as Uint8Array
enc.name, enc.modelVersion, enc.nVocab	jev-1.13, jev-1.13-20260917, 200019
estimateRequestTokens(state, questions?, options?)	Estimated usage.input_tokens; throws on budget violations
analyzeRequestTokens(state, questions?, options?)	Counts, effective limits and all violations without throwing for limits
createJevGuard(options?)	Reusable check() / assert() over an exact {state, questions} payload
stateCost(state, encoding?), questionCost(question, encoding?)	Individual contributions

state accepts JSON values. Questions are a record of noul, choice, or score definitions. choice.criteria maps option names to strings or null; score.criteria is a nonempty string array; optional noul.criteria contains true and false strings. Extra noul criteria keys do not contribute to the count.

Without questions, the upstream probe { q: { type: 'noul', instructions: 'x' } } is used; estimateRequestTokens('') === 267. Empty question records are invalid. Cycles, sparse arrays, non-JSON values, non-finite numbers, and lone UTF-16 surrogates are rejected. Shared, non-cyclic objects are accepted.

Structured state is not JSON text tokenization. Strings, keys, separators and containers contribute separately. Numbers follow upstream binary64 normalization and Python-style rendering. See the method.

Context budgets

import { analyzeRequestTokens, estimateRequestTokens, ContextLimitExceeded } from 'jevtok-ts';

const options = {
  safetyMargin: 256, // subtract 256 tokens from each limit
  limits: { total: 64000, stateAndLongestQuestion: 32736 },
};
const analysis = analyzeRequestTokens(state, questions, options);
// analysis.totalTokens, stateTokens, longestQuestionTokens, questionTokens,
// limits, violations, modelVersion

try {
  estimateRequestTokens(state, questions, options);
  // Call Jev using exactly the same state and questions.
} catch (error) {
  if (error instanceof ContextLimitExceeded) {
    console.log(error.violations); // [{ limit, actual, maximum }]
    // Trim or split in your application, then count again.
  } else {
    throw error;
  }
}

Defaults:

• state + longest question body <= 32,736: upstream's measured boundary.
• total estimated input <= 64,000: a conservative library policy, including accounting overhead. TypeSafe documents a 64k aggregate budget without an exact numeric boundary; 64,000 is not claimed as a recovered server cutoff.
• safetyMargin: 0. Counts are never modified by a margin; only admission thresholds change. Both budgets can be overridden explicitly.

ContextLimitExceeded exposes the same diagnostic fields as the analysis. Invalid input throws InvalidRequestError, malformed Unicode throws InvalidUnicodeError, and unsupported models throw UnknownEncodingError.

Preflight guard

Use the framework-agnostic guard immediately before the provider call. check() returns the full analysis and an ok flag; assert() returns the analysis or throws ContextLimitExceeded.

import { createJevGuard } from 'jevtok-ts';

const jevGuard = createJevGuard({ safetyMargin: 1024 });
const result = jevGuard.check({ state, questions });
if (result.ok) await callJev({ state, questions });
else console.log(result.analysis.violations);

Malformed request envelopes and invalid Jev payloads continue to throw InvalidRequestError. The guard delegates accounting to analyzeRequestTokens, so both existing limits and ContextLimitExceeded diagnostics stay identical.

Version mapping

Package	Encoder	Model reconstruction	Accepted static aliases
0.2.0	jev-1.13	jev-1.13-20260917	jev-1.13, jev-1.13.0, jev-1.13-20260917, each also with typesafe/ prefix
0.1.0	jev-1.13	jev-1.13-20260917	jev-1.13, jev-1.13.0, jev-1.13-20260917, each also with typesafe/ prefix

jev-latest, jev-preview, and unknown versions throw. A new model requires new parity evidence and an explicit definition. An alias accepted by this library identifies the recorded reconstruction; it cannot guarantee that a provider has not changed its implementation.

Next.js

The separate jevtok-ts/next export wraps a Jev provider call. It accepts the exact request as its first argument, or an explicit extract function for other call signatures. onLimit can map the typed error to an HTTP response:

import { withJevPreflight } from 'jevtok-ts/next';
import type { JevRequest } from 'jevtok-ts';

export const runtime = 'nodejs';

// callJev is your existing server-side provider function.
const guardedCallJev = withJevPreflight(async (input: JevRequest) => callJev(input), {
  safetyMargin: 1024,
  onLimit: (error) =>
    Response.json({ error: error.name, violations: error.violations }, { status: 413 }),
});

// Validate/normalize request.json() with your application schema first.
export async function POST(request: Request) {
  const input = (await request.json()) as JevRequest;
  return guardedCallJev(input);
}

Importing this subpath in a Client Component fails through Next.js's server-only boundary. See examples/nextjs for runnable App Router examples of request analysis and error mapping. No model call is made by the demo. Apply your normal request validation, request-size and access controls when exposing a public endpoint.

CLI

npx jevtok-ts count 'The tokenization of evidence'  # 7
npx jevtok-ts encode evidence                    # [68,1429,1082]
npx jevtok-ts encode --pieces evidence           # ["ZQ==","dmlk","ZW5jZQ=="]
npx jevtok-ts count -f input.txt
npx jevtok-ts estimate request.json --json

Text commands read stdin when no text/file is supplied. --pieces prints base64 to preserve partial UTF-8 bytes. estimate accepts { "state": ..., "questions": ... } and exits nonzero on a context violation.

Optional live verification

Only verify makes API calls. Set OPENROUTER_API_KEY or TYPESAFE_API_KEY in your environment; keys are never accepted as CLI arguments or written to reports.

npx jevtok-ts verify evidence --endpoint openrouter
npx jevtok-ts verify -f public-probes.txt --endpoint typesafe --report result.json
npx jevtok-ts verify --requests public-requests.json --endpoint openrouter

A text run checks both the measured text delta from an empty-state baseline and the full request count. A requests file is a JSON array of {state, questions} objects. The JSON report records the resolved model and expected/actual counts; mismatches or API failures exit nonzero. Text mismatches include the submitted text. Send only public or synthetic probes that you may disclose.

Each run allows at most 40 calls and a $0.10 estimated spend reserve, configurable downward with --max-calls and --max-cost-usd. There are no automatic retries. The reserve uses twice the documented $0.042/M input price plus a per-call allowance; this is a local guard, not a provider-enforced billing cap. Requests are rejected locally before exceeding the reserve. Unknown/missing response model versions fail verification.

Verification and performance

This port's offline suite passes 2,248 published upstream text fixtures and 371 published upstream request fixtures, with zero count mismatches. A separate differential check compares 3,792 texts (IDs and bytes), 5,009 binary64 values, and 471 requests directly with the pinned Python implementation.

The upstream author's larger research totals (20,034 validation strings and 403 accepted requests) describe their work, not the fixtures shipped here or an independent TypeScript live evaluation. Token IDs and boundaries are a reconstruction convention: the provider exposes counts, not token IDs. Exact agreement on tested inputs does not prove equality on all possible inputs or future models.

See verification results for measured live, package, Next.js and performance results. Run npm run benchmark on your own hardware; cold initialization and warm request cost differ.

Development

npm ci
npm run typecheck
npm test                    # offline; no Python or API keys needed
npm run test:package        # packed ESM/CJS/types/CLI consumer smoke
node scripts/nextjs-smoke.mjs
npm run benchmark

See CONTRIBUTING.md for Python differential checks, fixture provenance, data regeneration and release checks. CHANGELOG.md records package and encoder changes.

Attribution and license

MIT. This is an independent port of LabGuy94/jevtok, whose research recovered the lookup sets, merge restrictions and request formula. The o200k vocabulary comes from OpenAI tiktoken. This project is not an official TypeSafe tokenizer and is not affiliated with TypeSafe or OpenAI.

Upstream copyright and third-party notices are preserved in LICENSE and THIRD_PARTY_NOTICES.md.