Repository navigation
Feat/merge sieve - #4
Merged
Merged
Conversation
Annotate everything to pass mypy --strict at a py310 target, fix ruff findings (line lengths, zip strict=, import order), and move the runnable PiiModel smoke script from tests/ to examples/ — it is a demo that downloads the model, not a pytest test. No behavior changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bring over the quality gate that guarded the Sieve gateway repo: - ruff at py310 incl. T20 (no print in src/ — the privacy invariant that text being redacted never reaches stdout/stderr), tests/examples exempt - mypy --strict over src/ and tests/, ML libs treated as untyped - pytest config without the pythonpath hack (uv installs the project) - Makefile with the `make check` gate, CI matrix on 3.10 + 3.12, pre-commit hooks, and a committed uv.lock First step of merging Sieve (local PII gateway) into privacy-kit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merge Sieve's Span into the published one: same field order (start, end, label, score), now frozen with slots, plus text_of() for pulling the covered substring out of the source text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the stride-truncation chunking and strict-BIO decode with the algorithm that shipped in the Sieve gateway, while keeping the dependency-light loading (huggingface-hub + tokenizers + onnxruntime — still no torch/optimum): - overlapping token windows batched 8 per forward pass; on tokens covered by several windows the most confident prediction wins - subword predictions aggregated to whole words so partial tokens can never leak (e.g. "ample.com" surviving a masked email) - consecutive same-type words merged; spans below the threshold dropped, trailing punctuation trimmed (trim_spans), overlaps deduped - warmup() to keep first-inference latency off the hot path, labels property exposing the canonical entity-type set - ORT session tuning is now constructor-configurable; defaults changed from pinned single-thread/no-optimization to onnxruntime defaults (the old settings were a needless latency hit for gateway use) - explicitly disable the 512-token truncation baked into the shipped tokenizer.json — windowing handles long inputs, encode must not clip Real-model tests (PII_RUN_MODEL_TESTS=1) port Sieve's behavioral contract: expected types across 5 languages, whole-word coverage, valid offsets, no overlaps, and a >512-token document keeping an entity at its end. Verified span-level parity against the old optimum-based detector: identical on all language fixtures; the long document differs only in near-threshold boundary splits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port Sieve's Vault: a bidirectional, type-aware map between real PII values and [TYPE_N] placeholders. anonymize()/anonymize_into() replace detected spans (numbering follows reading order, splicing happens right-to-left so offsets stay valid); deanonymize() restores the real values. Same value -> same placeholder within a vault; idempotent. This is the reversible counterpart to Redactor: redact where the destination must never see values (observability), pseudonymize where a live LLM loop needs referential consistency and the caller needs the values back. Exported from privacy_kit top level. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Start porting the Sieve gateway. The `gateway` extra brings FastAPI, httpx, uvicorn, SQLModel, typer, and pydantic-settings; the base install stays lean (test_packaging now blocks those imports too). - gateway.config: pydantic-settings under the same PII_* env prefix the core library reads (PII_MODEL_ID, PII_THRESHOLD), plus PII_DB_PATH, PII_HOST/PII_PORT, upstream and OTLP-downstream URLs - gateway.store: SQLModel audit schema (interactions + per-type detections) and repository. Hard invariant: no column ever stores raw PII — asserted by a test that dumps every cell of the DB - privacy-kit console-script shim that points at the gateway CLI and fails with an install hint when the extra is missing - datetime.UTC -> timezone.utc for the py310 floor; CI installs the gateway extra Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The core of the gateway, ported from Sieve onto the unified core (Vault + Detector protocol): - proxy routes: Anthropic Messages (+ count_tokens) for Claude Code, OpenAI Chat Completions (Cursor/Codex chat), OpenAI Responses (Codex) - per request: pseudonymize every human-text field (incl. tool_use inputs, tool_result contents, tool-call arguments) off the event loop, forward with the client's own auth, rehydrate the response, write a metadata-only audit row - SSE streaming de-anonymization with cross-chunk placeholder buffering; stream usage audited; upstream refusals to stream pass through with their real status code - gzip/deflate (and optional br/zstd) request bodies decoded - OTLP/HTTP JSON sink on /v1/logs|traces|metrics: scrubs every stringValue one-way, audits logs, optional downstream re-export - source header renamed to x-privacy-kit-source; the legacy x-sieve-source is still honored for one release Tests: full stub-based suite (proxy round-trips per wire format, tool-block sanitization, streaming splits, usage capture, log-safety leak checks) plus real-model e2e for the Claude Code path, both non-streaming and 3-char-delta streaming (PII_RUN_MODEL_TESTS=1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- `privacy-kit serve` boots the gateway with the real detector warmed up; `setup claude-code|codex|cursor` prints the exact *_BASE_URL routing (incl. optional OTLP env); `report` summarizes the audit DB; `scan` does a one-off detection or --anonymize of a file/stdin - Dockerfile: python-slim + `pip install .[gateway]` + model baked in at build time — no torch at all, gigabytes slimmer than the Sieve image it replaces - Makefile: serve/docker-build/docker-run targets; test-model now also covers the e2e and CLI scan suites Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README now tells the unified story — privacy blocks at every trust boundary: redact vs pseudonymize, in-process (Redactor/Vault, Langfuse/LangChain) vs network path (gateway for Claude Code, Codex, Cursor, OTLP). Adds gateway quick-start, Vault section, the PII_* env reference, and a development section. CHANGELOG covers the detector pipeline upgrade, ORT session default change, and the Sieve->privacy-kit migration notes (CLI, env vars, deprecated x-sieve-source header). Verified from a clean venv: base wheel imports lean (no gateway/ML deps loaded) and the console script prints the extras hint; [gateway] wheel serves the CLI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… auth Claude Code's Max/Pro subscription requests carry an exact identifier as the first system block — "You are Claude Code, Anthropic's official CLI for Claude." — which Anthropic validates as an anti-spoofing check. The detector tags "Claude Code"/"Anthropic" there as PERSON_NAME, so the gateway was anonymizing the identifier and would have gotten OAuth requests rejected. The Anthropic request transform now keeps that identifier preamble verbatim (string or first text block) while still anonymizing everything after it — the user's CLAUDE.md, environment, and all messages. Auth headers (authorization, anthropic-beta, anthropic-version) already pass through untouched, so subscription auth now survives the proxy end to end. Verified with the real model: identifier byte-for-byte intact, surrounding PII still masked. `setup claude-code` now documents the subscription path (no API key; `claude setup-token` as the fallback for a subscription-backed token). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude Code applies the `env` block of ~/.claude/settings.json to every new session, so routing it through the gateway is one JSON edit instead of an export the user has to remember. New module gateway/route.py owns that edit and the CLI grows two entry points: - `privacy-kit setup claude-code --apply` writes ANTHROPIC_BASE_URL into the settings env block persistently (`--remove` undoes it) - `privacy-kit serve --route claude-code` applies the override on startup and restores the previous value on shutdown Safety rails: the CLI always prints exactly which file it overrides; corrupt settings JSON is never clobbered (fail with a clear error); and revert only touches the value if it is still the one we wrote — a manual user edit made while the gateway runs survives shutdown untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A basic local web UI served by the gateway itself — single inline HTML file, no build step, no external assets (a privacy tool must not pull JS from a CDN). Two panels: - Live preview: paste any text, see every detected span highlighted by entity type (with score) next to the pseudonymized version that would leave the machine. Processed in memory only — deliberately no audit row, no logging, nothing persisted (asserted by a test that greps the DB file). - Audit: the metadata-only story — totals by entity type and recent interactions from the audit store. JSON endpoints: POST /ui/api/preview (50k-char cap), GET /ui/api/summary, GET /ui/api/recent. `serve` now prints the UI URL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat: make setup
feat: add web component
…e-sieve # Conflicts: # Makefile # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.