A Chinese-language tech news platform that brings together RSS fetching, DeepL translation, and Gemini AI summaries in one place.
Fetches global tech news sources on a schedule → auto-translates English titles → presents them clearly grouped by source.

_{English · 简体中文}

Features · Architecture Overview · Quick Start · Project Structure

🎯 Features

🌐 Multi-source Aggregation Articles grouped by source, homepage sorted by priority, with lazy loading for stale sources	🌍 English Title Translation DeepL API automatically translates Latin-script titles to Chinese, falling back to the original on failure
🤖 AI Article Summaries One-click 50–80 character Chinese summaries (Gemini flash-lite), results cached in D1	🔥 V2EX Hot Topics Support Firecrawl scrapes the homepage hot list + fetches full content item by item, covering sources without RSS
⚡ Async Fetching Cloudflare Queues for async consumption, avoiding synchronous blocking on Pages	🔐 Hidden Admin Panel Triple-click the title to enter /fish, protected by GitHub login, admin-only access
👤 User Login & Read State GitHub OAuth login (better-auth), read status persisted in D1	⬇️ Works Offline Too Anonymous users fall back to localStorage for read state — the anonymous experience is unchanged

🏗️ Architecture Overview

Two deployable units + one shared library, with the fetch logic maintained in a single place:

• web (Cloudflare Pages): TanStack Start SSR + Hono API — both frontend and API server, and the Queue producer.
• fetcher (Cloudflare Workers): a cron-triggered fetching Worker, the Queue consumer, responsible for fetching RSS → dedup → DeepL translation → writing to D1.
• @repo/ingest (packages/ingest): the core fetching library, shared by fetcher and web.

⚡ Quick Start

Prerequisites

• Node.js ≥ 20 + pnpm 9
• Wrangler CLI logged in

1. Install

pnpm install

2. Create the D1 database

wrangler d1 create news-aggregator
# Fill the generated database_id into apps/web/wrangler.toml and apps/fetcher/wrangler.toml
wrangler d1 migrations apply news-aggregator --local

3. Create the Queue

wrangler queues create news-fetch-queue

4. Configure API keys

# Fetcher
cp apps/fetcher/.dev.vars.example apps/fetcher/.dev.vars
# Fill in DEEPL_API_KEY, FIRECRAWL_API_KEY

# Web — AI article summaries
echo "GEMINI_API_KEY=your_key" >> apps/web/.dev.vars

5. Start

# Terminal 1: frontend + API
pnpm dev:web

# Terminal 2: fetching Worker
pnpm dev:fetcher

Visit http://localhost:3000 for the homepage. The admin panel lives at /fish (triple-click the homepage title to enter).

Local development note: when the web .dev.vars sets DIRECT_FETCH=true, the API skips the Queue and fetches synchronously; locally the fetcher and web do not share a Queue.

📁 Project Structure

.
├── apps/
│   ├── web/                  # TanStack Start frontend + Hono API (Cloudflare Pages)
│   │   ├── app/routes/       # File routes (index.tsx / fish.tsx / api/$.ts)
│   │   ├── app/server/       # Hono routes (articles / sources / auth)
│   │   └── app/components/   # React components
│   └── fetcher/              # Cron fetching Worker (src/index.ts only, logic reused from @repo/ingest)
├── packages/
│   ├── db/                   # Drizzle schema + D1 client (sources / articles tables)
│   ├── ingest/               # Core fetching library (RSS / dedup / DeepL / Firecrawl / V2EX)
│   ├── shared/               # Shared types & utilities (normalizeUrl / shouldFetchNow, etc.)
│   └── telemetry/            # Structured logging (logger)
├── turbo.json                # build / dev / typecheck / lint / deploy
├── pnpm-workspace.yaml
└── .pnpmfile.cjs             # Pins @tanstack/* to 1.114.1

📊 Data Model

Two D1 tables (full definitions in packages/db/src/schema.ts):

Table	Key columns
sources	id / name / url / priority / fetch_frequency (hourly|twice_daily|daily) / is_active / last_fetched_at
articles	id / source_id (FK cascade) / title / translated_title / url (unique constraint) / published_at / summary (AI cache) / metadata (json)

Key indexes: articles_source_url_idx (per-source dedup across URLs), articles_source_published_idx, sources_priority_idx

📡 API Overview

All APIs are mounted under /api/* via app/routes/api/$.ts, with request validation via zValidator + Zod:

Method	Path	Description
GET	/api/articles/grouped?scope=active|stale	Articles grouped by source; active returns active sources only, with an X-Has-Stale response header
GET	/api/articles/:id/summary	AI summary (four-level fallback: D1 cache → live fetch with real browser UA → AMP variant → RSS intro → Browser Rendering)
POST	/api/sources/:id/fetch	Trigger a fetch for a single source
POST	/api/sources/fetch-all	Trigger a fetch for all sources
POST	/api/sources/detect	Auto-detect an RSS feed from a URL
*	/api/sources/*	Source CRUD / bulk enable-disable / bulk delete
GET	/api/read/articles	Get the current user's read article ID list
POST	/api/read/articles	Mark a single article as read
POST	/api/read/articles/batch	Mark multiple articles as read
*	/api/auth/*	better-auth: GitHub OAuth login / logout / session

🧪 Code Quality

husky runs the quality gates automatically before each commit (cannot be skipped with --no-verify):

pnpm lint        # Biome checks (formatting + import sorting + lint)
pnpm typecheck   # tsc --noEmit across all workspaces
pnpm test        # Vitest unit tests (node pool, pure logic)

Other common commands:

pnpm lint:fix    # Biome auto-fix + formatting
pnpm test:watch  # Unit tests in watch mode
pnpm test:e2e    # End-to-end tests (real workerd + real D1, run manually)
pnpm build       # Build all workspaces
pnpm cleanup     # Clean up orphaned workerd / esbuild processes
pnpm db:generate # Generate Drizzle migrations

🚀 Deployment

Resource preparation

1. Create the D1 database news-aggregator in the Cloudflare dashboard and copy its database_id into both wrangler.toml files
2. Create the Queue news-fetch-queue
3. Apply migrations: wrangler d1 migrations apply news-aggregator --remote
4. Configure secrets (wrangler secret put <NAME>):

Secret	Location	Purpose
DEEPL_API_KEY	fetcher	DeepL translation
FIRECRAWL_API_KEY	fetcher	V2EX anti-scraping fetch
GEMINI_API_KEY	web	AI article summaries
GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET	web	GitHub OAuth login
BETTER_AUTH_SECRET	web	Session signing (login state is lost after deploy if unset)

GitHub OAuth note: an OAuth App can only have one callback URL. Recommended: create two Apps (production + local), or temporarily swap credentials during local development.

Deploy commands

pnpm deploy:fetcher      # Deploy the fetching Worker
pnpm deploy:web          # Deploy the frontend + API

better-auth tables are created automatically on first request. After the first deploy, you may manually trigger a migration once (to avoid being swallowed by the /api/auth/* wildcard):

curl -X POST https://<your-domain>/api/admin-migrate

⚠️ Known Pitfalls

• Local D1 separation: web and fetcher each use their own sqlite file — after adding a migration, apply it to both local databases
• TanStack version pinning: .pnpmfile.cjs hard-pins @tanstack/* to 1.114.1; update that file in sync when upgrading
• Single static assets directory: favicon / logo / _headers live only in apps/web/public/; the root public/ has been removed
• Test content-extraction changes for real: live fetching for article summaries relies on @mozilla/readability — after changing packages/shared/src/extract.ts, always test against real article pages
• Local fetcher doesn't receive web's messages: locally the two processes don't share a Queue; web needs DIRECT_FETCH=true to fetch synchronously