Serina is a voice-activated AI assistant that lives on your desk as a physical ESP32-S3 device. Speak to it, and it thinks, talks back, and shows expressive animated robot eyes on its OLED screen. Backed by Claude AI, it can check the weather, read news, send WhatsApp messages, query your calendar, control smart home devices, and more.
┌─────────────────────────────────────────────────────────────────────┐
│ ESP32-S3 Device │
│ │
│ [BOOT Button] ──► Press to speak │
│ [INMP441 Mic] ──► I2S audio capture ──► 16kHz PCM chunks │
│ [MAX98357A ] ◄── I2S audio playback ◄── 16kHz PCM from server │
│ [SSD1306 OLED] ── RoboEyes animated eyes (50 fps, Adafruit GFX) │
│ │
│ WebSocket client ◄──────────────────► WiFi ──► Server :8000 │
└─────────────────────────────────────────────────────────────────────┘
│
Binary WebSocket (custom protocol)
│
┌─────────────────────────────────────────────────────────────────────┐
│ Python Server (PC) │
│ │
│ FastAPI + uvicorn ──► /ws WebSocket endpoint │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
│ │ VAD │ │ Wake Word │ │ Whisper STT │ │
│ │ (Silero) │──►│ (openWW) │──►│ (faster-whisper base) │ │
│ └─────────────┘ └──────────────┘ └────────────┬───────────┘ │
│ │ transcript │
│ ┌──────▼──────────┐ │
│ │ DOE Orchestrator│ │
│ │ (Claude Haiku) │ │
│ │ + 8 Tool Skills │ │
│ └──────┬──────────┘ │
│ │ text reply │
│ ┌──────▼──────────┐ │
│ │ Edge-TTS │ │
│ │ (en-US Aria) │ │
│ └──────┬──────────┘ │
│ │ PCM audio │
│ streamed back to ESP32 │
└─────────────────────────────────────────────────────────────────────┘
- Wake word — openWakeWord continuously monitors mic stream for
hey_jarvis - Listen — Press BOOT button; device streams raw 16kHz/16-bit PCM to server over WebSocket
- VAD stop — Silero VAD detects 0.8 s of silence → signals end-of-utterance to server
- STT — faster-whisper transcribes audio to text
- DOE Orchestrator — sends transcript + conversation history to Claude Haiku; Claude may call one or more tool functions (weather, news, calendar, …)
- Tool execution — Python scripts in
server/execution/run the actual API calls and return results - TTS — edge-tts generates MP3 audio, ffmpeg decodes to 16kHz/16-bit PCM
- Playback — PCM chunks streamed back over WebSocket; ESP32 plays via I2S speaker
- Face — server sends state commands (
SPEAKING,THINKING,IDLE, …) over WebSocket; ESP32 updates RoboEyes expression in real time
| Component | Spec | Pins |
|---|---|---|
| Brain | Xiaozhi AI Mini ESP32-S3, 16 MB flash, 8 MB PSRAM | — |
| Microphone | INMP441 (I2S MEMS) | WS=4, SCK=5, SD=6 |
| Speaker | MAX98357A (I2S Class-D amp) | DIN=7, BCLK=15, LRC=16 |
| Display | SSD1306 128×64 OLED (I2C) | SDA=41, SCL=42, addr=0x3C |
| Button | Built-in BOOT button (active LOW) | GPIO0 |
| Library | Purpose |
|---|---|
espressif32 Arduino framework |
ESP32-S3 HAL |
Adafruit SSD1306 @ ^2.5.10 |
OLED driver |
Adafruit GFX Library @ ^1.11.9 |
Graphics primitives used by RoboEyes |
FluxGarage RoboEyes |
Smooth animated vector robot eyes |
links2004/WebSockets @ ^2.4.1 |
WebSocket client for server comms |
bblanchon/ArduinoJson @ ^7.0.0 |
JSON handshake messages |
| Package | Purpose |
|---|---|
fastapi + uvicorn |
Async WebSocket server |
faster-whisper |
On-device speech-to-text (Whisper base) |
openwakeword |
Wake-word detection (hey_jarvis model) |
silero-vad (via torch) |
Voice activity detection — silence detection |
anthropic |
Claude Haiku LLM (DOE orchestrator) |
edge-tts |
Microsoft neural TTS (Aria voice) |
torch + torchaudio |
ML inference backend for VAD/wake word |
pydantic-settings |
Config management via .env |
| Component | Detail |
|---|---|
| LLM | claude-haiku-4-5, max 512 tokens |
| STT | faster-whisper base model, CPU inference |
| TTS | en-US-AriaNeural via edge-tts, decoded with ffmpeg |
| Wake word | hey_jarvis (openWakeWord built-in model) |
| Skill | Requires |
|---|---|
get_weather |
OPENWEATHER_API_KEY |
get_news |
NEWS_API_KEY |
get_calendar |
Google OAuth2 credentials |
get_email_summary |
Google OAuth2 credentials |
send_whatsapp |
WHATSAPP_API_KEY, WHATSAPP_PHONE_ID |
set_timer |
No key needed |
smart_home |
HOME_ASSISTANT_URL, HOME_ASSISTANT_TOKEN |
morning_briefing |
Combines weather + news + calendar |
AI Bot Project/
├── server/ # Python server
│ ├── main.py # FastAPI app + WebSocket endpoint
│ ├── session.py # Per-device session state machine
│ ├── doe_orchestrator.py # Claude Haiku + tool dispatch
│ ├── stt.py # faster-whisper STT
│ ├── tts.py # edge-tts → PCM pipeline
│ ├── vad.py # Silero VAD
│ ├── wake.py # openWakeWord detector
│ ├── llm.py # Anthropic client wrapper
│ ├── memory_db.py # In-memory conversation history
│ ├── tool_registry.py # Tool definitions for Claude
│ ├── script_runner.py # Executes skill scripts
│ ├── directive_loader.py # Loads skill docs into system prompt
│ ├── protocol.py # WebSocket message types
│ ├── config.py # All settings (reads from .env)
│ ├── requirements.txt
│ ├── directives/ # Skill instruction docs (markdown)
│ │ ├── AGENTS.md # Serina personality + rules
│ │ ├── get_weather.md
│ │ └── ...
│ └── execution/ # Skill Python scripts
│ ├── get_weather.py
│ ├── get_news.py
│ └── ...
│
└── firmware/ # ESP32-S3 Arduino firmware
├── platformio.ini # Build config + library dependencies
├── src/
│ ├── main.cpp # FreeRTOS tasks, button, state machine
│ ├── display.cpp # RoboEyes face expressions
│ ├── display.h
│ ├── audio_capture.cpp # I2S mic → WebSocket PCM stream
│ ├── audio_capture.h
│ ├── audio_playback.cpp # WebSocket PCM → I2S speaker
│ ├── audio_playback.h
│ ├── websocket_client.cpp
│ ├── websocket_client.h
│ └── protocol.h # Shared state enum + message types
└── include/
└── config.h # WiFi, server IP, pins, audio params
- Python 3.11 installed
- PlatformIO CLI installed (
pip install platformio) - ffmpeg on system PATH (for TTS decoding)
- Git installed (PlatformIO fetches libraries from GitHub)
- A WiFi network that both your PC and ESP32 can join
cd "d:\AI Bot Project"Create a file called .env in the project root (d:\AI Bot Project\.env) with the following content:
# Required
ANTHROPIC_API_KEY=sk-ant-...
# Optional — only needed for the corresponding skills
OPENWEATHER_API_KEY=
NEWS_API_KEY=
WHATSAPP_API_KEY=
WHATSAPP_PHONE_ID=
HOME_ASSISTANT_URL=http://homeassistant.local:8123
HOME_ASSISTANT_TOKEN=
# TTS / STT tuning (defaults work fine)
WHISPER_MODEL=base
WHISPER_DEVICE=cpu
EDGE_TTS_VOICE=en-US-AriaNeural
WAKE_WORD=hey_jarviscd "d:\AI Bot Project\server"
pip install -r requirements.txtWindows note:
torchmay take several minutes to install. If you have a GPU, optionally swapWHISPER_DEVICE=cpu→cuda.
Open firmware/include/config.h and update:
#define WIFI_SSID "YourWiFiName"
#define WIFI_PASSWORD "YourWiFiPassword"
#define WS_HOST "192.168.x.x" // your PC's local IP (run ipconfig)
#define WS_PORT 8000Connect the ESP32-S3 via USB, then:
cd "d:\AI Bot Project\firmware"
pio run --target uploadPlatformIO will automatically download all libraries and flash the device. After flashing, the OLED will show tired/sleepy eyes (waiting for server connection).
cd "d:\AI Bot Project\server"
python main.pyThe server starts on 0.0.0.0:8000. You should see:
[INFO] Models loaded
[INFO] Uvicorn running on http://0.0.0.0:8000
Once the ESP32 connects over WebSocket, the OLED switches to the idle (neutral) expression.
| Action | Result |
|---|---|
| Say "Hey Jarvis" | Serina wakes up (excited eyes), ready to listen |
| Press BOOT button | Immediately start listening (skip wake word) |
| Speak your request | Eyes show listening expression |
| Pause (0.8 s silence) | Serina starts thinking (confused-shake eyes) |
| Serina replies | Eyes show happy+laugh expression while speaking |
Example requests:
- "What's the weather in London?"
- "Read me the top news headlines"
- "Set a timer for 5 minutes"
- "Send a WhatsApp to Mum saying I'll be late"
- "Give me my morning briefing"
| Bot State | RoboEyes Expression | Description |
|---|---|---|
| Disconnected | Tired | Droopy eyelids, slow blinks |
| Idle | Default + auto-blink | Neutral, gaze wanders gently |
| Waking | Happy + Curious | Wide excited eyes, one enlarges looking sideways |
| Listening | Default + Curious + Idle | Attentive scanning gaze |
| Thinking | Default + Confused (looping) | Eyes shake side to side |
| Speaking | Happy + Laugh (looping) | Eyes bounce up and down |
| Error | Angry + H-Flicker | Furrowed brows, subtle tremor |
Eye animations run at 50 fps, driven by FluxGarage RoboEyes — pure vector maths, no bitmaps.
| Symptom | Fix |
|---|---|
| ESP32 shows sleepy eyes, never connects | Check WS_HOST IP in config.h, ensure server is running |
Could not open COM8 when flashing |
Close PlatformIO serial monitor / Arduino IDE |
| TTS returns 403 error | Upgrade edge-tts: pip install edge-tts --upgrade |
| Wake word never triggers | Lower threshold in config.py: vad_threshold = 0.35 |
| No audio playback | Check MAX98357A wiring; ensure ffmpeg is on PATH |
ANTHROPIC_API_KEY error |
Add your key to .env |
| Google Calendar / Gmail not working | Run OAuth flow: place credentials.json in server/ |
All server settings live in server/config.py and are overridable via .env:
| Setting | Default | Description |
|---|---|---|
claude_model |
claude-haiku-4-5 |
LLM model |
claude_max_tokens |
512 |
Max reply length |
claude_history_turns |
10 |
Conversation memory turns |
whisper_model |
base |
STT model size (tiny/base/small) |
vad_threshold |
0.5 |
Silence detection sensitivity |
silence_duration |
0.8 |
Seconds of silence before processing |
edge_tts_voice |
en-US-AriaNeural |
TTS voice |
wake_word |
hey_jarvis |
Wake word model name |
server_port |
8000 |
WebSocket server port |