Real-time inference @ Meta Superintelligence Labs · San Francisco
Real-time multimodal inference: CUDA graphs, batching and KV cache under memory pressure, and the RL and robotics stacks that run on them

I work on real-time inference at Meta Superintelligence Labs, which I joined through the WaveForms AI acquisition. I built the real-time inference stack behind full-duplex voice in Muse Realtime Voice (Bringing Your Muse to Life).

Before that:

• WaveForms AI: distributed training infrastructure for 1,000+ GPUs across data centers.
• Kumo (acquired by NVIDIA): platform, cloud and security for Graph ML, on-prem and SaaS.
• Founding teams at Grainite (acquired by MongoDB) and Lacework: infrastructure, security and compliance.

In open source I work on inference engines end to end: CUDA-graph capture and piecewise replay, continuous and dynamic batching, paged KV-cache admission and eviction under memory pressure, zero-copy tensor transport between workers over shared-memory arenas and Mooncake RDMA, tensor-parallel lockstep execution, and control-plane hardening.

Core contributor to M*.

• Performance: 2.3x faster streaming Wan VAE decode (#425) and 1.74x on Wan 2.2 with the same decoder (#437); Qwen3-Omni streams audio after 12 frames instead of 25 (#443); faster first audio for VoxCPM2 and Qwen3-TTS through CUDA-graph capture (#659, #661); no per-step host sync in the denoise loops of Cosmos3, Wan 2.2 and OmniVoice (#453).
• Batching and KV cache under memory pressure: a refused admit unwinds exactly what it reserved (#561), the engine recovers without stranding requests (#560), and a refused batch retries smaller when nothing can be evicted (#651). A full /dev/shm, a late message or one failed request no longer takes down the server or the batch (#546, #557, #559).
• Voice: Fish Audio S2-Pro (#492) and Fun-CosyVoice3 (#495) voice cloning, Meta Omnilingual ASR in 1,600+ languages (#515), server and semantic turn detection for /v1/realtime (#669, #672), AudioSeal watermarking (#469).
• World models: LingBot-World-V2 camera-controlled image-to-world (#455) and Cosmos3-Edge streamed-rollout fixes (#622).
• Security: removed pickle from the control-plane wire, closing an unauthenticated RCE (#407); optional API key on every route (#626); models load without running checkpoint repo code (#654); about 20 more hardening and denial-of-service fixes (request size and media limits, file permissions, upload cleanup, CVE floors).
• Observability and evals: Prometheus metrics across the API server, conductor, workers and transport with a Grafana dashboard (#736 to #755); an eval framework (#502) with a Full-Duplex-Bench suite (#520); a base vs PR regression matrix across GPUs, transports, stress and perceptual quality (#571).

• LeRobot: pi0.5 denoise: read-only prefix cache and opt-in CUDA graph, 154 to 56 ms per chunk on one H200 at the same LIBERO success rate (#5148, #5152, #5153)
• TorchRL: GSPO loss for LLM post-training (#4653); LeRobot policy loading fix (#4651)
• RLinf: Robotics test reliability (#1750); ALOHA env and LeRobot ACT eval (#1786); eval against running action servers (#1751)
• M*: Pause, drain and hot weight reload for RL rollouts (#845)
• vLLM-Omni: Copy-on-write KV session fork for AR-diffusion (#4909); action-conditioned robot world model (#4904)
• vLLM: Priority scheduling: preempt for higher-priority requests at max_num_seqs (#45561)
• Mooncake: Check every task in transfer_check_status (#4913)