GemmaLung is a web-based, AI-powered lung CT analysis platform designed for radiologists and clinicians. Think of it as an intelligent, conversational ITK-SNAP — combining interactive 3D visualization with a MedGemma-driven AI assistant that can retrieve, quantify, and interpret pulmonary findings through natural language.
GemmaLung provides a browser-based 3D WebGL viewer that renders multi-structure segmentation masks (lung, cyst, nodule, airway, vessel, pleural effusion) as interactive 3D meshes. Clinicians can:
- Select rendering combos — choose which structures to visualize (e.g., "Lung + Cyst", "Overview (All)")
- Rotate, zoom, and pan — full 3D manipulation in the browser via mouse/trackpad
- Click-to-locate — double-click any point on the 3D mesh to instantly generate linked tri-planar slices (axial, coronal, sagittal) at that exact anatomical location
- Rapidly assess spatial distribution — understand 3D lesion spread without scrolling through hundreds of 2D slices
This human-centered design directly addresses how radiologists work: instead of mentally reconstructing 3D anatomy from sequential 2D slices, they can see the full picture first, then drill into specific regions of interest.
GemmaLung integrates Google MedGemma 1.5 4B as an intelligent conversational agent with 5 composable tools. The agent autonomously decides which tools to invoke, chains multi-step reasoning, and produces structured clinical analysis — all grounded in actual CT evidence.
The AI assistant supports 7 validated clinical scenarios:
| # | Scenario | What It Does |
|---|---|---|
| 1 | Targeted Slice Analysis | Retrieve a specific CT slice with overlay, analyze visible lesions |
| 2 | Patient Overview | Quantitative summary of all segmented structures with clinical impression |
| 3 | Multi-view Comprehensive Analysis | Auto-find max-burden slices, retrieve 3-plane overlay images, synthesize findings |
| 4 | Risk Assessment | Identify most concerning findings, provide follow-up recommendations |
| 5 | Left-Right Laterality | Quantitative comparison of lesion burden between left and right lungs |
| 6 | Individual Lesion Details | Top-K largest lesions with volume, centroid, and bounding box |
| 7 | Multi-Target Overlay | Simultaneously visualize multiple lesion types on a single slice |
┌──────────────────────────────────────────────────────────────┐
│ Streamlit Frontend │
│ ┌────────────────────┐ ┌────────────────────────────────┐ │
│ │ Case_Viewer │ │ Diagnosis Chat │ │
│ │ - 3D WebGL Viewer │ │ - Natural language input │ │
│ │ - Combo Selector │ │ - Quick question buttons │ │
│ │ - Key Slices │ │ - Overlay image display │ │
│ │ - Quant. Metrics │ │ - Markdown-rendered answers │ │
│ └────────────────────┘ └────────────────────────────────┘ │
└──────────────────────────────┬───────────────────────────────┘
│ REST API
┌──────────────────────────────▼───────────────────────────────┐
│ FastAPI Backend │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ AgentController (tool-calling loop, auto-chaining) │ │
│ ├──────────────────────────────────────────────────────┤ │
│ │ EvidenceToolService │ │
│ │ 5 tools: overview, slice, max_burden, instances, │ │
│ │ laterality │ │
│ ├──────────────────────────────────────────────────────┤ │
│ │ WebGL Mesh Service (mask → 3D mesh → browser) │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────────┬───────────────────────────────┘
│ OpenAI-compatible API
┌──────────────────────────────▼───────────────────────────────┐
│ vLLM Server (GPU) │
│ google/medgemma-1.5-4b-it (multimodal: text + images) │
└──────────────────────────────────────────────────────────────┘
- Python 3.10+
- NVIDIA GPU with >=40GB VRAM (A100 80GB recommended)
- Hugging Face access token with MedGemma access
pip install -r requirements.txtexport HF_TOKEN=your_huggingface_token
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \
--model google/medgemma-1.5-4b-it \
--dtype bfloat16 \
--max-model-len 8192 \
--port 8000 \
--trust-remote-codepython -m uvicorn backend.main:app --host 0.0.0.0 --port 8100streamlit run frontend/app.py --server.port 8501 --server.headless trueNavigate to http://localhost:8501, select a case from Case Setup, and enter Diagnosis Chat.
GemmaLung/
├── backend/
│ ├── agent/ # MedGemma agent runtime
│ │ ├── controller.py # Tool-calling loop + auto-chain + system prompt
│ │ ├── llm_client.py # OpenAI-compatible multimodal LLM client
│ │ ├── tool_registry.py # 5 tool definitions
│ │ └── tool_executor.py # Tool execution + trace collection
│ ├── api/v1/routes/ # REST API endpoints
│ ├── services/
│ │ ├── evidence_tool_service.py # Imaging tool implementations
│ │ ├── metrics_compute.py # Quantitative metrics engine
│ │ └── webgl_mesh_service.py # 3D mesh generation for WebGL
│ ├── core/config.py
│ └── schemas/
├── frontend/
│ ├── app.py # Streamlit entry point
│ ├── pages/
│ │ └── diagnosis_chat.py # Main diagnosis interface
│ └── ui_components/
│ └── webgl_viewer.py # 3D WebGL viewer component
├── preprocess_pipeline/ # CT segmentation pipeline
├── data/cases/ # Case data (gitignored)
├── docs/ # Writeup and video guide
├── requirements.txt
├── LICENSE
└── README.md
- This public repository should contain code + docs only. Do not publish patient-level data, derived medical images, or runtime logs.
- Run the preflight check before publishing:
bash scripts/oss_preflight_check.sh- Detailed release checklist (Chinese):
docs/OPEN_SOURCE_RELEASE_GUIDE_ZH.md
5 generic, composable tools that MedGemma autonomously orchestrates:
| Tool | Purpose |
|---|---|
get_case_overview |
Patient metadata + quantitative metrics for all structures |
get_slice |
Extract CT slice — raw or with color-coded segmentation overlay |
get_max_burden_slices |
Find the most-affected slice index for each anatomical plane |
get_lesion_instances |
Connected-component analysis — individual lesion measurements |
get_laterality |
Left vs. right lung quantitative comparison |
GemmaLung leverages MedGemma's unique strengths over general-purpose models:
- Multimodal medical reasoning — MedGemma directly views CT overlay images and provides image-grounded analysis, not hallucinated descriptions
- Agentic tool orchestration — autonomously selects which tools to call and chains multi-step workflows, adapting to each clinical question
- Medical domain knowledge — pre-trained on medical literature and imaging data, enabling clinically relevant observations about lesion morphology, distribution patterns, and differential considerations
- Efficient deployment — 4B parameters runs on a single GPU, suitable for on-premise clinical environments where data privacy matters
@misc{gemmalung2026,
title={GemmaLung: AI-Powered Interactive Lung CT Analysis Platform},
year={2026},
url={https://github.com/JQW2006/GemmaLung}
}Apache License 2.0 — see LICENSE for details.