Release: 3.7.6 Status: Active Prototype
AGOUTIC is a general-purpose agent for analyzing and interpreting long-read genomic data (Nanopore/PacBio). It uses a Dual Interface architecture (REST + MCP) to allow both human users and AI agents to orchestrate complex bioinformatics pipelines.
The system is composed of:
- Cortex: Agent Engine - AI-powered orchestration and user interaction
- Atlas: Consortium Data Integration - Registry-driven ENCODE and IGVF retrieval via MCP
- Launchpad: Execution Engine - Dogme/Nextflow pipeline management (local + remote SLURM)
- Analyzer: Analysis Engine - Results analysis and QC reporting
- edgePython: Differential Expression β Bulk/single-cell RNA-seq DE via edgePython
- XgenePy MCP: Cis/trans regulatory modeling β local XgenePy execution with canonical artifacts
- UI: Web interface for monitoring and control
Current status: database infrastructure centralized in common/database.py
with Alembic migrations. Gene annotation and enrichment tools moved from
edgePython to Analyzer. The analyzer/server4 adapter layer proxies remaining
edgePython MCP calls upstream. Atlas now exposes both ENCODE and IGVF
consortium MCP servers through the same schema-aware routing and formatting
layer. Shared-project collaboration foundations now use the existing
owner/editor/viewer project RBAC consistently for conversation history,
project-scoped file routing, analyzer/job access, collaborator management APIs,
and the Streamlit sharing UI.
AGOUTIC projects now support multi-user collaboration with project-scoped viewer, editor, and owner roles.
- Owners can add collaborators directly from chat or from the Projects page, and can transfer ownership to an existing viewer or editor collaborator.
- Editors can work in shared projects and keep using normal mutating project actions.
- Viewers can open shared conversation history, inspect project files and jobs, and browse project state in a read-only mode.
- Admins can manage collaborators for any project.
The current collaboration UI surfaces are:
- Chat: natural-language requests such as
share this project with alice@example.comopen a dedicated sharing flow, andlist users/list collaboratorsdisplay the current project roster with role and activity labels. - Sidebar and chat shell: owned vs shared projects are labeled explicitly,
viewer-only sessions have chat submission, uploads, and clear-chat disabled, and
a
list userssidebar button exposes the current collaborator roster. Shared owner/admin chat views also keep a pinned collaborator summary available while the conversation scrolls. - Projects page: the project table shows access labels, owner/admin-only mutating controls stay gated, and the Collaborators tab supports add, update, remove, and ownership-transfer actions. Collaborator rosters are grouped by editors and viewers to make role-based access easier to scan.
Collaboration is project-scoped rather than conversation-scoped. Authorized members can reopen shared project conversations and continue follow-up work in the same project when their role allows it.
AGOUTIC supports dual execution modes:
- Local: Runs Nextflow/Dogme pipelines directly on the local machine (default, original behavior)
- Remote SLURM: Submits jobs to a remote SLURM cluster via SSH
Remote execution features:
- Saved SSH profiles β per-user connection profiles with secure key references (no raw secrets stored). Supports local OS user key access through a per-session broker launched under that Unix account with
su(password used transiently, never stored) - Separate rsync transfer host β optionally stage inputs and download outputs through a dedicated host while SLURM submission, polling, and remote commands continue through the configured SSH host; blank transfer-host settings fall back automatically
- SLURM resource management β configurable account, partition, CPUs, memory, walltime, GPUs with validation
- Shared OpenChromatin GPU runtime defaults β DNA SLURM runs now default to the shared Dogme OpenChromatin GPU container and task-scoped runtime wiring instead of the older custom host-mounted modkit path
- Remote base path model β a single
remote_base_pathanchorsref/,data/, and per-workflow remote directories - Remote browsing and stage-only intake β browse saved-cluster paths and stage references/input data without submitting a job
- Completed-workflow import β adopt an already-run local or remote Dogme workflow into the current project as the next
workflowN, infer metadata from the workflow.config, and keep provenance for later explicit resync - Stage transfer controls β running stage-only transfers can be refreshed, cancelled, resumed, and failed staging cards can delete their reserved local workflow folders directly from the UI
- Result destination policy β keep results remote-only, copy back locally, or both
- Staged approval prompts β Cortex collects details progressively, presents summary before submission
- Run and staging status tracking β dedicated staging tasks plus remote execution stage labels through
completed, including byte-level transfer progress, current-file details, and faster live refresh while transfers are active - Scheduler integration β SLURM job ID tracking, state polling via sacct/squeue, cancellation via scancel
AGOUTIC now supports the wf_pore_c workflow family end to end behind WF_PORE_C_ENABLED. Phase 3 is feature-complete across local preview, local execution, remote SLURM staging/submission, Analyzer recognition, automatic summary generation, UI approval/run cards, and backward-compatible conversation/job reconstruction. The flag still defaults to off in both Launchpad and Cortex configs, so existing Dogme behavior remains the default until operators explicitly enable wf-pore-c.
Current validation status:
- Full closure sweeps passed in both flag states:
1988 passedwithWF_PORE_C_ENABLED=falseand1988 passedwithWF_PORE_C_ENABLED=trueacross Cortex, Launchpad, Analyzer, and UI. - The identical pass count in both states confirms the flag changes runtime behavior, not test inclusion.
- Production UI rendering no longer carries the earlier AST-harness fallback branch; missing wf-pore-c helper wiring now fails loudly instead of silently degrading to Dogme metadata.
- Legacy chats and jobs that only stored
modestill normalize toworkflow_key="dogme"on both the slow history-rebuild path and the fast cached-state restore path.
Manual cluster validation is still required before operational rollout. Use these docs for the wf-pore-c validation path:
docs/wf_pore_c_plan.mdβ phased implementation record and closure notesdocs/wf_pore_c_smoke_test.mdβ local Phase 2 smoke pathdocs/wf_pore_c_remote_smoke_test.mdβ real-cluster Phase 3 SLURM smoke path
In any existing project, you can import a workflow that Dogme already ran earlier:
- Slash command:
/import-workflow /scratch/youruser/agoutic/project-alpha/workflow12 --remote - Natural language:
import remote workflow from /scratch/youruser/agoutic/project-alpha/workflow12
Optional flags:
--full-copyto copy the entire workflow directory instead of the normal result subset--sample-name,--mode,--reference, and--modificationsif you want to override metadata inferred from the workflow.config
Imported workflows are assigned the next workflowN folder in the current project. If the source workflow is still incomplete, AGOUTIC keeps a warning on the execution card and you can continue the copy later with /sync-workflow workflow12.
Phase 1 limitation: Analyzer operates on local-accessible files only. Remote results must be copied back before downstream analysis.
See docs/remote_execution_architecture.md for architecture details, docs/cluster_slurm_setup.md for setup, docs/user_guide_execution_modes.md for usage, and TUTORIAL.md for an end-to-end user walkthrough.
AGOUTIC combines computational workflow execution with agent-guided biological interpretation. After a pipeline finishes, the platform helps users move from raw output folders to scientific insight, including isoform behavior, modification patterns, pathway shifts, and gene-level functional context.
AGOUTIC is designed to help users:
- Interpret isoform discovery and transcript structure outputs from long-read workflows
- Summarize RNA and DNA modification signals from workflow artifacts
- Review QC metrics across runs and identify quality or completeness issues
- Read text, markdown, and HTML workflow reports directly from chat or with
/read-file - Inspect gene-level and transcript-level result tables with context
- Run downstream differential expression and enrichment analysis
- Compare outputs across samples, conditions, and workflows
-
Quality Control (QC) analysis
- Parse run summaries, count/stat tables, alignment summaries, and basecalling outputs
- Validate run completeness and surface troubleshooting context from logs and artifacts
- Generate QC summaries for quick review across workflow outputs
-
Transcriptomic analysis
- Explore gene- and transcript-level quantification outputs
- Support isoform-aware interpretation from long-read RNA/cDNA pipelines
- Review splice-aware and transcript-structure-relevant outputs
-
Epitranscriptomic and epigenomic analysis
- Summarize RNA modification outputs from direct RNA workflows
- Summarize DNA modification outputs from DNA workflows
- Parse and interpret
bedMethyloutputs, including region- and gene-linked review where applicable
-
Differential analysis
- Run bulk and single-cell RNA-seq differential expression through edgePython
- Compare grouped samples directly from reconciled workflow abundance tables or saved dataframes
- Use the stateful DE flow: load β filter β normalize β design β fit β test β results
- Prefer workflow-local
reconciled.gtfannotation when present so transcript-aware plots and summaries stay aligned with the active workflow output - Filter by FDR/logFC and report annotated top genes for interpretation
-
Functional interpretation
- Run GO enrichment (BP, MF, CC)
- Run Reactome and KEGG pathway enrichment
- Translate Ensembl IDs and normalize symbols for human and mouse datasets
- Build and reuse colocated GTF-backed gene/transcript caches for workflow-local or custom annotations
For deeper tool-level details, see analyzer/README.md,
skills/differential_expression/SKILL.md,
skills/enrichment_analysis/SKILL.md, and
SKILLS.md.
Pipeline execution β Result discovery β QC parsing and summary generation β Expression / isoform / modification result extraction β Optional differential expression analysis β Functional enrichment β Agent-guided visualization and biological interpretation
Once parsing and summaries are complete, AGOUTIC can pivot into follow-up interpretation tasks such as cross-workflow comparison, condition-focused differential analysis, and targeted functional hypotheses (for example, pathway-level shifts or biologically coherent gene programs).
The analysis layer consumes:
- Pipeline result folders (for example
workflow1/,workflow2/) - CSV/TSV/BED/bedMethyl files
- Counts and summary/statistics tables
- Annotation and quantification outputs
- User-selected files from workflow subdirectories
The analysis layer returns:
- Parsed result tables
- QC summaries and run-validation context
- Annotated gene/transcript lists
- Differential expression result tables
- GO/pathway enrichment tables
- Interactive plots and chart-ready summaries
- Chat-readable scientific interpretation for downstream decisions
Summarize the QC for workflow2List the important files in workflow1/annotParse the bedMethyl output and summarize methylation patternsHaplotype RNA workflow7 with file /data/parents.vcf.gzHaplotype mouse sample B6 Cast F1 workflow7/haplotype DNA workflow5 /data/family.vcf.gz/haplotype RNA workflow7 --vcf-sample B6,CASTShow the top expressed genes from this result fileRun differential expression between control and treatmentCompare the treated samples treated_1 and treated_2 to the control samples ctrl_1 and ctrl_2Compare treated_1 and treated_2 to ctrl_1 and ctrl_2 from DF1 at transcript level/de treated=treated_1,treated_2 vs control=ctrl_1,ctrl_2Annotate these Ensembl IDsRun GO enrichment on the upregulated genesCompare workflow1 and workflow2 outputs
AGOUTIC can haplotype long-read DNA, RNA, or cDNA workflow BAMs against a VCF through the haplotype_with_vcf skill.
- Slash command:
/haplotype RNA workflow7 /data/parents.vcf.gz - Natural language:
haplotype RNA workflow7 with file /data/parents.vcf.gz - Mouse founder-mode natural language:
haplotype mouse sample B6 Cast F1 workflow7 - Mouse founder-mode slash command:
/haplotype RNA workflow7 --vcf-sample B6,CAST - Cross-project workflow reference:
haplotype B6CASTF1 RNA mouse sample otherproject:workflow7 - DNA workflows resolve mapped BAMs from
bams/, RNA/cDNA workflows resolve annotated BAMs fromannot/, and reconcile workflows use root-level*.annotated.bamfiles. - Cross-project haplotype requests can target another project's workflow with
project_name:workflowNwhile still writing outputs into a new workflow under the active project unless you explicitly override the destination. - Plain
.vcfinputs are compressed to.vcf.gzand indexed automatically when AGOUTIC can write beside the source file;.vcf.gzinputs missing.tbior.csiare auto-indexed. - Mouse/mm39 founder-panel requests can omit the VCF. In that case AGOUTIC resolves
mgp_REL2021_snps_founders.vcf.gzfrom the same directory as the configured mm39 reference FASTA. - Mouse founder aliases are case-insensitive and ignore
/,_,-, and spaces.ref,B6,C57BL6, andC57BL6/Jall resolve toC57BL_6J;CAST,CAST/J, andCAST_EiJresolve toCAST_EiJ. - Founder-pair restrictions can be written as repeated
--vcf-sampleflags, comma-separated--vcf-sample B6,CAST, F1 shorthand such asB6CastF1orB6 Cast F1, or natural language such ashaplotype mouse between B6 and CAST workflow7. - The approval gate lists the exact BAM names, the selected VCF sample or founder subset, the resolved VCF path, assignment labels, and the destination
workflowNbefore execution starts.
- Inline Plotly visualizations directly in chat
- Interactive bar, scatter, heatmap, box, histogram, pie, venn, and upset plots from conversation dataframes
- Automatic plotting from parsed tables when chartable data are detected
- Cross-workflow open-chromatin overlap requests can run as approval-gated background workflows that write workflow-scoped overlap CSVs and manifests for downstream plotting
- Saved venn/upset overlap plots over earlier chat dataframes reload the full source table from analyzer provenance or matching project files when only preview rows are available, so overlap counts stay accurate and two-set venn diagrams remain readable for large overlaps
- edgePython-backed DE and enrichment plots now default to 600 dpi raster export with an SVG companion for project-scoped artifact output
- Publication-style volcano and MD plots route through edgePython, while generic dataframe charts remain an interactive Plotly path rather than a server-side publication export path
- Analyzer currently requires local-accessible files
- Remote-only results must be copied back before downstream analysis
- Some analysis pathways are file-format dependent and assume expected output conventions
- Cross-run comparison is strongest when workflows use consistent references and naming
- Interpretation depth depends on pipeline completeness and annotation availability
See docs/remote_execution_architecture.md
for remote execution constraints and staging/copy-back behavior.
AGOUTIC enforces access control at every layer:
- Authentication: Google OAuth 2.0 with session cookies (
httponly,samesite=lax,securein production) - Authorization: Role-based access (owner / editor / viewer) checked on every endpoint via
require_project_access(). Admins bypass all project-level checks; public projects allow viewer access. - Project collaboration: Shared projects expose conversation history and other project-scoped surfaces to members according to project role. Viewer access is read-only; editor access is required for project mutations such as linking jobs to conversations or writing project files.
- Run-level access: Each job still records the submitting
user_id, butrequire_run_uuid_access()now authorizes via the job'sproject_idfirst so shared-project collaborators can inspect project jobs; directuser_idownership remains a fallback for legacy rows. - File isolation: Project files resolve through the canonical shared project directory, while downloaded/uploaded source files remain in the acting user's central data folder (
AGOUTIC_DATA/users/{username}/data/). Projectdata/entries are symlinks into that central store, preserving per-user private storage while letting collaborators work in one shared project tree. Legacy{user_id}/{project_id}paths are still supported for backward compatibility. - Server-side project IDs: UUIDs generated server-side via
uuid4()β clients never control the ID. - Project management: Full dashboard for browsing projects, viewing stats/files/jobs, renaming, archiving, and permanent deletion with cascading cleanup.
- Bootstrap & admin scripts: Run
python scripts/cortex/init_db.pyfor a fresh database bootstrap,python scripts/cortex/set_usernames.py autoto derive usernames from email addresses on an existing instance, andpython scripts/cortex/bootstrap_project_tasks.pyto seed persistent project tasks from existing workflow history.
# Create environment
conda env create -f environment.yml
conda activate agoutic_core
# Or update an existing environment in place
conda env update -n agoutic_core -f environment.yml --pruneThe environment now includes htslib (bgzip, tabix) and bcftools so AGOUTIC can auto-compress and index VCF inputs for haplotyping and operators still have standard VCF command-line tooling available. bcftools is resolved from bioconda, so one-off manual installs should use conda-forge plus bioconda with strict channel priority.
AGOUTIC now supports two UI topologies:
- Hosted Streamlit: Streamlit and Cortex run on the same server or network boundary and the browser uses the existing cookie-based login flow.
- Local Streamlit against remote Cortex: Streamlit runs on the user's machine,
AGOUTIC_API_URLpoints at the remote Cortex server, and login returns to the local Streamlit page so it can exchange a bearer session.
In both modes, the UI talks only to Cortex. End-user UI environments do not need LAUNCHPAD_REST_URL or INTERNAL_API_SECRET.
# Recommended: start the full backend stack
./agoutic_servers.sh --start
# Then start the UI separately
streamlit run ui/appUI.py --server.address 0.0.0.0 --server.port 8501For local development, you can still run services manually:
# Terminal 1: Start Launchpad REST
uvicorn launchpad.app:app --host 0.0.0.0 --port 8003 --reload
# Terminal 2: Start Launchpad MCP
python -m launchpad.mcp_server --host 0.0.0.0 --port 8002
# Terminal 3: Start Cortex
uvicorn cortex.app:app --host 0.0.0.0 --port 8000 --reload
# Terminal 4: Start UI
cd ui && streamlit run appUI.pyFor a local UI talking to a remote Cortex server:
export AGOUTIC_API_URL=http://remote-cortex-host:8000
streamlit run ui/appUI.py --server.port 8501When the UI is running on localhost or 127.0.0.1, the login button automatically requests the local-client auth flow. Cortex redirects back to the local Streamlit URL with a short-lived auth code, and the UI exchanges that code for a bearer session.
Note: running python ui/appUI.py directly will not work correctly because the UI
auth flow depends on Streamlit request context plus either browser cookies (hosted mode) or the local auth-code exchange (local mode).
# Check Cortex health
curl http://localhost:8000/health
# Check Launchpad health
curl http://localhost:8003/health
# Test Atlas connection
python cortex/atlas_mcp_client.py
# Expected: Connection success and K562 search resultsAGOUTIC now maintains a persistent project task list instead of relying on a hard-coded checklist in the UI.
- Tasks are projected from durable workflow records, mainly
ProjectBlockstate plus job progress payloads. - The chat page groups tasks into pending, running, follow-up, and completed sections.
- Parent tasks can include child tasks for workflow stages, per-file download progress, analysis completion, and result review.
- Existing history can be backfilled safely with:
python scripts/cortex/bootstrap_project_tasks.py
# Optional: seed only one project
python scripts/cortex/bootstrap_project_tasks.py --project-id <project_id>ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β AGOUTIC System v3.7.2 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β ββββββββββββ β
β β Web UI β (Streamlit) β
β ββββββ¬ββββββ β
β β REST API β
β β β
β ββββββββββββββββββββββββββββββββββββββββββ β
β β Cortex (Agent Engine) β β
β β AI Orchestration + Coordination β β
β ββββββ¬βββββββββββββ¬βββββββββββββ¬ββββββββββ β
β β β β β
β β MCP β REST β MCP β
β β β β β
β ββββββββββ ββββββββββββ ββββββββββββ β
β βAtlasβ β Launchpad β β Analyzer β β
β βENCODE β β Nextflow β β Analysis β β
β β Portal β β Pipeline β β Engine β β
β ββββββ¬ββββ βββββββ¬βββββ βββββββ¬βββββ β
β β β β β
β β β β β
β ENCODE Dogme Results β
β Portal Pipelines Files β
β β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
- Role: Central orchestrator with LLM reasoning
- Tech: FastAPI + OpenAI-compatible LLM
- Features:
- Chat interface with skill-based workflows
- Coordinates Atlas, Launchpad, and Analyzer
- Block-based project timeline
- Persistent project task list with child-task projection for downloads, workflow stages, analysis, and follow-up review
- Background job monitoring with stop/cancel buttons
- Download cancel button β "π Cancel Download" on running downloads with partial-file cleanup
- "List my data" command β chat-based central data folder listing (DB + disk fallback)
- Post-cancel workflow management β Delete / Resubmit buttons on cancelled jobs; chat-based deletion via natural language
- User authentication
- Role-based authorization gates on all endpoints
- Server-side project CRUD (
POST/GET/PATCH /projects) [[PLOT:...]]tag parsing βAGENT_PLOTblocks for inline Plotly charts (histogram, scatter, line, area, bar, box, violin, strip, heatmap, pie, venn, upset)- Per-message and per-conversation token tracking β every LLM response records
prompt_tokens,completion_tokens,total_tokens, andmodel_namein the database; exposed viaGET /user/token-usage(own data) andGET /admin/token-usage(all users) find_fileecho recovery β when a weak model emits afind_fileJSON result verbatim instead of a[[DATA_CALL:...]]tag, the pipeline intercepts the response, auto-chains toparse_csv_file/parse_bed_file/read_file_content, and strips the bad block from conversation history to prevent looping- ENCODE tool routing guards β structural checks prevent LLM misrouting (e.g. cell-line names sent to
get_experiment); assay-only queries are routed tosearch_by_assay; assay name aliases (e.g.RNA-seqβtotal RNA-seq,ChIP-seqβTF ChIP-seq) are resolved before MCP dispatch - IGVF dispatch guards β malformed IGVF calls now backfill missing required
sample_term/assay_titlevalues from the user message when possible, and schema validation drops still-invalid calls before they hit MCP - Tool Schema Contracts β machine-readable JSON Schema for every MCP tool, fetched at startup from
/tools/schemaendpoints on all servers. Injected into the system prompt as a compact reference and used for pre-call param validation (strip unknown params, check required fields, normalise enums). - Structured Conversation State β typed
ConversationStateJSON (skill, project, sample, experiment, dataframes, workflows) built each turn and injected as[STATE]...[/STATE]so the LLM always sees current context - Error-Handling Playbook β deterministic failure rules in the system prompt + structured
[TOOL_ERROR]blocks + single-retry for transient failures - Output Contract Validator β post-LLM validation catches malformed
DATA_CALLtags, duplicateAPPROVAL_NEEDED, unknown tools, and mixed sources - Provenance Tags β
[TOOL_RESULT: source, tool, params, rows, timestamp]headers on every tool result for auditability; persisted in AGENT_PLAN blocks - Plan-Execute-Observe-Replan β structured multi-step planning layer that now runs through manifest-first classification and composition for core deterministic flows.
SkillManifestmetadata supplies planner triggers, expected inputs, required services, runtime hints, and MCP tool chains;plan_composer.pybuilds DE, enrichment, and XgenePy plans from that metadata; legacy templates remain as deterministic fallback for unmigrated flows; and CHECK_EXISTING guards still skip expensive operations when results already exist. - Deterministic skill-management commands β
/skills,/skill <skill_key>, and/use-skill <skill_key>expose the live skill catalog, describe individual skills, and persist manual skill switching across turns without relying on a freeform model response. - Gene Annotation & ID Translation β offline Ensembl gene ID β symbol translation (human + mouse) via pre-built lookup tables. Auto-annotates gene symbols when DE data is loaded; all downstream outputs (top genes, heatmaps, summaries) automatically use readable symbols instead of raw Ensembl IDs. Bidirectional
lookup_genetool answers "what is the Ensembl ID for TP53?" style queries. Pre-LLM auto-skill detection routes gene questions to the correct skill from any context (including Welcome). MCP tools:annotate_genes(edgePython, DE-stateful),translate_gene_idsandlookup_gene(Analyzer). - Robust DATA_CALL tag parsing β bracket-aware parameter parser handles JSON arrays inside DATA_CALL tags (e.g.
gene_symbols=["TP53", "BRCA1"]). Mistral-native[TOOL_CALLS]DATA_CALL:format is auto-normalized to standard[[DATA_CALL:...]]tags. - Skill-defined plan chains β skill authors can declare multi-step workflows in skill Markdown files under a
## Plan Chainssection. A single message like "get K562 experiments and make a plot by assay type" is detected at classify-time and produces both a data search and a visualization. Trigger phrases support multi-phrasing (AND/OR keyword groups) for flexible matching. SeeSKILLS.mdfor the full authoring guide. - Skills system documentation β new top-level
SKILLS.mddocuments the complete skills framework: skill file structure, routing patterns,[[DATA_CALL:...]]/[[PLOT:...]]tag system, plan chains format, and a step-by-step guide for creating new skills. - Inline Plotly visualizations with deduplication β
[[PLOT:...]]tags produce interactive bar, scatter, line, area, violin, strip, pie, histogram, heatmap, venn, and upset charts rendered directly in chat. A three-layer pipeline guarantees chart generation: chain context injection β second-pass PLOT tag instructions β post-DataFrame fallback. Deduplication and prompt-intent selection prevent duplicate/overlapping traces and stale style leakage across turns. Supports explicit colors (color=green), grouped or stacked bar modes, and boolean-column overlap plots for presentation-friendly set comparisons. - DF inspection quick commands β
list dfslists all dataframes in the conversation with their metadata;head df1(orhead df3 5) shows the first N rows as a markdown table. Both bypass the LLM entirely β zero token cost. - In-memory dataframe actions β filter, subset, select columns, rename, sort, melt, aggregate, join, and pivot existing conversation dataframes without going through analyzer file calls. Saved transforms can also appear as block-specific
PENDING_ACTIONcontrols in the UI.
- Role: Registry-driven public consortium data retrieval
- Tech: fastmcp + Atlas registry/configuration with ENCODE and IGVF MCP servers under
atlas/ - Features:
- Search ENCODE experiments by biosample/organism/target
search_by_assayβ assay-first ENCODE search (e.g. "how many RNA-seq experiments") across both organisms, returning combined counts and per-organism lists- Search IGVF measurement sets, analysis sets, prediction sets, files, genes, and samples
- Download and file-metadata helpers for both consortium integrations
- Metadata caching, result formatting, and tool/parameter alias repair
- 30+ MCP tools across the current consortium integrations
- Agent routing guards in Cortex prevent structural misrouting and block invalid required-param calls before MCP execution
- Extension pattern:
atlas/mcp_server.pyextends ENCODELIB's FastMCP server, whileatlas/igvf_mcp_server.pyandatlas/launch_igvf.pyprovide the parallel IGVF HTTP MCP server path. - Tool schemas:
atlas/tool_schemas.pyandatlas/igvf_tool_schemas.pydefine JSON Schema contracts for the ENCODE and IGVF tools, served via/tools/schemaGET endpoints. - Docs: ATLAS_IMPLEMENTATION.md
- Role: Nextflow pipeline execution
- Tech: FastAPI + Nextflow + Dogme
- Features:
- Submit Dogme DNA/RNA/cDNA pipelines
- Shared OpenChromatin GPU container defaults for SLURM DNA runs with task-scoped runtime injection for OpenChromatin work
- Real-time job monitoring
- Log streaming
- User-jailed working directories
- Job cancellation β SIGTERM-based cancel with cooperative
.nextflow_cancelledmarker; properly displays CANCELLED (not FAILED) in UI - Workflow folder deletion β DELETE endpoint removes work directory and sets status to DELETED; block status updated immediately so UI reflects deletion
- Job resume β resubmit cancelled/failed jobs with Nextflow
-resumeflag to reuse cached task results in the same workflow directory instead of starting fresh - Stage-only transfer lifecycle controls β refresh, cancel, resume, and failed-stage cleanup actions are available in both the workflow UI and Task Center
- Live staging telemetry β running transfers surface current file, transferred bytes, total size, and faster refresh cadence during brokered or direct rsync activity
delete_job_dataMCP tool β enables chat-based deletion ("delete workflow1")
- Docs: launchpad/README.md
- Role: Bulk and single-cell RNA-seq differential expression analysis
- Tech: FastMCP + edgePython
- Features:
- Full DE pipeline: load β filter β normalize β design β dispersion β fit β test β results β plots
- Gene list filtering from DE results by FDR, logFC, and direction (up/down/all)
- Gene annotation (annotate_genes) on DE results in-place
- Workflow-local
reconciled.gtfpreference for annotation when available, with shared reference caches as fallback - Stateful pipeline β each step builds on previous results within a session
- Volcano, MDS, MA, BCV, heatmap plot generation
- TSV/CSV/JSON result export
- JSON Schema tool contracts via
/tools/schema
- Docs: edgepython_mcp/
- Role: Results analysis, QC reporting, gene annotation, and GO/pathway enrichment
- Tech: fastmcp + Python analysis tools + g:Profiler
- Features:
- Parse pipeline outputs (CSV, TSV, BED files)
- Generate QC reports and analysis summaries
- File discovery and content reading
- Workflow folder browsing via
list_job_files - Gene ID translation (
translate_gene_ids) and bidirectional lookup (lookup_gene) via Ensembl reference tables - GTF-backed gene and transcript annotation with colocated caches for shared references, workflow-local outputs, or custom user-provided GTFs
- GO enrichment (BP/MF/CC) and pathway enrichment (KEGG/Reactome) via g:Profiler
- Per-conversation enrichment state management
- Species auto-detection from gene ID prefixes (ENSG β human, ENSMUSG β mouse)
- Workflow Directory Layout:
$AGOUTIC_DATA/users/{username}/{project-slug}/ βββ data/ # Uploaded input data βββ workflow1/ # First job's output β βββ annot/ # Annotations, final stats, counts β βββ bams/ # BAM alignment files β βββ bedMethyl/ # Methylation BED output β βββ fastqs/ # FASTQ files β βββ ... βββ workflow2/ # Second job's output - Agent Commands (handled automatically by Cortex's safety net):
list my data/list my filesβ lists all files in your central data folderlist workflowsβ lists all workflow folders in the projectlist files/list files in workflow2/annotβ lists files in a workflow or subfolderparse annot/File.csvβ finds and parses a file by relative pathparse workflow2/annot/File.csvβ parses a file in a specific workflow
- Docs: analyzer/README.md
agoutic/
βββ README.md # This file
βββ environment.yml # Conda environment specification
βββ alembic.ini # Alembic migration configuration
βββ CONFIGURATION.md # Path configuration guide
βββ ATLAS_IMPLEMENTATION.md # Atlas integration guide
βββ ATLAS_QUICKSTART.md # Atlas quick reference
β
βββ cortex/ # Agent Engine
β βββ README.md # Cortex documentation
β βββ app.py # FastAPI application
β βββ agent_engine.py # AI agent orchestration
β βββ skill_manifest.py # Skill capability registry for routing + planning
β βββ plan_classifier.py # Manifest-first request classification
β βββ plan_composer.py # Manifest-driven deterministic plan builder
β βββ planner.py # Planner orchestration + fallback selection
β βββ plan_executor.py # Deterministic step execution engine
β βββ plan_replanner.py # Failure recovery + plan adjustment
β βββ dependencies.py # Auth gates (require_project_access, require_run_uuid_access)
β βββ user_jail.py # Path traversal guards & file isolation
β βββ auth.py # Google OAuth 2.0 + cookie hardening
β βββ models.py # Database models
β βββ schemas.py # Request/response schemas
β βββ config.py # Configuration
β βββ db.py # Database connection
β βββ prompt_templates/ # LLM system prompts (first-pass, planning, second-pass)
β βββ routes/ # Extracted REST route modules
β
βββ launchpad/ # Execution Engine
β βββ README.md # Launchpad documentation
β βββ app.py # FastAPI application
β βββ nextflow_executor.py # Nextflow wrapper
β βββ mcp_tools.py # MCP tool definitions
β βββ mcp_server.py # MCP server
β βββ models.py # Database models
β βββ schemas.py # Request/response schemas
β βββ config.py # Configuration
β βββ db.py # Database connection
β βββ quickstart.sh # Quick start setup
β βββ DUAL_INTERFACE.md # REST + MCP architecture
β βββ IMPLEMENTATION_SUMMARY.md # Implementation details
β
βββ scripts/ # Manual admin and operational utilities
β βββ cortex/
β β βββ init_db.py # Fresh database bootstrap utility
β β βββ set_usernames.py # Username/slug admin CLI
β β βββ bootstrap_project_tasks.py # Backfill persistent project tasks
β βββ launchpad/
β β βββ debug_job.py # Job inspection helper
β β βββ submit_real_job.py # Manual job submission helper
β βββ build_gene_reference.py # One-time Gencode GTF β TSV builder
β
βββ ui/ # Web Interface
β βββ README.md # UI documentation
β βββ app.py # Streamlit main app (chat, sidebar, auto-refresh)
β βββ pages/
β βββ projects.py # Projects dashboard (stats, files, bulk actions)
β βββ results.py # Job results analysis (auto-lists project jobs)
β βββ admin.py # Admin user management
β
βββ atlas/ # ENCODE MCP Extension
β βββ launch_encode.py # HTTP launcher (imports mcp_server for extensions)
β βββ mcp_server.py # Extends ENCODELIB FastMCP with search_by_assay + /tools/schema
β βββ tool_schemas.py # JSON Schema contracts for all 16 ENCODE tools
β βββ config.py # Atlas configuration
β βββ result_formatter.py # Result formatting helpers
β
βββ edgepython_mcp/ # edgePython DE Server
β βββ edgepython_server.py # FastMCP tool definitions (DE + filtering)
β βββ mcp_server.py # Server wrapper + /tools/schema endpoint
β βββ launch_edgepython.py # HTTP launcher
β βββ tool_schemas.py # JSON Schema contracts for DE tools
β βββ config.py # Configuration
β
βββ common/ # Shared Utilities
β βββ database.py # Centralized DB infrastructure (Base, engines, sessions)
β βββ gene_annotation.py # Ensembl gene ID β symbol translation (bidirectional)
β βββ mcp_client.py # Shared MCP HTTP client
β βββ logging_config.py # Structured logging setup
β βββ logging_middleware.py # Request logging middleware
β
βββ skills/ # Workflow Definitions
β βββ welcome/SKILL.md # New-user onboarding
β βββ ENCODE_Search/SKILL.md # ENCODE search skill + routing rules
β βββ ENCODE_LongRead/SKILL.md # ENCODE pipeline definition
β βββ run_dogme_dna/SKILL.md # DNA pipeline definition
β βββ run_dogme_rna/SKILL.md # RNA pipeline definition
β βββ run_dogme_cdna/SKILL.md # cDNA pipeline definition
β βββ analyze_local_sample/SKILL.md # Sample intake workflow
β βββ analyze_job_results/SKILL.md # Post-pipeline results analysis
β βββ download_files/SKILL.md # File download workflow
β βββ differential_expression/SKILL.md # edgePython DE skill
β βββ enrichment_analysis/SKILL.md # GO & pathway enrichment skill
β βββ remote_execution/SKILL.md # Remote SLURM workflow
β βββ shared/
β βββ SKILL_ROUTING_PATTERN.md # Shared skill routing reference
β βββ DOGME_QUICK_WORKFLOW_GUIDE.md # Shared workflow parsing guide
β
βββ data/ # Data & Database (created at runtime)
βββ database/
β βββ agoutic_v24.sqlite
βββ reference/ # Gene annotation reference files
β βββ human_genes.tsv
β βββ mouse_genes.tsv
βββ launchpad_work/ # Job execution directories
βββ launchpad_logs/ # Server logs
βββ users/ # Per-user jailed project dirs
AGOUTIC provides two complementary interfaces:
-
REST API - For web clients, dashboards, and scripting
- Traditional HTTP endpoints
- Easy integration with existing tools
- Language-agnostic clients
-
MCP Protocol - For LLM agents and AI orchestration
- Model Context Protocol (MCP)
- Tools exposed as structured capabilities
- Seamless AI agent integration
See launchpad/DUAL_INTERFACE.md for detailed architecture.
User Request
β
Cortex (Agent)
- Interprets intent
- Plans workflow
- (Optional) Requests approval
β
Launchpad (Executor)
- Receives job
- Generates Nextflow config
- Submits to cluster/local
- Monitors progress
β
Dogme Pipeline
- Basecalling
- Alignment
- Quantification
- Modification calling
β
Results & Reports
- Return to Agent
- Display in UI
DNA Mode
- Genomic DNA and Fiber-seq analysis
- Includes modification calling (5mC, 6mA, etc.)
- Full basecalling β alignment β quantification pipeline
RNA Mode
- Direct RNA-seq analysis
- Native RNA modification calling (m6A, pseU, etc.)
- Splice-aware alignment
cDNA Mode
- Polyubiquitin cDNA and isoform analysis
- No modification calling (faster processing)
- Transcript quantification focus
AGOUTIC uses two root path variables with sensible defaults:
# Where source code lives (auto-detected)
export AGOUTIC_CODE=/path/to/agoutic
# Where data/database/jobs live (defaults to $AGOUTIC_CODE/data)
export AGOUTIC_DATA=/path/to/storageNo required environment variables for default local setup. Defaults work automatically, but AGOUTIC_CODE and AGOUTIC_DATA can be overridden if needed. See CONFIGURATION.md for detailed configuration options.
AGOUTIC_CODE/
βββ cortex/ # Agent engine
βββ launchpad/ # Execution engine
βββ ui/ # Web interface
βββ skills/ # Workflow definitions
AGOUTIC_DATA/
βββ database/ # SQLite database
βββ launchpad_work/ # Job working directories
βββ launchpad_logs/ # Server logs
βββ logs/ # Structured logs (all servers)
βββ users/ # Per-user jailed project dirs
βββ {username}/ # e.g. eli/
βββ {project-slug}/ # e.g. k562-atac-seq/
βββ data/
βββ results/
βββ workflow1/
AGOUTIC uses structlog for unified structured logging across all servers. Every log entry is a JSON object written to both per-server and unified log files.
Logs are written to $AGOUTIC_DATA/logs/:
$AGOUTIC_DATA/logs/
βββ agoutic.jsonl # Unified log (all servers)
βββ cortex.jsonl # Cortex only
βββ launchpad-rest.jsonl # Launchpad REST API
βββ launchpad-mcp.jsonl # Launchpad MCP
βββ analyzer-rest.jsonl # Analyzer REST API
βββ analyzer-mcp.jsonl # Analyzer MCP
βββ encode-mcp.jsonl # ENCODE MCP server
βββ *.log # Raw stdout/stderr (safety net)
βββ *.YYYYMMDD_HHMMSS.* # Rotated previous logs
# Stream the unified log
tail -f $AGOUTIC_DATA/logs/agoutic.jsonl | jq .
# Filter by server
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.server == "cortex")'
# Filter by log level
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.level == "error")'
# Filter requests by path
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.path == "/chat")'
# Find slow requests (>1s)
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.duration_ms > 1000)'
# Trace a request across servers by request_id
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.request_id == "some-uuid")'Every HTTP request receives a unique X-Request-ID header. This ID is:
- Bound to all log entries emitted during the request
- Returned in the response
X-Request-IDheader - Available at
request.state.request_idin route handlers
When servers are started or restarted via agoutic_servers.sh, existing log files are automatically renamed with a timestamp (e.g., cortex.20260213_143052.jsonl). Empty log files are skipped.
| Variable | Default | Description |
|---|---|---|
AGOUTIC_LOG_FORMAT |
json |
Set to dev for coloured human-readable console output |
AGOUTIC_API_URL |
http://127.0.0.1:8000 |
Cortex base URL used by Streamlit. Set this when running the UI against a remote Cortex server. |
FRONTEND_URL |
http://localhost:8501 |
Hosted Streamlit origin used by Cortex for the browser-cookie login flow. |
GOOGLE_REDIRECT_URI |
http://localhost:8000/auth/callback |
OAuth callback URL exposed by Cortex. Must match the value registered in Google Cloud. |
LOCAL_UI_ALLOWED_ORIGINS |
unset | Optional comma-separated extra local UI origins allowed for auth-code return targets. Loopback origins (localhost, 127.0.0.1) work without this. |
# Recommended for a full local stack
./agoutic_servers.sh --start
# Start the UI separately
streamlit run ui/appUI.py --server.port 8501For a local Streamlit client against a remote Cortex deployment:
export AGOUTIC_API_URL=http://remote-cortex-host:8000
streamlit run ui/appUI.py --server.port 8501The Streamlit UI now proxies remote profile management through Cortex, so the local UI environment does not need direct Launchpad REST credentials.
If you need manual development startup:
# Terminal 1: Start Launchpad
cd /path/to/agoutic
uvicorn launchpad.app:app --port 8003 --reload
# Terminal 2: Start Launchpad MCP
python -m launchpad.mcp_server --host 0.0.0.0 --port 8002
# Terminal 3: Start Cortex
uvicorn cortex.app:app --port 8000 --reload
# Terminal 4: Start UI (if using Streamlit)
cd ui && streamlit run appUI.pyfrom cortex.mcp_client import LaunchpadMCPClient
# Connect to MCP server
client = LaunchpadMCPClient()
await client.connect()
# Submit a job
job = await client.submit_dogme_job(
project_id="proj_001",
sample_name="liver_dna",
mode="DNA",
input_directory="/data/pod5"
)
# Monitor progress
status = await client.check_nextflow_status(job["run_uuid"])# Submit job
curl -X POST http://localhost:8003/jobs/submit \
-H "Content-Type: application/json" \
-d '{
"project_id": "proj_001",
"sample_name": "liver_dna",
"mode": "DNA",
"input_directory": "/data/pod5"
}'
# Check status
curl http://localhost:8003/jobs/{run_uuid}/status
# Get results
curl http://localhost:8003/jobs/{run_uuid}- cortex/README.md - Agent Engine documentation
- launchpad/README.md - Execution Engine documentation
- ui/README.md - Web UI documentation
- CONFIGURATION.md - Full configuration guide
- QUICK_REFERENCE.md - Path configuration quick start
- docs/DATAFRAMES.md - Dataframe commands, transforms, plotting, and memory guide
- docs/DATAFRAMES.md - Dataframe commands, transforms, plotting, and memory guide
- launchpad/DUAL_INTERFACE.md - REST + MCP architecture
- launchpad/IMPLEMENTATION_SUMMARY.md - Implementation details
The project has 1068 tests providing comprehensive coverage.
# Run the full test suite (1068 tests)
pytest tests/ -q
# Cortex tests only
pytest tests/cortex/ -q
# With coverage report
pytest tests/cortex/ --cov=cortex/app --cov-report=term-missing
# Other components
pytest tests/atlas/ tests/common/ tests/analyzer/ tests/launchpad/ tests/ui/ -q
# Single test file
pytest tests/cortex/test_chat_data_calls.py -x -q
# Focused task lifecycle and hierarchy coverage
pytest tests/cortex/test_project_endpoints.py -q- In-memory SQLite with
StaticPoolfor fast, isolated tests - Mocked LLM via
AgentEnginepatches (no real model calls) - Mocked MCP via
MCPHttpClientpatches (no real service connections) - 37 cortex test files covering: chat endpoint, approval gates, background tasks, project management, block endpoints, conversations, auth, admin, downloads, uploads, pure helpers, tool routing, skill detection, validation, planning
- Task coverage includes persistent task projection, task actions,
download-file children, and workflow-stage children in
tests/cortex/test_project_endpoints.py - Shared fixtures in
tests/conftest.pyfor DB engine, sessions, mock users
If you deploy this release onto a server that already has projects and stored workflow blocks, run the task bootstrap once after the application starts:
python scripts/cortex/bootstrap_project_tasks.pyThe script is idempotent: it reconciles the current workflow state and is safe to re-run.
# Interactive demo for Launchpad
python launchpad/demo_launchpad.py- Check port availability:
lsof -i :8003(Launchpad REST),lsof -i :8002(Launchpad MCP), orlsof -i :8000(Cortex) - Check database connectivity:
python -c "from launchpad.db import SessionLocal; SessionLocal()" - Check Python version:
python --version(requires 3.12+)
- Check Nextflow process:
ps aux | grep nextflow - Check logs:
tail -f $AGOUTIC_DATA/launchpad_logs/*.log - Cancel job:
curl -X POST http://localhost:8003/jobs/{run_uuid}/cancel
- Verify configuration:
python -c "from launchpad.config import *; print(f'Code: {AGOUTIC_CODE}')" - Check paths:
ls -la $AGOUTIC_DATA/launchpad_work
Control the number of simultaneous GPU tasks (dorado basecalling, openChromatin) within a single pipeline run. Configurable in the approval form or via environment variable:
export DEFAULT_MAX_GPU_TASKS=8 # Optional explicit limit. Leave unset for no maximum. Range: 1-16If DEFAULT_MAX_GPU_TASKS is unset, Launchpad omits Nextflow maxForks for GPU-bound Dogme processes and lets Nextflow manage concurrency. Users can also override per-job in the approval form dropdown or via chat ("limit dorado to 3 concurrent tasks").
Limit concurrent jobs to avoid resource exhaustion:
export MAX_CONCURRENT_JOBS=2 # Adjust based on server capacity- Development: SQLite (default, no setup required β
create_all()at startup) - Production: PostgreSQL with Alembic migrations (
alembic upgrade head)
# Set via environment variable
DATABASE_URL = "postgresql://user:pass@localhost/agoutic"Pre-defined bioinformatics workflows are available in skills/:
- welcome/SKILL.md - New-user onboarding
- ENCODE_Search/SKILL.md - ENCODE search and data discovery
- ENCODE_LongRead/SKILL.md - ENCODE consortium workflow
- run_dogme_dna/SKILL.md - Genomic DNA analysis workflow
- run_dogme_rna/SKILL.md - Direct RNA-seq workflow
- run_dogme_cdna/SKILL.md - cDNA isoform workflow
- analyze_local_sample/SKILL.md - Sample intake and validation
- analyze_job_results/SKILL.md - Post-pipeline results analysis
- download_files/SKILL.md - File download orchestration
- differential_expression/SKILL.md - edgePython DE pipeline (with gene annotation)
- enrichment_analysis/SKILL.md - GO & pathway enrichment analysis
- reconcile_bams/SKILL.md - Cross-workflow annotated BAM reconciliation
- haplotype_with_vcf/SKILL.md - Workflow-aware DNA/RNA/cDNA haplotyping with an indexed VCF
- remote_execution/SKILL.md - Remote execution workflow
- shared/SKILL_ROUTING_PATTERN.md and shared/DOGME_QUICK_WORKFLOW_GUIDE.md - Shared reference docs
- Use type hints throughout
- Write tests for new features
- Document configuration changes
- Update this README for major changes
# Run full test suite (1040+ tests)
pytest tests/ -q
# With coverage
pytest tests/ --cov=cortex --cov=launchpad --cov-report=html- Check CONFIGURATION.md for configuration issues
- Check launchpad/README.md for execution engine issues
- Check cortex/README.md for agent engine issues
- Release: 3.7.6 β adds optional separate rsync transfer hosts for remote SLURM profiles, with automatic fallback to the SLURM host when no transfer host is configured
- Python: 3.12+
- FastAPI: Latest (from environment.yml)
- SQLAlchemy: 2.0+
- Nextflow: >= 23.0
- Status: Active Development