Skip to content

Latest commit

Β 

History

345 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

AGOUTIC: Automated Genomic Orchestrator

Release: 3.7.6 Status: Active Prototype

🧬 Overview

AGOUTIC is a general-purpose agent for analyzing and interpreting long-read genomic data (Nanopore/PacBio). It uses a Dual Interface architecture (REST + MCP) to allow both human users and AI agents to orchestrate complex bioinformatics pipelines.

The system is composed of:

  • Cortex: Agent Engine - AI-powered orchestration and user interaction
  • Atlas: Consortium Data Integration - Registry-driven ENCODE and IGVF retrieval via MCP
  • Launchpad: Execution Engine - Dogme/Nextflow pipeline management (local + remote SLURM)
  • Analyzer: Analysis Engine - Results analysis and QC reporting
  • edgePython: Differential Expression β€” Bulk/single-cell RNA-seq DE via edgePython
  • XgenePy MCP: Cis/trans regulatory modeling β€” local XgenePy execution with canonical artifacts
  • UI: Web interface for monitoring and control

Current status: database infrastructure centralized in common/database.py with Alembic migrations. Gene annotation and enrichment tools moved from edgePython to Analyzer. The analyzer/server4 adapter layer proxies remaining edgePython MCP calls upstream. Atlas now exposes both ENCODE and IGVF consortium MCP servers through the same schema-aware routing and formatting layer. Shared-project collaboration foundations now use the existing owner/editor/viewer project RBAC consistently for conversation history, project-scoped file routing, analyzer/job access, collaborator management APIs, and the Streamlit sharing UI.

Project Sharing And Collaboration

AGOUTIC projects now support multi-user collaboration with project-scoped viewer, editor, and owner roles.

  • Owners can add collaborators directly from chat or from the Projects page, and can transfer ownership to an existing viewer or editor collaborator.
  • Editors can work in shared projects and keep using normal mutating project actions.
  • Viewers can open shared conversation history, inspect project files and jobs, and browse project state in a read-only mode.
  • Admins can manage collaborators for any project.

The current collaboration UI surfaces are:

  • Chat: natural-language requests such as share this project with alice@example.com open a dedicated sharing flow, and list users / list collaborators display the current project roster with role and activity labels.
  • Sidebar and chat shell: owned vs shared projects are labeled explicitly, viewer-only sessions have chat submission, uploads, and clear-chat disabled, and a list users sidebar button exposes the current collaborator roster. Shared owner/admin chat views also keep a pinned collaborator summary available while the conversation scrolls.
  • Projects page: the project table shows access labels, owner/admin-only mutating controls stay gated, and the Collaborators tab supports add, update, remove, and ownership-transfer actions. Collaborator rosters are grouped by editors and viewers to make role-based access easier to scan.

Collaboration is project-scoped rather than conversation-scoped. Authorized members can reopen shared project conversations and continue follow-up work in the same project when their role allows it.

πŸ–₯️ Execution Modes

AGOUTIC supports dual execution modes:

  • Local: Runs Nextflow/Dogme pipelines directly on the local machine (default, original behavior)
  • Remote SLURM: Submits jobs to a remote SLURM cluster via SSH

Remote execution features:

  • Saved SSH profiles β€” per-user connection profiles with secure key references (no raw secrets stored). Supports local OS user key access through a per-session broker launched under that Unix account with su (password used transiently, never stored)
  • Separate rsync transfer host β€” optionally stage inputs and download outputs through a dedicated host while SLURM submission, polling, and remote commands continue through the configured SSH host; blank transfer-host settings fall back automatically
  • SLURM resource management β€” configurable account, partition, CPUs, memory, walltime, GPUs with validation
  • Shared OpenChromatin GPU runtime defaults β€” DNA SLURM runs now default to the shared Dogme OpenChromatin GPU container and task-scoped runtime wiring instead of the older custom host-mounted modkit path
  • Remote base path model β€” a single remote_base_path anchors ref/, data/, and per-workflow remote directories
  • Remote browsing and stage-only intake β€” browse saved-cluster paths and stage references/input data without submitting a job
  • Completed-workflow import β€” adopt an already-run local or remote Dogme workflow into the current project as the next workflowN, infer metadata from the workflow .config, and keep provenance for later explicit resync
  • Stage transfer controls β€” running stage-only transfers can be refreshed, cancelled, resumed, and failed staging cards can delete their reserved local workflow folders directly from the UI
  • Result destination policy β€” keep results remote-only, copy back locally, or both
  • Staged approval prompts β€” Cortex collects details progressively, presents summary before submission
  • Run and staging status tracking β€” dedicated staging tasks plus remote execution stage labels through completed, including byte-level transfer progress, current-file details, and faster live refresh while transfers are active
  • Scheduler integration β€” SLURM job ID tracking, state polling via sacct/squeue, cancellation via scancel

wf-pore-c Status

AGOUTIC now supports the wf_pore_c workflow family end to end behind WF_PORE_C_ENABLED. Phase 3 is feature-complete across local preview, local execution, remote SLURM staging/submission, Analyzer recognition, automatic summary generation, UI approval/run cards, and backward-compatible conversation/job reconstruction. The flag still defaults to off in both Launchpad and Cortex configs, so existing Dogme behavior remains the default until operators explicitly enable wf-pore-c.

Current validation status:

  • Full closure sweeps passed in both flag states: 1988 passed with WF_PORE_C_ENABLED=false and 1988 passed with WF_PORE_C_ENABLED=true across Cortex, Launchpad, Analyzer, and UI.
  • The identical pass count in both states confirms the flag changes runtime behavior, not test inclusion.
  • Production UI rendering no longer carries the earlier AST-harness fallback branch; missing wf-pore-c helper wiring now fails loudly instead of silently degrading to Dogme metadata.
  • Legacy chats and jobs that only stored mode still normalize to workflow_key="dogme" on both the slow history-rebuild path and the fast cached-state restore path.

Manual cluster validation is still required before operational rollout. Use these docs for the wf-pore-c validation path:

Import Completed Workflows From Chat

In any existing project, you can import a workflow that Dogme already ran earlier:

  • Slash command: /import-workflow /scratch/youruser/agoutic/project-alpha/workflow12 --remote
  • Natural language: import remote workflow from /scratch/youruser/agoutic/project-alpha/workflow12

Optional flags:

  • --full-copy to copy the entire workflow directory instead of the normal result subset
  • --sample-name, --mode, --reference, and --modifications if you want to override metadata inferred from the workflow .config

Imported workflows are assigned the next workflowN folder in the current project. If the source workflow is still incomplete, AGOUTIC keeps a warning on the execution card and you can continue the copy later with /sync-workflow workflow12.

Phase 1 limitation: Analyzer operates on local-accessible files only. Remote results must be copied back before downstream analysis.

See docs/remote_execution_architecture.md for architecture details, docs/cluster_slurm_setup.md for setup, docs/user_guide_execution_modes.md for usage, and TUTORIAL.md for an end-to-end user walkthrough.

πŸ”¬ Analysis Capabilities

AGOUTIC combines computational workflow execution with agent-guided biological interpretation. After a pipeline finishes, the platform helps users move from raw output folders to scientific insight, including isoform behavior, modification patterns, pathway shifts, and gene-level functional context.

Analysis Goals

AGOUTIC is designed to help users:

  • Interpret isoform discovery and transcript structure outputs from long-read workflows
  • Summarize RNA and DNA modification signals from workflow artifacts
  • Review QC metrics across runs and identify quality or completeness issues
  • Read text, markdown, and HTML workflow reports directly from chat or with /read-file
  • Inspect gene-level and transcript-level result tables with context
  • Run downstream differential expression and enrichment analysis
  • Compare outputs across samples, conditions, and workflows

Supported Analysis Types

  • Quality Control (QC) analysis

    • Parse run summaries, count/stat tables, alignment summaries, and basecalling outputs
    • Validate run completeness and surface troubleshooting context from logs and artifacts
    • Generate QC summaries for quick review across workflow outputs
  • Transcriptomic analysis

    • Explore gene- and transcript-level quantification outputs
    • Support isoform-aware interpretation from long-read RNA/cDNA pipelines
    • Review splice-aware and transcript-structure-relevant outputs
  • Epitranscriptomic and epigenomic analysis

    • Summarize RNA modification outputs from direct RNA workflows
    • Summarize DNA modification outputs from DNA workflows
    • Parse and interpret bedMethyl outputs, including region- and gene-linked review where applicable
  • Differential analysis

    • Run bulk and single-cell RNA-seq differential expression through edgePython
    • Compare grouped samples directly from reconciled workflow abundance tables or saved dataframes
    • Use the stateful DE flow: load β†’ filter β†’ normalize β†’ design β†’ fit β†’ test β†’ results
    • Prefer workflow-local reconciled.gtf annotation when present so transcript-aware plots and summaries stay aligned with the active workflow output
    • Filter by FDR/logFC and report annotated top genes for interpretation
  • Functional interpretation

    • Run GO enrichment (BP, MF, CC)
    • Run Reactome and KEGG pathway enrichment
    • Translate Ensembl IDs and normalize symbols for human and mouse datasets
    • Build and reuse colocated GTF-backed gene/transcript caches for workflow-local or custom annotations

For deeper tool-level details, see analyzer/README.md, skills/differential_expression/SKILL.md, skills/enrichment_analysis/SKILL.md, and SKILLS.md.

Typical Analysis Workflow

Pipeline execution ↓ Result discovery ↓ QC parsing and summary generation ↓ Expression / isoform / modification result extraction ↓ Optional differential expression analysis ↓ Functional enrichment ↓ Agent-guided visualization and biological interpretation

What Happens Next in Analysis

Once parsing and summaries are complete, AGOUTIC can pivot into follow-up interpretation tasks such as cross-workflow comparison, condition-focused differential analysis, and targeted functional hypotheses (for example, pathway-level shifts or biologically coherent gene programs).

Analysis Inputs

The analysis layer consumes:

  • Pipeline result folders (for example workflow1/, workflow2/)
  • CSV/TSV/BED/bedMethyl files
  • Counts and summary/statistics tables
  • Annotation and quantification outputs
  • User-selected files from workflow subdirectories

Analysis Outputs

The analysis layer returns:

  • Parsed result tables
  • QC summaries and run-validation context
  • Annotated gene/transcript lists
  • Differential expression result tables
  • GO/pathway enrichment tables
  • Interactive plots and chart-ready summaries
  • Chat-readable scientific interpretation for downstream decisions

Example Analysis Requests

  • Summarize the QC for workflow2
  • List the important files in workflow1/annot
  • Parse the bedMethyl output and summarize methylation patterns
  • Haplotype RNA workflow7 with file /data/parents.vcf.gz
  • Haplotype mouse sample B6 Cast F1 workflow7
  • /haplotype DNA workflow5 /data/family.vcf.gz
  • /haplotype RNA workflow7 --vcf-sample B6,CAST
  • Show the top expressed genes from this result file
  • Run differential expression between control and treatment
  • Compare the treated samples treated_1 and treated_2 to the control samples ctrl_1 and ctrl_2
  • Compare treated_1 and treated_2 to ctrl_1 and ctrl_2 from DF1 at transcript level
  • /de treated=treated_1,treated_2 vs control=ctrl_1,ctrl_2
  • Annotate these Ensembl IDs
  • Run GO enrichment on the upregulated genes
  • Compare workflow1 and workflow2 outputs

Haplotype Reads With A VCF

AGOUTIC can haplotype long-read DNA, RNA, or cDNA workflow BAMs against a VCF through the haplotype_with_vcf skill.

  • Slash command: /haplotype RNA workflow7 /data/parents.vcf.gz
  • Natural language: haplotype RNA workflow7 with file /data/parents.vcf.gz
  • Mouse founder-mode natural language: haplotype mouse sample B6 Cast F1 workflow7
  • Mouse founder-mode slash command: /haplotype RNA workflow7 --vcf-sample B6,CAST
  • Cross-project workflow reference: haplotype B6CASTF1 RNA mouse sample otherproject:workflow7
  • DNA workflows resolve mapped BAMs from bams/, RNA/cDNA workflows resolve annotated BAMs from annot/, and reconcile workflows use root-level *.annotated.bam files.
  • Cross-project haplotype requests can target another project's workflow with project_name:workflowN while still writing outputs into a new workflow under the active project unless you explicitly override the destination.
  • Plain .vcf inputs are compressed to .vcf.gz and indexed automatically when AGOUTIC can write beside the source file; .vcf.gz inputs missing .tbi or .csi are auto-indexed.
  • Mouse/mm39 founder-panel requests can omit the VCF. In that case AGOUTIC resolves mgp_REL2021_snps_founders.vcf.gz from the same directory as the configured mm39 reference FASTA.
  • Mouse founder aliases are case-insensitive and ignore /, _, -, and spaces. ref, B6, C57BL6, and C57BL6/J all resolve to C57BL_6J; CAST, CAST/J, and CAST_EiJ resolve to CAST_EiJ.
  • Founder-pair restrictions can be written as repeated --vcf-sample flags, comma-separated --vcf-sample B6,CAST, F1 shorthand such as B6CastF1 or B6 Cast F1, or natural language such as haplotype mouse between B6 and CAST workflow7.
  • The approval gate lists the exact BAM names, the selected VCF sample or founder subset, the resolved VCF path, assignment labels, and the destination workflowN before execution starts.

Visualization Support

  • Inline Plotly visualizations directly in chat
  • Interactive bar, scatter, heatmap, box, histogram, pie, venn, and upset plots from conversation dataframes
  • Automatic plotting from parsed tables when chartable data are detected
  • Cross-workflow open-chromatin overlap requests can run as approval-gated background workflows that write workflow-scoped overlap CSVs and manifests for downstream plotting
  • Saved venn/upset overlap plots over earlier chat dataframes reload the full source table from analyzer provenance or matching project files when only preview rows are available, so overlap counts stay accurate and two-set venn diagrams remain readable for large overlaps
  • edgePython-backed DE and enrichment plots now default to 600 dpi raster export with an SVG companion for project-scoped artifact output
  • Publication-style volcano and MD plots route through edgePython, while generic dataframe charts remain an interactive Plotly path rather than a server-side publication export path

Current Analysis Limitations

  • Analyzer currently requires local-accessible files
  • Remote-only results must be copied back before downstream analysis
  • Some analysis pathways are file-format dependent and assume expected output conventions
  • Cross-run comparison is strongest when workflows use consistent references and naming
  • Interpretation depth depends on pipeline completeness and annotation availability

See docs/remote_execution_architecture.md for remote execution constraints and staging/copy-back behavior.

πŸ”’ Security & Multi-User Isolation

AGOUTIC enforces access control at every layer:

  • Authentication: Google OAuth 2.0 with session cookies (httponly, samesite=lax, secure in production)
  • Authorization: Role-based access (owner / editor / viewer) checked on every endpoint via require_project_access(). Admins bypass all project-level checks; public projects allow viewer access.
  • Project collaboration: Shared projects expose conversation history and other project-scoped surfaces to members according to project role. Viewer access is read-only; editor access is required for project mutations such as linking jobs to conversations or writing project files.
  • Run-level access: Each job still records the submitting user_id, but require_run_uuid_access() now authorizes via the job's project_id first so shared-project collaborators can inspect project jobs; direct user_id ownership remains a fallback for legacy rows.
  • File isolation: Project files resolve through the canonical shared project directory, while downloaded/uploaded source files remain in the acting user's central data folder (AGOUTIC_DATA/users/{username}/data/). Project data/ entries are symlinks into that central store, preserving per-user private storage while letting collaborators work in one shared project tree. Legacy {user_id}/{project_id} paths are still supported for backward compatibility.
  • Server-side project IDs: UUIDs generated server-side via uuid4() β€” clients never control the ID.
  • Project management: Full dashboard for browsing projects, viewing stats/files/jobs, renaming, archiving, and permanent deletion with cascading cleanup.
  • Bootstrap & admin scripts: Run python scripts/cortex/init_db.py for a fresh database bootstrap, python scripts/cortex/set_usernames.py auto to derive usernames from email addresses on an existing instance, and python scripts/cortex/bootstrap_project_tasks.py to seed persistent project tasks from existing workflow history.

πŸš€ Quick Start

Installation

# Create environment
conda env create -f environment.yml
conda activate agoutic_core

# Or update an existing environment in place
conda env update -n agoutic_core -f environment.yml --prune

The environment now includes htslib (bgzip, tabix) and bcftools so AGOUTIC can auto-compress and index VCF inputs for haplotyping and operators still have standard VCF command-line tooling available. bcftools is resolved from bioconda, so one-off manual installs should use conda-forge plus bioconda with strict channel priority.

Run the System

AGOUTIC now supports two UI topologies:

  • Hosted Streamlit: Streamlit and Cortex run on the same server or network boundary and the browser uses the existing cookie-based login flow.
  • Local Streamlit against remote Cortex: Streamlit runs on the user's machine, AGOUTIC_API_URL points at the remote Cortex server, and login returns to the local Streamlit page so it can exchange a bearer session.

In both modes, the UI talks only to Cortex. End-user UI environments do not need LAUNCHPAD_REST_URL or INTERNAL_API_SECRET.

# Recommended: start the full backend stack
./agoutic_servers.sh --start

# Then start the UI separately
streamlit run ui/appUI.py --server.address 0.0.0.0 --server.port 8501

For local development, you can still run services manually:

# Terminal 1: Start Launchpad REST
uvicorn launchpad.app:app --host 0.0.0.0 --port 8003 --reload

# Terminal 2: Start Launchpad MCP
python -m launchpad.mcp_server --host 0.0.0.0 --port 8002

# Terminal 3: Start Cortex
uvicorn cortex.app:app --host 0.0.0.0 --port 8000 --reload

# Terminal 4: Start UI
cd ui && streamlit run appUI.py

For a local UI talking to a remote Cortex server:

export AGOUTIC_API_URL=http://remote-cortex-host:8000
streamlit run ui/appUI.py --server.port 8501

When the UI is running on localhost or 127.0.0.1, the login button automatically requests the local-client auth flow. Cortex redirects back to the local Streamlit URL with a short-lived auth code, and the UI exchanges that code for a bearer session.

Note: running python ui/appUI.py directly will not work correctly because the UI auth flow depends on Streamlit request context plus either browser cookies (hosted mode) or the local auth-code exchange (local mode).

Verify Installation

# Check Cortex health
curl http://localhost:8000/health

# Check Launchpad health
curl http://localhost:8003/health

# Test Atlas connection
python cortex/atlas_mcp_client.py

# Expected: Connection success and K562 search results

Task System

AGOUTIC now maintains a persistent project task list instead of relying on a hard-coded checklist in the UI.

  • Tasks are projected from durable workflow records, mainly ProjectBlock state plus job progress payloads.
  • The chat page groups tasks into pending, running, follow-up, and completed sections.
  • Parent tasks can include child tasks for workflow stages, per-file download progress, analysis completion, and result review.
  • Existing history can be backfilled safely with:
python scripts/cortex/bootstrap_project_tasks.py

# Optional: seed only one project
python scripts/cortex/bootstrap_project_tasks.py --project-id <project_id>

πŸ“‹ Architecture Overview

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   AGOUTIC System v3.7.2                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                              β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                                               β”‚
β”‚  β”‚  Web UI  β”‚ (Streamlit)                                   β”‚
β”‚  β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜                                               β”‚
β”‚       β”‚ REST API                                            β”‚
β”‚       ↓                                                      β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                β”‚
β”‚  β”‚        Cortex (Agent Engine)         β”‚                β”‚
β”‚  β”‚     AI Orchestration + Coordination    β”‚                β”‚
β”‚  β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                β”‚
β”‚       β”‚            β”‚            β”‚                           β”‚
β”‚       β”‚ MCP        β”‚ REST       β”‚ MCP                       β”‚
β”‚       ↓            ↓            ↓                           β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                   β”‚
β”‚  β”‚Atlasβ”‚  β”‚ Launchpad β”‚  β”‚ Analyzer β”‚                   β”‚
β”‚  β”‚ENCODE  β”‚  β”‚ Nextflow β”‚  β”‚ Analysis β”‚                   β”‚
β”‚  β”‚ Portal β”‚  β”‚ Pipeline β”‚  β”‚  Engine  β”‚                   β”‚
β”‚  β””β”€β”€β”€β”€β”¬β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜                   β”‚
β”‚       β”‚            β”‚              β”‚                         β”‚
β”‚       ↓            ↓              ↓                         β”‚
β”‚    ENCODE      Dogme          Results                      β”‚
β”‚    Portal    Pipelines         Files                       β”‚
β”‚                                                              β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ”§ System Components

Cortex: Agent Engine (Port 8000)

  • Role: Central orchestrator with LLM reasoning
  • Tech: FastAPI + OpenAI-compatible LLM
  • Features:
    • Chat interface with skill-based workflows
    • Coordinates Atlas, Launchpad, and Analyzer
    • Block-based project timeline
    • Persistent project task list with child-task projection for downloads, workflow stages, analysis, and follow-up review
    • Background job monitoring with stop/cancel buttons
    • Download cancel button β€” "πŸ›‘ Cancel Download" on running downloads with partial-file cleanup
    • "List my data" command β€” chat-based central data folder listing (DB + disk fallback)
    • Post-cancel workflow management β€” Delete / Resubmit buttons on cancelled jobs; chat-based deletion via natural language
    • User authentication
    • Role-based authorization gates on all endpoints
    • Server-side project CRUD (POST/GET/PATCH /projects)
    • [[PLOT:...]] tag parsing β†’ AGENT_PLOT blocks for inline Plotly charts (histogram, scatter, line, area, bar, box, violin, strip, heatmap, pie, venn, upset)
    • Per-message and per-conversation token tracking β€” every LLM response records prompt_tokens, completion_tokens, total_tokens, and model_name in the database; exposed via GET /user/token-usage (own data) and GET /admin/token-usage (all users)
    • find_file echo recovery β€” when a weak model emits a find_file JSON result verbatim instead of a [[DATA_CALL:...]] tag, the pipeline intercepts the response, auto-chains to parse_csv_file/parse_bed_file/read_file_content, and strips the bad block from conversation history to prevent looping
    • ENCODE tool routing guards β€” structural checks prevent LLM misrouting (e.g. cell-line names sent to get_experiment); assay-only queries are routed to search_by_assay; assay name aliases (e.g. RNA-seq β†’ total RNA-seq, ChIP-seq β†’ TF ChIP-seq) are resolved before MCP dispatch
    • IGVF dispatch guards β€” malformed IGVF calls now backfill missing required sample_term/assay_title values from the user message when possible, and schema validation drops still-invalid calls before they hit MCP
    • Tool Schema Contracts β€” machine-readable JSON Schema for every MCP tool, fetched at startup from /tools/schema endpoints on all servers. Injected into the system prompt as a compact reference and used for pre-call param validation (strip unknown params, check required fields, normalise enums).
    • Structured Conversation State β€” typed ConversationState JSON (skill, project, sample, experiment, dataframes, workflows) built each turn and injected as [STATE]...[/STATE] so the LLM always sees current context
    • Error-Handling Playbook β€” deterministic failure rules in the system prompt + structured [TOOL_ERROR] blocks + single-retry for transient failures
    • Output Contract Validator β€” post-LLM validation catches malformed DATA_CALL tags, duplicate APPROVAL_NEEDED, unknown tools, and mixed sources
    • Provenance Tags β€” [TOOL_RESULT: source, tool, params, rows, timestamp] headers on every tool result for auditability; persisted in AGENT_PLAN blocks
    • Plan-Execute-Observe-Replan β€” structured multi-step planning layer that now runs through manifest-first classification and composition for core deterministic flows. SkillManifest metadata supplies planner triggers, expected inputs, required services, runtime hints, and MCP tool chains; plan_composer.py builds DE, enrichment, and XgenePy plans from that metadata; legacy templates remain as deterministic fallback for unmigrated flows; and CHECK_EXISTING guards still skip expensive operations when results already exist.
    • Deterministic skill-management commands β€” /skills, /skill <skill_key>, and /use-skill <skill_key> expose the live skill catalog, describe individual skills, and persist manual skill switching across turns without relying on a freeform model response.
    • Gene Annotation & ID Translation β€” offline Ensembl gene ID ↔ symbol translation (human + mouse) via pre-built lookup tables. Auto-annotates gene symbols when DE data is loaded; all downstream outputs (top genes, heatmaps, summaries) automatically use readable symbols instead of raw Ensembl IDs. Bidirectional lookup_gene tool answers "what is the Ensembl ID for TP53?" style queries. Pre-LLM auto-skill detection routes gene questions to the correct skill from any context (including Welcome). MCP tools: annotate_genes (edgePython, DE-stateful), translate_gene_ids and lookup_gene (Analyzer).
    • Robust DATA_CALL tag parsing β€” bracket-aware parameter parser handles JSON arrays inside DATA_CALL tags (e.g. gene_symbols=["TP53", "BRCA1"]). Mistral-native [TOOL_CALLS]DATA_CALL: format is auto-normalized to standard [[DATA_CALL:...]] tags.
    • Skill-defined plan chains β€” skill authors can declare multi-step workflows in skill Markdown files under a ## Plan Chains section. A single message like "get K562 experiments and make a plot by assay type" is detected at classify-time and produces both a data search and a visualization. Trigger phrases support multi-phrasing (AND/OR keyword groups) for flexible matching. See SKILLS.md for the full authoring guide.
    • Skills system documentation β€” new top-level SKILLS.md documents the complete skills framework: skill file structure, routing patterns, [[DATA_CALL:...]] / [[PLOT:...]] tag system, plan chains format, and a step-by-step guide for creating new skills.
    • Inline Plotly visualizations with deduplication β€” [[PLOT:...]] tags produce interactive bar, scatter, line, area, violin, strip, pie, histogram, heatmap, venn, and upset charts rendered directly in chat. A three-layer pipeline guarantees chart generation: chain context injection β†’ second-pass PLOT tag instructions β†’ post-DataFrame fallback. Deduplication and prompt-intent selection prevent duplicate/overlapping traces and stale style leakage across turns. Supports explicit colors (color=green), grouped or stacked bar modes, and boolean-column overlap plots for presentation-friendly set comparisons.
    • DF inspection quick commands β€” list dfs lists all dataframes in the conversation with their metadata; head df1 (or head df3 5) shows the first N rows as a markdown table. Both bypass the LLM entirely β€” zero token cost.
    • In-memory dataframe actions β€” filter, subset, select columns, rename, sort, melt, aggregate, join, and pivot existing conversation dataframes without going through analyzer file calls. Saved transforms can also appear as block-specific PENDING_ACTION controls in the UI.

Atlas: Consortium Data MCPs (ENCODE 8006 / IGVF 8009)

  • Role: Registry-driven public consortium data retrieval
  • Tech: fastmcp + Atlas registry/configuration with ENCODE and IGVF MCP servers under atlas/
  • Features:
    • Search ENCODE experiments by biosample/organism/target
    • search_by_assay β€” assay-first ENCODE search (e.g. "how many RNA-seq experiments") across both organisms, returning combined counts and per-organism lists
    • Search IGVF measurement sets, analysis sets, prediction sets, files, genes, and samples
    • Download and file-metadata helpers for both consortium integrations
    • Metadata caching, result formatting, and tool/parameter alias repair
    • 30+ MCP tools across the current consortium integrations
    • Agent routing guards in Cortex prevent structural misrouting and block invalid required-param calls before MCP execution
  • Extension pattern: atlas/mcp_server.py extends ENCODELIB's FastMCP server, while atlas/igvf_mcp_server.py and atlas/launch_igvf.py provide the parallel IGVF HTTP MCP server path.
  • Tool schemas: atlas/tool_schemas.py and atlas/igvf_tool_schemas.py define JSON Schema contracts for the ENCODE and IGVF tools, served via /tools/schema GET endpoints.
  • Docs: ATLAS_IMPLEMENTATION.md

Launchpad: Execution Engine (Ports 8003 REST / 8002 MCP)

  • Role: Nextflow pipeline execution
  • Tech: FastAPI + Nextflow + Dogme
  • Features:
    • Submit Dogme DNA/RNA/cDNA pipelines
    • Shared OpenChromatin GPU container defaults for SLURM DNA runs with task-scoped runtime injection for OpenChromatin work
    • Real-time job monitoring
    • Log streaming
    • User-jailed working directories
    • Job cancellation β€” SIGTERM-based cancel with cooperative .nextflow_cancelled marker; properly displays CANCELLED (not FAILED) in UI
    • Workflow folder deletion β€” DELETE endpoint removes work directory and sets status to DELETED; block status updated immediately so UI reflects deletion
    • Job resume β€” resubmit cancelled/failed jobs with Nextflow -resume flag to reuse cached task results in the same workflow directory instead of starting fresh
    • Stage-only transfer lifecycle controls β€” refresh, cancel, resume, and failed-stage cleanup actions are available in both the workflow UI and Task Center
    • Live staging telemetry β€” running transfers surface current file, transferred bytes, total size, and faster refresh cadence during brokered or direct rsync activity
    • delete_job_data MCP tool β€” enables chat-based deletion ("delete workflow1")
  • Docs: launchpad/README.md

edgePython: Differential Expression (Port 8007)

  • Role: Bulk and single-cell RNA-seq differential expression analysis
  • Tech: FastMCP + edgePython
  • Features:
    • Full DE pipeline: load β†’ filter β†’ normalize β†’ design β†’ dispersion β†’ fit β†’ test β†’ results β†’ plots
    • Gene list filtering from DE results by FDR, logFC, and direction (up/down/all)
    • Gene annotation (annotate_genes) on DE results in-place
    • Workflow-local reconciled.gtf preference for annotation when available, with shared reference caches as fallback
    • Stateful pipeline β€” each step builds on previous results within a session
    • Volcano, MDS, MA, BCV, heatmap plot generation
    • TSV/CSV/JSON result export
    • JSON Schema tool contracts via /tools/schema
  • Docs: edgepython_mcp/

Analyzer: Analysis Engine (Ports 8004 REST / 8005 MCP)

  • Role: Results analysis, QC reporting, gene annotation, and GO/pathway enrichment
  • Tech: fastmcp + Python analysis tools + g:Profiler
  • Features:
    • Parse pipeline outputs (CSV, TSV, BED files)
    • Generate QC reports and analysis summaries
    • File discovery and content reading
    • Workflow folder browsing via list_job_files
    • Gene ID translation (translate_gene_ids) and bidirectional lookup (lookup_gene) via Ensembl reference tables
    • GTF-backed gene and transcript annotation with colocated caches for shared references, workflow-local outputs, or custom user-provided GTFs
    • GO enrichment (BP/MF/CC) and pathway enrichment (KEGG/Reactome) via g:Profiler
    • Per-conversation enrichment state management
    • Species auto-detection from gene ID prefixes (ENSG β†’ human, ENSMUSG β†’ mouse)
  • Workflow Directory Layout:
    $AGOUTIC_DATA/users/{username}/{project-slug}/
    β”œβ”€β”€ data/              # Uploaded input data
    β”œβ”€β”€ workflow1/         # First job's output
    β”‚   β”œβ”€β”€ annot/         # Annotations, final stats, counts
    β”‚   β”œβ”€β”€ bams/          # BAM alignment files
    β”‚   β”œβ”€β”€ bedMethyl/     # Methylation BED output
    β”‚   β”œβ”€β”€ fastqs/        # FASTQ files
    β”‚   └── ...
    └── workflow2/         # Second job's output
    
  • Agent Commands (handled automatically by Cortex's safety net):
    • list my data / list my files β€” lists all files in your central data folder
    • list workflows β€” lists all workflow folders in the project
    • list files / list files in workflow2/annot β€” lists files in a workflow or subfolder
    • parse annot/File.csv β€” finds and parses a file by relative path
    • parse workflow2/annot/File.csv β€” parses a file in a specific workflow
  • Docs: analyzer/README.md

πŸ“ Project Structure

agoutic/
β”œβ”€β”€ README.md                     # This file
β”œβ”€β”€ environment.yml               # Conda environment specification
β”œβ”€β”€ alembic.ini                  # Alembic migration configuration
β”œβ”€β”€ CONFIGURATION.md              # Path configuration guide
β”œβ”€β”€ ATLAS_IMPLEMENTATION.md     # Atlas integration guide
β”œβ”€β”€ ATLAS_QUICKSTART.md         # Atlas quick reference
β”‚
β”œβ”€β”€ cortex/                      # Agent Engine
β”‚   β”œβ”€β”€ README.md                # Cortex documentation
β”‚   β”œβ”€β”€ app.py                   # FastAPI application
β”‚   β”œβ”€β”€ agent_engine.py          # AI agent orchestration
β”‚   β”œβ”€β”€ skill_manifest.py        # Skill capability registry for routing + planning
β”‚   β”œβ”€β”€ plan_classifier.py       # Manifest-first request classification
β”‚   β”œβ”€β”€ plan_composer.py         # Manifest-driven deterministic plan builder
β”‚   β”œβ”€β”€ planner.py               # Planner orchestration + fallback selection
β”‚   β”œβ”€β”€ plan_executor.py         # Deterministic step execution engine
β”‚   β”œβ”€β”€ plan_replanner.py        # Failure recovery + plan adjustment
β”‚   β”œβ”€β”€ dependencies.py          # Auth gates (require_project_access, require_run_uuid_access)
β”‚   β”œβ”€β”€ user_jail.py             # Path traversal guards & file isolation
β”‚   β”œβ”€β”€ auth.py                  # Google OAuth 2.0 + cookie hardening
β”‚   β”œβ”€β”€ models.py                # Database models
β”‚   β”œβ”€β”€ schemas.py               # Request/response schemas
β”‚   β”œβ”€β”€ config.py                # Configuration
β”‚   β”œβ”€β”€ db.py                    # Database connection
β”‚   β”œβ”€β”€ prompt_templates/        # LLM system prompts (first-pass, planning, second-pass)
β”‚   └── routes/                  # Extracted REST route modules
β”‚
β”œβ”€β”€ launchpad/                      # Execution Engine
β”‚   β”œβ”€β”€ README.md                # Launchpad documentation
β”‚   β”œβ”€β”€ app.py                   # FastAPI application
β”‚   β”œβ”€β”€ nextflow_executor.py    # Nextflow wrapper
β”‚   β”œβ”€β”€ mcp_tools.py            # MCP tool definitions
β”‚   β”œβ”€β”€ mcp_server.py           # MCP server
β”‚   β”œβ”€β”€ models.py               # Database models
β”‚   β”œβ”€β”€ schemas.py              # Request/response schemas
β”‚   β”œβ”€β”€ config.py               # Configuration
β”‚   β”œβ”€β”€ db.py                   # Database connection
β”‚   β”œβ”€β”€ quickstart.sh           # Quick start setup
β”‚   β”œβ”€β”€ DUAL_INTERFACE.md       # REST + MCP architecture
β”‚   └── IMPLEMENTATION_SUMMARY.md # Implementation details
β”‚
β”œβ”€β”€ scripts/                     # Manual admin and operational utilities
β”‚   β”œβ”€β”€ cortex/
β”‚   β”‚   β”œβ”€β”€ init_db.py           # Fresh database bootstrap utility
β”‚   β”‚   β”œβ”€β”€ set_usernames.py     # Username/slug admin CLI
β”‚   β”‚   └── bootstrap_project_tasks.py # Backfill persistent project tasks
β”‚   β”œβ”€β”€ launchpad/
β”‚   β”‚   β”œβ”€β”€ debug_job.py         # Job inspection helper
β”‚   β”‚   └── submit_real_job.py   # Manual job submission helper
β”‚   └── build_gene_reference.py  # One-time Gencode GTF β†’ TSV builder
β”‚
β”œβ”€β”€ ui/                          # Web Interface
β”‚   β”œβ”€β”€ README.md               # UI documentation
β”‚   β”œβ”€β”€ app.py                  # Streamlit main app (chat, sidebar, auto-refresh)
β”‚   └── pages/
β”‚       β”œβ”€β”€ projects.py         # Projects dashboard (stats, files, bulk actions)
β”‚       β”œβ”€β”€ results.py          # Job results analysis (auto-lists project jobs)
β”‚       └── admin.py            # Admin user management
β”‚
β”œβ”€β”€ atlas/                      # ENCODE MCP Extension
β”‚   β”œβ”€β”€ launch_encode.py        # HTTP launcher (imports mcp_server for extensions)
β”‚   β”œβ”€β”€ mcp_server.py           # Extends ENCODELIB FastMCP with search_by_assay + /tools/schema
β”‚   β”œβ”€β”€ tool_schemas.py         # JSON Schema contracts for all 16 ENCODE tools
β”‚   β”œβ”€β”€ config.py               # Atlas configuration
β”‚   └── result_formatter.py     # Result formatting helpers
β”‚
β”œβ”€β”€ edgepython_mcp/              # edgePython DE Server
β”‚   β”œβ”€β”€ edgepython_server.py    # FastMCP tool definitions (DE + filtering)
β”‚   β”œβ”€β”€ mcp_server.py           # Server wrapper + /tools/schema endpoint
β”‚   β”œβ”€β”€ launch_edgepython.py    # HTTP launcher
β”‚   β”œβ”€β”€ tool_schemas.py         # JSON Schema contracts for DE tools
β”‚   └── config.py               # Configuration
β”‚
β”œβ”€β”€ common/                      # Shared Utilities
β”‚   β”œβ”€β”€ database.py            # Centralized DB infrastructure (Base, engines, sessions)
β”‚   β”œβ”€β”€ gene_annotation.py      # Ensembl gene ID ↔ symbol translation (bidirectional)
β”‚   β”œβ”€β”€ mcp_client.py           # Shared MCP HTTP client
β”‚   β”œβ”€β”€ logging_config.py       # Structured logging setup
β”‚   └── logging_middleware.py   # Request logging middleware
β”‚
β”œβ”€β”€ skills/                      # Workflow Definitions
β”‚   β”œβ”€β”€ welcome/SKILL.md                 # New-user onboarding
β”‚   β”œβ”€β”€ ENCODE_Search/SKILL.md           # ENCODE search skill + routing rules
β”‚   β”œβ”€β”€ ENCODE_LongRead/SKILL.md         # ENCODE pipeline definition
β”‚   β”œβ”€β”€ run_dogme_dna/SKILL.md           # DNA pipeline definition
β”‚   β”œβ”€β”€ run_dogme_rna/SKILL.md           # RNA pipeline definition
β”‚   β”œβ”€β”€ run_dogme_cdna/SKILL.md          # cDNA pipeline definition
β”‚   β”œβ”€β”€ analyze_local_sample/SKILL.md    # Sample intake workflow
β”‚   β”œβ”€β”€ analyze_job_results/SKILL.md     # Post-pipeline results analysis
β”‚   β”œβ”€β”€ download_files/SKILL.md          # File download workflow
β”‚   β”œβ”€β”€ differential_expression/SKILL.md # edgePython DE skill
β”‚   β”œβ”€β”€ enrichment_analysis/SKILL.md     # GO & pathway enrichment skill
β”‚   β”œβ”€β”€ remote_execution/SKILL.md        # Remote SLURM workflow
β”‚   └── shared/
β”‚       β”œβ”€β”€ SKILL_ROUTING_PATTERN.md     # Shared skill routing reference
β”‚       └── DOGME_QUICK_WORKFLOW_GUIDE.md # Shared workflow parsing guide
β”‚
└── data/                        # Data & Database (created at runtime)
    β”œβ”€β”€ database/
    β”‚   └── agoutic_v24.sqlite
    β”œβ”€β”€ reference/               # Gene annotation reference files
    β”‚   β”œβ”€β”€ human_genes.tsv
    β”‚   └── mouse_genes.tsv
    β”œβ”€β”€ launchpad_work/            # Job execution directories
    β”œβ”€β”€ launchpad_logs/            # Server logs
    └── users/                   # Per-user jailed project dirs

πŸ”‘ Key Concepts

Dual Interface Architecture

AGOUTIC provides two complementary interfaces:

  1. REST API - For web clients, dashboards, and scripting

    • Traditional HTTP endpoints
    • Easy integration with existing tools
    • Language-agnostic clients
  2. MCP Protocol - For LLM agents and AI orchestration

    • Model Context Protocol (MCP)
    • Tools exposed as structured capabilities
    • Seamless AI agent integration

See launchpad/DUAL_INTERFACE.md for detailed architecture.

Job Submission Workflow

User Request
    ↓
Cortex (Agent)
  - Interprets intent
  - Plans workflow
  - (Optional) Requests approval
    ↓
Launchpad (Executor)
  - Receives job
  - Generates Nextflow config
  - Submits to cluster/local
  - Monitors progress
    ↓
Dogme Pipeline
  - Basecalling
  - Alignment
  - Quantification
  - Modification calling
    ↓
Results & Reports
  - Return to Agent
  - Display in UI

Supported Analysis Modes

DNA Mode

  • Genomic DNA and Fiber-seq analysis
  • Includes modification calling (5mC, 6mA, etc.)
  • Full basecalling β†’ alignment β†’ quantification pipeline

RNA Mode

  • Direct RNA-seq analysis
  • Native RNA modification calling (m6A, pseU, etc.)
  • Splice-aware alignment

cDNA Mode

  • Polyubiquitin cDNA and isoform analysis
  • No modification calling (faster processing)
  • Transcript quantification focus

βš™οΈ Configuration

AGOUTIC uses two root path variables with sensible defaults:

# Where source code lives (auto-detected)
export AGOUTIC_CODE=/path/to/agoutic

# Where data/database/jobs live (defaults to $AGOUTIC_CODE/data)
export AGOUTIC_DATA=/path/to/storage

No required environment variables for default local setup. Defaults work automatically, but AGOUTIC_CODE and AGOUTIC_DATA can be overridden if needed. See CONFIGURATION.md for detailed configuration options.

Path Resolution

AGOUTIC_CODE/
β”œβ”€β”€ cortex/          # Agent engine
β”œβ”€β”€ launchpad/          # Execution engine
β”œβ”€β”€ ui/               # Web interface
└── skills/           # Workflow definitions

AGOUTIC_DATA/
β”œβ”€β”€ database/         # SQLite database
β”œβ”€β”€ launchpad_work/     # Job working directories
β”œβ”€β”€ launchpad_logs/     # Server logs
β”œβ”€β”€ logs/             # Structured logs (all servers)
└── users/            # Per-user jailed project dirs
    └── {username}/   # e.g. eli/
        └── {project-slug}/  # e.g. k562-atac-seq/
            β”œβ”€β”€ data/
            β”œβ”€β”€ results/
            └── workflow1/

οΏ½ Logging & Observability

AGOUTIC uses structlog for unified structured logging across all servers. Every log entry is a JSON object written to both per-server and unified log files.

Log Files

Logs are written to $AGOUTIC_DATA/logs/:

$AGOUTIC_DATA/logs/
β”œβ”€β”€ agoutic.jsonl          # Unified log (all servers)
β”œβ”€β”€ cortex.jsonl          # Cortex only
β”œβ”€β”€ launchpad-rest.jsonl     # Launchpad REST API
β”œβ”€β”€ launchpad-mcp.jsonl      # Launchpad MCP
β”œβ”€β”€ analyzer-rest.jsonl     # Analyzer REST API
β”œβ”€β”€ analyzer-mcp.jsonl      # Analyzer MCP
β”œβ”€β”€ encode-mcp.jsonl       # ENCODE MCP server
β”œβ”€β”€ *.log                  # Raw stdout/stderr (safety net)
└── *.YYYYMMDD_HHMMSS.*   # Rotated previous logs

Reading Logs

# Stream the unified log
tail -f $AGOUTIC_DATA/logs/agoutic.jsonl | jq .

# Filter by server
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.server == "cortex")'

# Filter by log level
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.level == "error")'

# Filter requests by path
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.path == "/chat")'

# Find slow requests (>1s)
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.duration_ms > 1000)'

# Trace a request across servers by request_id
cat $AGOUTIC_DATA/logs/agoutic.jsonl | jq 'select(.request_id == "some-uuid")'

Request Tracing

Every HTTP request receives a unique X-Request-ID header. This ID is:

  • Bound to all log entries emitted during the request
  • Returned in the response X-Request-ID header
  • Available at request.state.request_id in route handlers

Log Rotation

When servers are started or restarted via agoutic_servers.sh, existing log files are automatically renamed with a timestamp (e.g., cortex.20260213_143052.jsonl). Empty log files are skipped.

Environment Variables

Variable Default Description
AGOUTIC_LOG_FORMAT json Set to dev for coloured human-readable console output
AGOUTIC_API_URL http://127.0.0.1:8000 Cortex base URL used by Streamlit. Set this when running the UI against a remote Cortex server.
FRONTEND_URL http://localhost:8501 Hosted Streamlit origin used by Cortex for the browser-cookie login flow.
GOOGLE_REDIRECT_URI http://localhost:8000/auth/callback OAuth callback URL exposed by Cortex. Must match the value registered in Google Cloud.
LOCAL_UI_ALLOWED_ORIGINS unset Optional comma-separated extra local UI origins allowed for auth-code return targets. Loopback origins (localhost, 127.0.0.1) work without this.

οΏ½πŸš€ Running the System

Development Mode

# Recommended for a full local stack
./agoutic_servers.sh --start

# Start the UI separately
streamlit run ui/appUI.py --server.port 8501

For a local Streamlit client against a remote Cortex deployment:

export AGOUTIC_API_URL=http://remote-cortex-host:8000
streamlit run ui/appUI.py --server.port 8501

The Streamlit UI now proxies remote profile management through Cortex, so the local UI environment does not need direct Launchpad REST credentials.

If you need manual development startup:

# Terminal 1: Start Launchpad
cd /path/to/agoutic
uvicorn launchpad.app:app --port 8003 --reload

# Terminal 2: Start Launchpad MCP
python -m launchpad.mcp_server --host 0.0.0.0 --port 8002

# Terminal 3: Start Cortex
uvicorn cortex.app:app --port 8000 --reload

# Terminal 4: Start UI (if using Streamlit)
cd ui && streamlit run appUI.py

Using MCP Interface

from cortex.mcp_client import LaunchpadMCPClient

# Connect to MCP server
client = LaunchpadMCPClient()
await client.connect()

# Submit a job
job = await client.submit_dogme_job(
    project_id="proj_001",
    sample_name="liver_dna",
    mode="DNA",
    input_directory="/data/pod5"
)

# Monitor progress
status = await client.check_nextflow_status(job["run_uuid"])

Using REST API

# Submit job
curl -X POST http://localhost:8003/jobs/submit \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_001",
    "sample_name": "liver_dna",
    "mode": "DNA",
    "input_directory": "/data/pod5"
  }'

# Check status
curl http://localhost:8003/jobs/{run_uuid}/status

# Get results
curl http://localhost:8003/jobs/{run_uuid}

πŸ“š Documentation

Component Documentation

Configuration & Setup

Architecture & Details

πŸ§ͺ Testing

Run All Tests

The project has 1068 tests providing comprehensive coverage.

# Run the full test suite (1068 tests)
pytest tests/ -q

# Cortex tests only
pytest tests/cortex/ -q

# With coverage report
pytest tests/cortex/ --cov=cortex/app --cov-report=term-missing

# Other components
pytest tests/atlas/ tests/common/ tests/analyzer/ tests/launchpad/ tests/ui/ -q

# Single test file
pytest tests/cortex/test_chat_data_calls.py -x -q

# Focused task lifecycle and hierarchy coverage
pytest tests/cortex/test_project_endpoints.py -q

Test Architecture

  • In-memory SQLite with StaticPool for fast, isolated tests
  • Mocked LLM via AgentEngine patches (no real model calls)
  • Mocked MCP via MCPHttpClient patches (no real service connections)
  • 37 cortex test files covering: chat endpoint, approval gates, background tasks, project management, block endpoints, conversations, auth, admin, downloads, uploads, pure helpers, tool routing, skill detection, validation, planning
  • Task coverage includes persistent task projection, task actions, download-file children, and workflow-stage children in tests/cortex/test_project_endpoints.py
  • Shared fixtures in tests/conftest.py for DB engine, sessions, mock users

Bootstrap Existing History

If you deploy this release onto a server that already has projects and stored workflow blocks, run the task bootstrap once after the application starts:

python scripts/cortex/bootstrap_project_tasks.py

The script is idempotent: it reconciles the current workflow state and is safe to re-run.

Run Demo

# Interactive demo for Launchpad
python launchpad/demo_launchpad.py

πŸ› Troubleshooting

Server Won't Start

  • Check port availability: lsof -i :8003 (Launchpad REST), lsof -i :8002 (Launchpad MCP), or lsof -i :8000 (Cortex)
  • Check database connectivity: python -c "from launchpad.db import SessionLocal; SessionLocal()"
  • Check Python version: python --version (requires 3.12+)

Job Stuck in RUNNING

  • Check Nextflow process: ps aux | grep nextflow
  • Check logs: tail -f $AGOUTIC_DATA/launchpad_logs/*.log
  • Cancel job: curl -X POST http://localhost:8003/jobs/{run_uuid}/cancel

Configuration Issues

  • Verify configuration: python -c "from launchpad.config import *; print(f'Code: {AGOUTIC_CODE}')"
  • Check paths: ls -la $AGOUTIC_DATA/launchpad_work

πŸ“Š Performance

GPU Concurrency (Per-Pipeline)

Control the number of simultaneous GPU tasks (dorado basecalling, openChromatin) within a single pipeline run. Configurable in the approval form or via environment variable:

export DEFAULT_MAX_GPU_TASKS=8  # Optional explicit limit. Leave unset for no maximum. Range: 1-16

If DEFAULT_MAX_GPU_TASKS is unset, Launchpad omits Nextflow maxForks for GPU-bound Dogme processes and lets Nextflow manage concurrency. Users can also override per-job in the approval form dropdown or via chat ("limit dorado to 3 concurrent tasks").

Concurrent Jobs

Limit concurrent jobs to avoid resource exhaustion:

export MAX_CONCURRENT_JOBS=2  # Adjust based on server capacity

Database Selection

  • Development: SQLite (default, no setup required β€” create_all() at startup)
  • Production: PostgreSQL with Alembic migrations (alembic upgrade head)
# Set via environment variable
DATABASE_URL = "postgresql://user:pass@localhost/agoutic"

πŸ“ Workflow Skills

Pre-defined bioinformatics workflows are available in skills/:

  • welcome/SKILL.md - New-user onboarding
  • ENCODE_Search/SKILL.md - ENCODE search and data discovery
  • ENCODE_LongRead/SKILL.md - ENCODE consortium workflow
  • run_dogme_dna/SKILL.md - Genomic DNA analysis workflow
  • run_dogme_rna/SKILL.md - Direct RNA-seq workflow
  • run_dogme_cdna/SKILL.md - cDNA isoform workflow
  • analyze_local_sample/SKILL.md - Sample intake and validation
  • analyze_job_results/SKILL.md - Post-pipeline results analysis
  • download_files/SKILL.md - File download orchestration
  • differential_expression/SKILL.md - edgePython DE pipeline (with gene annotation)
  • enrichment_analysis/SKILL.md - GO & pathway enrichment analysis
  • reconcile_bams/SKILL.md - Cross-workflow annotated BAM reconciliation
  • haplotype_with_vcf/SKILL.md - Workflow-aware DNA/RNA/cDNA haplotyping with an indexed VCF
  • remote_execution/SKILL.md - Remote execution workflow
  • shared/SKILL_ROUTING_PATTERN.md and shared/DOGME_QUICK_WORKFLOW_GUIDE.md - Shared reference docs

🀝 Contributing

Code Structure

  • Use type hints throughout
  • Write tests for new features
  • Document configuration changes
  • Update this README for major changes

Testing Requirements

# Run full test suite (1040+ tests)
pytest tests/ -q

# With coverage
pytest tests/ --cov=cortex --cov=launchpad --cov-report=html

πŸ“ž Support

πŸ“¦ Version Information

  • Release: 3.7.6 β€” adds optional separate rsync transfer hosts for remote SLURM profiles, with automatic fallback to the SLURM host when no transfer host is configured
  • Python: 3.12+
  • FastAPI: Latest (from environment.yml)
  • SQLAlchemy: 2.0+
  • Nextflow: >= 23.0
  • Status: Active Development

About

Bioinformatic agent for transcriptomics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages