A Multi-Agent MCP Server for Cross-Modal Biomedical Embedding Alignment Evaluated on Tumor Microenvironment Characterization

Nasim Sanati, M.S. Cameron Watson, M.S.

Department of Biomedical Engineering, OHSU Knight Cancer Institute

Dr. Allison Creason Lab

Abstract

We propose a focused multi-agent architecture for cross-modal biomedical reasoning, evaluated on co-registered histopathology and spatial transcriptomics data from 92 triple-negative breast cancer patients (Wang et al. 2024, Zenodo). UNI2 and Novae produce strong modality-specific embeddings, but their latent spaces are incompatible, limiting integrated analysis. This project delivers a reusable Model Context Protocol (MCP) server in which modality-specialized agents, orchestrated with LangGraph, align H&E morphology embeddings (1536-d) and spatial transcriptomics embeddings (256-d) into a shared 512-d space using cross-modal contrastive alignment. The aligned space enables richer cohort characterization than either modality alone: integrated morphology–expression profiles across TME compartments, readability maps revealing where modalities reinforce versus diverge, and pathway-morphology spatial maps linking Reactome signals to tissue architecture. Outcomes quantify retrieval (Recall@K, MRR, median rank), separability (alignment gap, AUC), representation alignment (CKA), and coherence versus unaligned baselines. The primary contribution is an open-source agent coordination pattern that turns representation alignment into an experimental tool for tumor microenvironment interpretability.

1. Overview

Scientific Rationale

Biomedical foundation models generate rich embeddings for imaging and molecular modalities but lack standardized mechanisms for cross-modal integration. UNI2 (1) produces 1536-dimensional morphological embeddings from H&E tiles via a ViT-H/14 architecture trained on over 200 million pathology images. Novae (2) produces 256-dimensional expression embeddings from spatial transcriptomics spots via a graph neural network. These representations are powerful within their respective modalities but incompatible: they live in different vector spaces with different dimensionality, training objectives, and semantic content.

Current agentic biomedical platforms such as Biomni (3) demonstrate autonomous analysis across 25 subfields but explicitly identify cross-modal integration as a key limitation. Cross-modal contrastive alignment via InfoNCE (Information Noise-Contrastive Estimation) loss has proven effective in computational pathology, where CONCH (4) learned a shared space between pathology images and clinical text that transferred to 14 downstream tasks. We apply the same contrastive principle to a different and complementary pairing: H&E morphology and spatial gene expression. We apply the same contrastive principle to a clinically grounded pairing. Histology images are routinely used clinically by pathologists and give us information about gross morphology but lack molecular information; whereas spatial transcriptomics is research-use-only and tells us about cell molecular states, trajectories, and high-resolution tissue residency, but is not validated for clinical application. Combining the two allows us to contextualize the more complex and exploratory modality within a more complete biological space, with downstream utility for tumor profiling, cohort stratification, and risk prediction.

The proposed MCP server evaluates whether agent orchestration can function as a reusable coordination layer for cross-modal biomedical reasoning by orchestrating modality-specific embedding extraction, alignment, and evaluation with full provenance tracing. Scope is intentionally targeted to two modality agents and a single alignment workflow; additional components are treated as evaluation modules rather than core dependencies.

Success is defined as reproducible execution of the full workflow with measurable improvement in retrieval and/or representation structure relative to unaligned baselines, independent of downstream biological interpretation outcomes. Each agent is modality-specific by design, carrying dedicated context, identity, and pluggable tooling with reusable orchestration patterns and MCP interface layer, which can be extended to new modality agents.

Agent Innovation

Our primary contribution lies in agent coordination and orchestration rather than development of new foundation models or novel alignment algorithms. Each agent is modality-specific but encoder-pluggable, invoking foundation models as swappable tools (ex, UNI2 vs Virchow2 for H&E); this separation allows evaluation of agent orchestration independent of specific encoder choice. We selected models based on demonstrated modality-specific performance and compatibility with agent-based orchestration workflows rather than novelty of architecture.

The system implements: (i) modality-specialized agents with persistent execution context; (ii) orchestration via LangGraph StateGraph with conditional routing; (iii) Claude Sonnet for intent classification, plan decomposition, and agent routing; (iv) Claude Opus for hypothesis evaluation and synthesis across H1-H3; (v) LangSmith tracing for decision-path inspection across orchestration loops; (vi) MCP interfaces callable by MCP-compatible clients; and (vii) human-in-the-loop escalation at confidence boundaries. Context engineering is treated as a first-class system component: each agent emits structured traces capturing intermediate decisions and state transitions, enabling debugging and iterative calibration.

Evaluation Scenario: Tumor Microenvironment

The tumor microenvironment provides an ideal benchmark because histopathology morphology and spatial gene expression encode complementary biological information, co-registered datasets exist with precise spatial pairing between modalities, and known spatial archetypes provide structured evaluation targets (5). The biological objective provides evaluation context for assessing agent-mediated cross-modal reasoning.

Research Setting

This work is based in the Creason Lab at OHSU Knight Cancer Institute, where we have deployed agent-based infrastructure for cancer data workflows. Biological context is informed by lab experience supporting SMMART clinical trial workflows and tumor microenvironment interpretation; the study itself uses only de-identified public data and focuses on agent-orchestrated cross-modal alignment. Our combined expertise spans AI systems engineering, graph embeddings, biomedical schema harmonization, foundation model integration, computational biology, and spatial transcriptomics analysis. Sanati previously developed gpath2vec (6) for pathway embedding generation and contributed to the Reactome-IDG web portal for the NIH Illuminating the Druggable Genome program (13, 14) in separate work at OHSU. Watson developed and validated COEXIST (17), a machine learning method for integration of spatial omics datasets, contributed to the benchmarking of VAE-based spatial proteomics imputation methods (18), and is a lead developer of Galaxy-ME, an open-source software platform for biomedical image analysis (19). All datasets used will be publicly available and de-identified; no new data collection or wet-lab experiments are required.

Team Role Allocation

Phase	Sanati (Lead)	Watson (Lead)	Shared
Data preparation & infrastructure	MCP server implementation; agent orchestration; embedding extraction pipelines; QC execution; workflow documentation	Biological context validation; review of QC assumptions against TNBC/TME biology	Dataset scope decisions; exclusion reporting
Alignment & evaluation	Contrastive alignment training; evaluation metrics; ablation experiments; retrieval analysis	Biological validation of alignment outputs using MCP tool	Interpretation discussions
Pathway analysis	gpath2vec projection; pathway correlation analysis; sensitivity testing	Validation of pathway-morphology findings against known biological patterns	Result synthesis
Packaging & release	MCP packaging; reproducible notebooks; trace logging; GitHub release	Usability validation from biological perspective	Manuscript preparation; open-science deliverables

* Dr. Allison Creason (PI, Creason Lab): scientific oversight, TME domain guidance, and clinical trial expertise across all phases.

Estimated Timeline

Months 1-2: Data preparation and embedding extraction. Months 3-4: Contrastive alignment training and cross-modal evaluation. Month 5: Pathway projection analysis and ablation studies. Month 6: MCP server packaging, documentation, and open-source release. All work, including manuscript submission, completed within the 6-month grant period.

Data Sharing Commitment

All code released under MIT license on GitHub as a pip-installable MCP server. Alignment checkpoint with model card published on Hugging Face. Reproducible evaluation notebooks and execution traces will be included as output logic. Processed embeddings archived on Zenodo with persistent DOI. Completed manuscript submitted to ResearchHub Journal.

2. Introduction

Research Question

Can a multi-agent architecture enable effective cross-modal reasoning by aligning embeddings from histopathology and spatial transcriptomics into a shared latent space?

Foundation models such as UNI2 (1) for histopathology and Novae (2) for spatial transcriptomics produce high-dimensional embeddings that encode complementary biological information. However, these latent representations lack direct correspondence, limiting integrated analysis. From a systems perspective, cross-modal reasoning becomes an orchestration challenge rather than purely a modeling problem. Agent architectures provide a mechanism to coordinate modality-specific workflows while preserving reproducibility and traceability.

The Model Context Protocol (MCP) provides a framework for building composable AI tools, yet no MCP-based system currently supports cross-modal biomedical data fusion. This work addresses that gap by combining modality-specific embedding agents, cross-modal contrastive alignment into a shared latent space as demonstrated by CONCH (4) for computational pathology, and pathway-level interpretability probes via gpath2vec (6) connected to Reactome biology (7, 8).

Evaluation Context

Triple-negative breast cancer accounts for 15-20% of breast cancers and is among the most heterogeneous subtypes (9, 10). The Wang et al. TNBC dataset (5) provides 92 patients with co-registered Visium spatial transcriptomics and H&E whole-slide images on the same tissue sections (Zenodo, doi:10.5281/zenodo.14204217). The dataset includes 17 expert-annotated morphological categories, 9 spatial archetypes representing distinct TME configurations, TNBC molecular subtypes mapped per spot, and TLS gene signatures scored spatially. These archetypes provide ground truth for evaluating whether aligned representations capture shared biological signal across modalities.

Hypotheses

H1: Cross-Modal Alignment. Contrastive learning will improve cross-modal retrieval and representation alignment relative to unaligned baselines (CCA (Canonical Correlation Analysis) projection, late fusion concatenation, and raw unaligned concatenation), as measured by retrieval metrics (Recall@K, MRR, median rank), alignment separability (alignment gap, AUC), and representation structure (CKA). Based on preliminary experiments and prior literature, cross-modal similarity improvements are expected within an approximate range of 0.65-0.75, though evaluation emphasizes relative improvement over baselines rather than fixed absolute thresholds.

H2: Structural Coherence. The nine spatial archetypes (5) will cluster more coherently in the aligned latent space compared to individual modality spaces, as measured by adjusted Rand index and silhouette score. Alignment quality will vary across TME compartments: morphologically distinct compartments (ex. tumor, TLS, necrosis) will show higher matched-pair cosine similarity than morphologically ambiguous compartments (ex. high-TIL vs. low-TIL stroma).

H3: Pathway Interpretability. Reactome pathway embeddings computed via gpath2vec (6) for five cancer-relevant pathways (TGF-β Signaling, Immune System, Extracellular Matrix Organization, Cell Cycle, Programmed Cell Death) will correlate with specific directions in the shared latent space via canonical correlation analysis, indicating preservation of interpretable biological signal.

Impact Statement

Regardless of outcome, this work produces: a reusable MCP-based agent architecture for cross-modal biomedical reasoning; empirical evidence regarding where agent-mediated alignment succeeds or fails across TME compartments; per-compartment signal-to-noise profiles revealing which biological features transfer between modalities; and open-source tooling for reproducible evaluation workflows. Negative or partial results establish the noise floor and signal limits of current foundation models for agent-mediated cross-modal reasoning.

Biological Deliverables

The aligned latent space produces three concrete outputs beyond the reusable software infrastructure. First, integrated multi-modal cohort characterization: the shared embedding space combines morphological and transcriptomic signals into a joint representation that is more informative for TME characterization than either modality alone, enabling cross-modal retrieval that links tissue architecture to molecular state across patients. Second, tumor microenvironment readability profiles: per-compartment signal-to-noise analysis quantifies where morphology and expression reinforce each other (ex, tumor nests, tertiary lymphoid structures) versus where they diverge (ex, immune gradient stroma), revealing the complementary information structure across modalities. Third, pathway-morphology spatial maps: Reactome pathway embeddings projected into the shared space reveal where specific programs (immune activation, extracellular matrix remodeling) co-localize with tissue architecture, supporting hypothesis generation for tumor microenvironment biology.

3. Methods

Dataset

Wang et al. (2024), 92 TNBC patients with co-registered Visium spatial transcriptomics and H&E whole-slide images on the same tissue sections (doi:10.5281/zenodo.14204217, 58 GB). Estimated >18,000 co-registered spatial spots passing QC across diverse TME compartments. Because all eligible samples in the public cohort are included (n=92), no recruitment is performed. Uncertainty for retrieval and separability metrics is quantified via patient-level bootstrapping (1,000 resamples) to report 95% confidence intervals.

Inclusion: samples with matched H&E and ST data on the same tissue section. Exclusion: tissue with extensive artifacts (pen marks, folds, blur >30% area, flagged by H&E Agent); ST spots with <200 unique molecular identifiers or mitochondrial gene fraction >25%. All exclusions reported with counts.

System Architecture

The MCP server implements four components: a histology modality agent (tool: UNI2 encoder, 1536-d tile embeddings), a spatial transcriptomics modality agent (tool: Novae GNN encoder, 256-d spot embeddings), a contrastive alignment module, and an evaluation module. Each agent maintains persistent execution context and exposes encoder capabilities as structured tool metadata, enabling orchestration logic to reason about available integration strategies.

Novae natively supports multimodal fusion by incorporating H&E foundation model features as node-level attributes during graph construction - an early-fusion approach where modalities are integrated within a single encoder. Rather than committing to a single integration paradigm, the MCP architecture externalizes integration strategy into the orchestration layer, where integration mode becomes a configurable experimental dimension rather than a fixed modeling assumption. The system supports early-fusion representations (Novae native), modular post-encoding contrastive alignment, and baseline alternatives (CCA projection, late fusion, unaligned concatenation). This study evaluates these as discrete experimental conditions; the reusable contribution is an orchestration design that treats integration strategy as an empirical question rather than a fixed pipeline decision.

Execution is coordinated via LangGraph StateGraph with conditional routing informed by analysis goals and system state. Claude Sonnet handles intent classification, plan decomposition, and agent routing. Routing decisions are fully traceable through LangSmith execution logs, and orchestration checkpoints permit reassessment of integration strategy when human-in-the-loop review introduces new hypotheses or alters evaluation priorities. This design balances adaptive decision-making with deterministic execution, ensuring reproducibility and full provenance tracking.

The histology agent supports multi-scale embedding extraction at cell (single tile), niche (local neighborhood), and region (tissue-level) resolutions; this study focuses on cell-scale extraction with multi-scale capability reserved for future extension. System outputs include aligned embedding representations, cross-modal retrieval results, TME readability profiles, and pathway-morphology spatial maps.

Figure 1. Multi-Agent MCP Server Architecture. MCP server exposes modality agents (H&E via UNI2, ST via Novae), an alignment module (InfoNCE contrastive, 512-d shared space), and an evaluation module (Claude Opus for hypothesis evaluation across H1-H3). LangGraph governs routing; LangSmith captures execution traces; HITL review is triggered at confidence thresholds. Outputs include aligned embeddings, cross-modal retrieval results, TME readability profiles, and pathway-morphology spatial maps.

Alignment

Learned projection heads (2-layer MLPs per modality: H&E 1536 to 512, ST 256 to 512, with ReLU and batch normalization) map both modalities into a shared 512-dimensional space using InfoNCE contrastive loss (11), applying the cross-modal contrastive representation alignment approach demonstrated by CONCH (4) for computational pathology, with analogous contrastive alignment validated in unimodal histopathology settings (16). Matched H&E-ST pairs from co-registered tissue locations serve as positive examples; cosine similarity is the alignment metric with temperature τ = 0.07, with loss computed symmetrically and averaged across directions.

Hard negatives are mined within-slide from different tissue compartments to control for slide-level batch effects; cross-patient negatives are excluded from the hard negative pool to avoid confounding biological variation with alignment failure. Training uses batch size 256, max 100 epochs, early stopping on validation cosine similarity (patience=10), with an 85/15 train/test split stratified by patient.

Pathway Projection (Prototype)

A lightweight module links the shared latent space to pathway mechanisms using Reactome (7, 8) and gpath2vec (6), which converts gene-set feature lists into pathway-level embedding vectors encoding both enrichment strength and inter-pathway network topology. Pathway embeddings are correlated with shared latent dimensions via canonical correlation analysis for five target pathways, with spatial maps generated showing pathway-morphology correspondence.

Planned Analyses

H1: Alignment Quality

Cosine similarity distributions on matched versus unmatched test pairs. Bidirectional cross-modal retrieval measured via Recall@{1,5,10}, MRR (Mean Reciprocal Rank), and median rank to quantify both top-K success and overall ranking quality. Alignment separability assessed via alignment gap (mean matched-pair similarity minus mean mismatched-pair similarity) and AUC for classifying matched versus mismatched pairs, with negatives sampled within-slide from different tissue compartments to control for batch effects. CKA (Centered Kernel Alignment) computed between modality embeddings before and after projection into the shared space to assess representation-level structural alignment beyond retrieval. Ablations compare InfoNCE contrastive alignment against CCA projection baseline, late fusion concatenation, and unaligned concatenation; encoder sensitivity is assessed via UNI2 versus Virchow2 (15) as a drop-in replacement under the pluggable tool interface.

H2: Structural Coherence

Adjusted Rand index and silhouette score for nine spatial archetypes across three embedding spaces: H&E-only, ST-only, shared aligned. Per-compartment mean cosine similarity across 17 tissue categories produces a signal-to-noise profile revealing which TME compartments are morphologically readable versus expression-dependent, accompanied by UMAP visualizations.

H3: Pathway Signal

Canonical correlations between gpath2vec pathway embeddings and shared latent dimensions for five target pathways, with spatial maps of pathway-morphology associations.

Contingencies

If alignment performance falls below expected range: increase projection head depth to 3-layer MLP with curriculum learning. If UNI2 shows poor tissue-type separation: benchmark Virchow2 as drop-in replacement via the agent's pluggable encoder design. If spot-level pathway embeddings are too noisy: aggregate to spatial neighborhood level (k=15 neighbors). All contingency results reported alongside primary results.

Ethics and Data Management

Publicly available, de-identified human subjects data from Zenodo. Original collection was approved by the ethics committee of Institut Jules Bordet, Université Libre de Bruxelles (5). No additional IRB approval is required. All code released on GitHub (MIT), checkpoint on Hugging Face, embeddings on Zenodo with DOI.

4. Pilot Data

No pilot data from the proposed TNBC dataset has been generated. The evidence below demonstrates infrastructure readiness and technical feasibility rather than preliminary results from the main study.

We have deployed a production ready agentic MCP server at OHSU Knight Cancer Institute using LangChain orchestration with conditional routing, HITL review, and per-agent execution tracing, where multi-component embedding fusion achieved 70-75% accuracy versus 25% for the best single component on an unrelated biomedical schema harmonization task (12). We also have validated operational UNI2, Virchow2, and Novae GPU inference pipelines on internal datasets, confirming that the proposed encoder stack runs end-to-end. Preliminary contrastive alignment on pancreatic cancer H&E-ST pairs (separate dataset, separate cancer type) produced cosine similarity >0.70 on matched regions, providing a rough feasibility signal for the alignment approach but not a direct estimate of performance on the TNBC cohort.

Sanati developed gpath2vec (6) for pathway embedding generation and contributed to the Reactome-IDG web portal for the NIH Illuminating the Druggable Genome program (13, 14) in prior work at OHSU, providing direct domain expertise in pathway biology and embedding-based analysis. We both contribute computational biology expertise, spatial transcriptomics workflows, and tumor microenvironment characterization.

5. Budget

Budget will be allocated toward LLM API tokens for agent orchestration and reasoning workflows, cloud GPU compute costs for embedding extraction, alignment training, and evaluation analysis, and minor storage and processing overhead.

Budget Breakdown (Estimated)

Category	Description	Est. Allocation
LLM API usage (~85%)	Estimated token usage based on iterative agent runs and evaluation cycles	~$850
GPU cloud compute (~10%)	Embedding extraction, alignment training, evaluation runs leveraging institutional GPU access where available	~$100
Storage & data processing (~3%)	Temporary storage, intermediate artifacts, processing overhead	~$30
Contingency (~2%)	Unexpected compute or iteration costs	~$20
Total requested	Minimum viable budget	$1,000

Budget note (minimum scope + optional expansion). This budget reflects the minimum allocation required to complete the proposed work under existing OHSU support and available institutional compute resources. No personal funds are allocated. If additional funding becomes available upon approval, we would apply it directly toward expanding compute and evaluation coverage, for example, additional ablation runs (more encoder comparisons and multi-scale extraction), extended cross-patient sensitivity analyses (including negative-mining variants), and additional HITL evaluation cycles to strengthen reproducibility and robustness, without changing the core technical scope.

6. References

Chen, R.J. et al. Towards a general-purpose foundation model for computational pathology. Nat. Med. 30, 850–862 (2024).
Conan-Guez, B. et al. Novae: a graph neural network framework for spatial transcriptomics. bioRxiv (2024).
Huang, K. et al. Biomni: a comprehensive biomedical AI agent with 150 tools across 25 subfields. Nature (2025).
Lu, M.Y. et al. A visual-language foundation model for computational pathology. Nat. Med. 30, 863–874 (2024).
Wang, X. et al. Spatial transcriptomics reveals substantial heterogeneity in triple-negative breast cancer with potential clinical implications. Nat. Commun. 15, 10232 (2024).
Sanati, N. gpath2vec: Gene-set to biological pathway embeddings with enrichment analysis attributes. GitHub, https://github.com/teslajoy/gpath2vec (2024).
Jassal, B. et al. The Reactome pathway knowledgebase. Nucleic Acids Res. 48, D498–D503 (2020).
Gillespie, M. et al. The Reactome pathway knowledgebase 2022. Nucleic Acids Res. 50, D419–D426 (2022).
Denkert, C. et al. Molecular alterations in triple-negative breast cancer. Lancet 389, 2430–2442 (2017).
Bianchini, G. et al. Treatment landscape of triple-negative breast cancer. Nat. Rev. Clin. Oncol. 19, 91–113 (2022).
Oord, A.v.d. et al. Representation learning with contrastive predictive coding. arXiv:1807.03748 (2018).
Sanati, N. Agentic AI for biomedical schema harmonization via multi-component embedding fusion. OHSU Knight Cancer Institute Technical Report (2025).
Nguyen, D.-T. et al. Pharos: Collating protein information to shed light on the druggable genome. Nucleic Acids Res. 45, D483–D491 (2017).
Beavers, D., Brunson, T., Sanati, N. et al. Illuminate the functions of dark proteins using the Reactome-IDG web portal. Curr. Protoc. 3, e845 (2023).
Vorontsov, E. et al. Virchow2: Scaling self-supervised mixed magnification models in pathology. arXiv:2408.00738 (2024).
Lazard, T. et al. Contrastive representation alignment for unimodal histopathology. arXiv:2508.05084 (2025).
Watson, C.W. et al. COEXIST: Coordinated single-cell integration of serial multiplexed tissue images. PLoS Comput. Biol. (2025). doi:10.1371/journal.pcbi.1013325
Watson, C.W. et al. Imputing single-cell protein abundance in multiplex tissue imaging. Nat. Commun. 16, 4685 (2025). doi:10.1038/s41467-025-59788-x
Creason, A., Watson, C.W., Gu, Q. et al. A web-based software resource for interactive analysis of multiplex tissue imaging datasets. bioRxiv (2022). doi:10.1101/2022.08.18.504436

A Multi-Agent MCP Server for Cross-Modal Biomedical Embedding Alignment Evaluated on Tumor Microenvironment Characterization

Funding Progress

Nonprofit Recipient

Topics

DOI

Funding Progress

Nonprofit Recipient

Topics

DOI

A Multi-Agent MCP Server for Cross-Modal Biomedical Embedding Alignment Evaluated on Tumor Microenvironment Characterization

Funding Progress

Nonprofit Recipient

Topics

DOI