How a European Orchestrator Learned to Command a US Research Cloud Across the Atlantic

Building a shared computing environment that spans two continents has long been one of the most stubborn obstacles in international science. Research teams on either side of the Atlantic work under different identity systems, security policies, and allocation models, and stitching those together has typically required weeks of manual engineering before a single experiment can run. Now a team at the Instituto de Instrumentación Para Imagen Molecular, a joint center of the Spanish National Research Council and the Universitat Politècnica de València, has shown that the process can be compressed from days to hours. During the DISCOVER-US exchange program, Germán Moltó and colleagues spent April and May 2025 at Argonne National Laboratory integrating the Infrastructure Manager, an open-source cloud orchestrator developed in Europe, with Chameleon, the prominent US research cloud. The result, published in Frontiers in Complex Systems, is a reproducible blueprint for transatlantic computational testbeds that respects the governance rules of both continents without forcing either infrastructure to change.

The Infrastructure Manager, known as IM, is a TOSCA-based orchestrator for the computing continuum, meaning it accepts declarative blueprints written in the OASIS TOSCA standard and turns them into fully configured deployments across virtualized data centers, public clouds, and edge devices. Unlike general-purpose tools such as Terraform or Cloudify, IM was designed with research communities in mind: it supports multi-cloud portability, integrates with the Ansible contextualization engine widely used in scientific workflows, and exposes an API, a command-line interface, and a web dashboard. Chameleon, meanwhile, is a nationally funded, reconfigurable platform built on OpenStack that exposes both virtual machines and bare-metal nodes for computer science at scale. Its emphasis on configurability and experiment repeatability has made it a reference platform for US systems research, but its policies, resource reservation model, and authentication flows differ substantially from those of European e-infrastructures such as the EGI Federated Cloud.

The central technical challenge was identity. European projects typically rely on community-managed Virtual Organizations, while Chameleon foregrounds institution-backed project leadership and allocation accountability. Rather than establishing a trust relationship between the two identity systems, the team adopted a staged credential workflow that preserves the independence of each side. Visiting researchers authenticated through Chameleon’s identity federation, requested Principal Investigator status from the Chameleon operations team, and once approved created a dedicated project with Service Unit quotas and role-based collaborator invitations. Each collaborator then generated OpenStack application credentials restricted to the project scope, with expiration times and revocation capabilities, following the principle of least privilege. These scoped credentials were stored in the IM Vault service and exposed to automation, delivering traceability and a clear mapping between human identities and automated actions. It is the user, not the infrastructure, who bridges both sides by obtaining credentials from each federation and configuring them in IM.

On the engineering side, the team extended IM with a Chameleon-specific provider plugin that encapsulates the Keystone endpoints and region labels exposed by Chameleon sites, such as KVM@TACC, operated by the Texas Advanced Computing Center. The plugin authenticates via application credentials, fetches flavor and image metadata, and maps IM’s node states of Pending, Running, Configuring, and Configured onto Nova’s provisioning lifecycle. Special handling was introduced for resources that require reservation identifiers, such as bare-metal nodes leased through Chameleon’s Blazar service, and IM’s error-handling routines were enhanced to surface Chameleon’s verbose fault messages so researchers can diagnose quota breaches, missing allocations, or hypervisor-side issues. The plugin was decomposed into components for Keystone interactions, Nova and Glance queries, networking, and optional Blazar reservations, with caching mechanisms that minimize redundant API calls and structured logging that captures request identifiers, execution timings, and retry counts. The integration was contributed upstream to the production instance of IM offered in the EGI catalog of services.

To validate the approach, the team deployed five demonstrators of increasing complexity. The baseline was single-node virtual machines with pre-configured scientific Python stacks, which provisioned in an average of four minutes from template submission to SSH availability, matching IM’s performance on European OpenStack sites. Next came a production-ready Galaxy bioinformatics environment, configured through Ansible roles with curated reference datasets, whose pipeline execution times were comparable to European deployments when data resided locally. To mitigate transatlantic transfer overheads, the team prototyped caching strategies using S3 buckets replicated across geographical regions, positioning Galaxy as a shared scientific gateway with IM handling lifecycle management and Chameleon providing elastic compute capacity.

The most striking demonstrator involved serverless computing across the ocean. The team orchestrated two OSCAR clusters, one on the EGI Federated Cloud at the LIP site in Portugal and one on Chameleon at KVM@TACC in Texas, each comprising Kubernetes nodes, an OSCAR control plane, and MinIO storage back-ends. The workload was an AI-based fish detection pipeline from the OBSEA submarine observatory, which ingests underwater imagery, applies model inference, and publishes annotated outputs to shared storage. In independent tests with images of roughly 100 kilobytes, processing took 47 seconds on the US cluster and 74 seconds on the EU cluster, a difference attributed to the superior computational resources of the US site, while event download times were around 120 milliseconds on each cluster because events resided on local MinIO servers. When seven invocations were launched on the EU cluster and three were delegated to the US cluster, execution on the US side averaged 49 seconds, a slight increase caused by the need to download events from the EU MinIO server across the Atlantic. The team concluded that federated serverless platforms can tolerate long-distance links when offloading job executions.

The fourth demonstrator brought digital twin technology into the testbed. The team integrated the DT-Flood workflow from the interTwin project, part of the FloodAdapt Digital Twin by Deltares, which combines three numerical models: WFLOW for river discharge, SFINCS for flood-extent mapping, and RA2CE for assessing disruptions to infrastructure networks. Using Common Workflow Language descriptions, hydrological preprocessing and the WFLOW and SFINCS simulations ran on the European cluster to remain close to regional datasets, while the RA2CE impact-analysis stage was offloaded to the US cluster to exploit additional capacity. A customized Python script invoked by the CWL workflow used the oscar-python library to stage data, trigger model execution, monitor progress, and retrieve outputs across both clusters, enabling seamless distributed execution while maintaining data-locality constraints and increasing resilience.

The final demonstrator was a hybrid SLURM cluster that placed the head node and one worker on the EGI Federated Cloud while scaling additional workers on Chameleon. The TOSCA template described network connectivity, NFS exports, and SLURM daemon registration across both sites, and batch jobs submitted via the front-end node executed seamlessly on resources from both continents. Performance analysis revealed that compute-bound tasks scaled nearly linearly, whereas I/O-heavy workloads suffered throughput degradation because the shared filesystem traversed the transatlantic link, a reminder that physics still constrains distributed computing. Deployment timing measurements also exposed site-level differences: the OSCAR cluster deployed in 12 minutes 7 seconds at KVM@TACC versus 39 minutes 35 seconds at the EGI LIP site, and Galaxy deployed in 13 minutes 37 seconds versus 39 minutes 26 seconds, with analysis attributing the gap to disk performance, as the Chameleon site reported write access three times faster and read access four times faster than the European site.

The work also documented honest operational constraints. Creating reservation leases on behalf of end users still requires manual intervention via the Chameleon helpdesk, delaying access to specialized hardware. Bare-metal deployments do not yet expose resource metadata through the OpenStack flavor catalog, hindering IM’s ability to tailor contextualization scripts automatically. Kubernetes clusters spanning continents struggle with NodePort-based service exposure, causing liveness probes to fail under high latency. Addressing these issues, the authors argue, will require deeper integration with reservation APIs, enhanced discovery mechanisms for bare-metal nodes, and overlay networking solutions better suited to high-latency links. Cultural and organizational challenges surfaced as well, as European teams accustomed to VO-based governance had to align expectations with Chameleon’s PI-led allocation model, demanding clear communication about responsibilities, escalation paths, and acceptable use.

Looking ahead, the team envisions extending the integration to additional US sites, including facilities exposing specialized accelerators or edge devices such as CHI@EDGE, and exploring cost-aware orchestration strategies that consider credit budgets alongside performance and policy constraints. To maximize reuse, the templates, scripts, and best-practice guides have been packaged as a publicly accessible collection, including an entry in Chameleon’s Trovi artifact repository, licensed under permissive terms with metadata describing dependencies and maintainers. The deeper significance of the work lies in its demonstration that research infrastructures from different continents can interoperate without trust relationships between their identity management systems, requiring no changes to the underlying infrastructures. That preservation of independence while enabling seamless cross-cloud operations, the authors suggest, is a key requirement for transnational scientific collaboration, and the wide range of TOSCA-based templates now available shows that a single orchestration framework can support bioinformatics, serverless pipelines, digital twins, and hybrid HPC across the Atlantic, turning what was once a bespoke engineering effort into a durable, shared foundation for joint experimentation.

Subject of Research: Integration of the Infrastructure Manager orchestrator with Chameleon cloud to deploy reproducible transatlantic computational testbeds for scientific workloads.

Article Title: Deployment of transatlantic computational testbeds via the infrastructure manager

Article References: Moltó, G., Caballer, M., Parcero, E., & Rodríguez, V. (2026). Deployment of transatlantic computational testbeds via the infrastructure manager. Frontiers in Complex Systems, 4, Article 1724679. https://doi.org/10.3389/fcpxs.2026.1724679

Image Credits: AI Generated

DOI: 10.3389/fcpxs.2026.1724679

Keywords: cloud computing, cloud orchestration, Chameleon Cloud, EGI Federated Cloud, Infrastructure Manager, TOSCA, serverless computing, OSCAR, Kubernetes, digital twin, SLURM, transatlantic testbeds