Python Interface Bridges Gap in Solar-Terrestrial Physics Simulation

Researchers have developed a Python interface for the Code-To-Code Adapter (CoToCoA) library, a framework designed to facilitate direct communication between distinct solar-terrestrial physics simulation codes. Published in Earth, Planets and Space, the technical report details the creation of this interface to enhance the usability of CoToCoA, which previously supported only C and Fortran environments. By integrating Python, the team aims to overcome limitations in applying advanced data analytics and machine-learning techniques to coupled computations in solar-terrestrial physics. The study presents a comprehensive assessment of two development approaches and their performance relative to the native C implementation.

The motivation for this development stems from the prominent role Python plays in scientific computing, data analysis, and the development of artificial intelligence applications. The original CoToCoA library, developed to eliminate the necessity for intermediate file exchanges in coupled computations, was restricted to C and Fortran. This restriction limited its applicability in emerging research areas that demand the advanced data processing capabilities offered by Python. The new interface allows Python programs to function as Workers within the CoToCoA framework, thereby enabling seamless data exchange across programming languages and combining the computational efficiency of C with Python’s analytical tools.

The CoToCoA framework operates through an architecture comprising three distinct program types: the Requester, the Coupler, and the Worker. The Requester and Worker are implemented as multi-process programs, while the Coupler acts as a single-process moderator between them. Data flow within this system is strictly unidirectional, proceeding from the Requester to the Worker, analogous to the transmission of solar wind parameters to a magnetosphere model. The Coupler coordinates interactions by forwarding requests and temporarily storing data in a queue if no Worker is immediately available, thereby reducing the modifications required for integrating existing simulation codes.

To develop the Python interface, the researchers employed two distinct methodologies. The first approach involved a direct implementation of CoToCoA functionality in Python utilizing the mpi4py library, which provides Python bindings for the Message Passing Interface (MPI) standard. The second approach leveraged the ctypes module to enable Python programs to access the original CoToCoA functions implemented in C. Both methods were used to construct a dedicated Python class named CTCAWorker, which enables Python applications to participate as Workers within the framework, managing initialization, request polling, and data reading or writing operations.

Performance evaluations were conducted on a single node of the Laurel 3 supercomputer at Kyoto University to compare the efficiency of the original C interface, the mpi4py-based Python interface, and the ctypes-based Python interface. The measurements focused on two primary data transfer functions: pollreq, which retrieves computation requests from the Coupler, and readarea, which facilitates direct reading from the Requester’s registered memory area. These tests were performed across data sizes ranging from 1 to 16 megabytes, with each test repeated ten times to minimize experimental error and calculate mean execution times.

For the pollreq function, the results indicated that the ctypes-based Python interface demonstrated performance comparable to the native C implementation, particularly as data sizes increased. At smaller data sizes of 1 and 2 megabytes, the ctypes interface achieved the shortest execution times, outperforming both the C and mpi4py interfaces. However, as data volumes grew to 16 megabytes, the C implementation achieved the shortest execution time, with the ctypes interface closely following.

In the evaluation of the readarea function, the C implementation consistently produced the lowest execution times for all tested data sizes, reflecting its native integration with the MPI environment. The ctypes-based Python interface performed competitively, showing better scalability and improved efficiency relative to the mpi4py implementation as data sizes increased. At 16 megabytes, the ctypes interface was faster than the mpi4py interface and closely matched the C implementation, indicating that ctypes offers lower runtime overhead and performance approaching that of the native C code for larger data volumes.

The researchers noted that the measured execution times for the pollreq function include synchronization latency, as the function internally calls MPI_Probe to wait for incoming data. This waiting period can inflate the measured runtime, particularly for the C implementation which may enter the waiting state earlier due to more efficient initialization. Consequently, the observation that Python interfaces achieved shorter latencies at smaller data sizes should not be interpreted as an inherent throughput advantage of Python over C. The architectural design of CoToCoA, specifically the use of non-blocking reception via MPI_Iprobe and a request queue, ensures that these differences in data transfer speed do not significantly affect the overall system operation.

To demonstrate the practical utility of the developed interface, the team applied it to the online visualization of results from a magnetohydrodynamics (MHD) simulation code for planetary magnetospheres. In this application, the MHD simulation code functioned as the Requester, while a Python program served as the Worker, performing post-processing and visualization tasks. The simulation was executed on four compute nodes of the Laurel 3 supercomputer, with the Coupler aggregating data from each process to construct a complete grid dataset for the Python Worker.

Latency analysis of this visualization workflow revealed that the average computation time of the MHD simulation was 1.151 seconds per step, while the average time required for one visualization update was 0.5548 seconds. This indicates that the visualization-side processing time corresponds to approximately 48.2% of the simulation time per step, leaving an average margin of about 0.597 seconds. The simulation produced approximately 0.868 updates per second, whereas the visualization processed approximately 1.802 updates per second, demonstrating that the pipeline can keep up with the simulation without sustained queue growth under the tested conditions.

The study concludes that the Python interface utilizing ctypes effectively bridges the gap between Python’s high-level accessibility and the low-level efficiency of C, especially for large datasets. While the mpi4py interface exhibited greater execution times compared to C, the ctypes-based interface consistently approximated the C interface’s performance across varying data sizes. The framework’s potential for extension to integration with other Python tools, such as machine-learning or artificial intelligence models, is identified as a significant advantage, although such integration is noted as future work rather than an established capability in the present study.

This technical report highlights the importance of interoperability in modern scientific computing, where diverse physical processes operating at various scales require efficient data exchange mechanisms. By providing a robust Python interface for CoToCoA, the researchers have enhanced the framework’s versatility and facilitated seamless data exchange across programming languages. The developed interface allows for the use of advanced analytical tools in simulation analysis, supporting the growing need for integration between high-performance simulation codes and modern data science workflows in solar-terrestrial physics research.

The CoToCoA framework distinguishes itself from other multi-physics coupling tools by prioritizing low-overhead data exchange with minimal modifications to existing simulation codes. While frameworks like preCICE and MOOSE offer sophisticated numerical strategies or monolithic simulation environments, they generally lack native Python interfaces. CoToCoA’s design specifically addresses the need for seamless interoperability among disparate software packages in solar-terrestrial physics, where diverse physical processes at various scales are intrinsically linked. By eliminating the need for intermediate file exchanges, the framework reduces the input/output overhead and data duplication traditionally associated with file-based data transfer methods. This approach is particularly advantageous for coupled computations that require seamless interoperability to model complex phenomena, such as the transmission of solar wind parameters to a magnetosphere model.

The development of the Python interface involved creating a dedicated class named CTCAWorker, which encapsulates several key methods corresponding to functions in the original C or Fortran code. These methods include initialization, which sets up the Worker’s MPI environment and generates commonly used tables, and process control, which manages the workflow within the framework. The class also supports the registration of memory areas, enabling direct access to designated regions of the Requester’s memory. This one-sided communication capability allows the Worker to read or write data without synchronization between the Coupler or Worker and the Requester, further enhancing the efficiency of inter-program data exchange. The initialization routines for the Requester, Coupler, and Worker are executed sequentially to establish the inter-program communication environment, ensuring that each component is properly configured before operations begin.

Process synchronization within the CoToCoA framework is maintained through three principal states: Fin, Idle, and Running. Each process verifies its current state prior to executing any operation, raising an error if the state is not appropriate. The Coupler uses these states to determine the allocation of incoming requests, with Workers alternating between Busy and Idle states. When a request is received, the Coupler identifies the appropriate Worker based on its identifier and forwards the request accordingly. If all Workers are occupied, the Coupler temporarily stores the request in a queue. The Coupler periodically checks this queue and, when a Worker becomes available, retrieves and dispatches the request. This mechanism ensures that the system can handle asynchronous execution and manage the flow of data efficiently, even when multiple Workers are involved in the simulation.

The performance evaluation also highlighted the differences between the pollreq and readarea functions. The pollreq function, which is based on MPI_Recv and MPI_Probe, requires the Worker to wait for requests from the Requester before proceeding, introducing additional latency. In contrast, the readarea function is built on MPI_Get, a component of MPI one-sided communication, which permits direct access to the memory of the Requester and eliminates the need to wait for a request to be sent. This architectural difference explains why readarea consistently achieves lower execution times across all interfaces and data sizes. The additional complexity of pollreq, including probing and request management, contributes to its higher processing overhead, particularly as data sizes increase.

Subject of Research: Development and performance evaluation of a Python interface for the CoToCoA solar-terrestrial physics simulation framework

Article Title: Development of a python interface for a framework of cross-reference solar-terrestrial physics simulation

Article References: Fukazawa, K., Zhao, H., Nanri, T., Miyake, Y., & Katoh, Y. (2026). Development of a python interface for a framework of cross-reference solar-terrestrial physics simulation. Earth, Planets and Space, 78(1), Article 206. https://doi.org/10.1186/s40623-026-02487-8

Image Credits: AI Generated

DOI: 10.1186/s40623-026-02487-8

Keywords: Solar-Terrestrial Physics, Python Interface, CoToCoA, Magnetohydrodynamics, Scientific Computing, Data Transfer, MPI, Coupled Simulations, Development, python, interface, framework