The research team, led by Professor Masayuki Tanaka and Professor Masatoshi Okutomi from the Department of Systems and Control Engineering at Science Tokyo, addressed one of the most persistent failure modes in multi-object tracking: the identity switch. When a camera loses sight of a person because of occlusion by another person, an object, or any other obstruction, the tracking pipeline often treats the reappearing individual as a brand-new subject and assigns a fresh identifier. Multiply this across several cameras and the result is a fragmented record in which the same person appears as multiple phantom individuals. Maintaining a consistent identity across camera views therefore remains a central challenge for anyone building systems that must follow people through real spaces, from security operators to transportation planners.

The key insight behind the new method is that two complementary clues can be combined to solve the association problem. The first clue is geometric. When two cameras observe the same scene from different positions, the mathematical relationship between their views is captured by what computer vision researchers call epipolar geometry. This geometry constrains where an object seen in one image can possibly appear in the other: the candidate location must lie along a specific line, known as the epipolar line, determined by the relative positions and orientations of the two cameras. The researchers exploit this constraint by calculating an epipolar distance for each potential match, a measure of how far a candidate deviates from the geometrically consistent region. Candidates that fall too far from the expected line can be eliminated outright, dramatically shrinking the pool of possible matches before any visual comparison is attempted.

The second clue is appearance. Once geometry has narrowed the field, the system compares visual features extracted from the images of the remaining candidates. These features, drawn from pre-trained models, encode what a person looks like, their clothing, build, and other visible characteristics, allowing the system to distinguish between multiple people who all happen to lie along the same epipolar line. The order of operations matters. By using epipolar geometry to verify spatial consistency first and only then applying appearance-based matching, the system combines the strengths of both signals: geometry rules out physically impossible pairings, while appearance resolves the remaining ambiguity. As Tanaka explains, combining epipolar geometry with appearance similarity allows the geometric relationship between cameras to constrain possible matches, after which visual information distinguishes between them.

A particularly attractive feature of the approach is that it does not require building a new tracking system from scratch. Instead, it is designed to plug into existing single-camera tracking pipelines. Each camera independently detects and tracks people, producing short sequences of detections called tracklets. These tracklets are often fragmented, broken whenever a person is temporarily lost from view. The proposed method then performs cross-camera association, deciding which tracklet fragments from different cameras most likely belong to the same individual. Because the association step relies on the known geometric relationships between cameras together with pre-trained appearance features, it does not require additional training for each new environment. Tanaka notes that this means cross-camera track association can be deployed without environment-specific retraining, a significant practical advantage over approaches that must be tuned to the particular layout and lighting of every installation.

To evaluate the method rigorously, the team tested it on two established multi-camera tracking benchmarks: MMPTrack and CAMPUS. Both datasets contain synchronized video from multiple camera views and are specifically designed to measure how well tracking systems preserve person identities over time. The primary metric for identity consistency is IDF1, which scores how faithfully a system maintains the correct identity of each person across frames. The researchers also reported results on Higher Order Tracking Accuracy, or HOTA, a metric that jointly assesses two distinct capabilities: how accurately people are detected in the first place, and how consistently their identities are tracked once detected. Reporting both metrics matters because a system can excel at one while failing at the other, and real-world deployments need both to succeed simultaneously.

The results showed clear gains on identity preservation. On MMPTrack, the proposed method achieved an average IDF1 score of 65.48, compared with 62.30 for MCTR, an existing multi-camera tracking method. On HOTA, the new approach scored 56.92, essentially matching the 55.77 achieved by MCTR, indicating that the improvement came specifically from better identity association rather than from changes in detection behavior. On the CAMPUS benchmark, the method reached an average IDF1 of 47.37, outperforming ByteTrack, a well-known single-camera tracking method, which scored 44.72. Taken together, the numbers suggest that adding geometric and appearance-based cross-camera association on top of standard single-camera trackers yields measurable improvements in exactly the area where multi-camera systems struggle most: keeping identities stable across views.

The evaluation also surfaced an honest limitation that the researchers themselves highlight. Severe occlusion can cause people to be missed entirely during detection, meaning no tracklet is generated for the association stage to work with. If a person is never detected in one of the camera views, no amount of clever matching can link them across cameras, because there is simply nothing to link. This observation underscores an important structural point about multi-camera tracking: reliable performance depends not only on accurately matching observations across views but also on consistently detecting people in the first place. Detection and association are chained together, and a weakness in the first link caps the performance of the second, no matter how sophisticated the matching algorithm becomes.

Looking forward, the researchers see two natural directions for extending the work. The first is improving person detection itself, particularly under the severe occlusion conditions that currently cause missed detections. The second is refining cross-camera track association so that it remains robust in increasingly crowded and complex environments, where many people move through overlapping fields of view and occlusions are frequent rather than exceptional. Progress on both fronts could extend the approach to settings that are far more challenging than current benchmarks, such as dense pedestrian zones, transit hubs during peak hours, or large public events where hundreds of people cross camera boundaries every minute.

The potential applications extend well beyond the laboratory. Any system that must maintain consistent identities across multiple viewpoints stands to benefit, including security monitoring, transportation management, facility operations, and pedestrian-flow analysis. In security contexts, stable identities mean that a person of interest remains a single coherent record as they move between cameras, rather than dissolving into a confusing set of fragments. In transportation and facility management, accurate pedestrian-flow statistics depend on counting each person once, not several times over, which requires precisely the kind of cross-camera identity consistency this method provides. By combining the physical rigor of epipolar geometry with the discriminative power of modern appearance features, and by doing so in a way that integrates with existing tracking infrastructure without retraining, the Science Tokyo team has offered the field a template for multi-camera tracking that is both technically principled and practically deployable.

Subject of Research: Multi-camera multi-object tracking using epipolar distance and appearance similarity

Article Title: Improving identity-tracking across multiple cameras with geometry and appearance

Article References: Improving identity-tracking across multiple cameras with geometry and appearance. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: multi-camera tracking, computer vision, epipolar geometry, appearance similarity, identity switches, tracklet association, IDF1, HOTA, person re-identification, occlusion, surveillance, Institute of Science Tokyo

The study was led by Professor Rachel Kolodny and PhD candidate Guy Yanai of the University of Haifa, together with Professor Nir Ben-Tal and graduate student Gabriel Axel of Tel Aviv University, and Specially Appointed Associate Professor Liam M. Longo of ELSI. Kolodny also spent five months as a visiting researcher at ELSI, developing methods to analyze the new model. Their creation, dubbed CLSS for Contrastive Learning Sequence-Structure, is a protein language model designed to overcome a stubborn problem that has limited previous computational approaches: the awkward relationship between what a protein’s sequence says and what its structure actually does.

Scientists have long organized proteins into hierarchical groups based on relatedness, much like the genus and species categories biologists use to classify organisms. These curated systems, such as the widely used ECOD and CATH databases, distill decades of expert knowledge. But with artificial intelligence now capable of generating ’embeddings’—numerical representations in which proteins with similar properties receive nearby coordinates, like postal codes on a map—researchers can visualize relationships across millions of proteins at once, producing what the team calls a protein world map. The catch is that sequence and structure do not map neatly onto each other. Unrelated sequences can fold into similar shapes, while even identical sequences can sometimes adopt wildly different structures.

Most existing protein language models treat sequence and structure as separate worlds, processing one or the other independently. Even hybrid models that incorporate both kinds of data rarely place the sequence and the structure of the same protein at the same location on a global map, leaving researchers with two conflicting atlases of protein space. CLSS was engineered specifically to resolve this discordance. Using a machine learning strategy known as contrastive learning, the model is trained on pairs of protein sequences and their corresponding structures, learning to pull matching sequence-structure pairs together in the embedding space while pushing unrelated pairs apart.

The result is a single shared map in which a protein occupies essentially the same location whether the model is given its sequence or its structure. When benchmarked against other state-of-the-art protein language models, CLSS succeeded in producing a cohesive unified representation, something its predecessors could not achieve. Remarkably, the model’s maps closely reproduced the relationships recorded in the expert-curated ECOD and CATH classification systems, even though those classifications were never shown to the model during training. In direct classification tests, CLSS also performed strongly, demonstrating that merging sequence and structure information yields genuinely more informative protein representations.

Perhaps the most exciting feature of CLSS is its ability to handle fragments. Most protein language models require a complete sequence or structure to generate a meaningful embedding, but CLSS showed that short sequence fragments can in many cases be positioned meaningfully alongside full-length proteins and structures. This capability matters enormously for evolutionary studies, because small pieces of proteins have been repeatedly reused and rearranged throughout the history of life. Some fragments may even have served as the primordial building blocks from which the earliest protein domains were assembled, meaning that similar fragments appearing in otherwise unrelated proteins can hint at ancient evolutionary connections.

The maps produced by CLSS also revealed sweeping patterns across protein space that were previously difficult to see. When the researchers overlaid biological properties onto the maps, proteins associated with organic cofactors turned out to cluster in particular regions, while metal-binding proteins were scattered more broadly. Such patterns illustrate how global protein maps can serve not only as classification tools but as instruments for exploring the interplay between sequence, structure, function, and deep evolutionary history, potentially exposing large-scale patterns invisible to conventional pairwise comparison methods.

‘This gives us a way to look at the protein universe through sequence and structure at the same time, rather than treating them as separate worlds,’ said Longo. ‘What is particularly exciting for us is the possibility of using these maps to uncover large-scale evolutionary patterns that are difficult to recognise using conventional approaches.’ The team ultimately envisions unified sequence-structure representations opening new frontiers in database searches, protein engineering, and the reconstruction of evolutionary trajectories—offering a fresh window onto how the staggering diversity of proteins found in life today emerged over nearly four billion years of evolution.

Subject of Research: A contrastive-learning protein language model that unifies protein sequence and structure representations to map the protein universe

Article Title: Uniting sequence and structure to map the protein universe

Article References: Uniting sequence and structure to map the protein universe. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: protein language model, CLSS, protein evolution, contrastive learning, protein structure, amino acid sequence, ECOD, CATH, protein embeddings, evolutionary biochemistry, protein classification, Institute of Science Tokyo

In a groundbreaking development, scientists from the Institute of Science Tokyo have achieved a significant milestone in enhancing the applicability of the wurtzite crystal structure by expanding its scope to include heterovalent ternary nitrides. Their research, published online in Advanced Electronic Materials, highlights the fabrication of magnesium silicon nitride (MgSiN₂) as the first-ever heterovalent nitride with a wurtzite structure that exhibits promising piezoelectric properties. This innovative study was spearheaded by Professor Hiroshi Funakubo, alongside a team that includes researchers from various esteemed institutions, most notably Pennsylvania State University and Tohoku University.

Traditionally, wurtzite-structured crystals have been predominantly composed of trivalent cations. However, this has presented challenges due to high coercive electric fields that inhibit polarization switching necessary for effective piezoelectric charge generation. By incorporating heterovalent cations with different valencies, the researchers discovered a means to adjust the structural rigidity of these materials, subsequently facilitating polarization and reducing the coercive field. This foundational exploration into the effects of heterovalent doping in crystal structures marks a pivotal shift in the understanding of how these materials can be optimized.

The synthesis of MgSiN₂ was particularly innovative. This compound typically crystallizes in a different configuration, specifically the orthorhombic β-NaFeO₂ structure. The research team successfully managed to stabilize MgSiN₂ in the desired wurtzite phase through a highly controlled process known as reactive RF magnetron sputtering of magnesium and silicon ions at a consistent temperature of 600 °C within a nitrogen-rich environment. The process not only altered the crystal structure but also introduced a form of random cationic ordering which contributes to the material’s unique properties.

Collectively, this endeavor represents a major leap forward in the domain of piezoelectric materials. As highlighted by Professor Funakubo, the successful realization of MgSiN₂ in a wurtzite structure could catalyze the development of high-performance materials tailored for specific electronic applications. The implications of this research extend into numerous fields, including sensors, actuators, and energy harvesting systems, where materials that can effectively convert mechanical energy into electrical energy are crucial.

Advanced characterization techniques were employed to confirm the piezoelectric characteristics of the newly synthesized wurtzite-MgSiN₂ structure. Techniques such as X-ray diffraction, transmission electron microscopy, and piezoresponse force microscopy were instrumental in revealing the material’s properties. The results indicated a converse piezoelectric coefficient of approximately 2.3 pm/V, which aligns closely with the metrics observed in traditional simple nitrides. The significance of this finding lies in the material’s ability to effectively translate mechanical stress into electrical charge, thus enhancing its viability for diverse applications.

In addition to its piezoelectric attributes, the MgSiN₂ compound demonstrated an impressive wide bandgap of about 5.9 eV for direct transitions and 5.1 eV for indirect transitions. Such a wide bandgap is akin to that of established piezoelectric materials like wurtzite AlN, suggesting that MgSiN₂ possesses robust insulating properties. The ability to restrict electron mobility between the valence and conduction bands is indicative of a material that not only exhibits durability but is also stable under varying environmental conditions, further reinforcing its potential as a candidate for next-generation electronic devices.

The research team’s future trajectory remains focused on delving deeper into the realm of heterovalent ternary nitrides with piezoelectric and ferroelectric properties. They aim to refine the deposition parameters used during the synthesis process to uncover even greater improvements in polarization switching. This continued exploration will not only serve to validate the initial findings regarding the ferroelectric behavior of MgSiN₂ but will also enhance the understanding of how structural variations can influence functional properties in similar materials.

Overall, the emergence of wurtzite-structured MgSiN₂ represents an exciting frontier in material science, especially regarding piezoelectric and ferroelectric applications. As researchers delve deeper into novel material phases, the prospects for advancing electronic technologies become increasingly promising. The work carried out by the Institute of Science Tokyo not only demonstrates the profound potential of heterovalent doping strategies in improving material properties but also sets a new benchmark for future research endeavors aimed at harnessing the power of novel materials for technological advancement.

As a result, the integration of MgSiN₂ within the electronics landscape could lead to the development of innovative devices with higher efficiencies and tailored functionalities. This work exemplifies the critical interplay between material science and practical engineering, showcasing how fundamental research can pave the way for revolutionary advancements in technology. Whether it’s enhancing the capabilities of existing applications or crafting new technologies altogether, the implications of this research will undoubtedly resonate across several fields and industries.

In conclusion, the future looks bright for the promising applications of wurtzite-structured materials like MgSiN₂. Continued investigation into their properties and potential applications holds the key to unlocking the next generation of piezoelectric and ferroelectric technologies. The efforts of the researchers at the Institute of Science Tokyo lay a solid foundation for ongoing advancements, ushering in an era of enhanced performance and innovation in electronic materials.

Subject of Research: Heterovalent ternary nitrides with piezoelectric properties
Article Title: Realization of Non-Equilibrium Wurtzite Structure in Heterovalent Ternary MgSiN2 Film Grown by Reactive Sputtering
News Publication Date: 6-Feb-2025
Web References: https://advanced.onlinelibrary.wiley.com/doi/10.1002/aelm.202400880
References: 10.1002/aelm.202400880
Image Credits: Institute of Science Tokyo

Keywords

Wurtzite structure, piezoelectric materials, heterovalent nitrides, magnesium silicon nitride, advanced materials, electronic properties, material science, structural properties, semiconductor technology, energy harvesting.