Saturday, November 21, 2026
Science
No Result
View All Result
  • Login
  • HOME
  • SCIENCE NEWS
  • CONTACT US
  • HOME
  • SCIENCE NEWS
  • CONTACT US
No Result
View All Result
Scienmag
No Result
View All Result
Home Science News

Tactile Sensor Data Enhances Robot Point Cloud Completion in 3D Simulations

November 21, 2026
in Science News
Denise Maddox
By Denise Maddox Scienmag Editorial Profile - Mechanical Engineering
Reading Time: 6 mins read
0
Tactile Sensor Data Enhances Robot Point Cloud Completion in 3D Simulations

Tactile Sensor Data Enhances Robot Point Cloud Completion in 3D Simulations

65
SHARES
587
VIEWS
Share on FacebookShare on Twitter
ADVERTISEMENT

Researchers have introduced a deep learning method that combines tactile information with visual point clouds to enhance incomplete three-dimensional shape completion in simulated robotic environments. In computer vision and robotics, obtaining complete spatial models of physical objects is a critical task, especially when clutter, occlusion, or single-view limitations obscure key surface regions. While conventional point cloud completion algorithms rely primarily on visual data, they frequently fail to accurately restore local geometric details due to the lack of descriptive information for missing parts. To address this limitation, the study integrates visual point clouds with contact data gathered through tactile sensing, showing that physical touch imposes strict geometric constraints on object boundaries. Tested on the 3DVT benchmark dataset and simulated robotic arm interactions, the multimodal network achieved measurable performance improvements, effectively recovering fine edges and structural features that purely visual completion methods struggle to reconstruct.

Developing an accurate spatial understanding of three-dimensional environments is fundamental for autonomous robots interacting with surrounding objects. The widespread availability of low-cost lidar, depth cameras, and single-photon 3D imaging has made spatial data increasingly common in robotics. The authors noted that their team previously developed a multi-stage synergistic recovery network to reconstruct accurate depth maps in single-photon 3D imaging. Nevertheless, in cluttered environments, inevitable occlusion between objects makes it difficult to obtain complete 3D spatial data. Missing spatial information complicates downstream tasks such as robotic grasping and path planning. While earlier studies attempted to reconstruct 3D shapes from RGB or RGB-D images, photographs lose substantial 3D information, and their implicit, entangled semantic attributes make reconstructing detailed structures more difficult than completing objects directly from 3D point clouds.

Existing machine learning models routinely downsample and encode raw point coordinates, discarding portions of the original surface structure and requiring specialized mechanisms to mitigate detail loss. For instance, FBNet implements a feedback-driven completion network that reuses output representations to enhance feature learning. SoftPool++ introduces a convolution operator that learns point cloud embeddings via soft-pooling of activations, extracting permutation-invariant properties to preserve fine geometric nuances. Similarly, PoinTr uses a set-to-set Transformer equipped with a local attention mechanism that models neighborhood relationships through k-nearest neighbors. SnowflakeNet stacks multiple snowflake point deconvolution modules to generate missing regions like growing structures, while SeedFormer introduces patch seeds to preserve regional details. Despite these advances, the authors note that encoder compression continues to cause shape degradation, as visual methods lack precise local geometric constraints.

To supply missing local shape constraints, the investigators turned to vision-based tactile sensing, an active area of robotic perception. Tactile sensors directly probe occluded object surfaces, providing physical contact measurements that offset visual blind spots. In automated manufacturing and assembly, robots must accurately perceive workpiece shapes; combining tactile information with incomplete point clouds could enhance reconstruction precision, assembly planning, and production efficiency. Prior research on tactile-assisted 3D reconstruction has largely focused on processing 2D tactile images. For example, Wang and colleagues used neural networks to learn shape priors from single-view images before refining shapes with tactile perception, while Smith and colleagues used graph convolutional networks to project visual cues onto charts deformed by tactile predictions. However, the authors emphasize that inferring 3D shape from 2D tactile images remains difficult compared to using tactile point clouds, which inherently contain 3D spatial structure.

To implement their tactile-assisted framework, the researchers designed an acquisition procedure using a simulated robotic arm with motorized grippers, each fitted with a DIGIT tactile simulator. In simulation, target objects were placed at fixed positions in front of the arm, with the gripper initially positioned above the object. The robotic arm moved and closed its jaws to touch the object’s surface until the DIGIT sensor generated touch and depth images. Upon contact, the arm stopped moving, and the system recorded the depth distribution H, the sensor’s spatial position x, and a unit rotation vector r. To generate the tactile point cloud, the pipeline established a planar point cloud, P_I, at the origin of the robotic coordinate system. Superimposing depth information H onto this plane yielded an intermediate tactile point cloud, P_c. Using the rotation vector r and its modulus length theta, the framework computed an orthogonal rotation matrix R_t and applied spatial transformation to generate the final tactile point cloud P_t.

Integrating tactile points with visual scans requires spatial calibration across reference frames to ensure that both modalities share an identical coordinate system. The methodology assumes that the world and robot coordinate systems overlap, reducing the problem to aligning the object and robot coordinate systems. Theoretically, determining the transformation involves selecting three reference points in the robot frame, recording their positions in the object frame, and solving linear equations to find a rotation matrix R_z and translation vector T_z. To simplify calculations during simulations, the authors assumed that the rotation matrix between the object and robot coordinate systems is an identity matrix, requiring only a translation vector. By placing the object at a fixed position offset of w_0 = (0.5, 0, 0)^T along the x-axis of the robot frame, any point coordinate p_i in the object system was mapped to the robot coordinate p_j simply by adding this translation vector.

Once coordinates were aligned, the framework processed visual and tactile point clouds through distinct neural pathways to extract complementary representations. For the tactile point cloud, the authors engineered a feature extraction module based on U-Net. Tactile features were downsampled twice using max-pooling layers and upsampled twice by deconvolution, with skip connections at each stage to preserve low-level features and support multiscale fusion. In parallel, the incomplete optical point cloud was processed using a Dynamic Graph Convolutional Neural Network, or DGCNN. By constructing local neighbor graphs and applying EdgeConv operations, DGCNN refines local point features, with stacked EdgeConv layers expanding the receptive field to capture complex shapes and semantic information. Instead of processing vision and touch independently, the network concatenated the U-Net tactile features F_t with the DGCNN incomplete point cloud features F_p to form a fused feature representation. Additionally, multilayer perceptrons encoded spatial positions for both point clouds, yielding concatenated positional encodings that enabled comprehensive learning of spatial geometry.

The researchers validated their method on a newly constructed benchmark termed the 3DVT, or 3D Visual and Touch, dataset. Commonly used point cloud datasets such as KITTI, ShapeNet, and PCN lack tactile point clouds and cannot support multimodal evaluation. Building on earlier research by Smith that paired complete surface point clouds with DIGIT sensor data, the team created 3DVT with 12,732 objects. Each object profile contains an 8,192-point ground-truth model and five to eight tactile point clouds from distinct contact locations. The dataset was split into training and testing sets in a 4:1 ratio. Implemented in PyTorch, the network was trained on an NVIDIA 3090 GPU using the AdamW optimizer with an initial learning rate of 0.0005 and weight decay of 0.0005. Evaluation relied on the average L2 Chamfer distance, measuring the bidirectional Euclidean distance between reconstructed and ground-truth point sets.

Experimental evaluations on the 3DVT dataset examined three completion difficulty tiers based on missing data ratios: easy with 25 percent missing, medium with 50 percent missing, and difficult with 75 percent missing. The team compared baseline PoinTr performance without tactile input, direct concatenation of tactile and incomplete point clouds, and their U-Net-based tactile fusion scheme. Results demonstrated that direct concatenation offered only minimal improvements over visual-only completion. In contrast, the U-Net-based tactile feature extraction and fusion module significantly outperformed direct concatenation, achieving the lowest Chamfer distance across all three missing ratios. Visual comparisons confirmed that the U-Net fusion scheme learned fine local details and effectively restored shape structure. Instead of producing scattered local edge points typical of visual-only completion, the tactile-assisted model formed tighter point clusters whose reconstructed boundaries closely matched ground-truth geometry.

To explore broader applications, the authors conducted simulation trials on object categories selected from the ShapeNet dataset, evaluating single-contact and multi-contact setups. The tests revealed that the value of tactile data depended on geometric complexity: while simple, symmetrical objects like spheres and cubes achieved good completion from visual inputs alone, complex shapes like buckets and wine bottles showed substantial reconstruction gains when tactile point clouds were added. Furthermore, increasing the number of tactile point clouds from one to three yielded progressive improvements in reconstruction quality. Visual evaluations of wine bottles, chairs, and guitars confirmed that additional contact samples restored missing bottle tops, reconstructed chair armrests, and reduced scattered boundary points on guitar bodies. To evaluate noise tolerance, researchers introduced controlled noise to the sensors, confirming that completion performance remained robust. However, the authors noted that their study relied on simulation data, acknowledging that physical contact in real-world environments introduces differences that must be addressed in future research.

Subject of Research: Robotic 3D point cloud completion assisted by vision-based tactile sensor feature fusion

Article Title: Enhancing the performance of point cloud completion assisted with tactile information

Article References: Lu, M., Shi, J.-H., Li, J., Shen, H.-R., Wang, H.-Y., Zhang, C., Wang, P.-Z., & Wang, Q. (2024). Enhancing the performance of point cloud completion assisted with tactile information. EURASIP Journal on Image and Video Processing, 2024(1), Article 43. https://doi.org/10.1186/s13640-024-00659-8

Image Credits: AI Generated

DOI: 10.1186/s13640-024-00659-8

Keywords: Point cloud completion, Tactile sensing, 3D reconstruction, U-Net architecture, DGCNN, DIGIT tactile sensor, 3DVT dataset, Chamfer distance, Feature fusion, EdgeConv, Robotic grasping, ShapeNet, Coordinate system alignment, Multimodal perception, Single-photon 3D imaging, Geometric detail restoration, Robotic arm simulation

Cite Scienmag News

Denise Maddox. (November 21, 2026). Tactile Sensor Data Enhances Robot Point Cloud Completion in 3D Simulations. Scienmag. https://scienmag.com/tactile-sensor-data-enhances-robot-point-cloud-completion-in-3d-simulations/

Denise Maddox. "Tactile Sensor Data Enhances Robot Point Cloud Completion in 3D Simulations." Scienmag, 21 November 2026, https://scienmag.com/tactile-sensor-data-enhances-robot-point-cloud-completion-in-3d-simulations/. Accessed 21 November 2026.

Denise Maddox. "Tactile Sensor Data Enhances Robot Point Cloud Completion in 3D Simulations." Scienmag. November 21, 2026. https://scienmag.com/tactile-sensor-data-enhances-robot-point-cloud-completion-in-3d-simulations/

Tags: 3D reconstruction3DVT datasetChamfer distanceCoordinate system alignmentDGCNNDIGIT tactile sensorEdgeConvfeature fusionGeometric detail restorationmultimodal perceptionpoint cloud completionRobotic arm simulationrobotic graspingShapeNetSingle-photon 3D imagingtactile sensingU-Net Architecture
Share26Tweet16
Previous Post

Hybrid Fiber Reinforcement Boosts Seawater Concrete Strength by Thirty Percent

Next Post

Smartphone Mood Ratings and Wrist Actigraphy Improve Bipolar Depression Relapse Detection

Related Posts

Design Science Study Couples Customer Strategy with Business Intelligence in Construction Supplier
Science News

Design Science Study Couples Customer Strategy with Business Intelligence in Construction Supplier

November 21, 2026
BRCA1 Promoter Methylation Shows No Link to Germline Variants in Polish Study
Cancer

BRCA1 Promoter Methylation Shows No Link to Germline Variants in Polish Study

November 21, 2026
East Kolkata Wetland Bacteria Degrade Soil Microplastics by Seventy Percent
Science News

East Kolkata Wetland Bacteria Degrade Soil Microplastics by Seventy Percent

November 21, 2026
Scoping Review Identifies Mechanisms and Contexts Shaping Outcomes in Home Rehabilitation
Medicine

Scoping Review Identifies Mechanisms and Contexts Shaping Outcomes in Home Rehabilitation

November 21, 2026
PSMA PET/CT Reliably Identifies Prostate Cancer Recurrence Before the Phoenix Threshold
Medicine

PSMA PET/CT Reliably Identifies Prostate Cancer Recurrence Before the Phoenix Threshold

November 21, 2026
Machine Learning and Ultrasound Predict ICU-Acquired Weakness Within Twenty-Four Hours
Science News

Machine Learning and Ultrasound Predict ICU-Acquired Weakness Within Twenty-Four Hours

November 21, 2026
Next Post
Smartphone Mood Ratings and Wrist Actigraphy Improve Bipolar Depression Relapse Detection

Smartphone Mood Ratings and Wrist Actigraphy Improve Bipolar Depression Relapse Detection

  • Mothers who receive childcare support from maternal grandparents show more optimized

    Mothers who receive childcare support from maternal grandparents show more parental warmth, finds NTU Singapore study

    27656 shares
    Share 11059 Tweet 6912
  • University of Seville Breaks 120-Year-Old Mystery, Revises a Key Einstein Concept

    1061 shares
    Share 424 Tweet 265
  • Bee body mass, pathogens and local climate influence heat tolerance

    682 shares
    Share 273 Tweet 171
  • Researchers record first-ever images and data of a shark experiencing a boat strike

    546 shares
    Share 218 Tweet 137
  • Groundbreaking Clinical Trial Reveals Lubiprostone Enhances Kidney Function

    531 shares
    Share 212 Tweet 133
Science

Embark on a thrilling journey of discovery with Scienmag.com—your ultimate source for cutting-edge breakthroughs. Immerse yourself in a world where curiosity knows no limits and tomorrow’s possibilities become today’s reality!

RECENT NEWS

  • Design Science Study Couples Customer Strategy with Business Intelligence in Construction Supplier
  • BRCA1 Promoter Methylation Shows No Link to Germline Variants in Polish Study
  • East Kolkata Wetland Bacteria Degrade Soil Microplastics by Seventy Percent
  • Scoping Review Identifies Mechanisms and Contexts Shaping Outcomes in Home Rehabilitation

Categories

  • Agriculture
  • Anthropology
  • Archaeology
  • Athmospheric
  • Biology
  • Biotechnology
  • Blog
  • Bussines
  • Cancer
  • Chemistry
  • Climate
  • Earth Science
  • Editorial Policy
  • Marine
  • Mathematics
  • Medicine
  • Pediatry
  • Policy
  • Psychology & Psychiatry
  • Science Education
  • Science News
  • Social Science
  • Space
  • Technology and Engineering

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 5,151 other subscribers

© 2026 Scienmag - Science Magazine

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • HOME
  • SCIENCE NEWS
  • CONTACT US

© 2026 Scienmag - Science Magazine

Discover more from Science

Subscribe now to keep reading and get access to the full archive.

Continue reading