The appeal of 3D data is easy to grasp. A photograph flattens the world onto a grid of pixels, discarding the very depth information that lets humans judge distance, volume and shape. Three-dimensional data, by contrast, carries precise spatial measurements, allowing an algorithm to know not just that a pedestrian is present but exactly how far away they stand and how much space they occupy. The survey identifies five dominant ways of representing this information: point clouds, voxel grids, depth and normal maps, polygonal meshes and neural fields. Each comes with trade-offs in computational cost, storage demands and expressive power, and choosing the right representation is often the first critical decision in any 3D pipeline.

Point clouds deserve particular attention because they dominate real-world sensing. Produced by LiDAR scanners and depth cameras, a point cloud is simply a set of points floating in space, each carrying a 3D coordinate and sometimes color or intensity. The trouble is that these points are irregular and unordered, which makes them fundamentally incompatible with the convolutional neural networks that revolutionized 2D image analysis. The field’s answer is sparse convolution, a clever trick that performs computation only on non-empty regions of a voxelized space. Because most of a scanned scene is empty air, sparse convolution slashes the computational burden dramatically. Its cousin, sub-manifold sparse convolution, goes further by restricting computation to kernel centers that already contain data, preventing the dilation that would otherwise destroy the input’s sparsity, though the dilation property itself can be useful for giving models contextual awareness of their surroundings.

With these foundations in place, the survey turns to the heart of modern 3D research: pre-training. The idea borrows directly from the playbook that made 2D vision so successful. Models pre-trained on ImageNet’s fourteen million labeled images learned general-purpose features that transferred effortlessly to new tasks. But 3D data lacks an equivalent of ImageNet, largely because annotating millions of 3D scans is prohibitively expensive. The solution has been self-supervised learning, in which models generate their own training signals from raw data. The survey organizes these efforts into three families: contrastive methods, masked autoencoder approaches and rendering-based techniques.

Contrastive learning teaches a network to pull similar data points together in an embedding space while pushing dissimilar ones apart, typically through an InfoNCE loss. In the 3D world this takes several forms. PointContrast, a landmark 3D-to-3D method, encourages networks to learn representations that remain stable under different viewpoints and noise, directly boosting downstream detection and segmentation. Other approaches reach across modalities: CrossPoint and SLidR bridge point clouds and rendered images, transferring knowledge from 2D backbones trained on vast image collections into 3D networks. Perhaps most strikingly, methods like PointCLIP and its successor PointCLIPV2 connect 3D perception to language, projecting point clouds into multi-view images and aligning them with the text embeddings of CLIP-style models. This enables zero-shot recognition, where a model classifies objects it has never been trained on simply by matching visual features to textual descriptions.

Masked autoencoder methods take a different route, inspired by the masked image modeling that powered recent breakthroughs in 2D vision. The core idea is to corrupt or hide parts of the input and force the model to reconstruct them, thereby learning the underlying structure of 3D shapes. VoxelMAE converts unstructured points into voxel grids and predicts the occupancy of masked voxels. PointMAE, borrowing a page from BERT, tokenizes point clouds using farthest point sampling and K-nearest-neighbor grouping, then trains with an L2 loss on masked token features; its descendant PointM2AE adds pyramid architectures that capture both fine geometric detail and high-level semantics, achieving state-of-the-art linear classification on the ModelNet40 benchmark. GD-MAE extends the paradigm to convolutional backbones favored in autonomous driving, with a masking strategy designed to prevent knowledge leakage during downsampling.

The third family is the most visually intuitive: rendering-based pre-training. Differentiable rendering makes the process of drawing an image from a 3D model mathematically transparent, so gradients can flow backward from the rendered image to the 3D representation itself. The Ponder framework describes 3D surfaces within a volume and optimizes them against 2D image projections using a NeRF-like structure, while PonderV2 scales the idea to outdoor scenes and can pre-train both 2D and 3D backbones. According to the survey, this rendering-based approach outperforms both contrastive and masked-autoencoder methods, suggesting that grounding 3D learning in the physics of image formation may be the field’s most promising direction.

These pre-trained representations then feed a rich ecosystem of downstream tasks. Classification, the field’s founding problem, began with PointNet’s pioneering use of shared multi-layer perceptrons on raw points and matured through architectures like DGCNN’s EdgeConv, which builds local graphs to learn edge embeddings. Segmentation assigns a category to every point in a scan and has spawned a fierce architectural competition: MinkowskiUNet applies U-Net-style designs to voxel data, Cylinder3D replaces cubic partitions with cylindrical ones to handle LiDAR’s varying point density, SphereFormer exploits the transformer’s global receptive field, and RangeFormer brings powerful 2D-style transformers to range-view representations. Detection, crucial for autonomous vehicles, spans LiDAR-based methods like PointRCNN and PVRCNN, camera-based systems like BEVFormer that follow Tesla’s vision-centric pipeline, and fusion approaches like PointPainting and LoGoNet that marry the precise geometry of LiDAR with the rich texture of images. Tracking, matching and registration round out the toolkit, with methods like GeoTransformer and robust variants of the classic ICP algorithm enabling machines to align separate 3D scans into coherent wholes.

Progress on all these fronts is measured against a demanding set of benchmarks. ModelNet40 and ShapeNet test synthetic shape understanding, ScanNetV2 and S3DIS probe indoor scene parsing, while KITTI, nuScenes, Waymo and ONCE push algorithms to their limits in real-world driving scenarios containing millions of annotated frames. Evaluation relies on metrics such as mean average precision, computed from intersection-over-union thresholds between predicted and ground-truth boxes, and mean intersection-over-union for segmentation, with registration tasks judged by rotation error, translation error and registration recall. The survey’s careful cataloging of these standards provides newcomers with an immediate map of how the field keeps score.

Looking forward, the authors identify several frontiers that could define the next decade. Large language models, brimming with world knowledge distilled from vast text corpora, are only beginning to be connected to 3D perception through systems like Uni3D-LLM. Knowledge transfer from the data-rich 2D and language domains into the data-starved 3D world remains a wide-open opportunity. Synthetic data could relieve the crushing cost of manual annotation, though bridging the gap between simulated and real scenes will demand advances in domain adaptation and neural rendering. Most ambitiously, the field awaits its own foundation model: while 2D vision has been reshaped by systems like Segment Anything, no equivalent exists for 3D, largely because large-scale 3D benchmarks are still lacking. Building one, the survey argues, could dramatically cut the design and deployment costs of countless 3D applications and pave the way toward a genuine era of 3D artificial general intelligence. For a field that measures the world in three dimensions, the trajectory is unmistakably upward.

Subject of Research: Self-supervised pre-training methods and downstream tasks for 3D point cloud perception

Article Title: Advances in 3D pre-training and downstream tasks: a survey

Article References: Hou, Y., Huang, X., Tang, S., He, T., & Ouyang, W. (2024). Advances in 3D pre-training and downstream tasks: a survey. Vicinagearth, 1(1), Article 6. https://doi.org/10.1007/s44336-024-00007-4

Image Credits: AI Generated

DOI: 10.1007/s44336-024-00007-4

Keywords: 3D perception, point clouds, self-supervised learning, pre-training, masked autoencoders, contrastive learning, sparse convolution, LiDAR, autonomous driving, neural rendering, foundation models, benchmarks

The first half of the innovation lies in the backbone architecture, which the authors call a 3D large sparse multi-directional convolution. To understand why this matters, it helps to recall how modern LiDAR networks process point clouds. Raw LiDAR data is irregular: each laser return is a point floating in space, with no grid structure that a conventional convolutional neural network can slide over. The standard solution is voxelization—dividing space into small cubic cells and treating occupied cells as sparse entries in a three-dimensional tensor. Sparse convolutions then operate only on the occupied cells, which keeps computation tractable even when a single scan contains well over a hundred thousand points.

The catch is that ordinary sparse convolutions use small kernels, typically limited to a few neighboring voxels. That gives each layer a narrow receptive field, so the network must stack many layers before any single voxel can “see” enough surrounding context to decide whether it belongs to a curb, a wall, or a distant vehicle. The Shenzhen team attacks this bottleneck with dynamic large sparse kernels, which allow the convolutional footprint to grow dramatically while still operating only on the sparse set of occupied voxels. Because the kernel is dynamic, its shape can adapt to the local geometry of the scene rather than remaining fixed, letting the network concentrate its receptive field along structures such as elongated roads or the flat planes of building facades.

Large kernels alone, however, can still miss context that does not align with the kernel’s orientation. The second ingredient of the backbone, multi-directional convolution, addresses this by sweeping the receptive field along several directions at once. The combination creates what the authors describe as large effective receptive fields: regions of the scene over which a single computational unit can integrate evidence. In practice, this means that a voxel at the edge of a scan can draw on information from far across the street, which is precisely the kind of long-range context needed to resolve ambiguous geometry in outdoor driving scenes.

Before any of this convolution happens, the raw points must be organized, and here the framework introduces a further refinement called radially non-uniform cylindrical voxelization. Standard cubic voxelization treats all regions of space equally, but LiDAR sensors do not sample the world equally. Points near the sensor are dense, while points at the edge of the sensor’s range are spread thinly, so a uniform grid leaves near-field cells crowded and far-field cells nearly empty. Cylindrical voxelization, organized around the sensor’s axis, naturally matches the geometry of the scan, and making the radial divisions non-uniform—finer near the sensor, coarser farther out—balances the number of points per voxel across the entire scene. The result is a more even representation that prevents the network from being dominated by nearby structures and helps it learn features that transfer across distances.

The second half of the framework is where the approach becomes genuinely distinctive. During training, the 3D backbone is accompanied by two auxiliary streams: one that processes 2D RGB images, and one that processes the point cloud after it has been serialized into a 1D sequence using a space-filling curve. Space-filling curves, such as the curves that wind through a volume while visiting neighboring cells consecutively, flatten a 3D structure into an ordered sequence that preserves local adjacency, allowing lightweight 1D networks to extract patterns from the point cloud. The RGB stream, meanwhile, captures the appearance information that cameras provide but LiDAR cannot—texture, color, and fine visual boundaries.

Knowledge from these two auxiliary modalities is then distilled into the 3D backbone through a multi-layer bidirectional polarity-aware linear cross-attention mechanism. Cross-attention lets each stream query the others, so features from the image stream and the serialized point stream can inform the features being learned by the 3D convolutional network. The bidirectional design allows information to flow in both directions between streams, and the polarity-aware component distinguishes the sign or orientation of the features being aligned, which helps the mechanism match corresponding structures across modalities that represent the world in very different ways. Crucially, all of this exchange happens only during training. Once the network has been trained, the auxiliary streams are discarded, and the deployed model is the 3D backbone alone—meaning the extra accuracy gained from images and serialized point sequences comes at zero additional inference cost.

This zero-cost distillation is the aspect of the work with the clearest practical significance. Real-time perception systems on autonomous vehicles operate under strict computational budgets, and any module that must run on board—however accurate—must also be fast. Techniques that fuse camera and LiDAR features at inference time can improve accuracy but add latency and hardware complexity. By confining the multi-modal interaction to the training phase, the Shenzhen framework captures much of the benefit of multi-modal learning while keeping the deployed model as lean as a single-modality network. For engineering teams weighing the cost of additional sensors and compute on a production vehicle, that distinction could be decisive.

The reported benchmark numbers place the framework among the strongest results on both datasets. SemanticKITTI, collected in Karlsruhe, Germany, is the canonical benchmark for outdoor LiDAR segmentation, with sequences spanning tens of kilometers of urban and highway driving; nuScenes, developed for autonomous driving research, pairs LiDAR with camera, radar, and other sensors across a thousand scenes in Singapore and Boston. Mean intersection-over-union, the standard accuracy metric, measures the overlap between predicted and ground-truth labels for each class and averages across them, so gains of even a few points are meaningful. Achieving 74.4 percent on SemanticKITTI and 83.5 percent on nuScenes reflects the combined effect of the large receptive fields, the balanced voxelization, and the dual-stream distillation.

The research was supported by the Shenzhen Science and Technology Program under grants JCYJ20220818102215034 and 20231129112637001, and in part by the National Key Research and Development Program of China under grant 2024YFB4709503. The article was received on 15 June 2026, accepted on 7 September 2026, and published on 28 September 2026 under a Creative Commons license that permits non-commercial sharing with attribution. As with any benchmark-driven advance, the ultimate test will be how the framework transfers to the messier conditions of production deployments—different sensor configurations, adverse weather, and novel cities—but the paper offers a clear demonstration that a 3D network can be taught by its richer siblings during training and still run solo, and fast, once it hits the road.

Subject of Research: Knowledge distillation and large sparse multi-directional convolution for 3D LiDAR point cloud semantic segmentation

Article Title: Dual stream knowledge distillation for 3D LiDAR segmentation model with large sparse multi-directional convolution

Article References: Gu, Y., Xiao, H., Chen, G., & Liang, L. (2026). Dual stream knowledge distillation for 3D LiDAR segmentation model with large sparse multi-directional convolution. Complex & Intelligent Systems. https://doi.org/10.1007/s40747-026-02523-w

Image Credits: AI Generated

DOI: 10.1007/s40747-026-02523-w

Keywords: LiDAR segmentation, point clouds, knowledge distillation, sparse convolution, autonomous driving, SemanticKITTI, nuScenes, 3D scene understanding, cross-attention, voxelization, computer vision, deep learning