<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>density-based clustering &#8211; Science</title>
	<atom:link href="https://scienmag.com/tag/density-based-clustering/feed/" rel="self" type="application/rss+xml" />
	<link>https://scienmag.com</link>
	<description></description>
	<lastBuildDate>Sat, 03 Oct 2026 16:47:23 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.2</generator>

<image>
	<url>https://scienmag.com/wp-content/uploads/2024/07/cropped-scienmag_ico-32x32.jpg</url>
	<title>density-based clustering &#8211; Science</title>
	<link>https://scienmag.com</link>
	<width>32</width>
	<height>32</height>
</image> 
<site xmlns="com-wordpress:feed-additions:1">73899611</site>	<item>
		<title>Entropy-Guided Clustering Tackles Robust Fitting of Multiple Structures in Noisy Data</title>
		<link>https://scienmag.com/entropy-guided-clustering-tackles-robust-fitting-of-multiple-structures-in-noisy-data/</link>
		
		<dc:creator><![CDATA[Denise Maddox]]></dc:creator>
		<pubDate>Sat, 03 Oct 2026 16:47:23 +0000</pubDate>
				<category><![CDATA[Technology and Engineering]]></category>
		<category><![CDATA[adaptive clustering algorithms]]></category>
		<category><![CDATA[adaptive threshold]]></category>
		<category><![CDATA[Cluster Computing]]></category>
		<category><![CDATA[clustering]]></category>
		<category><![CDATA[computer vision]]></category>
		<category><![CDATA[density-based clustering]]></category>
		<category><![CDATA[dominant sets]]></category>
		<category><![CDATA[entropy-based clustering]]></category>
		<category><![CDATA[geometric pattern separation]]></category>
		<category><![CDATA[information entropy]]></category>
		<category><![CDATA[information-theoretic methods]]></category>
		<category><![CDATA[model fitting]]></category>
		<category><![CDATA[multi-model fitting]]></category>
		<category><![CDATA[multi-structure data processing]]></category>
		<category><![CDATA[neighborhood graph]]></category>
		<category><![CDATA[noisy data analysis]]></category>
		<category><![CDATA[outlier detection]]></category>
		<category><![CDATA[outlier removal]]></category>
		<category><![CDATA[preference analysis]]></category>
		<category><![CDATA[RANSAC]]></category>
		<category><![CDATA[RANSAC limitations]]></category>
		<category><![CDATA[robust model fitting]]></category>
		<category><![CDATA[structure from motion]]></category>
		<guid isPermaLink="false">https://scienmag.com/?p=230938</guid>

					<description><![CDATA[Researchers at Xiamen University of Technology have developed an entropy-driven clustering method that automatically tunes its own thresholds to separate inliers from outliers when fitting multiple geometric models to heavily contaminated data.]]></description>
										<content:encoded><![CDATA[<p>Computer vision systems face a deceptively simple question every time they look at the world: which points in the data actually belong to the same underlying structure? A self-driving car tracking lane markings, a drone stitching together aerial photographs, or a robot arm reconstructing a scene all depend on the ability to separate genuine geometric patterns from a swarm of misleading outliers. A new study published in Cluster Computing by Fei Chen, Disai Yang, Hanlin Guo, Minjian Wei, Jiaqi Chen, and Zhisheng Lv of Xiamen University of Technology addresses this problem head-on with a method called Information-theoretic Adaptive Clustering-based Fitting, or IACF. The work, published on 27 September 2026 as volume 29, article 802 of the journal, tackles one of the most stubborn weaknesses of modern model fitting: the reliance on fixed, hand-tuned thresholds that collapse when data becomes severely contaminated.</p>
<p>To understand why this matters, it helps to recall how geometric model fitting has evolved. Since Martin Fischler and Robert Bolles introduced the Random Sample Consensus paradigm, or RANSAC, in 1981, the standard approach has been to repeatedly sample small subsets of points, hypothesize a model such as a line, a plane, or a homography, and count how many points agree with it. When a scene contains only one structure and a moderate number of outliers, this strategy works remarkably well. But real scenes are rarely so cooperative. An image pair of a building contains dozens of planes at different depths; a point cloud of a street contains ground surfaces, vehicle bodies, and facades simultaneously. Multi-structure fitting requires the algorithm not only to find models but to segment the data, deciding which points belong to which model instance and which are simply noise.</p>
<p>In recent years, researchers have increasingly turned to density-based clustering algorithms to perform this segmentation. Methods such as DBSCAN, introduced by Ester and colleagues in 1996, group points based on how densely they pack together, and their robustness to noise made them attractive for fitting pipelines. However, as the authors of the new study point out in their abstract, these algorithms carry a critical limitation: they depend on a fixed threshold parameter, conventionally called MinPts, which specifies the minimum number of neighbors required for a point to be considered part of a dense region. This threshold often lacks adaptability to varying data characteristics. When input data is severely contaminated with outliers, determining suitable threshold values becomes genuinely difficult, and a poorly chosen value leads directly to suboptimal fitting results, with inlier structures fragmented or outliers absorbed into spurious clusters.</p>
<p>The core innovation of IACF is to let information theory decide the threshold automatically. Entropy, in the sense introduced by Claude Shannon, quantifies the disorder or uncertainty of a distribution. The researchers propose an entropy-based adaptive strategy that measures the disorder of local neighborhoods in the data and uses this measurement to dynamically determine an optimal clustering threshold, denoted MinPts*, for each specific dataset without any prior knowledge. In other words, instead of an engineer guessing how many neighbors should define a dense region, the algorithm inspects the statistical structure of the data itself and derives the answer. This is a meaningful shift: the parameter that most strongly controls density-based clustering behavior is no longer a fixed constant but a data-driven quantity that adapts to the contamination level and geometric complexity of each problem instance.</p>
<p>The method unfolds in three coordinated stages. First, the authors construct a reliable neighborhood graph by integrating motion and preference information. Preference analysis, a technique well established in the multi-model fitting literature through works such as T-linkage by Magri and Fusiello and accelerated hypothesis generation by Chin, Yu, and Suter, represents each data point by its pattern of agreement with a large set of sampled hypotheses. Points belonging to the same structure share similar preference vectors, so combining this preference signature with motion consistency information allows the graph-building step to prune invalid edges while preserving the potential local inlier structures that matter for fitting. This graph pruning is the first line of defense against outliers, removing connections that would otherwise let mismatched points contaminate each other&#8217;s neighborhoods.</p>
<p>Second, the entropy-based adaptive strategy operates on this graph, quantifying the disorder of local neighborhoods to set the optimal MinPts* threshold for the dataset at hand. Third, building on both components, the authors develop an Entropy-Adaptive Density-based Dominant Set Clustering algorithm, or EADSC. Dominant set clustering, formalized by Pavan and Pelillo in their influential work on pairwise clustering, is a graph-theoretic framework in which clusters emerge as dominant sets, subsets of vertices that are internally coherent with respect to their external environment. By making the dominant set search entropy-adaptive, EADSC can autonomously detect model-related subgraphs corresponding to individual model instances and accurately partition inliers from outliers, without the analyst supplying the number of structures in advance.</p>
<p>The experimental evaluation reported in the paper covers several challenging datasets, and the authors state that the proposed IACF method outperforms several state-of-the-art fitting methods in terms of both fitting accuracy and computational efficiency. That dual improvement is notable because robustness and speed are usually in tension in this field. Methods that achieve high accuracy on heavily contaminated data, such as hypergraph optimization approaches, energy minimization and mode-seeking frameworks, and residual-based consensus techniques, often pay a steep computational price, while faster methods sacrifice accuracy when outlier ratios climb. An adaptive threshold that correctly identifies inlier structure on the first pass reduces wasted hypothesis sampling, which is where much of the computational cost of fitting pipelines accumulates.</p>
<p>The context in which this work appears is a field under growing pressure from real-world deployment. The reference list of the paper traces the trajectory of the problem: from RANSAC and its many guided-sampling descendants, through preference-analysis methods and hypergraph formulations, to recent deterministic fitting approaches such as latent semantic consensus and spatial clustering guided two-view fitting, and graph-based techniques for mismatch removal. Applications cited in the surrounding literature range from multimodal image matching and feature matching with large numbers of outliers, to instantaneous frequency estimation of multi-component signals with multi-sensor consensus, to 3D point cloud map merging for large-scale environments and defect detection in printed-circuit boards. Each of these tasks shares the same mathematical skeleton: data generated by multiple geometric models plus noise, and an algorithm that must recover the models without knowing in advance how many there are or how badly the data is corrupted.</p>
<p>What makes the entropy angle particularly compelling is its generality. The authors connect their contribution to a broader line of research on autonomous clustering, including recent work by Yang and Lin on clustering by fast find of mass and distance peaks and toward autonomous distributed clustering. The shared ambition across these efforts is to remove human tuning from the clustering loop, letting statistical measures of structure determine algorithmic parameters. In the IACF framework, entropy serves as a universal yardstick of neighborhood disorder: a well-organized inlier neighborhood has low entropy, while a neighborhood polluted by outliers exhibits high disorder. By searching for the threshold that best separates these regimes, the algorithm effectively asks the data how much contamination it contains and adjusts its density criteria accordingly. This is precisely the kind of self-configuration that matters for systems operating in uncontrolled environments, where the outlier ratio can swing dramatically between frames or between scenes.</p>
<p>The practical implications extend across the computer vision stack. Robust multi-structure fitting sits upstream of fundamental estimation tasks: fundamental matrix and homography estimation in two-view geometry, motion segmentation in video, plane detection in point clouds, and registration of 3D scans. Improving the accuracy and speed of the fitting stage propagates improvements into every downstream application, from augmented reality anchoring to industrial inspection. The authors have made the datasets used in the study publicly available through the Adelaide RMF benchmark repository and a project page, and state that the custom code is available from the corresponding author upon reasonable request, which should make the method straightforward for other groups to evaluate against their own pipelines.</p>
<p>Funded by the Natural Science Fund of Fujian Province, the Natural Science Foundation of Xiamen, the National Natural Science Foundation of China, and several Xiamen municipal science and technology programs, the research reflects a sustained investment in the geometric foundations of machine perception. Whether entropy-adaptive thresholds will become a standard component of future fitting pipelines remains to be seen, but the study makes a clear case that the era of fixed MinPts values in density-based fitting may be drawing to a close. As vision systems are asked to operate in ever messier environments, algorithms that measure the disorder of their own inputs and adapt on the fly offer a persuasive path toward fitting that is both more accurate and more efficient, no matter how many structures the world throws at them.</p>
<p><strong>Subject of Research:</strong> Information-theoretic adaptive density clustering for robust multi-structure geometric model fitting in computer vision</p>
<p><strong>Article Title:</strong> Information-theoretic adaptive clustering for robust multi-structure model fitting</p>
<p><strong>Article References:</strong> Information-theoretic adaptive clustering for robust multi-structure model fitting. (n.d.). <a href="https://doi.org/10.1007/s10586-026-06563-2" rel="noopener noreferrer">https://doi.org/10.1007/s10586-026-06563-2</a></p>
<p><strong>Image Credits:</strong> AI Generated</p>
<p><strong>DOI:</strong> <a href="https://doi.org/10.1007/s10586-026-06563-2" rel="noopener noreferrer">10.1007/s10586-026-06563-2</a></p>
<p><strong>Keywords:</strong> model fitting, density-based clustering, information entropy, adaptive threshold, dominant sets, RANSAC, computer vision, outlier removal, preference analysis, multi-model fitting, neighborhood graph, Cluster Computing</p>
]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">230938</post-id>	</item>
		<item>
		<title>Widely Used Data-Cleaning Step in Chronic Disease AI Fails Rigorous Testing</title>
		<link>https://scienmag.com/widely-used-data-cleaning-step-in-chronic-disease-ai-fails-rigorous-testing/</link>
		
		<dc:creator><![CDATA[Denise Maddox]]></dc:creator>
		<pubDate>Thu, 01 Oct 2026 12:18:20 +0000</pubDate>
				<category><![CDATA[Technology and Engineering]]></category>
		<category><![CDATA[best practices for data preparation in healthcare AI]]></category>
		<category><![CDATA[chronic disease data cleaning]]></category>
		<category><![CDATA[chronic disease prediction]]></category>
		<category><![CDATA[class imbalance]]></category>
		<category><![CDATA[clinical data]]></category>
		<category><![CDATA[cross-validation]]></category>
		<category><![CDATA[data leakage]]></category>
		<category><![CDATA[data leakage in clinical machine learning]]></category>
		<category><![CDATA[DBSCAN]]></category>
		<category><![CDATA[density-based clustering]]></category>
		<category><![CDATA[effects of resampling techniques like SMOTE-ENN on model accuracy]]></category>
		<category><![CDATA[evaluation of outlier removal methods in health data]]></category>
		<category><![CDATA[impact of data preprocessing on disease prediction models]]></category>
		<category><![CDATA[influence of data leakage on reported machine learning performance]]></category>
		<category><![CDATA[limitations of common]]></category>
		<category><![CDATA[Machine learning]]></category>
		<category><![CDATA[machine learning bias in healthcare]]></category>
		<category><![CDATA[outlier detection]]></category>
		<category><![CDATA[outlier detection in medical datasets]]></category>
		<category><![CDATA[pitfalls of standard data cleaning steps in chronic disease prediction]]></category>
		<category><![CDATA[reproducibility]]></category>
		<category><![CDATA[rigorous testing of clinical data preprocessing pipelines]]></category>
		<category><![CDATA[SMOTE]]></category>
		<category><![CDATA[YadroSeg]]></category>
		<guid isPermaLink="false">https://scienmag.com/?p=222606</guid>

					<description><![CDATA[A leakage-free benchmark of ten chronic-disease cohorts finds that no outlier-removal method beats doing nothing, while data-cleaning steps silently delete rare patients and inflated evaluation protocols manufacture up to 61 points of phantom accuracy.]]></description>
										<content:encoded><![CDATA[<p>Machine-learning pipelines that predict diabetes, hypertension, chronic kidney disease and heart disease from routine clinical records often begin with a seductive idea: delete the outliers, balance the classes, and the classifier will see the signal more clearly. A new study in Results in Engineering puts that idea under the microscope and finds it wanting. In a leakage-free benchmark spanning ten public chronic-disease cohorts, twelve outlier detectors, and 6,500 per-fold evaluation records, researchers Rifkat R. Davronov and Fatima T. Adilova report that not one cleaning method significantly outperformed simply doing nothing at all. Worse, the cleaning step preferentially deletes exactly the patients a screening model exists to find.</p>
<p>The paper&#8217;s most striking number concerns how these pipelines have been measured, not how they perform. When feature selection, scaling, outlier removal and the popular SMOTE-ENN resampling technique are all fitted on the full dataset before cross-validation — a practice the authors trace through much of the published literature — reported F1 scores are inflated by an average of 28.9 percentage points, with a range from 0.4 to 61.2 points depending on the cohort. The dominant culprit is resampling before splitting: SMOTE synthesises new minority-class points by interpolating between existing ones, so a test-fold patient can become a near-duplicate of training data, while the accompanying Edited Nearest Neighbours rule deletes borderline majority records using their labels. The evaluation, in effect, grades the model on questions it has already seen.</p>
<p>The inflation is largest precisely where the underlying task is hardest. On a small hypertension cohort of 175 patients, where blood pressure is essentially unpredictable from anthropometric measurements alone, the flawed protocol reports an F1 of 93.75 percent; the corrected protocol yields 35.5 percent, with the area under the ROC curve hovering near chance. On an easy, well-posed chronic kidney disease diagnosis task, the difference between protocols is a mere 0.4 points. Leakage, the authors conclude, does not add a constant bonus — it manufactures performance in proportion to how little genuine signal exists, which is why mid-to-high-nineties accuracy figures on small clinical cohorts deserve deep suspicion.</p>
<p>Under the corrected protocol, with every fitted step confined inside the training fold, the benchmark&#8217;s verdict on outlier removal is unambiguous. Twelve detectors — including tuned DBSCAN, Local Outlier Factor, Isolation Forest, One-Class SVM, robust covariance estimation, HDBSCAN, and the recent distribution-based methods ECOD and COPOD — were compared against a no-cleaning control over 50 folds per cell, using the Nadeau–Bengio corrected resampled t-test with Holm adjustment to account for overlapping training sets. Across all 130 method-by-cohort cells, no detector significantly beat the control. The single statistically significant result in either direction was a loss: HDBSCAN on the diabetic retinopathy cohort, dropping F1 by 9.37 points.</p>
<p>The authors did not merely tune their own detectors carefully and declare victory. To test the objection that poor configuration explains the null result, they granted an oracle: for each cohort, the best of 44 parameter settings of their proposed method was selected with hindsight on the very folds being reported. Even this impossible advantage yielded a mean gain of only 1.24 points — less than the 3.20 points that picking the best of 44 genuinely equivalent settings would produce from fold-to-fold noise alone. On two cohorts, the oracle&#8217;s chosen configuration removed nothing whatsoever. Across all 440 evaluated configurations, F1 correlated negatively with the fraction of data removed on seven of nine informative cohorts, as strongly as r = −0.99. Within this family of methods, removing more is monotonically worse.</p>
<p>If cleaning were merely useless, it would be a wasted step. The study&#8217;s forensic analysis shows it is actively hazardous. Every detector examined removes minority-class records preferentially — the proposed YadroSeg method deleted 32.8 percent of minority-class patients against 18.5 percent of majority-class records, a pattern significant on 8 of 10 cohorts, while Isolation Forest showed the most extreme ratio at 4.4 to 1. Triaging the deleted rows against physiological ranges, only about 15 percent contained impossible values that could be called genuine data errors; roughly 63 percent were rare but clinically plausible patients. The mechanism is not a defect in any particular algorithm: removal correlates strongly with local sparsity in feature space (r between 0.43 and 0.69) and barely at all with label impurity. Density-based detectors find sparse regions, and in clinical data sparse regions are populated by uncommon patients, not corrupted records.</p>
<p>The study is also a remarkable act of scientific self-correction. An earlier version of this work reported that its graph-based detector, YadroSeg, achieved the best F1 on three of five datasets, reaching 98.87 percent — a figure produced by the very leakage the revised paper dissects. The same configuration scores 100.00 under the flawed protocol and 58.51 under the corrected one. The authors pre-registered a series of tests to understand why their density-variation analysis, adapted from image segmentation, failed to transfer to tabular data, and report that two of their own registered explanations were refuted by the experiments designed to check them. The final finding is structural: driving the coring threshold until no core vertex survives leaves the set of removed records bit-identical on all ten cohorts, because the density sequence over a constructed clinical feature graph lacks the small set of dominant boundaries that thresholding presupposes.</p>
<p>What survives the transfer is the density ordering itself. When exposed as a continuous score rather than collapsed into a threshold, the peeling sequence recovered injected corruption better than the published algorithm on ten of ten cohorts, ranking second of nine scorers behind LOF. Measuring this honestly required the authors to correct their own evaluation a second time: their initial comparison scored detectors on the arbitrary masks they emitted, at flag rates differing by more than an order of magnitude. At a matched flag budget, every detector tested ranked corrupted records two to three times above chance — though none exceeded a detection F1 of 0.275, confirming that none of them is a reliable corruption detector, particularly for flipped labels, which leave a record exactly where it was in feature space.</p>
<p>The corrected implementation also brings a scalability lesson. The original formulation required a dense distance matrix that scaled as N to the power 2.67 and became infeasible beyond 32,000 records; the revised version, using radius queries on a spatial index, measures at N to the power 1.48 — competitive with DBSCAN and LOF, though an order of magnitude behind Isolation Forest&#8217;s effectively flat profile. The authors withdraw their earlier claim of O(N log N) scaling as unsupported by measurement, and confirm the exponents on two real clinical registries of over 100,000 and 250,000 records. A large-cohort confirmation arm on those registries, reaching nearly nine times the sample size of the largest primary cohort, reproduced the central null: cleaning never helped, and where differences reached significance they ran against it.</p>
<p>The practical recommendations are pointed. Every fitted preprocessing step should live inside the cross-validation loop, ideally enforced structurally through pipeline tools rather than manual sequencing. Any study adopting outlier removal should include a no-cleaning control and report the class composition of what it deletes. Calibration should be reported alongside discrimination, since resampling is known to damage probability calibration — a property the study observed across all configurations. And for reviewers, the single most informative check on a paper using the cleaning-plus-resampling-plus-ensemble template is whether the resampler is fitted inside or outside the validation loop. On the balance of evidence, the authors recommend against routine density-based outlier removal in clinical tabular prediction: a measurable risk of discarding rare patients, with no measurable benefit in return.</p>
<p><strong>Subject of Research:</strong> Evaluation leakage and outlier removal in machine-learning prediction of chronic diseases from clinical tabular data</p>
<p><strong>Article Title:</strong> Graph-based density-variation outlier removal for chronic disease prediction: A leakage-free benchmark and component-wise ablation</p>
<p><strong>Article References:</strong> Davronov, R. R., &amp; Adilova, F. T. (2026). Graph-based density-variation outlier removal for chronic disease prediction: A leakage-free benchmark and component-wise ablation. <em>Results in Engineering, 32</em>, Article 112842. <a href="https://doi.org/10.1016/j.rineng.2026.112842" rel="noopener noreferrer">https://doi.org/10.1016/j.rineng.2026.112842</a></p>
<p><strong>Image Credits:</strong> AI Generated</p>
<p><strong>DOI:</strong> <a href="https://doi.org/10.1016/j.rineng.2026.112842" rel="noopener noreferrer">10.1016/j.rineng.2026.112842</a></p>
<p><strong>Keywords:</strong> machine learning, outlier detection, data leakage, chronic disease prediction, SMOTE, class imbalance, DBSCAN, clinical data, cross-validation, YadroSeg, reproducibility, density-based clustering</p>
]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">222606</post-id>	</item>
	</channel>
</rss>
