Comparative Analysis of Unsupervised Concept Drift Detection Techniques in High-Dimensional Biomedical Data Streams

Abstract

In the area of real-time analytics, the ability to detect concept drift shifts in data distribution over time is vital for maintaining the reliability of predictive models. This analysis presents a comprehensive comparative analysis for five Unsupervised Concept Drift Detection Algorithms Adaptive Boosting (AdaBoost), Diversity-Induced Ensemble (DIE), Adaptive Sliding Window (ADWIN), Sequential Probability Ratio Test (SPRT), and Page-Hinkley Test (PHT) with a focus on high-dimensional biomedical data streams. The evaluation is conducted using three large-scale and diverse biomedical datasets: MIMIC-III/IV, UK Biobank, and MedMNIST, each representing a distinct challenge in terms of dimensionality, temporal variability and data type (tabular, genomic, and imaging). Performance is assessed across key metrics including Detection Delay, Memory Usage, Execution Time, and post-drift classification effectiveness (Precision, Recall, F1-Score, and Accuracy). Both synthetic and real-world drifts are incorporated to simulate dynamic environments. The findings reveal that ensemble-based methods such as AdaBoost and DIE outperform statistical approaches in handling noisy, sparse, and high-dimensional streams, offering superior adaptability and robustness. This research contributes a systematic evaluation framework and empirical insights to guide the deployment of unsupervised drift-aware systems in healthcare and other data-intensive domains.