Performance Optimization of High-Speed Wireless Communication Systems Using Machine Learning

Abstract

High-speed wireless communication systems underpin the data-intensive demands of contemporary 5G deployments and the emergent 6G paradigm, yet sustaining high throughput, low latency, and dependable connectivity in channels that shift rapidly remains an open engineering problem. In this paper, we present an ML-driven framework that combines a deep reinforcement learning (DRL) scheduler with a hybrid CNN-LSTM channel predictor to jointly optimize radio resource allocation, modulation order, transmit power, and bandwidth assignment in real time. Evaluated across full-buffer, bursty, mixed-traffic, and high-mobility scenarios under a 3GPP TR 38.901-compliant simulator, the proposed scheduler achieved 23.0% higher throughput, 35.0% lower latency, and 18.1% better energy efficiency than Proportional Fair scheduling, while reducing QoS violations by 75.9% and packet loss by 76.3%, with a Jain's fairness index of 0.887. The companion CNN-LSTM channel predictor achieved 94.7% prediction accuracy and a normalized mean squared error of −16.2 dB, reducing CSI feedback overhead by 41.3% relative to reactive schemes and outperforming standalone CNN and LSTM baselines by 28% under high-mobility conditions. These results indicate that coupling predictive channel-state modeling with multi-objective reinforcement learning is a practical, quantifiable path toward meeting next-generation wireless performance targets.