Performance Optimization of High-Speed Wireless Communication Systems Using Machine Learning

3.1. Deep Reinforcement Learning for Resource Allocation

The resource allocation problem was formulated as a Markov Decision Process (MDP). This enabled the development of a DRL algorithm that learns to make allocation decisions within the MDP through interaction with a simulated communication network. At the core of the algorithm is a Deep Q-Network (DQN) model that maps observed system states to allocation decisions—specifically power levels, MCS selection, and RB assignment.

At every scheduling instant t in the simulation, the state (t) consists of normalized channel state information matrices for all UEs in the network, current SINR measurements, and current traffic statistics. The hybrid neural network architecture comprises convolutional layers that learn spatial correlations from the CSI matrices, as well as LSTM layers that capture the evolution of channel and traffic conditions over time. Actions include both discrete variables, such as the RB assignment matrix Apᵇ ∈ {0,1}ⁿˣᴹ, and continuous ones, such as the power allocation vector P ∈ ℝⁿ with Σp_i ≤ P_max.

3.1.1. Action Representation

The action space combines three discrete decision variables: resource block assignment, modulation and coding scheme (MCS) selection, and transmit power allocation. At each decision interval, the DRL agent selects an appropriate combination of these actions based on the observed network state to maximize the cumulative reward. Resource block assignment allocates available spectrum resources to active users; the modulation and coding scheme adapts the transmission rate to channel conditions; and transmit power allocation optimizes energy consumption while maintaining reliable communication. This joint action representation enables efficient utilization of wireless resources, improves throughput, minimizes latency, and enhances overall Quality of Service (QoS). The per-user transmit-power range is therefore quantized into L discrete levels, uniformly spaced between a minimum transmit power P_min and a per-user maximum P_max defined in Table 2:

$$\mathrm{p}\_\mathrm{i}\in \{\mathrm{P}\_\min +\mathrm{k}·(\mathrm{P}\_\max -\mathrm{P}\_\min )/(\mathrm{L}-1):\mathrm{k}=0,1,\dots ,\mathrm{L}-1\}$$

with L = 8 in the reported experiments, a choice that balances control granularity against action-space growth.

Table 2:

Simulation environment parameters.

Parameter	Value/Configuration
Network topology	Single macro-cell, 500 m radius; 4 small cells
Number of UEs	20–100 (varied per scenario)
Carrier frequency	3.5 GHz (sub-6 GHz); 28 GHz (mmWave)
System bandwidth	100 MHz
Number of resource blocks	50 (sub-6 GHz); 66 (mmWave)
Channel model	3GPP TR 38.901 UMa / Umi
UE velocity	0–120 km/h
Traffic types	eMBB, URLLC, mMTC (mixed)
Transmit power range	10–40 dBm
Noise figure	7 dB
DRL training episodes	Up to 50,000
Baseline schedulers	Proportional Fair, Maximum Throughput

Factored action-space design. Rather than forming a single joint Q-head over the full Cartesian product of RB assignment, MCS, and power level—which would be intractably large for the UE counts in Table 2—the action space is factored into three per-user discrete sub-actions sharing a common state encoder (the CNN-LSTM backbone described above):

• Resource-block assignment, a_rb ∈ {1, …, M}, selecting one of M available resource blocks per scheduling opportunity (M = 50 for sub-6 GHz, M = 66 for mmWave, per Table 2);

• MCS selection, a_mcs ∈ {QPSK, 16-QAM, 64-QAM, 256-QAM, 1024-QAM}, five discrete options;

• Power level, a_pow ∈ {0, …, L − 1}, the quantized index above.

Each sub-action is produced by a dedicated output head reading from the shared fully connected trunk, rather than a single flattened head of size M × 5 × L. For N simultaneously scheduled users, the per-user action cardinality is M × 5 × L; with M = 50 and L = 8, this is 50 × 5 × 8 = 2000 discrete actions per user, evaluated independently across up to N = 100 users per scheduling slot—several orders of magnitude smaller than the undiscretized joint space a monolithic Q-head would require.

Constraint masking. Two hard constraints from the problem formulation—the per-cell power budget Σ_i p_i ≤ P_max and the one-user-per-resource-block rule—are enforced by masking infeasible actions before the arg-max over Q-values, rather than relying solely on the reward penalty to discourage them. At each decision step, any combination that would violate the power budget or reassign an already-occupied resource block is assigned Q = −∞ prior to action selection, guaranteeing that only feasible allocations are ever executed regardless of the current policy’s learned preferences. This masking is applied identically during both training and inference, so the ε-greedy exploration process described in Section 3.1 samples only from the feasible action set at every step.

Figure 1 illustrates the process of continuous interaction among state observation, agent learning, action selection and execution, and performance evaluation in the closed feedback loop. Learning takes place by employing ε-greedy policy exploration with exponential decay, gradually transitioning from exploration to policy exploitation. The experience replay memory stores up to one million transitions, which are sampled in minibatches of 64 to reduce temporal correlation and improve sample efficiency.

The reward function considers four aspects in proportion to weights: R_throughput(t) = Σ_i log(1 + R_i(t)) is a sum-log utility for proportional fairness summed over users i, where R_i(t) is the instantaneous achieved rate for user i, R_latency(t) = −(1/N)Σ_i max(0, D_i(t) − D_i^target), a hinge-style penalty on delay overshoot per user, with an additional fixed penalty term applied if any user breaches its latency ceiling. R_energy(t) = −Σ_i P_i(t) / Σ_i R_i(t), i.e., negative power-per-bit, or equivalently a positive efficiency ratio Σ_i R_i(t) / Σ_i P_i(t) if the sign convention is meant to reward higher efficiency directly. R_fairness(t) = (Σ_i R_i(t))² / (N · Σ_i R_i(t)²), bounded in (0,1], with an additional penalty subtracted whenever a designated priority user falls below its guaranteed minimum rate. The target network follows the standard DQN approach, maintaining a fixed copy of the network parameters θ⁻ for C = 1000 iterations, so that the target Q value is calculated as y = r + γ maxₐ’Q(s’, a’; θ⁻). This avoids instability when updating both estimates and targets concurrently.

The dataset includes 5G NR and post-5G wireless scenarios spanningsub-6 GHz and millimeter-wave frequency bands. As shown in Figure 2, the DRL scheduler consistently outperforms the baseline Proportional Fair algorithm across the key criteria of throughput, latency, energy efficiency, and resource allocation fairness.

The model comprises four convolutional layers with 64, 128, 256, and 512 filters; two LSTM layers with 256 and 128 units, respectively, for temporal modeling; and three fully connected layers with 512, 256, and |A| units, respectively, to produce the Q-values. Convolutions are followed by batch normalization to speed up training. Dropout is used on the LSTM layers with p = 0.3 to avoid overfitting. The Adam optimizer uses an initial learning rate of α = 0.001, which is multiplied by 0.95 every 10,000 episodes, together with a gradient-norm threshold of 10.0.

Prioritized experience replay samples the most informative transitions based on their temporal-difference error. Training is stopped if the moving average score of the last 100 episodes does not improve by more than 0.1% over the next 500 episodes, or if the maximum number of episodes reaches 50,000. Allocation decisions during inference take less than 5 ms, which is within the coherence time of wireless channels. After training, the experiments showed a 23% increase in throughput, a 35% reduction in latency, and an 18% increase in energy efficiency compared with proportional-fair scheduling, with Jain’s fairness index exceeding 0.85 under full-buffer, bursty, and mixed eMBB/URLLC/mMTC workloads.

3.2. Hybrid CNN-LSTM for Channel Prediction

High-precision CSI forecasting is necessary for proactive resource management; accordingly, an efficient hybrid CNN-LSTM architecture was designed to capture both the spatial properties of the antenna array and the temporal evolution of channel conditions. The proposed model processes the input through two different streams from the historical channel sequence H(t−T), …, H(t).

One stream processes the input through three convolutional layers with 64, 128, and 256 filters to extract higher-order spatial features from the channel matrix. The resulting feature maps undergo batch normalization and max pooling to reduce dimensionality while preserving predictive information. The vectorized output is then passed to two LSTM layers with 256 and 128 units, respectively.

The final layer computes H_pred(t + Δt) as a linear function over a time horizon of Δt equaling 5–10 ms, which corresponds to the coherence time for millimeter-wave systems. Model training uses an objective function consisting of mean squared error as a measure of amplitude prediction error and a penalty on phase coherence, optimized with the Adam optimizer and learning-rate scheduling.

During online operation, the model achieves a normalized mean squared error below −15 dB, enabling preemptive link adjustment that mitigates channel aging and reduces feedback exchanges by more than 40% compared with reactive approaches. In particular, in cases of high mobility, where both spatial and temporal effects come into play, the hybrid approach achieved 28% better prediction accuracy than either the standalone CNN or LSTM architectures.

The output layer predicts H_pred(t + Δt) with linear activation, where the prediction horizon Δt is scenario-conditioned rather than fixed. Using the Doppler-based approximation T_c ≈ 9/(16π·f_d), Δt is set to approximately 10–20% of the coherence time for the corresponding frequency-mobility combination, ranging from Δt ≈ 1.8–3.7 ms for sub-6 GHz pedestrian conditions down to Δt ≈ 5.8–11.5 µs for mmWave high-mobility conditions. The 5–10 ms horizon reported in earlier sections of this work applies specifically to the sub-6 GHz, low-mobility regime; it is not representative of the mmWave, high-mobility case, where a substantially shorter horizon is required to remain within the channel’s coherence interval.

4.1. Simulation Setup and Evaluation Scenarios

The experiments were performed in a dedicated Python-based discrete-event simulator, in accordance with the 3GPP TR 38.901 channel model. Table 2 provides the fundamental parameters used in the simulations. Three traffic types were considered: eMBB, URLLC, and mMTC. For each scenario, ten experiments were performed using varying random seeds, and the results are presented as averages. Run-to-run variation did not exceed 1.5%, indicating low variability across experiments.

For comparison, two classical scheduling schemes—proportional fair (PF) and maximum throughput (MT)—were used as baselines. These two policies were selected due to their extreme positions in the classic efficiency-versus-fairness trade-off problem and their extensive deployment in practical 5G networks. The reinforcement learning scheme was trained from scratch for all three scenarios. No transfer learning was employed; therefore, the reported performance improvements reflect cold-start training for each scenario.

4.2. Throughput Performance

Table 3 presents downlink throughput performance averaged over five test conditions. Under the full-buffer sub-6 GHz setting, DRL achieved 387.4 Mbps, compared with 314.9 Mbps for PF, yielding a 23.0% gain. DRL also exceeded MT, which achieved 341.2 Mbps. Similarly, DRL outperformed MT by 387.4 Mbps versus 341.2 Mbps. The throughput gain was greater in the 28 GHz mmWave setting, where DRL achieved 1124.6 Mbps and surpassed PF by 24.8%. Under the 120 km/h high-mobility condition, DRL maintained a throughput of 271.3 Mbps, an 18.1% gain over PF. The CNN-LSTM channel predictor contributed significantly to DRL robustness in this setting.

Figure 3 visualizes the comparison results. The consistent gap between DRL and the two baselines across all five conditions is consistent with the proposed scheduler’s use of predicted future channel states, whereas PF and MT allocate resources using current channel conditions.

4.3. Latency Reduction Across Traffic Classes

Table 4 shows the latency results. For URLLC flows, the DRL algorithm achieved an average latency of 0.81 ms, which is well below the 1 ms requirement and 34.7% lower than the 1.24 ms recorded by PF. For eMBB flows, the algorithm showed a similar advantage over PF, reducing latency from 7.11 ms to 4.62 ms. In peak-hour mixed loads, DRL achieved an average latency of 6.14 ms, compared with 9.48 ms for PF.

DRL’s ability to provide latency guarantees for URLLC flows without starving low-priority flows can be attributed directly to the fairness factor in the reward function. Figure 4 illustrates the latency results. The URLLC result is particularly notable: a latency of 0.81 ms provides a 190 μs margin below the 1 ms requirement.

4.4. Energy Efficiency and Fairness

Table 5 summarizes the results for energy efficiency and fairness. The DRL solution yielded 12.4 Mbps/W, representing an 18.1% improvement over PF and a 26.5% improvement over MT. Jain’s fairness index reached 0.887, exceeding both baselines and indicating that the multi-objective reward mechanism prevented the agent from disproportionately favoring the best-performing users over cell-edge devices. The QoS violation rate decreased to 2.1%, while the packet loss rate was 0.9%.

Figure 5 shows the relationship between energy efficiency and fairness using a dual-axis chart. Although MT yields relatively high throughput, its fairness and energy-efficiency values are lower than those of both other schedulers (0.731 and 9.8 Mbps/W, respectively). As shown in Figure 6, the difference between the schedulers in QoS violation and packet loss rates is substantial.

4.5. Channel Prediction Accuracy

Table 6 compares the performance of the CNN-LSTM-based predictor with a simple CNN predictor, an LSTM predictor, and a traditional LS estimator. The hybrid model achieved an NMSE of −16.2 dB and an accuracy of 94.7%. The standalone CNN achieved −12.8 dB and 87.4%, respectively, whereas the standalone LSTM achieved −11.9 dB and 85.1%, respectively. An additional benefit is a 41.3% reduction in feedback overhead, which frees channel capacity for payload data.

Figure 7 illustrates these results as three horizontal bar charts, from which the margin by which the proposed solution surpasses the baselines in NMSE, accuracy, and feedback overhead reduction can be readily seen. Additionally, the 2.1 ms inference delay remains below the 5 ms scheduling-slot period. Finally, under high-mobility scenarios, the hybrid model achieved an NMSE of −14.8 dB, whereas the CNN predictor degraded to −10.5 dB, supporting the importance of LSTM temporal memory under short coherence times.

4.6. Ablation Study

The ablation study in Table 7 demonstrates the effect of removing individual components from the full DRL model. Excluding prioritized experience replay resulted in a 6.8% reduction in throughput and an 18.5% increase in latency. Excluding the CNN-LSTM state encoder led to the largest effect: throughput dropped to 338.6 Mbps while latency increased to 1.09 ms. Removing the attention mechanism reduced throughput to 356.8 Mbps and increased latency to 0.93 ms, a smaller degradation than that caused by removing the CNN-LSTM state encoder. Removing the fairness reward slightly increased throughput but reduced Jain’s fairness index to 0.724, demonstrating the trade-off between throughput maximization and fairness.

The radar chart shown in Figure 8 compares these metrics as polygons. The ‘Without Fairness Reward’ polygon retains high normalized throughput and energy-efficiency values but drops sharply on the fairness axis, illustrating the need for all four reward terms. The ‘Without CNN-LSTM State’ polygon shows the broadest degradation across the evaluated metrics.

4.7. Signal Strength Variation Across Propagation Environments

Signal strength varies considerably depending on the location at which it is measured. In open spaces, the absence of obstructions ensures that the RSSI value remains high, while path loss stays low. However, in suburban areas, shadowing introduced by buildings leads to some degradation in signal quality. Urban environments experience even greater degradation because reflections, diffraction, and shadowing from high-rise buildings cause large RSSI fluctuations and produce a steep path-loss curve.

This phenomenon is illustrated in Figure 9, where indoor/urban attenuation is much higher than in open outdoor environments, and the difference increases as the distance between the transmitter and receiver grows. For example, at a distance of 100 m, indoor RSSI was almost 45 dB lower than in open outdoor conditions. The learned DRL policy reflected this behavior, showing a preference for lower-order modulation and higher transmit power under indoor conditions during post-training policy evaluation.

4.8. Convergence and Training Stability

The DRL algorithm converged consistently across all tested scenarios, reaching a stable performance plateau after approximately 18,000–24,000 training episodes, depending on scenario complexity. The target network and prioritized experience replay reduced reward fluctuations commonly observed during basic DQN training. After training, the policy remained consistently stable over 10,000 timesteps, with an episode reward standard deviation below 2% of the mean.

Figure 10 illustrates the relationship between the normalized episode rewards and the training episodes of the full architecture and its three ablations. Two observations are evident. First, removing the CNN-LSTM state encoder lowers the asymptotic reward and delays convergence: the curve plateaus at approximately 30,000 episodes, compared with 20,000 episodes for the full model. Second, the architecture without prioritized experience replay exhibits a much higher variance, implying that, without guided sampling, the agent frequently revisits low-information states rather than efficiently utilizing rare yet informative events. The proposed hybrid CNN-LSTM and Deep Reinforcement Learning framework demonstrates the potential of intelligent scheduling for next-generation wireless communication systems. The obtained results agree with recent advances in AI-assisted wireless networking, highlighting the growing importance of machine learning techniques for achieving scalable, adaptive, and energy-efficient communication in Beyond-5G and 6G environments [27-30].