The advent of the 5th generation (5G) networks has facilitated various Industry 4.0 applications, including connected and autonomous vehicles, remote surgery, industrial automation, and mission-critical Internet-of-Things (IoT) services. These applications, classified as Ultra-Reliable Low-Latency Communication (URLLC) in 5G, demand stringent Quality-of-Service (QoS) requirements, notably low latency and high reliability [1].

Two key technologies have emerged to support the flexible and cost-effective provisioning of applications while ensuring QoS requirements, namely Network Function Virtualization (NFV) and Multi-Access Edge Computing (MEC). On the one hand, NFV has become fundamental for service provisioning, enabling software-defined virtual networks to offer flexibility and scalability. Delivering a specific service requires composing and mapping an ordered chain of Virtual Network Functions (VNFs) to the underlying network infrastructure [2]. On the other hand, MEC alleviates resource demands and reduces user-perceived latency by extending cloud computing capabilities and IT services to the edge of the network [3]. Despite their advantages, MEC nodes possess comparatively limited storage and computational resources compared to the cloud.

In this context, a significant challenge in deploying VNFs for URLLC applications in MEC networks arises from the dynamic nature of user demands, application requirements, and traffic conditions, which may lead to several violations of QoS requirements. Furthermore, the typically ephemeral nature of VNFs means that deployed instances become inactive or must be deactivated once the related service is completed, thereby incurring additional costs or leading to inefficient resource usage [4].

In the literature, typical solutions for NFV deployment in MEC networks involve modifying the deployed VNFs through various redeployment schemes, including VNF migration, VNF sharing, and scaling [5,6,7,8,9,10]. VNF migration is frequently used to achieve load balancing in congested networks, conserve energy in underutilized segments, ensure reliability during physical node failures, and minimize costs. However, VNF migration has to guarantee stringent latency requirements between VNFs within a VNF chain. Traditional migration strategies typically move a single VNF at a time, which incurs high transmission costs and introduces potential service disruptions due to the resource overhead required for migration, such as processing power and bandwidth to transfer VNF states. These inefficiencies can degrade the user experience, strain network resources, and potentially lead to service-level agreement (SLA) violations [11]. To mitigate these drawbacks, recent research by Afrasiabi et al. [12] has proposed cluster migration, where groups of coupled VNFs are migrated together within a single physical node or across multiple nodes. This approach addresses some inefficiencies associated with single VNF migrations, yet the challenge of maintaining reliable service delivery remains a significant concern. Additionally, frequent migrations, whether from single VNFs or clusters, can lead to resource fragmentation, further reducing overall network efficiency and complicating resource management.

In contrast, reusing VNFs ensures consistent service delivery, minimizes the risk of interruptions, reduces downtime, and eliminates the need to migrate VNF states and data across nodes. It leverages tested and stable configurations to enhance the reliability and reduce the complexity of fault management. VNF reuse is cost-efficient as it avoids migration-related transmission and operational costs, maintains low latency by keeping VNFs in optimal locations, and enhances resource utilization. However, only a few studies explored reuse schemes [4,13,14], and generally overlooked the impact of traffic redirection on the operation and reliability of Service Function Chains (SFCs). Moreover, most existing work proposed heuristic algorithms that may require long execution times, yielding only suboptimal solutions and facing exponential computational complexity with network scaling.

Consequently, this paper focuses on solving the problem of SFC/VNF placement in MEC networks for mission-critical applications, aiming to guarantee QoS requirements through VNF redeployment and reuse schemes. Given that the SFC/VNF placement problem is NP-hard [15,16,17], the complexity of our problem, which includes the placement, recomposition, and redeployment of SFC/VNF, is also NP-hard. To address it, we formulate the problem as a Markov Decision Process (MDP) and solve it using a novel reinforcement learning (RL)-based framework. Specifically, we propose a novel and intelligent VNF/SFC placement strategy based on the deep Transformer Q-network (DTQN) algorithm. This approach is designed to learn complex data representations, enabling accurate and efficient decision-making. In particular, we aim to minimize the costs associated with VNF placement/redeployment (e.g., migration, removal) while favoring VNF reuse to reduce service disruptions and satisfy QoS requirements.

Our contributions can be summarized as follows:

• We formulate the VNF/SFC placement problem in MEC networks for mission-critical applications using Mixed-Integer Non-Linear Programming (MINLP).

• Given the NP-hardness of the formulated problem, we propose a novel DTQN-based approach to solve it.

• We perform extensive simulations that demonstrate the efficiency of our solution in terms of cost, VNF placement, reuse, and migration of VNF, as compared to the baseline approaches.

The remainder of the paper is organized as follows: Section 2 discusses the related work. In Section 3, we present the system model, while Section 4 formulates the VNF placement problem. Section 5 details our proposed solution, and Section 6 provides the simulation results. Finally, Section 7 concludes the paper.

To overcome the critical challenges of URLLC application provisioning in MEC networks, it is essential to meet key requirements such as minimizing latency to fulfill stringent QoS demands, reducing operational costs to ensure scalability, and maintaining the reliability of services through effective VNF reuse and minimal service disruptions. Comprehensive overviews of NFV placement strategies within MEC networks have been provided in surveys [18,19,20,21].

In the context of latency minimization and cost efficiency, the advent of NFV and MEC as fundamental technologies for agile network service deployment in 5G has led to extensive research. Early studies primarily employed heuristic and optimization-based approaches to address these challenges. For example, Yala et al. [22] used a genetic algorithm to balance the conflicting objectives of minimizing latency and maximizing availability in NFV-MEC networks. Kiran et al. [14] focused on reducing placement and resource costs through coordinated VNF placement, demonstrating its effectiveness in lowering overall costs. Jin et al. [23] addressed VNF chain deployment in MEC environments by formulating the problem as a Mixed Integer Linear Programming (MILP) model to optimize resource consumption. They proposed a two-phase deployment scheme: a constrained depth-first search algorithm to identify feasible paths and a path-based greedy algorithm to maximize resource reuse while minimizing new allocations. Simulations revealed that this method achieved near-optimal performance, reducing resource consumption by up to 25.6% compared to heuristic baselines while meeting latency constraints. Similarly, Afrasiabi et al. [12] proposed clustering coupled VNFs and migrating them within or across physical nodes using two look-ahead heuristics to reduce embedding costs and avoid local optima. Despite their merits, these approaches do not address reliability issues comprehensively.

The increasing complexity and dynamism of network environments necessitate scalable and adaptive solutions. To sustain service performance, researchers have widely studied reconfiguration-based strategies, such as VNF migration. Li et al. [11] introduced a latency-aware migration strategy to reduce end-to-end latency in SFC deployments, while Cho et al. [24] investigated optimization techniques for VNF migration in dynamic cloud environments. Kuo et al. [25] proposed a joint VNF placement and path selection approach to maximize served traffic demands, using a stress-testing-inspired algorithm to adapt resource allocation dynamically. This method demonstrated superior performance in terms of link and Virtual Machine (VM) resource utilization, outperforming traditional heuristics. However, scalability remains a challenge, as many of these solutions struggle to maintain optimal performance in highly dynamic conditions. Zhu et al. [26] similarly noted that while these strategies can be effective, they often fall short in environments requiring rapid adaptation to changing conditions.

An alternative strategy to enhance reliability and reduce frequent redeployments or migrations involves reusing VNFs. Addressing this gap, Doan et al. [27] proposed the Subchain-Aware NFV Service Placement (SAP) model, which optimizes network function reuse in MEC environments. By focusing on subchain reuse, SAP reduces configuration and deployment costs while enhancing reliability. The authors developed Tabu-SAP, a Tabu search-based solution, and implemented the Automated Provisioning framework for MEC (APMEC) on OpenStack. Tabu-SAP demonstrated scalability and supported eight times more SFC requests, achieving over a 50 percent cost reduction compared to start-of-the-art methods.

To address the limitations of traditional approaches, recent studies have incorporated machine learning (ML) and RL methods, which offer improved adaptability and scalability. For instance, Abouaomar et al. [28] applied deep reinforcement learning to optimize service migration in MEC vehicular environments, minimizing latency and service disruptions. Similarly, Subramanya et al. [29] utilized neural networks to enable auto-scaling of VNFs based on traffic demand and latency requirements. While these approaches show promise, traditional ML methods often face challenges in stability, scalability, and efficiently capturing long-term dependencies in dynamic settings.

Transformer-based techniques have emerged as a promising solution to address these challenges. Wu et al. [30] introduced the Double Deep Q-Network Decision Transformer (D3T), which combines a Decision Transformer (DT) with a Double Deep Q-Network (DDQN). The DDQN generates trajectory data for offline training, while the DT models sequences to optimize placement decisions, effectively handling high-dimensional state-action distributions and mitigating issues such as error propagation and value overestimation. D3T achieved a 25% reduction in rejection ratio and a 7% reduction in delay compared to previous methods. However, it does not fully address reliability or redeployment strategies to ensure continuous service delivery.

In summary, the existing body of research on NFV and MEC highlights substantial progress in optimizing service deployment for 5G applications, particularly in latency-sensitive and cost-efficient scenarios. Early heuristic and optimization-based approaches provided foundational insights but struggled with scalability and adaptability to dynamic environments. More recent reinforcement learning methods have improved decision-making in such dynamic environments, yet often fail to fully address temporal dependencies and the complexity of large-scale, mission-critical applications. Transformer-based models have shown promise in overcoming these limitations, offering superior capabilities for handling high-dimensional and temporally dependent data. In this context, our work advances the state of the art by introducing a novel DTQN-based framework that leverages the strengths of transformers and reinforcement learning. This approach uniquely focuses on optimizing VNF placement through reuse and intelligent redeployment, addressing critical gaps in scalability, cost efficiency, reliability, and QoS adherence.

We consider a two-tier hierarchical MEC network model. The lowest tier consists of user devices such as sensors, smart gadgets, and cameras that generate and transmit real-time data. This data is forwarded to edge nodes in the above MEC layer for processing. Indeed, the MEC nodes can compute tasks, store data, and network with other nodes.

We assume that time is discretized into periods $\mathcal{T}=\{1,2,\dots ,T\}$ , with each period denoted by $t\in \mathcal{T}$ . At the start of each period, placement decisions are made to ensure the provision of NFV-based services within predefined QoS requirements. The system must decide, for each computing request, whether to utilize existing VNF instances or create new ones at time t. Reusing instances incur a reuse cost, which is generally lower than the instantiation cost of a new VNF instance. The length of each period strikes a balance between the system’s responsiveness to dynamic changes in user demands and network conditions and the computational overhead incurred.

The physical network is modeled as a directed graph $\mathcal{G}=(\mathcal{N},\mathcal{L})$ , where $\mathcal{N}$ represents the physical MEC nodes and $\mathcal{L}$ represents the physical links. Each node $n\in \mathcal{N}$ is characterized by the tuple [Math Processing Error] $\{c_n,{\mathcal{C}}_n,{\alpha }_n\}$ , where $c_n$ represents the maximum computing resource capacity, ${\mathcal{C}}_n$ the unit resource cost, and ${\alpha }_n$ , its reliability. Each link $l\in \mathcal{L}$ is characterized by the tuple [Math Processing Error] $\{b_l,d_l,{\mathcal{C}}_l,{\alpha }_l\}$ , where $b_l$ denotes the available bandwidth capacity, $d_l$ indicates the link delay, ${\mathcal{C}}_l$ represents the unit bandwidth cost, and ${\alpha }_l$ represents the reliability of the link l.

Let $\mathcal{K}$ be the set of available VNF types. Assume a licensing model where a maximum number $I^k$ of VNF instances may be instantiated across the network for VNF k, with [Math Processing Error] $i\in \{1,2,\dots ,I^k\}$ . Each VNF type $k\in \mathcal{K}$ is characterized by the tuple [Math Processing Error] $\{q^k,u^k,c^k,{\mathcal{C}}_n^k,{\mathcal{C}}_n^{i,k},{\alpha }^k\}$ , where $q^k$ represents the processing demand, $u^k$ and $c^k$ represent the resource and processing capacities, ${\mathcal{C}}_n^k$ is the instantiation cost for creating a new instance on node n, ${\mathcal{C}}_n^{i,k}$ is the cost of reusing instance i on node n, and ${\alpha }^k$ represents the reliability of VNF k.

Define the binary matrix $\mathbf{X}\left(t\right)\in {\{0,1\}}^{\left|\mathcal{N}\right|\times \left|\mathcal{K}\right|\times I^k}$ to indicate whether a new instance i of VNF k is deployed on node n at time t. Let the binary matrix $\mathbf{M}\left(t\right)\in {\{0,1\}}^{\left|\mathcal{N}\right|\times \left|\mathcal{K}\right|\times I^k}$ indicate whether instance i of VNF k is reused on node n at time t or not, and $\Gamma \in {\mathbb{R}}^{\left|\mathcal{N}\right|\times \left|\mathcal{K}\right|\times I^k}$ denotes the processing load already assigned to instance i of VNF k on node n.

Moreover, let $\mathcal{R}$ be the set of requests. A request $r\in \mathcal{R}$ is modeled as a directed graph [Math Processing Error] ${\mathcal{G}}_r=({\mathcal{K}}_r,{\mathcal{L}}_r)$ , where ${\mathcal{K}}_r$ is the set of VNFs to be installed on MEC nodes and ${\mathcal{L}}_r$ is the set of virtual edges. A request r is characterized by its delay threshold $d_r$ and reliability requirement ${\alpha }_r$ . Each virtual edge $l_r\in {\mathcal{L}}_r$ is characterized by a tuple [Math Processing Error] $\{d_{l,r},b_{l,r}\}$ , where $d_{l,r}$ and $b_{l,r}$ represent the delay and bandwidth requirement, respectively. Finally, the binary matrix $\mathbf{Y}\left(t\right)\in {\{0,1\}}^{\left|\mathcal{L}\right|\times |{\mathcal{L}}_r|}$ indicates whether a virtual link $l_r$ is mapped to a physical link l at time t or not.

In this section, we first outline the constraints necessary for our problem formulation, then provide detailed formulations of the associated costs, and finally formulate the objective function.

In this section, we propose a solution to the problem (P1), which addresses the total cost of SFC placement in MEC networks with redeployment schemes. We propose a DTQN-based approach that learns a policy for placing and routing network functions to ensure QoS requirements, considering real-time network conditions and operational objectives.

This section presents the simulations conducted to evaluate the performance of our proposed solution against existing baselines. We first describe the simulation environment, followed by an analysis of the obtained results.

6.2. Simulation Results and Discussion

We begin by examining the impact of increasing the number of nodes and the arrival rates $\delta$ for each algorithm. The average rewards per episode for each algorithm are presented in Figure 1, which considers 5 nodes with $\delta =30$ , while in Figure 2, we consider 10 nodes with $\delta =100$ . It is important to note that in each episode, network link mappings among nodes vary to capture the variability and dynamism of the network topology. Furthermore, strict adherence to QoS requirements is enforced for all requests, and partial satisfaction is not accepted. Consequently, only fully accepted requests are included in the results presented in Figure 1 and Figure 2.

For any $\delta$ (Figure 1 and Figure 2), the proposed DTQN solution consistently achieves the highest rewards, despite its greater computational demands and longer training times, compared to baseline algorithms such as DQN and Heuristic. While DTQN and D3T exhibit similar reward performance at a smaller scale, DTQN surpasses D3T as the number of users increases. In contrast, the Heuristic approach consistently yields the lowest rewards. This trend is further supported by Figure 3, which presents the average rejection rates over five simulation runs with $\delta =50$ . The results indicate that DTQN achieves the lowest rejection rate, followed by D3T and DQN, while the Heuristic approach has the highest rejection rate.

Notably, the performance gap between DTQN, D3T, and DQN in Figure 1 and Figure 2 widen as the number of nodes in the topology and $\delta$ increase. This is likely due to the enhanced ability of DTQN to capture temporal dependencies, eventually leading to more accurate SFC placement decisions that prioritize reuse over migration. As the request arrival rate $\delta$ increases and network resources become saturated, optimizing VNF placement and reuse decisions becomes crucial to avoid network overloading.

This observation is further validated by Figure 4 ( $\mathcal{N}=10$ , $\delta =30$ ) and Figure 5 ( $\mathcal{N}=10$ , $\delta =100$ ), which illustrate the SFC embedding strategies adopted by each evaluated approach. The Heuristic method, which achieves the lowest average rewards, exhibits a uniform distribution across new VNF instantiation, migration, and reuse placement decisions, regardless of the arrival rate $\delta$ . In contrast, the DQN baseline demonstrates a more adaptive strategy, adjusting the balance between VNF placement and reuse based on the value of $\delta$ . Meanwhile, DTQN and D3T-based solutions consistently prioritize VNF instance reuse across all values of $\delta$ . As the number of users increases, DTQN further distinguishes itself from D3T by prioritizing VNF instance reuse even more.

This preference can be attributed to several factors. First, the simulation parameters set the reuse cost lower than the migration cost, making VNF reuse more cost-effective than instantiating new VNFs. Moreover, migration induces extra bandwidth consumption, potentially jeopardizing QoS requirements and incurring penalties, particularly at high $\delta$ . By prioritizing VNF reuse, the DTQN-based solution minimizes unnecessary resource allocations and bandwidth usage, thereby maintaining a more stable and efficient MEC network. Furthermore, the enhanced ability of our solution to capture temporal dependencies enables more informed placement decisions, striking a strategic balance between immediate resource utilization and long-term network performance.

Figure 6, which illustrates the average costs of each algorithm as a function of the number of nodes over five simulation runs, supports the previous statement ( $\delta =50$ ). Despite having the highest rejection rate compared to DTQN, D3T and DQN, the Heuristic approach incurs the highest costs due to its inadequate placement strategy and excessive bandwidth consumption. At smaller scales, DQN incurs slightly lower costs than DTQN. However, it is imperative to note that DQN has a higher rejection rate, resulting in fewer satisfied requests. Moreover, as the number of nodes increases, DTQN’s costs decrease relative to DQN’s, since DQN’s suboptimal decisions have a greater impact than the difference in the number of requests served.

These findings suggest that Transformer-based RL models, such as DTQN and D3T, hold significant promise for complex decision-making tasks in dynamic networks, particularly in optimizing VNF placement strategies. By prioritizing VNF reuse over new instantiation and migration, DTQN achieves superior performance, cost efficiency, and resource optimization, especially under varying network demands.

This paper presents a comprehensive solution for VNF placement and utilization in MEC networks, specifically tailored for mission-critical applications with stringent QoS requirements. Our DTQN-based approach effectively addressed the dynamic nature of MEC environments by leveraging advanced RL techniques to optimize VNF placement strategies. Simulation results demonstrated the superiority of the proposed method in achieving cost efficiency, minimizing service disruptions, and ensuring better resource utilization compared to baseline approaches. The emphasis on reusing VNFs rather than creating new instances or migrating VNFs highlights the crucial role of smart resource management in supporting dynamic networks and minimizing operational expenses. These findings suggest that Transformer-based RL models are promising for solving complex decision-making tasks in dynamic and resource-constrained network environments. In future work, we will investigate the scalability challenge of the proposed DTQN-based solution in large-scale network environments.