A Unified Snort–OSSEC Hybrid Intrusion Detection Framework for Cloud Security

1.1. Research Objectives

The objectives of this study are as follows:

1. To design and implement a hybrid IDS framework that integrates Snort and OSSEC for cloud environments.

2. To simulate attack scenarios and evaluate the detection performance.

3. To compare the performance of Snort-only, OSSEC-only, and the hybrid IDS for accuracy, detection rate, and false alarm rate.

4. To benchmark the proposed framework against previous IDS studies to show its novelty and practical contribution.

1.2. Scope of the Study

This study was conducted on a virtualized Ubuntu-based cloud testbed running OwnCloud, Apache, and MariaDB. The hybrid intrusion detection system was implemented using two widely adopted open-source tools: Snort for network-level monitoring and OSSEC for host-level monitoring. Four attack categories, port scanning, denial-of-service (DoS), brute-force, and user-to-root (U2R), are the basis of the assessment, which represent common threats in cloud environments. The focus of this study was on improving detection performance and reducing false alarms through cross-layer correlation. The need for scalability to large multi-tenant deployments and integration with prevention mechanisms was outside the present scope and is recommended for future research.

2.1. Cloud Services and Security Challenges

Cloud services are typically categorized into Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), and Software-as-a-Service (SaaS). The users are exposed to VM-to-VM attacks and hypervisor vulnerabilities through IaaS, Insecure API and multi-tenancy issues are mostly from PaaS, while confidentiality and compliance issues are mostly the concerns from SaaS based deployments. Previous studies outlined these risks, but often the emphasis is always on governance and policy over the integration of IDS mechanisms [5]. More recent studies predict sustained growth in cloud service adoption, but give little or no attention to the implementation of IDS within these contexts [6]. It became necessary to address these gaps by developing and deploying adaptable intrusion detection solutions that span service layers.

2.2. Attack Vectors in Cloud Environments

Cloud infrastructures face various attack vectors. Side-channel exploits, which are virtual-machine-based attacks, demonstrate how easily privilege escalation can occur across tenants. User-to-root (U2R) exploits have been studied with classifiers such as Naïve Bayes and ANN, even though detecting these exploits remains difficult given their low frequency [7]. The prevalence of DoS and DDoS threats continues to rise, with distributed detection frameworks often burdened by computational overhead [1]. Datasets with Backdoors and adversarial attack vectors represent growing risks for IDS robustness, which shows the need for anomaly-based detection that can withstand the evolving threats [3,8].

There have been Hybrid IDS approaches that attempt to integrate N-IDS and H-IDS techniques. The early designs showed great promise but could not be validated at the host level, and in some cases were tested only on simulated datasets [9]. In a study by [10], a two-phase hybrid IDS was proposed that improved detection rates; however, the work suffers from high false-positive rates. A recent survey on AI-driven models reported very high accuracy on benchmark datasets, and some hybrid CNN–RNN designs have achieved accuracy levels exceeding 98% [11]. However, these systems often compromise interpretability and require high computation, which limits deployment in resource-constrained cloud environments [12].

Recent and advanced studies emphasize the use of adaptive and federated approaches. [1], in their research introduced a hybrid IDS with the combination of XGBoost and deep learning components on various datasets. This model improved accuracy but also raised concerns about generalizability. The study by [4] complemented transformer-based models with graph features, which achieved almost perfect accuracy, but the implementation wasn’t validated in a real-world environment. In [7], a deep learning IDS for IoT networks was introduced; however, its ability to perform optimally and effectively in cloud environments remains uncertain. These studies highlight the importance of adaptive, explainable, and scalable IDS for detecting unknown attacks and reducing false alarms, but they also indicate a persistent gap in practical, open-source deployments. From this review, three main gaps were noted:

1. Most IDS frameworks place more emphasis on either known signature detection or anomaly-based detection. The combination of both techniques is rarely explored, even though it is effective in a real cloud testbed.

2. Previous hybrid systems lack empirical validation of cross-layer integration that consists of network and host to reduce false positives.

3. Recent AI-based IDS studies tend to achieve high accuracy but often compromise deployability and interpretability in cost-sensitive environments.

This study addresses these gaps with the implementation and validation of a Snort–OSSEC hybrid IDS with correlated alerting, tested against multiple attack classes, and the performance evaluated based on metrics such as accuracy, detection rate, and false alarm rate.

3.1. Study Aim and Design

The primary aim of this study was to design and evaluate a hybrid intrusion detection system (IDS) that integrates Snort, a network-based IDS, and OSSEC, a host-based IDS, to enhance cloud security. A quasi-experimental testbed design was used, which involves controlled attack simulations against a cloud-based environment. The performance of three configurations Snort-only, OSSEC-only, and Snort–OSSEC hybrid was also compared to determine the relative advantages of the hybridization technique employed.

Figure 1 illustrates how Snort N-IDS and OSSEC H-IDS operate independently and then correlate alerts through a combined analysis engine, depicted as the central controller. It shows the flow of data from network traffic and system logs to detection and correlation.

3.2. Study Setting

The study was conducted in an isolated laboratory environment that was configured to replicate a realistic cloud service. An Ubuntu 20.04 LTS server was deployed as the primary cloud platform, running OwnCloud to simulate cloud storage and collaboration services. Supporting services included MariaDB for database management and the Apache HTTP Server for web hosting. This setup provided a clear multi-tier cloud architecture for testing intrusion detection mechanisms under real-world conditions.

3.3. Materials and Tools

The following are the materials used in this study that include both software and hardware:

1. IDS Tools: Snort version 2.9.17 for packet inspection, OSSEC version 3.7.0 for host log and integrity monitoring.

2. Traffic Capture and Analysis: Wireshark version 3.4.7.

3. Attack Tools: Nmap for port scanning, Hping3 for DoS traffic generation, Hydra for brute-force login attempts, and Metasploit for U2R privilege escalation.

4. Dataset Source: The primary dataset used was generated from simulated attacks on the isolated testbed, which consists of 14,072 labeled connections.

5. Hardware Environment: Physical and virtualized PC running Ubuntu as the server and Attacker PC on VMware ESXi with 8-core CPU, 32 GB RAM, and 100 GB allocated storage.

Figure 2 shows the workflow: network traffic is first filtered by protocol type (TCP, UDP, ICMP, HTTP/FTP), then passed to a processing queue for analysis against a pattern database by the detection engine. All matches generate alerts that are logged and trigger notifications, while regular traffic is archived. This layered process of detection and reporting shows the integration of Snort (N-IDS) and OSSEC (H-IDS) for cross-validation and improved accuracy.

The attack simulation structure is detailed in Figure 3, which shows the complete experimental process.

3.3.1. Attack Simulation Process

Controlled simulations were conducted to validate the system’s detection capabilities across multiple attack categories. Open-source security tools were employed to generate attacks, ensuring reproducibility. Nmap was used for port scanning by probing open ports and identifying vulnerable services; Hping3 generated TCP/ICMP floods targeting the OwnCloud server to simulate Denial-of-Service (DoS) attacks; Hydra attempted repeated password logins against FTP services for brute-force attacks; and Metasploit modules were utilized to simulate privilege escalation on the Ubuntu server for user-to-root exploits.

Each attack was executed against the testbed with corresponding Snort rules and OSSEC alerts monitored in real time. By including both frequent (port scans, DoS) and rare (U2R) categories, the review covered a wide range of threat scenarios.

The testbed was deployed on an Ubuntu environment, which hosts the OwnCloud server with Apache and MariaDB, as shown in Figure 3. Snort was configured as a network intrusion detection system (N-IDS) to monitor packet-level traffic, while OSSEC was configured as a host intrusion detection system (H-IDS) to log and monitor system integrity. An attacker machine was configured to generate malicious traffic using Nmap, Hping3, Hydra, and Metasploit, while Wireshark was used for packet capturing and analysis. This setup enabled controlled evaluation of the hybrid IDS under real-world cloud attack scenarios.

This diagram, as shown in Figure 4, indicates the process by which attack simulations, such as port scanning, denial-of-service (DoS), brute-force login, and user-to-root (U2R), are launched from the attacker machine against the cloud testbed. Snort monitors network traffic, while OSSEC analyzes system-level logs. The alerts triggered from both sources are forwarded to the hybrid correlation module, where a cross-validation technique is in place to improve detection accuracy and reduce false alarms.

In this study, ‘unknown threats’ refer to attacks that are not included in the initial Snort signature sets or the predefined OSSEC rules. Detection was enabled through anomaly-based mechanisms: Snort leveraged preprocessors to identify unusual traffic behaviors, such as abnormal packet sizes and protocol misuse, while OSSEC used integrity checks and system log correlation to capture deviations from baseline user activities.

This dual-layer anomaly detection ensured that threats absent from signature databases could still be flagged as suspicious, supporting claims of “unknown” detection capability.

3.4. Data Collection Procedures

The benign traffic was initially generated by cloud services during regular user activities. While Wireshark captures network-layer packets, OSSEC simultaneously monitors host-level system events and file integrity, providing comprehensive visibility across both network and host environments. Upon completion of baseline profiling, controlled attacks were carried out against the testbed using the tools listed above. Every attack scenario was repeated several times for consistency. Captured traffic and logs were stored in packet capture (pcap) and syslog formats for preprocessing. All experiments were carried out in an isolated laboratory testbed that had no connection to external networks. By doing that, it ensured that simulated attacks posed no risk to production systems or third-party infrastructure.

3.5. Data Preprocessing and Feature Extraction

Initially, 31 attributes were recorded from the Snort logs; after normalization and feature selection, only 26 were retained. These attributes include protocol type, source and destination ports, connection duration, TCP flags, packet size, and byte counts. Additionally, the OSSEC host-based IDS (H-IDS) contributed features such as failed login attempts, file integrity modifications, and unauthorized root access events. All the features were then presented in a tabular form to create a structured dataset that is suitable for classification as shown in Table 1.

3.6. Interventions and Comparisons

Three system configurations were tested:

1. Snort-only (N-IDS)—network-level detection only.

2. OSSEC-only (H-IDS)—host-level detection only.

3. Hybrid Snort–OSSEC IDS—correlated alerts across host and network layers.

Each configuration underwent an evaluation phase based on identical attack scenarios, allowing for a comparative analysis of accuracy, detection rate, and false alarm rate.

3.7. Classifier Justification: Naïve Bayes

The justification for the selection of the Naïve Bayes classifier for the evaluation of the detection performance is due to its suitability for categorical and discrete data that are derived from Snort and OSSEC logs. Its advantages include:

1. Computational efficiency: suitable for real-time IDS applications with large datasets.

2. Noise tolerance: Its effectiveness in handling imperfect or incomplete network traffic data.

3. Transparency: its ability to provide interpretable probability outputs, enhancing explainability in cloud security.

While deep learning methods have achieved higher accuracy in some IDS studies, they usually require substantial computational resources and lack interpretability. The lightweight and explainable nature of Naïve Bayes makes it appropriate for practical, cost-sensitive cloud deployments.

3.8. Evaluation Metrics

The Naïve Bayes classifier was used to evaluate detection performance based on its efficiency, noise tolerance, and interpretability [12]. Performance metrics were calculated from confusion matrices using the following formulas (Equations (1)–(3)):

• i.

  Accuracy

• ii.

  Detection Rate

• iii.

  False Alarm Rate

4.1. Experimental Setup in Relation to Methods

As outlined in Section 3.4, the evaluation was conducted in an Ubuntu OwnCloud testbed where four attack scenarios, port scanning, DoS, brute-force, and U2R, were simulated using Nmap, Hping3, Hydra, and Metasploit. The three IDS configurations described in Section 3.6 Interventions and Comparisons were tested separately: Snort-only, OSSEC-only, and the hybrid Snort–OSSEC framework. The primary dataset used for evaluation comprised 14,072 labeled connections generated during these experiments.

4.2. Classifier Evaluation: Confusion Matrix

Using the dataset described in Section 3.5 and the Naïve Bayes classifier introduced in Section 3.7, the classifier’s performance was first evaluated in terms of its confusion matrix. Table 2 presents the distribution of classification outcomes across normal and anomalous traffic. Out of a total of 7,446 normal connections, 7,430 were correctly classified, while only 16 were misclassified as anomalies. For the anomalous connections, out of 6626 connections, 6239 were correctly detected, while 387 were misclassified as usual.

4.3. Classifier Evaluation: Performance Metrics

From the confusion matrix, the standard performance measures (accuracy, detection rate, false alarm rate, precision, F-measure, MCC, ROC area, PRC area) were calculated using the formulas defined in Section 3.8. The results are shown in Table 3.

These results demonstrate the classifier’s ability to discriminate effectively between normal and anomalous connections, with ROC and PRC areas above 0.99, confirming robustness.

4.4. Class and Protocol-Based Analysis

To further analyze the distribution of detection, classification outcomes were visualized.

Figure 5, the Connection Class Chart, illustrates the classification split between normal and anomalous traffic, showing a strong bias toward correct predictions in both categories.

Additionally, Figure 6 presents the traffic analysis by protocol (TCP, UDP, ICMP, HTTP/FTP), illustrating the system’s coverage of diverse network activities. The illustration shows that the IDS successfully handled protocol heterogeneity, validating the importance of protocol-aware detection in hybrid frameworks.

4.5. Standalone IDS Performance

The performance of Snort and OSSEC when deployed independently was assessed. As detailed in Section 3.6, both IDSs were evaluated using the same traffic dataset. Table 4 summarizes the results.

4.6. Hybrid IDS Performance

When Snort and OSSEC were integrated, the hybrid IDS achieved significantly higher performance. Table 5 and Figure 7 and Figure 8 compare the hybrid IDS to the standalone systems.

Table 5 compares Snort-only, OSSEC-only, and the proposed hybrid IDS in terms of accuracy, detection rate, and false alarm rate. While the standalone systems performed well individually, each exhibited critical blind spots: Snort failed to detect host-level exploits, whereas OSSEC was less effective against reconnaissance attacks. The hybrid IDS overcame these gaps, achieving the highest accuracy (97.1%) and detection rate (94.1%) while reducing the false alarm rate to just 0.2%. These results confirm the advantage of cross-layer correlation in delivering a more reliable intrusion detection solution for cloud environments.

The grouped bar chart, as shown in Figure 7, compares Snort-only, OSSEC-only, and the hybrid IDS across accuracy, detection rate, and false alarm rate. The hybrid IDS clearly outperformed standalone systems, achieving the highest accuracy (97.1%) and detection rate (94.1%), while reducing the false alarm rate to just 0.2%.

The line chart, as shown in Figure 8, illustrates the trend of performance metrics across IDS configurations. While both Snort and OSSEC performed well individually, the hybrid IDS consistently achieved superior results across all metrics, confirming the advantage of cross-layer correlation.

The hybrid IDS clearly outperforms the standalone systems, achieving both higher detection rates and significantly lower false alarms.

4.7. Comparative Analysis with Previous Studies

Finally, the performance of the hybrid IDS was benchmarked against prior studies.

The bar chart in Figure 9 compares the performance of the proposed hybrid IDS with five representative studies [1,4,10,11]. While previous work achieved strong accuracy, false alarm rates remained relatively high, ranging from 0.9% to 2.1%. By contrast, the proposed hybrid IDS achieved a balanced performance with 97.1% accuracy and a 0.2% false alarm rate, demonstrating superior efficiency and reliability in a real-world testbed environment.

5.1. Case-Based Evaluation of Hybrid Correlation

Case-specific examples from the attack simulations in Section 3.4 highlight the advantages of the hybrid framework. In the case of a port scan, Snort detected a TCP SYN scan with medium confidence while OSSEC logged multiple failed login attempts; together, the correlated alerts confirmed malicious intent and eliminated what would have been a false positive in Snort-only mode. During the brute-force login attempt, OSSEC detected multiple failed FTP login attempts, which Snort corroborated by identifying abnormal TCP connection patterns, thereby distinguishing the incident from ordinary user error. For a user-to-root (U2R) exploit, OSSEC detected privilege escalation while Snort traced suspicious pre-attack reconnaissance, producing a more comprehensive forensic trail. Results from the cases suggest that hybrid correlation improves detection rates, validates alerts, reduces false positives, and enhances forensic analysis.

Comparative analysis with previous studies, as shown in Figure 9 further positions this work within the state of the art. Deep learning-based systems, such as those proposed by [1,4], achieved high accuracy but are computationally expensive and often lack transparency. Similarly, hybrid ML–DL frameworks [7,11] were optimized for IoT environments rather than cloud infrastructures. The Snort-OSSEC hybrid IDS offers balanced accuracy and efficiency through a practical, open-source alternative accessible to enterprises with limited computational resources.

The Snort–OSSEC hybrid model also presents certain limitations. The system was evaluated within a small-scale cloud testbed and restricted to four attack types, excluding more sophisticated adversarial techniques such as ransomware and advanced persistent threats (APTs). Moreover, while the hybrid IDS demonstrated high accuracy, it lacks adaptive learning capabilities for detecting zero-day threats and operates strictly as an IDS rather than an IPS. The limitations that come with the hybrid model are a prerequisite for future research as outlined in Section 6, which includes scaling the framework to multi-tenant clouds, widening the attack coverage, inclusion of adaptive learning, and evolving toward automated prevention mechanisms.

5.2. Open Challenges and Future Work

Although the proposed hybrid IDS demonstrated strong results in a controlled testbed, several limitations highlight opportunities for future research. First, the evaluation was restricted to a small-scale virtualized cloud environment. Future research should explore the validation of the framework on a large-scale, multi-tenant infrastructure to examine the scalability and performance of the model when used in a diverse workload. Furthermore, simulation was carried out on four categories of attacks: port scanning, denial-of-service (DoS), brute-force login, and user-to-root (U2R). Expanding the model’s attack set to include ransomware, insider threats, and advanced persistent threats (APTs) would enable a critical evaluation of system effectiveness.

The study also lacks adaptive learning mechanisms. Although the hybrid model effectively reduced false alarms, its capacity in detecting zero-day attacks remains limited. The inclusion of online learning, federated learning, or reinforcement learning can further enhance the model’s adaptability against evolving threats. Finally, the system functionality is currently just an intrusion detection system. Transitioning the system into an intrusion prevention system (IPS) by automating responses such as virtual machine isolation or IP blocking would substantially enhance its practical utility, but it also introduces potential concerns regarding service continuity. Addressing these open challenges will be critical to advancing hybrid IDS solutions into enterprise-ready, next-generation cloud security frameworks.

5.3. Contribution of the Study

This study contributes to cloud security research by proposing and validating a hybrid intrusion detection system (IDS) that integrates Snort (N-IDS) and OSSEC (H-IDS) for cross-layer alert correlation. Unlike previous works that relied solely on either standalone signature-based systems or computationally intensive deep learning frameworks, the proposed model demonstrates cost-effectiveness, interpretability, and scalability. It provides a transparent detection process, reduces false alarms through dual validation, and offers a strong foundation for forensic analysis by combining network- and host-level mechanisms. The research further contributes by offering an open-source, real-world testbed implementation, thus bridging the gap between theoretical models and practical deployment in enterprise cloud infrastructures.

5.4. Key Findings

The results confirm three major findings:

1. Superior Performance of Hybrid IDS: The hybrid framework achieved 97.1% accuracy, 94.1% detection rate, and a remarkably low 0.2% false alarm rate, outperforming both Snort-only and OSSEC-only deployments.

2. Validated Case-Based Detection: Attack simulations showed that hybrid correlation not only improved detection but also reduced false positives and enriched forensic trails, particularly in cases such as port scanning, brute-force login, and user-to-root exploits.

3. Balanced Contribution to State of the Art: Compared with previous IDS research, the hybrid IDS delivered competitive accuracy while maintaining efficiency and interpretability, offering a practical middle ground between lightweight standalone systems and complex deep learning frameworks.