Cloud Network Firewall

In Q1 2025, CyberRatings.org conducted independent evaluations of ten leading cloud firewall solutions using the Cloud Firewall Test Methodology v3.0. These assessments included offerings from cloud service providers (CSPs) and independent security vendors, typically available through CSP marketplaces.

The evaluation covered key performance metrics: security effectiveness, false positive management, TLS/SSL 1.2 and 1.3 support, and system stability under adverse conditions. The firewalls were tested using real-world attack scenarios, enterprise-grade workloads, and adversarial evasion techniques to measure resilience, reliability, and performance.

CyberRatings evaluated firewall security by testing for evasion detection at three separate layers of the Open Systems Interconnection (OSI) model, specifically Layers 3, 4, and 7. Missing lower-layer evasions had the greatest impact on the overall score because these layers form the foundation of firewall security at the fundamental networking level, and when these lower layers are compromised, the firewall’s primary protective function is undermined. Points were deducted based on the firewall’s ability—or inability—to detect evasions:

• A missed evasion from the Layer 3 level resulted in a 50% deduction per category, up to a potential category maximum reduction of 100%.
• Missing a Layer 4 evasion led to a 20% deduction per category, up to a potential category maximum reduction of 60%.
• A miss at Layer 7 incurred only a 1% deduction per category, up to a potential category maximum reduction of 10%.

Layers 3 and 4 evasions are particularly concerning since all modern applications rely on IP and TCP. Vulnerabilities at these layers can be exploited across a wide range of systems—from cloud services to enterprise applications.

For the Q1 2025 publication, ten market leading Cloud Network Firewall vendors were in this comparative test. Six products were Recommended, and four received a Caution rating. Security Effectiveness scores ranged from 0% to 100%.

In the Cloud Service Provider Native Firewall test from November 2024 only 522 exploits were used in the Part 1 “Mini-Test”, but not evasions. For this round of testing, a greater number of exploits were deployed, and evasions were introduced to the test samples:

• False Positives: 2,760 samples from various business-critical files and applications, ensuring security measures did not disrupt legitimate traffic.
• Exploits: 2,028 attack samples from widely exploited vulnerabilities in enterprise environments.
• Evasion Techniques: 2,500 attacks spanning 27 evasion techniques tested across multiple network layers to bypass firewall defenses.
• Performance Metrics: 46 different stress and capacity tests under diverse workloads.
• Stability & Reliability: Seven extended tests simulating prolonged real-world attack and operational scenarios.

The test reports are provided in two ways:

• In-depth individual test reports for each vendor
• A comparative report summarizing test results for all of the vendors in the test.

The methodology and the comparative SVM graphic are provided to the community at no cost, while the reports are available for purchase. Of the individual reports, those that received a “Caution” rating are available to all members free of charge.

We would like to thank Keysight Technologies for providing their CyPerf and BreakingPoint tools for us to test security, performance, TLS functionality, and stability of Cloud Network Firewalls.