Austin, TX – April 2, 2025 – CyberRatings.org (CyberRatings), the non-profit entity dedicated to providing confidence in cybersecurity product efficacy, today released its Q1 2025 Comparative Test Report on Cloud Network Firewalls (CNFW), along with separate, in-depth reports for each of the ten cloud firewall solutions tested. Security effectiveness results ranged from 0% to 100%.
Key Findings:
- Third-party firewalls from Check Point, Fortinet, Juniper Networks, Palo Alto Networks, and Versa Networks demonstrated the highest security effectiveness blocking exploits and evasion tactics. Results ranged from 99.61% to 100%.
- Native cloud firewalls from Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure offer a convenient alternative, but all received 0% Security Effectiveness as they allowed attacks to bypass existing defenses.
- Google Cloud Platform’s Next Generation Firewall (NGFW) service leverages Palo Alto Networks technology. We attribute the differences in security effectiveness and performance results between the two platforms to each provider independently selecting and deploying different software versions based on their own criteria.
- A total of six firewall solutions were Recommended and four received Caution ratings.
In the Cloud Service Provider Native Firewall test from November 2024 only 522 exploits were used in the Part 1 “Mini-Test”, but not evasions. For this round of testing, a greater number of exploits were deployed, and evasions were introduced to the test samples:
- False Positives: 2,760 samples from various business-critical files and applications, ensuring security measures did not disrupt legitimate traffic.
- Exploits: 2,028 attack samples from widely exploited vulnerabilities in enterprise environments.
- Evasion Techniques: 2,500 attacks spanning 27 evasion techniques tested across multiple network layers to bypass firewall defenses.
- Performance Metrics: 46 different stress and capacity tests under diverse workloads.
- Stability & Reliability: Seven extended tests simulating prolonged real-world attack and operational scenarios.
CyberRatings evaluated firewall security by testing for evasion detection at three separate layers of the Open Systems Interconnection (OSI) model, specifically Layers 3, 4, and 7. Missing lower-layer evasions had the greatest impact on the overall score because these layers form the foundation of firewall security at the fundamental networking level, and when these lower layers are compromised, the firewall’s primary protective function is undermined. Points were deducted based on the firewall’s ability—or inability—to detect evasions:
- A missed evasion from the Layer 3 level resulted in a 50% deduction per category, up to a potential category maximum reduction of 100%.
- Missing a Layer 4 evasion led to a 20% deduction per category, up to a potential category maximum reduction of 60%.
- A miss at Layer 7 incurred only a 1% deduction per category, up to a potential category maximum reduction of 10%.
Layers 3 and 4 evasions are particularly concerning since all modern applications rely on IP and TCP. Vulnerabilities at these layers can be exploited across a wide range of systems—from cloud services to enterprise applications.
“Until cloud service provider native firewalls provide better protection, customers should be looking to third parties for their cloud security needs,” said Vikram Phatak, CEO of CyberRatings.org. “Traditional third-party security vendors have demonstrated that they bring significant value to customers.”
Below is a summary of the Ratings:
The cloud firewalls were tested using Keysight’s CyPerf v5.0 software testing platform in addition to CyberRatings’ in-house developed test tools. Enterprises can easily perform similar testing with a 2-week free trial from Keysight. Further details of the CyPerf strike library can be found here: https://www.keysight.com/us/en/products/network-test/cloud-test/cyperf.html