How effective are the Cloud Service Provider (CSP) native cloud firewall offerings?

Introduction

In today’s cloud-centric environment, businesses often face a critical choice regarding the security of their cloud infrastructure. They can rely on firewalls offered directly by Cloud Service Providers (CSPs) or use independent security vendor firewall offerings typically available through the respective CSP’s marketplace.

Multiple factors influence this decision, including integration, billing convenience, security capabilities, operational needs, and cost. Security effectiveness is a crucial factor in selecting the right firewall solution, as it directly impacts the organization’s ability to protect against cyber threats. This makes objective testing essential to ensure firewalls are protecting as expected.

Background

In April 2024, we published the results of our annual Cloud Network Firewall test. In that test, the AWS Network Firewall exhibited a mere 5.39% Security Effectiveness score, the lowest result in our comparison. At that time, it was also the only CSP native firewall under test. That extremely low Security Effectiveness score was not considered to meet any reasonably acceptable standard, and it was concerning enough that we decided to re evaluate AWS’ offering six months later to see if any improvements might have been made.

For this round, we expanded our testing to include Microsoft Azure Firewall and Google Cloud Platform (GCP) Cloud NGFW. Combined, the “Big Three” now account for two-thirds of the growing cloud market, estimated to have an annualized run rate of a $300-billion market growing at 21% and is expected to “double in size in the next four years.” ¹

This testing was conducted as part one of a two-part series examining cloud network firewalls. The firewalls were tested against an array of exploits using Keysight’s CyPerf v5.0 software testing platform, offering an evidence-based look at how well these native solutions withstand real-world security threats. We chose Keysight’s CyPerf testing tool so that enterprises could easily replicate our results. This first phase provides end-users with insights into the effectiveness of CSP native firewall offerings across three CSPs, helping organizations understand each CSP’s security offerings and capabilities. These findings also set the stage for part two, which will feature a series of upcoming tests evaluating cloud network firewall solutions from select industry leading vendors. Together, these two phases will enable organizations to compare firewalls based on tested effectiveness, helping them make well-informed decisions when selecting the most appropriate firewall solution for their cloud environment.

Part 2 testing of third-party vendor cloud network firewall offerings is scheduled for publication in Q1 of 2025.

Key Findings

• Protection ranged from 0.38% to 50.57% for security effectiveness.
• Based on the results, Security Effectiveness varies significantly between Amazon, Microsoft, and Google, while remaining low overall.
• Enterprises may be accepting reduced security effectiveness when choosing a CSP native firewall over a third-party firewall.

Recommendations

• Organizations using CSP native firewall solutions should scrutinize whether the performance exhibited by these offerings will meet their needs.
• Customers considering CSP native firewalls should gather as much testing and data as possible on their effectiveness to understand whether the offerings align with their needs.

Test Topology

For this test, we employed 522 attacks (exploits), focusing on exploit types that target servers and are typically relevant to cloud workload deployments.

We used exploits from the last ten years, focusing on attacks with a severity of medium or higher. The attacks used included those targeting enterprise applications that businesses may be running and that could potentially be migrated to a cloud platform. This set included attacks targeting Apache, HPE, Joomla, Cisco, Microsoft, Oracle, PHP, VMware, WordPress, and Zoho ManageEngine.

This part one test was not intended to be a comprehensive security evaluation of the vendor platforms’ full capabilities or overall effectiveness. Instead, it focused on the exploit protection delivered by vendors for the set of vulnerabilities tested. The results of this particular test should not be construed as representative of the overall effectiveness or capabilities of the firewall platforms tested.

The figure below shows the testing topology.

1. Amazon Maintains Cloud Lead as Microsoft Edges Closer