In April 2024, we published the results of our annual Cloud Network Firewall test. In that test, the AWS Network Firewall exhibited a mere 5.39% Security Effectiveness score, the lowest result in our comparison. That extremely low Security Effectiveness score was not considered to meet any reasonably acceptable standard, and it was concerning enough that we decided to re-evaluate AWS’ offering six months later to see if any improvements might have been made.
We expanded our testing to include Microsoft Azure Firewall and Google Cloud Platform (GCP) Cloud NGFW and published those results on November 26, 2024. Combined, the “Big Three” now account for two-thirds of the growing cloud provider market. Our goal was to assess their native firewall capabilities and understand their strengths and limitations against a variety of attacks.
Part One of a Two-Part Test
This testing was conducted as part one of a two-part series examining cloud network firewalls. It was limited to a subset1 of exploits from Keysight’s CyPerf v5.0 software testing platform. In part two, we will increase the difficulty of the test by widening the scope and depth of testing. The second part of the test will also compare cloud service provider native offerings against market leading third-party cloud network firewall providers. Publication of part 2 will be in Q1 of 2025.
Retesting AWS Network Firewall
Here’s a rundown of our journey with the AWS Network Firewall, including our steps and what we discovered along the way. We decided to publish these additional details so that customers of AWS Firewalls can investigate how the firewall would work in their environments.
Setting Up and Initial Testing
First, we set up the AWS Network Firewall with its basic firewall features, such as access control rules. We wanted to ensure that the fundamental functionalities were working correctly before diving deeper. During our initial testing, these features functioned as expected, effectively controlling access according to our configurations. Then, we executed a series of attacks to test the firewall’s threat detection capabilities. Surprisingly, the AWS Network Firewall only blocked two out of all the attacks we launched. This unexpected result prompted us to question whether the firewall was correctly configured or if there was an issue with our setup.
Double-Checking the Configuration
To rule out any misconfigurations on our part, we carefully double-checked our firewall settings against the official AWS documentation. We reviewed each configuration parameter, ensuring everything matched exactly as prescribed. Our review confirmed that the configuration was indeed correct.
Validating the Firewall Deployment
We conducted additional tests to verify that the AWS Network Firewall was deployed correctly and that all network traffic was routed through it rather than bypassing the firewall and going directly to the internet. We transmitted both normal and malicious traffic across the firewall to observe its behavior.
All traffic forwarded or blocked by the firewall rules was accurately logged in the AWS flow and alert logs via AWS CloudWatch. This validation step reassured us that the firewall was operational and actively monitoring the network traffic as intended.
Diving Deeper with Detailed Analysis
Next, we tested specific threat detection capabilities to confirm that the firewall was functioning correctly. We selected three Suricata rules from the AWS-managed ruleset and generated traffic that matched the malicious content described in those rules.
Case Study: The daddy.linkpc.net Domain
One of the domains we tested was daddy.linkpc.net. The Suricata signature associated with this domain is designed to drop any traffic attempting to access it. Here were the steps we took:
-
- Attempted to Resolve the Domain: We tried to resolve daddy.linkpc.net, mimicking a user or system attempting to access a malicious domain.
- Firewall Detection: The AWS Network Firewall detected this malicious activity using the predefined security signature (Signature ID 2853242).
- Action Taken: The firewall blocked the DNS query, effectively preventing any communication with the malicious domain.
- Observation: The alert triggered by this activity was visible in the AWS Alert Logs, confirming that the firewall’s detection mechanisms were working as expected for this case.
At this stage, we theorized that most of the attacks bypassed the firewall because it lacked specific rules for those particular attacks. To investigate this possibility, we decided to delve deeper into the firewall’s ruleset.
Matching CVEs from AWS Suricata Rules to CyPerf Attacks
Suricata is a high-performance, open-source network analysis and threat detection software. It’s widely used by both private and public organizations and is embedded by major vendors to protect their assets. Suricata uses a signature-based approach to detect threats, which relies on predefined rules and patterns.
CVE stands for Common Vulnerabilities and Exposures. It is a glossary that classifies vulnerabilities. Its purpose is to identify, define, and catalog publicly disclosed cybersecurity vulnerabilities.
To test our theory, we needed to determine whether the AWS Network Firewall had rules corresponding to the attacks Cyperf was using. Here’s how we approached it:
- Extracted CVEs from Suricata Rules: We extracted all the CVEs listed in the Suricata rules used by the AWS Network Firewall.
- Matched CVEs with CyPerf Attacks: We compared these CVEs with the attacks available in CyPerf to identify those that matched.
- Executed Matched Attacks: We then executed only the attacks with corresponding CVEs in the firewall’s ruleset.
The Result: We found very few rules matched CVEs. This is because many Suricata rules did not contain CVE details, and out of those that did, the AWS Firewall only blocked two attacks. Further, most of the signatures were not relevant for cloud environments.
Observations and Concerns
Through our analysis, we observed the following about AWS Network Firewall’s capabilities:
- Reliance on Basic Regular Expressions: Many signatures in the firewall’s ruleset rely on simple regular expressions. While effective for straightforward patterns, this approach can be insufficient against more sophisticated or obfuscated attacks.
- Design Focus: Many signatures appear to be designed for home or small office environments rather than for cloud or server workloads. This could limit the firewall’s effectiveness in enterprise or cloud-native scenarios.
- Total Signatures and Limitations: As of October 2024, the signatures contained over 21,500 preconfigured rules. These signatures are organized into groups, each containing thousands of signatures. Therefore, to enable one, you must enable all in the group. The AWS Firewall also limits the number of signatures you can enable. The maximum number of signatures is 30,000 for stateful and stateless firewall
- Distribution of Rules:
- Approximately 10% of the rules protect web browsers.
- Around 17,500 rules monitor outbound connections (post-infection activities).
- Only about 4,000 rules focus on inbound traffic.
- Lack of CVE Information: Most signatures do not contain CVE details, making it challenging to assess the firewall’s coverage against known vulnerabilities.
Assessing the Severity of Limitations
To illustrate the potential impact of these limitations, let’s consider an example involving domain-based signatures:
Example: Circumventing Domain-Based Signatures
- Signature Limitation: Suppose there’s a signature designed to block traffic to cnn.com.
- Potential Circumvention: An attacker could easily bypass this signature by slightly altering the domain name to cnn.com, using a subdomain like news.cnn.com, or adding a character before or after cnn.com.
- Issue: If the signature only matches the exact domain cnn.com or tries to match a precise number of characters, it fails to detect these variations, allowing malicious traffic to pass undetected.
- Implication: This example demonstrates how attackers can exploit simple signatures by making minor changes, highlighting the need for more robust and comprehensive detection mechanisms.
Our Journey with Other Cloud Service Provider Firewalls
Azure
Unfortunately, we came into this test blind; Azure seems to use a propriety, closed-source security stack. It provides little visibility into how it works or how to change anything. You pay for it, turn it on, and it does “stuff.” This limitation is frustrating, given the results, as it removes the user’s ability to adjust protection based on their unique deployment.
The lack of visibility into rules and/or related CVE information limits the user’s ability to understand how well the protection aligns with their environment, applications, and workloads.
Google Cloud Firewall (GCP)
Like Azure, this is also closed-source security software. However, we discovered that GCP’s firewall is running software from Palo Alto Networks. We have worked with Palo Alto Networks for many years and expected much better results than we saw reflected in the scores. GCP directs you to the Palo Alto Networks website for information on the deployed protection.
Recommendations:
- Third-Party Firewall: Until Cloud Service Provider’s Native Firewalls offer adequate protection, we recommend using a third-party cloud firewall (offered through the respective CSP’s marketplace).
- Custom Rule Creation: Consider creating custom rules tailored to your specific environment and attack surface.
- Supplementary Security Measures: Consider using additional security technologies to complement the Cloud Network
- Update & Test: Update your firewall regularly and test each update. Attackers are continuously innovating; defenses that work today may not work tomorrow. It is important that your defenses are current and that you verify they work as expected.
- Supplementary Security Measures: Consider using additional security technologies to complement the Cloud Network Firewalls based on test results.
Our journey highlighted the importance of not solely relying on default configurations and the need for a proactive approach to network security. By understanding the capabilities and limitations of tools like Cloud Service Provider Firewalls, organizations can better protect their assets and respond to evolving threats.
For further information on how to best configure these products, please reference Best Practices for Cloud Network Firewall Deployment in 2024: Cloud Service Providers (CSP).
[1] We limited the scope of exploits to those that: 1) targeted servers, 2) were applicable to applications or workloads that could be run on a cloud virtual machine or bare metal platform, 3) were for known exploits over the last 10 years with a CVSS score of medium to high.