Tag: Enterprise Firewall

InsideCybersecurity: Cyber assessment firm identifies evasion vulnerabilities in enterprise firewall products

Posted on by CyberRatings.org

A nonprofit cyber assessment firm found vulnerabilities in the ability of widely used enterprise firewall products to block transport and network-layer evasions commonly deployed by cyber attackers, in a report examining the effectiveness of security offerings.

“Enterprise Firewalls are constantly evolving to combat new attacker techniques and tools but sometimes that evolution takes a wrong turn. A vendor can have a near-perfect detection engine but if attackers can bypass that engine it gives them a clear path through your defenses,” CyberRatings.org CEO Vikram Phatak sad in a Nov. 5 release.

CyberRatings is a nonprofit organization conducting independent testing of cybersecurity products through its testing partner firm, NSS Labs.

CyberRatings evaluated the “security effectiveness” of seven firewall products in 55 performance tests using 3,326 exploits, 11,311 malware samples, 5,752 evasion techniques in 53 evasion categories and 6,481 false-positive samples,” according to the report.

Read the full article here.

CyberRatings.org and NSS Labs Announce Follow-On Enterprise Firewall Results

Posted on by CyberRatings.org

Austin, TX – November 25, 2025 – CyberRatings.org (CyberRatings), the non-profit organization dedicated to providing confidence in cybersecurity products and services through independent testing, today announced Follow-On Test Results for the Fortinet FortiGate-200G and Palo Alto Networks PA-1410 Enterprise Firewalls.

Both products have improved their ratings from Caution to Recommended following submissions to NSS Labs to retest after developing new builds to address their earlier evasion resistance deficiencies published on November 5, 2025.

“Both Fortinet and Palo Alto Networks responded quickly and transparently to our original findings, issuing updates within days and requesting immediate retesting,” said Vikram Phatak, CEO of NSS Labs. “The speed at which these vendors addressed and resolved critical issues shows their commitment to their customers’ security.”

Fortinet Follow-On Results

During the initial test of Fortinet’s v7.6.4 build3596 with IPS v7.01154 (33.00064), NSS Labs was able to bypass protection using Layer 4 TCP evasions. Fortinet responded quickly to develop an updated IPS signature package. After retesting, NSS Labs confirmed that the update addressed all exploit evasion resistance deficiencies.

Exploit evasion resistance increased from 60% to 100%, elevating the overall Security Effectiveness from 79.24% to 99.24%. Organizations running IPS version v7.01154 (33.00064) or earlier should upgrade immediately to v7.01165 (33.00064) to ensure protection against evasion techniques as detailed in the November 5 publication.

Palo Alto Networks Follow-On Results

During the initial test of PAN-OS 11.2.8-c537, NSS Labs was able to bypass protection using Layer 3 IP and Layer 4 TCP evasions. Palo Alto Networks responded quickly to develop an updated PAN-OS firmware package (PAN-OS 11.2.10-c37) to ensure that the problem had been fixed. After retesting, NSS Labs confirmed that the updated firmware addressed all exploit evasion resistance deficiencies, providing substantial improvements in protection.

Exploit evasion resistance increased dramatically from 0% to 100%, elevating the overall Security Effectiveness from 46.37% to 96.07%.

NSS Labs notes that it is not unusual for vendors to submit pre-release software or firmware intended for imminent release, which NSS Labs requires to be scheduled for general availability within 90 days following a test. Palo Alto Networks confirmed that PAN-OS version 11.2.10-c37 was provided as a pre-release and will be designated as PAN-OS 11.2.10 upon reaching general availability.

Organizations running PAN-OS 11.2.8-c537 or earlier should immediately request PAN-OS 11.2.10 to ensure protection against evasion techniques as detailed in the November 5 publication.

Context and Vendor Accountability

These follow-on results reaffirm the importance of independent testing and vendor accountability. Both vendors’ prompt response demonstrates how transparency and rapid engineering benefit customers.

To accompany these follow-on reports, NSS Labs published a blog titled When Firewalls Fail Gracefully: Why Vendor Responsiveness Matters as Much as Security Effectiveness, highlighting the importance of transparency and quick remediation in cybersecurity engineering.

Testing Methodology

The follow-on tests were conducted using the same methodology and datasets employed in the original Q4 2025 Enterprise Firewall Comparative Report, which evaluated seven leading products under real-world conditions. The updated results now place Fortinet and Palo Alto Networks in the Recommended category alongside Check Point, Juniper Networks, and Versa Networks.

Tests were conducted by NSS Labs developed technologies and Keysight’s CyPerf tool to evaluate security, performance, TLS functionality, and stability. The updated test reports are available at no cost on the CyberRatings.org website.

When Firewalls Fail Gracefully

Posted on by CyberRatings.org

The latest NSS Labs Enterprise Firewall Comparative Report was published this month and, as usual, provided a deep insight into the state of the enterprise firewall market.

Seven of the most widely deployed products were tested using real-world attack scenarios, enterprise-grade workloads, and adversarial evasion techniques to measure their resilience, reliability, and performance.

The results reveal a security landscape that remains uneven: most products blocked the majority of exploits and malware, but a few stumbled when exposed to modern, and not so modern, evasion techniques.

However, the story doesn’t end with the Comparative Security Map – it is also a case study in vendor accountability. How vendors respond when weaknesses are exposed in independent tests such as this tells us a lot about how they are likely to support their enterprise customers in a pinch. It also tells us how seriously they take engineering challenges that could result in serious failures, or even breaches, when installed in live environments.

Palo Alto Networks and Fortinet, though not the highest-scoring participants, stand out precisely because they treated the findings as an opportunity to rectify shortcomings in their products that could have a serious impact on their customers. Within days of publication, both vendors confirmed patches for the issues identified and scheduled retests for the affected products. That kind of responsiveness deserves as much attention as raw test scores.

Read the full blog from NSS Labs: https://nsslabs.com/media/blog/when-firewalls-fail-gracefully/

Inside Cybersecurity: Cyber Assessment Firm Identifies Evasion Vulnerabilities in Enterprise Firewall Products

Posted on by CyberRatings.org

A nonprofit cyber assessment firm found vulnerabilities in the ability of widely used enterprise firewall products to block transport and network-layer evasions commonly deployed by cyber attackers, in a report examining the effectiveness of security offerings.

“Enterprise Firewalls are constantly evolving to combat new attacker techniques and tools but sometimes that evolution takes a wrong turn. A vendor can have a near-perfect detection engine but if attackers can bypass that engine it gives them a clear path through your defenses,” CyberRatings.org CEO Vikram Phatak said in a Nov. 5 release.

Read the full article here.

CyberRatings.org and NSS Labs Announce 2025 Enterprise Firewall Test Results

Posted on by CyberRatings.org

Austin, TX – November 5, 2025 – CyberRatings.org (CyberRatings), the non-profit organization dedicated to providing confidence in cybersecurity products and services through independent testing, today announced the results of its latest Enterprise Firewall (EFW) evaluation.  Tests were conducted by NSS Labs and are now available at no cost on the CyberRatings.org website.

NSS Labs performed independent evaluations of seven leading Enterprise Firewall products using the Enterprise Firewall Test Methodology v3.0. The testing revealed a striking disparity in performance — Security Effectiveness ranged from 46.37% to 99.59%.

Firewalls were tested under encrypted enterprise-grade workloads using 3,326 exploits, 11,311 malware samples, 5,752 evasion techniques spanning 53 evasion categories, 6,481 false-positive samples, and 55 performance tests. Each firewall was required to maintain operational stability throughout testing.

Key Findings

  • Attackers Are Bypassing Defenses: While average exploit and malware block rates exceeded 96%, three widely deployed vendors failed critical evasion tests that significantly reduced their effectiveness. Only three of seven products earned aRecommended
  • Evasion Vulnerabilities: Common transport and network-layer evasions, techniques that can be applied to nearly every attack, bypassed some of the world’s most widely used firewalls.
  • Encrypted Threats: More than 95% of global web traffic is encrypted. Detecting attacks hidden within TLS/SSL sessions remains a crucial differentiator; some products showed marked performance degradation when inspecting encrypted traffic.
  • Accuracy Matters: One product recorded only 80% false-positive accuracy, potentially increasing operational costs and reducing trust in security alerts as customers disable protections to reduce noise.

“Enterprise Firewalls are constantly evolving to combat new attacker techniques and tools but sometimes that evolution takes a wrong turn,” said Vikram Phatak, CEO of CyberRatings.org. “A vendor can have a near-perfect detection engine but if attackers can bypass that engine it gives them a clear path through your defenses.”

The test results are as follows:

NSS Labs is the Official Testing Partner of CyberRatings, generating the test results and reports for CyberRatings publications. NSS Labs developed tools and Keysight’s CyPerf tool were used to test the security, performance, TLS functionality, and stability of Enterprise Firewalls.

The Enterprise Firewall Test Reports, Comparative Report and Security Map are available at CyberRatings.org.

Best Practices for Cloud Network Firewall Deployment in 2024: Cloud Service Providers (CSP)

Posted on by CyberRatings.org

This guide supports the Cloud Network Firewall (CNFW) mini-test, which compares the security effectiveness of native firewall solutions from Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).

This guide should supplement what vendors already provide to their customers. Please see the links below for the best practices and guides for each product we tested. We have also included information for Keysight’s CyPerf v5.0 software testing platform, enabling enterprises to easily replicate our results.

Cloud Network Firewall Test Topology

AWS Network Firewall

Product, best practices, and documentation:

Following AWS’s documentation, the CyberRatings team deployed the AWS Network Firewall instance in routing mode to inspect both inbound and outbound traffic. We set up and configured our threat testing harness using CyPerf, which was installed using the AWS Marketplace.

Deployment Steps:

  1. Route Tables: The route tables must be configured to properly route traffic across the AWS Firewall (AWS FW). Three necessary routing tables are required, and they are described below.
  2. Control Subnet: routes traffic from and to the Control Subnet, the subnet on which the AWS FW endpoint has been deployed.
  3. Customer Subnet: routes traffic from and to the Customer Subnet, which is the subnet on which the trusted clients and servers are deployed, i.e., the LAN side of the Firewall.
  4. Public Internet Gateway: routes traffic from and to the Customer Subnet, which is the subnet on which the trusted clients and servers are deployed, i.e., the WAN side of the Firewall.
  5. Firewall Policies: create rule groups for various traffic types, including allowed and blocked IPs and protocols.
  6. Firewall Subnet: configure in each VPC to direct traffic through the firewall.
  7. Logging & Monitoring: Lastly, we enabled logging to store data in CloudWatch for auditing and monitoring purposes (S3 can also be utilized).

For AWS Network Firewall, multiple steps are required:

  1. Enable logging (both flow and alert logs)
  2. Forward logging output to one or more AWS services (CloudWatch, S3 Storage, etc.)
  3. Then, we analyzed the logs (in JSON format). They can also be exported to multiple formats to be viewed locally.

CyPerf: We used AWS Marketplace and installation was straightforward.

Google Cloud NGFW Enterprise Firewall

Product, best practices, and documentation:

We followed their best practices and documentation and deployed the instance in routing mode to inspect both inbound and outbound traffic. We set up and configured our threat testing harness using CyPerf. Using the provided instructions and documentation, it was easy to deploy.

Cloud NGFW’s threat detection and prevention capabilities are powered by Palo Alto Networks threat prevention technologies[1]. To help protect your network, Cloud NGFW supports a default set of threat signatures with predefined severity levels. Users can view all the threat signatures configured in Cloud NGFW in the threat vault.

Firewall Endpoints:

Firewall Policy:

CyPerf: The CyPerf agents were deployed on GCP using Terraform. The procedure for deploying using Terraform is shown below.

Installation of Google Cloud Software Development Kit (SDK):

  1. Reference: Install the gcloud CLI  |  Google Cloud CLI Documentation
  2. Next step: https://github.com/Keysight/cyperf/tree/main/deployment/gcp/terraform
  3. Once the GCP SDK has been installed successfully, install the agents using Terraform. (These were obtained from Cyperf): https://github.com/Keysight/cyperf/tree/main/deployment/gcp/terraform

Microsoft Azure Firewall Premium

Product, best practices, and documentation:

We followed their best practices and documentation for Microsoft Azure Premium Firewall and deployed the instance in routing mode to inspect both inbound and outbound traffic. We set up and configured our threat testing harness using CyPerf.

Microsoft Azure Firewall Premium uses Microsoft’s closed-source signatures. As of October 2024, its ruleset contained over 67,000 rules in over 50 categories.

Deployment Steps:

  1. First, we installed Terraform: https://learn.hashicorp.com/tutorials/terraform/install-cli
  2. Then we installed Azure CLI: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli
  3. Firewall Policy:
  4. Firewall IDPS Policy:
  5. Firewall Threat Intelligence Policy:
  6. Lastly, we set up logging and forwarded logs to NetWatcher, another service in Azure. The logs could then be analyzed in multiple formats.

CyPerf: https://github.com/Keysight/cyperf/tree/main/deployment/azure/terraform/controller_and_agent_pair

[1] https://cloud.google.com/firewall/docs/about-threats

CyberRatings Announces Enterprise Firewall Test Results

Posted on by CyberRatings.org

Austin, TX – June 27, 2024 – CyberRatings.org (CyberRatings), the non-profit entity dedicated to providing confidence in cybersecurity products and services through its research and testing programs, has completed an independent test of eight market leading enterprise firewall vendors. Seven products were Recommended, and one received a Caution rating.

Enterprise firewalls are used to protect a trusted network from an untrusted network while allowing authorized communications to pass from one side to the other, thus facilitating secure business use of the Internet. Protection rate tests verified how effectively the firewall protected control network access, applications, and users while preventing threats (exploits and evasions), blocking malicious traffic under extended load, and remaining resistant to false positives.

Key Findings:

  • When an exploit is blocked by a firewall, applying an evasion technique to that exploit is often easier for an attacker than finding a new exploit that isn’t blocked by that firewall.
  • Threat actors apply evasion techniques to disguise and modify attacks to avoid detection by security products. Missing a type of evasion means a hacker can use an entire class of exploits to circumvent the security product. CyberRatings used multiple exploits for each evasion technique to see how each product defended against these combinations.
  • Vendors have made progress towards “Secure by Default.” For the products and versions CyberRatings tested, if a vendor’s pre-defined high security configuration is selected, then firewall evasion defenses will be on by default. For other security configurations evasion defenses may not be enabled by default.
  • Encryption matters: Roughly 80% of web traffic is encrypted. The top four cipher suites account for over 95% of HTTPS traffic. It should be noted that decryption is not on by default. Firewalls will not see attacks delivered via HTTPS unless configured to do so.
  • Variants from well-known exploits are not always covered by vendors. At times, CyberRatings found multiple signatures/rules for the same Common Vulnerabilities and Exposures (CVE), with some offering more protection than others. Vendors may attempt to provide rapid coverage for high profile vulnerabilities by creating multiple exploit-specific signatures. If vendors don’t follow up with more comprehensive defenses, this approach can lead to gaps in protection.

To our knowledge, this was the most comprehensive evasion test performed to date. We have accelerated our research into evasion techniques as attackers increasingly bypass defenses,” said Vikram Phatak, CEO of CyberRatings.org. “An attacker can bypass protection if a firewall fails to detect a single form of evasion.”

The following products were tested and rated:

Keysight provided their CyPerf and BreakingPoint tools to test performance, TLS functionality and stability. TeraPackets provided their Threat Replayer tool for packet replay, and CyberRatings used its own proprietary tools for live exploits and evasions.

The Enterprise Firewall Test Reports, Comparative and Security Value Map™ are available at CyberRatings.org.

 

CyberRatings.org Announces “Spot Check” for Security Service Edge (SSE).

Posted on by CyberRatings.org

Austin, TX – January 31, 2024 – CyberRatings.org (CyberRatings), the non-profit entity dedicated to providing confidence in cybersecurity products and services through its research and testing programs, has launched “Spot Check,” a verification of Security Service Edge (SSE) Threat Protection to help enterprises answer the question, “How do you know?”

Cloud delivered security such as SSE provides users with seamless secure access to applications and data regardless of location. Being a cloud technology, organizations are no longer burdened with day-to-day operational management. However, since a third party is now delivering security, oversight is key. This involves:

  • Ensuring the SSE provider maintains the system effectively.
  • Assessing the impact of policy changes on security.
  • Measuring the effectiveness of the SSE solution within the organization’s security framework.

“Often times cybersecurity is a black box; and SSE is a black box in a black box,” said Vikram Phatak, CEO of CyberRatings.org. “How do they know that their SSE is defending against the latest threats, or their policy modifications aren’t adversely impacting their security?” adds Phatak.

SSE solutions leverage the cloud’s scalability, flexibility, and operational benefits to deliver security – Access Control, Authentication and Identity, Data Loss Prevention (DLP), DNS Protection, Encryption (TLS/SSL), Exploit Detection and Prevention, Malware and Phishing Protection (including via Browser Isolation), Cloud Access / Application Control (CASB), and the ability to implement Zero Trust Network Access (ZTNA). It’s a lot harder to test SSE than traditional network security products, and many enterprises don’t have the time or expertise to build a test environment.

What will be tested:

  • Cipher Suite Support: Which cipher suites are supported?
  • False Positive Rate: What is the rate at which the SSE blocks legitimate traffic?
  • Exploits & Malware Delivered Over HTTP: What is the rate at which exploits & malware delivered over HTTP are blocked?
  • Exploits & Malware Delivered Over HTTPS: What is the rate at which exploits & malware delivered over HTTPS are blocked?
  • Evasions: Threat actors use evasion techniques to disguise and modify attacks at the point of delivery to avoid detection by security products. Which ones can be used to bypass protection?

“Spot Check” operates as a virtual employee that is added to the SSE policy being used by an organization. Using the customer’s SSE configuration and CyberRatings’ live network and targets of exploits, malware downloads and evasions, the testing service provides an independent evaluation of SSE solutions, verifying that they are delivering on their promise of protection.

CyberRatings Members with a Premium Membership will receive one free “Spot Check” annually.

CyberRatings has an active test program in 2024 with group test results on Cloud Network Firewall to be announced in early February. Test programs are also currently underway for SD-WAN, SSE Threat Protection, ZTNA, and Enterprise Firewall.

Additional Resources:

Enterprise Firewall Comparative Test Results Show That Encryption and Evasions Matter

Posted on by CyberRatings.org

AUSTIN, Texas – RSAC 2023 – April 25, 2023 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, has completed an independent test of eight market leading security vendors in its Enterprise Firewall comparative evaluation. Six products received Recommended ratings with high security effectiveness scores ranging from 94.05% to 99.94%.

Security Effectiveness tests measured how well the enterprise firewall controlled network access/applications and prevented exploits/evasions, all while remaining resistant to false positives. Products were subjected to thorough testing to determine their support for TLS/SSL 1.2 and 1.3 cipher suites, how they defended against 1,724 exploits, whether protection could be bypassed by any of 1,482 evasions, and if the devices would remain stable under adverse conditions.

Performance was measured using both clear text and encrypted traffic in order to provide more realistic ratings that are based on modern network traffic. Performance was measured with security enabled, and security effectiveness was measured while under moderate performance load. This was to ensure vendors did not take security shortcuts to improve performance nor enable overly aggressive security protections that would adversely impact performance. Connection rates and throughput of TLS 1.2 and TLS 1.3 encrypted traffic were significantly lower. Average connection rates of encrypted traffic were between 65% to 86.5% lower than unencrypted traffic.

Evasions were measured by taking several previously blocked attacks and then applying evasion techniques to those baseline samples. This ensured that any misses were due to the evasions, not the baseline samples. Several vendors missed evasions, with one vendor missing 72 evasions.

Key Findings:

  • Encryption matters: Roughly 80% of web traffic is encrypted. The top four cipher suites account for over 95% of HTTPS traffic.
    • Decryption is not on by default: Firewalls will not see attacks delivered via HTTPS unless configured to do so.
    • There is a performance cost when TLS/SSL is turned on. Sometimes performance is significantly different.
  • When a “known good” exploit is blocked by a firewall, applying an evasion technique to that exploit is often easier for an attacker than finding a new exploit that isn’t blocked by that firewall.
    • Many firewall evasion defenses are not on by default, potentially leaving customers at significant risk.
    • Most enterprises are not testing for evasions.
    • Some products have concerning gaps when it comes to evasions.
  • At times, CyberRatings found multiple signatures/rules for the same CVE, with some more effective than others.
    • Attempts to provide rapid coverage for vulnerabilities that are not fully understood can result in multiple exploit-specific signatures that may be inaccurate, ineffective, or prone to false positives.
    • A single poorly written signature/rule can significantly impact performance.

“Firewalls are the keystone of most network security programs,” said Vikram Phatak, CEO of CyberRatings.org. “It is concerning that some market share leaders are falling behind. CISOs should put pressure on those vendors to improve and look at alternatives in case they don’t.”

The following products were evaluated:

  • Check Point Quantum QLS250 Lightspeed R81.20
  • Cisco Firepower 2130 v7.3.1-19
  • Forcepoint 2205 NGFW version 7.0.1.28052
  • Fortinet FortiGate 600F v6.4.12 build5431 (GA)
  • Juniper Networks SRX4600 22.3R1.12
  • Palo Alto Networks PA-3220 v10.2.3
  • Sangfor NGAF 5300 AF8.0.47.1004
  • Versa Networks CSG5000 versa-flexvnf-22.1.1-B

CyberRatings.org Invites Industry Participation in Forthcoming Enterprise Firewall and Data Center Firewall Tests

Posted on by CyberRatings.org

AUSTIN, Texas – October 6, 2022 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, will begin testing Enterprise Firewalls and Data Center Firewalls this fall with group test scores and ratings to be released in 2023. Test methodologies have been published at CyberRatings.org and are provided at no charge.

CyberRatings invites all leading vendors to submit their offerings to be tested for free. Products with significant market share, as well as challengers with innovative technologies, will be included at CyberRatings’ discretion. The scope of the methodologies includes security effectiveness, performance, stability and reliability, and total cost of ownership.

Enterprise Firewalls (also known as Next Generation Firewalls) are one of the largest and most mature markets in the industry, projected to grow 10% year over year. A firewall is the first line of defense that monitors incoming and outgoing network traffic. Its purpose is to protect the network by allowing or blocking data packets based on a set of security rules. Without a firewall, networks and data can be vulnerable to thousands of attacks.

“While Cloud Firewalls and SASE are new categories that garner a lot of attention, the Enterprise Firewall market is vast, and while mature, is still growing. Most firewalls remain at the enterprise perimeter or data center,” said Vikram Phatak, CEO of CyberRatings.org. “Our test methodologies are designed to address the challenges faced by enterprise security and IT professionals in selecting and managing security products. We welcome feedback from the community about which firewalls should be tested.”

CyberRatings.org test reports are primarily reserved for members with a paid subscription. In observance of Cybersecurity Awareness Month and courtesy of Keysight CyPerf, CyberRatings.org is making its recent Cloud Network Firewall test reports for Fortinet, Forcepoint and Juniper Networks available for free. Additional products are currently being tested with ratings and a comparative report to be published in the coming weeks.

If you would like to see an Enterprise Firewall or Data Center Firewall product tested, please contact us at info@cyberratings.org.