CyberRatings.org Announces Test Results for VMware VeloCloud SD-WAN by Broadcom

Austin, TX – October 22, 2024 – CyberRatings.org (CyberRatings), the non-profit entity dedicated to providing confidence in cybersecurity products and services through its research and testing programs, has completed an independent test of VMware VeloCloud SD-WAN by Broadcom.

This is the third Software-Defined Wide Area Network (SD-WAN) test to be announced by CyberRatings this year. In August, CyberRatings published the SD-WAN test results for Forcepoint and Versa Networks.

SD-WAN technology helps organizations achieve operational savings by enabling remote configuration of new locations rather than requiring engineers to be onsite. Many vendors offer zero-touch provisioning (ZTP), where onsite engineering expertise is optional, other than the ability to connect a device to the appropriate internal and external links and power up the device. Once online, the device will call “home” to either headquarters or a cloud configuration service to download the operational configuration.

We tested five devices, a high availability pair at corporate headquarters and then three remote locations: 1) a regional office, 2) a remote branch office and 3) a retail storefront. Each use case was tested for traffic content, throughput, transport, and impairments to see how the SD-WAN performed. A “AAA” score was based on Management, Routing & Access Control, Stability & Reliability, and the Mean Opinion Score (MOS).

Maximum MOS scores for Voice Over IP (VoIP/audio) are 4.41 and 4.53 for video. The VMware VeloCloud SD-WAN solution achieved a 4.39 MOS score for audio and a 4.39 score for video.

“VMware’s SD-WAN is a full-featured solution that proved to be highly efficient,” said Vikram Phatak, CEO of CyberRatings.org. “Enterprises should have this product on their shortlist.”

Key Findings for VMware VeloCloud SD-WAN by Broadcom:

  • Management and Deployment: Effectively supported Zero Touch Provisioning (ZTP), facilitating remote configuration and deployment across multiple branches.
  • Routing and Policy Enforcement: Successfully enforced simple and complex network policies, including site-to-site VPNs and direct internet access, ensuring secure and efficient traffic routing.
  • Quality of Experience: Delivered mostly consistent MOS for audio and video, even when subjected to various WAN impairments such as packet loss, delay, and reordering.
  • Capacity and Performance: Handled traffic loads well and achieved high throughput with minimal latency.
  • Stability and Reliability: Maintained operational stability under extended adverse conditions and high traffic loads preserving state and ensuring the continued passage of traffic.

Keysight provided its CyPerf, BreakingPoint and Network Emulator tools to test performance, TLS functionality, stability and impairment.

The in-depth test report with scores for Management, Routing & Access Control, Stability & Reliability, and the Mean Opinion Score (MOS) are available at CyberRatings.org.

CyberRatings.org Announces SD-WAN Test Results for Forcepoint and Versa Networks.

Austin, TX – August 26, 2024 – CyberRatings.org (CyberRatings), the non-profit entity dedicated to providing confidence in cybersecurity products and services through its research and testing programs, has completed an independent test of two market-leading Software-Defined Wide Area Network (SD-WAN) products: Forcepoint FlexEdge Secure SD-WAN and Versa Networks Secure SD-WAN.

SD-WAN technology helps organizations achieve operational savings by enabling remote configuration of new locations rather than requiring engineers to be onsite. Many vendors offer zero-touch provisioning (ZTP), where onsite engineering expertise is optional, other than the ability to connect a device to the appropriate internal and external links and power up the device. Once online, the device will call “home” to either headquarters or a cloud configuration service to download the operational configuration.

The testing combination of devices consisted of a high availability pair at the head-end, corporate headquarters representing Branch 1, a regional office (Branch 2), and a retail outlet (Branch 3). Each use case was tested for traffic content, throughput, transport, and impairments to see how the SD-WAN performed. A “AAA” score was based on Management, Routing & Access Control, Stability & Reliability, and the Mean Opinion Score (MOS).

Maximum MOS scores for Voice Over IP (VoIP/audio) are 4.41 and 4.53 for video. Forcepoint achieved a 4.40 MOS score for audio and a 4.48 score for video. Versa’s MOS score for audio was 4.41 and 4.37 for video.

“These are mature, full-featured SD-WAN products that proved to be highly efficient,” said Vikram Phatak, CEO of CyberRatings.org. “Enterprises should have them on their shortlist.”

Key Findings for both the Forcepoint and Versa Networks products:

  • Management and Deployment: Effectively supported Zero Touch Provisioning (ZTP), facilitating remote configuration and deployment across multiple branches.
  • Routing and Policy Enforcement: Successfully enforced simple and complex network policies, including site-to-site VPNs and direct internet access, ensuring secure and efficient traffic routing.
  • Quality of Experience: Delivered consistent MOS for audio and video, even when subjected to various WAN impairments such as packet loss, delay, and reordering.
  • Capacity and Performance: Handled traffic loads well and achieved high throughput with minimal latency.
  • Stability and Reliability: Maintained operational stability under extended adverse conditions and high traffic loads preserving state and ensuring the continued passage of traffic.

Keysight provided its CyPerf, BreakingPoint and Network Emulator tools to test performance, TLS functionality, stability and impairment.

The in-depth Forcepoint and Versa Networks SD-WAN test reports with scores for Management, Routing & Access Control, Stability & Reliability, and the Mean Opinion Score (MOS) are available at CyberRatings.org.

Best Practices for Enterprise Firewall Deployment in 2024

As previously announced, the security industry is working towards a secure-by-default configuration.

This is still an ongoing process; however, we already see vendors making improvements from when we published the cloud network firewall group test. In that test, we found that not all products were secure by default. Therefore, we documented the changes we made and published them. We are doing the same for this group test.

Last year, the Cybersecurity & Infrastructure Security Agency (CISA), along with ten U.S. and international partners, published guidelines for their “Secure by Design, Secure by Default” principle. In their April 2023 publication, they stated the following:

“Secure-by-Default” means products are resilient against prevalent exploitation techniques out of the box without additional charge. These products protect against the most prevalent threats and vulnerabilities without end-users having to take additional steps to secure them. Secure-by-Default products are designed to make customers acutely aware that when they deviate from safe defaults, they are increasing the likelihood of compromise unless they implement additional compensating controls.

This guide should complement what the vendors already provide to their customers. Please refer to the links below for the best practices and guides for each vendor we tested. We have also included extra information for one vendor: Cisco.

The following steps were taken for each firewall:

  • Deploy the firewall in our lab in Austin, Texas (we are using Fortinet for this example).
  • Connect the interfaces required for the topology. This information can be found in each vendor guide.
  • Register the device to the centralized management system where needed. For our test, we only used centralized management for Cisco and Versa Networks. For Cisco, it is required to get some functionality working; more on this below. For Versa Networks, it’s both recommended and needed from a licensing perspective. This is information that is found in each vendor guide.
  • Validation of licenses, which, in turn, enable software updates, threat updates, etc. This information is found in each vendor guide.
  • Define access policies:
    • Trust to untrust
    • Untrust to trust
  • Define IPS policies:
    • Enable threat signatures, advanced protection, cloud lookup, etc. Each product handles this differently, but this information is in their guides.
  • Upload the required server certs, keys, and CA certs if necessary. This information is available in each vendor guide.
  • Define TLS decryption policies for versions 1.2 and 1.3. Configure them to decrypt all traffic. We make a few exceptions to test if the product can bypass decryption based on specific IP addresses or domain names. If something cannot be decrypted or is using an older TLS/SSL version or an insecure cipher, then the product is set to block.
  • Link IPS and TLS policies to the overall access policy.
  • Validate configuration:
    • Make sure you can pass traffic.
    • Make sure you can block attacks by sending something malicious. Tune out false positives where possible. If we couldn’t do so without disabling security or if it was practically impossible, we listed the false positive rate in the test report. Please refer to individual test reports for more details.

For each firewall listed below, we have included a link to best practices and additional information.

Firewalls Tested:

 

Check Point Quantum Force 19200 plus

https://www.checkpoint.com/downloads/products/quantum-force-19200-datasheet.pdf

Firmware: R81.20 Jumbo Hotfix Take 45
IPS Version: 635242922
Configuration: 2 x 40G – 1 port-pairs

Follow their instructions; the product doesn’t require any special configuration.

Documentation: https://support.checkpoint.com

 

Cisco Firepower 2130

https://www.cisco.com/c/en/us/products/collateral/security/firepower-2100-series/datasheet-c78-742473.html

Firmware: Threat Defense v7.3.1 (build 19)
IPS Version: 384
Configuration: 4 x 10G – 2 port-pairs

Follow their instructions; the product requires special configuration; see below.

Documentation: https://www.cisco.com/c/en/us/support/security/firepower-ngfw/products-installation-and-configuration-guides-list.html

We also registered the Cisco Secure Firewall Threat Defense into the Cisco Secure Firewall Management Center (FMC). We enabled TLS 1.2 and 1.3 following Cisco’s instructions. This included updating from Snort v2 to Snort v3, which is required to enable TLS 1.3—as per Cisco: “You must be using Snort 3 to match TLS 1.3 connections.” See https://www.cisco.com/c/en/us/td/docs/security/firepower/730/fdm/fptd-fdm-config-guide-730/fptd-fdm-ssl-decryption.html for more details. This link also provides information about how to make TLS 1.2 and TLS 1.3 work, while also blocking other SSL/TLS version.

Note: Cisco does not support the CHACHA20 cipher suites despite claiming otherwise.

The following screenshot shows the instructions required for achieving our test’s use case.

 

Forcepoint 3410 NGFW

https://www.forcepoint.com/sites/default/files/resources/datasheets/datasheet-forcepoint-ngfw-3400-series-appliance-en_0.pdf

Firmware: 7.1.1 build 29059
IPS Version: 1707
Configuration: 2 x 40G – 1 port-pairs

Follow their instructions; the product doesn’t require any special configuration.

Note: From version 7.1, Forcepoint Next Generation Firewall is rebranded to Forcepoint FlexEdge Secure SD-WAN.

Documentation: https://support.forcepoint.com/s/article/FlexEdge-Secure-SD-WAN

 

Fortinet FortiGate-900G

https://www.fortinet.com/content/dam/fortinet/assets/data-sheets/fortigate-900g-series.pdf

Firmware: v7.4.4 GA
IPS Version: 27.00783
Configuration: 4 x 10G – 2 port-pairs

Follow their instructions; the product doesn’t require any special configuration.

Documentation: https://docs.fortinet.com/document/fortigate/7.4.4/administration-guide/954635/getting-started

 

Juniper Networks SRX4600

https://www.juniper.net/us/en/products/security/srx-series/srx4600-firewall-datasheet.html

Firmware: JUNOS 22.4X3.1 srx4600
IPS Version: 3701
Configuration: 2 x 40G – 2 port-pairs

Follow their instructions; the product doesn’t require any special configuration.

Documentation: https://www.juniper.net/documentation/product/us/en/srx4600/junos-os/

 

Palo Alto Networks PA-450

https://docs.paloaltonetworks.com/hardware/pa-400-hardware-reference/pa-400-firewall-specifications

Firmware: 11.1.1
IPS Version: Threat Version: 2024-05-14 (8849-8746)

AntiVirus Version: 2024-05-14 (4818-5336)

Configuration: 4 x 1G – 2 port-pairs

Follow their instructions; the product doesn’t require any special configuration.

Evasion defenses are now enabled by default, using their latest update. To verify this is the case, please follow the instructions below.

Documentation: https://docs.paloaltonetworks.com/best-practices

Next, you will have to follow the detailed instructions as documented by Palo Alto Networks: https://docs.paloaltonetworks.com/advanced-threat-prevention/administration/threat-prevention/best-practices-for-securing-your-network-from-layer-4-and-layer-7-evasions

After following those instructions, issue the commands in the command line interface (CLI).

To do this:

  • You will have to enable SSH on the device.
  • Then, log in to the device with your admin credentials.
  • Then, run the following commands:

Set system setting ctd block-on-base64-decode-error enable
set system setting ctd block-on-bdat-chunk-decode-error enable
set system setting ctd block-on-chunk-decode-error enable
set system setting ctd block-on-qp-decode-error enable
set system setting ctd block-on-utf-decode-error enable
set system setting ctd block-on-uu-decode-error enable
set system setting ctd block-on-zip-decode-error enable

set deviceconfig setting session resource-limit-behavior bypass

 

Sangfor NGAF 5300

https://www.sangfor.com/sites/default/files/2022-06/NGAF_DS_P_NGAF53-Datasheet_20220531.pdf

Firmware AF 8.0.85.1029 Build 20240423
IPS Version 2024-04-23 (Vulnerability Database)
Configuration 2 x 10G – 1 port-pairs

Follow their instructions; the product doesn’t require any special configuration.

Documentation: https://community.sangfor.com/plugin.php?id=sangfor_databases%3Aindex#?Product=NGAF&Document=Configuration%20Guide&Language=English

 

Versa Networks CSG5000

https://versa-networks.com/documents/datasheets/versa-csg5000-series.pdf

Firmware versa-flexvnf-20240405-041659-5186a33-22.1.4-B
IPS Version 6446
Configuration 5 x 10G – 5 port-pairs (limited to 40G)

Follow their instructions; the product doesn’t require any special configuration.

Documentation: https://academy.versa-networks.com/versa-academy-library/

Documentation: https://docs.versa-networks.com

2024 Best Practices for Cloud Network Firewall Deployment

The security industry is working towards a secure by default configuration. However, when setting up products for the 2024 Cloud Network Firewall group test, we found that not all products are secure by default. We, therefore, documented the changes we made and are publishing them in this guide.

Last year, the Cybersecurity & Infrastructure Security Agency (CISA), along with ten U.S. and international partners, published guidelines for their “Secure by Design, Secure by Default” principle. In their April 2023 publication, they stated the following:

“Secure-by-Default” means products are resilient against prevalent exploitation techniques out of the box without additional charge. These products protect against the most prevalent threats and vulnerabilities without end-users having to take additional steps to secure them. Secure-by-Default products are designed to make customers acutely aware that when they deviate from safe defaults, they are increasing the likelihood of compromise unless they implement additional compensating controls.

This guide should be used as a supplement to what the vendors already make available to their customers. Please see the links below to the best practices and guides for each product we tested. We have also provided additional information for three products: Cisco Secure Firewall Threat Defense Virtual, Palo Alto Networks VM-Series, and Amazon Web Services (AWS) Network Firewall.

Cloud Network Firewall Test Topology

The following steps were taken for each firewall:

Deploy the firewall in Amazon Web Services (AWS). For this example, we are using Fortinet:

  1. Review Firewall options and pricing, then Click to Subscribe.

2. Once you have picked your instance type, Continue to Configuration.

3. Review selection and Continue to Launch.

4. Launch the software from the website, or the EC2 dashboard – selected in the drop-down menu.

5. In the EC2 dashboard, define who has access to the firewall, launch Instance.

  • Connect the interfaces required for the topology. This is information that is found in each vendor guide.
  • Register the device to the centralized management system. This is information that is found in each vendor guide.
  • Validation of license/s, which in turn enabled software updates, threat updates, etc. This is information that is found in each vendor guide.
  • Define access policies:
    • Trust to untrust
    • Untrust to trust

  • Define IPS policies:
    • Enable threat signatures, advanced protection, cloud lookup, etc. Each product handles this differently, but this information is in their guides.
  • Upload the required server certs, keys, and CA certs if needed. This is information that is found in each vendor guide.
  • Define TLS decryption policies (1.2 and 1.3). Configure it to decrypt all traffic. We make a few exceptions to test whether the product can bypass decryption based on specific IP addresses or domain name(s). If something cannot be decrypted or is using an older TLS/SSL version and or an insecure cipher then the product is set to block.
  • Link IPS and TLS policies to the overall access policy.
  • Validate configuration:

Make sure you can pass traffic.

Make sure you can block attacks by sending something malicious.

Tune out false positives.

Firewall Configurations

For each firewall listed below, we have included:

  • Link to the AWS page where the firewall can be selected and deployed
  • Link to best practices and additional information

Amazon Web Services (AWS) Network Firewall

Product, best practices, and documentation: https://aws.amazon.com/network-firewall/

Following the best practices resulted in a deployment that was unable to stop any threats in our Cloud Network Firewall testing harness. After consulting AWS, we were instructed to make changes to the policy. The resulting deployment was successful in blocking 5.39%.

The configuration suggested by AWS:

  • No stateless rules

  • No default stateful action

  • Only Alert actions in the unmanaged signatures

This configuration should send all traffic through the inspection engine.

Our configuration that was tested before the conversation with AWS was slightly different. We had a rule for all stateless traffic to be forwarded to the stateful engine. Under the Stateful default actions, we had “Drop established,”“Alert all,” and “Alert established enabled. For rules we had the following:

alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:”allow-ssm”;flow:established,to_server;sid:88889;rev:1;)
pass tcp $HOME_NET any -> $EXTERNAL_NET any (msg:”allow-ssm”;flow:established,to_server;sid:8888;rev:1;)
pass tcp any any -> any any (msg:”allow-ssm”;flow:established,to_server;sid:888888;rev:1;)
pass UDP any any -> any any (msg:”allow-udp”;sid:88881;rev:1;)
pass ICMP any any -> any any (msg:”allowing-ourbuddyping”;sid:88882;rev:1;)

Both configurations resulted in the same security block rate of 5.39%.

Barracuda CloudGen Firewall

Follow their instructions; the product doesn’t require any special configuration.

Product page: https://aws.amazon.com/marketplace/pp/prodview-nf2s254wcmqfw

Documentation: https://campus.barracuda.com/product/cloudgenfirewall/doc/98209931/overview

Check Point CloudGuard

Follow their instructions; the product doesn’t require any special configuration.

Product page: https://aws.amazon.com/marketplace/pp/prodview-3xp7nph2367yc

Documentation: https://support.checkpoint.com

Cisco Secure Firewall Threat Defense Virtual

Follow their instructions; the product requires special configuration. See below.

We followed the best practices for deploying Cisco Secure Firewall Threat Defense Virtual including updating the software from 7.2.0 to 7.2.5-208 and then to 7.3.0-69.

We also registered the Cisco Secure Firewall Threat Defense into the Cisco Secure Firewall Management Center (FMC). We enabled TLS 1.2 and 1.3 following Cisco’s instructions. This included updating from Snort v2 to Snort v3, which is required to enable TLS 1.3—as per Cisco: “You must be using Snort 3 to match TLS 1.3 connections.” See https://www.cisco.com/c/en/us/td/docs/security/firepower/730/fdm/fptd-fdm-config-guide-730/fptd-fdm-ssl-decryption.html for more details.


After deploying the firewall following their best practices, we were able to verify that TLS 1.2 was decrypting properly, as seen in this screenshot:

Even though we followed the instructions provided by Cisco to enable TLS 1.3, the Cisco Firewall failed to decrypt TLS 1.3 and the logs issued an “SSL_Version_Not_Supported” error, as seen in this screenshot:

Unless you look at the logs, the user interface does not indicate that TLS 1.3 is not being appropriately As shown in the above screenshots. you can check a box that is supposed to turn on TLS 1.3 support, but it does not turn it on. Moreover, while Cisco says TLS 1.3 is on, and when TLS 1.3 traffic is sent across the Cisco cloud firewall, the logs clearly state that the TLS/Cipher suites are not supported and do not decrypt the data. In addition, Cisco does not support the CHACHA20 cipher suites despite claiming otherwise.

Forcepoint NGFW

Follow their instructions; the product doesn’t require any special configuration.

Product page: https://aws.amazon.com/marketplace/pp/prodview-svzncd5l73lu2

Documentation: https://help.forcepoint.com/ngfw/en-us/7.0.4/index.html

Fortinet FortiGate-VM

Follow their instructions; the product doesn’t require any special configuration.

Product page: https://aws.amazon.com/marketplace/pp/prodview-wory773oau6wq

Documentation: https://docs.fortinet.com/document/fortigate/7.2.6/administration-guide/954635/getting-started

Juniper Networks vSRX

Follow their instructions; the product doesn’t require any special configuration.

Product page: https://aws.amazon.com/marketplace/pp/prodview-z7jcugjx442hw

Documentation: https://www.juniper.net/documentation/us/en/software/vsrx/vsrx-consolidated-deployment-guide/vsrx-consolidated-deployment-guide.pdf

Palo Alto Networks VM-Series Next-Generation Firewall w/ Advanced Threat Prevention

Follow their instructions; the product requires special configuration. See below.

Evasion defenses are not enabled by default. Please follow the instructions below. These are a combination of their best practices guide as well as additional instructions per the Palo Alto Engineers we worked with.

Product page: https://aws.amazon.com/marketplace/pp/prodview-mn63yjbq37n4c

Documentation: https://docs.paloaltonetworks.com/best-practices/security-policy-best-practices/security-policy-best-practices

Next, you will have to follow the detailed instructions as documented by Palo Alto Networks: https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/threat-prevention/best-practices-for-securing-your-network-from-layer-4-and-layer-7-evasions

After following those instructions, then issue the following commands in the command line interface (CLI).

To do this:

  • Enable SSH on the device.
  • Log in to the device with your admin credentials.
  • Run the following commands:

Set system setting ctd block-on-base64-decode-error enable
set system setting ctd block-on-bdat-chunk-decode-error enable
set system setting ctd block-on-chunk-decode-error enable
set system setting ctd block-on-qp-decode-error enable
set system setting ctd block-on-utf-decode-error enable
set system setting ctd block-on-uu-decode-error enable
set system setting ctd block-on-zip-decode-error enable

set deviceconfig setting session resource-limit-behavior bypass

request plugins vm_series set-cores dp-cores 3

While the below configuration change will not improve your evasion protection, it may improve your performance. If you are worried about client-side attacks, do not make this change!

Disable Server Response Inspection (DSRI) – see screenshot below.

To Enable or disable DSRI:

  1. Go to Policies>Security>untrust-to-trust>Action.
  2. Check the “Other Settings “section for the DSRI option.
  3. Select the checkbox to enable.
  4. Unselect the checkbox to disable.

We have been informed that an upcoming version of the Palo Alto OS will enable these evasions by default.

Sophos Firewall

Follow their instructions; the product doesn’t require any special configuration.

Product page: https://aws.amazon.com/marketplace/pp/prodview-ga4qvij427bvw

Documentation: https://docs.sophos.com/nsg/sophos-firewall/20.0/help/en-us/webhelp/onlinehelp/index.html

Versa Networks NGFW

Follow their instructions; the product doesn’t require any special configuration.

Unlike the other products in this guide, Versa only offers a bring-your-own license (BOYL). This means products can be deployed using Versa Director (their central management system) or through EC2 on AWS.

Product page: https://aws.amazon.com/marketplace/pp/prodview-egtkta2usoxfa

Documentation: https://academy.versa-networks.com/versa-academy-library/

Documentation: https://docs.versa-networks.com

WatchGuard Firebox Cloud

Follow their instructions; the product doesn’t require any special configuration.

Product page: https://aws.amazon.com/marketplace/pp/prodview-5qg2dngtf3fgu

Documentation: https://www.watchguard.com/help/docs/help-center/en-US/Content/en-US/_intro/home.html

Exploring the Landscape of Cloud Network Firewalls Available on AWS

Today we focus on the diverse array of Cloud Network Firewalls available on Amazon Web Services (AWS). This comprehensive overview aims to inform IT professionals, network administrators, security analysts, and cybersecurity enthusiasts about the various firewall options on AWS, beyond just AWS’s native offerings.

Cloud Network Firewalls on AWS: A Broad Spectrum

AWS hosts a range of third-party cloud network firewalls, each offering unique features and capabilities tailored to different organizational needs. Here’s a rundown of some key players:

  1. Arista Networks Cloud Network Firewall
    Arista Networks provides a cloud network firewall offering for AWS environments, with a focus on traffic management and security. Its features include firewall capabilities, detailed traffic inspection, and policy enforcement tools. The offering is designed for compatibility with complex network architectures, providing various deployment options to meet diverse cloud security requirements.
  2. Barracuda CloudGen Firewall
    The Barracuda CloudGen Firewall, designed for AWS, offers security for cloud-connected networks. Its features encompass threat protection, VPN connectivity, and application-based traffic management. The firewall is developed to adapt to the evolving requirements of cloud infrastructures and includes centralized management for administration across distributed network setups.
  3. Check Point CloudGuard
    Check Point CloudGuard is a network security offering for AWS, including features like intrusion prevention, identity awareness, and anti-bot technology. It is designed for AWS service integration, aiming to protect cloud assets. CloudGuard supports auto-scaling to adapt its security measures in response to network traffic variations. This offering is intended for cloud environments, providing capabilities for managing security policies and handling network traffic.
  4. Cisco Cloud Network Firewall
    Known for its firepower series and ASA (Adaptive Security Appliances), Cisco offers a cloud network firewall tailored for AWS environments, equipped with features to manage network traffic and enforce security policies. This offering includes capabilities for inspecting and controlling traffic flow, as well as implementing security rules across cloud deployments. Designed to integrate with AWS, Cisco’s firewall aims to provide network security management suited to various cloud infrastructure requirements.
  5. Forcepoint Cloud Network Firewall
    Forcepoint’s cloud network firewall for AWS offers capabilities like SD-WAN integration and centralized management to safeguard network perimeters in cloud environments. The Forcepoint offering is structured to provide security management for cloud-based networks. It incorporates features for monitoring network activities and implementing security protocols to address potential threats.
  6. Fortinet FortiGate-VM: The Fortinet FortiGate-VM is a virtual firewall solution tailored for AWS environments, providing a range of network security capabilities. It offers features such as intrusion prevention, web filtering, and SSL inspection, aimed at safeguarding virtualized and cloud infrastructures. Key aspects include its ability to scale dynamically with AWS workloads, integration with AWS services for enhanced management and monitoring, and support for centralized control through Fortinet’s FortiManager. FortiGate-VM is designed to address various security requirements for AWS deployments, from basic VPC protection to advanced threat prevention, catering to diverse network architectures and compliance needs. As with any cloud network firewall solution, its effectiveness and suitability can vary based on specific organizational requirements and network configurations.
  7. Hillstone Networks: Focused on visibility and control, Hillstone offers advanced features for threat detection and mitigation. The offering includes capabilities for inspecting network activities and enforcing relevant security measures. Hillstone’s firewall is developed to support the security needs of cloud deployments, providing functionalities that facilitate the management of network traffic and the implementation of security policies in cloud environments.
  8. Juniper Networks vSRX Cloud Network Firewall
    Juniper Networks offers the vSRX Cloud Network Firewall for AWS, providing network traffic management and policy enforcement. This firewall includes features for monitoring network activities and implementing security protocols. The vSRX offering is designed for AWS environments, aiming to address various network security management needs in cloud infrastructures.
  9. Palo Alto Networks VM-Series
    The VM-Series from Palo Alto Networks is a cloud network firewall available on AWS, focusing on network traffic security and policy management. It offers features for inspecting network traffic and applying security rules. The VM-Series is developed to integrate with AWS, providing network security capabilities for different cloud deployment scenarios.
  10. Sophos UTM and XG Firewalls
    Sophos offers the UTM and XG Firewalls for AWS environments, delivering features to manage network security and traffic. These firewalls include tools for network activity monitoring and security protocol enforcement. Both the UTM and XG Firewalls by Sophos are structured to support security management in cloud-based networks, with functionalities aimed at maintaining network integrity and implementing necessary security measures.
  11. Versa Networks Cloud Network Firewall
    Versa Networks specializes in next-generation firewall capabilities integrated with SD-WAN, suitable for enterprises looking for a combination of security and network optimization. Its cloud network firewall solution for AWS is equipped to handle network security and traffic control. Versa Networks’ firewall is tailored for AWS cloud environments, focusing on meeting diverse network security management requirements in cloud infrastructures.

Choosing the Right Cloud Network Firewall on AWS

Selecting the right cloud network firewall on AWS depends on specific security requirements, scalability needs, and integration capabilities. Factors to consider include:

  • Security Features: Assess the firewall’s capability to protect against the specific threats your organization faces.
  • Performance and Scalability: Ensure the firewall can handle your current and projected traffic volumes without compromising performance.
  • Integration with AWS Services: Look for firewalls that offer seamless integration with other AWS services for streamlined security management.
  • Cost: Consider both upfront and ongoing costs associated with each firewall solution.

Conclusion

The choice of a cloud network firewall on AWS should be guided by your organization’s unique security, performance, and budgetary requirements. Each of the mentioned firewalls brings distinct advantages and specialties to the table, catering to a wide range of cloud-based security needs.

Stay tuned for our Cloud Network Firewall test results coming in March.

Enterprise Firewall Comparative Test Results Show That Encryption and Evasions Matter

AUSTIN, Texas – RSAC 2023 – April 25, 2023 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, has completed an independent test of eight market leading security vendors in its Enterprise Firewall comparative evaluation. Six products received Recommended ratings with high security effectiveness scores ranging from 94.05% to 99.94%.

Security Effectiveness tests measured how well the enterprise firewall controlled network access/applications and prevented exploits/evasions, all while remaining resistant to false positives. Products were subjected to thorough testing to determine their support for TLS/SSL 1.2 and 1.3 cipher suites, how they defended against 1,724 exploits, whether protection could be bypassed by any of 1,482 evasions, and if the devices would remain stable under adverse conditions.

Performance was measured using both clear text and encrypted traffic in order to provide more realistic ratings that are based on modern network traffic. Performance was measured with security enabled, and security effectiveness was measured while under moderate performance load. This was to ensure vendors did not take security shortcuts to improve performance nor enable overly aggressive security protections that would adversely impact performance. Connection rates and throughput of TLS 1.2 and TLS 1.3 encrypted traffic were significantly lower. Average connection rates of encrypted traffic were between 65% to 86.5% lower than unencrypted traffic.

Evasions were measured by taking several previously blocked attacks and then applying evasion techniques to those baseline samples. This ensured that any misses were due to the evasions, not the baseline samples. Several vendors missed evasions, with one vendor missing 72 evasions.

Key Findings:

  • Encryption matters: Roughly 80% of web traffic is encrypted. The top four cipher suites account for over 95% of HTTPS traffic.
    • Decryption is not on by default: Firewalls will not see attacks delivered via HTTPS unless configured to do so.
    • There is a performance cost when TLS/SSL is turned on. Sometimes performance is significantly different.
  • When a “known good” exploit is blocked by a firewall, applying an evasion technique to that exploit is often easier for an attacker than finding a new exploit that isn’t blocked by that firewall.
    • Many firewall evasion defenses are not on by default, potentially leaving customers at significant risk.
    • Most enterprises are not testing for evasions.
    • Some products have concerning gaps when it comes to evasions.
  • At times, CyberRatings found multiple signatures/rules for the same CVE, with some more effective than others.
    • Attempts to provide rapid coverage for vulnerabilities that are not fully understood can result in multiple exploit-specific signatures that may be inaccurate, ineffective, or prone to false positives.
    • A single poorly written signature/rule can significantly impact performance.

“Firewalls are the keystone of most network security programs,” said Vikram Phatak, CEO of CyberRatings.org. “It is concerning that some market share leaders are falling behind. CISOs should put pressure on those vendors to improve and look at alternatives in case they don’t.”

The following products were evaluated:

  • Check Point Quantum QLS250 Lightspeed R81.20
  • Cisco Firepower 2130 v7.3.1-19
  • Forcepoint 2205 NGFW version 7.0.1.28052
  • Fortinet FortiGate 600F v6.4.12 build5431 (GA)
  • Juniper Networks SRX4600 22.3R1.12
  • Palo Alto Networks PA-3220 v10.2.3
  • Sangfor NGAF 5300 AF8.0.47.1004
  • Versa Networks CSG5000 versa-flexvnf-22.1.1-B

CyberRatings.org Announces Results from First-of-its-Kind Comparative Test on Cloud Network Firewall

AUSTIN, Texas – December 1, 2022 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, has completed an independent test of eight market leading security vendors in its first-ever Cloud Network Firewall comparative evaluation. Forcepoint, Fortinet and Juniper’s test reports were published earlier in the year, all with ‘AAA’ ratings. In this latest release of test reports, Check Point and Versa Networks received a ‘AAA’ rating. Palo Alto Networks received an ‘AA,’ Sophos an ‘A,’ and Cisco ‘CC.’

The test covered capabilities considered essential in a firewall including basic routing, access control, SSL / TLS decryption, threat prevention (exploits), evasion, performance, stability and reliability, and management. Amazon Web Services (AWS) was the public cloud service chosen to run the test. Ratings were calculated using a scale from 0 to 800.

Key Findings include:

  • Cloud services assume a shared security model, where cloud providers are responsible for the infrastructure and customers are responsible for securing the applications running on the infrastructure.
  • Roughly 80% of web traffic is encrypted and firewall decryption is not on by default: Firewalls will not see/block attacks delivered via (encrypted) HTTPS unless configured to do so.
  • Security vendors are used to controlling the platform on which their products are installed. In the cloud, they do not have that control; vendors are learning how to operate under these new conditions and there will be challenges.
  • Supply Chain attacks are on the rise. Using the cloud means relying on third parties to maintain software supply chain integrity. APIs, code reuse, open-source libraries, not maintained code, and other shared resources introduce unknown risks.

Security effectiveness scores ranged from 27% to 100%. The security effectiveness tests verified how effectively the firewall protected control network access, applications, and users while preventing threats (exploits and evasions), blocking malicious traffic while under extended load, and remaining resistant to false positives. Exploit block rates ranged from 88.3% to 100%. All products achieved 100% for resistance to evasion techniques.

“Security is your problem, not Amazon’s,” said Vikram Phatak, CEO of CyberRatings.org. “If you are migrating your data center to the cloud, create a plan for securing it,” Phatak added. “And if you needed a firewall for your data center, you probably need one for your cloud deployment.”

There are different ways consumers can purchase security products for the cloud. The individual test reports reflect the bring-your-own-license model while the comparative report illustrates the pay-as-you-go pricing. Both pricing models provide consumers with options to compare pricing on items important to their own organizations.

CyberRatings.org Invites Industry Participation in Forthcoming Enterprise Firewall and Data Center Firewall Tests

AUSTIN, Texas – October 6, 2022 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, will begin testing Enterprise Firewalls and Data Center Firewalls this fall with group test scores and ratings to be released in 2023. Test methodologies have been published at CyberRatings.org and are provided at no charge.

CyberRatings invites all leading vendors to submit their offerings to be tested for free. Products with significant market share, as well as challengers with innovative technologies, will be included at CyberRatings’ discretion. The scope of the methodologies includes security effectiveness, performance, stability and reliability, and total cost of ownership.

Enterprise Firewalls (also known as Next Generation Firewalls) are one of the largest and most mature markets in the industry, projected to grow 10% year over year. A firewall is the first line of defense that monitors incoming and outgoing network traffic. Its purpose is to protect the network by allowing or blocking data packets based on a set of security rules. Without a firewall, networks and data can be vulnerable to thousands of attacks.

“While Cloud Firewalls and SASE are new categories that garner a lot of attention, the Enterprise Firewall market is vast, and while mature, is still growing. Most firewalls remain at the enterprise perimeter or data center,” said Vikram Phatak, CEO of CyberRatings.org. “Our test methodologies are designed to address the challenges faced by enterprise security and IT professionals in selecting and managing security products. We welcome feedback from the community about which firewalls should be tested.”

CyberRatings.org test reports are primarily reserved for members with a paid subscription. In observance of Cybersecurity Awareness Month and courtesy of Keysight CyPerf, CyberRatings.org is making its recent Cloud Network Firewall test reports for Fortinet, Forcepoint and Juniper Networks available for free. Additional products are currently being tested with ratings and a comparative report to be published in the coming weeks.

If you would like to see an Enterprise Firewall or Data Center Firewall product tested, please contact us at info@cyberratings.org.

Cloud Network Firewall (CNFW) Test Update

Today we published our test report of Forcepoint’s Cloud Network Firewall (CNFW). This follows last month’s publication of Fortinet’s Cloud Network Firewall at the RSA Conference in San Francisco. These are the first two publications from the Cloud Network Firewall group test. Testing covered Management & Reporting Capabilities, Routing and Policy Enforcement, SSL/TLS Functionality, Threat Prevention and Performance. Amazon Web Services (AWS) was the cloud provider.

We have been asked who else is in the test, and we want to let everyone know there is more coming!! We expect several more products to be added to the test before we publish our comparative report in a few months. The next reports to be published will be Juniper and Versa, which are currently being tested. I don’t mean to be coy about the specifics of when and who all will be published; this is a new test and like anything new, testing the first few products takes time. We ask everyone to bear with us while we go through these growing pains.

As a reminder, we ask that you please tell us which technologies and vendors you would like to see us test. The easiest way is to email us at members@cyberratings.org.

Thank you,
Vikram Phatak
CEO

CyberRatings.org Issues AAA Rating on Forcepoint’s Cloud Network Firewall

AUSTIN, Texas – July 13, 2022 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, has completed an independent test of the Forcepoint Cloud Network Firewall (CNFW). Forcepoint received the highest possible overall rating of ‘AAA’ with excellent security and performance. The instance used for this test was c5.9xlarge (36 vCPU, 72 GB memory, and 10+ Gbps network Bandwidth).

Forcepoint’s test report is the second in a series of CNFW publications from CyberRatings. Management & Reporting Capabilities, Routing and Policy Enforcement, SSL/TLS Functionality, Threat Prevention and Performance were all tested using Amazon Web Services (AWS) as the cloud provider. More products are being added to the test before the comparative report is published in the second half of 2022. The group test is the first-ever cloud network firewall evaluation provided to the market.

The CyberRatings exploit repository contains exploits that demonstrate a wide range of protocols and applications. Exploit sets for individual tests are selected based on CVSS score (how widely used is an application + what can an attacker do?), use case, and relevance to customers. Forcepoint’s Threat Protection was rated excellent, blocking 35 out of 35 evasion techniques, 977 out of 977 exploits, with an excellent combined Rated Throughput of 943 Mbps.

While the firewall market is one of the largest and most mature security technology segments, this latest evolution virtualizes this functionality to provide scalable and elastic policy enforcement in a cloud environment.

“We were impressed with Forcepoint’s security and performance,” said Vikram Phatak, CEO of CyberRatings.org. “This product is a worthy contender and should be considered by enterprises,” added Phatak.

A blog addressing the timeline for more CNFW to be published was issued today. To read the CyberRatings in-depth report on the various CNFW capabilities offered by Forcepoint, go to CyberRatings.org.