Tag: Attack

CyberRatings.org Announces Test Results for Zscaler Zero Trust Exchange

Posted on by CyberRatings.org

Austin, TX – June 2, 2025 – CyberRatings.org (CyberRatings), the non-profit organization dedicated to providing insight into the capabilities of cybersecurity products and services through independent testing, has released additional results from its Security Service Edge (SSE) and Zero Trust Network Access (ZTNA) testing. This latest test focused on another leading product: Zscaler Zero Trust Exchange (ZTE).

Zscaler achieved a Security Effectiveness score of 100%, successfully blocking 100% of exploits, malware and evasions in the SSE test. The test report provides details on product performance across multiple threat categories, with scoring weighted by attack severity. The SSE evaluation covered:

  • TLS/SSL: Top 5 Ciphers used (accounts for ~97% of HTTPS traffic).
  • Malware: 6,184 attack samples sourced from current malware campaigns.
  • Exploits: 205 attack samples from widely exploited vulnerabilities in enterprise environments.
  • Evasions: 1,154 attacks spanning 37 evasion techniques.
  • False Positives: 1,514 samples from various business-critical files and applications, ensuring security measures did not disrupt legitimate traffic.

The ZTNA results confirmed that Zscaler demonstrated strong capabilities by effectively enforcing policies and managing access according to predefined rules, policies, and user roles, achieving 100% in all categories tested. The ZTNA test covered:

  • Authentication & Identity
  • Routing & Access Control
  • Resource Access (Zero Trust Network Access capabilities)
  • TLS/SSL Support

Of the SSE test criteria, meeting the threshold of blocking evasions had the most impact on scores.  Evasion techniques are used by attackers to disguise or obfuscate attacks so that they bypass detection. SSE products must not be tricked by evasions—failure exposes organizations to entire classes of undetected threats. Zscaler scored 100% in blocking all 1,154 evasion attempts.

Security Service Edge is a complex multi-layered security technology built on top of complex, ever-changing cloud technologies. Customers have minimal visibility into their operation and architecture, and testing is challenging. This double-layered opacity limits an organization’s ability to diagnose performance issues, fine-tune policy enforcement, or validate security outcomes.

“The only way to know if an SSE offering works properly is to test it,” said Vikram Phatak, CEO of CyberRatings.org. “Our test determined that Zscaler provides exceptional security effectiveness and strong coverage across a wide variety of threat categories.”

CyberRatings is on track to test several other SSE vendors for Threat Protection along with a Comparative Report to be published this summer.

In addition to in-house testing technologies, CyberRatings used Keysight’s CyPerf tool to test performance and TLS/SSL functionality as well as TeraPackets Threat Replayer tool for exploit packet capture replay.

CyberRatings.org Announces Test Results for Cisco Umbrella and Palo Alto Networks Prisma Access

Posted on by CyberRatings.org

Austin, TX – May 15, 2025 – cyberratings.org/ (CyberRatings), the non-profit organization dedicated to providing insight into the capabilities of cybersecurity products and services through independent testing, has released additional results from its Security Service Edge (SSE) testing. These latest tests focused on two leading products: Cisco Umbrella and Palo Alto Networks Prisma Access.

Palo Alto Networks Prisma achieved a Security Effectiveness score of 98.89%, successfully blocking 100% of evasions. In contrast, Cisco Umbrella scored 12.44%, primarily due to its failure to detect evasive threats. Full test reports detail product performance across multiple threat categories, with scoring weighted by attack severity.

The evaluation covered:

  • TLS/SSL: Top 5 Ciphers used (accounts for ~97% of HTTPS traffic).
  • Malware: 6,184 attack samples sourced from current malware campaigns.
  • Exploits: 205 attack samples from widely exploited vulnerabilities in enterprise environments.
  • Evasions: 1,154 attacks spanning 37 evasion techniques.
  • False Positives: 1,514 samples from various business-critical files and applications, ensuring security measures did not disrupt legitimate traffic.

Evasion techniques are used by attackers to disguise or obfuscate attacks so that they bypass detection. SSE products must not be tricked by evasions—failure exposes organizations to entire classes of (undetected) threats.

“Missing just one type of evasion allows attackers to use entire categories of malware or exploits undetected,” said Vikram Phatak, CEO of cyberratings.org/.

Security Service Edge is a complex multi-layered security technology built on top of complex, ever-changing cloud technologies. Customers have minimal visibility into their operation and architecture, and testing is challenging. This double-layered opacity limits an organization’s ability to diagnose performance issues, fine-tune policy enforcement, or validate security outcomes.

“These are closed systems—what I think of as a black box in a black box—that force executives to make risk decisions based on trust rather than evidence,” Phatak added. “That’s why it is critical that independent testing provides evidence-based data on which executives can make decisions.”

CyberRatings is on track to test several other SSE vendors for Threat Protection along with a Comparative Report to be published this summer.

In addition to in-house testing technologies, CyberRatings used Keysight’s CyPerf tool to test performance and TLS/SSL functionality as well as TeraPackets Threat Replayer tool for exploit packet capture replay.

How CyberRatings Tests Security

Posted on by CyberRatings.org

At CyberRatings, our mission is to provide objective, real-world validation of cybersecurity products, ensuring enterprises have the insights they need to make informed security decisions. To achieve this, we employ a rigorous, industry-leading testing methodology, leveraging a combination of commercial, open-source, and in-house developed tools.

The Tools Behind Our Testing

Our testing arsenal consists of both commercial and custom-developed platforms, allowing us to generate realistic and complex network conditions. One of our key technology partners is Keysight Technologies, whose platforms—including CyPerf, PerfectStorm, BreakingPoint, and Network Emulator—enable us to simulate high-scale network traffic and sophisticated attack scenarios. Additionally, we have partnered with TeraPackets, utilizing their advanced traffic replay tools when needed.

Comprehensive Threat Testing

A critical aspect of cybersecurity validation is ensuring products can detect and block real-world threats without excessive false positives. To accomplish this, we curate threat packages from a vast range of sources, including:

  • Exploits from 240,000+ known vulnerabilities documented in the CVE database. These exploits are carefully selected based on the focus of each test program, ensuring alignment with real-world security concerns. Our library includes both commercially sourced and in-house developed exploit samples.
  • Malware from over 1,000,000+ live samples publicly available across the internet, covering a wide range of threat actors and attack techniques.
  • Advanced evasion techniques, meticulously crafted to test how well security products handle obfuscation and bypass attempts. Our in-house techniques allow us to apply hundreds of evasion layers on malware and exploit samples, creating millions of potential threat variations.
  • Legitimate application traffic to assess the rate of false positives, ensuring that security solutions effectively distinguish between malicious and benign activity.

Business and Technical Value

Our approach to testing provides tangible business and technical benefits for enterprises, security teams, and vendors alike:

  • Enterprises gain confidence in the products they deploy, knowing they have been tested against real-world threat conditions rather than artificial lab environments.
  • Security teams receive actionable insights, helping them choose solutions that offer strong protection while minimizing disruptions from false positives.
  • Vendors benefit from transparent, data-driven validation, allowing them to optimize product performance and improve detection efficacy based on objective results.

By leveraging industry-leading tools, an extensive threat database, and advanced evasion techniques, CyberRatings sets the standard for security validation, helping organizations navigate the evolving cybersecurity landscape with confidence.

CyberRatings.org Publishes Test Results on Cloud Network Firewalls

Posted on by CyberRatings.org

Austin, TX – April 2, 2025 – CyberRatings.org (CyberRatings), the non-profit entity dedicated to providing confidence in cybersecurity product efficacy, today released its Q1 2025 Comparative Test Report on Cloud Network Firewalls (CNFW), along with separate, in-depth reports for each of the ten cloud firewall solutions tested. Security effectiveness results ranged from 0% to 100%.

Key Findings:

  • Third-party firewalls from Check Point, Fortinet, Juniper Networks, Palo Alto Networks, and Versa Networks demonstrated the highest security effectiveness blocking exploits and evasion tactics. Results ranged from 99.61% to 100%.
  • Native cloud firewalls from Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure offer a convenient alternative, but all received 0% Security Effectiveness as they allowed attacks to bypass existing defenses.
  • Google Cloud Platform’s Next Generation Firewall (NGFW) service leverages Palo Alto Networks technology. We attribute the differences in security effectiveness and performance results between the two platforms to each provider independently selecting and deploying different software versions based on their own criteria.
  • A total of six firewall solutions were Recommended and four received Caution ratings.

In the Cloud Service Provider Native Firewall test from November 2024 only 522 exploits were used in the Part 1 “Mini-Test”, but not evasions. For this round of testing, a greater number of exploits were deployed, and evasions were introduced to the test samples:

  • False Positives: 2,760 samples from various business-critical files and applications, ensuring security measures did not disrupt legitimate traffic.
  • Exploits: 2,028 attack samples from widely exploited vulnerabilities in enterprise environments.
  • Evasion Techniques: 2,500 attacks spanning 27 evasion techniques tested across multiple network layers to bypass firewall defenses.
  • Performance Metrics: 46 different stress and capacity tests under diverse workloads.
  • Stability & Reliability: Seven extended tests simulating prolonged real-world attack and operational scenarios.

CyberRatings evaluated firewall security by testing for evasion detection at three separate layers of the Open Systems Interconnection (OSI) model, specifically Layers 3, 4, and 7. Missing lower-layer evasions had the greatest impact on the overall score because these layers form the foundation of firewall security at the fundamental networking level, and when these lower layers are compromised, the firewall’s primary protective function is undermined. Points were deducted based on the firewall’s ability—or inability—to detect evasions:

  • A missed evasion from the Layer 3 level resulted in a 50% deduction per category, up to a potential category maximum reduction of 100%.
  • Missing a Layer 4 evasion led to a 20% deduction per category, up to a potential category maximum reduction of 60%.
  • A miss at Layer 7 incurred only a 1% deduction per category, up to a potential category maximum reduction of 10%.

Layers 3 and 4 evasions are particularly concerning since all modern applications rely on IP and TCP. Vulnerabilities at these layers can be exploited across a wide range of systems—from cloud services to enterprise applications.

“Until cloud service provider native firewalls provide better protection, customers should be looking to third parties for their cloud security needs,” said Vikram Phatak, CEO of CyberRatings.org. “Traditional third-party security vendors have demonstrated that they bring significant value to customers.”

Below is a summary of the Ratings:

The cloud firewalls were tested using Keysight’s CyPerf v5.0 software testing platform in addition to CyberRatings’ in-house developed test tools. Enterprises can easily perform similar testing with a 2-week free trial from Keysight. Further details of the CyPerf strike library can be found here: https://www.keysight.com/us/en/products/network-test/cloud-test/cyperf.html

The Hidden Danger of Evasions

Posted on by CyberRatings.org

Imagine you’ve installed the latest and greatest security system for your home—smart locks, cameras, motion detectors—the whole package. You sleep soundly at night, confident that nothing can get past your defenses. But what if a burglar figured out a way to slip past your security without triggering any alarms?

That’s exactly what happens in cybersecurity when an evasion is successful.

What Is an Evasion?

In cybersecurity, an evasion is a sneaky trick that allows attackers to slip past security measures undetected. Think of it like someone disguising themselves as a delivery driver to walk right through a guarded entrance. If a security product can’t recognize or stop an evasion, then all the protections it promises can be bypassed—making it effectively useless in that moment.

At cyberratings.org/, we rigorously test security products against evasions, focusing on two key areas:

  • Network Layer Evasions (OSI Layers 3 and 4) – Attackers manipulate how data is sent across the network to avoid detection.
  • Application Layer Evasions (OSI Layer 7) – Attackers disguise malicious content within legitimate-looking traffic to fool security products.

Network Layer Evasions: The Invisible Backdoor (Layers 3 & 4)

At the network level, data moves in packets—small chunks of information that get routed between devices. Security products like firewalls, intrusion prevention systems (IPS), and secure web gateways (SWG) inspect these packets to detect threats. However, what if attackers manipulate the way these packets are sent?

How Attackers Evade at This Level:

  • Fragmentation: Breaking malicious payloads into tiny pieces across multiple packets, making it harder for security products to reassemble and recognize the threat.
  • Traffic Spoofing: Making malicious traffic appear like normal web browsing, using tricks like fake source IPs, sending invalid packet formats, or mimicking trusted protocols.
  • Obfuscation with Tunnels: Hiding malware traffic inside legitimate protocols (e.g., sending bad stuff over HTTPS or DNS).

Application Layer Evasions: The Master of Disguise (Layer 7)

At the application level, attackers take things up a notch. Instead of just manipulating how data moves, they manipulate what’s inside the data itself. This tricks security tools like web proxies, email security gateways, and endpoint security solutions into thinking the traffic is safe.

How Attackers Evade at This Level:

  • Encoding and Encryption: Hiding malicious payloads inside harmless-looking data, like base64-encoded text or password-protected ZIP files.
  • Polymorphic Malware: Changing the structure of the malware slightly with each attack to evade signature-based detection.
  • User-Agent Spoofing: Making malicious traffic look like it’s coming from a trusted browser or application.

The Bigger the Evasion, the Bigger the Impact

The lower the OSI layer where an evasion happens, the broader the attack’s potential impact:

  • Network layer evasions (Layers 3 & 4) allow attackers to bypass security at a fundamental level, enabling them to deliver any type of malicious traffic to their target. Since all modern applications rely on IP and TCP, this means an attacker could exploit vulnerabilities across a wide range of systems, from cloud services to enterprise apps.
  • Application layer evasions (Layer 7) are more constrained because they only affect specific protocols, applications, or services. While they can be highly effective for targeted attacks, they depend on network-layer evasion techniques to ensure that the attack traffic reaches the target without being blocked.

A Better Way to Think About It:

  • A network-layer evasion is like slipping through an unguarded side door of a skyscraper. Once inside, you can access any office or floor, allowing for a wide range of attacks.
  • An application-layer evasion is like using a fake employee badge to enter a specific office. It only gets you into that department, meaning the impact is more focused on a particular application or system.

Why Evasion Testing Matters for Security Buyers

If you’re evaluating a firewall, secure web gateway, or SSE platform, don’t just look at feature lists—look at how well it handles evasions.

A security solution that fails against evasions is like a locked door with a hidden back entrance—it might look secure, but it’s easy to bypass.

This is why we test security products against real-world evasion techniques—so enterprises, IT teams, and decision-makers can choose solutions that actually stop threats instead of just looking good on paper.

Want to see how your security products hold up? Check out cyberratings.org/’s independent test results to find out!

Endpoint Protection / Anti-Virus Products Tested for Malware Protection

Posted on by CyberRatings.org

AUSTIN, Texas – August 25, 2022 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, has published results of its Q2 2022 Endpoint Protection Comparative Test.

Focused on endpoint products that feature anti-virus protection, the products tested were Avast Free Antivirus, AVG AntiVirus Free, ESET Internet Security, McAfee Total Protection, Norton 360, Microsoft Defender, Sophos Home Premium and Trend Micro Maximum Security.

“The bad guys are getting bolder and malware / ransomware campaigns continue to get more sophisticated,” said Vikram Phatak, CEO of CyberRatings.org. “Most infections occur in the first few hours after a new campaign is launched. The time it takes for a security product to block the attack matters a lot,” adds Phatak. “That is why we tested not only how much malware a product blocks, but how quickly it blocks an attack.”

Over 40,000 live tests were performed on each product, providing a ±0.49% margin of error. Trend Micro Maximum Security offered the most protection, blocking 97.97% of malware. Sophos Home Premium provided the second-highest protection, blocking 97.47%, followed by Microsoft Defender at 97.13%. Sophos was the quickest to add protection for previously unblocked malware, closely followed by Trend Micro.

With more businesses embracing remote work, a user’s protection is likely limited to the web browser and their endpoint protection product. Therefore, it’s important to be informed about which products are performing as advertised.

The Comparative Test Reports provide metrics for products blocking malware over time, average time a product added protection and average time it took a product to add protection.

The test was funded by CyberRatings.org and no vendor paid to be in or out of the test. As a service to the community, CyberRatings.org is providing these reports for free.

The following endpoint protection / anti-virus products were tested:

  • Avast Free Antivirus – v22.4.6011 (build 22.4.7175.725)
  • AVG AntiVirus Free – v22.4.3231 (build 22.4.7175.725)
  • ESET Internet Security – v15.1.12.0
  • McAfee Total Protection – v16.0 R46
  • Norton 360 (latest updates)
  • Sophos Home Premium – v4.1.0
  • Trend Micro Maximum Security – v17.7.1243
  • Windows Defender – Antimalware Client v4.18.2203.5