Our Adversarial Approach to Testing

When enterprise organizations invest in cybersecurity products, they need confidence that these solutions will perform effectively under real-world conditions. Traditional vendor claims and marketing materials often lack the depth and rigor needed to validate these capabilities. That’s where independent third-party cybersecurity product evaluations and validation testing come in.

At CyberRatings, we take an adversarial approach to testing—an approach that mirrors the tactics used by real-world attackers. Unlike certification-style testing, which provides vendors with predefined test cases to optimize for, our evaluations challenge cybersecurity products against a dynamic set of real threats, real exploits, and real evasion techniques.

The Value of Adversarial Testing

Our adversarial testing methodology is designed to give enterprise buyers unbiased, comprehensive insights into a product’s actual security effectiveness. Here’s why this matters:

  1. Testing Under Realistic Attack Conditions
    Cyberattacks don’t follow a script, and neither should security testing. Instead of providing vendors with exact test details in advance, CyberRatings evaluates products against thousands of curated exploits, malware samples, and evasion techniques. This ensures that solutions are tested in conditions that mirror the unpredictable, constantly evolving threat landscape.
  2. No “Teaching to the Test”
    Many certification-based testing programs allow vendors to optimize their products for a narrow set of predefined scenarios. This can create a false sense of security for enterprise buyers. CyberRatings’ adversarial approach prevents vendors from tailoring their defenses to a limited test environment and instead evaluates how well their solutions perform when facing the unknown—just as they would in the real world.
  3. Clear, Actionable Insights for Decision-Makers
    Our rigorous testing methodologies provide enterprise buyers with objective, real-world data about a product’s strengths and weaknesses. This transparency enables organizations to make informed purchasing decisions based on validated security effectiveness, rather than relying solely on vendor claims.
  4. Confidence in Security Investments
    With independent validation from CyberRatings, organizations can trust that the cybersecurity solutions they deploy are capable of defending against modern attacks. Whether it’s network firewalls, cloud security platforms, or endpoint protection solutions, our testing ensures that the technologies you invest in can withstand real adversarial threats.

A Higher Standard for Cybersecurity Validation

Enterprise buyers cannot afford to gamble on security solutions that only work under controlled, artificial test conditions. By using an adversarial testing methodology, CyberRatings provides a higher standard of validation—one that reflects the real-world challenges businesses face every day.

When evaluating cybersecurity products, demand realistic, rigorous, and independent testing—because attackers won’t follow a script, and neither should security evaluations.

Want to learn more about CyberRatings’ approach? See how we tested Cloud Network Firewalls.

CyberRatings.org Revives NSS Labs Research

AUSTIN, Texas – November 29, 2022 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, has launched The NSS Labs archive, a library of over 800 test reports, analyst briefs and research published by NSS Labs from 2013 – 2020. Once available only to paid subscribers, CyberRatings.org is providing The NSS Labs archive to the community at no charge.

NSS Labs was an independent analysis and testing company recognized for its fact-based cybersecurity guidance that ceased operations on October 15, 2020. Based in Austin, Texas, NSS Labs tested security products protecting networks, data centers and endpoints for security effectiveness, evasions, performance, stability and usability. It was in the early stages of producing methodologies for cloud testing when the company closed.

Following the NSS Labs closure, CyberRatings.org acquired some of the assets from the custodians of NSS Labs including domain name, test data and reports previously published on their website. The website also holds press releases and blogs over the last six years of the company’s existence showing a timeline for the former company’s development.

“The NSS Labs team produced a tremendous amount of volume year over year that can now serve as historical reference on security product efficacy,” says Vikram Phatak, CEO of CyberRatings.org. “For example, some product scores demonstrated improvement over time while others went the opposite direction. We believe this is a good way to help consumers see how products have evolved.”

Product reports and comparative reports from 2013 – 2020 listed below are now available in the following technology verticals:

  • Data Center Security
  • Endpoint Security (including Advanced Endpoint Protection and Browser)
  • Next Generation Firewall (NGFW)
  • Next Generation Intrusion Prevention System
  • Secure Sockets Layer / Transport Layer Security (SSL / TLS)
  • Software-Defined Wide Area Network (SD-WAN)

The NSS Labs archive can be accessed through www.nsslabs.com and www.cyberratings.org.

Endpoint Protection / Anti-Virus Products Tested for Malware Protection

AUSTIN, Texas – August 25, 2022 – CyberRatings.org, the non-profit entity dedicated to providing transparency on cybersecurity product efficacy, has published results of its Q2 2022 Endpoint Protection Comparative Test.

Focused on endpoint products that feature anti-virus protection, the products tested were Avast Free Antivirus, AVG AntiVirus Free, ESET Internet Security, McAfee Total Protection, Norton 360, Microsoft Defender, Sophos Home Premium and Trend Micro Maximum Security.

“The bad guys are getting bolder and malware / ransomware campaigns continue to get more sophisticated,” said Vikram Phatak, CEO of CyberRatings.org. “Most infections occur in the first few hours after a new campaign is launched. The time it takes for a security product to block the attack matters a lot,” adds Phatak. “That is why we tested not only how much malware a product blocks, but how quickly it blocks an attack.”

Over 40,000 live tests were performed on each product, providing a ±0.49% margin of error. Trend Micro Maximum Security offered the most protection, blocking 97.97% of malware. Sophos Home Premium provided the second-highest protection, blocking 97.47%, followed by Microsoft Defender at 97.13%. Sophos was the quickest to add protection for previously unblocked malware, closely followed by Trend Micro.

With more businesses embracing remote work, a user’s protection is likely limited to the web browser and their endpoint protection product. Therefore, it’s important to be informed about which products are performing as advertised.

The Comparative Test Reports provide metrics for products blocking malware over time, average time a product added protection and average time it took a product to add protection.

The test was funded by CyberRatings.org and no vendor paid to be in or out of the test. As a service to the community, CyberRatings.org is providing these reports for free.

The following endpoint protection / anti-virus products were tested:

  • Avast Free Antivirus – v22.4.6011 (build 22.4.7175.725)
  • AVG AntiVirus Free – v22.4.3231 (build 22.4.7175.725)
  • ESET Internet Security – v15.1.12.0
  • McAfee Total Protection – v16.0 R46
  • Norton 360 (latest updates)
  • Sophos Home Premium – v4.1.0
  • Trend Micro Maximum Security – v17.7.1243
  • Windows Defender – Antimalware Client v4.18.2203.5

CyberRatings.org Announces 2021 Enterprise Firewall Product Ratings

AUSTIN, Texas – February 9, 2021 – CyberRatings.org has begun its publication of the 2021 Enterprise Firewall + SSL/TLS Product Ratings with the first two reports for Palo Alto Networks and Cisco published today. Eleven market leaders are part of this group comparative test that will culminate with the Comparative Rating announcement later this month. CyberRatings determines a vendor’s inclusion in a group test based on an analysis of the market and an understanding of the criteria important to consumers. Elements considered are:

  • Vendor market presence 
  • Products identified by industry analysts covering the specific technology area
  • Consumer requests for a product to be included in a test
  • Innovative technology or solution (requires internal vetting for emerging vendors)

Dedicated to providing transparency on cybersecurity product efficacy, CyberRatings has already initiated new testing programs to fill the void left by the closure of NSS Labs. Tests currently underway are Browser Protection and Cloud Network Firewall. Cloud Native Security, Cloud XDR, Endpoint Protection and SASE are also on the test road map. “The team and I have been touched by the outreach from the cybersecurity community,” said Vikram Phatak, Chairman and CEO at CyberRatings. “It’s not often you get a second chance to do something right. We’re looking forward to working with the community to identify where we can be helpful,” Phatak added. Community Members have free access to the 2021 Enterprise Firewall Methodology. A $100 PAID Membership is required to gain access to the detailed reports. To participate in an upcoming test, email info@www.cyberratings.org.

Additional Resources: