Google Play Protect receives perfect 18/18 score on AV-Test. Now let’s build the “Holy Grail” of AV Benchmarks.
As you may have read in some of my other blogs and 2012 security book, I’ve long been a proponent of measurable security and the need for comparisons that help security technologists, regulators, enterprises, and other stakeholders make more informed decisions about the products they use and their relative risks and benefits. In my earliest days as a software engineer in the 90s, the compilers I was working on would routinely outperform the compilers from the CPU manufacturers themselves on real-world applications written by our users, yet benchmarks like Dhrystone, Whetstone, and Spec were easily gamed and not reflective of real-world performance. Comparisons are important, but good ones can be difficult to achieve. Anti-virus (AV) is one area where many attempts have been made to create useful comparisons, but unfortunately, the results have not been great.
In fact, for more than two decades, for-profit AV benchmarking companies have received the ire of leading AV technology companies (e.g. Kaspersky, Symantec, Cylance), who have accused these firms of operating “pay-to-play” schemes that do not accurately represent protection quality.
In 2017, Google was brought into the fray.
The big bagel
AV-Test, without Google’s involvement, began benchmarking Google Play Protect in Q3 2017, giving it a score of zero which persisted for several years, a misleading score given the massive amount of malware GPP detects and blocks every day. The same reports show a long list of other AVs having perfect scores, which - as I’ll explain - is also misleading. From conversations with our friends working on AV at various companies to understand their experiences and views towards commonly referenced AV benchmarks, it was clear they felt the benchmarks are not good measurements of real-world, real-time detection capability, and they lamented having to spend significant resources managing the scoring optics. We discussed the potential for an improved benchmarking service. This became a personal goal (more on that later).
Subscribing to benchmarks felt … wrong, so for a long while we didn’t do much about them. But the periodic scores kept coming, and with it, questions from OEMs and enterprises about the relative quality of Play Protect. So we changed tactics.
OK, let’s play
In 2022 we did what many other AVs have done, which is bite the bullet and subscribe. We did this not because it was our goal to win against other AVs - in fact, we partner closely with leading Android AVs through the App Defense Alliance, to crowdsource protections for Android users together. Why would these Android AVs want to partner with Google for Android ecosystem protection? Our ADA partners offer enterprise features on top of AV, such as app reputation-based enterprise management controls, distinct from Google Play Protect, whose goal is to provide core malware detection for over 3 billion consumer devices. If you’re an enterprise looking for enterprise-focused AV, you can consult with other independent sources, such as Gartner, for comparisons. Furthermore, our AV partners gain valuable intelligence from Play Protect, each other, and the rest of our safety program and infrastructure that helps them improve their capabilities, which we all want for our users. We subscribed because we wanted to understand what the benchmark was really testing under the hood, the effectiveness of its methodology in comparison to our real-world understanding of risk, and to see if we could use the knowledge gained via subscription to improve GPP and achieve the highest possible protection score.
This was a part-time project for a few folks, and we recently reached the perfect score (18 of 18, requiring 99.8% recall). Here’s how Google Play Protect’s AV-Test protection score improved during the project:
Achieving the perfect protection score did not require significant innovation in core detection capabilities - e.g. static program analysis, dynamic analysis, and AI. The experience reinforced what we expected, which is that subscribing enabled us to understand the sources of benchmark malware samples to ensure they are more rapidly added to the database. Even though Play Protect “sees” malware from many sources, there is a lot of malware across the Internet we don’t immediately see, and massaging these feeds made a significant difference.
The illusion of “perfect” protection scores
It has never ceased to amaze me how many tech pubs simply accept the results of such benchmarks as representative of comparative ability to detect new malware. 100% recall never seemed overly sanguine.
Computer scientists realize just how misleading a “perfect” protection score is. Understanding program behavior is a well-known undecidable computing problem. Sadly, it’s impossible to detect all erroneous behavior in software. There are wonderful code analyzers in the world that find lots of bugs using heuristics, formal methods, and AI. I’ve worked on developing them and most engineers working on complex systems software employ several. Wouldn’t it be great if software analyzers could find all our bugs? But we know this isn’t feasible because math. Studies have shown that code analysis can only realistically find a small percentage of the actual bugs in commercial software. And genAI isn’t going to magically solve bug finding either.
Detecting malicious program behavior (malware) is similarly difficult and undecidable, but with one important difference: the author doesn’t want the bad code to be detected, so they often intentionally obfuscate and cloak malicious behavior to evade the analyzers (AV). And they will use AV as an oracle to check detection in real-time in order to tweak behavior until their malware passes through. Because of this adversarial component, malware detection is more difficult than software bug detection, which already can’t be accomplished with very high recall at the scale of large commercial software projects, by any known method.
Yet AV benchmarks will only award the top protection score if 99.8% of malware is detected.
It is naive to think this kind of recall is feasible with novel malware at scale. Yet these AV benchmarks lead people to believe it is routinely achievable for leading subscribers vendors.
How subscribing helped
I didn’t expect that achieving the benchmark’s top score would significantly improve Google Play Protect. However, there was utility in going through the process. A good AV benchmark would measure performance against novel malware rather than an arbitrary set of samples. However, it’s valid that a more comprehensive database helps to reduce malware distribution. Malware that evades initial detection will still be rendered less impactful by more comprehensive and timely blocklist updates. Our work on the benchmark aided improvements to the speed of blocklist updates and the development of more on-device checks for perturbations to known malware. Also, some of our recent work to invoke more powerful server-side AI algorithms in real-time, which is especially helpful in detecting sideloaded malware, also helped improve the benchmark score.
Holy grail?
Thirteen years ago Eugene Kasperky opined that the “holy grail” of AV benchmarking would never materialize. That may well be true, as it’s even difficult to gain consensus about what metrics matter the most. In Android, we measure a large set of metrics to help inform risk and progress, including malware infection rates, recall and precision of various detection algorithms, and length of time between malware creation and detection, to name a few. But I agree with Eugene that traditional AV benchmarks leave much for improvement:
In order to show in “scanning tests” a near-100% result, an AV firm doesn’t need to go to all the trouble of actually raising the quality of its technology. No sir. All it needs do is detect everything that shows up in the tests. - Eugene Kaspersky
So how can we better test the quality of AV technology and its ability to detect novel malware? Well, it turns out that because Google Play is the most popular app store on the planet, it’s an attractive target for malware authors, who constantly - literally every day - attempt to publish new malware on the store. This stream of novel malware submissions could serve as a benchmark source for the AV tools that run their detection engines on each app submitted to the store. So each publish attempt could act as a novel malware test since the app is not yet on blocklists, and AV must rely on the quality of its algorithmic detection.
And the even better news is that Google Play already has a set of AV engines that run their detection algorithms on every new app submitted to the store. These are the AV developers that have partnered with Google in the aforementioned malware mitigation arm of the App Defense Alliance, under the Linux Foundation. The way it works is simple: new apps published to Google Play are sent for review to all the AV engines - Google Play Protect and our App Defense Alliance AV partners - using a standardized API and SLO. Apps flagged as malware by any of the engines are reviewed and confirmed by reverse engineers, generating relative detection rates between the AV engines for all detected malware. These metrics could be used as the benchmark. While humans are not perfect, any (rare) errors in judgment should be evenly spread over time. Also, an independent audit of the review process should be done to ensure fairness with respect to all the AVs under test. While we’re discussing the ADA, huge shout out: we are grateful for the incremental detections contributed by our AV partners.
Benchmarking additional protections
When it comes to detecting and preventing harmful apps, GPP handles much more than the traditional categories of malware (e.g. spyware, phishing, trojans, etc.), and unfortunately these aspects have not been benchmarked, as AVs generally do not offer comparable solutions. For example, GPP includes a wide range of algorithms for detecting Google Play policy non-compliant content, including restricted content, impersonation, inappropriate use of sensitive permissions, and violations of kids and families policies. GPP also detects attempts to exploit users through OS vulnerabilities. And GPP has sophisticated developer risk and reputation algorithms that leverage developer signals, including interactions with Google Play store, interactions with the developer’s Google account, and interactions with billing and finance systems. Content and developer risk management represents a significant portion of our investment in protecting users against harmful apps and bad actors yet are difficult to benchmark due to reliance on signals that live outside of the app code itself. We are considering potential expansion of our ADA partner APIs to expose store and account metadata that could enable AVs to more easily contribute in these areas, which in turn could then be used to augment benchmark comparisons that help all AVs improve.
AV benchmarks have a less-than-stellar history, but the future for more helpful, realistic measurement and comparison is bright.