The whole point of installing an antivirus or security suite product is to protect you and your data from a multitude of security risks and annoyances. But how do you know it’s doing its job? It’s not like a motion-sensing camera. You can’t just wave at it and see it working. When evaluating these products for review, we put their claims to the test in many ways. Each review reports the results of our tests, as well as hands-on experience with the product. This article will explain just how these tests work.
Of course, not every test is appropriate for every product. Many antivirus utilities include protection against phishing, for example, but some don’t. Many suites include parental control, but others omit this feature. Some nominally standalone antivirus products add firewall protection as a bonus. Whatever features a given product offers, we put them to the test.
Testing Real-Time Antivirus
Every full-powered antivirus tool includes two core features. The on-demand scanner seeks out and destroys existing malware infestations, and the real-time monitor fends off new attacks. There’s only one way to be sure these protective features work, and that’s to hit them with real-world malware. We use virtual machines for this testing, so there’s no risk of spreading any missed infections.
Each year in the spring, when most security vendors have finished their yearly update cycle, we gather a new collection of malware samples for this test. We start with a feed of the latest malware-hosting URLs, download thousands of samples, and winnow them down to a manageable number.
We analyze each sample using various hand-coded tools. Some of the samples detect when they’re running in a virtual machine and refrain from malicious activity; we simply don’t use those. We look for a variety of different types, and for samples that make changes to the file system and Registry. Once we’ve pared the collection down to something reasonable, we record exactly what system changes each sample makes.
To test a product’s malware-blocking abilities, we start by opening the folder of samples. Real-time protection in some products kicks in immediately, wiping out known malware. If necessary to trigger real-time protection, we single-click each sample, or copy the collection to a new folder, or download the samples from cloud storage—whatever it takes. We note which samples the antivirus eliminates on sight.
Next, we launch each surviving sample and watch how the antivirus handles it. We record the total percentage detected, regardless of when detection happened.
Detection of a malware attack is necessary but not sufficient; the antivirus must actively prevent the attack. A small in-house program checks the system to determine whether the malware managed to make any Registry changes or install any of its files. In the case of executable files, it also checks whether any of those processes are running. And as soon as measurement is complete, we shut down the virtual machine.
If a product prevents installation of all executable traces by a malware sample, it earns 8, 9, or 10 points, depending on how well it prevented cluttering the system with non-executable traces. Detecting malware but failing to prevent installation of executable components gets half-credit, 5 points. Finally, if, despite the antivirus’s attempt at protection, we find one or more malware processes running, that’s worth a mere 3 points. The average of all these scores becomes the product’s final malware protection score.
Testing Web-Level Protection
The best time to annihilate malware is before it ever reaches your computer. Many antivirus products integrate with your browsers and steer them away from known malware-hosting pages. If protection doesn’t kick in at that level, there’s always an opportunity to wipe out the malware payload during or immediately after download.
While our basic malware protection test uses the same set of samples for a season, the malware-hosting URLs we use to test Web-based protection are different every time. We get those links from a feed of the very newest malicious URLs detected by London-based MRG-Effitas and typically use URLs that are no more than a few days old.
Using a small purpose-built utility, we go down the list, launching each URL in turn. We discard any that don’t point to a malware download, and any that return error messages. For the rest, we note whether the antivirus prevents access to the URL, wipes out the download, or sits by idly doing nothing. After recording the result, the utility jumps to the next URL in the list that isn’t at the same domain. We do skip any files larger than 5MB, and we skip files that have already appeared in the same test. We keep at it until we’ve accumulated data for many dozens of verified malware-hosting URLs.
The score in this test is simply the percentage of URLs for which the antivirus prevented downloading malware, whether by cutting off access to the URL completely or by wiping out the downloaded file. Perfect 100% scores are not uncommon, and most tested antivirus tools manage 90% or better protection.
Testing Phishing Detection
Why resort to elaborate data-stealing Trojans when you can just trick people into giving up their passwords? That’s the mindset of malefactors who create and manage phishing websites. These fraudulent sites mimic banks and other sensitive sites. If you enter your login credentials, you’ve just given away the keys to the kingdom. And phishing is platform-independent; it works on any operating system that supports browsing the Web.
These fake websites typically get blacklisted not long after their creation, so for testing we use only the very newest phishing URLs. We gather these from phishing-oriented websites, favoring those that have been reported as frauds but not yet verified. The best security programs detect these new fakes using real-time analysis. Those that just rely on simple-minded blacklists typically earn lower scores.
We use four virtual machines for this test, one protected by the product under testing, and one each using the phishing protection built into Chrome, Edge, and Firefox. A small utility program launches each URL in the four browsers. If any of the four returns an error message, we discard that URL. If the resulting page doesn’t actively attempt to imitate another site, or doesn’t attempt to capture username and password data, we discard it. For the rest, we note whether each product detected the fraud.
As with the web-level protection test, scores vary widely. Some products achieve 100% detection, while others can’t even outscore the protection built into the three browsers.
Testing Spam Filtering
These days email accounts for most consumers have the spam vacuumed out of them by the email provider, or by a utility running on the email server. In fact, the average consumer’s need for spam filtering is slim to none. Austrian test lab AV-Comparatives tested antispam functionality back in 2016, finding that even Microsoft Outlook alone blocked almost 90 percent of spam, and most suites did better, some of them much better. But the lab hasn’t published another such test. According to one of the principals there, “end consumers use hosted mail solutions like Gmail, so no spam filter is needed anymore.”
Years ago, we ran our own antispam tests using a real-world account that we carefully tuned so it would receive plenty of spam but also plenty of valid email. To test a spam filter, we would download thousands of messages and manually check whether any spam slipped into the Inbox or, worse, valid mail got marked as spam. This test took more time and effort than any of our other hands-on tests. Expending maximal effort on a feature of minimal importance no longer makes sense.
There are still important points to report about a suite’s spam filter. What email clients does it support? Can you use it with an unsupported client? Is it limited to POP3 email accounts, or does it also handle IMAP, Exchange, or even Web-based email? At present, we carefully consider each suite’s antispam capabilities, but we longer suffer through downloading and analyzing thousands of emails
Testing Security Suite Performance
When your security suite is busily watching for malware attacks, defending against network intrusions, preventing your browser from visiting dangerous websites, and so on, it’s clearly using some of your system’s CPU and other resources to do its job. Many, many years ago, security suites deservedly got a reputation for sucking up so much of your system resources that your own computer use was affected. If users turn off protection because it’s a performance drag, that’s no kind of protection. Security companies shaped their products up, so you hardly ever see a noticeable slowdown. We do still run some simple tests to get an insight into each suite’s effect on system performance.
Security software needs to load as early in the boot process as possible, lest it find malware already in control. But users don’t want to wait around any longer than necessary to start using Windows after a reboot. Our test script runs immediately after boot and starts asking Windows to report the CPU usage level once per second. After 10 seconds in a row with CPU usage no more than 5 percent, it declares the system ready for use. Subtracting the start of the boot process (as reported by Windows) we know how long the boot process took. We run many repetitions of this test and compare the average with that of many repetitions when no suite was present.
In truth, you probably reboot no more than once per day. A security suite that slowed everyday file operations might have a more deleterious impact on your activities. To check for that kind of slowdown, we time a script that moves and copies a large collection of large-to-huge files between drives. Averaging several runs with no suite and several runs with the security suite active, we can determine just how much the suite slowed these file activities. A similar script measures the suite’s effect on a script that zips and unzips the same file collection.
There are still a few suites that slow one or more of these tests appreciably, but their numbers are dwindling. At the other end of the spectrum, we’ve found several instances in which the tests ran faster after installing the suite.
Testing Firewall Protection
The typical personal firewall has two jobs, protecting the computer from outside attack and ensuring that programs don’t misuse the network connection. To test protection against attack, we use a physical computer that connects through the router’s DMZ port. This gives the effect of a computer connected directly to the Internet. That’s important for testing, because a computer that’s connected through a router is effectively invisible to the Internet at large. We hit the test system with port scans and other Web-based tests. In most cases we find that the firewall completely hides the test system from these attacks, putting all ports in stealth mode. The built-in Windows firewall handles stealthing all ports, so this test is just a baseline.
Program control in the earliest personal firewalls was painfully interactive. Every time an unknown program tried to access the network, the firewall popped up a query asking the user whether to allow or block access. This approach isn’t very effective, since the user generally has no idea what action is correct. Most will just allow everything. Others will click Block every time, until they break some important program; after that they allow everything.
It’s becoming more and more common for suite firewalls to offer this kind of detailed program control without making it the default. For those products, we turn it on before testing. Also, many firewalls come pre-configured with access permissions for known programs. For our hands-on check of program control functionality, we use an app that’s guaranteed to be unknown—a tiny browser utility coded in house and not present anywhere else.
At the other end of the spectrum, the best firewalls automatically configure network permissions for known good programs, eliminate known bad programs, and step up surveillance on unknowns. Only if an unknown program attempts a suspicious connection does the firewall kick in to stop it. That’s not something we can test, since we don’t write malware, but we can observe this functionality during malware protection testing.
Software isn’t and can’t be perfect, so the bad guys work hard to find security holes in popular operating systems, browsers, and applications. They devise exploits to compromise system security using any vulnerabilities they find. Naturally the maker of the exploited product issues a security patch as soon as possible, but until you apply that patch, you’re vulnerable.
The smartest firewalls intercept these exploit attacks at the network level, so they never even reach your computer. Even for those that don’t scan at the network level, in many cases the antivirus component wipes out the exploit’s malware payload. We use the CORE Impact penetration tool to hit each test system with about 30 recent exploits and record how well the security product fended them off.
Finally, we run a sanity check to see whether a malware coder could easily disable security protection. We look for an on/off switch in the Registry and test whether it can be used to turn off protection (though it’s been years since we found a product vulnerable to this attack). We attempt to terminate security processes using Task Manager and third-party task killer utilities. And we check whether it’s possible to stop or disable the product’s essential Windows services.
Testing Parental Control
The category of parental control and monitoring covers a wide variety of programs and features. The typical parental control utility keeps kids away from unsavory sites, monitors their Internet usage, and lets parents determine when and for how long the kids are allowed to use the Internet each day. Other features range from limiting chat contacts to patrolling Facebook posts for risky topics.
We always perform a sanity check to make sure the content filter correctly blocks inappropriate websites. As it turns out, finding porn sites for testing is a snap. Just about any URL composed of a size adjective and the name of a normally covered body part is already a porn site. Very few products fail this test.
We use a tiny in-house browser utility to verify that content filtering is browser independent. We issue a three-word network command (no, we’re not publishing it here) that disables some simple-minded content filters. And we check whether we can defeat the filter by using a secure anonymizing proxy website.
Imposing time limits on the children’s computer or Internet use is only effective if the kids can’t interfere with timekeeping. We verify that the time-scheduling feature works, then try foiling it by resetting the system date and time. The best products don’t rely on the system clock for their date and time.
After that, it’s simply a matter of testing the features that the program claims to have. If it promises the ability to block use of specific programs, we engage that feature and try to break it by moving, copying, or renaming the program. If it says it strips out bad words from email or instant messaging, we add a random word like “fnord” to the block list and verify that it doesn’t get sent. If it claims it can limit instant messaging contacts, we set up a conversation between two of our accounts and then ban one of them. Whatever control or monitoring power the program promises, we do our best to put it to the test.
Interpreting Antivirus Lab Tests
We don’t have the resources to run the kind of exhaustive antivirus tests performed by independent labs around the world, so we pay close attention to their findings. We follow four labs that release scored test results on a regular basis, using their results to help inform our reviews.
Based in Magdeburg, Germany, the AV-Test Institute continuously puts antivirus programs through a variety of tests. The one we focus on is a three-part test that awards up to 6 points in each of three categories: Protection, Performance, and Usability. To reach certification, a product must earn a total of 10 points and have no zero scores in any of the categories. The very best products take home a perfect 18 points in this test.
To test protection, the researchers expose each product to AV-Test’s reference set of over 100,000 samples, and to several thousand extremely widespread samples. Products get credit for preventing the infestation at any stage, be it blocking access to the malware-hosting URL, detecting the malware using signatures, or preventing the malware from running. The best products often reach 100 percent success in this test.
Performance is important—if the antivirus noticeably puts a drag on system performance, some users will turn it off. AV-Test’s researchers measure the difference in time required to perform more than a dozen common system actions with and without the security product present. Among these actions are downloading files from the Internet, copying files both locally and across the network, and running common programs. Averaging multiple runs, they can identify just how much impact each product has.
The Usability test isn’t necessarily what you’d think. It has nothing to do with ease of use or user interface design. Rather, it measures the usability problems that occur when an antivirus program erroneously flags a legitimate program or website as malicious, or suspicious. Researchers actively install and run an ever-changing collection of popular programs, noting any odd behavior by the antivirus. A separate scan-only test checks to make sure the antivirus doesn’t identify any of over 600,000 legitimate files as malware.
We track results from three of the many tests regularly released by AV-Comparatives, which is based in Austria and works closely with the University of Innsbruck. Security tools that pass a test receive Standard certification; those that fail are designated as merely Tested. If a program goes above and beyond the necessary minimum, it can earn Advanced or Advanced+ certification.
This lab’s file detection test is a simple, static test that checks each antivirus against about 100,000 malware samples. A parallel false-positives test ensures accuracy—too many false positives can impact a program’s rating. And the performance test, much like AV-Test’s, measures any impact on system performance. We consider the dynamic whole-product test to be the most significant. This test aims to simulate as closely as possible an actual user’s experience, allowing all components of the security product to participate in fighting the malware.
Where AV-Test and AV-Comparatives typically include a couple dozen products in testing, SE Labs generally reports on no more than 10. That’s in large part because of the nature of this lab’s test. Researchers capture real-world malware-hosting websites and use a replay technique so that each product encounters precisely the same drive-by download or other Web-based attack. It’s extremely realistic, but arduous.
A program that totally blocks one of these attacks earns three points. If it sprang into action after the attack began but managed to remove all executable traces, that’s worth two points. And if it merely terminated the attack, without full cleanup, it still gets one point. In the unfortunate event that the malware runs free on the test system, the product under testing loses five points. Because of this, some products (notably earlier editions of Windows Defender) have received below-zero scores.
In a separate test, the researchers evaluate how well each product refrains from erroneously identifying valid software as malicious, weighting the results based on each valid program’s prevalence, and on how much of an impact the false positive identification would have. They combine the results of these two tests and certify successful products at one of five levels: AAA, AA, A, B, and C.
As noted above, we use a feed of samples supplied by MRG-Effitas in our hands-on malicious URL blocking test. We also follow quarterly results from two of this lab’s ongoing tests. The 360 Assessment & Certification test simulates real-world protection against current malware, much like the dynamic real-world test used by AV-Comparatives. A product that completely prevents any infestation by the sample set receives Level 1 certification. Level 2 certification means that at least some of the malware samples planted files or other traces on the test system, but these traces were eliminated later. Any product that doesn’t achieve one of these levels simply fails. The Online Banking Certification very specifically tests for protection against financial malware and botnets, on a similar pass/fail basis.
Coming up with an overall summary of lab results isn’t easy, since the labs don’t all test the same collection of programs and they all use different scoring systems. We’ve devised an algorithm that normalizes each lab’s scores to a value from 0 to 10. Our aggregate lab results chart reports the average of these scores, as well as the number of labs testing. The best possible result is a perfect 10 score based on results from all four labs. If just one lab includes a product in testing, we consider that to be insufficient information for an aggregate score.
You may have noted that this list of testing methods doesn’t cover virtual private networks, or VPNs. Testing a VPN is very different from testing any other portion of a security suite, so we’ve supplied a separate explanation for how we test VPN services.
