Spam filtering review and compare

Clean up your inbox with these no-cost junk email fighters.
 
Learn more
 
 
 
 
 
 

01 .Introduction

Spam

Test results for seven popular free spam filters and four email clients with built-in spam filtering capabilities

Anyone using email these days will eventually encounter spam — junk email that can clog up your inbox. Spam is not only annoying, but also potentially dangerous. Its often used for spreading malicious software such as viruses, adware or spyware, or even leading to the theft of your personal and financial information.

The good news is that a good anti-spam program is as far away as a free download. Spam filters are the guardians of your email inbox, identifying and diverting spam emails automatically, weeding out the ‘suspect’ from the ‘safe’ (also known as ham).

If you’re using Windows XP and Outlook Express, which doesn’t have sophisticated spam filtering built-in, you should consider a good add-on spam filter. If you don’t need to use a particular email client such as Outlook Express, you can use a free email program (client) that has spam filtering built-in.

Please note: this information was current as of March 2009 but is still a useful guide to today's market.


How spam filters work

Aside from an email client with built-in spam filtering there are three types of add-on anti-spam program:

Plug-in — operates within the email client to process spam as it is received. Plug-ins are easy to use because you don’t have to switch between email client and a separate filter.

Pre-step — a pre-step program is a program you run before you use your email client. It interacts only with the mail server, identifying and removing spam before you fetch mail in your chosen email client. Since it doesn’t interact with an email client like a plug-in, a pre-step program is compatible with all clients.

Proxy — a proxy sits between the email client and the mail server. All email passes from the email server to the filter proxy (where it is deleted or categorised) then to the email client. Proxies, like pre-step programs, should also be compatible with all email clients.

Spam filters tested

Add-ons for Windows

  • Bullguard Spamfilter 8
  • MailWasher Free 6.3
  • Popfile 1.0.1
  • SpamBayes 1.0.4
  • SPAMfighter Standard 6.4.23
  • Spamihilator 0.9.9.43
  • SpamPal 1.594

Full version free email clients

  • Windows Mail 6.0.6000.16386 (Windows Vista default email client).
  • Mac Mail 3.5 (OS X 10.5 default email client).
  • Evolution 2.24.1.1 (Ubuntu Linux default email client).
  • Thunderbird 2.0.0.17 (Third-party email client for Linux, Windows and Mac OS X)

We also tested four additional free spam filters but couldn’t get them to work satisfactorily and so dropped them from this test. They were: Sinapsi Antispam,SpamAware, SpamExperts Desktop Limited and xTerminator.

How you get spam

Even if you are careful about who you email and respond to, chances are your email address will still eventually be targeted by spam. Spam emails pop up in your inbox because spammers can get hold of your email address once it’s been exposed publicly on the net. Any email address you use has to ‘travel’ across the internet and it can be captured and used to send you spam. Email addresses are ‘harvested’ from chat rooms, websites, newsgroups and other online places.

They can also be collected by viruses that get onto PCs and copy users’ address books. Or they may even be ‘guessed’ by automated software. So, even if you do everything right, your address can still be captured from the contact list of someone you’ve emailed and sold on to spammers. So, you can’t stop spam, but you can stop it from being a nuisance by using an effective anti-spam program. Not all anti-spam programs work with all email programs so it’s important to check that the one you select will work with your operating system and email client.

 
 

Sign up to our free
e-Newsletter

Receive FREE email updates of our latest tests, consumer news and CHOICE marketing promotions.

 
The following models scored the best results in our test.
What to use
Brand Price
Free add-on filters
Popfile Free
Bullguard Spamfighter Free
Email clients
Mac Mail Free with Mac OS X
Thunderbird  Free for Mac, Linux, Windows
Evolution Free for Linux

Spam options

If you use Outlook Express you need an add-on spam filter and Popfile topped our list for performance, and supports a wide range of other email clients as well. Bullguard Spamfilter was also easy to use and effective. The default clients for Mac and Ubuntu Linux, Mac Mail and Evolution, did an excellent job as did Thunderbird, which has the advantage of having free versions for Mac, Linux and Windows.

What we found

Getting a spam filter set up to work properly is important or you’ll never get the most out of it. Unfortunately, the easiest to set up and use are not always the best performers. We looked at how easy each program was to set up and use, as well as how well it stopped spam.

Ease of setup

Of all the programs tested, the email clients Evolution, Mac Mail and Windows Mail were the easiest to set up because there was no need to download and install a separate program. Each of them was installed as part of their respective operating system and spam detection and filtering was enabled by default.

MailWasher Free scored a 90 (excellent) for ease of setup as it was easy to install and configured itself to work with the email client. Spamihilator and Thunderbird scored an 80 (very good). They were both easy to install, although Spamihilator required a reboot and Thunderbird required manual configuration of the email client.

Bullguard Spamfilter and SPAMfighter Standard were both scored at 70 (good) because although relatively simple to set up, they required a registration email address. The remaining filters all scored 60 (OK) primarily because although the installation was easy they required manually making spam rule and folder changes to the email client.

Ease of use

For ease of setting up and subsequent filtering of emails, Bullguard Spamfilter, Spamihilator, Evolution and Mac Mail were all rated as excellent.

It should be noted that although MailWasher Free, SPAMfighter Standard and SpamPal also scored highly for ease of use, partly because there was less for the user to do, as they were not trainable. This is not necessarily a good thing, as it ultimately results in less performance.

Popfile and SpamBayes scored a ranking of good for ease of use because although the training process was relatively straightforward, they had to be trained one email at a time. The programs don’t allow you to select a group of emails and classify them as spam with a single click.

Windows Mail also scored a ranking of good (70) for ease of use because of its simple spam level setting. Thunderbird scored even lower, with a rating of OK (60) because although easy to use, accidentally clicking on the program during the filtering would abort the process, which could be a problem if having to filter a large amount of email at a time.

Performance

Once trained, most filters were very good at letting legitimate messages through and not mistakenly blocking them as spam. The notable exception being MailWasher Free which scored very poorly in this test. See How we tested — False positive errors. All the others scored a 100 except for SpamPal with a 98, Windows Mail an 86 and Spamihilator a 71. Results for spam blocking varied much more, with scores from 22 to 100. Note that our testing was done using the default settings of each program as much as possible. Tweaking individual settings could achieve a better result.

Thunderbird scored the best result for spam blocking, with a score of 100, followed by Popfile with a score of 98 and Mac Mail with 93, all in the excellent range. In the very good range of scores for spam blocking were Evolution with 82 and Windows Mail with 80. Bullguard Spamfilter was just in the ‘good’ range next with a score of 73. The last four filters ranged from 52 down to 22.

The Overall score takes into account spam blocked, legitimate email not blocked, and the ease of setup and use. Four programs we tested were rated as excellent, with a score over 90. Top of the list was email client Mac Mail with 96, followed by the best-placed of the add-on programs Popfile on 93. Next were the open-source cross-platform email client Thunderbird (92) and the Linux email client Evolution (91).

Add-on program Bullguard Spamfilter was ranked very good overall with 84, as was Windows Mail with 81. Lowest scored overall was MailWasher Free (52), because even though it scored well in blocking spam, it also wrongly blocked a large amount of legitimate email.

Results table

Full results for all models are shown in the table below.

  SCORE SPECIFICATIONS FEATURES
Program Version Tested Operating system Overall Spam blocked (40%) Legitimate
emailblocked (40%)
Ease of use (20% Operating system(s)
supported
Protocols Program class Multiple accounts Trainable Remote invalidation
Website
Whitelist from address book
Popfile
www.getpopfi le.org
1.0.1 XP 93 98 100 70 Windows (versions unspecified), OS X, other unspecified platforms POP3 proxy  
Bullguard Spamfilter
www.bullguard.com
8 XP 84 73 100 73 Windows 2000 / XP / Vista IMAP,
POP3
plug-in
[a]
SPAMfi ghter Standard www.spamfi ghter.com 4/5/2019 XP 75 52 100 73 Windows 98, ME, 2000, XP (32 bit) and Vista (32 bit) POP3 plug-in
[b]
SpamBayes
spambayes.sourceforge.net
1.0.4 XP 67 36 100 63 Windows Vista or XP, Linux, OS X IMAP,
POP3
proxy  
SpamPal
www.spampal.org
1.594 XP 64 22 98 80 Windows 95, 98, ME, NT, 2000, 2003 or XP IMAP4,
POP3
proxy
Spamihilator
www.spamihilator.com
0.9.9.43 XP 58 36 71 77 Windows 2000 / 2003 Server / XP / Vista IMAP,
POP3
proxy
MailWasher Free
www.mailwasher.net
6.3 XP 52 86 0 90 Windows 95 / 98 / Me / NT4 / 2000 / XP / Vista IMAP, POP3, Hotmail, AOL pre-step
EMAIL CLIENTS
Mac Mail
www.apple.com.au
3.5
(929.4/ 929.2)
OS X 96 93 100 93 OS X 10.5 IMAP, POP3 client email  
Thunderbird
www.mozilla.com
2.0.0.17 XP 92 100 100 60 Windows Vista or XP, Linux, OS X IMAP, POP3 client email  
Evolution
projects.gnome.org
2.24.1.1 Linux 91 82 100 93 Linux IMAP, POP3 client email
Windows Mail
www.microsoft.com
6.0.6000. 16386 Vista 81 80 86 73 Vista IMAP, POP3 client email
 

 

Table notes

1 Score Overall includes Spam blocked (40%), Legitimate emails not blocked (40%) and Ease of use (20%).

2 Specifications Operating system(s) supported the operating system(s) on which the program will run; Protocols the type of mail server protocols supported; Program class the program architecture, whether plug-in, pre-step or proxy.

3 Features Multiple accounts if the program works with more than one email account concurrently; Trainable if the program accepts user input to improve accuracy; Remote invalidation if the program uses internet blacklists to classify messages as spam; Whitelist from address book if the program can add existing address book contacts to a user-defined whitelist.

Footnotes:
[a] Outlook, Outlook Express, Thunderbird, Windows Mail
[b] Outlook 2000/2002 (XP) 2003/2007 or Outlook Express 5.5 or later and Windows Mail (32 bit)

How we tested

We downloaded the latest version of each spam filter from the developer’s website in November. Each dedicated spam filter was installed in turn on a newly built Windows XP PC and configured to work with Outlook Express. For the cross-platform Thunderbird, the Windows version was installed on Windows XP. For testing Mac Mail (OS X), Windows Mail (Vista) and Evolution (Linux) we installed a fresh copy of the operating system, which includes the email client.

To serve spam and non-spam emails to the programs we set up our own mail server with two pre-set mail archives consisting of over 8000 spam and ham (legitimate) emails each. We trained all the trainable filters with a separate training list for both spam and ham. Then we tested all using the ham and spam test lists. Scoring results were based on performance after training. We also assessed how easy it was to install and configure each filter or email client to communicate with the mail server. When testing we used the default settings as much as possible. We assessed the help available from the software itself, its documentation and/or website.

False negative errors

Spam filters can make two kinds of errors: false positives and false negatives. A false negative is a spam message incorrectly identified as a legitimate email and let through to the inbox. Allowing a few spam messages through to your inbox (false negatives) is undesirable, but happens occasionally. These emails can usually be fairly easily identified as spam and eliminated on a case by case basis as part of the ongoing training process. So, to assess the performance of each filter we recorded the number of legitimate emails allowed through and the number of spam that were stopped. To score the spam stopping ability we used the percentage of spam stopped as a score out of 100 (see ‘Spam blocked’ in the table ).

False positive errors

False positives, however, are legitimate emails incorrectly classified as spam and these can be a far more serious problem. A filter which falsely classifies legitimate email as spam may remove it before you can see it, resulting in you missing important messages.

Therefore, the effectiveness of each filter program’s ability to correctly classify legitimate email was judged against a more demanding scale. In scoring the results, we applied a scaling factor which allowed for less tolerance for this type of error. This calculation set a 90% success rate in correctly identifying legitimate email as the lowest acceptable figure. Thus, allowing 90% of good email through (allowing 10% of legitimate email to be identified as spam) would result in a rating of just ‘ok’. MailWasher Free was the only program to fail to meet this requirement, receiving a score of 0.

Allowing 100% of legitimate email through (not incorrectly identifying any of it as spam) would produce a score of 100. (see ‘Legitimate emails not blocked’ in the table ).

03.Troubleshooting and dictionary

 

Troubleshooting

You can’t argue with the price when a program is free, but don’t expect the sort of help and support that you would get with a commercial program. So what happens when something goes wrong with your free filter? First thing to do is check the built-in help, if available.

MailWasher Free, SpamPal, Evolution and Mac Mail all scored an excellent rating (90) for their help, which was available within the program itself and was comprehensive and easy to navigate. Though the help for SpamPal has to be downloaded as a separate file. Popfile’s help was just as good, but it was only based on-line so it scored slightly lower at 80.

Bullguard Spamfilter, SPAMfighter Standard, Spamihilator and SpamBayes all scored in the okay range (60) because although most relevant information was present, It was either stored in multiple files or required browsing through the website to locate.

Windows Mail scored even lower at 50, in the borderline range, because its help detail was sparse. Thunderbird’s help was rated ‘poor’, scoring only 40 because it was a remote knowledgebase which we found to be inconsistent.

Of course Mac Mail and Windows Mail aren’t completely ‘free’ — they come bundled with their operating system, which you have to purchase, but they come with the full technical support options of a commercial software package.

Thunderbird and Evolution are both open source programs. They do have support from the developers and open source community, largely through support forums to which anybody can contribute:

Dictionary

  • Email clientA: a program on a computer for sending and receiving email messages over the internet.
  • Ham: Legitimate, or genuine, email so-named to complement spam, or unsolicited email messages.
  • IMAP: Internet Message Access Protocol, a standard for retrieving email messages.
  • Mail server: a networked computer that works as your virtual post office, often located with your ISP (Internet Service Provider).
  • POP: Post Office Protocol — a standard for retrieving email from a mail server.
  • Spam: electronic junk mail also known as unsolicited bulk email.

Join the fight against spam

Another way to help in the fight against email spam is to download the SpamMATTERS spam reporting software developed by The Australian Communications and Media Authority (ACMA).

The software works with Microsoft Outlook and Outlook Express and allows you to report spam that has slipped through to your inbox.

Note, however, that SpamMATTERS is not a spam filter in itself and won’t stop you receiving spam, it simply reports spam to ACMA. The information gathered can be used to identify spammers and may even be used in court proceedings against spammers.