September 16, 2011

How Email Spam Filters Really Work

by Chris Kolbenschlag, Director of Deliverability at Bronto

Spam FilterWhy Are Spam Filters Blocking Your Message?

Spam filters can be complicated, fuzzy and a challenge for legitimate marketers to maneuver through.  Even the best marketer, with clear permissions, triple opt-in and a notarized document signed by the end user is not exempt from having emails hit these filters and land in the bulk folder or get quarantined.  This is called a 'false-positive', where your legitimate email gets flagged as spam and placed in jail and never makes it to the inbox of the end user.  Why does this happen and what can you do?  Let's break down both of these questions on how these filters work and what you can do to avoid them as best as possible.

Marketers who send business-to-business emails are more susceptible to these spam filters as they are commonly used by corporations, schools and other businesses that don’t use Internet Service Providers (ISPs) such as AOL, Hotmail, Yahoo etc.  The major ISPs battle spam through more sophisticated anti-spam technology that are geared more toward user feedback (people clicking on the 'this is spam' button').  But for businesses that do have small (or non-existent) IT departments, commercial spam filters are their main line of defense against the massive amount of spam coming into their servers.  These filters run on several levels, filtering mail before they hit the server, after the server and then again on an end user's desktop.  This creates three levels of defense for the end user and three levels of challenges for marketers.  Many of these spam filters are looking at the content of the email to make the decision of whether the email it is reviewing is spam or wanted email.  As your email comes into to the filter, it is evaluated in many ways, such as words, phrases, links and image sizes and is assigned a score.  The score is then evaluated against the threshold setting determined by the end user (i.e. low, med or high). If the score surpasses the threshold, the email is filtered into the spam folder.

The big question this poses to a marketer is "What words or phrases are having the most impact on the email’s overall score?"  The developers of email filters do not share the heuristics of how they calculate their scores (and they are changing their calculation as spammer techniques change).  Some products, such as Spam Assassin do disclose some information that can be helpful in pinpointing trouble words/phrases.  Also, the filter heuristics are constantly learning from each and every email to reduce the amount of unwanted email, aka spam, that gets through.  Another type of filter common with large ISPs, thus more typical in the B2C space, focuses on reputation by evaluating header information. These reputation-based filters evaluate a sender by user feedback provided via the 'this is spam' button.  With reputation-based filters, the volume of spam complaints determines your reputation, and whether your future messages get delivered into the inbox, regardless of content.

Other filters, such as those used by many corporations, judge the purpose of a message so that anything sent to their employees that is not business-related will be filtered out as junk. Just like corporations blocking Facebook from employees, if they feel an email is not related to the business, it will be filtered out on those rules.

Finally, the last filter I call the 'buddy filter.'  This is when the end user will add you to their contact list, bypassing all these filters and cannot be overridden by the filters or ISP.  Obtaining this permission is golden to a marketer and something to communicate to end-users as often as possible to educate them on the benefits of adding you to their contact list.

Now that you have a better idea how these filters work, here are some best practices to avoid getting a 'false positive':

  1. Do not send an all-image email - it may look beautiful, but not to spam filters. Review your image to text ratio in the email. Filters will assume that the image might be hiding something.  Plus, if subscribers have images turned off they won't see anything when they open the email.
  2. Avoid the classic spammy words such as: FREE, sex, BUY NOW, Viagra, ACT NOW, call now, apply now, great offer, buy one, join etc.
  3. Avoid the overuse of common punctuations such as exclamation points, dollar signs and question marks (!!!!, $$$$, ???,)
  4. Do not leave your CAPS LOCK on for too long!  Having subject lines or words with a lot of capitalization will cause high spam scores.
  5. Keep subject lines simple and short, no one wants to read the entire offer in the subject line.  Use your brand name to build positive points.
  6. Avoid links that land you on a page of just an image, which is considered suspect by filters.  Make the landing page legit with text so you do not appear to be hiding anything.
  7. Do not send emails with attachments! Some filters cannot get into them to read and this is also a very common way for malicious viruses to spread and is always suspicious to filters.
  8. Make sure you are sending relevant email to people who have explicitly and deliberately opted-in to your list.  Creating complaints creates higher scores against you.

Here are some common questions you may be asking right now:

Question - How do I know before I send the email if it will get filtered out?
Answer – In Bronto, you can run a Spam Test prior to sending that will scan your content and provide feedback on your email.

Question -  What filter setting does my end user have?
Answer - Since these are personal settings, it is impossible to know. If you assume all end users have the highest and most sensitive setting and test your emails against that setting, you will increase your chances of getting through.

Question - What can I do if I am being filtered?
Answer - The best way to combat spam filters is to get the end user to add you to their contact list. Educate your subscribers on the importance of this and do it on the web page sign up and not via an email since they may never get that email. Create a splash page that says "Thanks for signing up with us!  In order that you do not miss out on important notifications/great deals, please make sure you add us to your contact list."   Second, test your emails.  If you are being filtered out, you will need to systematically dissect your email to identify the culprit.  Start with the subject line using different words.  If that doesn't work, cut the body of the email in two and send each half to see which is causing the issue. Once you know which half is the problem, continue to dissect that portion until you have identified the guilty word, words or phrase.

Question - How do I know I am being filtered?
Answer - Since spam filters act differently, you will need to check two places to identify if there is a spam filter issue. The first location will be lower open rates since people are not receiving your email. Second, some filters will just bounce your mail. A review of your bounce reasons within your email application will show if you are being filtered/blocked.  Third, your users may notify you that they are not receiving your messages.

There is no perfect spam filter out there that sorts every email you want to the inbox and every email you don’t want to the spam folder. 'False positives' are a way of life for marketers, but being educated on how these filters work can be a good weapon in minimizing them.

Chris Kolbenschlag
Director of Deliverability at Bronto

About the Author
Chris Kolbenschlag's picture

Chris Kolbenschlag, Director of Deliverability at Bronto

Read about Chris

Comments

Hi Chris,

Great post. According to Return Path's "Sender Reputation Report", the three metrics that are most likely to reduce your inbox placement rate are: complaint rate, spam trap hit rate and unknown user rate. In your opinion, what role do the latter two play in Spam Filters?

Thanks Craig!
Since every ISP has their own formulas on how they rate senders, the impact of unknown users and spam traps can vary. Some ISPs have this factor weighted more than another but keeping the unknown users below 8% is a good goal. For the spam traps, ISPs do keep careful watch on these but I dont see too much filtering going on with this metric. For example, in Microsoft's SNDS reporting we can view the number of traps hit but yet we commonly have seen no impact to open rates or conversions indicating no filtering due to these traps. Whereas with Spamcop, if you hit a single trap you are blacklisted for 24 hours and mail can be affected to any domain who will query Spamcop's blacklist. I've learned a lot in the past 6 months on spam traps and if the ISPs are seeing what I have been seeing, I hope they are not using them strongly in the metric against a sender's reputation since they are so unreliable.
While I believe all 3 you mentioned are importing to pay attention to, user complaints outweigh the other 2 by a good amount.

Join the Conversation