February 9, 2011

How to Measure Email Deliverability

by Chris Kolbenschlag, Director of Deliverability at Bronto

Email DeliveryEmail deliverability is a key issue in email marketing.  You could have a great message, a beautiful email, a tempting incentive, but it doesn't mean anything if the email doesn't reach the inbox.  That's why as Director of Deliverability at Bronto my job is to not only monitor Bronto's deliverability but help clients improve their deliverability over time. At Bronto we publish our deliverability rates along with our average open rates and clickthroughs.  These metrics are important to our clients, so we want to make sure that information is always accessible. Whether you use Bronto or not, it's important that you actively monitor your deliverability.  However it's not a clear cut science,  there is currently no method to track every single email and where it landed, but there are several ways to gauge how your campaigns are performing as well as measure the reputation of your IP.  Here are some challenging factors when measuring if your message makes it to the inbox and some methods to evaluate deliverability.

Challenges of Measurement

We are all waiting for the day when we will be able to track every single email to see just where the email landed.  Deliverability of an email is measure by some as simply 'did the ISP accept it'. Some measure it by where the emails landed (spam folder=not delivered, inbox=delivered).   A sender can have a 99.9% delivered statistic but where did all of those 99.9% emails delivered end up? That is today's challenge with measuring deliverability. ReturnPath Certified clients can enjoy such a report that will show where emails landed (inbox vs. spam folder)... for Yahoo! only.  This is a great report to see exact results with no guessing for Yahoo!.  Unfortunately, what your campaign does for Yahoo! isn’t necessarily what is occurring at the other ISPs. Each ISP views senders differently, so results can vary widely.  There is also the heavy challenge that the ISPs measurements on the reputation of a sender can vary unannounced and that is something we cannot measure.  A send may be going out and arriving at the inbox and at any given moment, that send can be immediately routed to the spam folder for crossing a threshold of a determined % of complaints, bounces, traps etc. We have found that using seed addresses within a campaign is a great way to measure where that campaign is ending up since the seeds are part of the actual send.

Methods of Measurement

Inbox Monitoring - Also known as seed list tests,  we will periodically send out email on each of our IP addresses to a group of test email addresses that represent the top ISPs our customers are sending to. This generates a report that will inform us where each of those emails landed at each of the test addresses via our reporting tools with Unica's Pivotal Veracity.  We are able to see if the emails landed in the inbox, spam folder or never made it to the test account at all. This is the best measurement on how well the IP is performing since the data is showing real results of where the emails are landing from an actual campaign.  Over the past 6 months, Bronto's IP addresses are averaging 99.2% inbox delivery on these Unica's Pivotal Veracity reports.

Sender Score -  ReturnPath's Sender Score is a measurement on IP reputation collected from many resources that ReturnPath is able to gather that is generally unavailable to the public.  Sender Score is a good resource to check on an IP address and its performance but it is not the end all result on how your campaigns are performing. Sender Score calculates the score each IP by several measurements: volume, bad address, spam complaints, and spam traps.  ReturnPath also reports on any blacklists the IP may be listed on as well as a good check on authentication methods the IP is signing (DKIM, SenderID, SPF).  The challenge with this scoring is what one ISP may look as a key reputation indicator may not be what another ISP measures.  For example, some ISPs may look very closely at sending volume and consistency while another may not factor that measurement in to their methods with such a high measurement.  Seeing a Sender Score of 80-90 may not reflect on what exactly your campaigns performance is doing.

Open Rates- One other place to check on your results is something easily accessible to a sender within their own reports which is your open rates.   Open rates are a great way to see into your performance on whether your campaign has landed in the spam folder for a specific send. Typically, if you are seeing consistent open rates of 10% to a specific domain and all of a sudden you see a campaign had only 1% open rate to that same domain, this can indicate your campaign was bulk mailed and very few people saw the email to open it because the emails were routed to the spam folder.

What Matters - While we can look at ratings and scores of IP addresses from good sources of data points, in the end what counts is where exactly is your email landing.  Monitoring actual data points such as seed list testing and open rates will tell you the true story of your campaigns. If you want to improve that story - keep ensuring that your messages are relevant and that you're sending to subscribers who signed up for your messages.  Have you found other helpful ways to measure deliverability, if so please comment below.

Chris Kolbenschlag
Director of Deliverability at Bronto

About the Author
Chris Kolbenschlag's picture

Chris Kolbenschlag, Director of Deliverability at Bronto

Read about Chris

Comments

Submitted by Brian Adkins (not verified) on

How does Bronto measure open rates on emails? Is this done via a beacon pixel?... and, if so, what about email clients that do not show images by default?

Submitted by Chris Kolbenschlag on

Hey Brian,
Opens are tracked by a pixel embedded in the email body content. It's not the most accurate but a good steady measure. Most email readers have images off by default so unless the reader turns on images, it wont register if a reader just viewed the email. Thanks

Join the Conversation