How we analyze spam and email deliverability issues

While 99% of GMass users experience the highest email deliverability they’ve ever experienced through any email marketing platform, sometimes users will ask us for help with a deliverability issue, because they believe that some of their email is landing in people’s Spam folders rather than Inboxes. GMass Campaign Reports do show blocks, which happens at the SMTP level, and is indicative of a content or domain-based blocking issue, but in cases where an email isn’t outright blocked and then generates a bounce, it’s helpful to know whether the email is making it to the actual Inbox or not.

Over time, I’ve built an array of tools to help our Support Team diagnose where a user’s emails are landing, the Inbox or Spam folder. When a user contacts us with a suspected deliverability issue, we first analyze the account and look at the open rates of the most recent campaigns. Some users have open rates of greater than 50%, and if we see that, we can usually conclude that there isn’t a widespread deliverability problem. If the open rate is lower, or we want to be thorough, then we conduct a seed list test. We have a simple way of taking any campaign in your GMass account and sending it from your Gmail account to several “seed list” addresses that we maintain. This list of about ten secret addresses are addresses across multiple email providers and email filtering systems, and they include at least one address in each of the following systems:

  • Gmail and G Suite (obviously)
  • GoDaddy Hosted Email
  • Outlook.com
  • Yahoo! Mail
  • Comcast
  • AOL
  • Barracuda
  • Symantec Email Security (formerly known as MessageLabs)
  • Mimecast

The first six (Gmail, GoDaddy, Outlook.com, Yahoo! Mail, Comcast, and AOL) are consumer email providers, and it’s easy to test Inbox placement at these domains, simply because anybody can sign up for a free email account at those providers, and then send their email campaign to these addresses to see if the email makes it to the Inbox. The next three (Barracuda, Symantec, and Mimecast), however, are the world’s three most popular corporate email filters, and surprisingly, most email marketers have never heard of them. I’ve secret-shopped all three such that I have at least one email address at each provider, so that I can test whether their filters are blocking or accepting a particular email message.

It’s also important to understand that the three corporate email filters don’t host email — they simply filter email and then pass the email to the email service that is the host, which for most corporations, is either G Suite or Microsoft Office 365. Therefore, it’s possible to host your company’s email on G Suite, but also, as an additional layer of protection, implement Mimecast or Barracuda or Symantec as the first line of defense.

How do you know if a domain has implemented such a setup? You need only look up the domain’s MX record. One of my domains is chromecompete.com, and I’m using Barracuda to filter its email. Here’s the MX lookup for chromecompete.com:

The MX lookup for chromecompete.com reveals that Barracuda is filtering its email.

So while you can discern that any @chromecompete.com address is using Barracuda to filter email, you cannot discern who the true email provider is…whether it’s G Suite, Microsoft Office 365, or something else. It turns out that I’m actually running my own Windows mail server (a platform called IceWarp) that hosts the email for @chromecompete.com.

As I mentioned, we have the ability to take a campaign from your account and send it, from your Gmail or G Suite account, to this list of addresses. Then we manually check each address to see whether the email arrived or didn’t arrive, and if it arrived, whether it landed in the Inbox, the Spam folder, or whether it was blocked by one of the corporate email filters. Additionally, our seed list tool allows us to easily vary elements of your email campaign, such as whether open and click tracking is on, what tracking domain is used, and the From Address. Even better, soon we’ll be making this tool available to you with the click of a button, and we’ll program our software to check if the email made it to the various email accounts and send a report back to you in a few minutes. Next, I’ll dig into some of the specific spam filters the seed-testing tool examines.

The Gmail Spam Filter

If your email campaign ends up in the Gmail Spam filter, Gmail accompanies it with an explanation that sheds some light on why it ended up there. Gmail has been doing this since 2012. We’ve generally seen five reasons when we see that a user’s emails are ending up in the Spam folder:

Generic content issue:

An example of an email in spam because of a content-related issue. This could be the result of a domain present in the body. Strange though, because this is a transactional email from Stripe, a well known payment processor, and coincidentally, the one GMass uses.

From Domain Issue:

An example of Gmail sending email to Spam because of the From Domain, which may be a G Suite hosted domain. If you’re a GMass user facing this issue, you should get in touch with G Suite support.

DKIM Domain Issue:

An example of Gmail sending email from its own paying G Suite customer to spam. If you’re facing this issue, we encourage you to contact G Suite support, because GMass staff doesn’t have the power to resolve this for you.

What’s shocking about this DKIM sample is it shows Google’s willingness to place email from its own paying G Suite customer in the Spam folder. As it’s explained on this page about how G Suite DKIM-signs emails, the gappssmtp.com domain is G Suite’s internal domain, meaning this email was sent by a G Suite customer and ended up in Spam folder.

Generic Issue:

An example of Gmail sending email to Spam based on heuristic data. Strange though, since Evite is a well known brand.

Phishing/Scam Issue:

An example of Gmail sending email to Spam for being a phishing email.

Given the variety of reasons Gmail may route an email to the Spam folder, even when it’s an email sent from a Gmail or G Suite account, this is how we would advise our users in each scenario:

  • From Domain or DKIM Domain Issue: Contact G Suite Support, since it’s your domain that’s been flagged, and it’s unrelated to anything GMass adds to your email.
  • Content-Related: We’ll use our seed testing tool to vary elements of your email to see if we can get it to the Inbox and what changes were necessary to make that happen.
  • Heuristic-Related: We’ll again use our tool to vary the content to see what works.

The Barracuda Filter

Inside the Barracuda control panel, I can highlight any message that’s been blocked by a Barracuda filter, and Barracuda will state why it’s been blocked, and even better, they’ll tell you if it was a particular domain that caused the block.

In this example, Barracuda blocks an email because of an SPF policy failure.
In this example, Barracuda blocks an email because of the presence of a particular domain that is on its internal block list.

Based on the Barracuda reason, we’ll advise our user to take the appropriate action. In these examples, if it’s an SPF failure, we’ll advise our user to set up the appropriate SPF record, which is an easy fix. If the block is because of the Barracuda Real-Time System (BRTS) and a domain is specified, we’ll advise our user to eliminate that domain from the emails, and if the domain happens to be the tracking domain, we’ll have you swap out that tracking domain for another as an immediate workaround while we submit the blocked domain for delisting to Barracuda Support in the interim.

The Symantec/MessageLabs Filter

In this example, the Symantec/MessageLabs filter blocks a message based on the content.

In our experience with Symantec/MessageLabs, we’ve noticed little domain-based blocking, and when we have, it’s been tied to the domain being listed on a public blacklist, like SURBL or Spamhaus. We haven’t seen evidence of MessageLabs maintaining their own domain blacklist, so most blocks we see are content-based. So in cases of these blocks, we again vary the email with different tracking domains and tracking options to see if the email gets past the filter.

Google’s Postmaster Tools

In addition to use the seed-list testing tool, Google’s Postmaster Tools provides helpful data to determine what is happening to your emails sent to @gmail.com, @googlemail.com, and G Suite email addresses.

Set up Google Postmaster Tools for your sending domain and share the data with me at ajay[at]wordzen.com.
We’ll take a close look at the Postmaster Tools in a future article, but you can set this up easily by adding your sending email domain to your account, and then sharing the data with us. Doing so will help us diagnose what is happening with your emails sent to Google-hosted addresses.

What if I’m on a blacklist?

There are two kinds of blacklists, IP-based and domain-based. It’s unlikely that your sending IP is blacklisted. In most cases, your emails are sent from Gmail’s own servers, so it’s almost impossible that your deliverability issue is caused by an IP issue, because only a foolish network administrator would ever block Gmail’s IP addresses. The exception to the rule is if you’re sending via an alias From Address you have set up inside your Gmail account. Gmail now requires you to specify an external SMTP server by which to send emails with an Alias From Address, so it is possible that the SMTP server you specify has an IP address that is on an IP blacklist.

If you’re experiencing lower open-rates than normal, then it’s possible one of the domains in your email is on a domain blacklist. Domain blacklists are either publicly searchable or private and internal to an organization. GMass encourages the use of dedicated tracking domains to isolate your domain reputation from that of other GMass users, since GMass isn’t a fully monitored system and it’s possible that a shared tracking domain ends up on a public or private domain blacklist. If your email isn’t outright blocked but instead ends ends up in Spam, it’s usually a content issue that involves either the actual text of the email, the From Domain, or the tracking domain present in the body. And if it’s the tracking domain causing the issue, we’ll pursue delisting of the tracking domain and replacement of the tracking domain during the time it takes for the delisting to happen.

In conclusion…

GMass employs a number of tactics to resolve the rare email deliverability issue that a user faces. We do the following:

  • Check your domain’s SPF records
  • Ensure your emails are DKIM-signed
  • Maintain seed addresses at every major email provider, both consumer and corporate

Additionally, you should share your Google Postmaster Tools data with us to give us an additional set of information to analyze. We are soon going to be releasing our automated “seed list testing” tool, so that you can conduct much of this analysis yourself, with the click of a button. Stay tuned!

2 Replies to “How we analyze spam and email deliverability issues”

  1. Jesse Hernandez

    Awesome post tons of valuable content, I really like the screenshots from Barracuda that explain exactly why the email was rejected.

Leave a Reply

Your email address will not be published. Required fields are marked *