Issues sending emails via Mandrill service. Sometimes messages disappear.

[dc2000]

Hi everyone:

I've been using the Mandrill service to deliver automated emails from my web site (using their php APIs.) The service seems to work, except sometimes I get questions from my customers that claim that they never received my automated emails. Note that I do not send more than 4-5 of those automated messages a day. So the outflow is very low. The email itself is the registration instructions along with the registration code. So it shouldn't have anything too spammy in it.

So I decided to make a test. I used my own hotmail account to send a test email to. So I sent the first email. It never got into my inbox. I checked spam folder, inbox, been refreshing... nothing. I waited until the next morning. Still nothing. I then tried to send another email and in that case it made through to the same hotmail account just fine.

I then went to my control panel on the Mandrill web site and checked the "Outbound activity" and both of my test messages showed as "delivered", when the first one was clearly not. Worse still, it did not even show up in my Spam folder. It just disappeared without a trace.

So I'm curious, if anyone else had the same issue? And if so, how to address it?

[jarland]

I've not seen this happen before with Mandrill. You've confirmed that it's going from you to Mandrill, but not from Mandrill to the recipient. At that point I would say the investigation is complete when you have a statement from Mandrill and the host of the recipient. One of them either has a log that explains what happens, or one of them has an obligation to explain why they do not. If Mandrill says the recipient server took it from their hands, the recipient host has an obligation to explain why they physically took the mail and refused to deliver it or bounce it.

[dc2000]

Originally Posted by jarland

I've not seen this happen before with Mandrill. You've confirmed that it's going from you to Mandrill, but not from Mandrill to the recipient. At that point I would say the investigation is complete when you have a statement from Mandrill and the host of the recipient. One of them either has a log that explains what happens, or one of them has an obligation to explain why they do not. If Mandrill says the recipient server took it from their hands, the recipient host has an obligation to explain why they physically took the mail and refused to deliver it or bounce it.

Thanks for your reply. The Mandrill web site is laid out in such a way that it's hard to find their support email. I did manage to find it (via Google though) and sent them this very same question. But, so far, no answer... and thus I'm posting it here.

As for the Hotmail (or now Outlook) logs, how can you get those?

[jarland]

Originally Posted by dc2000

As for the Hotmail (or now Outlook) logs, how can you get those?

You have to go here: http://answers.microsoft.com/en-us/o...m/forum?auth=1

Rest assured that eventually, they may reply to you there, ask you to send them a private message, and then they'll tell you how to send an e-mail. In short, they have no desire to help.

[dc2000]

Originally Posted by jarland

You have to go here: http://answers.microsoft.com/en-us/o...m/forum?auth=1

Rest assured that eventually, they may reply to you there, ask you to send them a private message, and then they'll tell you how to send an e-mail. In short, they have no desire to help.

Thanks I'll try it.

In either case (even if I get those logs from Hotmail) what is the way to address this issue? I mean, I can't change the way Hotmail operates. Is there something I need to do (on my end) additionally to make it more reliable?

I mean I'm OK if a user's mail settings filter off my email into Spam folder. I can always tell them to look there. I'm not OK though with messages disappearing completely.

[jarland]

Originally Posted by dc2000

Thanks I'll try it.

In either case (even if I get those logs from Hotmail) what is the way to address this issue? I mean, I can't change the way Hotmail operates. Is there something I need to do (on my end) additionally to make it more reliable?

I mean I'm OK if a user's mail settings filter off my email into Spam folder. I can always tell them to look there. I'm not OK though with messages disappearing completely.

You can ask them to whitelist you in their Outlook.com settings and they should get the e-mail. If they're just treating Mandrill unfairly you might try using a different SMTP source. If it's intermittent then surely the problem can't take anything too extreme to resolve.

[dc2000]

Originally Posted by jarland

You can ask them to whitelist you in their Outlook.com settings..

Well, it's not that simple. Let me give you an example. Say, you're my potential customer. You go to my web site and sign up for a service. This basically involves my site directing you to a PayPal page that takes your payment. Then the PayPal API sends a notification to the php script on my site that in turn validates it and then upon success is supposed to email you the registration details (name and code.) In my current approach I was relying on Mandrill to deliver such registration emails.

So obviously I don't know you and you don't know me. Thus I cannot expect you to add my email into your whitelist. But when you pay money and register for my service and then don't receive an email from me (even in your spam folder) you would probably start saying bad things about my business, wouldn't you? And that's what I'm trying to avoid.

[jarland]

Originally Posted by dc2000

Well, it's not that simple. Let me give you an example. Say, you're my potential customer. You go to my web site and sign up for a service. This basically involves my site directing you to a PayPal page that takes your payment. Then the PayPal API sends a notification to the php script on my site that in turn validates it and then upon success is supposed to email you the registration details (name and code.) In my current approach I was relying on Mandrill to deliver such registration emails.

So obviously I don't know you and you don't know me. Thus I cannot expect you to add my email into your whitelist. But when you pay money and register for my service and then don't receive an email from me (even in your spam folder) you would probably start saying bad things about my business, wouldn't you? And that's what I'm trying to avoid.

I would agree completely, whitelisting is a viable technical solution but not at all a reasonable request to your customers, especially up front as the initial interaction with a new customer.

[dc2000]

You know, just to update. All those "missing" emails showed up now. Almost 24 hrs later! It just baffles me. Is it Hotmail, Mandrill, something else?

[n5bb]

Originally Posted by dc2000

All those "missing" emails showed up now. Almost 24 hrs later! It just baffles me. Is it Hotmail, Mandrill, something else?

Just look at the full headers on those messages you finally received. In live.com/outlook.com you do this by opening the received message, clicking ... at the top, and selecting View message source.

• The Received headers show when each server received the message along the path from the sender to the receiver.
• New headers are added at the top of the list, so you read the headers from bottom to top.
• Start near the end of the headers with the Date header, which shows you when the message was originated by the sender. The Date format is usually similar to the following:

  Date: Sat, 4 Oct 2014 19:22:32 -0700

• Then move up the header list from Date to the first Received header. You should see something similar to:

  Received: (from XXXX) by YYYY; Sat, 4 Oct 2014 23:55:49 -0500

• Write down the complete Date header date/time, and each of the Received date/time values in order. The last (top) Received header will show the time the message was received at outlook/live.
• When I sent a test message from my Fastmail account to my hotmail/live/outlook account, I got these date/time stamps (which I have reordered):

  Date: Sun, 05 Oct 2014 00:07:06 -0500
  Received: by (Fastmail internal server); Sun,  5 Oct 2014 01:07:06 -0400 (EDT)
  Received: from (Fastmail internal server) by (Fastmail internal server); Sun, 05 Oct 2014 01:07:06 -0400
  Received: from (Fastmail internal server) by (Fastmail internal server); Sun,  5 Oct 2014 01:07:06 -0400 (EDT)
  Received: from (Fastmail outbound server) by COL004-MC6F1.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.22712);Sat, 4 Oct 2014 22:07:07 -0700

• You have to algebraically subtract the timezone (in parenthesis) from the date/time to get the Greenwich (international) time. In the Americas the time zones are negative, so this results in a later time. The date/times become:

  Date: Sun, 05 Oct 2014 05:07:06
  Received: Sun,  5 Oct 2014 05:07:06
  Received: Sun,  5 Oct 2014 05:07:06
  Received: Sun,  5 Oct 2014 05:07:06
  Received: Sat,  4 Oct 2014 29:07:07 = Sun,  5 Oct 2014 05:07:07

• So that message was reported as delivered 1 second after it was sent. In this example, I used the Fastmail webmail system to create the message, so this timing relies only on the Fastmail and Outlook/Live/Hotmail server clocks. Since the clocks only have 1 second resolution, even if they are set perfectly all we can say is that the delivery delay was between about 0 and 2 seconds.

Bill

[dc2000]

Thanks, Bill.

Here's the header from one of the messages that was delayed (with actual IDs and email addresses edited out):

x-store-info:J++/JTCzmObr++wNraA4Pa4f5Xdfjfdjdsosdjjdj+obapv9lRm1QO5aFa6j9l8xH/AWlPX8Wdss442ewltmQg=
Authentication-Results: hotmail.com; spf=pass (sender IP is 205.201.131.142) smtp.mailfrom=bounce-md_30010351.542fb85b.v1-58dddjskjsdjjdjhcf0e38528@mandrillapp.com; dkim=permerror header.d=mydomain.com; x-hmca=none header.id=myemail@mydomain.com
X-SID-PRA: myemail@mydomain.com
X-AUTH-Result: NONE
X-SID-Result: NONE
X-Message-Status: n:n
X-Message-Delivery: Vj0xLjksieushyhwuie9MTtHRD0xO1NDTD0w
X-Message-Info: NhFq/7gR1vRFsKNdsfsdfsdfA7uIyzG1gYGo6q6H/uw7GZ0RWwzsvsdfsdfn8yuOVIJ3pkqIQFSYl30fPryOANbiFdLosdf+OUlhUmzEUKfsdfsdf1Ug98KcdfsdfNZURsdfbO3TFYu
Received: from mail142.wdc04.mandrillapp.com ([205.201.131.142]) by SNT004-MC2F24.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.22712);
	 Sat, 4 Oct 2014 14:50:29 -0700
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=mandrill; d=mydomain.com;
 h=From:Subject:To:Message-Id:Date:MIME-Version:Content-Type:Content-Transfer-Encoding; i=myemail@mydomain.com;
 bh=7QyCm9xvdfvdvdsWqlX8E=;
 b=Umuy8D+JVy42T/PJ2ufxNuvpase7bwhmMC+w+3VV0cxvsvxcvxvpI/vFNrRjf
   L+TtWM/Qs5aOOpnjqfBtLxcvdsdfdfgOAJA5MTdfgfdgJ9dsOBWAO
   Rgpdfgdfg0hrJM1cE=
DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=mandrill; d=mydomain.com;
 b=ecQbQNdtRws/DB3mly741Goxcv9iqdYwL/ykVnLLsdfsdfrPPZv
   svTh1vYp0bsdfljKe3mdLV+ZIpk66Xm3czrcxvxcvPBzi2E
   Ob6sdvvvmudhF0=;
Received: from pmta05.wdc01.mailchimp.com (127.0.0.1) by mail142.wdc04.mandrillapp.com id h61lq81jvjgv for <test_hotmail_acct@hotmail.com>; Sat, 4 Oct 2014 09:05:31 +0000 (envelope-from <bounce-md_30010351.542fb85b.v1-58ddf711bea94db4ae9dadecf0e38528@mandrillapp.com>)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandrillapp.com; 
 i=@mandrillapp.com; q=dns/txt; s=mandrill; t=1412413531; h=From : 
 Subject : To : Message-Id : Date : MIME-Version : Content-Type : 
 Content-Transfer-Encoding : From : Subject : Date : X-Mandrill-User : 
 List-Unsubscribe; bh=+4MA2Jxcvxsdsvsvvsz+NvyHvmhOE=; 
 b=IriAQsdvsdvmC7iwGvYEkqwh9JFmkyVE+F6xcvxcvxcvtsv
 OTBLuVTBuyuaxcvdsdfsdfsdfAzBZ7gzkzsQoNsdfsdfsdfyg178
 44cxvdsvfvdfvdfvXLbNUjZ3s=
From: =?UTF-8?B?U2jdusahyshsuY29t?= <myemail@mydomain.com>
Subject: =?UTF-8?B?U2NyaXB0cjdushsyehyehyW5mbw==?=
Return-Path: <bounce-md_30010351.542fb85b.v1-58dkdjhdhgdh@mandrillapp.com>
To: =?UTF-8?B?QXBwIFVzZXI=?= <test_hotmail_acct@hotmail.com>
Received: from [192.64.117.186] by mandrillapp.com id 5s938338; Sat, 04 Oct 2014 09:05:31 +0000
X-Report-Abuse: Please forward a copy of this message, including all headers, to abuse@mandrill.com
X-Report-Abuse: You can also report abuse here: http://mandrillapp.com/contact/abuse?id=30010351.5
X-Mandrill-User: userid
Message-Id: <30010351.20141004090531.542fb85bd79ea3.06249963@mail142.wdc04.mandrillapp.com>
Date: Sat, 04 Oct 2014 09:05:31 +0000
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 04 Oct 2014 21:50:29.0163 (UTC) FILETIME=[3697DFB0:01CFE01D]

[n5bb]

According to the munged headers you posted:

• The message was received by the mandrillapp.com server on Sat, 4 Oct 2014 09:05:31 +0000
• The message was received by the hotmail.com server on Sat, 4 Oct 2014 14:50:29 -0700, which corrected for timezone is Sat, 4 Oct 2014 21:50:29 +0000
• So the message was delayed by exactly 12 hours + 45 minutes (within 2 seconds). My guess this is due to greylisting by Hotmail.
• The greylisting is probably due to reputation and/or spam rejection. Your message was sent from a bulk sending service, and unless you include the legally required opt-out information (which I can't confirm just looking at the headers), Hotmail might consider it spam.
• The message does not pass reputation tests at Hotmail:
  • X-AUTH-Result: NONE
  • X-SID-Result: NONE
  • Authentication-Results: hotmail.com; spf=pass ... but dkim=permerror
• See: Hotmail accounts do not receive New order emails
• Are you using authenticated SMTP from your script? If not, this might be the cause.
• I suggest that you read through the Mandrill help or contact them to determine why Hotmail is giving permanent DKIM errors and not passing reputation tests.

Bill

[dc2000]

Thanks again, Bill.

While working on it yesterday I did discover that for some unknown to me reason the hosting company reset my DKIM settings, which might have caused some of those issues. Interestingly enough, after correcting the DKIM and SPF I resent a couple of test emails and the first one got through just fine (within seconds) while it took the second one almost 6 hours to arrive. There might be still something up with it... Man, it's all so complicated.

I'm not sure about the authenticated SMTP though. Here's the actual PHP script that sends the email through Mandrill servers:

		$mandrill = new Mandrill($MandrillKey);

		$message = array(
		    'subject' => '=?UTF-8?B?'.base64_encode($subject).'?=',
		    'from_email' => $fromEmail,
		    'from_name' => '=?UTF-8?B?'.base64_encode($fromName).'?=',
		    'text' => $msg,
		    'to' => array(array('email' => $toEmail, 'name' => '=?UTF-8?B?'.base64_encode($toName).'?=')),
		    'track_opens' => true,
		    'track_clicks' => false,
		    'auto_text' => false,
		    'auto_html' => false
		);

		$resAPI = $mandrill->messages->send($message);

		$sentStatus = $resAPI[0]["status"];

		if(strcasecmp($sentStatus, 'sent') == 0)
		{
			//Sent OK
		}

I got it from the official Mandrill API docs page. So I don't think it uses authenticated SMTP. What difference would it make, can you explain?

Also I'm curious about the "legally required opt-out information." Can you give some details on that? I'll include it if it's required.

[n5bb]

If you are sending commercial messages to persons who you are not in an ongoing business relationship with, you need to let them unsubscribe from your list. If you are just sending out single replies to purchases, this shouldn't be needed. I was just pointing our that since Mandrill normally is used for large commercial mailings, Hotmail might have put in special searches for opt-out messages.

If you send email which appears to be from an email client but isn't authenticated (is sent from an open relay without a password), you will be blocked by many email systems.

Bill

[dc2000]

Well, since it's all automate, is there any special format for the opt-out message?

PS. I explained my need of sending these emails here.