So I rather blew it. As part of the move to the new look site I took the opportunity to move from MailChimp to Buttondown for my newsletter. However things did not go according to plan…
This turned out to be a bit of a journey with some interesting lessons that I thought were worth documenting.
Let’s start at the begining:
I chose Buttondown as it came recommended by my friend Dave Smyth from Work Notes; he had recently migrated his own list over. Some googling and general checking out and it seemed to be a good fit for me. Simple email service that allowed me to just write emails.
A small independent with a privacy focus, It’s not perfect, the documentation is in Notion and some of the interface is probably best described as intentionally minimal.
Migration from MailChimp
The migration from MailChimp was actually a pretty painless exercise. If anything, I would argue the onboarding is easier than the migration between Tiny Letter and MailChimp, a process I had done earlier that year.
From within MailChimp I spent a few minutes fighting its UI but eventually worked out how to download my audience. Then it was a case of uploading them to Buttondown.
OK, this was particularly easy – win to Buttondown!
Setting up Buttondown
OK, so Buttondown is not complicated but it does have settings in multiple locations and I will be honest. not all of them are very logical to me. Some options are “Premium” as well and the GUI doesn’t really do much to help you work out what is what, barring the occasional tool tip.
One set of settings is the ability to specify which email you want the email to be sent from. This aids in deliverability, tick
So now we need to set some DNS, and a little bit of the magic is revealed, as, well, Buttondown is a GUI wrapper for Mailgun under the hood, as the DNS records we are pointing to are all Mailgun related.
This is a good thing, sending email is HARD. I don’t want someone trying to do it, so I am much happier knowing a specialist company is doing the hard work. However, I can’t help but think if it was my project I would have put those DNS records under my vanity nameservers, even if I just forwarded on all requests. As now, should Buttondown try to migrate to a different mail provider (or branch to multiple) we do have a bit of an issue, and lots of DNS records will need updating.
Long term I see a potential painful “upgrade” is going to be needed.
One of the DNS changes needed was to create a CNAME email.timnash.co.uk note “email” not “mail”, I will skip over how long it took me to notice I had pointed to “mail” by accident. Basically the email.timnash.co.uk is a CNAME to mailgun.com
dig email.timnash.co.uk
; <<>> DiG 9.10.6 <<>> email.timnash.co.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24929
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;email.timnash.co.uk. IN A
;; ANSWER SECTION:
email.timnash.co.uk. 180 IN CNAME mailgun.org.
mailgun.org. 60 IN A 44.236.174.11
mailgun.org. 60 IN A 54.186.54.23
;; Query time: 15 msec
;; SERVER: 192.168.1.254#53(192.168.1.254)
;; WHEN: Mon Dec 07 18:00:21 GMT 2020
;; MSG SIZE rcvd: 105confirms it was set up correctly. OK, so I will admit at this point I only did a dig, I made no HTTP request, I was in my head sorting some DNS bits out. At this stage an alarm bell should have been ringing in my head.
However, it, and the other records such as SPF, all set up and confirmed both by me and Buttondown interface, we are good to go.
Writing the first email
So I write the email in MarkText then paste it into a Google Doc, for Babs to edit, just like this post. Then I copy and paste back out into the mail client, in this case Buttondown, and fix any weird foibles that might occur. With MailChimp this was a nightmare as Google Docs and MailChimp both tried to be clever, so it was nice to just be able to straight copy and paste and the markdown be automatically added. Generally it looked OK. I deliberately didn’t play too much with making things fancy, I figured I wanted to build such functionality up and I also wanted to experiment with the plainer looking email. What I did do is test! I sent myself 5 separate preview emails which I checked on:
- Gmail Web Client
- Android Gmail App
- K9Mail a Android App
- Apple Mail
Each time I was checking things like DO THE LINKS WORK and they did!
OK, time to send an email. It was at this point I found tags, as, until then, I hadn’t actually realised I could tag users. This would be useful later, and would have been really useful earlier.
Anyway, I hit send immediately and off went the email to my inbox.
I checked my inbox, sure enough there was an email!
- Deliverability tick
Happy it had sent, I started stabbing around and realised the analytics were a little more complete then I was expecting. After all I had set up so I wouldn’t be collecting such stats.
It’s OK,I thought, it’s probably a bit of an interface bug.
Then I got a Tweet…
That’s strange, why would Claire be having issues? Of course my first thought was there was an issue with my site and I jumped to the Ransomware Article. OK, so there is no issue with the site itself, phew!
On opening the email that had been sent to me, I clicked on the Ransomware post and, yep, I had an identical issue!
Looking at the link:
http://email.timnash.co.uk/c/eJyNjkluxCAURE8DOywmMyxYJGn7GhaTY5TGRAarldsH9y7KpqW_qdJ7pR8Ml6NgFCZDMcWEYok1k1gMZBCznOm7mqaPm8ATHwHHLeXd1m3wZTi_4GY8H4MLXPjAgibK2bBKqn1UJPrY5-DdbK19V8DeAJ37_RnoebMV_ZTzQFupDX0WdNi9lvywR7wENp8tL7UDPgJ2a5cLqLjKHEM6cy9jtukOD1NTi_9ebEY4YplmCrnIGeJREqS8d8gqq70fx7CyFbqztbKH8tiX59ySwmvik34J_QX8EXAEHmmm OK, that apparently is a tracking URL. My first thought was that the domain is not working properly so I quickly CURL’d the response
curl -I http://email.timnash.co.uk/c/eJyNjkluxCAURE8DOywmMyxYJGn7GhaTY5TGRAarldsH9y7KpqW_qdJ7pR8Ml6NgFCZDMcWEYok1k1gMZBCznOm7mqaPm8ATHwHHLeXd1m3wZTi_4GY8H4MLXPjAgibK2bBKqn1UJPrY5-DdbK19V8DeAJ37_RnoebMV_ZTzQFupDX0WdNi9lvywR7wENp8tL7UDPgJ2a5cLqLjKHEM6cy9jtukOD1NTi_9ebEY4YplmCrnIGeJREqS8d8gqq70fx7CyFbqztbKH8tiX59ySwmvik34J_QX8EXAE
HTTP/1.1 302 FOUND
Content-Length: 383
Content-Type: text/html; charset=utf-8
Date: Tue, 08 Dec 2020 20:19:11 GMT
Location: https://timnash.co.uk/has-your-host-go-ransomware/?utm_source=tnash&utm_medium=email
Server: nginx
Connection: keep-aliveOK, so that is hitting that domain and is apparently working, but it doesn’t in the browser. By now I had already worked out why, but just to be sure I did:
curl -I https://email.timnash.co.uk/c/eJyNjkluxCAURE8DOywmMyxYJGn7GhaTY5TGRAarldsH9y7KpqW_qdJ7pR8Ml6NgFCZDMcWEYok1k1gMZBCznOm7mqaPm8ATHwHHLeXd1m3wZTi_4GY8H4MLXPjAgibK2bBKqn1UJPrY5-DdbK19V8DeAJ37_RnoebMV_ZTzQFupDX0WdNi9lvywR7wENp8tL7UDPgJ2a5cLqLjKHEM6cy9jtukOD1NTi_9ebEY4YplmCrnIGeJREqS8d8gqq70fx7CyFbqztbKH8tiX59ySwmvik34J_QX8EXAE
curl: (7) Failed to connect to email.timnash.co.uk port 443: Connection refusedSpot the difference?
Over HTTPS that url is failing to connect, but what is causing the insecure connection to be upgraded?
HTTP Strict Transport Header
The HTTP Strict Transport Header (HSTS) is a special Header sent by a server (in this case mine, or more specfically Netlify on my main domain) to say only try to connect over HTTPS. The first time a browser tries to connect it can do it over HTTP or HTTPS but then it’s presented with this header in the first response and from then on, if a browser gets a HTTP link for the site it will automatically jump to using HTTPS version.
So my site (timnash.co.uk) has the HSTS header set that looks like this:
strict-transport-security: max-age=31536000; includeSubDomains; preloadThis basically say that you should always come to this site over HTTPS:
- For the next year max-age
- This policy includes Subdomains includeSubDomains so that means it effects email.timnash.co.uk
- And that where possible it should be preloaded by browser preload
In addition my site is registered with https://hstspreload.org/, a website run by Google that allows sites that meet the conditions (basically the above configuration) to request that they are preloaded into Chrome and other Browsers. Meaning the browser will use the HTTPS version only, even if it’s never visited the site before.
So now we have a problem:
the email.timnash.co.uk subdomain was running over HTTP but not HTTPS but virtually every visitor would have a browser saying it knows NOT to run any subdomain over HTTP and to only go over HTTPS. So all the tracking URLs are broken, which means all the links in my newsletter are broken.
AAAAAGGGGHHHHHHHHHHH
BUT BUT BUT, I checked this!
OK, slightly panicking I checked this. All the links were working in the preview, why?
Going back to the preview emails, and checking the links, the preview email links were not going through the email.timnash.co.uk domain. Presumably to stop issues where they get caught up in tracking!
But why was it tracking? I didn’t want it tracking. Glancing at the analytics I had some tracking information (presumably from email apps, rather than browsers) .
OK, so the issue appears to be the tracking, the tracking I don’t want.
How do I turn the tracking off? When you are doing things in a bit of a hurry you get grumpy. I was quite grumpy when I eventually tracked down the tickbox for:
Would you like to opt out of analytics? YES!
Ok so I have opt-ed out, now how to test, as when I tested previously I couldn’t see this issue as it wasn’t including the tracking links.
Buttondown tags
Tags seemed like the answer, and actually this is where simplicity really did win, and is one of the reasons I still really like Buttondown. I was able to generate a TEST tag, attach it to a single user and then send the email again by using the “reuse” option to resend. Except you have to change at least something in the subject, and body or it doesn’t send anything and instead you get an email telling you off for duplicate emails.
So with the copy re-sent, I could confirm no domain hopping, meaning no problems. So now I could reuse the email once more, add an apology at the top and resend.
Phew…
Lessons from all this!
So what have I learnt? This was frustrating as I feel like I did my side of the testing and it was just unfortunate it went wrong but there are still a few good lessons for anyone to take away.
- In 2020 NOTHING should default over http, there just is not a valid reason to not provide a HTTPS endpoint. This is not Buttondown’s fault particularly but Mailgun’s and it’s inexcusable. With more and more sites using HSTS (including whole TLDs like .dev) and with Firefox having a HTTPS only mode. This is going to start being a real issue.
- If you are providing something as Privacy First I read that as privacy by default. Some of that is on me, the documentation does say you can turn off, rather than turn on. But the option is somewhat hidden and finding it was reminiscent of trying to find plans for Intergalactic Highways. I really don’t want to moan about Buttondown, that’s not the point of this article but if I was to moan, finding information is not easy, I mean the pricing page doesn’t even have, well, pricing! It is a 1-man project with all that comes with it. Anyway it was on me because I misread, but I feel also a little bit like it was cheeky sales material.
- I give talks on security headers, I manage a WordPress Security Headers plugin, I know HSTS headers can cause issues, but I will be honest this is the first time I have actually had this happen where it’s anything but a little annoying. If you are enabling HSTS header, you are enabling it for “life” and for everything on that domain. I still believe that’s a good thing, but it’s important to understand and account for.
- Apparently people don’t mind you screwing up, in fact it was a bit of a double whammy, because while I was tweeting apologies and fixing the issue, folks seeing those tweets were signing up to the email except…
https://twitter.com/nickwilmot/status/1335901225364172802
This was entirely my own fault, I managed to forget to includename="email"so it wasn’t passing the email field. DOH! Worse, it was working fine on the homepage, the homepage which should have been the same reusable Block, except I, for some reason, hard coded one but not the other!
Oh well, all in all far worse things have happened but it was a good learning exercise and, do you know what, I actually still really like Buttondown.
So will leave you with this final thought.