Building a CDN over SSL with CloudFront and SNI

One of the major disadvantage with using CDN is when you combine with an SSL certificate as the requirement for only a single certificate per IP makes it prohibitively expensive. CloudFront one of the most popular CDN networks has recently slashed its costs for using your own SSL by using SNI to allow multiple certificates on a single IP address. Having just gone through the process of migrating static assets off this little dedicated box which I host this site on and over to Amazon CloudFront this article is a combination walkthrough/how to and a bit of commentary thrown in.

This article covers:

• Setting up CloudFront and using your own custom origin server (no requirement for Amazon S3)
• Setting up CloudFront to use your own SSL certificate without a static IP, and the large monthly costs

Last week I gave a talk at WordPress Sheffield, and when I originally put together the talk I was expecting to give it to a small group of the regular attendees so it was a really nice to see a wider than normal audience at the event. The downside: my rather egotistical title “My Blog” presentation had to be quickly reworked on the fly to make it hopefully more useful to the wider audience. One of the things I discussed was how this site’s biggest issue is serving static assets. The little Atom processor and small amount of RAM means it’s not really suited to being a file server. While there are ways to improve performance of serving static assets the simplest solution is to push these assets to a CDN. When I set up the site again, I realised I would be collecting personal information both through the comments and on the left hand side via the newsletter signup (If you haven’t signed up, do it now :) it will only take a second, we will wait). Unfortunately while CDN pricing has come down in recent years, these prices shoot back up when you want to work with SSL, especially if you want to use your own SSL certificates rather than a shared certificate. For this site, the cost to use MaxCDN who have a great CDN solution or similar with SSL was just too costly as a personal project – though it’s a great solution for projects with a budget.

Enter CloudFront and its new SNI options

CloudFront, like most CDNs, has offered the option to use either their shared certificate or a dedicated IP and your own certificate at cost for some time. Recently they rolled out support for SNI (Server Name Indication). SNI has been unfairly been referred to as a hack, but really it’s an extension to TLS protocol and has been implemented and back ported into most server stacks and browsers. At the start of the handshake between the browser and server, SNI defines which hostname the handshake is for. This means a single IP can have multiple certificates associated with it, and without the need to have a dedicated IPs for every instance of an SSL certificate the costs rapidly go down. In Amazon CloudFront’s case, they have removed the monthly fees for supporting an SSL certificate if you use SNI. Sounds great!

Hold your horses, this is to good to be true!

Ah yes, like all the cool things, it’s not supported across all browsers… want to guess which browser doesn’t play ball?

On Windows Vista or higher, SNI is supported by IE 7+, but unfortunately it’s not supported in IE at all on Windows XP and earlier. Of course no one uses Windows XP so that’s ok [sarcasm]. It’s also not supported on Android default browser before V2.2, but other than that amongst major browsers support is pretty universal. Chrome has supported it since V5, Firefox since V2 and Safari since V3. Unfortunately there is no easy way to spot if a browser supports SNI, as by the time you start the handshake it’s to late to break out of it and almost certainly the browser is going to be presented with the wrong certificate. Because we can’t check for support, the only real way to ensure compatibility is to check if the browser is on a known list of not supporting browsers, which is neither graceful nor 100%.

This sticking point can be a complete no no for some use cases, if you are working on a corporate site or e-commerce store and have to support older browsers for example, and in such cases you may wish to consider going down the dedicated IP route, or potentially dropping SSL support though this should always be a last resort.

However for this blog, with zero IE 7 or earlier users, SNI is perfect and in case analytics is lying or more likely something is misrepresenting its user agent, and I can catch the few stragglers, if it becomes an issue.

There seem to be hundreds of guides to setting up CloudFront, and of course there’s the Amazon documentation, but the truth is it’s really simple to set up and contrary to virtually all the tutorials you DON’T need to use Amazon S3 to host content.

Here is how I set up cdn.timnash.co.uk

• Set up a Custom Origin Server
• Generated a certificate
• Created a CloudFront Distribution
• Set up WP-Stack CDN on the site
• Made a cup of tea

Set up a custom origin server

By default Amazon encourages you to use Amazon S3 to host your static content as the origin server however you can use any web server, so in theory you could just point CloudFront to your server and let it act as a proxy server, passing all your content. However I wanted to just serve my static file assets, so I created a new ‘server’ in my Nginx config (for apache users, this is like a new virtual host). This server reads the same directory tree as my main server but it’s set to not serve php, txt or html files. Unlike timnash.co.uk or www.timnash.co.uk my origin server does not run over https which means when files are being called by CloudFront it’s not doing any additional handshakes. However when CloudFront serves the file to the end user this is done over https.

Once you have set up your origin server or if you are happy for all your files to be served through CloudFront doing nothing, you will need to generate a certificate. If you are not planning on serving your CDN over SSL jump to creating a CloudFront Distribution step.

So why does Amazon push Amazon S3 as an origin server if you can run your own? A cynical person might suggest it was for the money, but there is one very practical reason to use Amazon S3 and that’s when you want to securely deliver files to specific individuals. To make use of Amazon CloudFront, a non Amazon S3 origin needs to not have any authentication, so your files are open for all to see. You could potentially lock down by IP to just Amazons IPs but basically its going to be wide open. On the other hand, you can use CloudFront with a private bucket on S3 and with signed keys download a time limited URL.

One of the nice things is you can have more then one origin server, so could host assets on Amazon S3 a custom origin server and EC2 if you wished. So while you may not wish Amazon S3 to be your primary origin server, you may wish to store some assets in a bucket (error pages for example) in case your primary origin server goes down.

If you are trying to work out what to host where you might want to look at this article on CloudFront design patterns.

Generating a Certificate and sending it to Amazon

Depending on your setup you may well already have a suitable certificate, if your domains SSL certificate is a wildcard and it uses at least 2048 bit encryption. Otherwise you will need to purchase one to cover the domain you will be using, in my case being a cheap skate, I purchased a second rapidSSL from NameCheap for cdn.timnash.co.uk. It’s worth emphasising we are uploading the certificate and private key to Amazon, as such you may wish to generate a separate key and certificate from the rest of your network regardless, if at additional cost.

Generating the certificate is exactly the same as you would do normally:
Generate Private key
Generate CSR
Go through your CA required approval steps, and collect your shiny certificate and intermediate certificate which along with your private key we will need in a bit.

Once we have our certificate and intermediate certificates, we need to upload them to Amazon to do this you need to use AWS Command Line tool.

If you haven’t already installed it the easiest way is via Python PIP system, on Debian/Ubuntu you can install this with:

sudo apt-get install python-pip

Probably don’t need to say it but the above needs Python also installed, but you knew that already ;)

Once up install AWS CLI:

sudo pip install awscli

Then you will need to associate your Amazon Access key and ID by running

aws configure

You don’t need to specify the region, though you can if you want, you will also need to have signed up for Amazon Web Services and CloudFront specifically.

Once we configured the tool we need to upload the certificate to Amazon. To do this we use the following command:

aws iam upload-server-certificate --server-certificate-name YOURNAME --certificate-body file://YOURSIGNEDCERT.pem --private-key file://YOURPRIVATEKEY.pem --certificate-chain file://INTERMEDIATECERT.pem --path /cloudfront/

so splitting it down, calling AWS and their identity service:

• YOURNAME – a way for you to identify the certificate once it’s uploaded in the admin area
• YOURSIGNEDCERT – the certificate sent to you from your CA
• YOURPRIVATEKEY – your private key, which you generated as part of your certificate process
• INTERMEDIATECERT – The intermediate key provided by your CA

We have add the extra –path /cloudfront/ to let Amazon know we will be using this on CloudFront, and it can be used across multiple distributions, though in reality it will be used on only one.

We can test that everything is fine by running:

aws iam get-server-certificate --server-certificate-name YOURNAME

You should get your certificate information back.

Creating a CloudFront Distribution

Next log into the Amazon Web Services console and select CloudFront and create Distribution and select a new Web Distribution.

On the next page follow the setup wizard I have highlighted a few steps below:

• Origin Domain Name – put the domain name of your origin server in here, you created earlier, just the name
• Origin ID – This is automatically generated when you put in a domain name, but can be customised
• Viewer Protocol Policy – I set this to Redirect HTTP to HTTPS just on the off-chance I screwed something up and a user pulls a request via http.

As you can see in the image, I have set the Price Class to just cover US and Europe which makes up just over 90% of my traffic, and as you can see I have set my CNAME to be cdn.timnash.co.uk. If everything has gone according to plan, under SSL Certificate option, you should see your certificate identified by the name we gave it earlier in the drop down.

Finally select “Only Clients that Support Server Name Indication” if you want to make use of SNI rather than a dedicated and costly IP.

That’s it, once you hit create distribution it will take a while to update, and while waiting you can add the CloudFront domain that’s been generated as a CNAME via your DNS server.

Creating custom error pages and logging

By default Cloudfront simply acts like a proxy server, and returns what ever it’s sent, so if your origin server sends a default Nginx 403 error message than that will be forwarded to the user. This is not ideal, partially because this is going to be cached by CloudFront but also because it’s going to get confusing both for the end user and for us when debugging.
Luckily CloudFront allows us to display custom error messages and manipulate the headers sent back to the user.

When editing a CloudFront distribution, select error pages, and create new error page, then select which error your origin server would serve, and what you want CloudFront to serve.

For example on timnash.co.uk I have it set so 403 by the origin as well as 404 from the origin server show as 404 on CloudFront. The Response page path is the path on the origin error server where the error messages can be found. You might want to create a Amazon S3 bucket that maps to your error page bucket, so when the origin times out you still display some sort of error message. It’s worth noting that CloudFront charges for showing the error message, at the cost it takes to show the message not the original resource. Great news if the 404 is a terabyte video, but not so good if its a 1kb file.

In a similar way to error messages we can create logging for CloudFront, with logging you MUST use a Amazon S3 bucket, I create a specific private bucket for my logging, you can specify a prefix if you are planning on putting multiple log files into a single bucket. Logging costs both to access the logs and for the data being written to the Amazon S3 bucket. On some sites this could quickly get expensive, but is the only reliable way to get logging information about your static assets.

Setup WP-Stack CDN

The final aspect is simply to upload WP-Stack CDN, this plugin replaces the domain name, of various assets based on their file extensions.

After uploading the plugin, either to plugins or mu-plugins folder, you just need to add a couple of lines in your wp-config.php

In my case:

define( 'WP_STACK_CDN_DOMAIN', 'cdn.timnash.co.uk' );
define( 'WP_stage', 'production' );

That’s it (well don’t forget to flush your WordPress cache if you have one set up). All the static assets are being served from cdn.timnash.co.uk while the rest of the site is coming from timnash.co.uk, both being served over SSL but with separate certificates.

Result: one happy Tim.

Going forward

So it’s not all roses, and there are a couple of things I need to tweak, specifically to support IE <7 and those still on WinXP as well as on older Android. While WP_STACK has several filters I can hook on to prevent the swapping of domains based on user agent (so at first glance this would seem the ideal solution), unfortunately because I'm using BatCache to do full page caching this isn't really a quick solution, as any checking would be overridden by the cache. To get round this I would need to invalidate the cache for the individual user, which will be pain. Given currently less than 0.01% of reported visitors are using IE <7 or Android 2.0 it's not a massive priority but is something I will be looking at implementing in the coming weeks. Likewise Amazon CloudFront, has no real hot-linking protection and while I can disable the distribution or change the CNAME if I have a particularly erroneous leacher I would like more control. The answer is in CloudFront signed URLs which allows me to create time limited or IP restricted URLs for all content. This would mean, that instead of making a call to say file.jpg it would be made to signedurl which returns file.jpg. If the url is invalid, because it's expired or IP doesn't match then the user is returned an error. This is often used to secure downloads but I intend to create signed downloads, with an expiry time of twice my current cache expiry. This means, that each time BatCache regenerates a cached page, it will also generate new signed signatures for any assets. If someone directly links to the signed URL it will invalidate after the first hour. There are a few edge cases to be worked out, such as images in feeds that maybe cached, and some social networks, link to images and store a cached version of the link, rather than the file itself. So until I have had a chance to test thoroughly I won't be deploying the CDN with signed URLs.

So has it actually made a performance difference?

Only a small one, but that wasn’t the reason for implementation but rather to reduce the resource requirements on the small dedicated box. As such where the biggest gain has come is when looking at rush based tests (where you send lots of concurrent user requests from multiple locations) and this has seen a rise from 250 concurrent users per second to 375 concurrent users, with a drop in errors to below <0.5% at the 300 user mark. It's unlikely this site is ever going to see 375 concurrent users, let alone for any length of time but should it happen the box will stay up (fingers crossed) and it's worth remembering how small this little box is, chances are your mobile phone is more powerful than this server. So was it worth it? Yep, it was easy to implement, and feels like a robust solution, I have some edge cases to sort out but definitely seems like the way to go. There are still lot's of little things that can be done to optimise the server performance wise and tweaks to be made to CloudFront as well so this is still an ongoing project. The next step is to look at implementing SPDY support on Nginx though by serving nearly all static files through CloudFront it may not have the performance gains reported by many.

TLDR;

Amazon CloudFront is remarkably simple to set up, you don’t need a S3 account, simply use your own HTTP server as the origin. SNI allows multiple SSL certificates to be associated with a single IP but is not 100% supported. Amazon CloudFront supports using your own SSL certificate with SNI. The combination means you can quickly build a CDN for your site, assuming you don’t want to ever serve Window XP users, and who would?

WAIT!

Before you go off to setup your Amazon CloudFront CDN, I would love you to do 3 things for me

1. If you haven’t already please do sign up to the newsletter, it’s monthly and is packed with useful information and the mail going out the end of this month (April) will have some additional CDN/Performance bits in it
2. I will be doing a follow up to this post, with answers to any questions, but to do this I need your thoughts and questions so please do leave me a comment or message either here or on social media of your choice. I will try my best to answer as many queries as possible.
3. Finally please share this article, you can use the social media buttons below, or copy and paste the URL :p but if you found this useful, others might as well so please share.

That’s a wrap, thanks for reading, don’t forget to comment, tweet and sign up!