Managing SSL certificates and HTTPS configuration at scale

Our multi-network multisite WordPress installation at WSU has 1022 sites spread across 342 unique domain names. We have 481 SSL certificates on the server to help secure the traffic to and from these domains. And we have 1039 unique server blocks in our nginx configuration to help route that traffic.

Configuring a site for HTTPS is often portrayed as a difficult process. This is mostly true depending on your general familiarity with server configuration and encryption.

The good thing about process is only having to figure it out a few times before you can automate it or define it in a way that makes things less difficult.

Pieces used during SSL certification

A key—get it—to understanding and defining the process of HTTPS configuration is to first understand the pieces you’re working with.

  • Private Key: This should be secret and unique. It is used by the server to sign encrypted traffic that it sends.
  • Public Key: This key can be distributed anywhere. It is used by clients to verify that encrypted traffic was signed by your private key.
  • CSR: A Certificate Signing Request. This contains your public key and other information about you and your server. Used to request digital certification from a certificate authority.
  • Certificate Authority: The issuer of SSL certificates. This authority is trusted by the server and clients to verify and sign public keys. Ideally, a certificate authority is trusted by the maximum number of clients. (i.e. all browsers)
  • SSL Certificate: Also known as a digital certificate or public key certificate. This contains your public key and is signed by a certificate authority. This signature applies a level of trust to your public key to help clients when deciding its validity.

Of the files and keys generated, the most important for the final configuration are the private key and the SSL certificate. The public key can be generated at any time from the private key and the CSR is only a vessel to send that public key to a certificate signing authority.

Losing or deleting the SSL certificate means downloading the SSL certificate again. Losing or deleting the private key means restarting the process entirely.

Obtaining an SSL certificate

The first step in the process is to generate the private key for a domain and a CSR containing the corresponding public key.

openssl req -new -newkey rsa:2048 -nodes -sha256 -keyout jeremyfelt.com.key -out jeremyfelt.com.csr

This command will generate a 2048 bit RSA private key and a CSR signed with the SHA-256 hash algorithm. No public key file is generated as it is inserted directly into the CSR file.

Next, submit the CSR to a certificate signing authority. The certificate signing authority will sign the public key and return a digital certificate including the signature, your public key, and other information.

The certificate signing authority is often the part of the process that is annoying and difficult to automate.

If you’re purchasing the signature of a certificate through a certificate authority or reseller such as GoDaddy or Namecheap, the steps to purchase the initial request, submit the CSR, and download the correct certificate file can often be confusing and very time consuming.

Luckily, in WSU’s case, we have a university subscription to InCommon, a reseller of Comodo certificates. This allows us to request as many certificates as we need for one flat annual fee. It also provides a relatively straight forward web interface for requesting certificates. Similar to other resellers, we still need to wait as the request is approved by central IT and then generated by Comodo via InCommon.

Even better is the new certificate authority, Let’s Encrypt, which provides an API and a command line tool for submitting and finishing a certificate signing request immediately and for free.

Configuring the SSL certificate

This is where the process starts becoming more straight forward again. And where I’ll only focus on nginx as my familiarity with Apache disappeared years ago.

A cool thing about nginx when you’re serving HTTP requests is the flexibility of server names. It can use one server block in the configuration to serve thousands of sites.

server {
    listen 80;
    server_name *.wsu.edu wsu.io jeremyfelt.com foo.bar;
    root /var/www/wordpress;
}

However, when you serve HTTPS requests, you must specify which files to use for the private key and SSL certificate:

server {
    listen 443 ssl http2;
    server_name jeremyfelt.com;
    root /var/www/wordpress;

    ssl on;
    ssl_certificate /etc/nginx/ssl/jeremyfelt.com.cer;
    ssl_certificate_key /etc/nginx/ssl/jeremyfelt.com/key;
}

If you are creating private keys and requesting SSL certificates for individual sites as you configure them, this means having a server block for each server name.

There are three possibilities here:

  1. Use a wildcard certificate. This would allow for one server block for each set of subdomains. Anything at *.wsu.edu would be covered.
  2. Use a multi-domain certificate. This uses the SubjectAltName portion of a certificate to list multiple domains in a single certificate.
  3. Generate individual server blocks for each server name.

A wildcard certificate would be great if you control the domain and its subdomains. Unfortunately, at WSU, subdomains point to services all over the state. If everybody managing multiple subdomains also had a wildcard certificate to make it easier to manage HTTPS, the likelihood of that private key and certificate leaking out and becoming untrustworthy would increase.

Multi-domain certificates can be useful when you have some simple combinations like www.site.foo.bar and site.foo.bar. To redirect an HTTPS request from www to non-www, you need HTTPS configured for both. A minor issue is the size of the certificate. Every domain added to a SubjectAltName field increases the size of the certificate by the size of that domain text.

Not a big deal with a few small domains. A bigger deal with 100 large domains.

The convenience of multi-domain certificates also depends on how frequently domains are added. Any time a domain is added to a multi-domain certificate, it would need to be re-signed. If you know of several in advance, it may make sense.

If you hadn’t guessed yet, we use option 3 at WSU. Hence the 1039 unique server blocks! 🙂

From time to time we’ll request a small multi-domain certificate to handle the www to non-www redirects. But that too fits right into our process of putting the private key and certificate files in the proper place and generating a corresponding server block.

Using many server blocks in nginx for HTTPS

Private keys are generated, CSRs are submitted, SSL certificates are generated and downloaded.

Here’s what a generated server block at WSU looks like:

# BEGIN generated server block for fancy.wsu.edu
#
# Generated 2016-01-16 14:11:15 by jeremy.felt
server {
    listen 80;
    server_name fancy.wsu.edu;
    return 301 https://fancy.wsu.edu$request_uri;
}

server {
    server_name fancy.wsu.edu;

    include /etc/nginx/wsuwp-common-header.conf;

    ssl_certificate /etc/nginx/ssl/fancy.wsu.edu.cer;
    ssl_certificate_key /etc/nginx/ssl/fancy.wsu.edu.key;

    include /etc/nginx/wsuwp-ssl-common.conf;
    include /etc/nginx/wsuwp-common.conf;
}
# END generated server block for fancy.wsu.edu

We listen to requests on port 80 for fancy.wsu.edu and redirect those to HTTPS.

We listen to requests on port 443 for fancy.wsu.edu using a common header, provide directives for the SSL certificate and private key, and include the SSL configuration common to all server blocks.

wsuwp-common-header.conf

This is the smallest configuration file, so I’ll just include it here.

listen 443 ssl http2;
root /var/www/wordpress;

Listen on 443 for SSL and HTTP2 requests and use the directory where WordPress is installed as the web root.

These directives used to be part of the generated server blocks until nginx added support for HTTP2 and immediately deprecated support for SPDY. I had to replace spdy with http2 in all of our server blocks so instead decided to create a common config and include it.

WSU’s wsuwp-common-header.conf is open source if you’d like to use it.

wsuwp-ssl-common.conf

This is my favorite configuration file and one I often revisit. It contains all of the HTTPS specific nginx configuration.

# Enable HTTPS.
ssl on;

# Pick the allowed protocols
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

# And much, much more...

This is a case where so much of the hard stuff is figured out for you. I regularly visit things like Mozilla’s intermediate set of ciphers and this boilerplate nginx configuration and then make adjustments as they make sense.

WSU’s wsuwp-ssl-common.conf is open source if you’d like to use it.

wsuwp-common.conf

And the configuration file for WordPress and other things. It’s the least interesting to talk about in this context. But! It too is open source if you’d like to use it.

The process of maintaining all of this

At the beginning I mentioned defining and automating the process as a way of making it less difficult. We haven’t yet reached full automation at WSU, but our process is now well defined.

  1. Generate a private key and CSR using our WSUWP TLS plugin. This provides an interface in the main network admin to type in a domain name and generate the required files. The private key stays on the server and the CSR is available to copy so that it can be submitted to InCommon.
  2. Submit the CSR through the InCommon web interface. Wait.
  3. Upon receipt of the approval email, download the SSL certificate from the embedded link.
  4. Upload the SSL certificate through the WSUWP TLS interface. This verifies the certificate’s domain, places it on the server alongside the private key, and generates the server block for nginx.
  5. Deploy the private key, SSL certificate, and generated server block file. At the moment, this process involves the command line.
  6. Run nginx -t to test the configuration and service nginx reload to pull it into production.
  7. In the WSUWP TLS plugin interface, verify the domain responds on HTTPS and remove it from the list.

Looking at the steps above, it’s not hard to imagine a completely automated process, especially if your certificate authority has a way of immediately approving request and responding with a certificate. And even without automation, having this process well defined allows several members of our team to generate, request, and deploy certificates.

I’d love to know what other ways groups are approaching this. I’ve often hoped and spent plenty of time searching for easier ways. Share your thoughts, especially if you see any holes! 🙂

Previously:

My first Let’s Encrypt certificate

The timing of the Let’s Encrypt beta could not be more perfect as my previous certificate expires on November 18th. I purposely purchased only a 1 year certificate because I knew Let’s Encrypt was coming. Let’s see how this works!

6:00pm

In my email, I have an invite to Let’s Encrypt for 3 whitelisted domains—jeremyfelt.com, www.jeremyfelt.com, and content.jeremyfelt.com. Per the documentation, I cloned the git repository to a spot on my server—I chose /home/jeremyfelt/—so that I could use the client.

I admit that I haven’t read any documentation up until this point, so I’m flying blind and impatient like normal. 🙂

My first attempt at running the ./letsencrypt-auto command was interesting, but kind of a failure. A ton of dependencies were installed, which is good. I have an outdated version of Python apparently, which is annoying.

WARNING: Python 2.6 support is very experimental at present…
if you would like to work on improving it, please ensure you have backups and then run this script again with the –debug flag!

It took me several attempts before I finally read the message above and figured out that I was supposed to run the Let’s Encrypt command as ./letsencrypt-auto --debug to even pass to the next level. If you have Python not 2.6, this probably won’t be an issue.

Ok. Figured that out, then that crazy fake linux light blue GUI comes up… progress! Go through a couple steps and get this:

Screen Shot 2015-11-12 at 6.09.52 PM

Right then. By default, the Let’s Encrypt client wants to be a web server so that it can properly communicate authority. This would be excellent if I didn’t already have a web server running.

At this point, I read down the page of documentation a bit and realized I could (a) use a config file and (b) use text only instead of ncurses. Sweet!

When I setup the default config file as /etc/letsencrypt/cli.ini, I noticed an option for webroot authenticator. This looked more promising as a way to handle authentication through Nginx. I enabled this and tried again.

And failed! “Failed authorization procedure” to be exact. My client told the other side to verify at http://jeremyfelt.com/.well-known/acme-challenge/BIGLONGSTRING, but my default Nginx configuration blocks public access to all hidden files.

I added a location block to Nginx specifically to allow .well-known and tried again.

Success! Authorization worked and a bunch of files were generated that look like what I need. I went into my Nginx configuration and updated the ssl_certificate directive to point at fullchain.pem and the ssl_certificate_key directive to point to privkey.pem. nginx -t has no complaints… let’s restart the server.

Failure! Big red X, invalid everything! The issuing CA is….. happy hacker fake CA.

Oh. Quick Google search and sure enough:

“happy hacker fake CA” is the issuer used in our staging/testing server. This is what the Let’s Encrypt client currently uses when you don’t specify a different server using the--server option like you did in the original post. Because of this, I believe the --server flag was not included when you ran the client. Try running the client again, but make sure you include the --server option from your original post.

Thank you, bmw!

I failed to update the cli.ini file that I had copied from the web to use the production API instead of the staging API.

Fix the server URL, try again. Success! And for real this time.

Screen Shot 2015-11-12 at 6.24.04 PM

I repeated the process with www.jeremyfelt.com and content.jeremyfelt.com, making fewer mistakes along the way and that’s that.

  • Here’s the final cli.ini file that worked for me.
  • And the final command line arguments: ./letsencrypt-auto --config /etc/letsencrypt/cli.ini --debug --agree-dev-preview certonly

6:36pm

I have 3 new certificates. I still have an A+ in SSL Labs. Nothing is broken. The future is here.

Thank you to everyone at Let’s Encrypt!

OpenSSL commands that came in useful today

When nginx -t complained about a certificate/key mismatch this afternoon, I first assumed that the problem was on our end during our automated CSR/key generation or our certificate request process. I took a closer look at all three pieces to look for the source of the error using “The Most Common OpenSSL Commands“:

openssl rsa -in example.test.key -check

The info from the key check was pretty unhelpful, but it was a valid key. See the section below for how to better compare that.

openssl req -text -noout -verify -in example.test.csr

The CSR check was somewhat helpful as I was able to verify that the correct domain name and other request information was in place.

openssl x509 -in example.test.cer -text -noout

The certificate check was most helpful as I was able to diff the results of this with the results of a working certificate. This showed me that nothing was off and all data was formatted as expected, just different.

I turned to searching for the verbose error instead.

Via “SSL Library Error: 185073780 key values mismatch“, I used these commands to compare a certificate and private key to see if they were indeed not matching:

  • openssl x509 -noout -modulus -in example.test.cer | openssl md5
  • openssl rsa -noout -modulus -in example.test.key | openssl md5

Each of these generated an md5 hash that I was able to compare. In my case, the error reported by nginx -t was correct and the certificate generated by Comodo did not match my private key. I double checked this by comparing a working certificate/key pair that resulted in matching md5 hashes.

Bah. This is nice because it’s likely not our fault. This is not nice because now we have less control over fixing it. 😞

I do have a set of commands that may come in useful again. 😃