Last month we announced that we had finally completed the move to HTTPS everywhere for Envato Market. This was no easy feat since we are serving over 170 million page views a month that includes about 10 million products listed and are all user generated content. Along the way we have learnt many valuable lessons that we want to share with the wider community and hopefully make other HTTPS moves easier and encourage a better adoption of HTTPS everywhere.
Behind the scenes, the groundwork for the HTTPS rollout started back in 2014 with a couple of the engineers implementing a feature toggle which allowed staff to opt-in for HTTPS. For a long time, this sat dormant and unused by most staff until earlier this year when a few engineers got together and decided it was time to give HTTPS everywhere another push and get it to general availability.
HTTPS isn’t just about the having a padlock or green indicator shown in the browser. It’s about creating a trusted connection between the end user and your services via three protection layers:
- Encryption: Securing the exchanged data to prevent eavesdropping on the connections.
- Data integrity: Confidence that the data has not been altered mid transit without being detected.
- Authentication: Assuring the website you are connecting to is who you expect them to be.
An added side effect of migrating to HTTPS is that you can unlock HTTP/2 and features like request multiplexing and server push which are great news for performance! Last year, Google announced HTTPS as a ranking signal so by migrating to HTTPS you get a boost in your search results too!
User managed content
We have a lot of user managed content. The problem here is that many of our authors don’t have time or additional funds to implement things like content delivery network (CDN for short) caching so most of our user managed content requests would end up needing to hit their origin servers to fulfill requests. This was bad for a few reasons:
- Many authors use shared hosting or very small instances for storing these assets. During the testing phase, we generated a low amount of traffic for a particular set of assets that would sometimes take over 20 seconds to complete! The result of these slow load times is a very poor experience for buyers and would result in many people looking elsewhere because they couldn’t see previews or screenshots of the product quickly enough.
- Very little HTTPS adoption. If we intended to serve our pages under HTTPS, we needed to ensure the assets on the page were also served securely. The issue here is that it’s very unrealistic for Envato to force users to spend time (and potentially money) on updating all of their assets to be served via HTTPS to avoid seeing mixed content warnings on the item pages.
To solve both of these issues, we decided to use an approach that consisted of an image proxy and a CDN. The image proxy would rewrite all of the non-secure links at render time to point at our CDN which would help speed up response times and allow us to take some of the load off author origins by caching the assets.
Initially we used camo which was built by Corey Donohoe
when he was at GitHub where they needed to solve a similar
issue. This worked well for us until we started
trying to scale it to handle more traffic. GitHub solved the scaling
issue by adding more worker processes however we decided to try adding
clustering support so that we could utilise more of
the hardware we already had in place. This didn’t solve the problem for
long and we eventually ended up back in the same position and needed to
resize our hardware to account for the additional load. We determined
this wasn’t going to be a viable path going forward and we needed a
better solution. After some looking around we found go-camo
which is a Go port of Corey’s original project. For a while we ran the
two implementations side by side and discovered that
go-camo was able
to better utilise all of the existing hardware (due to its ability to
use more than a single operating system thread) and was easier to debug
when issues popped up. After a couple of weeks of load testing we
decided to completely swap to
go-camo and start ramping up the number
of users who were using this.
As you may know, Envato Market is built using Ruby on Rails and out of the box Rails offers the ability to define how you
handle your cookies. To continue with our
incremental rollout, we needed to allow user cookies to be able to be
accessible on HTTP or HTTPS depending on which protocol their request
was served via. This was achieved by omitting the
Secure flag on
cookies until we were confident post rollout that we were not going to
One of the big concerns from teams looking to undertake HTTPS migrations is that they will incur a performance hit once it’s live and in most cases, it’s just not true. Deploying to modern hardware/software setup and using a suitable cipher suite mitigates many of the performance bottlenecks that used to be associated with HTTPS. This definitely doesn’t mean you shouldn’t collect metrics around these areas and monitor them, it just means that “HTTPS is slow” is not a valid excuse.
In Envato’s case, we haven’t seen any performance impacts and our end user time is consistent with the weeks prior to the HTTPS rollout.
One of the most important things you can do during a HTTPS migration is Monitor All The Things. By having insight into the changes within your stack during the migration you can quickly detect an issue before all your users do. During our rollout some of the metrics we kept a very close eye on were:
- Exception rate
- Time spent in network requests
- End user response time
- Application response time
- Instance resource utilisation (CPU specifically)
- Total number of requests
- Edge network requests and the count by status code
During our rollout we identified a couple of issues, most notably a load balancer misconfiguration. We were seeing a CPU spike on a small subset of web instances that were missed in all of our testing which we managed to catch before rolled it out to all of our users.
Here are two that we put together to keep everyone informed about how far through the rollout we were. The top one was the initial rollout (mostly just staff usage) and the second is when we decided to cut everyone over to HTTPS.
2016 has been a big year for SEO at Envato. We’ve kicked off many initatives targetting better visibility for search engines into our author products and during early discussions it was decided we needed to be extra careful during the migration not to undo all of the hard work we’ve put into the last 7 months. To ensure we didn’t do go backwards we took a couple of steps that has helped us stay on top and improve our searh engine rankings:
- Submit both HTTP and HTTPS sitemaps to Google webmaster tools: In the week leading up to the swap over, we took a snapshot of our sitemaps and uploaded them into Google webmaster tools as a new set of sitemaps. This was done to ensure that when we swapped over to HTTPS Google would have access to a HTTP and HTTPS sitemap source and would allow Google to continue crawling the HTTP sitemap but at the same time be lead into the HTTPS version of the site.
- Ensure we maintained 1:1 redirects: This helped ensure our users (and bots) still knew where to find us even though we moved to HTTPS.
In taking these steps, 61% of high volume terms we track have remained stable or improved their rankings since the HTTPS release. The remaining terms that have moved backwards were not on page 1 and have not actually lost us traffic or revenue.
The migration wasn’t completed overnight and took longer than we would have liked but we’ve managed to roll this out without any negative impacts on our users or application which is something we are extremely proud of. Hopefully by publishing our journey this will give others information about migrating to HTTPS along with the added benefits that come with it and remove the stigma that HTTPS is only a painful experience.