Last week the Envato Marketplaces reached a milestone: for the first time we handled over 70 million page requests on our Rails stack in one week … that’s 10 million a day!
Our Rails app powers seven separate Marketplace sites, the most well-known of which is Themeforest.net and based on this article it ranks in the top 5 Rails sites in the world by traffic. The marketplaces are for buying and selling a wide range of digital items, and we sell an item roughly every 10 seconds. A couple of our sellers–we call them authors–have sold $2,000,000 worth of items and another six have sold over $1,000,000.
The marketplace code base has been around for a while. In fact, the first commit was made before Rails 1.0 was released back in 2005. Since then, we’ve lovingly cared and tended to the app, with no wholesale rewrites. This certainly makes for challenging and lengthy major framework upgrades as this post details.
The reason we managed to scale Rails consistently is because we’ve been keeping our stack simple and stayed focused on maintaining our back-end performance. Today, our app is running faster than ever before, averaging a 117 millisecond response time across more than 70 million weekly requests. Our front-end performance isn’t too shabby either, as we average an end-user page load time of around 2.5 seconds in the US.
Even more important to us than these performance figures, is that we want to be able to manage this traffic while moving fast. One of the ways we measure this by how often we ship new versions of our app to our end users. The week we processed 70 million requests, we also deployed a new version of the app 17 times and over the last year we’ve deployed our app 753 times!
How are we doing this? The truth is: because this is the way it’s always been. We’ve been fortunate that there has been no “Dev Ops” divide to bring together or lengthy release cycles to shorten. The development workflow started with small changes deployed multiple times a day and supported by the developers that wrote the code from the day, 5 years ago, that the application was created. The key ingredients are:
- The developers “build and run” the sites and they participate in a primary on-call rotation, so they are “wearing the beeper”. If they cowboy changes, they take the call when things go wrong. We have no operations team to watch our back.
- We have a huge and comprehensive automated test suite.
- We have an engaged, tech savvy and vocal user base. If things go wrong, they tell us and we can revert quickly.
- We have substantial monitoring tools in place, and the development team keeps an eye on application performance continuously.
The number of deploys we do a day is one of the key metrics to gauge the agility of the team. Wrapped up in this number is a whole lot of information about how we go about our work.
- If our build tooling is flakey or slow, fewer builds leads to fewer merges into master and fewer deploys.
- If our test suite is not picking up bugs, this leads to failed deploys, which leads to fewer deploys.
- If our internal agile delivery processes become too heavyweight, this leads to fewer deploys.
- If our site is not reliable, it leads to fewer deploys as we fear changes will break the site.
In lean thinking, the rate of deployments is a classic measurement of flow. When continuous delivery is at the heart of your process—which it is at Envato—this number is a key metric to watch. The development team has nearly tripled in size in the last year, and we’ve kept a close eye on this deployment metrics during that time. As we focus on getting new team members up to speed on our codebase and domain, we’ve managed to maintain that throughput, despite the significant changes to personnel and process:
In subsequents posts we’ll discuss our architecture in more detail and the developer workflow that underpins our continuous delivery approach.
John Viner Software Development Manager, Envato Marketplaces