How we tracked down Ruby heap corruption in amongst 35 million daily requests

Back in November 2015, one of the Envato Market developers made a startling discovery - our exception tracker was overrun with occurrences of undefined method exceptions with the target classes being NilClass and FalseClass. These type of exceptions are often a symptom that you’ve written some Ruby code and not accounted for a particular case where the data you are accessing is returning nil or false. For our users, this would manifest itself as our robot error page letting you know that we encountered an issue. This was a particularly hairy scenario to be in because the exceptions we were seeing were not legitimate failures and replaying the requests never reproduced the error and code inspection showed values could never be set to nil or false.

Running Headless JavaScript Testing with Electron On Any CI Server

Background

Since the end of 2015, the Envato Front End team has been working on bringing a modern development workflow to our stack. Our main project repo powers sites like themeforest.net and serves around 150 million Page Views a month, so it is quite a challenge to re-architect our front end while maintaining a stable site. In addition, the codebase in 9 years old, so it contains the code from many developers and multiple approaches.

We recently introduced our first React based component into the code base when we developed an autosuggest search feature on the homepage of themeforest.net and videohive.net. The React component was written with ES6, and uses Webpack to bundle the JavaScript code.

As I mentioned above, it’s a 9 year old code base and nobody can guarentee that introducing something new won’t break the code, so we began all the work with tests in mind. This post documents our experiences developing the framework for testing the React based autosuggestion component.

Introducing StackMaster - The missing CloudFormation tool

StackMaster

CloudFormation is an Amazon (AWS) service for provisioning infrastructure as “stacks”, described in a JSON template. We use it a lot at Envato, and initially I hated it. Typing out JSON is just painful (literally!), and the APIs exposed in the AWS CLI are very asynchronous and low level. I wanted something to hold my hand and provide more visibility into stack updates.

Today I’d like to introduce a project we’ve recently open-sourced: StackMaster is a tool to make working with multiple CloudFormation stacks a lot simpler. It solves some of the problems we’ve experienced while working with the CloudFormation CLI directly. The project is a refinement of some existing tooling that we have been using internally at Envato for most of this year, and it was built during one of Envato’s previous “Hack Fortnights”.

See the changes you are making to a stack before you apply them:

When applying a stack update with StackMaster, it does a few things. First you’ll see a text diff of the proposed template JSON and the template that currently exists on CloudFormation. This helps sanity-check the changes and abort if something doesn’t look right. It also shows a diff of any parameter changes. After confirming the change, StackMaster will display the log of stack events until CloudFormation has finished applying the change.

StackMaster Apply Demo

Easy ways to connect stacks:

StackMaster provides a number of helper functions to deal with parameters. One allows you to easily take the output from one stack and use it as the input to another, without having to hardcode it. We call these helpers parameter resolvers.

Make it easy to keep secrets secret.

Another parameter resolver transparently decrypts encrypted parameters before supplying them to CloudFormation, meaning you don’t need to worry about plain-text secrets.

Make your parameters easier to understand by using names instead of IDs.

Another set of parameter resolvers StackMaster offers allow you to refer to Amazon Simple Notification Service (SNS) topics and security groups by descriptive names, instead of obscure and hard to maintain ID numbers.

Here’s an example parameter file using those features:

1
2
3
4
5
6
7
8
9
InstanceType: t2.micro
VpcId:
  stack_output: my-vpc/VpcId
DbPassword:
  secret: db_password
ssh_sg:
  security_group: SSHSecurityGroup
notification_topic:
  sns_topic: PagerDuty

Make it easy to customise stacks in different environments

StackMaster will load and merge parameters for a given stack from multiple YAML files to allow for region- or environment-specific overrides. You can, for example, set defaults in one YAML file and then use an environment specific YAML file to tailor as required. We use this to do things like use a smaller instance type in our staging environment.

Apply descriptive labels to regions

Think in terms of environments instead of region names. StackMaster allows you to operate on your staging stack, rather than on your ap-southeast-2 stack, reducing the chance of applying changes where they are not desired.

For more details and examples check out our StackMaster repository on GitHub.

How Envato defined the expectations of our developers

The Envato development team has always had a strong sense of what we stand for, how we work together and what we expect of each other … at least that is what many of us thought. Around 9 months ago our company participated in the Great Places to Work survey, which gauges how our employees feel about Envato as a place to work. Each department received a breakdown of their feedback, and whilst much of our feedback was great, one statement was a clear outlier “Management makes its expectations clear”. This was a trigger to question our assumptions about those expectations. This post tells the story of that journey.

The Response

Step 1 - Review What We Have

We reviewed where we stated expectations, how consistent and available that information is, and how we applied that information. The conclusion was it is patchy and not consistent. Our position description alluded to some expectations, our annual review question didn’t at all, and were broad and generic across the whole company, and goals that line managers set for developers were unique to each developers and did not relate to a common set of expectations.

Step 2 - Organise a Working Group

Envato has around 60 developers right now, and about 45 when we started this work, so consulting everyone as a large group was not going to be productive. We got together a working group of 7 developers that was a cross section of disciplines and seniority.

Step 3 - Come Up with a Framework / Classification

We reviewed all the information we had and came up with a basic classification system of expectations:

  • Tech Competency
  • Learning and Teaching
  • Collaboration and Ownership
  • Envato Values

Step 4 - A Starting Set of Values / Expectation

Next the working group had a go at coming up with a set of these statements, as a template for more broader consultation. We came up with a way of stating them which was a first person assertion, so that a developer could “ask themselves” if they satisfy this expectation, not just so that a manager could “judge” the developer … e.g. “I define the problems I am solving before the solutions”

An Example Developer Expectations Trello Card

We came up with 13 statements in this first draft, although we chose not to be exhaustive, but more to provide starting points for the rest of the team.

Step 5 - Get Ideas all Team Members

The entire dev team of 35 staff members was split up into groups and and assigned to one working group member. There were 9 separate sessions ran.

They ran a workshop to come up with a unique set of statements themselves. Ideas were collected in individual trello boards, and categorised either with the existing categories or new categories.

There were 220 statements, around 20 from each group, generated in these sessions.

Step 6 - Merging all Input to One Master List

The working group got back together to attempt to synthesise all this feedback. We created a new “Master” board with 5 major lists. The working group moved cards from their workshop boards into the Master board. The lists we came up with, with number of cards in each list was:

  • Technical Competence - 86
  • Collaboration - 57
  • Learning and Teaching - 42
  • Envato Values - 19
  • What Makes and Awesome Team Mate - 8
  • Miscellaneous - 7

With everyone’s separate lists on the one board it looked like this! All Developer Expectations Trello Cards

Step 7 - Consolidating Ideas

The working group split into 3 groups to consolidate one list each. The outcome of the consolidation was to represent common themes in the input. For example the Collaboration list had 57 cards and was consolidated to 10 major themes, covering about 50 of the cards, with 7 being marked for review.

Consolidating Expectations

We linked these consolidated cards back to the original card so we could trace individual input through to final statements.

Linking Source Statements

Consolidating all our ideas revealed some outliers that were not common across the working groups. We wanted to reduce the final set of expectations to a workable number, so outliers were cut.

Step 8 - Finalise the List

After much re-wording we came up with our final set of lists and cards that we considered a small enough but not too large list of expectations.

Final Consolidated Expectations

Step 9 - Convert to a Github Repo and Request Reviews

Once we were happy with our trello cards, it was time to publish these expectations and open them up for comment and review. We decided to use the development process that serves us well for code, which is github hosted repositories and pull requests. After presenting our content to the entire team at a ‘code party’ we found team members started to start conversations via Pull Requests.

Expectation Pull Requests

An Expectation Pull Request

And finally, publishing our living set of developer expectations to the public. You can find them here at Developer Expectation.

How to organise i18n without losing your translation_not_found

I’ve written before about Working with Locales and Time Zones in Rails, but I often feel the i18n library (short for internationalisation) is underused (appreciated?). Perhaps it is even avoided because of the perception it is more effort to develop with and harder to maintain.

This article will, I hope, open your mind to the idea that you will be better off using i18n in your application (even for a single language) and that it can be maintainable with some simple organisational pointers.

Envato Market Structure Styleguide

Today we have released the Envato Market ‘Structure Styleguide’ to the public.

https://market.styleguide.envato.com

A Structure Styleguide is a special breed of living styleguide, designed to document an application’s UI logic and all of the possible permutations of the UI.

The goal is to have a complete and shared understanding of how an application behaves in any situation, even in the most uncommon edge cases, without having to laboriously recreate each scenario by hand.

Your puppet code base makes you fear the apocalypse

Let me paint you a picture. At some point in time someone said ‘hey, wouldn’t it be great if we could manage our servers with that new puppet thing’. ‘Great’, said everyone, ‘Let’s do that and the way we have always done it.’. And that, my friends, is how you end up where we are.

Reading our puppet code base reads much like a bad folding story. Everyone had a plot line and tried to weave into a flowing narrative. And like all badly written stories it has many dead ends, twists, continuity issues and obscure meanings. You get that from many different authors - all code bases face the same problem.

Creating Form Objects with ActiveModel and Virtus

You might have heard of “fat models” referring to (mostly) ActiveRecord models turning into huge classes responsible for everything from user authentication to accepting attributes for entirely different models. This violates one of my favourite rules of [SOLID][solid], the single responsibility principle.

One of the patterns you can use to help balance your SOLID karma is with form objects. We use form objects extensively to back our forms and encapsulate the data submitted by a user.

It’s often tempting to reach for the ever ready ActiveRecord model when building forms in Rails because it Just Works™ with the Rails Form Helpers. I’m going to show you how to have your cake and eat it by backing your forms with objects.

Push Button Deployments

Envato is becoming a large company, with several teams working on many different products spread across various technology stacks.

Each of our teams are responsible for their own deployment process, which means each has ended up with its own way to deploy changes. It’s complicated to grant a new developer access to all the tools and services they need to deploy a single project, let alone multiple projects.

A Better Way to do Rails with Aldous

About a year ago the Tuts team encountered an issue with our Rails codebases becoming more unwieldy as they got bigger. Our development pace was slowing down and new features and fixes would cause regressions in unexpected parts of the system. This is a problem that many of us have encountered before, and we often treat it as part and parcel of doing business. This time, however, we decided to see if we could find ways to improve the situation.