Techniques for mobile and responsive cross-browser testing: An Envato case study.

Not so long ago, cross-browser testing meant firing up different versions of Internet Explorer, Chrome, Safari, Firefox and (possibly) Opera on multiple operating systems. Add in the ever-growing multitude of mobile devices now available, and it can be a real challenge developing your site to deliver a consistent experience to all.

On the Envato Marketplaces we get over 16 million monthly visits, 6% of which are from mobile devices. We’re currently working on making the sites responsive, so having the right tools and cross-browser testing on a range of mobile devices is incredibly important. I’m going to take you through some of the options I’ve explored while setting up the cross-browser testing suite we use for the marketplaces, and some of the workflows and techniques that make us more efficient.

Goals and Challenges

When testing mobile and responsive sites on mobile devices there are basic factors to test for. Ask yourself:

  1. Does the site look ok?
  2. Are there any bugs?
  3. Does the user interaction feel good?
  4. Is the site optimised for mobile?

The ultimate goal is to deliver a consistent experience to users on a variety of devices and browsers. We’re using traffic data and survey results to help narrow down the devices on which we focus our mobile browser support, so that the changes we make assist the bulk of our users. Reviewing your own mobile traffic data and researching device stats usage is a great place to start deciding which devices and browsers to support.

When developing the marketplaces we run multiple local development servers which makes normal cross-browser testing complicated. Until recently we used a combination of Pow and xip.io on our development machines to serve the marketplaces to our testing devices and VMs.

Since beginning our journey to responsive enlightenment, this testing process has become even slower and more complicated. Due to the nature of the marketplaces, we want to focus on testing the same actions across a range of mobile devices and browsers. This means being signed in simultaneously on each device and simulating different scenarios.

Luckily there are some quality tools and techniques available to help alleviate the pain of developing and testing responsive websites, whether you own any testing devices or not.


Synchronized Testing

Synchronised testing is an efficient way to automatically perform the same action across a variety of browsers and devices simultaneously. Products like Ghostlab and Adobe Edge Inspect CC solve this problem and help speed up testing with the added bonus of being able to remotely inspect and debug each connected browser.

Ghostlab

Earlier this year I came across Ghostlab and was immediately attracted to the product by their promise that it “synchronizes scrolls, clicks, reloads and form input across all connected clients.”

Ghostlab

Available for Mac, Ghostlab allows you to connect any device (computer, tablet, smartphone, etc.) as long as you are on the same network. If a device has JavaScript enabled then Ghostlab can run on it.

Ghostlab includes the Weinre remote web inspector that lets you inspect the DOM and debug JavaScript on any connected device. To remotely inspect a device it’s as simple and double-clicking the device listed in the sidebar and then clicking ‘debug’.

I haven’t had a device yet that Ghostlab couldn’t remotely inspect, however, I did try to use it to inspect desktop versions of Internet Explorer with no luck.

Ghostlab has lived up to its promise and is now an invaluable tool in our cross-browser testing toolkit. Not only do we use it for testing on mobile devices, but we also use it for our regular desktop cross-browser testing as well.

Adobe Edge Inspect CC

A few years ago Adobe released a product called Adobe Shadow which was revolutionary as it was the first that easily allowed you to synchronise and remotely inspect your testing across multiple devices. Since then it has since been rebranded as Adobe Edge Inspect CC and requires a paid subscription to Adobe Creative Cloud for all the features.

“Wirelessly pair multiple iOS and Android devices to your computer. With Edge Inspect, you browse in Chrome, and all connected devices will stay in sync.”

It is still pretty awesome, however, the downside is that it is limited to Chrome on iOS and Android. It also requires installing software, a browser plugin and apps on each device.

Other mentions


Web-based Cross-browser Testing

Web-based services such as BrowserStack and Crossbrowsertesting.com offer the ability to test your site across a massive collection of mobile OS and browser combinations.

BrowserStack

We have a subscription to BrowserStack and find it really handy for when we want to test devices we don’t have physical access to. Although it’s great to have access to almost every conceivable OS and browser combination, I find it can be very sluggish and hard to test interactions. I usually resort to using it for bug fixing rather than regular website testing.

BrowserStack relies on mobile device emulators and is served with Flash, so it can’t be relied on 100% for a native experience. However, the benefit of nearly no installation makes it another great tool in our cross-browser testing toolkit.

BrowserStack Automate is a separate service which allows you to automatically run your Selenium test scripts across multiple browsers in parallel.

“Automate removes the need for manual testing of common workflows such as registrations and logins, search queries, web-based admin tasks etc”

Crossbrowsertesting.com

Crossbrowsertesting.com is another fantastic cross-browser testing tool and with over 1,000 combinations of browsers, OS, and plugins it markets itself as having “More features than any competitor.”

Other mentions

There are a few similar services that offer desktop browsers testing such as Browserling, however, currently they don’t offer mobile browser testing.


Browser Developer Tools

All modern browsers have really good built-in developer tools and in the last couple of years a few have added mobile specific features.

Chrome

The Google Chrome DevTools are my go-to inspector tools for developing the marketplaces. An awesome addition has been the ‘override’ settings, which allow Chrome to emulate a mobile device.

In DevTools > Settings > Overrides you can currently override the following settings:

  • User Agent
  • Device Metrics
  • Override Geolocation
  • Device Orientation
  • Emulate touch events
  • Emulate CSS media (i.e. handheld, tv etc)

Google Chrome Devtools

Protip:
The ‘User Agent’ override is really handy if you want to inspect a website that uses device detection to serve up the mobile site. However, since it’s just mimicking a different browser it will not replicate the associated bugs and inconsistencies.

Firefox

Firefox has a ‘Responsive Design Mode’ which allows you to change the size of the content area without having to change the window size.

  • On the Tools menu, choose Web Developer, and then Responsive Design View.
  • On the keyboard, press ctrl-shift-M on Windows or Linux, or cmd-option-M on Macintosh.

This is great as it doesn’t squash the window and the developer tools whilst testing out different viewport sizes and orientations.


Remote Debugging

As web-developers we have taken for granted the ability to right-click and ‘inspect element’ since Firebug hit the scene in 2006. Developing on mobile devices brings its obvious challenges and debugging would be at the top of the list. Fortunately there are a handful of solutions which solve this problem for us, some easier than others.

Using Ghostlab is definitely the easiest way to remotely inspect and debug almost any device and browser. Although it’s possible to use the Weinre remote web inspector on it’s own, it’s probably not worth the headache.

Other mentions


Viewport Resizers

If you want to test or show-off how your website looks and responds at different viewport sizes, there is no shortage of online tools to help.

Bookmarklets

Interactive resizers

Side-by-side comparison

Protip:
Almost all work with localhost addresses, but if you’re developing a static site then you’ll need to run a server. If you have a copy of Ghostlab then this should be no problem, however another simple way is to startup a basic server. For Mac users open Terminal and to cd into the directory that your files are located in and then run:

python -m SimpleHTTPServer

You should now be able to access your static site at http://localhost:8000/


Screenshots

If you need to rapidly test how your site renders on different devices and browsers then screenshots are a great way to automate testing.

BrowserStack

In addition to its Live Testing and Automate products, BrowserStack offers another product called BrowserStack Screenshots. Screenshots’ pricing is independent of Live Testing but does includes BrowserStack Responsive, which is another useful responsive screenshot tool.

Crossbrowsertesting.com

Crossbrowsertesting.com offers automated screenshots across all browsers, operating systems and devices as part of a regular subscription. It seems to offer a few more features than BrowserStack Screenshots and some examples can be seen here.

PhantomJS

PhantomJS is a “headless WebKit with JavaScript API”, which means that it can run web pages and capture screenshots without actually loading a browser. This is great for testing that your website renders correctly in WebKit, but it’s not ideal for detecting device and browser specific issues.

Here are a couple of good write ups on using PhantomJS to automate responsive screenshot capturing:


Simulators & Emulators

If you can’t get your hands on physical devices to test on, then a great option is to test your site on a simulator/emulator. This is what BrowserStack uses and if you are after speed and responsiveness I would use these first.

Mobile Emulators & Simulators: The Ultimate Guide

Maximiliano Firtman has put together a fantastic write up of all mobile emulators and simulators and is definitely worth bookmarking. http://www.mobilexweb.com/emulators

Other mentions

Keynote MITE “MITE is a desktop tool that lets you interactively test and verify mobile content by emulating over 2,200 devices and 12,000 device profiles”

If you have a spare $5,000 per licence then Keynote MITE Pro looks like a promising tool, however it only runs on Windows.


Open Device Labs

Open Device Labs (ODLs) are a non-profit community driven collection of devices that are open to developers to visit so that they can physically test their work on a broader range of devices.

According to opendevicelab.com there are 66 ODLs across 22 countries with 1,052 devices accessible at the time of writing.

For more information about visiting, sponsoring or contributing to a ODL, check out opendevicelab.com and lab-up.org.

Related articles

Open Device Labs: Why Should We Care? (Smashing Magazine)


Envato’s Testing Suite

After lots of experimenting my current cross-browser testing setup looks a bit like this:

  • Machine: 27inch iMac (OS X Mountain Lion)
  • Dev browser: Google Chrome and the DevTools
  • Testing browsers:
    • Mac: Latest Chrome, Safari, Firefox & occasionally Opera
    • Windows: Same as Mac but also Internet Explorer 7-10 (VMs thanks to www.modern.ie)
  • Devices:
    • iPhone 4 (iOS 5.1)
    • iPhone 5 (iOS 6.1)
    • iPad 3rd gen (iOS 6.1)
    • Nokia Lumia 800 (Windows Phone 7.8)
    • Nokia Lumia 920 (Windows Phone 8.0)
    • Samsung Galaxy S4 (Android 4.2)
    • MacBook Pro Retina
    • We are continually growing our collection
  • Synchronised testing: Ghostlab
  • Remote debugging: Ghostlab
  • Edge case browsers: BrowserStack
  • Screenshots: BrowserStack
  • Viewport resizing: Viewport Resizer

Cross-browser testing at Envato


Conclusion

There is no right or wrong way to test a mobile or responsive site. Having good tools is definitely an advantage but there are plenty of free resources available to achieve the same goal.

If you’re unable to get your hands on testing devices then using a device simulator/emulator is great for speed. Otherwise a web-based tool like BrowserStack or Crossbrowsertesting.com is fine.

Whether you’re testing on physical devices, emulator/simulators, or VMs, I highly recommend using Ghostlab to ease the pain of synchronised testing and remote inspection and debugging.

If your goal is to simply review your site at different viewport dimensions rather than as rendered by specific browsers; I recommend using a tool like Viewport Resizer.

For taking browser specific screenshots, BrowserStack Screenshots or Crossbrowsertesting.com are both good options. If you’re trying to take screenshots at a range of viewport dimensions, then a solution that uses PhantomJS is ideal.



Envato & the Responsive Design Community

One of Envato’s key values is to help people learn by sharing our knowledge and engaging with a wide community of designers and coders. I co-founded Be Responsive Melbourne and am hosting the third meetup here in the cool new offices at Envato — come along if you’re local!