Frontend Performance Testing

The purpose of any testing is to protect users from a degraded or broken experience, and poor website performance is one of the quickest ways to give your users a degraded and broken experience. Therefore, performance testing, while not a test that points out system or visual regressions, is an important part of our testing arsenal.

Performance testing measures key metrics that affect a user’s ability to use your website, including page weight, number of requests, time to first byte (TTFB), load time, and scrolling performance.

The key to performance testing is to set a proper budget and stick to it. How you set the budget and stick to it will determine how effective the tests will be in your project.

Setting a Performance Budget

Creating a performance budget means setting target values for each key metric and then continually testing those metrics before each code merge or deployment. If any of the tests fail, the new feature will need to be adjusted, or some other feature may need to be removed.

As with financial budgets, very few people are really excited about the prospect of performance budgets. To most, a budget means spending less, getting less, having less fun, and most importantly...less! Less isn’t much fun in a world where we are always being told that we deserve more. As designers, we feel that our creativity is being stifled if we can’t toss around hi-res images and full-screen video with reckless abandon. As developers, we think that we can’t do our job without a CSS framework, a couple JavaScript frameworks, and dozens of jQuery plug-ins. Less is no fun!

As a person that has been living within a financial budget for the past four years, I certainly understand what it means to not get everything I want. On the other hand, when I do make a large, budgeted purchase, I do so without a single bit of guilt or debt. In the same way, performance budgets allow us to “spend” our budget responsibly, and without regret.

Just like fiscal discipline and financial budgets, UX discipline and a performance budget can help us to achieve our ultimate goals, which include a performant website and an engaged user.

While a financial budget is typically based off one’s income, a performance budget has more to do with external factors than internal ones.

Competitive Baseline

One method of determining your performance budget is to look at your competition. While saying “at least I’m better than so-and-so” is no excuse for a poorly performing website, it does ensure that you have a competitive advantage.

Start by looking at a few of your key competitors’ homepages and other key landing pages, and then compare load times, page weight, and other key metrics with your own website. The goal here isn’t to just match their metrics. You want to make sure you are beating them by 20% or more. So if your competitor’s product listing page loads in 3 seconds, make sure that your site loads its product listing page in 2.4 seconds or less. This 20% advantage over your competitor is the difference required for a user to recognize the difference between the two tasks.

This is not something you do just once, but something that needs to be monitored regularly. You can be assured that your competitors are looking for ways to improve and optimize their own sites. And if they had been looking at your site to determine their budgets, you’ve now pushed them to reduce their budgets as well!

Averaged Baseline

Regardless of your competition, it is always important to compare your performance baselines to industry averages and general best practices. There is no reason to settle for mediocre just because your competition is throwing off the curve.

HTTPArchive is a great service that measures and records the average value of various website metrics across almost half a million websites. As of April 2015, here are a few values of note:

Page weight: 2,061 KB Total requests: 99 Cacheable resources: 46% Therefore, if you want your website to feel faster than most websites, you might consider setting a goal of having a 1,648 KB website, that is served with 79 requests, of which 44 are cacheable. This will put you 20% ahead of the average website.

So now that we know a few methods for setting our budget, what are the budget items we need to consider when setting our tests up?

Raw Metrics

The most basic test of website performance is to look at the assets that are required to render it. How heavy are those assets, and how many of them are there?

Page Weight

Websites are getting fatter! Between April 2014 and April of 2015, the average website grew from 1,762 to 2,061 kilobytes, a 17% increase year over year. Reaching back to April of 2011, the average page was a skimpy 769 KB!

While page weight is not the only factor affecting the load time of your website, it certainly plays a large part. Page weight also has another side effect as we remember that more and more people are accessing our sites on mobile devices, and they are paying to download those bytes. The heavier your page, the more you are going to be costing your customers, especially in developing nations. Consider checking out What Does My Site Cost? to see what that new carousel is costing your mobile customers in Germany.

When looking to reduce the weight of your pages, there are a few obvious places to start:

Images make up 61% of an average website’s page weight. Optimize your PNG files. Consider reducing the quality of some JPEG files. Take advantage of the new responsive image tag and srcset attribute to download appropriately sized images. Set a budget and don’t add image weight without removing another image. Too many custom fonts will quickly weigh your page down. Set a font budget and consider not adding that second or third font. Consider necessary font weights, as each font weight adds kilobyte weight to the font file. While icon fonts are great, be mindful of the file size, as they can grow large quite quickly. Split the font up if one set is used for one section of the website, and another set for others. Also consider using inline SVGs instead, as you’ll gain many of the benefits of icon fonts while only needing to load the required SVGs. JS frameworks, jQuery plug-ins, and CSS frameworks often add a great deal of weight for little reward. Many sites are moving away from jQuery, as vanilla JS is sufficient for their needs, especially if targeting modern browsers. jQuery plug-ins, while they might offer some “wiz-bang” functionality, can often add significant weight to your site. Consider if the same thing could be done with CSS for modern browsers with reasonable fallbacks for older ones. Large JS frameworks like Angular or Ember might accomplish what you need done, but might come with more weight than required to get the job done. If all you need from Angular is the view layer, you might be better off using React or even Mustache. CSS frameworks are often a kitchen sink. They include every little imaginable style you could ever possibly need. While this might be great for prototyping, starting your website with hundreds of kilobytes of CSS and JS is putting yourself in quite a hole before you even write a line of code. Take advantage of minification and compression. JavaScript can be programmatically minified during your build process, and your servers can be set up to gzip files before sending them to the browser. These are both vital steps to reducing page weight.

Number of HTTP Requests

The browser is required to perform an HTTP request for every single file needed to fully render a page. Because each browser has a limited number of concurrent HTTP requests it can make, a large number of individual files means that the browser has to make numerous round trips to the server. The effect of these rounds trips is compounded on slower networks, so limiting the number of round trips needed to gather the required files will pay off greatly.

You can reduce the number of round trips in a few ways:

Reduce the number of HTTP requests. Instead of serving up dozens of individual CSS and JavaScript files, concatenate them into single files. Combine individual image files into a single image map or icon font. You’ll find many great tools to do this for you automatically (Compass, Grunt/Gulp plug-ins). Lazy-load assets not required for initial page load. This could be JavaScript that isn’t needed until the user interacts with the page, or images that are far below the initial load window. Increase the number of assets retrieved per round trip. Splitting up your assets across different servers (or CDNs) will allow the browser to pull down more assets per round trip, as the limitations on concurrent connections is per server.

Timing Metrics

Regardless of the number and size of your site’s assets, there are a number of other timing metrics that impact a user’s perceptions of your site’s performance.

Time to first byte (TTFB) Time to first byte is the number of milliseconds between the browser requesting the web page, and the first byte being received by the browser. It is a measurement of the paths between the browser and the server, including DNS lookup, initial connection, and data receipt. This value is not the best judge of a website’s performance, but it is a valuable number to keep an eye on. Time to start render A more useful time measurement is the “time to start render.” This measurement is the time at which the user starts to see content on the page. This means that any blocking files have been loaded and the browser is able to start drawing out the DOM. You can improve this number by deferring blocking JS/CSS, putting critical CSS inline in the page head, replacing image assets with data URIs, and lazy-loading any blocking content that it loads after the document has completely rendered. Time to document complete Once all of the initially requested assets have been loaded, the document is considered to be “complete.” Time to document complete doesn’t include assets pulled in by JavaScript, so lazy-loading assets won’t increase this metric

Hybrid Metrics

Hybrid metrics don’t measure a discrete value; they are scores based on numbers performance indicators.

PageSpeed Score

PageSpeed is a website tool and Chrome extension made by Google that analyzes the performance and usability of a website, providing a score out of 100 and explaining ways that the user can improve that score. Tests include:

Presence of render-blocking JavaScript or CSS Landing page redirects Image optimization File minification Server response time Server compression Browser caching Tap target size Viewport properly configured Legible font sizes

Speed Index

As stated on the project page, the speed index is the average time at which visible parts of the page are displayed. It is expressed in milliseconds and dependent on size of the view port.

This hybrid timing metric provides a score that takes into account many of the metrics just discussed, and combines them with a measurement of what the user is actually able to see of your site as it loads. Speed index is one of the best measurements of actual end-user experience.

Setting Up Performance Tests

Now that we know what types of metrics we can test and how to set performance budgets, let’s take a quick look at a few methods for automating the testing process. Whether you are testing a single website or dozens of them, no one wants to perform these measurements manually.

Grunt PageSpeed

The first tool we’ll look at for automating this workflow is Grunt PageSpeed. As the name implies, this is a Grunt plug-in that allows us to run Google’s PageSpeed test on our website. So rather than plugging your URL into the test page or using a Chrome extension, you can run this Grunt task before every merge request or continuous integration build.

To set up Grunt PageSpeed, start with the standard commands to install and require our plug-in:

$ npm install grunt-pagespeed --save-dev

// Added to Gruntfile.js
grunt.loadNpmTasks('grunt-pagespeed');

// Added to grunt.initConfig inside of Gruntfile.js
 pagespeed: {
  options: {
    nokey: true,
    url: "http://redhat.com"
  },
  desktop: {
    options: {
      paths: ["/en", "/en/services"],
      locale: "en_US",
      strategy: "desktop",
      threshold: 80
    }
  },
  mobile: {
    options: {
      paths: ["/en", "/en/services"],
      locale: "en_US",
      strategy: "mobile",
      threshold: 80
    }
  }
}

This code will allow us to automatically run both desktop and mobile tests on an array of pages inside of our base URL (in this case, http://www.redhat.com/). As long as our score comes back over 80, the tests will pass. If it is below 80, we’ll get a failing test, which signifies that changes need to be made to hit our threshold again.

Grunt Perfbudget

Another great Grunt tool is Grunt Perfbudget. This Grunt plug-in taps into Marcel Duran’s WebPageTest API, allowing us to programmatically pull results from WebPageTest and compare them with our set budgets. If you haven’t used WebPageTest yet, you’ll be in for a treat. It is able to test numerous metrics for your site while simulating different types of connections and locations around the globe. I won’t get into everything the site can do, but after five minutes of viewing the results for your own site, I’m confident you’ll love the wide array of information it provides.

So let’s see what this looks like set up in Grunt:

npm install grunt-perfbudget --save-dev

// Added to Gruntfile.js
grunt.loadNpmTasks("grunt-perfbudget");

perfbudget: {
  default: {
    options: {
      url: 'http://www.redhat.com/en',
      key: 'SEE_NOTE_ABOVE',
      budget: {
        visualComplete: '4000',
        SpeedIndex: '1500'
      }
    }
  }
}

This setup allows us to automatically run the Red Hat homepage through the entire WebPageTest suite of tests, and check the returned values against the budgets I have set. In this case, I have set my Visually Complete timing metric to 4,000 milliseconds and the Speed Index to 1,500. If either of those tests comes back above our budget, we get a big error message telling us to revisit the most recent code push and see what we did to break our budget.

Conclusion

With some proper automated testing, and a competitive budget in place, you’ll be in a good position to continue developing features and making improvements to your website while being sure that none of the changes you push out ever break your budget.