Load Testing

Home / Load Testing

Load Testing

Page updated: June 2017

What is load testing? What is the definition of load testing?

Load testing is the practical application of load upon a system.  A load test may be generated either manually or automatically.  A load test may consist of a single test or a series of tests over time.  While there are many different types of load tests based upon the system being tested, the most common type of load test typical performed are load tests on software or hardware systems.

What is load testing of a website?

Load testing a website is the process of generating user traffic on a website.  Load tests can be accomplished through several means, including manual load testing with real users, automated load testing with scripts or bots, synthetic load testing performed by a third party, or stress testing, which pushes a load test to the limits of the system, usually through automated methods.

Is a stress test the same thing as a load test?

A stress test is simply a load test that attempts to pushes a system beyond its limits and causes the system to break in some fashion.  While stress tests can be considered a sub group of load tests, not all load tests can be considered a stress test.  A load test with the purpose of capacity planning may be run with a known maximum number of simultaneous users that the website infrastructure is supposed to be able to handle, thus not causing the website infrastructure to fail.

The definition of load testing a website is generating a specified amount of load on a website.  Typically load tests are performed in a fashion that imitates real users visiting the website.  More advanced load tests may perform a series of steps on the website such as clicking on key elements, buttons and fields and navigating through the site.

What is load testing in software?

Load testing software consists of generating user load on a software system.  As software systems can be very complex with many layers and components, software load tests may consist of many different types of load tests.  Software load tests may also run on a system or they may isolate one or more components of the software architecture.  Such components can include user interfaces, APIs, database connections or even underlying hardware such as servers, routers, firewalls, and load balancers.

Software may be built on a variety of platforms from traditional windows forms to java applets and web applications.  Load testing windows forms applications may require a very different testing infrastructure than load testing web applications.  Web applications may be testable from external environments if the application is on a public facing website while windows forms must typically be installed on one or more local computers to perform a test.  In the case of some local applications you can still perform a load test by simply bypassing the user interface and making calls directly to the underlying APIs, databases, or data access layers.  How you chose to load test software really depends upon what aspects you are concerned about at the time of the load test.

What is load testing in .NET? What is load testing in Java?

Load testing .NET applications can refer to several different types of load tests because .NET (typically C#) is used on the back end for both windows form or desktop applications and for web applications.

Load testing in Java can also refer to several different types of tests as Java can natively be run in many different environments on Java virtual machines.  Java applets can also be run from a website.

For both .NET and Java load testing you may want to use a load testing suite to setup different load tests that can be repeatedly run with different settings.  Such software suites or services often integrate with code management and automation software such as Jenkins.

What is load testing in QA? What is performance testing in software testing?

Load testing in QA means you are applying a certain number of simultaneous users to a software system during quality assurance.  Load testing for quality assurance ensures that each iteration of the software is capable of handling at least a minimum number of simultaneous users, and at most, it will identify the maximum number of simultaneous users a software system can handle.  QA load testing is often performed using software automation such as Jenkins within an environment using continuous integration.

What is the meaning of throughput in performance testing?

When running performance tests, in general, throughput refers to the amount of data over time that can be transferred between the front end of the application and the back end. Specifically, throughput in performance testing can refer to several different choke points such as network bandwidth, database I/O, simultaneous users, maximum memory constraints, and disk reads and writes.  Each of these components could theoretically become a bottleneck for the throughput of data from the client to the server and vice versa.  Monitoring the throughput of these various bottlenecks as the load applied to the software increases can identify causes of system speed degradation.

What is load testing in JMeter? What is load testing in LoadRunner?

JMeter is an open source application by Apache for testing applications.  JMeter can generate heavy load on an application to analyze performance with different types of load on applications including web applications, web services and databases queries among many other protocols.

LoadRunner is a tool for application testing by HP. LoadRunner can be used to test different layers of an application by generating messages directly to a service or by simulating user interaction with a UI by recording scripts of mouse movements and button presses and keyboard entries.  LoadRunner can be used to test a variety of applications including Microsoft .NET and Java applications. LoadRunner can also interface directly with databases and even network protocols.

What is load testing in SoapUI? What is a spike test?

SoapUI is used to perform functional tests on Application Performance Interfaces.  SoapUI can test both SOAP and RESTful APIs.  Load testing an API creates multiple connections or requests to the API in order to measure the performance under a different number of simultaneous requests to the SOAP or REST API server.

A Spike test is a specific type of load test that performs a rapidly increasing number of simultaneous requests in order to simulate large spikes in traffic on a system. Spike tests can be used to load test an API or application for bottlenecks during periods of rapid growth or high numbers of concurrent users.  Performing spike tests before a high number of simultaneous users actually hits a website or application can identify the bottlenecks that could cause the site to crash or the application performance to slow to a crawl.

Who is responsible for load testing? Who performs load testing?

Often times a QA team, DevOps, or sometimes even marketing is responsible for load testing their websites or web applications.  QA typically handles the majority of testing when it comes to software and web apps in a testing environment while DevOps needs to be sure that the software runs properly on production hardware.  Marketing is in charge of driving high numbers of users to a website or product, and therefore will be concerned with whether the website infrastructure can handle high loads from events such as product releases or sales.

Load testing is typically performed by these same groups- QA, DevOps and marketing.  Sometimes load testing can also be performed by the developers or development team to guarantee the application will scale up under heavy loads.  It may, however, be the case that the dev team simply does not have access to the resources to spin up enough machines to perform a proper load test.

Which load testing tool should I use? Which is the best load testing tool?

There are a variety of load testing tools available, and they basically break down into to categories: free and paid. The paid tools are obviously more robust and have a wider range of data and features, while the free tools are often open source and are available at no cost.

Examples of free tools are Apache JMeter and HP LoadRunner (to an extent). Examples of paid tools are LoadImpact, LoadView, and Apica.

Who is responsible for load testing? Who performs load testing?

Often times a QA team, DevOps, or sometimes even marketing is responsible for load testing their websites or web applications.  QA typically handles the majority of testing when it comes to software and web apps in a testing environment while DevOps needs to be sure that the software runs properly on production hardware.  Marketing is in charge of driving high numbers of users to a website or product, and therefore will be concerned with whether the website infrastructure can handle high loads from events such as product releases or sales.

Load testing is typically performed by these same groups- QA, DevOps and marketing.  Sometimes load testing can also be performed by the developers or development team to guarantee the application will scale up under heavy loads.  It may, however, be the case that the dev team simply does not have access to the resources to spin up enough machines to perform a proper load test.

When should I start load testing? When is load testing done?

Start load testing before you think you need it.  By the time many people start load testing, there is not enough time to make the necessary changes to help handle additional load.

If load tests are performed at major milestones in the development process with proper foresight, the web infrastructure should have much less problem scaling to handle heavy load once it is in production (assuming action is taken to optimize performance as a result of the load test).

Load testing should be performed before any major event that could potentially bring large amounts of simultaneous traffic to a website.  Ideally load tests should be performed in a production environment during off peak hours before a public release of any new code or updates to a website or web application.  This should be completed well enough ahead of the public release to allow for last minute tweaks to the system if the load test fails.

Often times, a load test is performed right before the release of a new product, a new campaign, a major event or a system update.  Since the goal of such a load test is to ensure the website infrastructure can handle expected load, this may not give you enough time to react to the results of the load test.

Why perform load testing? Whey is load testing important?

You want to perform load tests so that you are not caught off guard when real load is placed on your website.  You should know what the maximum number of simultaneous users that your servers can handle is, and you should be prepared with what to do if that number is reached.  If you are not prepared for heavy traffic, there is a good chance you will end up losing business due to your website being slow or even crashing when too many visitors are on the site at once.

Load Testing is important because it identifies points of failure in your system, and can also show you how performance of your site degrades as more simultaneous users are added.  If you know what your peak traffic should be, load testing gives you greater insight into how your web application or site will perform once that level of traffic is reached.

What is the objective of load testing?

Load testing is typically performed for three reasons:

  1. Capacity Planning

If you have a rough idea of the size and volume of traffic your website should be able to handle, you can setup tests to slowly increase until you reach that limit.  This type of load test can help you plan for expected capacity of the website.

  1. Identifying critical failure points

Often times you just want to know how many simultaneous users can hit your web application until there is a critical failure.  At this point you can troubleshoot the failure to identify the root cause and hopefully mitigate the failing component or at least flag the bottleneck for future fixes.

  1. Test whether new changes introduce scalability bugs or other unforeseen consequences

Every time you update anything in production there is a potential to introduce bugs or other unintended changes.  Load testing can help eliminate some of those potential liabilities by running a load test after every code or content update.

What is performance testing?

Performance testing typically gauges different metrics related to the performance of a website or web application.  Website performance testing can measure page load speeds, time to first byte, time to interaction and other performance metrics.  Performance testing as it relates to load testing typically records how these performance metrics are impacted by adding additional load to the system.  There are a number of variables that can affect the results of performance testing, from the underlying hardware running the system (including CPU, RAM, dick I/O), available bandwidth, database reads and writes, and complexity of the code used in the system.  After running a performance test, you may be able to identify which components impacted the results and tweak those components to improve the performance of the web site.

What are the different types of performance testing?

Types of performance testing include:

Smoke testing

Smoke testing performs low level background tests to ensure the system is capable of handling the bare minimum requirements.

Baseline speed testing

Baseline speed testing establishes standard baseline performance metrics.  These baselines are often used to compare more advanced load tests when monitoring differences in performance under load.

Scalability testing

Scalability testing typically introduces more and more users to the system over time to measure whether the system can scale up to meet user needs as load increases.

Stability testing

Stability testing may include a number of different scenarios that should always work, even after a system is changed.  If stability tests begin to fail then either the system has become unstable or the stability test needs to be adjusted to handle changes to the system.

Capacity testing

Capacity testing identifies the maximum capacity of a website such as the highest possible number of simultaneous requests and then ramps up to that number by adding more users over time.  If the system successfully responds to all requests within the required time, the system has successfully passed a capacity test.

Stress testing

A Stress test takes a capacity test a step further and continues to increase the number of simultaneous suers until the system performance degrades or completely fails.

Spike testing

A spike test sends large bursts of traffic to a website to see if the servers can handle the bursts of traffic.

Endurance testing

Endurance tests send various amounts of traffic to a web application over longer periods of time to see if any anomalies arise due to extended use of system resources.  Sometimes bugs can appear during an endurance test due to memory overflow, improper garbage collection or a number of other reasons.

Individual component testing

Individual component testing isolates individual parts of a system, such as a web service or a database call and performs various tests on that component outside of the rest of the system.  A component test may also test 1 aspect of a system such as a database search, or a database write.

When should I use performance testing?

All the time.  Performance testing should be used during the development of web applications, during quality assurance, after new releases, prior to new product releases, prior to marketing initiatives, and pretty much any time something about the system is going to change.

When is performance testing not required?

When you do not care about the quality of your user experience, when no one is going to visit your website, or when you do not care if your website is up or down.  If a user interface is constantly changing it may not be practical to setup automated performance tests because you would have to constantly change the tests themselves.

What is the performance testing process?

The performance testing process depends upon how thoroughly you wish to test your website.  It can be as simple as a one-time test to see how fast your web page loads, or it could be as intimate as an automated set of tests built into your source code management platform that runs every single time an update is checked in to the system.  For websites that experience high volumes of traffic, performance tests also consist of regular load testing that applies increasing numbers of simultaneous users to both predict capacity issues as traffic increases and to identify bottlenecks or limitations of the system under extremely heavy loads.

How can performance testing be automated?

Performance testing can be automated using a number of third party solutions.  These solutions typically allow you to build test cases and schedule the test cases to run based upon a number of factors from regularly scheduled occurrences every day to automatic instances triggered by code check-ins.   Many software development tools such as Microsoft Visual studio Team Foundation Server also contain components or add-ins that can be used to perform automated performance testing.

How do performance testing tools work?

Performance testing tools work by performing actions upon a website or application and recording the results.  Many different metrics can be recorded by a performance testing tool such as page load times, time to interact, and responsiveness to user interaction.  Depending on what part of the application performance you are measuring, different tests can be run to focus specifically on certain elements such as reading data from a database, running javascript, or loading images from a file store.

Why should we automate performance testing?

Automating performance testing frees up the tester to perform more advanced test, and spend more time analyzing results. Automated tests can be repeatedly run after every code change with minimal user interaction. Automated testing can often catch issues before end users experience a problem, because automated tests are running on a consistent basis.

What is the performance testing life cycle?

The performance testing lifecycle describes the process of arriving at one or more of a series of tests that should be performed to measure performance metrics of an application over time.

The performance testing lifecycle begins while a website or application is still in development.

Up front, you need to determine the goal of the application.  Identify your audience, and the size of the audience.  Identify the ideal average number of users that might simultaneously use the product and then identify the maximum number of simultaneous users during a heavy load.

Next you need to identify how the average user will use the application, and write out scripts for some of the common use paths of a user.  You will also want to include the most demanding possible use as one of your scenarios.

Next you need to establish benchmark performance results for each scenario by simply running and recording the results of each individual test under ideal conditions.

As soon as a prototype or first draft of the application has been built, performance testing should be performed to detect limitations in the system as early as possible.  As a product matures and is deployed into a production environment performance testing should be run to gauge average performance metrics under different scenarios such as load spikes, steady increases in traffic, and extreme endurance load (load over an extended period of time).  Performance tests should continue to be run with each change to the system to ensure there is no degradation in system performance.

As the results of each scenario are analyzed, performance testing can help evolve the infrastructure to support the most extreme load that can be expected by identifying and eliminating bottlenecks in the system.

How do I do performance testing on for mobile applications?

Performance testing on mobile applications can be performed in the same manner as performance testing for desktop or web based applications.  Typically, the performance tests are not run from actual mobile devices.  Instead they are often run in a simulation by an emulator.  If the application is simply local and does not require a network connection, then load testing may not be a very valid form of performance testing on a mobile application.  For an application that connects to a remote backend, spinning up a number of simultaneous users on a mobile application can generate load with the potential to slow the system down.  Mobile performance testing may also throw a curve ball into the performance metrics due to the type of connection the mobile user has to the internet.  If the user is in a geographic zone where high speed data is not available, that can also limit the speed of a performance test.  Some load testing systems allow you to select what type of connection to emulate during a load test.  This is done by artificially limiting the bandwidth used by the application.

How is load testing performed?

Load testing is most often performed using automated systems to generate requests to a server or simulate real users in a system.  It can be performed internally using hardware and software within your own network or it can be performed externally with 3rd party testing systems.  Load tests measure the performance and response times of a system as load on a system increases.  Load tests are most appropriate to perform on a production system to measure the real responsiveness of the system under simulated conditions.  Load tests can be scheduled or run on demand but are typically planned for a time when traffic to the site is low so that any problems caused by the heavy load will affect the least amount of actual customers.

How do I load test a desktop application?

Desktop application load testing may be slightly different than load testing for a web-based application.  If the entire desktop application resides on the users machine and does not contact a central server or database, the effects of running multiple users at once will likely not have much of an effect on the performance of the application.  If the application communicates with a server or database, then load testing may be a viable means of testing performance.  Some types of load testing might simply simulate the desktop app in order to send requests without actually running the front end GUI.  These tests are much more scalable than instantiating multiple GUIs due to the need for system resources on the client machines.

Can load testing be done manually?

Load testing can be done manually by simply having a number of real users active in a system at once, however manual load testing will likely not return as valuable data as automated load testing that can gather and aggregate all of the performance metrics for the system.

How to perform load testing of a website

First, establish what aspects of the system you are going to test.  Based upon these requirements, select a load testing platform that is capable of performing every aspect of the tests you wish to perform.  Once you have familiarized yourself with the testing platform of your choice, you can design scripts or load scenarios that accurately simulate the use cases to be tested.  Some scenarios may simulate real users in the system while others may simply generate massive amounts of simultaneous GET requests. The type of test depends upon your end goal.  If you are trying to simulate real users in the system to plan for future capacity, you might run a very different load test than someone that is looking to identify how many users a system can handle before it fails.

After establishing your load scenarios you need to figure out your target load numbers as well as how and where you are going to initiate the load from.  Some systems will generate the load right from your local machine while others may require multiple machines in order to generate large numbers of simultaneous users.  Many systems allow you to spin up multiple scenarios at once per machine and even help you spin up multiple machines in the cloud.

Once you initiate the load test you probably want to record and pay attention to the performance counters on the servers hosting the website so you can see if any of the common monitors discover bottlenecks such as CPU, RAM dick IO or bandwidth.  Some load testing systems have components built into the platform that you can install on the servers in order to capture this data and correlate it to the number of simultaneous users hitting the site and the average response times of those users.  Large increases or spikes in the response times may be a good indicator that something was running less than optimal in the system.  You can often use these indications to drill down and find the exact cause for the slowdown.

How to do a load test using Selenium & JMeter

Selenium and JMeter are two examples of software that can be used to perform load testing.  Selenium can record user actions within a browser and replay them.  Selenium scripts can also be edited using different programming languages and testing platforms.  Selenium Grid allows you to perform multiple tests at once from multiple machines.  While selenium was not designed strictly for load testing, it can be used to generate scripts that integrate directly with a number of load testing platforms.

JMeter is an open source platform by Apache designed specifically for load testing web applications.  JMeter does not work on the browser level, it simply works at the protocol level.  While it appears to be a browser from the web servers persoective, it cannot execute all actions that a browser could perform such as actually rendering the page and executing javascript.  JMeter is good for generating requests on a website from many simultaneous users during a load test.

How load testing tools work

Most load testing tools let you record or write a script that interacts with a website or application.  Those scripts are then put into test scenarios or scheduled load tests.  You then identify the scope and the quantity of users required in the load test, and the length of time to run the test.  Some tests allow you to scale up the load over time while others my simply identify the maximum number of simultaneous users in the test and spin them all up at once.

How to load test single page applications

Single page applications can be trickier to load test in terms of measuring KPIs because the page doesn’t necessarily reload each time the user performs an action.  Some load testing platforms will let you measure time between actions such as button clicks and the resultant data displayed on the screen while other systems can only measure the total time it took to complete a script.