COMMON FAILURES IN PERFORMANCE TESTING AND THEIR CAUSES

As Defined by the ASTQB/GTB:

Slow response under all load levels

In some cases, response time is unacceptable regardless of load. This may be caused by underlying performance issues, including, but not limited to, bad database design or implementation, network latency, and other background loads. These problems can be detected during functional and usability testing, not just performance testing, so test analysts should know to look for them and report them.

Slow response under moderate-to-heavy load levels

In some cases, response degrades unacceptably with moderate-to-heavy load, even when such loads are entirely within normal, expected, allowed ranges. Underlying defects include saturation of one or more resources and varying background loads.

Degraded response over time

In some cases, response degrades gradually or severely over time. Underlying defects include memory leaks, disk fragmentation, increasing network load over time, and database growth.

Inadequate or graceless error handling under heavy or over-limit load

In some cases, response time is acceptable but error handling degrades at high and beyond-limit load levels. Underlying defects include insufficient resource pools, undersized queues and stacks, and too rapid time-out settings. Specific examples of the general types of failures listed above include:
  • A web-based application that provides information about a company’s services does not respond to user requests within seven seconds. The user goes to another company that provides similar services.
  • A system crashes or is unable to respond to user inputs when subjected to a sudden large number of user requests (e.g., ticket sales for a major sporting event). The capacity of the system to handle this number of users is inadequate.
  • System response is significantly degraded when users submit requests for large amounts of data (e.g., a large and important report is posted on a website for download). The capacity of the system to handle the generated data volumes is insufficient.
  • Batch processing is unable to complete before online processing is needed. The execution time of the batch processes is insufficient for the time period allowed.
  • A real-time system runs out of RAM when parallel processes generate large demands for dynamic memory which cannot be released in time. The RAM is not dimensioned adequately, or requests for RAM are not adequately prioritized.
  • A real-time system component A which supplies inputs to real-time system component B is unable to calculate updates at the required rate. The overall system fails to respond in time and may fail. Code modules in component A must be evaluated and modified (“performance profiling”) to ensure that the required update rates can be achieved.