The Basic Idea



Before going into the details I will start explaining the basic idea behind the approach. Let us start with how a software system works from the performance perspective. The model we are using here are queuing networks. A queuing network is a collection of resources (nodes) with queues for processes waiting to get a hold on those resources. The resources are connected to each other forming a network, including CPU, Memory, Disk and other limited resources like threads in thread pools, database connections or the network. How we use these resources affects performance and scalability. If we - for example - execute too many database statements, we will run into performance problem when load increases, as more requests have to wait for a database connection to become available. Tuning approaches like wait-based-tuning rely on these increasing wait times to identify performance problems.

If we run short of resources, we try to increase the amount of resources i.e. adding more CPUs or memory. This is referred to as vertical scaling if it is done within one system or horizontal scaling if is done by adding more nodes. From a scalability perspective CPU and Memory are scalable by hardware. Network and sequential shared resource access can only be scaled by changing the application behavior and architecture. Architectural changes often involve a lot of work and cost time and money. So our primary goal must be to discover architectural problems as early as possible. Based on my own experience and also other people I have talked to about 50 percent of later performance problems can already be found in development.

In addition to architectural problems the detection of performance regressions is our second major concern. Especially in large applications which are continuously improved - so all serious enterprise applications - changes get introduced by implementing a new feature which affect the performance of other features. This is then very often discovered late during final load tests. This is however not a developer fault. Very often a developer does not understand all interdependencies of a complex application. Respectively side effects are not obvious. Performance regression analysis should discover exactly these regression problems. This can happen as easy as introducing an additional field in a data transfer object, eventually resulting in additional megabytes being transferred over the wire in certain usage scenarios.

Bottom line Continuous Integration Performance Management avoids introducing architectural problems and performance regression. This process also must be automated as it has to be part of the Continuous Integration process.

Implementing Performance Management in Continuous Integration

Now as we are clear what we want to achieve we look at how to implement performance testing and management and integrate it into our existing processes.