Performance Testing Work Load Model Creation

Performance test workload model – This document is the baseline for Performance test scenarios which needs to be executed for testing the scalability, durability and availability of the application under test. This document contains the characteristics of the Vusers which will be executed to test the system performance. These characteristics include number of Vusers required, Pace times and Think times assigned to them.

Observations about the current system behavior and future predictions of the system are all dependent on how accurately Performance test workload model is documented.

If performance model is not correct the behavior of application under test cannot be predicted with desired accuracy.

Performance test workload model contains –

1)     List of transactions/services getting tested from performance testing point of view.

2)      Vusers, Think Time and Pace time settings used to test these services for different performance scenarios like baseline, peak or endurance. 

Inputs for creating a performance test work load model –

1)    Low Level Design document

2)  High Level Design document

3)  Current Production Load metrics (If system is already live)

4)  Anticipated Production Load

5)  Test Environment details

6) Production Environment details

Inputs detailed –

Low Level Design Document and High level Design documents helps to analyze new changes which are going to be deployed in production. These documents help to single out transactions which are getting impacted with current release of application in production. 

New Transactions - Transactions which are getting impacted due to current release and are getting code changes.

Regression Transactions – Transactions which are not getting impacted due to current release and were part of earlier test suits (If the application was tested earlier).

Current Production Load metrics – If the system which is going to be tested is live in production the details of live transactions need to be pulled from there. This will help us to understand the current user behavior in production and will help us to accurately emulate same in our performance model.

Anticipated Production Load – If the whole system or some part of it is new the anticipated production volumes for same are required to correct documentation of Performance test workload model.

Test Environment details – The performance testing environment details are required for coming up with a accurate and useful performance load model. Most of the times the testing environment of the application is not scaled up to or is not scalable to production environment hence we need to adjust our volumes accordingly. Getting to know the environment details of testing environment also helps in predicting the future hardware or system capabilities of current system.

Also there are many instances where whole of the production systems cannot be emulated in testing environment such as third party systems. So these systems needs to be virtualized using different virtualization software’s like iTKO LISA which needs further analysis for there bottlenecks. Since these systems are not equally capable and just is a work around to test a complete transaction flow we many need to incorporate these details in our production models.

Production Environment details – Just as testing environment details are required as detailed above for the same exact reasons we require the production environment details too for documenting accurate performance model.

Performance model documentation process:

1)      Document the changes going into production for the current release of application. If the application is going live for the first time document the business critical processes.

2)      Identify the transactions or business flows which require performance testing. Not all the code changes impact the performance of the application example if label of a button or a hyper link text is modified it will not impact the performance of the application in any way. So these kinds of code changes can be ignored from performance testing point of view. Make a list of transactions identified to be tested. This is generally done in an excel sheet.

3)      The list we created in the step 2 above form the basis of our performance model. These listed transactions are further divide into high, medium and low priority categories based on their impact of business flow and volumes. For example a service or transaction which is related to updating of account information of a customer or underwriting a insurance product is much more critical than a transaction or service which pulls out the company address of the customer. Similarly a service or a transaction which is executed 10 times per Second in production is more eligible for testing than a service or transaction which is executed 50 or 100 times a day.

4)      For all the live transactions the production load for peak time traffic needs to be documented. For new transactions this value can be asked from the application architects or developers. Since testing environment is not scaled to production environment these load or volumes needs to be scaled down relatively. This scaling down can be accomplished using Pace times, Think times or Vusers in the performance scenarios.  The values for Pace times, Think times and Vusers are generally modified either individually or collectively to create different performance scenarios like baseline, peak and endurance tests. All this information needs to be documented in performance model.

5)      For all the services Expected response time metrics should be specified which will make the baseline for all the testing. This response time should be taken from production. For new services SLA should be provided by application architect or developer.

Performance model creation explained with example –

 For a Brand New Banking Project For HTTP requests (Also called a Greenfield project) –

a)      Manually navigate through the application. Understand the User activities on the application by walking through it. Select the most important business activities based on its execution frequency on a high load business day.

Note – Ignore the negative flows. This is not a System integration testing and negative flows are rarely tested for performance.

Divide the flow into individual transactions. Transaction is generally a request from client to server. So whenever there is client interaction with the server that instance can be called as a transaction with the transaction name being the purpose of that interaction.

Now suppose we have following three business flows for our example banking application –

Flow 1 – User Check Balance

Flow 2 – User Transfer Money to other account

Flow 3 – User Requests Check book

For simplicity we take limited transactions for these flows as below:

1

  b)      Gather the information about each individual flow. The information required mostly is how much requests/hour application will be expecting in production. Also how will users use the application i.e. which business flow will be executed and at what frequency. For example in the given example let us assume Flow 1 will be executed 60% of the time while Flow 2 and Flow 3 will be executed 30% and 10% of time in production.

       Also assume we are expecting 72000 http requests per hour in production.

 This information can be provided either by marketing team or by application architects.

  Now suppose we have following data gathered by above process –

 Expected requests per hour in production – 72000 requests per hour.

 No of requests generated per Flow can be understood by tools like fiddler or any other http                  traffic analyzer tool.

 Once we have above information we can go ahead with starting work load model.

                  

 c) Calculations and derivations for Think time, Pacing and Number of Vusers settings.We need to find answer to a simple question for finding these values –

If a single user is given 1 hour of time to execute these Flows, how many requests will                   be generated?

For answering this question some mathematical deductions are required which are as                           follows.

Total http requests generated in a flow can be obtained by summing up the requests                       generate by each individual transaction in that flow.

Total time for a flow to complete is just sum of all the think times in that flow.

Now from total time taken by a Flow to complete we can calculate how many times that                 flow can be executed per hour.

For example in case of Flow 1 –

 This is the point where answer to the question we are finding solution to can be answered                    for this Flow 1.

 If single user is given an hour of time and he is executing only this flow he will make                          102.8571429 executions of this flow which will generate 102.8571429*15 requests.

  Since answer is incomplete we will move forward. 

  For Flow 2 and Flow 3 above values are as follows –

 Now since it is assumed that Flow 1, Flow 2 and Flow 3 are executed 60%, 30% and 10%                    in  production, user will devote as much time proportionally from his allocated hour.

 So for Flow 1 user will have 0.6 hour, for Flow 2 0.3 hour and for Flow 3 0.1 Hour.

 So in 0.6 hours user will execute Flow 1  0.6*102.65 times, Flow 2 0.3*60 times and Flow                  3 0.1*65.55 times.

 So in 1 Hour single user will execute following iterations of Flow 1, Flow 2 and Flow 3.

 If we multiply Number of executions of particular Flow with the total number of requests in                that flow we can get the answer to our question as follows -

  Now just add the number of requests in all the flows above and we will get the number of                   requests generated by single user in single hour executing Flow 1, Flow 2 and Flow 3 as per               production threshold.

   Moving forward from here is relatively easier.

   Since we know the we are expected to generate 72000 requests per hour and 1 user in 1 hour                generates 1013 requests total number of users required to generate expected requests will be                72000/1013 i.e. 71 Users.

    Now we just need to distribute these users among Flow 1, Flow 2 and Flow 3 as 60%, 30% and 10 %.

2) For a Existing  Project For HTTP requests – The process of creating workload model for existing performance testing project is similar to the one for green field project except that now all the details for creating one is already handy. There may be some new code changes or hardware changes which may affect the performance of the application so we need to test those on high priority. But since these new changes may affect the performance of whole system knowingly or unknowingly we may have to retest the whole application again as regression.

3) For Middleware application which contains only web services – Sometimes we may need to test a middleware application which requires us to test all the web service calls involved. For preparing work load model for this king of application we need a different simple approach. Steps involved are as follows –

a)      Gather the list of web services which are going to be tested and expected transactions per second which these services may be required to operate upon. Generally it is not possible to test all the web services in a particular application since list is huge for a enterprise application and also it is not possible to test some web services in standalone fashion.  Generally the approach is to include web services which can be tested as standalone and which are responsible for 90 or 95% of production volumes.

Suppose we have list of 15 web services with expected TPS as detailed below –

      Calculate the total TPS and 95% of TPS (If you are targeting 95% of production volume in performance test, change if you are targeting something else).

      Arrange the web services according to decreasing TPS. And select those web services from top of the list whose TPS sum makes 95% of the TPS calculated above. In our case total TPS is 20.4 and 95% TPS is 19.38, so our selection will include services highlighted in green in below list –

This will be the list of web services which we want to test rest can be ignored since those do not have considerable volume to be tested.

b)  Now since we have the list of services and there volumes which we need to emulate in our performance tests we will proceed to find the Think time and Number Vusers to be assigned to them for emulating correct load.

For this we can use Little’s law of queuing theory which is as follows –

Average number of customers in a system (over some time interval), is equal to their average arrival rate, multiplied by their average time in the system.

 To simplify this in performance testing perspective this statement is equal to following equation –

N = TPS * (RT + T)
N – Number of Vusers,
TPS – Transactions per Second (TPS),
RT – Response Time in seconds,
T – Think Time in seconds.

To proceed further we need the response times which we are expecting for these web services from business people. Once those are received we will document those in our workload model as below –

C) Calculations of Number of users and think time. Since web services are generally single requests think times and pacing can be used interchangeably since we create 1 action per web service in our scripts generally. If we have more than one web service in a action part of our script then pacing and think times are different entities.

Now to calculate the Number of Users and think time to be assigned to each web service we need to use Little’s law as follows –

Think time = (Number of Vusers/Transactions per Second) – Response Time
Since this is a variable equation involving four variables there are many solutions to this equation and hence many values for Vusers and Think time satisfy this equation.
To find the best solution we need to increase the value of Vusers from 1 to the value for which we get the lowest value for think time which is more than zero.
So for example for the first web service we will start increasing value for Vusers as follows –

For Vuser = 1, Think time will be = -0.75
For Vuser =2, Think time will be = -0.5
For Vuser = 3, Think time will be = -0.25
For Vuser = 4, Think time will be = 0
For Vuser = 5, Think time will be = 0.25

Hence we will document Vuser = 5, Think time = 0.25 for this web service.
Similarly for we need to calculate for other web services too.

 

This is how we can create a performance model for web services testing.