Mobile Application Performance Testing and HP Network Virtualization - Introduction

Mobile Application Performance Testing

Applications on mobile phones are installed on mobiles with different CPU, Memory, Screen sizes, Resolutions and Networks should perform well.

As in performance testing of web applications, standalone applications or database’s the  focus of mobile performance testing is to identify or anticipate the performance bottleneck in the application and identify and diagnose the reasons for such bottlenecks.

Performance Parameters for mobile performance testing are:

      1) Application response time – How fast application responds?

      2)  Reliability – Will application for a particular environment be able to handle the load? How long can application run without any issues? How any particular functionality of the application works when it is repeated again and again?

      3)  Configuration – Is this the best configuration for performance? If not which one is?

      4) Scalability – Will application be able to scale to higher load requirements if required in future?

      5) Transaction Response Time – How long it took to complete a particular business process?

      6) Bottleneck Identification – If any hardware or software environment changes what will be its impact on the application performance?

      7) Power management – Can the application use the battery better? How much power is used by the application while running for long time? How does the application behavior changes with respect to battery level?

      8) Resource Utilization – How much CPU or Memory or Network used by application? How much memory is used when the application is installed? How much memory application uses while it’s running? How much CPU is used by the application while it’s running? What is the throughput of the application? Amount of bandwidth application uses? And what is the latency in the network?

Focus areas of mobile performance testing-

Mobile performance testing can be performed from 4 different perspectives.

1)      Device perspective –

a)      Power management. Key Performance indicator (KPI) is Battery usage.

b)      CPU performance – KPI is CPU utilization percentage.

            c)       Memory – KPI’s are Cache memory, free memory and used memory. 

2) End user perspective – What we want in our device so that we justify our $’s we spent on that.

a)      Transition time

b)      Application launch time

c)       Load time

d)      Page component analysis

e)      Response time

For all these metrics the KPI is data/sec processed by the device i.e. all the indicators depend on how fast our device processes the data for the application taking into consideration the different hardware and software restrictions.

1    3) Network Perspective - 1)      Since most of the applications which are available on mobile devices use networks and there are many types of networks available for mobile devices following network features also require performance testing because these may severely hamper performance of application on a mobile device which needs a network.

a)      Connectivity strength – KPI’s are application features in low, medium and high power condition.

b)      Connection switching or handovers – How application behaves when network is changed from 3g to 2g or vice versa and other network scenarios.

c)       Calling use interference – How the application behaves when incoming voice calls happen while using.

      4) Server Perspective – Applications on mobile many a times uses a server which takes the requests from clients and processes those. We need to test this side of the application too.

a)      Backend server load – KPI’s are bandwidth, latency and transaction response time.

Measuring performance parameters for Mobile performance testing:

           1)      Server –

a)      Load

b)      Process time

c)       Bytes total

d)      User time

e)      Packet sent/receive

2) Network –

a)      Packets and Bytes sent

b)      Packets and Bytes received

c)       Average delay/Latency

d)      Packet drops

e)      Bandwidth usage

                  3)      Device –

a)      CPU and Memory usage

b)      Method level profiling

c)       Web application component level

d)      Response time

e)      Page rendering time

4) Transaction-

a)      Response time

b)      Throughput

There are many tools available in market for testing Mobile applications from different perspectives and provide different level of details while doing so. Some of the common tools and their features are explained below: 

           1)      HP Load runner ( Server Perspective)

-          Transaction response time

-          Bandwidth

-          Latency

                 2)      ARO (Device Perspective)

-          Battery level

-          Signal strength

-          Network packets

3)      Anritsu Simulator (Network Perspective)

-          Application behaviour for different network conditions

-          Handovers

-          Incoming call/SMS 

4)   Perfecto Mobile (End-User Perspective)

-          Screen transition time

-          Application launch time

-          CPU consumption
-          Memory consumption

Mobile Load Testing using HP Load Runner and HP Network Virtualization –

Most important aspect of mobile performance testing is network. As we have seen the network conditions in mobile ecosystems are very diverse from LTE to 4Gs and more. Also mobile networks also have coverage and signal strength limitations which make network one of the most important factors in mobile application performance testing.

Load runner provides us some basic features for network virtualization. We can find these options in Load runner run time settings as shown below.

If we go to Replay -> Run time settings -> Network -> Speed Simulation, we will find following three options for Network Speed simulation –

a)      Use the maximum bandwidth

b)      Use a predefined bandwidth (bps)

c)       Use a custom bandwidth (bps)

Bps here means bits per second.

As we can see above although Load Runner provides us with some basic network simulation tools these are not robust. Since networks have many issues which cannot be emulated using virtualizing only its bandwidth properties. Two of the issues which come to my mind right now are following –

a) Random network delay (not a consistent delay)

b) Network packet loss (data packets lost during transmission)

Hence we need more tools for correctly virtualizing network properties and make mobile performance tests more realistic. There are many tools in market to emulate network conditions but here I will discuss HP network virtualization tool.

The HP network virtualization agents are installed on the Load generator machines and network conditions are specified on them through these. Network virtualization software component called HP Network Editor is installed on the ALM/PC client machine or on controller host. HP Network Editor provides the different network simulation options for the performance tester to use in performance test scenarios. Some pre built sample networks are also there to be used as it is configurations.

HP Network Virtualization tool set contains following tools –

1)    HP Network Catcher – This tool helps to capture the real world network scenarios.This tool captures network condition from one location to another in real time.

There are tools available specifically to capture mobile network conditions. Once this tool captures the real world scenario from a mobile device like iOS or Android its captured files can be exported and imported using HP Network Editor for use in performance scenario.

For example HP Network Capture Express is a tool available on Apple store and Google Play store for iOS and Android phones respectively.

2) HP Network editor – This tool is used to modify network scenario files. Scenario files are the real world network condition files collected using HP Network Catcher ( .ntxx files).

3) HP Network Virtualization for Load Runner / Performance Center – This allows us to run load test with network virtualization enabled.

4) HP Analytics – This tool helps to understand the root cause of performance bottleneck from a single user perspective. Input to this tool is single user test output. It details load times, component download analysis, response time breakdown and errors.

5) HP Predictor – This tool help to compare application performance after real world network configurations are applied against the KPI’s. It analyzes the test results from HP Load Runner or HP Performance center tests with WAN emulation.

Following diagram show where these components are installed on a HP performance center system -

Why can’t we put our load generators on remote locations to emulate the mobile network conditions?

There are many reasons for us not to do this, some are as follows –

a) Remote load generators cannot be set up on a mobile network.

b) Remote load generators incur more cost in terms of hosting and support. Also remote load generators are difficult to manage.

c) If we need to emulate the behavior of network from a different location we need to deploy a load generator in that network location which is near to impossible for each location we want to test.

d) Limitation in volumes of users that can be tested and also the times at which tests can be executed.

For complete process of how we can performance test mobile application from network perspective using HP Network Virtualization I will create another blog and publish. Please wait for my next blog for this topic.