Evaluating Cloud based performance testing solutions – BlazeMeter, Cloud Test, NeoLoad & StormRunner

When the markets are going global, being confined to a single geography can be detrimental to the business. Enterprises now assume any user with an access to internet as a potential customer. Product and Service availability in physical locations is rapidly being replaced by access to the digital interfaces.

In such a competitive world, the best way to grow the market space is through global presence. Having the global presence mandates the organizations to ensure better performance for the users from each corner of the globe.

To measure the performance of the application from each geographical location, performance testing in cloud is the solution. The performance tests can be run with the load generators being across the world utilizing cloud service providers like Amazon, Rack space etc. Alternatively, Cloud based performance testing tools can be leveraged for the same need. This blog is about various cloud based performance testing solutions.

Tool selection is a decision which is taken after considering the suitability of the features to meet the requirements. This blog mentions the prominent features of the tools and is published for general information. Readers are recommended to consult the tool vendors for in depth description of the tool features and determine the feasibility.

This blog is a collection of personal observations in a highly challenging performance testing engagements. Among the existing cloud based performance testing tools, this blog aims to list out the features as well as some of the limitations of the following tools as on the date of publication.

• BlazeMeter
• Cloud Test – SOASTA
• NeoLoad – Neotys
• StormRunner Load – HP

• BlazeMeter is a commercial, self-service load testing platform as a service that is fully compatible with Apache JMeter (Backward compatibility up to 2.3.2).
• BlazeMeter is a pure play solution and can be accessed directly from the Web
• For simple static Web pages, BlazeMeter’s “FollowMe” can be utilized. This technology mimics the browser activity performed by the end user for the target user loads
• For all the web applications, JMeter scripts can be imported to BlazeMeter and load tests with the user loads up to 1 Million users can be executed.
• BlazeMeter can simulate the specified Bandwidth and packet loss to simulate the traffic from mobile devices with varied signal strengths (Low, Medium and High) and various networks (2G, 3G, 4G)
• The tool provides Sand box tests to validate the scripts for limited number of users for limited amount of time (varies based on the plans chosen)
• The tool provides application performance monitoring with New Relic and Amazon CloudWatch
• Test results can be downloaded for offline use and custom reports can be generated

Limitations

• The user load cannot be modified during the test run
• Except for the enterprise plan, the tool does not support an endurance test that runs for more than 3 hours
• JMeter and hence Blaze Meter supports only Web, Web Services and Citrix. It cannot be used with Flex, Silverlight or Multimedia protocols
• The tool lacks Pre-test SLA Configuration capability and does not provide analytical metrics based on SLAs

Evaluation of Blazemeter:

If the performance testing team has got enough expertise using JMeter to build complex scripts and if the requirement is to run several performance tests at high user loads (100K to 1 M), this tool can achieve its optimum.

• SOASTA’s cloud testing solution, CloudTest is a cloud based load and performance testing solution for mobile and web apps (Web and Web Services).
• CloudTest provides the users to choose the Load Generators from the list of Cloud providers and instance type(s) and hence provides more flexibility on the number of Load Generators needed for the performance tests.
• CloudTest offers bandwidth and latency simulation and hence can be used with the mobile application performance testing at varied signal strengths and varied networks
• CloudTest also offers the flexibility to add or drop users during the test
• CloudTest provides widget based real time monitoring and analysis of the servers, performance issues
• The results can be used to prepare custom reports

Limitations

• CloudTest cannot be used to automate Flex, Multimedia or Silver light based applications
• CloudTest doesn’t integrate with third party tools or custom tools that monitor the servers at Method level for pin pointing the performance issues
• SLA configuration is not available in CloudTest

Evaluation of CloudTest:

If the performance team can automate the navigations using CloudTest and performance tests are to be run for higher user loads with flexibility to choose the LGs from the Cloud providers (Amazon or Rack Space etc), this tool fits perfectly.

• NeoLoad is a load testing software solution designed for web and mobile applications to simulate user activity and monitor infrastructure behaviour
• NeoLoad offers Load Generators from the cloud that can be chosen for a performance test
• The tool can be used to automate the complete suite of Web2.0 bundle that includes Web, Web Services, Flex, Silver light and Multimedia
• The user has the flexibility to choose the number of Load Generators needed in the Cloud along with the LGs from LAN.
• NeoLoad provides flexibility to simulate mobile networks (2G, 3G, 4G and Wi-Fi) with varied signal strengths
• NeoLoad also provides the flexibility to add or drop users during the test run
• NeoLoad can be integrated with third party tools like Site Scope, Introscope, App Dynamics and Dynatrace to provide deep dive analysis on the servers
• The results can be saved to local machine and can be used for generating custom reports

Limitations

• SLA configuration is not available for analysing the results

Evaluation of NeoLoad

NeoLoad is the performance testing solution for Web and Mobile applications that can also be used to run the tests from cloud by having LGs from various locations. If there is a strong performance team that can analyse and correlate the test results with the raw data, then NeoLoad would be the right solution.

• StormRunner Load is the pure play cloud solution aimed at reducing the time for performance testing efforts in the Agile world
• StormRunner Load offers protocols that help developers and QA team create performance scripts very quickly
• StormRunner Load abstracts the concept of setting up the performance test bed
• SLAs can be configured before the test to identify the transactions that do not meet the SLA
• The tool can be integrated with monitoring tools like Site Scope for Hardware level and server level monitoring
• SLA violations and anomalies detection during the test enables the end users to identify performance issues
• Real time comparison of performance results with the previous runs helps the product owners assess the performance of ongoing sprints quickly

Limitations

• Supports only Web Protocol
• Users cannot be added or deleted during the test
• Bandwidth and latency simulation cannot be done and hence cannot be used to simulate mobile networks
• Method level monitoring tools cannot be integrated

Evaluation of StormRunner Load

StormRunner Load is the new player in the Cloud based performance testing solutions for web based applications. StormRunner Load is aimed at the Agile teams that have smaller sprint cycle times. If the Dev teams follow agile methodology and want to performance test their web application without the need for a performance tester, StormRunner Load is the solution.

Conclusion:

There is no single tool out there with a one-solution-fits-all perspective. The right choice of testing tools really depend on multiple parameters including client application architecture, context and client needs. This blog post is an attempt to analyse the tools in detail and understand their core strengths and limitations to enable informed decision making before taking any of them up for your enterprise. If you would like to assess which performance testing tool would best fit the needs of your organisation, Cigniti’s Test Advisory Services team would be glad to assist.

About the Author: