The selection of a performance and load testing tool is not one that you can easily make. It should be chosen based on a number of factors including application type, number of users, organisational preferences, timelines, technical expertise and the IT landscape (which includes existing and future technologies such as mobile, cloud, IoT and big data).
There are commercial load testing offerings that offer both, SaaS and on premise solutions. In fact, several SaaS-only solutions exist to generate user load. From an open source perspective, the Apache JMeter load testing tool has long been considered the go-to tool for performance testing. There are other less popular open source tools available on the market such as Gatling and Grinder.
Why Performance Test with JMeter
JMeter is a pure Java-based desktop application for performance testing and evaluating overall system performance under different loads. It works with many test category types such as load, performance, functional, regression and more. With so much in favour of it, let us look at the benefits and limitations of JMeter.
Key JMeter Advantages
One of the key advantage is that JMeter is open source and free. Being Java-based and platform independent, it can easily be integrated with a variety of plugins.
Server compatibility
JMeter is capable of load and performance testing across many different server/protocol types. This includes HTTP and HTTPS, web services, FTP, database (via JDBC), LDAP, message-oriented middleware (MOM) via JMS and MongoDB (NoSQL).
Extensible architecture
The open source community is very active on development around JMeter plugins. In fact, additional capabilities exist to extend its reporting, server resource monitoring and other feature sets. Users can write their own plugins if they want to.
Zero investment
The fact that there is no software cost involved is a boon to development teams with limited budget. This also works for management teams who prefer to invest in in-house expertise versus commercial tools.
Key JMeter Disadvantages
No vendor support
Since JMeter is open source, it lacks vendor support for application development or any of the other advantages that commercial applications bring. One ends up having to refer to forums and any available JMeter resources for help.
Unviable large scale testing
If you have a large-scale test to deliver, a farm of test servers, orchestrated by a central controller should be built. All this gets quite complicated if one uses JMeter. Since this tool requires dedicated hardware and network resources, and can cause latency, it is an unrealistic option for large scale testing.
No cloud capability
Like all on premise tools, JMeter does not offer cloud hosting and management opportunities. If the application is likely to scale significantly or have users distributed across geographies, a cloud-based service is a viable means of testing.
Excess memory consumption
JMeter can simulate heavy load and visualise the test report. This can lead to excessive memory consumption and result in the application running out of memory under heavy load.
Complex application recording
Complex applications that use dynamic content, workflows, role-based access, move data from one script to another script during runtime and JavaScript to alter requests can be difficult to test using JMeter. This is because these applications consume a significant duration of time and effort.
Poor JavaScript support
As JMeter is not a browser, it cannot run JavaScript in the web application. It has limited support for processing JavaScript or Ajax, which may affect the accuracy of the simulation.
Basic reporting capabilities
The JMeter analysis module does not have a comprehensive reporting functionality. For example, it is difficult to analyse and identify bottlenecks or critical KPI metrics, such as hit rates, requests per second and response times using common JMeter listeners during run time.
A customer framework can still be created on JMeter to enhance reporting and other capabilities. At Mastek, we have developed several such accelerators on JMeter for our customers.
In conclusion, the JMeter capabilities and limitations detailed above should serve as a guide for developers and testers who have to select a testing tool to performance test applications across different platforms and technologies.
