The selection of a performance and load testing tool is not one that you can easily make. It should be chosen based on a number of factors including application type, number of users, organisational preferences, timelines, technical expertise and the IT landscape (which includes existing and future technologies such as mobile, cloud, IoT and big data).
There are commercial load testing offerings that offer both, SaaS and on premise solutions. In fact, several SaaS-only solutions exist to generate user load. From an open source perspective, the Apache JMeter load testing tool has long been considered the go-to tool for performance testing. There are other less popular open source tools available on the market such as Gatling and Grinder.
Why Performance Test with JMeter
JMeter is a pure Java-based desktop application for performance testing and evaluating overall system performance under different loads. It works with many test category types such as load, performance, functional, regression and more. With so much in favour of it, let us look at the benefits and limitations of JMeter.
Key JMeter Advantages
One of the key advantage is that JMeter is open source and free. Being Java-based and platform independent, it can easily be integrated with a variety of plugins.
JMeter is capable of load and performance testing across many different server/protocol types. This includes HTTP and HTTPS, web services, FTP, database (via JDBC), LDAP, message-oriented middleware (MOM) via JMS and MongoDB (NoSQL).
The open source community is very active on development around JMeter plugins. In fact, additional capabilities exist to extend its reporting, server resource monitoring and other feature sets. Users can write their own plugins if they want to.
The fact that there is no software cost involved is a boon to development teams with limited budget. This also works for management teams who prefer to invest in in-house expertise versus commercial tools.
Key JMeter Disadvantages
No vendor support
Since JMeter is open source, it lacks vendor support for application development or any of the other advantages that commercial applications bring. One ends up having to refer to forums and any available JMeter resources for help.
Unviable large scale testing
If you have a large-scale test to deliver, a farm of test servers, orchestrated by a central controller should be built. All this gets quite complicated if one uses JMeter. Since this tool requires dedicated hardware and network resources, and can cause latency, it is an unrealistic option for large scale testing.
No cloud capability
Like all on premise tools, JMeter does not offer cloud hosting and management opportunities. If the application is likely to scale significantly or have users distributed across geographies, a cloud-based service is a viable means of testing.
Excess memory consumption
JMeter can simulate heavy load and visualise the test report. This can lead to excessive memory consumption and result in the application running out of memory under heavy load.
Complex application recording
Basic reporting capabilities
The JMeter analysis module does not have a comprehensive reporting functionality. For example, it is difficult to analyse and identify bottlenecks or critical KPI metrics, such as hit rates, requests per second and response times using common JMeter listeners during run time.
A customer framework can still be created on JMeter to enhance reporting and other capabilities. At Mastek, we have developed several such accelerators on JMeter for our customers.
In conclusion, the JMeter capabilities and limitations detailed above should serve as a guide for developers and testers who have to select a testing tool to performance test applications across different platforms and technologies.