1. 13

My company builds simple Java APIs using spring boot. We need to performance test these APIs so that they can be called some hundreds of times a second. In some cases more, some less. Some of these applications store state, so we must perform requests in a certain order for them to be realistic. We also in some cases need to configure settings like connection pooling, timeouts, etc in order to simulate the systems that will be calling our APIs.

Tools like JMeter don’t necessarily have good support for this configuration, and it’s not easy to have to manually alter your scenarios each time there’s a change in your API. Plus, we don’t really like defining scenarios in XML.

We have created an internal library for performance testing that an application can include in a test module. This library allows you to define the request order with great detail, as well as low level client settings like I mentioned.

An architect has brought up that there must be better tools for this, but we haven’t been able to find any. I have begun to think about this question a bit, but haven’t found much from searching online. Others must experience this problem, and our company has had issues doing great performance testing in the past.

So my question for you crustaceans is: what is the best practice? What do you use? Please talk about your setup in some detail as I’d like to know others perspective on this problem.

  1. 3

    I suggest to take a look at Stressgrid (disclaimer: I am a contributor). You can create elaborate behaviours with Elixir scripts, run them at scale in public clouds, and collect detailed telemetry.

    1. 1

      Not sure if this fits everything you listed but here’s the links -

      wrk - https://github.com/wg/wrk wrk2 - https://github.com/giltene/wrk2

      1. 1

        The only two things I’ve ever used to any profit are JMeter (with great difficulty as it’s not easy to configure) and using custom scripts run against a reduced size of deployment (less memory, less CPU) based on a rough statistical model of production traffic.

        I think the devil is in getting an accurate model of what you’re likely to face. If you’re in a pre-production state you’re likely to spend lots of time speeding up the wrong things.

        1. 1

          I’d thoroughly recommend https://gatling.io/ as we’re using it across both Java-based and non-Java-based APIs, and have found it pretty great.

          I know we’re not using nearly the power it affords, but it’s very good!

          You don’t need to know that much Scala too, it has a straightforward DSL before you get there

          (originally posted at https://www.jvt.me/mf2/2020/01/utfz2/ and hand syndicated)

          1. 1

            I’ve also used Gatling, and it should do what you need.

            I’m not as thrilled about it: parts of the API are a little hard to make work the way you want, but it has all the major capabilities I’ve needed.

            I also agree about Scala not being a huge obstacle if you work in Java, though when I’ve had to dig into the Gatling code to understand things, it has slowed me down a bit.

          2. 1

            If I’m not mistaken, Locust might be precisely the tool you are looking for.