1. 2
    CLI tools for capturing CPU & Memory metrics of running application? (doing HTTP loading testing) ☶ ask performance
    cliffordfajardo avatar authored by cliffordfajardo 8 months ago | 1 comment

Any recommendations for CLI tools which allow you to capture the memory and CPU utilization of a running application?

Background Context:

  • I am doing HTTP benchmarking on two different servers using a HTTP load testing tool. Both servers are written in 2 different languages. Ideally, I’d like to not write any app code to capture the system information (CPU, Memory) while HTTP load test is happening.
  • What I envision this looking like: run HTTP benchmarking tool on running server, while the other CLI tool is capturing system info (CPU / memory stats) without any modification to the code of each server

At a high level, i’m wondering if such a tool exists that has the ease of an HTTP benckmarking tool and simply spits out a summarized description of the load measured:

# this is a http benchmarking tool (but im looking for a tool for CPU & memory

➜ autocannon -c 100 -d 30 -p 10 http://localhost:3000/small-json-payload
Running 30s test @ http://localhost:3000/small-json-payload
100 connections with 10 pipelining factor


┌─────────┬────────┬────────┬────────┬────────┬───────────┬──────────┬────────┐
│ Stat    │ 2.5%   │ 50%    │ 97.5%  │ 99%    │ Avg       │ Stdev    │ Max    │
├─────────┼────────┼────────┼────────┼────────┼───────────┼──────────┼────────┤
│ Latency │ 460 ms │ 487 ms │ 667 ms │ 705 ms │ 500.59 ms │ 50.07 ms │ 781 ms │
└─────────┴────────┴────────┴────────┴────────┴───────────┴──────────┴────────┘
┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ Stat      │ 1%      │ 2.5%    │ 50%     │ 97.5%   │ Avg     │ Stdev   │ Min     │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Req/Sec   │ 1160    │ 1160    │ 2000    │ 2287    │ 1984.57 │ 219.1   │ 1160    │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Bytes/Sec │ 22.7 MB │ 22.7 MB │ 39.1 MB │ 44.7 MB │ 38.8 MB │ 4.28 MB │ 22.7 MB │
└───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 30

61k requests in 30.17s, 1.16 GB read

Or something similar to docker stats (i’m not using docker though)

  1. 1
    Ristovski avatar Ristovski 8 months ago | link

    One naive approach ala docker stats would simply be to look at htop, however this is as “eyeball”-y as looking at docker stats itself. Do note that both top and htop provide filtering based on process ID (PID).

    However, more advanced options exist.

    When it comes to profiling CPU usage, perf (https://perf.wiki.kernel.org/index.php/Main_Page) is probably the best you can get. It allows drilling into stuff like cache accesses, branch misses etc.

    A great resource is Brendan Greggs website: https://www.brendangregg.com/perf.html, which includes tons of examples as well.

    For profiling memory, I would say that the best tool that’s still pretty user friendly would be https://apps.kde.org/heaptrack, (AppImage for easy download-and-run available at https://download.kde.org/stable/heaptrack/1.4.0/), which includes a basic CPU usage graph as well, iirc. Given the binary is built with debugging symbols, this will show you exactly where the memory is being wasted (unless you are using nodejs et al, in which case narrowing it down gets a bit tricky).