blog

Personal blog
git clone git://git.laack.co/blog.git
Log | Files | Refs

basic-software-profiling-cpu-usage.md (18823B)


      1 # Basic Software Profiling: CPU Usage
      2 
      3 ## Date: 2026/10/17
      4 
      5 Software today is much slower than it should be. I believe this is, in part, because most software developers do not understand performance.
      6 
      7 An easily explainable aspect of performance is profiling, and in this post I'll discuss the basics of if. I'll be focusing on compiled programs, but the core ideas may be applied to interpreted programs too.
      8 
      9 ## Tools
     10 
     11 [Perf](https://en.wikipedia.org/wiki/Perf_%28Linux%29) is a suite of tools for program profiling on Linux. We will mainly use it to collect aggregate performance metrics and call stack information. Analogous tools exist for other Free operating systems, and many bespoke tools exist for interpreted programs.
     12 
     13 [Flamegraph](https://github.com/brendangregg/FlameGraph) is a suite of tools for generating flamegraphs, a type of graph designed for the visualization of performance data. I'll be using my [fork of flamegraph](https://git.laack.co/flamegraph) which doesn't add j\*\*\*s\*\*\*\*\* to generated SVGs, but apart from that they function identically.
     14 
     15 ## Reading a Flamegraph
     16 
     17 Look at the flamegraph and try to understand the performance characteristics of the program:
     18 
     19 <object data="/images/ex1-fg.svg" type="image/svg+xml" width="100%"></object>
     20 
     21 <details>
     22 <summary>Source code for the program that generated this flamegraph</summary>
     23 
     24 ```cpp
     25 uint32_t square(uint32_t input) {
     26     return input * input;
     27 }
     28 
     29 // helper function to show stack depths
     30 uint32_t squareHelper(uint32_t input) {
     31     uint32_t x = 100 * 100;
     32     return square(input);
     33 }
     34 
     35 uint32_t factorial(uint32_t input) {
     36     uint32_t acc = 1;
     37 
     38     while(input > 1) {
     39         acc *= input;
     40         input -= 1;
     41     }
     42     return acc;
     43 }
     44 
     45 int main() {
     46 
     47     uint32_t ub = std::sqrt(UINT32_MAX);
     48 
     49     for(uint32_t i = 0; i < ub; ++i) {
     50         for(uint32_t x = 0; x < 100000; ++x) {
     51             square(i);
     52         }
     53 
     54         for(uint32_t x = 0; x < 100000; ++x) {
     55             squareHelper(i);
     56         }
     57 
     58         for(uint32_t x = 0; x < 20000; ++x) {
     59             factorial(x % 10);
     60         }
     61     }
     62 }
     63 ```
     64 
     65 </details>
     66 
     67 The amount of time spent with the `main` function on the call stack is represented by the width of its bar. 
     68 
     69 Notice how there are multiple bars that say `square`. This is because we execute `square` at different call stack depths, and the width of each bar is proportional to the amount of time we spent with `square` on the call stack at a given stack depth.
     70 
     71 Given how the widths are calculated, we can deduce the top-down width of a bar is how much time the program spent executing a specific function. In the case of `squareHelper` and `square`, we see we spent more time executing `squareHelper` than `square`, evidenced by the fact that from a top-down view, `squareHelper` is wider. 
     72 
     73 The astute among you might've noticed `square` and `squareHelper` seem to have been executed a few times outside of the main loop. This is due to how `perf` calculates the stack depth. The specifics of this are more in the weeds than we'll discuss here, but it shouldn't meaningfully impact the high level information that can be obtained from a flamegraph.
     74 
     75 ## Perf
     76 
     77 I've been loose with my words in terms of the specific data being displayed by the flamegraph. The command I used to generate the data for the flamegraph doesn't technically track the amount of time spent with a function on the stack at a given depth, we are instead tracking the number of on-CPU samples taken with the function on the top of the call stack. This distinction is important because things like blocking I/O will not be reflected in our timings.
     78 
     79 This is the command I ran to generate the data:
     80 
     81 ```bash
     82 perf record -g ./ex1
     83 ```
     84 
     85 The `perf record` command is used to run a command and record its performance profile, which is written to the `perf.data` file. By default, `perf record` will sample at 1000Hz while the program is on-CPU. The final option, `-g`, is used to specify we are interested in having the call stack included in our data. 
     86 
     87 To view aggregate information about the data in the `perf.data` file we can run:
     88 
     89 ```bash
     90 perf report -g
     91 
     92 +  100.00%     0.00%  ex1      ex1                   [.] _start
     93 +  100.00%     0.00%  ex1      libc.so.6             [.] __libc_start_main
     94 +  100.00%     0.00%  ex1      libc.so.6             [.] call_init (inlined)
     95 +  100.00%     0.00%  ex1      libc.so.6             [.] 0x00007f6302a27781
     96 +   99.79%    39.46%  ex1      ex1                   [.] main
     97 +   34.97%    19.94%  ex1      ex1                   [.] squareHelper(unsigned int)
     98 +   29.99%    19.90%  ex1      ex1                   [.] square(unsigned int)
     99 +   21.61%    20.69%  ex1      ex1                   [.] factorial(unsigned int)
    100 ...
    101 ```
    102 
    103 If you are interested in information about which functions are calling each other, and the associated on-CPU times, you can run this command:
    104 
    105 ```bash
    106 perf report -g --stdio
    107 
    108 ...
    109 
    110 |--88.87%--main
    111 |          |
    112 |          |--19.84%--squareHelper(unsigned int)
    113 |          |          |
    114 |          |           --4.99%--square(unsigned int)
    115 |          |
    116 |          |--19.55%--factorial(unsigned int)
    117 |          |
    118 |           --10.07%--square(unsigned int)
    119 |
    120 |--5.02%--squareHelper(unsigned int)
    121 |
    122 |--4.98%--square(unsigned int)
    123 |
    124 --1.13%--factorial(unsigned int)
    125 ...
    126 ```
    127 
    128 I often find the `--stdio` command difficult to read because the report can get wide for non-trivial programs, and horizontal navigation in the terminal is unergonomic.
    129 
    130 The final command worth mentioning is:
    131 
    132 ```
    133 perf stat ./ex1
    134 
    135  Performance counter stats for './ex1':
    136 
    137                  0      context-switches:u              #      0.0 cs/sec  cs_per_second
    138                  0      cpu-migrations:u                #      0.0 migrations/sec  migrations_per_second
    139                132      page-faults:u                   #      4.1 faults/sec  page_faults_per_second
    140          32,436.48 msec task-clock:u                    #      nan CPUs  CPUs_utilized
    141            356,775      branch-misses:u                 #      0.0 %  branch_miss_rate           (88.89%)
    142     63,706,083,988      branches:u                      #   1964.0 M/sec  branch_frequency       (88.89%)
    143    134,126,037,336      cpu-cycles:u                    #      4.1 GHz  cycles_frequency         (88.89%)
    144    300,951,617,557      instructions:u                  #      2.2 instructions  insn_per_cycle  (88.89%)
    145          TopdownL1                                      #      4.4 %  tma_backend_bound
    146                                                         #     61.4 %  tma_bad_speculation        (88.89%)
    147                                                         #      0.0 %  tma_frontend_bound         (77.78%)
    148                                                         #     34.2 %  tma_retiring               (88.89%)
    149 
    150       32.440411262 seconds time elapsed
    151 
    152       32.234190000 seconds user
    153        0.002939000 seconds sys
    154 ```
    155 
    156 Which provides high level information about the program's entire execution.
    157 
    158 This is the command I run most frequently when validating optimizations that target functions making up a sizeable percentage of the program's CPU usage. If the program is uniformly expensive across many functions, `perf stat` is less useful than a full report / flamegraph.
    159 
    160 ## Case Study
    161 
    162 ### Amfora
    163 
    164 [Amfora](https://github.com/makew0rld/amfora) is a fancy terminal-based Gemini protocol browser with syntax highlighting. Gemini is a text-based protocol which primarily distributes [gemtext](https://geminiprotocol.net/docs/gemtext.gmi) documents. These documents use a similar syntax to markdown, but it's slightly simpler. I chose to profile this program because software in the Gemini ecosystem is often simple to understand, making it great for educational use cases.
    165 
    166 When I start profiling software I often find it useful to use an oracle to understand what's possible. An oracle in this context is a program that performs comparable tasks with decent performance characteristics. I chose [Foreseen](https://git.laack.co/foreseen/), a Gemini protocol browser I developed, as the oracle because it fulfils both criteria.
    167 
    168 I like to start by searching for worst case performance scenarios. Worst cases are often an amplification of an average case, so by improving worst case performance we can often improve the average case.
    169 
    170 To start our quest to find worse case scenarios for Amfora, we'll load a large text file from disk, killing the program once the first lines of text are drawn in the terminal. Since neither browser supports text streaming, they both have to load and parse the document and then send text to the terminal, making this is a fair comparison. To generate a baseline document I executed the following Python code:
    171 
    172 ```python
    173 with open("lf.txt", "w") as f:
    174     for i in range(0,200000):
    175         f.write(str(i) + "\n")
    176 ```
    177 
    178 The resulting file was ~1.3MB in size. I then ran both programs killing each program with Ctrl-C at first draw.
    179 
    180 ```
    181 perf stat amfora lf.txt
    182 
    183 Performance counter stats for 'amfora lf.txt':
    184 
    185                  0      context-switches:u               #      0.0 cs/sec  cs_per_second
    186                  0      cpu-migrations:u                 #      0.0 migrations/sec  migrations_per_second
    187              2,742      page-faults:u                    #   9306.7 faults/sec  page_faults_per_second
    188             294.63 msec task-clock:u                     #      nan CPUs  CPUs_utilized
    189          1,052,499      branch-misses:u                  #      0.2 %  branch_miss_rate           (90.02%)
    190        575,712,764      branches:u                       #   1954.1 M/sec  branch_frequency       (89.14%)
    191        955,922,330      cpu-cycles:u                     #      3.2 GHz  cycles_frequency         (87.42%)
    192      2,669,651,845      instructions:u                   #      2.8 instructions  insn_per_cycle  (88.36%)
    193          TopdownL1                                       #      6.7 %  tma_backend_bound
    194                                                          #     24.9 %  tma_bad_speculation        (90.11%)
    195                                                          #     16.4 %  tma_frontend_bound         (79.92%)
    196                                                          #     52.0 %  tma_retiring               (90.37%)
    197 
    198        0.473990805 seconds time elapsed
    199 
    200        0.272938000 seconds user
    201        0.021266000 seconds sys
    202 
    203 ```
    204 
    205 ```
    206 perf stat foreseen lf.txt
    207 
    208  Performance counter stats for 'foreseen lf.txt':
    209 
    210                  0      context-switches:u               #      0.0 cs/sec  cs_per_second
    211                  0      cpu-migrations:u                 #      0.0 migrations/sec  migrations_per_second
    212              8,936      page-faults:u                    # 104842.2 faults/sec  page_faults_per_second
    213              85.23 msec task-clock:u                     #      nan CPUs  CPUs_utilized
    214            209,148      branch-misses:u                  #      0.1 %  branch_miss_rate           (88.30%)
    215        176,970,330      branches:u                       #   2076.3 M/sec  branch_frequency       (88.29%)
    216        248,919,966      cpu-cycles:u                     #      2.9 GHz  cycles_frequency         (88.29%)
    217        736,194,375      instructions:u                   #      3.0 instructions  insn_per_cycle  (88.99%)
    218          TopdownL1                                       #      3.9 %  tma_backend_bound
    219                                                          #     29.7 %  tma_bad_speculation        (89.50%)
    220                                                          #     21.2 %  tma_frontend_bound         (77.84%)
    221                                                          #     45.2 %  tma_retiring               (88.33%)
    222 
    223        0.354029324 seconds time elapsed
    224 
    225        0.066861000 seconds user
    226        0.021186000 seconds sys
    227 ```
    228 
    229 This is very unscientific, especially given I was watching to see when to kill the programs, but this is how I normally start profiling. We can't conclude anything from these results as the timings are within margin of error.
    230 
    231 Since Amfora can highlight code let's see how that impacts performance. I pasted the Python code above at the top of the text file, encapsulated by gemtext preformatted text lines, and then renamed the file to `lf.gmi`. We then find:
    232 
    233 ```
    234 perf stat amfora lf.gmi
    235 
    236  Performance counter stats for 'amfora lf.gmi':
    237 
    238                  0      context-switches:u               #      0.0 cs/sec  cs_per_second
    239                  0      cpu-migrations:u                 #      0.0 migrations/sec  migrations_per_second
    240            900,174      page-faults:u                    #  12090.4 faults/sec  page_faults_per_second
    241          74,453.35 msec task-clock:u                     #      nan CPUs  CPUs_utilized
    242        229,042,926      branch-misses:u                  #      0.4 %  branch_miss_rate           (88.94%)
    243     62,713,218,442      branches:u                       #    842.3 M/sec  branch_frequency       (88.85%)
    244    153,193,058,619      cpu-cycles:u                     #      2.1 GHz  cycles_frequency         (88.97%)
    245    306,159,421,078      instructions:u                   #      2.0 instructions  insn_per_cycle  (88.90%)
    246          TopdownL1                                       #      0.6 %  tma_backend_bound
    247                                                          #     97.8 %  tma_bad_speculation        (88.83%)
    248                                                          #     -0.1 %  tma_frontend_bound         (77.86%)
    249                                                          #      1.7 %  tma_retiring               (88.91%)
    250 
    251       28.983517177 seconds time elapsed
    252 
    253       63.894262000 seconds user
    254        8.207151000 seconds sys
    255 ```
    256 
    257 ```
    258 perf stat foreseen lf.gmi
    259 
    260  Performance counter stats for 'foreseen lf.gmi':
    261 
    262                  0      context-switches:u               #      0.0 cs/sec  cs_per_second
    263                  0      cpu-migrations:u                 #      0.0 migrations/sec  migrations_per_second
    264              6,551      page-faults:u                    #  90396.7 faults/sec  page_faults_per_second
    265              72.47 msec task-clock:u                     #      nan CPUs  CPUs_utilized
    266            154,134      branch-misses:u                  #      0.1 %  branch_miss_rate           (88.98%)
    267        166,258,529      branches:u                       #   2294.2 M/sec  branch_frequency       (88.99%)
    268        232,523,911      cpu-cycles:u                     #      3.2 GHz  cycles_frequency         (88.96%)
    269        705,288,446      instructions:u                   #      3.0 instructions  insn_per_cycle  (88.93%)
    270          TopdownL1                                       #      7.6 %  tma_backend_bound
    271                                                          #     23.2 %  tma_bad_speculation        (88.96%)
    272                                                          #     18.8 %  tma_frontend_bound         (77.41%)
    273                                                          #     50.4 %  tma_retiring               (88.45%)
    274 
    275        0.381682961 seconds time elapsed
    276 
    277        0.058132000 seconds user
    278        0.015949000 seconds sys
    279 ```
    280 
    281 That's a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let's look at a flamegraph for this.
    282 
    283 <object data="/images/amfora-before.svg" type="image/svg+xml" width="100%"></object>
    284 
    285 So 68.34% of samples occurred while executing the GC. That's a bit surprising as there shouldn't be that many allocations occuring. We also see 25.47% of the samples occur when executing the RenderGemini function. Of this, most of it is spent on concatstring3, a function used to concatenate strings.
    286 
    287 Looking in the RenderGemini function for string concatenations we find:
    288 
    289 ```
    290 rendered += fmt.Sprintf("[%s]", config.GetColorString("preformatted_text")) +
    291     buf + fmt.Sprintf("[%s:%s:-]\r\n", config.GetColorString("regular_text"), config.GetColorString("bg"))
    292 
    293 ...
    294 
    295 rendered += buf + "\r\n"
    296 
    297 ...
    298 
    299 rendered += ren
    300 
    301 ...
    302 
    303 buf += strings.TrimSuffix(lines[i], "\r") + "\r\n"
    304 ```
    305 
    306 Yeah, this isn't great. Each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into `buf` until reaching a preformatted text switch line ("```" lines), we have to re-create the accumulator string 200,000 times to parse our file.
    307 
    308 Fortunately, this is easy to solve by using a strings.Builder which doesn't have this issue, and that's [what I did](https://github.com/makew0rld/amfora/pull/370/changes). After this change, we can perform our profiling again to find:
    309 
    310 ```
    311 perf stat amfora lf.gmi
    312 
    313  Performance counter stats for 'amfora lf.gmi':
    314 
    315                  0      context-switches:u               #      0.0 cs/sec  cs_per_second
    316                  0      cpu-migrations:u                 #      0.0 migrations/sec  migrations_per_second
    317              6,039      page-faults:u                    #   4871.9 faults/sec  page_faults_per_second
    318           1,239.55 msec task-clock:u                     #      nan CPUs  CPUs_utilized
    319          3,631,996      branch-misses:u                  #      0.1 %  branch_miss_rate           (89.02%)
    320      3,017,891,055      branches:u                       #   2434.7 M/sec  branch_frequency       (88.96%)
    321      4,654,531,574      cpu-cycles:u                     #      3.8 GHz  cycles_frequency         (88.93%)
    322     14,168,541,807      instructions:u                   #      3.1 instructions  insn_per_cycle  (88.81%)
    323          TopdownL1                                       #      2.2 %  tma_backend_bound
    324                                                          #     36.9 %  tma_bad_speculation        (88.80%)
    325                                                          #     18.5 %  tma_frontend_bound         (77.83%)
    326                                                          #     42.5 %  tma_retiring               (88.90%)
    327 
    328        1.185772969 seconds time elapsed
    329 
    330        1.193871000 seconds user
    331        0.039062000 seconds sys
    332 ```
    333 
    334 This is much better. We are now using ~33x less cycles, and it's ~25x faster than before. It's also only 3x slower than foreseen on this task.
    335 
    336 <object data="/images/amfora-after.svg" type="image/svg+xml" width="100%"></object>
    337 
    338 Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples.
    339 
    340 We could continue identifying hotspots, but now there are 4 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller impact on overall performance, so I'm going to leave this as is.
    341 
    342 ## What's Next
    343 
    344 This post only discussed on-CPU profiling for program execution. There are a few additional metrics worth profiling that I'll discuss in the future:
    345 
    346 1. Off-CPU profiling
    347 2. Memory profiling
    348 3. Build time profiling