blog

Personal blog
git clone git://git.laack.co/blog.git
Log | Files | Refs

commit 496c6674dbeb0854ff06423b3889013d085744b2
parent 6f07e7f6ba4402f7622dd8d9d589259c1c1eb352
Author: Andrew Laack <andrew@laack.co>
Date:   Sat, 17 Oct 2026 15:52:52 -0500

Finalized

Co-authored-by: Vim

Diffstat:
Aposts/entries/basic-software-profiling-cpu-usage.md | 348+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Aposts/gemini/basic-software-profiling-cpu-usage.md | 358+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mposts/gemini/feed.xml | 278++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
Mposts/gemini/index.gmi | 1+
Aposts/site/basic-software-profiling-cpu-usage.html | 375+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mposts/site/feed.xml | 278++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
Mposts/site/index.html | 1+
Dposts/wip/basic-software-profiling-cpu-usage.md | 346-------------------------------------------------------------------------------
8 files changed, 1637 insertions(+), 348 deletions(-)

diff --git a/posts/entries/basic-software-profiling-cpu-usage.md b/posts/entries/basic-software-profiling-cpu-usage.md @@ -0,0 +1,348 @@ +# Basic Software Profiling: CPU Usage + +## Date: 2026/10/17 + +Software today is much slower than it should be. I believe this is, in part, because most software developers do not understand performance. + +An easily explainable aspect of performance is profiling, and in this post I'll discuss the basics of if. I'll be focusing on compiled programs, but the core ideas may be applied to interpreted programs too. + +## Tools + +[Perf](https://en.wikipedia.org/wiki/Perf_%28Linux%29) is a suite of tools for program profiling on Linux. We will mainly use it to collect aggregate performance metrics and call stack information. Analogous tools exist for other Free operating systems, and many bespoke tools exist for interpreted programs. + +[Flamegraph](https://github.com/brendangregg/FlameGraph) is a suite of tools for generating flamegraphs, a type of graph designed for the visualization of performance data. I'll be using my [fork of flamegraph](https://git.laack.co/flamegraph) which doesn't add j\*\*\*s\*\*\*\*\* to generated SVGs, but apart from that they function identically. + +## Reading a Flamegraph + +Look at the flamegraph and try to understand the performance characteristics of the program: + +<object data="/images/ex1-fg.svg" type="image/svg+xml" width="100%"></object> + +<details> +<summary>Source code for the program that generated this flamegraph</summary> + +```cpp +uint32_t square(uint32_t input) { + return input * input; +} + +// helper function to show stack depths +uint32_t squareHelper(uint32_t input) { + uint32_t x = 100 * 100; + return square(input); +} + +uint32_t factorial(uint32_t input) { + uint32_t acc = 1; + + while(input > 1) { + acc *= input; + input -= 1; + } + return acc; +} + +int main() { + + uint32_t ub = std::sqrt(UINT32_MAX); + + for(uint32_t i = 0; i < ub; ++i) { + for(uint32_t x = 0; x < 100000; ++x) { + square(i); + } + + for(uint32_t x = 0; x < 100000; ++x) { + squareHelper(i); + } + + for(uint32_t x = 0; x < 20000; ++x) { + factorial(x % 10); + } + } +} +``` + +</details> + +The amount of time spent with the `main` function on the call stack is represented by the width of its bar. + +Notice how there are multiple bars that say `square`. This is because we execute `square` at different call stack depths, and the width of each bar is proportional to the amount of time we spent with `square` on the call stack at a given stack depth. + +Given how the widths are calculated, we can deduce the top-down width of a bar is how much time the program spent executing a specific function. In the case of `squareHelper` and `square`, we see we spent more time executing `squareHelper` than `square`, evidenced by the fact that from a top-down view, `squareHelper` is wider. + +The astute among you might've noticed `square` and `squareHelper` seem to have been executed a few times outside of the main loop. This is due to how `perf` calculates the stack depth. The specifics of this are more in the weeds than we'll discuss here, but it shouldn't meaningfully impact the high level information that can be obtained from a flamegraph. + +## Perf + +I've been loose with my words in terms of the specific data being displayed by the flamegraph. The command I used to generate the data for the flamegraph doesn't technically track the amount of time spent with a function on the stack at a given depth, we are instead tracking the number of on-CPU samples taken with the function on the top of the call stack. This distinction is important because things like blocking I/O will not be reflected in our timings. + +This is the command I ran to generate the data: + +```bash +perf record -g ./ex1 +``` + +The `perf record` command is used to run a command and record its performance profile, which is written to the `perf.data` file. By default, `perf record` will sample at 1000Hz while the program is on-CPU. The final option, `-g`, is used to specify we are interested in having the call stack included in our data. + +To view aggregate information about the data in the `perf.data` file we can run: + +```bash +perf report -g + ++ 100.00% 0.00% ex1 ex1 [.] _start ++ 100.00% 0.00% ex1 libc.so.6 [.] __libc_start_main ++ 100.00% 0.00% ex1 libc.so.6 [.] call_init (inlined) ++ 100.00% 0.00% ex1 libc.so.6 [.] 0x00007f6302a27781 ++ 99.79% 39.46% ex1 ex1 [.] main ++ 34.97% 19.94% ex1 ex1 [.] squareHelper(unsigned int) ++ 29.99% 19.90% ex1 ex1 [.] square(unsigned int) ++ 21.61% 20.69% ex1 ex1 [.] factorial(unsigned int) +... +``` + +If you are interested in information about which functions are calling each other, and the associated on-CPU times, you can run this command: + +```bash +perf report -g --stdio + +... + +|--88.87%--main +| | +| |--19.84%--squareHelper(unsigned int) +| | | +| | --4.99%--square(unsigned int) +| | +| |--19.55%--factorial(unsigned int) +| | +| --10.07%--square(unsigned int) +| +|--5.02%--squareHelper(unsigned int) +| +|--4.98%--square(unsigned int) +| +--1.13%--factorial(unsigned int) +... +``` + +I often find the `--stdio` command difficult to read because the report can get wide for non-trivial programs, and horizontal navigation in the terminal is unergonomic. + +The final command worth mentioning is: + +``` +perf stat ./ex1 + + Performance counter stats for './ex1': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 132 page-faults:u # 4.1 faults/sec page_faults_per_second + 32,436.48 msec task-clock:u # nan CPUs CPUs_utilized + 356,775 branch-misses:u # 0.0 % branch_miss_rate (88.89%) + 63,706,083,988 branches:u # 1964.0 M/sec branch_frequency (88.89%) + 134,126,037,336 cpu-cycles:u # 4.1 GHz cycles_frequency (88.89%) + 300,951,617,557 instructions:u # 2.2 instructions insn_per_cycle (88.89%) + TopdownL1 # 4.4 % tma_backend_bound + # 61.4 % tma_bad_speculation (88.89%) + # 0.0 % tma_frontend_bound (77.78%) + # 34.2 % tma_retiring (88.89%) + + 32.440411262 seconds time elapsed + + 32.234190000 seconds user + 0.002939000 seconds sys +``` + +Which provides high level information about the program's entire execution. + +This is the command I run most frequently when validating optimizations that target functions making up a sizeable percentage of the program's CPU usage. If the program is uniformly expensive across many functions, `perf stat` is less useful than a full report / flamegraph. + +## Case Study + +### Amfora + +[Amfora](https://github.com/makew0rld/amfora) is a fancy terminal-based Gemini protocol browser with syntax highlighting. Gemini is a text-based protocol which primarily distributes [gemtext](https://geminiprotocol.net/docs/gemtext.gmi) documents. These documents use a similar syntax to markdown, but it's slightly simpler. I chose to profile this program because software in the Gemini ecosystem is often simple to understand, making it great for educational use cases. + +When I start profiling software I often find it useful to use an oracle to understand what's possible. An oracle in this context is a program that performs comparable tasks with decent performance characteristics. I chose [Foreseen](https://git.laack.co/foreseen/), a Gemini protocol browser I developed, as the oracle because it fulfils both criteria. + +I like to start by searching for worst case performance scenarios. Worst cases are often an amplification of an average case, so by improving worst case performance we can often improve the average case. + +To start our quest to find worse case scenarios for Amfora, we'll load a large text file from disk, killing the program once the first lines of text are drawn in the terminal. Since neither browser supports text streaming, they both have to load and parse the document and then send text to the terminal, making this is a fair comparison. To generate a baseline document I executed the following Python code: + +```python +with open("lf.txt", "w") as f: + for i in range(0,200000): + f.write(str(i) + "\n") +``` + +The resulting file was ~1.3MB in size. I then ran both programs killing each program with Ctrl-C at first draw. + +``` +perf stat amfora lf.txt + +Performance counter stats for 'amfora lf.txt': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 2,742 page-faults:u # 9306.7 faults/sec page_faults_per_second + 294.63 msec task-clock:u # nan CPUs CPUs_utilized + 1,052,499 branch-misses:u # 0.2 % branch_miss_rate (90.02%) + 575,712,764 branches:u # 1954.1 M/sec branch_frequency (89.14%) + 955,922,330 cpu-cycles:u # 3.2 GHz cycles_frequency (87.42%) + 2,669,651,845 instructions:u # 2.8 instructions insn_per_cycle (88.36%) + TopdownL1 # 6.7 % tma_backend_bound + # 24.9 % tma_bad_speculation (90.11%) + # 16.4 % tma_frontend_bound (79.92%) + # 52.0 % tma_retiring (90.37%) + + 0.473990805 seconds time elapsed + + 0.272938000 seconds user + 0.021266000 seconds sys + +``` + +``` +perf stat foreseen lf.txt + + Performance counter stats for 'foreseen lf.txt': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 8,936 page-faults:u # 104842.2 faults/sec page_faults_per_second + 85.23 msec task-clock:u # nan CPUs CPUs_utilized + 209,148 branch-misses:u # 0.1 % branch_miss_rate (88.30%) + 176,970,330 branches:u # 2076.3 M/sec branch_frequency (88.29%) + 248,919,966 cpu-cycles:u # 2.9 GHz cycles_frequency (88.29%) + 736,194,375 instructions:u # 3.0 instructions insn_per_cycle (88.99%) + TopdownL1 # 3.9 % tma_backend_bound + # 29.7 % tma_bad_speculation (89.50%) + # 21.2 % tma_frontend_bound (77.84%) + # 45.2 % tma_retiring (88.33%) + + 0.354029324 seconds time elapsed + + 0.066861000 seconds user + 0.021186000 seconds sys +``` + +This is very unscientific, especially given I was watching to see when to kill the programs, but this is how I normally start profiling. We can't conclude anything from these results as the timings are within margin of error. + +Since Amfora can highlight code let's see how that impacts performance. I pasted the Python code above at the top of the text file, encapsulated by gemtext preformatted text lines, and then renamed the file to `lf.gmi`. We then find: + +``` +perf stat amfora lf.gmi + + Performance counter stats for 'amfora lf.gmi': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 900,174 page-faults:u # 12090.4 faults/sec page_faults_per_second + 74,453.35 msec task-clock:u # nan CPUs CPUs_utilized + 229,042,926 branch-misses:u # 0.4 % branch_miss_rate (88.94%) + 62,713,218,442 branches:u # 842.3 M/sec branch_frequency (88.85%) + 153,193,058,619 cpu-cycles:u # 2.1 GHz cycles_frequency (88.97%) + 306,159,421,078 instructions:u # 2.0 instructions insn_per_cycle (88.90%) + TopdownL1 # 0.6 % tma_backend_bound + # 97.8 % tma_bad_speculation (88.83%) + # -0.1 % tma_frontend_bound (77.86%) + # 1.7 % tma_retiring (88.91%) + + 28.983517177 seconds time elapsed + + 63.894262000 seconds user + 8.207151000 seconds sys +``` + +``` +perf stat foreseen lf.gmi + + Performance counter stats for 'foreseen lf.gmi': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,551 page-faults:u # 90396.7 faults/sec page_faults_per_second + 72.47 msec task-clock:u # nan CPUs CPUs_utilized + 154,134 branch-misses:u # 0.1 % branch_miss_rate (88.98%) + 166,258,529 branches:u # 2294.2 M/sec branch_frequency (88.99%) + 232,523,911 cpu-cycles:u # 3.2 GHz cycles_frequency (88.96%) + 705,288,446 instructions:u # 3.0 instructions insn_per_cycle (88.93%) + TopdownL1 # 7.6 % tma_backend_bound + # 23.2 % tma_bad_speculation (88.96%) + # 18.8 % tma_frontend_bound (77.41%) + # 50.4 % tma_retiring (88.45%) + + 0.381682961 seconds time elapsed + + 0.058132000 seconds user + 0.015949000 seconds sys +``` + +That's a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let's look at a flamegraph for this. + +<object data="/images/amfora-before.svg" type="image/svg+xml" width="100%"></object> + +So 68.34% of samples occurred while executing the GC. That's a bit surprising as there shouldn't be that many allocations occuring. We also see 25.47% of the samples occur when executing the RenderGemini function. Of this, most of it is spent on concatstring3, a function used to concatenate strings. + +Looking in the RenderGemini function for string concatenations we find: + +``` +rendered += fmt.Sprintf("[%s]", config.GetColorString("preformatted_text")) + + buf + fmt.Sprintf("[%s:%s:-]\r\n", config.GetColorString("regular_text"), config.GetColorString("bg")) + +... + +rendered += buf + "\r\n" + +... + +rendered += ren + +... + +buf += strings.TrimSuffix(lines[i], "\r") + "\r\n" +``` + +Yeah, this isn't great. Each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into `buf` until reaching a preformatted text switch line ("```" lines), we have to re-create the accumulator string 200,000 times to parse our file. + +Fortunately, this is easy to solve by using a strings.Builder which doesn't have this issue, and that's [what I did](https://github.com/makew0rld/amfora/pull/370/changes). After this change, we can perform our profiling again to find: + +``` +perf stat amfora lf.gmi + + Performance counter stats for 'amfora lf.gmi': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,039 page-faults:u # 4871.9 faults/sec page_faults_per_second + 1,239.55 msec task-clock:u # nan CPUs CPUs_utilized + 3,631,996 branch-misses:u # 0.1 % branch_miss_rate (89.02%) + 3,017,891,055 branches:u # 2434.7 M/sec branch_frequency (88.96%) + 4,654,531,574 cpu-cycles:u # 3.8 GHz cycles_frequency (88.93%) + 14,168,541,807 instructions:u # 3.1 instructions insn_per_cycle (88.81%) + TopdownL1 # 2.2 % tma_backend_bound + # 36.9 % tma_bad_speculation (88.80%) + # 18.5 % tma_frontend_bound (77.83%) + # 42.5 % tma_retiring (88.90%) + + 1.185772969 seconds time elapsed + + 1.193871000 seconds user + 0.039062000 seconds sys +``` + +This is much better. We are now using ~33x less cycles, and it's ~25x faster than before. It's also only 3x slower than foreseen on this task. + +<object data="/images/amfora-after.svg" type="image/svg+xml" width="100%"></object> + +Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples. + +We could continue identifying hotspots, but now there are 4 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller impact on overall performance, so I'm going to leave this as is. + +## What's Next + +This post only discussed on-CPU profiling for program execution. There are a few additional metrics worth profiling that I'll discuss in the future: + +1. Off-CPU profiling +2. Memory profiling +3. Build time profiling diff --git a/posts/gemini/basic-software-profiling-cpu-usage.md b/posts/gemini/basic-software-profiling-cpu-usage.md @@ -0,0 +1,358 @@ +# Basic Software Profiling: CPU Usage + +## Date: 2026/10/17 + +Software today is much slower than it should be. I believe this is, in part, because most software developers do not understand performance. + +An easily explainable aspect of performance is profiling, and in this post I'll discuss the basics of if. I'll be focusing on compiled programs, but the core ideas may be applied to interpreted programs too. + +## Tools + +Perf is a suite of tools for program profiling on Linux. We will mainly use it to collect aggregate performance metrics and call stack information. Analogous tools exist for other Free operating systems, and many bespoke tools exist for interpreted programs. + +=> https://en.wikipedia.org/wiki/Perf_%28Linux%29 + +Flamegraph is a suite of tools for generating flamegraphs, a type of graph designed for the visualization of performance data. I'll be using my fork of flamegraph which doesn't add j***s***** to generated SVGs, but apart from that they function identically. + +=> https://github.com/brendangregg/FlameGraph +-> https://git.laack.co/flamegraph + + +## Reading a Flamegraph + +Look at the flamegraph and try to understand the performance characteristics of the program: + +=> https://blog.laack.co/images/ex1-fg.svg + +Source code for the program that generated this flamegraph: + +```cpp +uint32_t square(uint32_t input) { + return input * input; +} + +// helper function to show stack depths +uint32_t squareHelper(uint32_t input) { + uint32_t x = 100 * 100; + return square(input); +} + +uint32_t factorial(uint32_t input) { + uint32_t acc = 1; + + while(input > 1) { + acc *= input; + input -= 1; + } + return acc; +} + +int main() { + + uint32_t ub = std::sqrt(UINT32_MAX); + + for(uint32_t i = 0; i < ub; ++i) { + for(uint32_t x = 0; x < 100000; ++x) { + square(i); + } + + for(uint32_t x = 0; x < 100000; ++x) { + squareHelper(i); + } + + for(uint32_t x = 0; x < 20000; ++x) { + factorial(x % 10); + } + } +} +``` + +The amount of time spent with the `main` function on the call stack is represented by the width of its bar. + +Notice how there are multiple bars that say `square`. This is because we execute `square` at different call stack depths, and the width of each bar is proportional to the amount of time we spent with `square` on the call stack at a given stack depth. + +Given how the widths are calculated, we can deduce the top-down width of a bar is how much time the program spent executing a specific function. In the case of `squareHelper` and `square`, we see we spent more time executing `squareHelper` than `square`, evidenced by the fact that from a top-down view, `squareHelper` is wider. + +The astute among you might've noticed `square` and `squareHelper` seem to have been executed a few times outside of the main loop. This is due to how `perf` calculates the stack depth. The specifics of this are more in the weeds than we'll discuss here, but it shouldn't meaningfully impact the high level information that can be obtained from a flamegraph. + +## Perf + +I've been loose with my words in terms of the specific data being displayed by the flamegraph. The command I used to generate the data for the flamegraph doesn't technically track the amount of time spent with a function on the stack at a given depth, we are instead tracking the number of on-CPU samples taken with the function on the top of the call stack. This distinction is important because things like blocking I/O will not be reflected in our timings. + +This is the command I ran to generate the data: + +```bash +perf record -g ./ex1 +``` + +The `perf record` command is used to run a command and record its performance profile, which is written to the `perf.data` file. By default, `perf record` will sample at 1000Hz while the program is on-CPU. The final option, `-g`, is used to specify we are interested in having the call stack included in our data. + +To view aggregate information about the data in the `perf.data` file we can run: + +```bash +perf report -g + ++ 100.00% 0.00% ex1 ex1 [.] _start ++ 100.00% 0.00% ex1 libc.so.6 [.] __libc_start_main ++ 100.00% 0.00% ex1 libc.so.6 [.] call_init (inlined) ++ 100.00% 0.00% ex1 libc.so.6 [.] 0x00007f6302a27781 ++ 99.79% 39.46% ex1 ex1 [.] main ++ 34.97% 19.94% ex1 ex1 [.] squareHelper(unsigned int) ++ 29.99% 19.90% ex1 ex1 [.] square(unsigned int) ++ 21.61% 20.69% ex1 ex1 [.] factorial(unsigned int) +... +``` + +If you are interested in information about which functions are calling each other, and the associated on-CPU times, you can run this command: + +```bash +perf report -g --stdio + +... + +|--88.87%--main +| | +| |--19.84%--squareHelper(unsigned int) +| | | +| | --4.99%--square(unsigned int) +| | +| |--19.55%--factorial(unsigned int) +| | +| --10.07%--square(unsigned int) +| +|--5.02%--squareHelper(unsigned int) +| +|--4.98%--square(unsigned int) +| +--1.13%--factorial(unsigned int) +... +``` + +I often find the `--stdio` command difficult to read because the report can get wide for non-trivial programs, and horizontal navigation in the terminal is unergonomic. + +The final command worth mentioning is: + +``` +perf stat ./ex1 + + Performance counter stats for './ex1': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 132 page-faults:u # 4.1 faults/sec page_faults_per_second + 32,436.48 msec task-clock:u # nan CPUs CPUs_utilized + 356,775 branch-misses:u # 0.0 % branch_miss_rate (88.89%) + 63,706,083,988 branches:u # 1964.0 M/sec branch_frequency (88.89%) + 134,126,037,336 cpu-cycles:u # 4.1 GHz cycles_frequency (88.89%) + 300,951,617,557 instructions:u # 2.2 instructions insn_per_cycle (88.89%) + TopdownL1 # 4.4 % tma_backend_bound + # 61.4 % tma_bad_speculation (88.89%) + # 0.0 % tma_frontend_bound (77.78%) + # 34.2 % tma_retiring (88.89%) + + 32.440411262 seconds time elapsed + + 32.234190000 seconds user + 0.002939000 seconds sys +``` + +Which provides high level information about the program's entire execution. + +This is the command I run most frequently when validating optimizations that target functions making up a sizeable percentage of the program's CPU usage. If the program is uniformly expensive across many functions, `perf stat` is less useful than a full report / flamegraph. + +## Case Study + +### Amfora + +Amfora is a fancy terminal-based Gemini protocol browser with syntax highlighting. Gemini is a text-based protocol which primarily distributes gemtext documents. These documents use a similar syntax to markdown, but it's slightly simpler. I chose to profile this program because software in the Gemini ecosystem is often simple to understand, making it great for educational use cases. + +=> https://github.com/makew0rld/amfora +=> https://geminiprotocol.net/docs/gemtext.gmi + +When I start profiling software I often find it useful to use an oracle to understand what's possible. An oracle in this context is a program that performs comparable tasks with decent performance characteristics. I chose Foreseen, a Gemini protocol browser I developed, as the oracle because it fulfils both criteria. + +=> https://git.laack.co/foreseen/ + +I like to start by searching for worst case performance scenarios. Worst cases are often an amplification of an average case, so by improving worst case performance we can often improve the average case. + +To start our quest to find worse case scenarios for Amfora, we'll load a large text file from disk, killing the program once the first lines of text are drawn in the terminal. Since neither browser supports text streaming, they both have to load and parse the document and then send text to the terminal, making this is a fair comparison. To generate a baseline document I executed the following Python code: + +```python +with open("lf.txt", "w") as f: + for i in range(0,200000): + f.write(str(i) + "\n") +``` + +The resulting file was ~1.3MB in size. I then ran both programs killing each program with Ctrl-C at first draw. + +``` +perf stat amfora lf.txt + +Performance counter stats for 'amfora lf.txt': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 2,742 page-faults:u # 9306.7 faults/sec page_faults_per_second + 294.63 msec task-clock:u # nan CPUs CPUs_utilized + 1,052,499 branch-misses:u # 0.2 % branch_miss_rate (90.02%) + 575,712,764 branches:u # 1954.1 M/sec branch_frequency (89.14%) + 955,922,330 cpu-cycles:u # 3.2 GHz cycles_frequency (87.42%) + 2,669,651,845 instructions:u # 2.8 instructions insn_per_cycle (88.36%) + TopdownL1 # 6.7 % tma_backend_bound + # 24.9 % tma_bad_speculation (90.11%) + # 16.4 % tma_frontend_bound (79.92%) + # 52.0 % tma_retiring (90.37%) + + 0.473990805 seconds time elapsed + + 0.272938000 seconds user + 0.021266000 seconds sys + +``` + +``` +perf stat foreseen lf.txt + + Performance counter stats for 'foreseen lf.txt': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 8,936 page-faults:u # 104842.2 faults/sec page_faults_per_second + 85.23 msec task-clock:u # nan CPUs CPUs_utilized + 209,148 branch-misses:u # 0.1 % branch_miss_rate (88.30%) + 176,970,330 branches:u # 2076.3 M/sec branch_frequency (88.29%) + 248,919,966 cpu-cycles:u # 2.9 GHz cycles_frequency (88.29%) + 736,194,375 instructions:u # 3.0 instructions insn_per_cycle (88.99%) + TopdownL1 # 3.9 % tma_backend_bound + # 29.7 % tma_bad_speculation (89.50%) + # 21.2 % tma_frontend_bound (77.84%) + # 45.2 % tma_retiring (88.33%) + + 0.354029324 seconds time elapsed + + 0.066861000 seconds user + 0.021186000 seconds sys +``` + +This is very unscientific, especially given I was watching to see when to kill the programs, but this is how I normally start profiling. We can't conclude anything from these results as the timings are within margin of error. + +Since Amfora can highlight code let's see how that impacts performance. I pasted the Python code above at the top of the text file, encapsulated by gemtext preformatted text lines, and then renamed the file to `lf.gmi`. We then find: + +``` +perf stat amfora lf.gmi + + Performance counter stats for 'amfora lf.gmi': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 900,174 page-faults:u # 12090.4 faults/sec page_faults_per_second + 74,453.35 msec task-clock:u # nan CPUs CPUs_utilized + 229,042,926 branch-misses:u # 0.4 % branch_miss_rate (88.94%) + 62,713,218,442 branches:u # 842.3 M/sec branch_frequency (88.85%) + 153,193,058,619 cpu-cycles:u # 2.1 GHz cycles_frequency (88.97%) + 306,159,421,078 instructions:u # 2.0 instructions insn_per_cycle (88.90%) + TopdownL1 # 0.6 % tma_backend_bound + # 97.8 % tma_bad_speculation (88.83%) + # -0.1 % tma_frontend_bound (77.86%) + # 1.7 % tma_retiring (88.91%) + + 28.983517177 seconds time elapsed + + 63.894262000 seconds user + 8.207151000 seconds sys +``` + +``` +perf stat foreseen lf.gmi + + Performance counter stats for 'foreseen lf.gmi': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,551 page-faults:u # 90396.7 faults/sec page_faults_per_second + 72.47 msec task-clock:u # nan CPUs CPUs_utilized + 154,134 branch-misses:u # 0.1 % branch_miss_rate (88.98%) + 166,258,529 branches:u # 2294.2 M/sec branch_frequency (88.99%) + 232,523,911 cpu-cycles:u # 3.2 GHz cycles_frequency (88.96%) + 705,288,446 instructions:u # 3.0 instructions insn_per_cycle (88.93%) + TopdownL1 # 7.6 % tma_backend_bound + # 23.2 % tma_bad_speculation (88.96%) + # 18.8 % tma_frontend_bound (77.41%) + # 50.4 % tma_retiring (88.45%) + + 0.381682961 seconds time elapsed + + 0.058132000 seconds user + 0.015949000 seconds sys +``` + +That's a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let's look at a flamegraph for this. + +=> https://blog.laack.co/images/amfora-before.svg + +So 68.34% of samples occurred while executing the GC. That's a bit surprising as there shouldn't be that many allocations occuring. We also see 25.47% of the samples occur when executing the RenderGemini function. Of this, most of it is spent on concatstring3, a function used to concatenate strings. + +Looking in the RenderGemini function for string concatenations we find: + +``` +rendered += fmt.Sprintf("[%s]", config.GetColorString("preformatted_text")) + + buf + fmt.Sprintf("[%s:%s:-]\r\n", config.GetColorString("regular_text"), config.GetColorString("bg")) + +... + +rendered += buf + "\r\n" + +... + +rendered += ren + +... + +buf += strings.TrimSuffix(lines[i], "\r") + "\r\n" +``` + +Yeah, this isn't great. Each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into `buf` until reaching a preformatted text switch line ("```" lines), we have to re-create the accumulator string 200,000 times to parse our file. + +Fortunately, this is easy to solve by using a strings.Builder which doesn't have this issue, and that's what I did. After this change, we can perform our profiling again to find: + +=> https://github.com/makew0rld/amfora/pull/370/changes + +``` +perf stat amfora lf.gmi + + Performance counter stats for 'amfora lf.gmi': + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,039 page-faults:u # 4871.9 faults/sec page_faults_per_second + 1,239.55 msec task-clock:u # nan CPUs CPUs_utilized + 3,631,996 branch-misses:u # 0.1 % branch_miss_rate (89.02%) + 3,017,891,055 branches:u # 2434.7 M/sec branch_frequency (88.96%) + 4,654,531,574 cpu-cycles:u # 3.8 GHz cycles_frequency (88.93%) + 14,168,541,807 instructions:u # 3.1 instructions insn_per_cycle (88.81%) + TopdownL1 # 2.2 % tma_backend_bound + # 36.9 % tma_bad_speculation (88.80%) + # 18.5 % tma_frontend_bound (77.83%) + # 42.5 % tma_retiring (88.90%) + + 1.185772969 seconds time elapsed + + 1.193871000 seconds user + 0.039062000 seconds sys +``` + +This is much better. We are now using ~33x less cycles, and it's ~25x faster than before. It's also only 3x slower than foreseen on this task. + +=> https://blog.laack.co/images/amfora-after.svg + +Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples. + +We could continue identifying hotspots, but now there are 4 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller impact on overall performance, so I'm going to leave this as is. + +## What's Next + +This post only discussed on-CPU profiling for program execution. There are a few additional metrics worth profiling that I'll discuss in the future: + +1. Off-CPU profiling +2. Memory profiling +3. Build time profiling diff --git a/posts/gemini/feed.xml b/posts/gemini/feed.xml @@ -7,9 +7,285 @@ <language>en-us</language> <managingEditor>andrew@laack.co</managingEditor> <webMaster>andrew@laack.co</webMaster> -<lastBuildDate>Sat, 17 Oct 2026 15:40:30 -0500</lastBuildDate> +<lastBuildDate>Sat, 17 Oct 2026 15:51:38 -0500</lastBuildDate> <atom:link href="gemini://blog.laack.co/feed.xml" rel="self" type="application/rss+xml"/> <item> +<title><![CDATA[Basic Software Profiling: CPU Usage]]></title> +<link>gemini://blog.laack.co/basic-software-profiling-cpu-usage.gmi</link> +<description><![CDATA[<p>Software today is much slower than it should be. I believe this is, in part, because most software developers do not understand performance.</p> +<p>An easily explainable aspect of performance is profiling, and in this post I’ll discuss the basics of if. I’ll be focusing on compiled programs, but the core ideas may be applied to interpreted programs too.</p> +<h2 id="tools">Tools</h2> +<p><a href="https://en.wikipedia.org/wiki/Perf_%28Linux%29">Perf</a> is a suite of tools for program profiling on Linux. We will mainly use it to collect aggregate performance metrics and call stack information. Analogous tools exist for other Free operating systems, and many bespoke tools exist for interpreted programs.</p> +<p><a href="https://github.com/brendangregg/FlameGraph">Flamegraph</a> is a suite of tools for generating flamegraphs, a type of graph designed for the visualization of performance data. I’ll be using my <a href="https://git.laack.co/flamegraph">fork of flamegraph</a> which doesn’t add j***s***** to generated SVGs, but apart from that they function identically.</p> +<h2 id="reading-a-flamegraph">Reading a Flamegraph</h2> +<p>Look at the flamegraph and try to understand the performance characteristics of the program:</p> +<object data="/images/ex1-fg.svg" type="image/svg+xml" width="100%"> +</object> +<details> +<summary> +Source code for the program that generated this flamegraph +</summary> +<pre class="cpp"><code>uint32_t square(uint32_t input) { + return input * input; +} + +// helper function to show stack depths +uint32_t squareHelper(uint32_t input) { + uint32_t x = 100 * 100; + return square(input); +} + +uint32_t factorial(uint32_t input) { + uint32_t acc = 1; + + while(input &gt; 1) { + acc *= input; + input -= 1; + } + return acc; +} + +int main() { + + uint32_t ub = std::sqrt(UINT32_MAX); + + for(uint32_t i = 0; i &lt; ub; ++i) { + for(uint32_t x = 0; x &lt; 100000; ++x) { + square(i); + } + + for(uint32_t x = 0; x &lt; 100000; ++x) { + squareHelper(i); + } + + for(uint32_t x = 0; x &lt; 20000; ++x) { + factorial(x % 10); + } + } +}</code></pre> +</details> +<p>The amount of time spent with the <code>main</code> function on the call stack is represented by the width of its bar.</p> +<p>Notice how there are multiple bars that say <code>square</code>. This is because we execute <code>square</code> at different call stack depths, and the width of each bar is proportional to the amount of time we spent with <code>square</code> on the call stack at a given stack depth.</p> +<p>Given how the widths are calculated, we can deduce the top-down width of a bar is how much time the program spent executing a specific function. In the case of <code>squareHelper</code> and <code>square</code>, we see we spent more time executing <code>squareHelper</code> than <code>square</code>, evidenced by the fact that from a top-down view, <code>squareHelper</code> is wider.</p> +<p>The astute among you might’ve noticed <code>square</code> and <code>squareHelper</code> seem to have been executed a few times outside of the main loop. This is due to how <code>perf</code> calculates the stack depth. The specifics of this are more in the weeds than we’ll discuss here, but it shouldn’t meaningfully impact the high level information that can be obtained from a flamegraph.</p> +<h2 id="perf">Perf</h2> +<p>I’ve been loose with my words in terms of the specific data being displayed by the flamegraph. The command I used to generate the data for the flamegraph doesn’t technically track the amount of time spent with a function on the stack at a given depth, we are instead tracking the number of on-CPU samples taken with the function on the top of the call stack. This distinction is important because things like blocking I/O will not be reflected in our timings.</p> +<p>This is the command I ran to generate the data:</p> +<pre class="bash"><code>perf record -g ./ex1</code></pre> +<p>The <code>perf record</code> command is used to run a command and record its performance profile, which is written to the <code>perf.data</code> file. By default, <code>perf record</code> will sample at 1000Hz while the program is on-CPU. The final option, <code>-g</code>, is used to specify we are interested in having the call stack included in our data.</p> +<p>To view aggregate information about the data in the <code>perf.data</code> file we can run:</p> +<pre class="bash"><code>perf report -g + ++ 100.00% 0.00% ex1 ex1 [.] _start ++ 100.00% 0.00% ex1 libc.so.6 [.] __libc_start_main ++ 100.00% 0.00% ex1 libc.so.6 [.] call_init (inlined) ++ 100.00% 0.00% ex1 libc.so.6 [.] 0x00007f6302a27781 ++ 99.79% 39.46% ex1 ex1 [.] main ++ 34.97% 19.94% ex1 ex1 [.] squareHelper(unsigned int) ++ 29.99% 19.90% ex1 ex1 [.] square(unsigned int) ++ 21.61% 20.69% ex1 ex1 [.] factorial(unsigned int) +...</code></pre> +<p>If you are interested in information about which functions are calling each other, and the associated on-CPU times, you can run this command:</p> +<pre class="bash"><code>perf report -g --stdio + +... + +|--88.87%--main +| | +| |--19.84%--squareHelper(unsigned int) +| | | +| | --4.99%--square(unsigned int) +| | +| |--19.55%--factorial(unsigned int) +| | +| --10.07%--square(unsigned int) +| +|--5.02%--squareHelper(unsigned int) +| +|--4.98%--square(unsigned int) +| +--1.13%--factorial(unsigned int) +...</code></pre> +<p>I often find the <code>--stdio</code> command difficult to read because the report can get wide for non-trivial programs, and horizontal navigation in the terminal is unergonomic.</p> +<p>The final command worth mentioning is:</p> +<pre><code>perf stat ./ex1 + + Performance counter stats for &#39;./ex1&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 132 page-faults:u # 4.1 faults/sec page_faults_per_second + 32,436.48 msec task-clock:u # nan CPUs CPUs_utilized + 356,775 branch-misses:u # 0.0 % branch_miss_rate (88.89%) + 63,706,083,988 branches:u # 1964.0 M/sec branch_frequency (88.89%) + 134,126,037,336 cpu-cycles:u # 4.1 GHz cycles_frequency (88.89%) + 300,951,617,557 instructions:u # 2.2 instructions insn_per_cycle (88.89%) + TopdownL1 # 4.4 % tma_backend_bound + # 61.4 % tma_bad_speculation (88.89%) + # 0.0 % tma_frontend_bound (77.78%) + # 34.2 % tma_retiring (88.89%) + + 32.440411262 seconds time elapsed + + 32.234190000 seconds user + 0.002939000 seconds sys</code></pre> +<p>Which provides high level information about the program’s entire execution.</p> +<p>This is the command I run most frequently when validating optimizations that target functions making up a sizeable percentage of the program’s CPU usage. If the program is uniformly expensive across many functions, <code>perf stat</code> is less useful than a full report / flamegraph.</p> +<h2 id="case-study">Case Study</h2> +<h3 id="amfora">Amfora</h3> +<p><a href="https://github.com/makew0rld/amfora">Amfora</a> is a fancy terminal-based Gemini protocol browser with syntax highlighting. Gemini is a text-based protocol which primarily distributes <a href="https://geminiprotocol.net/docs/gemtext.gmi">gemtext</a> documents. These documents use a similar syntax to markdown, but it’s slightly simpler. I chose to profile this program because software in the Gemini ecosystem is often simple to understand, making it great for educational use cases.</p> +<p>When I start profiling software I often find it useful to use an oracle to understand what’s possible. An oracle in this context is a program that performs comparable tasks with decent performance characteristics. I chose <a href="https://git.laack.co/foreseen/">Foreseen</a>, a Gemini protocol browser I developed, as the oracle because it fulfils both criteria.</p> +<p>I like to start by searching for worst case performance scenarios. Worst cases are often an amplification of an average case, so by improving worst case performance we can often improve the average case.</p> +<p>To start our quest to find worse case scenarios for Amfora, we’ll load a large text file from disk, killing the program once the first lines of text are drawn in the terminal. Since neither browser supports text streaming, they both have to load and parse the document and then send text to the terminal, making this is a fair comparison. To generate a baseline document I executed the following Python code:</p> +<pre class="python"><code>with open(&quot;lf.txt&quot;, &quot;w&quot;) as f: + for i in range(0,200000): + f.write(str(i) + &quot;\n&quot;)</code></pre> +<p>The resulting file was ~1.3MB in size. I then ran both programs killing each program with Ctrl-C at first draw.</p> +<pre><code>perf stat amfora lf.txt + +Performance counter stats for &#39;amfora lf.txt&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 2,742 page-faults:u # 9306.7 faults/sec page_faults_per_second + 294.63 msec task-clock:u # nan CPUs CPUs_utilized + 1,052,499 branch-misses:u # 0.2 % branch_miss_rate (90.02%) + 575,712,764 branches:u # 1954.1 M/sec branch_frequency (89.14%) + 955,922,330 cpu-cycles:u # 3.2 GHz cycles_frequency (87.42%) + 2,669,651,845 instructions:u # 2.8 instructions insn_per_cycle (88.36%) + TopdownL1 # 6.7 % tma_backend_bound + # 24.9 % tma_bad_speculation (90.11%) + # 16.4 % tma_frontend_bound (79.92%) + # 52.0 % tma_retiring (90.37%) + + 0.473990805 seconds time elapsed + + 0.272938000 seconds user + 0.021266000 seconds sys +</code></pre> +<pre><code>perf stat foreseen lf.txt + + Performance counter stats for &#39;foreseen lf.txt&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 8,936 page-faults:u # 104842.2 faults/sec page_faults_per_second + 85.23 msec task-clock:u # nan CPUs CPUs_utilized + 209,148 branch-misses:u # 0.1 % branch_miss_rate (88.30%) + 176,970,330 branches:u # 2076.3 M/sec branch_frequency (88.29%) + 248,919,966 cpu-cycles:u # 2.9 GHz cycles_frequency (88.29%) + 736,194,375 instructions:u # 3.0 instructions insn_per_cycle (88.99%) + TopdownL1 # 3.9 % tma_backend_bound + # 29.7 % tma_bad_speculation (89.50%) + # 21.2 % tma_frontend_bound (77.84%) + # 45.2 % tma_retiring (88.33%) + + 0.354029324 seconds time elapsed + + 0.066861000 seconds user + 0.021186000 seconds sys</code></pre> +<p>This is very unscientific, especially given I was watching to see when to kill the programs, but this is how I normally start profiling. We can’t conclude anything from these results as the timings are within margin of error.</p> +<p>Since Amfora can highlight code let’s see how that impacts performance. I pasted the Python code above at the top of the text file, encapsulated by gemtext preformatted text lines, and then renamed the file to <code>lf.gmi</code>. We then find:</p> +<pre><code>perf stat amfora lf.gmi + + Performance counter stats for &#39;amfora lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 900,174 page-faults:u # 12090.4 faults/sec page_faults_per_second + 74,453.35 msec task-clock:u # nan CPUs CPUs_utilized + 229,042,926 branch-misses:u # 0.4 % branch_miss_rate (88.94%) + 62,713,218,442 branches:u # 842.3 M/sec branch_frequency (88.85%) + 153,193,058,619 cpu-cycles:u # 2.1 GHz cycles_frequency (88.97%) + 306,159,421,078 instructions:u # 2.0 instructions insn_per_cycle (88.90%) + TopdownL1 # 0.6 % tma_backend_bound + # 97.8 % tma_bad_speculation (88.83%) + # -0.1 % tma_frontend_bound (77.86%) + # 1.7 % tma_retiring (88.91%) + + 28.983517177 seconds time elapsed + + 63.894262000 seconds user + 8.207151000 seconds sys</code></pre> +<pre><code>perf stat foreseen lf.gmi + + Performance counter stats for &#39;foreseen lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,551 page-faults:u # 90396.7 faults/sec page_faults_per_second + 72.47 msec task-clock:u # nan CPUs CPUs_utilized + 154,134 branch-misses:u # 0.1 % branch_miss_rate (88.98%) + 166,258,529 branches:u # 2294.2 M/sec branch_frequency (88.99%) + 232,523,911 cpu-cycles:u # 3.2 GHz cycles_frequency (88.96%) + 705,288,446 instructions:u # 3.0 instructions insn_per_cycle (88.93%) + TopdownL1 # 7.6 % tma_backend_bound + # 23.2 % tma_bad_speculation (88.96%) + # 18.8 % tma_frontend_bound (77.41%) + # 50.4 % tma_retiring (88.45%) + + 0.381682961 seconds time elapsed + + 0.058132000 seconds user + 0.015949000 seconds sys</code></pre> +<p>That’s a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let’s look at a flamegraph for this.</p> +<object data="/images/amfora-before.svg" type="image/svg+xml" width="100%"> +</object> +<p>So 68.34% of samples occurred while executing the GC. That’s a bit surprising as there shouldn’t be that many allocations occuring. We also see 25.47% of the samples occur when executing the RenderGemini function. Of this, most of it is spent on concatstring3, a function used to concatenate strings.</p> +<p>Looking in the RenderGemini function for string concatenations we find:</p> +<pre><code>rendered += fmt.Sprintf(&quot;[%s]&quot;, config.GetColorString(&quot;preformatted_text&quot;)) + + buf + fmt.Sprintf(&quot;[%s:%s:-]\r\n&quot;, config.GetColorString(&quot;regular_text&quot;), config.GetColorString(&quot;bg&quot;)) + +... + +rendered += buf + &quot;\r\n&quot; + +... + +rendered += ren + +... + +buf += strings.TrimSuffix(lines[i], &quot;\r&quot;) + &quot;\r\n&quot;</code></pre> +<p>Yeah, this isn’t great. Each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into <code>buf</code> until reaching a preformatted text switch line (“```” lines), we have to re-create the accumulator string 200,000 times to parse our file.</p> +<p>Fortunately, this is easy to solve by using a strings.Builder which doesn’t have this issue, and that’s <a href="https://github.com/makew0rld/amfora/pull/370/changes">what I did</a>. After this change, we can perform our profiling again to find:</p> +<pre><code>perf stat amfora lf.gmi + + Performance counter stats for &#39;amfora lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,039 page-faults:u # 4871.9 faults/sec page_faults_per_second + 1,239.55 msec task-clock:u # nan CPUs CPUs_utilized + 3,631,996 branch-misses:u # 0.1 % branch_miss_rate (89.02%) + 3,017,891,055 branches:u # 2434.7 M/sec branch_frequency (88.96%) + 4,654,531,574 cpu-cycles:u # 3.8 GHz cycles_frequency (88.93%) + 14,168,541,807 instructions:u # 3.1 instructions insn_per_cycle (88.81%) + TopdownL1 # 2.2 % tma_backend_bound + # 36.9 % tma_bad_speculation (88.80%) + # 18.5 % tma_frontend_bound (77.83%) + # 42.5 % tma_retiring (88.90%) + + 1.185772969 seconds time elapsed + + 1.193871000 seconds user + 0.039062000 seconds sys</code></pre> +<p>This is much better. We are now using ~33x less cycles, and it’s ~25x faster than before. It’s also only 3x slower than foreseen on this task.</p> +<object data="/images/amfora-after.svg" type="image/svg+xml" width="100%"> +</object> +<p>Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples.</p> +<p>We could continue identifying hotspots, but now there are 4 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller impact on overall performance, so I’m going to leave this as is.</p> +<h2 id="whats-next">What’s Next</h2> +<p>This post only discussed on-CPU profiling for program execution. There are a few additional metrics worth profiling that I’ll discuss in the future:</p> +<ol type="1"> +<li>Off-CPU profiling</li> +<li>Memory profiling</li> +<li>Build time profiling</li> +</ol>]]></description> +<pubDate>Sat, 17 Oct 2026 00:00:00 -0500</pubDate> +<guid>gemini://blog.laack.co/basic-software-profiling-cpu-usage.gmi</guid> +</item> +<item> <title><![CDATA[PyGame vs. Raylib]]></title> <link>gemini://blog.laack.co/pygame-vs-raylib.gmi</link> <description><![CDATA[<p>I implemented <a href="https://cp-algorithms.com/graph/mst_prim.gmi">Prim’s algorithm</a> to create a MST of a graph in Python. I also created a visualization of it, shown below.</p> diff --git a/posts/gemini/index.gmi b/posts/gemini/index.gmi @@ -2,6 +2,7 @@ => gemini://blog.laack.co/feed.xml +=> basic-software-profiling-cpu-usage.gmi basic-software-profiling-cpu-usage 2026/10/17 => pygame-vs-raylib.gmi pygame-vs-raylib 2026/09/20 => easily-hostable-protocols-over-applications.gmi easily-hostable-protocols-over-applications 2026/04/25 => config-files-suck.gmi config-files-suck 2025/11/19 diff --git a/posts/site/basic-software-profiling-cpu-usage.html b/posts/site/basic-software-profiling-cpu-usage.html @@ -0,0 +1,375 @@ +<!DOCTYPE html> +<html xmlns="http://www.w3.org/1999/xhtml"> +<head> + <meta charset="utf-8" /> + <meta name="generator" content="pandoc 3.10.2" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" /> + <title>basic-software-profiling-cpu-usage</title> + <style> + /* Default styles provided by pandoc. + ** See https://pandoc.org/MANUAL.html#variables-for-html for config info. + */ + span.smallcaps{font-variant: small-caps;} + div.columns{display: flex; gap: 1.5em;} + div.column{flex: auto;} + @media screen { + div.columns{gap: min(4vw, 1.5em);} + div.column{overflow-x: auto;} + } + div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;} + /* The extra [class] is a hack that increases specificity enough to + override a similar rule in reveal.js */ + ul.task-list[class]{list-style: none;} + ul.task-list li input[type="checkbox"] { + font-size: inherit; + width: 0.8em; + margin: 0 0.8em 0.2em -1.6em; + vertical-align: middle; + } + .display.math{display: block; text-align: center; margin: 0.5rem auto;} + /* CSS for syntax highlighting */ + html { -webkit-text-size-adjust: 100%; } + pre > code.sourceCode { white-space: pre; position: relative; } + pre > code.sourceCode > span { display: inline-block; line-height: 1.25; } + pre > code.sourceCode > span:empty { height: 1.2em; } + .sourceCode { overflow: visible; } + code.sourceCode > span { color: inherit; text-decoration: inherit; } + div.sourceCode { margin: 1em 0; } + pre.sourceCode { margin: 0; } + @media screen { + div.sourceCode { overflow: auto; } + } + @media print { + pre > code.sourceCode { white-space: pre-wrap; } + pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; } + } + pre.numberSource code + { counter-reset: source-line 0; } + pre.numberSource code > span + { position: relative; left: -4em; counter-increment: source-line; } + pre.numberSource code > span > a:first-child::before + { content: counter(source-line); + position: relative; left: -1em; text-align: right; vertical-align: baseline; + border: none; display: inline-block; + -webkit-touch-callout: none; -webkit-user-select: none; + -khtml-user-select: none; -moz-user-select: none; + -ms-user-select: none; user-select: none; + padding: 0 4px; width: 4em; + background-color: #ffffff; + color: #a0a0a0; + } + pre.numberSource { margin-left: 3em; border-left: 1px solid #a0a0a0; padding-left: 4px; } + div.sourceCode + { color: #1f1c1b; background-color: #ffffff; } + @media screen { + pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; } + } + code span { color: #1f1c1b; } /* Normal */ + code span.al { color: #bf0303; background-color: #f7e6e6; font-weight: bold; } /* Alert */ + code span.an { color: #ca60ca; } /* Annotation */ + code span.at { color: #0057ae; } /* Attribute */ + code span.bn { color: #b08000; } /* BaseN */ + code span.bu { color: #644a9b; font-weight: bold; } /* BuiltIn */ + code span.cf { color: #1f1c1b; font-weight: bold; } /* ControlFlow */ + code span.ch { color: #924c9d; } /* Char */ + code span.cn { color: #aa5500; } /* Constant */ + code span.co { color: #898887; } /* Comment */ + code span.cv { color: #0095ff; } /* CommentVar */ + code span.do { color: #607880; } /* Documentation */ + code span.dt { color: #0057ae; } /* DataType */ + code span.dv { color: #b08000; } /* DecVal */ + code span.er { color: #bf0303; text-decoration: underline; } /* Error */ + code span.ex { color: #0095ff; font-weight: bold; } /* Extension */ + code span.fl { color: #b08000; } /* Float */ + code span.fu { color: #644a9b; } /* Function */ + code span.im { color: #ff5500; } /* Import */ + code span.in { color: #b08000; } /* Information */ + code span.kw { color: #1f1c1b; font-weight: bold; } /* Keyword */ + code span.op { color: #1f1c1b; } /* Operator */ + code span.ot { color: #006e28; } /* Other */ + code span.pp { color: #006e28; } /* Preprocessor */ + code span.re { color: #0057ae; background-color: #e0e9f8; } /* RegionMarker */ + code span.sc { color: #3daee9; } /* SpecialChar */ + code span.ss { color: #ff5500; } /* SpecialString */ + code span.st { color: #bf0303; } /* String */ + code span.va { color: #0057ae; } /* Variable */ + code span.vs { color: #bf0303; } /* VerbatimString */ + code span.wa { color: #bf0303; } /* Warning */ + </style> + <link rel="stylesheet" href="style.css" /> +</head> +<body> +<h1 id="basic-software-profiling-cpu-usage">Basic Software Profiling: CPU Usage</h1> +<h2 id="date-20261017">Date: 2026/10/17</h2> +<p>Software today is much slower than it should be. I believe this is, in part, because most software developers do not understand performance.</p> +<p>An easily explainable aspect of performance is profiling, and in this post I’ll discuss the basics of if. I’ll be focusing on compiled programs, but the core ideas may be applied to interpreted programs too.</p> +<h2 id="tools">Tools</h2> +<p><a href="https://en.wikipedia.org/wiki/Perf_%28Linux%29">Perf</a> is a suite of tools for program profiling on Linux. We will mainly use it to collect aggregate performance metrics and call stack information. Analogous tools exist for other Free operating systems, and many bespoke tools exist for interpreted programs.</p> +<p><a href="https://github.com/brendangregg/FlameGraph">Flamegraph</a> is a suite of tools for generating flamegraphs, a type of graph designed for the visualization of performance data. I’ll be using my <a href="https://git.laack.co/flamegraph">fork of flamegraph</a> which doesn’t add j***s***** to generated SVGs, but apart from that they function identically.</p> +<h2 id="reading-a-flamegraph">Reading a Flamegraph</h2> +<p>Look at the flamegraph and try to understand the performance characteristics of the program:</p> +<object data="/images/ex1-fg.svg" type="image/svg+xml" width="100%"> +</object> +<details> +<summary> +Source code for the program that generated this flamegraph +</summary> +<div class="sourceCode" id="cb1"><pre class="sourceCode cpp"><code class="sourceCode cpp"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="dt">uint32_t</span> square<span class="op">(</span><span class="dt">uint32_t</span> input<span class="op">)</span> <span class="op">{</span></span> +<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> input <span class="op">*</span> input<span class="op">;</span></span> +<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span> +<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="co">// helper function to show stack depths</span></span> +<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a><span class="dt">uint32_t</span> squareHelper<span class="op">(</span><span class="dt">uint32_t</span> input<span class="op">)</span> <span class="op">{</span></span> +<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a> <span class="dt">uint32_t</span> x <span class="op">=</span> <span class="dv">100</span> <span class="op">*</span> <span class="dv">100</span><span class="op">;</span></span> +<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> square<span class="op">(</span>input<span class="op">);</span></span> +<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span> +<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb1-11"><a href="#cb1-11" aria-hidden="true" tabindex="-1"></a><span class="dt">uint32_t</span> factorial<span class="op">(</span><span class="dt">uint32_t</span> input<span class="op">)</span> <span class="op">{</span></span> +<span id="cb1-12"><a href="#cb1-12" aria-hidden="true" tabindex="-1"></a> <span class="dt">uint32_t</span> acc <span class="op">=</span> <span class="dv">1</span><span class="op">;</span></span> +<span id="cb1-13"><a href="#cb1-13" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb1-14"><a href="#cb1-14" aria-hidden="true" tabindex="-1"></a> <span class="cf">while</span><span class="op">(</span>input <span class="op">&gt;</span> <span class="dv">1</span><span class="op">)</span> <span class="op">{</span></span> +<span id="cb1-15"><a href="#cb1-15" aria-hidden="true" tabindex="-1"></a> acc <span class="op">*=</span> input<span class="op">;</span></span> +<span id="cb1-16"><a href="#cb1-16" aria-hidden="true" tabindex="-1"></a> input <span class="op">-=</span> <span class="dv">1</span><span class="op">;</span></span> +<span id="cb1-17"><a href="#cb1-17" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span> +<span id="cb1-18"><a href="#cb1-18" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> acc<span class="op">;</span></span> +<span id="cb1-19"><a href="#cb1-19" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span> +<span id="cb1-20"><a href="#cb1-20" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb1-21"><a href="#cb1-21" aria-hidden="true" tabindex="-1"></a><span class="dt">int</span> main<span class="op">()</span> <span class="op">{</span></span> +<span id="cb1-22"><a href="#cb1-22" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb1-23"><a href="#cb1-23" aria-hidden="true" tabindex="-1"></a> <span class="dt">uint32_t</span> ub <span class="op">=</span> <span class="bu">std::</span>sqrt<span class="op">(</span>UINT32_MAX<span class="op">);</span></span> +<span id="cb1-24"><a href="#cb1-24" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb1-25"><a href="#cb1-25" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span><span class="op">(</span><span class="dt">uint32_t</span> i <span class="op">=</span> <span class="dv">0</span><span class="op">;</span> i <span class="op">&lt;</span> ub<span class="op">;</span> <span class="op">++</span>i<span class="op">)</span> <span class="op">{</span></span> +<span id="cb1-26"><a href="#cb1-26" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span><span class="op">(</span><span class="dt">uint32_t</span> x <span class="op">=</span> <span class="dv">0</span><span class="op">;</span> x <span class="op">&lt;</span> <span class="dv">100000</span><span class="op">;</span> <span class="op">++</span>x<span class="op">)</span> <span class="op">{</span></span> +<span id="cb1-27"><a href="#cb1-27" aria-hidden="true" tabindex="-1"></a> square<span class="op">(</span>i<span class="op">);</span></span> +<span id="cb1-28"><a href="#cb1-28" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span> +<span id="cb1-29"><a href="#cb1-29" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb1-30"><a href="#cb1-30" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span><span class="op">(</span><span class="dt">uint32_t</span> x <span class="op">=</span> <span class="dv">0</span><span class="op">;</span> x <span class="op">&lt;</span> <span class="dv">100000</span><span class="op">;</span> <span class="op">++</span>x<span class="op">)</span> <span class="op">{</span></span> +<span id="cb1-31"><a href="#cb1-31" aria-hidden="true" tabindex="-1"></a> squareHelper<span class="op">(</span>i<span class="op">);</span></span> +<span id="cb1-32"><a href="#cb1-32" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span> +<span id="cb1-33"><a href="#cb1-33" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb1-34"><a href="#cb1-34" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span><span class="op">(</span><span class="dt">uint32_t</span> x <span class="op">=</span> <span class="dv">0</span><span class="op">;</span> x <span class="op">&lt;</span> <span class="dv">20000</span><span class="op">;</span> <span class="op">++</span>x<span class="op">)</span> <span class="op">{</span></span> +<span id="cb1-35"><a href="#cb1-35" aria-hidden="true" tabindex="-1"></a> factorial<span class="op">(</span>x <span class="op">%</span> <span class="dv">10</span><span class="op">);</span></span> +<span id="cb1-36"><a href="#cb1-36" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span> +<span id="cb1-37"><a href="#cb1-37" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span> +<span id="cb1-38"><a href="#cb1-38" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span></code></pre></div> +</details> +<p>The amount of time spent with the <code>main</code> function on the call stack is represented by the width of its bar.</p> +<p>Notice how there are multiple bars that say <code>square</code>. This is because we execute <code>square</code> at different call stack depths, and the width of each bar is proportional to the amount of time we spent with <code>square</code> on the call stack at a given stack depth.</p> +<p>Given how the widths are calculated, we can deduce the top-down width of a bar is how much time the program spent executing a specific function. In the case of <code>squareHelper</code> and <code>square</code>, we see we spent more time executing <code>squareHelper</code> than <code>square</code>, evidenced by the fact that from a top-down view, <code>squareHelper</code> is wider.</p> +<p>The astute among you might’ve noticed <code>square</code> and <code>squareHelper</code> seem to have been executed a few times outside of the main loop. This is due to how <code>perf</code> calculates the stack depth. The specifics of this are more in the weeds than we’ll discuss here, but it shouldn’t meaningfully impact the high level information that can be obtained from a flamegraph.</p> +<h2 id="perf">Perf</h2> +<p>I’ve been loose with my words in terms of the specific data being displayed by the flamegraph. The command I used to generate the data for the flamegraph doesn’t technically track the amount of time spent with a function on the stack at a given depth, we are instead tracking the number of on-CPU samples taken with the function on the top of the call stack. This distinction is important because things like blocking I/O will not be reflected in our timings.</p> +<p>This is the command I ran to generate the data:</p> +<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="ex">perf</span> record <span class="at">-g</span> ./ex1</span></code></pre></div> +<p>The <code>perf record</code> command is used to run a command and record its performance profile, which is written to the <code>perf.data</code> file. By default, <code>perf record</code> will sample at 1000Hz while the program is on-CPU. The final option, <code>-g</code>, is used to specify we are interested in having the call stack included in our data.</p> +<p>To view aggregate information about the data in the <code>perf.data</code> file we can run:</p> +<div class="sourceCode" id="cb3"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="ex">perf</span> report <span class="at">-g</span></span> +<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="ex">+</span> 100.00% 0.00% ex1 ex1 <span class="pp">[</span><span class="ss">.</span><span class="pp">]</span> _start</span> +<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a><span class="ex">+</span> 100.00% 0.00% ex1 libc.so.6 <span class="pp">[</span><span class="ss">.</span><span class="pp">]</span> __libc_start_main</span> +<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a><span class="ex">+</span> 100.00% 0.00% ex1 libc.so.6 <span class="pp">[</span><span class="ss">.</span><span class="pp">]</span> call_init <span class="er">(</span><span class="ex">inlined</span><span class="kw">)</span></span> +<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a><span class="ex">+</span> 100.00% 0.00% ex1 libc.so.6 <span class="pp">[</span><span class="ss">.</span><span class="pp">]</span> 0x00007f6302a27781</span> +<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a><span class="ex">+</span> 99.79% 39.46% ex1 ex1 <span class="pp">[</span><span class="ss">.</span><span class="pp">]</span> main</span> +<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a><span class="ex">+</span> 34.97% 19.94% ex1 ex1 <span class="pp">[</span><span class="ss">.</span><span class="pp">]</span> squareHelper<span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb3-9"><a href="#cb3-9" aria-hidden="true" tabindex="-1"></a><span class="ex">+</span> 29.99% 19.90% ex1 ex1 <span class="pp">[</span><span class="ss">.</span><span class="pp">]</span> square<span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb3-10"><a href="#cb3-10" aria-hidden="true" tabindex="-1"></a><span class="ex">+</span> 21.61% 20.69% ex1 ex1 <span class="pp">[</span><span class="ss">.</span><span class="pp">]</span> factorial<span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb3-11"><a href="#cb3-11" aria-hidden="true" tabindex="-1"></a><span class="ex">...</span></span></code></pre></div> +<p>If you are interested in information about which functions are calling each other, and the associated on-CPU times, you can run this command:</p> +<div class="sourceCode" id="cb4"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="ex">perf</span> report <span class="at">-g</span> <span class="at">--stdio</span></span> +<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="ex">...</span></span> +<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a></span> +<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span><span class="ex">--88.87%--main</span></span> +<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="kw">|</span></span> +<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="kw">|</span><span class="ex">--19.84%--squareHelper</span><span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="kw">|</span> <span class="kw">|</span></span> +<span id="cb4-9"><a href="#cb4-9" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="kw">|</span> <span class="ex">--4.99%--square</span><span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb4-10"><a href="#cb4-10" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="kw">|</span></span> +<span id="cb4-11"><a href="#cb4-11" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="kw">|</span><span class="ex">--19.55%--factorial</span><span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb4-12"><a href="#cb4-12" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="kw">|</span></span> +<span id="cb4-13"><a href="#cb4-13" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="ex">--10.07%--square</span><span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb4-14"><a href="#cb4-14" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span></span> +<span id="cb4-15"><a href="#cb4-15" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span><span class="ex">--5.02%--squareHelper</span><span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb4-16"><a href="#cb4-16" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span></span> +<span id="cb4-17"><a href="#cb4-17" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span><span class="ex">--4.98%--square</span><span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb4-18"><a href="#cb4-18" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span></span> +<span id="cb4-19"><a href="#cb4-19" aria-hidden="true" tabindex="-1"></a><span class="ex">--1.13%--factorial</span><span class="er">(</span><span class="ex">unsigned</span> int<span class="kw">)</span></span> +<span id="cb4-20"><a href="#cb4-20" aria-hidden="true" tabindex="-1"></a><span class="ex">...</span></span></code></pre></div> +<p>I often find the <code>--stdio</code> command difficult to read because the report can get wide for non-trivial programs, and horizontal navigation in the terminal is unergonomic.</p> +<p>The final command worth mentioning is:</p> +<pre><code>perf stat ./ex1 + + Performance counter stats for &#39;./ex1&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 132 page-faults:u # 4.1 faults/sec page_faults_per_second + 32,436.48 msec task-clock:u # nan CPUs CPUs_utilized + 356,775 branch-misses:u # 0.0 % branch_miss_rate (88.89%) + 63,706,083,988 branches:u # 1964.0 M/sec branch_frequency (88.89%) + 134,126,037,336 cpu-cycles:u # 4.1 GHz cycles_frequency (88.89%) + 300,951,617,557 instructions:u # 2.2 instructions insn_per_cycle (88.89%) + TopdownL1 # 4.4 % tma_backend_bound + # 61.4 % tma_bad_speculation (88.89%) + # 0.0 % tma_frontend_bound (77.78%) + # 34.2 % tma_retiring (88.89%) + + 32.440411262 seconds time elapsed + + 32.234190000 seconds user + 0.002939000 seconds sys</code></pre> +<p>Which provides high level information about the program’s entire execution.</p> +<p>This is the command I run most frequently when validating optimizations that target functions making up a sizeable percentage of the program’s CPU usage. If the program is uniformly expensive across many functions, <code>perf stat</code> is less useful than a full report / flamegraph.</p> +<h2 id="case-study">Case Study</h2> +<h3 id="amfora">Amfora</h3> +<p><a href="https://github.com/makew0rld/amfora">Amfora</a> is a fancy terminal-based Gemini protocol browser with syntax highlighting. Gemini is a text-based protocol which primarily distributes <a href="https://geminiprotocol.net/docs/gemtext.gmi">gemtext</a> documents. These documents use a similar syntax to markdown, but it’s slightly simpler. I chose to profile this program because software in the Gemini ecosystem is often simple to understand, making it great for educational use cases.</p> +<p>When I start profiling software I often find it useful to use an oracle to understand what’s possible. An oracle in this context is a program that performs comparable tasks with decent performance characteristics. I chose <a href="https://git.laack.co/foreseen/">Foreseen</a>, a Gemini protocol browser I developed, as the oracle because it fulfils both criteria.</p> +<p>I like to start by searching for worst case performance scenarios. Worst cases are often an amplification of an average case, so by improving worst case performance we can often improve the average case.</p> +<p>To start our quest to find worse case scenarios for Amfora, we’ll load a large text file from disk, killing the program once the first lines of text are drawn in the terminal. Since neither browser supports text streaming, they both have to load and parse the document and then send text to the terminal, making this is a fair comparison. To generate a baseline document I executed the following Python code:</p> +<div class="sourceCode" id="cb6"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">&quot;lf.txt&quot;</span>, <span class="st">&quot;w&quot;</span>) <span class="im">as</span> f:</span> +<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>,<span class="dv">200000</span>):</span> +<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a> f.write(<span class="bu">str</span>(i) <span class="op">+</span> <span class="st">&quot;</span><span class="ch">\n</span><span class="st">&quot;</span>)</span></code></pre></div> +<p>The resulting file was ~1.3MB in size. I then ran both programs killing each program with Ctrl-C at first draw.</p> +<pre><code>perf stat amfora lf.txt + +Performance counter stats for &#39;amfora lf.txt&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 2,742 page-faults:u # 9306.7 faults/sec page_faults_per_second + 294.63 msec task-clock:u # nan CPUs CPUs_utilized + 1,052,499 branch-misses:u # 0.2 % branch_miss_rate (90.02%) + 575,712,764 branches:u # 1954.1 M/sec branch_frequency (89.14%) + 955,922,330 cpu-cycles:u # 3.2 GHz cycles_frequency (87.42%) + 2,669,651,845 instructions:u # 2.8 instructions insn_per_cycle (88.36%) + TopdownL1 # 6.7 % tma_backend_bound + # 24.9 % tma_bad_speculation (90.11%) + # 16.4 % tma_frontend_bound (79.92%) + # 52.0 % tma_retiring (90.37%) + + 0.473990805 seconds time elapsed + + 0.272938000 seconds user + 0.021266000 seconds sys +</code></pre> +<pre><code>perf stat foreseen lf.txt + + Performance counter stats for &#39;foreseen lf.txt&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 8,936 page-faults:u # 104842.2 faults/sec page_faults_per_second + 85.23 msec task-clock:u # nan CPUs CPUs_utilized + 209,148 branch-misses:u # 0.1 % branch_miss_rate (88.30%) + 176,970,330 branches:u # 2076.3 M/sec branch_frequency (88.29%) + 248,919,966 cpu-cycles:u # 2.9 GHz cycles_frequency (88.29%) + 736,194,375 instructions:u # 3.0 instructions insn_per_cycle (88.99%) + TopdownL1 # 3.9 % tma_backend_bound + # 29.7 % tma_bad_speculation (89.50%) + # 21.2 % tma_frontend_bound (77.84%) + # 45.2 % tma_retiring (88.33%) + + 0.354029324 seconds time elapsed + + 0.066861000 seconds user + 0.021186000 seconds sys</code></pre> +<p>This is very unscientific, especially given I was watching to see when to kill the programs, but this is how I normally start profiling. We can’t conclude anything from these results as the timings are within margin of error.</p> +<p>Since Amfora can highlight code let’s see how that impacts performance. I pasted the Python code above at the top of the text file, encapsulated by gemtext preformatted text lines, and then renamed the file to <code>lf.gmi</code>. We then find:</p> +<pre><code>perf stat amfora lf.gmi + + Performance counter stats for &#39;amfora lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 900,174 page-faults:u # 12090.4 faults/sec page_faults_per_second + 74,453.35 msec task-clock:u # nan CPUs CPUs_utilized + 229,042,926 branch-misses:u # 0.4 % branch_miss_rate (88.94%) + 62,713,218,442 branches:u # 842.3 M/sec branch_frequency (88.85%) + 153,193,058,619 cpu-cycles:u # 2.1 GHz cycles_frequency (88.97%) + 306,159,421,078 instructions:u # 2.0 instructions insn_per_cycle (88.90%) + TopdownL1 # 0.6 % tma_backend_bound + # 97.8 % tma_bad_speculation (88.83%) + # -0.1 % tma_frontend_bound (77.86%) + # 1.7 % tma_retiring (88.91%) + + 28.983517177 seconds time elapsed + + 63.894262000 seconds user + 8.207151000 seconds sys</code></pre> +<pre><code>perf stat foreseen lf.gmi + + Performance counter stats for &#39;foreseen lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,551 page-faults:u # 90396.7 faults/sec page_faults_per_second + 72.47 msec task-clock:u # nan CPUs CPUs_utilized + 154,134 branch-misses:u # 0.1 % branch_miss_rate (88.98%) + 166,258,529 branches:u # 2294.2 M/sec branch_frequency (88.99%) + 232,523,911 cpu-cycles:u # 3.2 GHz cycles_frequency (88.96%) + 705,288,446 instructions:u # 3.0 instructions insn_per_cycle (88.93%) + TopdownL1 # 7.6 % tma_backend_bound + # 23.2 % tma_bad_speculation (88.96%) + # 18.8 % tma_frontend_bound (77.41%) + # 50.4 % tma_retiring (88.45%) + + 0.381682961 seconds time elapsed + + 0.058132000 seconds user + 0.015949000 seconds sys</code></pre> +<p>That’s a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let’s look at a flamegraph for this.</p> +<object data="/images/amfora-before.svg" type="image/svg+xml" width="100%"> +</object> +<p>So 68.34% of samples occurred while executing the GC. That’s a bit surprising as there shouldn’t be that many allocations occuring. We also see 25.47% of the samples occur when executing the RenderGemini function. Of this, most of it is spent on concatstring3, a function used to concatenate strings.</p> +<p>Looking in the RenderGemini function for string concatenations we find:</p> +<pre><code>rendered += fmt.Sprintf(&quot;[%s]&quot;, config.GetColorString(&quot;preformatted_text&quot;)) + + buf + fmt.Sprintf(&quot;[%s:%s:-]\r\n&quot;, config.GetColorString(&quot;regular_text&quot;), config.GetColorString(&quot;bg&quot;)) + +... + +rendered += buf + &quot;\r\n&quot; + +... + +rendered += ren + +... + +buf += strings.TrimSuffix(lines[i], &quot;\r&quot;) + &quot;\r\n&quot;</code></pre> +<p>Yeah, this isn’t great. Each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into <code>buf</code> until reaching a preformatted text switch line (“```” lines), we have to re-create the accumulator string 200,000 times to parse our file.</p> +<p>Fortunately, this is easy to solve by using a strings.Builder which doesn’t have this issue, and that’s <a href="https://github.com/makew0rld/amfora/pull/370/changes">what I did</a>. After this change, we can perform our profiling again to find:</p> +<pre><code>perf stat amfora lf.gmi + + Performance counter stats for &#39;amfora lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,039 page-faults:u # 4871.9 faults/sec page_faults_per_second + 1,239.55 msec task-clock:u # nan CPUs CPUs_utilized + 3,631,996 branch-misses:u # 0.1 % branch_miss_rate (89.02%) + 3,017,891,055 branches:u # 2434.7 M/sec branch_frequency (88.96%) + 4,654,531,574 cpu-cycles:u # 3.8 GHz cycles_frequency (88.93%) + 14,168,541,807 instructions:u # 3.1 instructions insn_per_cycle (88.81%) + TopdownL1 # 2.2 % tma_backend_bound + # 36.9 % tma_bad_speculation (88.80%) + # 18.5 % tma_frontend_bound (77.83%) + # 42.5 % tma_retiring (88.90%) + + 1.185772969 seconds time elapsed + + 1.193871000 seconds user + 0.039062000 seconds sys</code></pre> +<p>This is much better. We are now using ~33x less cycles, and it’s ~25x faster than before. It’s also only 3x slower than foreseen on this task.</p> +<object data="/images/amfora-after.svg" type="image/svg+xml" width="100%"> +</object> +<p>Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples.</p> +<p>We could continue identifying hotspots, but now there are 4 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller impact on overall performance, so I’m going to leave this as is.</p> +<h2 id="whats-next">What’s Next</h2> +<p>This post only discussed on-CPU profiling for program execution. There are a few additional metrics worth profiling that I’ll discuss in the future:</p> +<ol type="1"> +<li>Off-CPU profiling</li> +<li>Memory profiling</li> +<li>Build time profiling</li> +</ol> +</body> +</html> diff --git a/posts/site/feed.xml b/posts/site/feed.xml @@ -7,9 +7,285 @@ <language>en-us</language> <managingEditor>andrew@laack.co</managingEditor> <webMaster>andrew@laack.co</webMaster> -<lastBuildDate>Sat, 17 Oct 2026 15:40:30 -0500</lastBuildDate> +<lastBuildDate>Sat, 17 Oct 2026 15:51:38 -0500</lastBuildDate> <atom:link href="https://blog.laack.co/feed.xml" rel="self" type="application/rss+xml"/> <item> +<title><![CDATA[Basic Software Profiling: CPU Usage]]></title> +<link>https://blog.laack.co/basic-software-profiling-cpu-usage.html</link> +<description><![CDATA[<p>Software today is much slower than it should be. I believe this is, in part, because most software developers do not understand performance.</p> +<p>An easily explainable aspect of performance is profiling, and in this post I’ll discuss the basics of if. I’ll be focusing on compiled programs, but the core ideas may be applied to interpreted programs too.</p> +<h2 id="tools">Tools</h2> +<p><a href="https://en.wikipedia.org/wiki/Perf_%28Linux%29">Perf</a> is a suite of tools for program profiling on Linux. We will mainly use it to collect aggregate performance metrics and call stack information. Analogous tools exist for other Free operating systems, and many bespoke tools exist for interpreted programs.</p> +<p><a href="https://github.com/brendangregg/FlameGraph">Flamegraph</a> is a suite of tools for generating flamegraphs, a type of graph designed for the visualization of performance data. I’ll be using my <a href="https://git.laack.co/flamegraph">fork of flamegraph</a> which doesn’t add j***s***** to generated SVGs, but apart from that they function identically.</p> +<h2 id="reading-a-flamegraph">Reading a Flamegraph</h2> +<p>Look at the flamegraph and try to understand the performance characteristics of the program:</p> +<object data="/images/ex1-fg.svg" type="image/svg+xml" width="100%"> +</object> +<details> +<summary> +Source code for the program that generated this flamegraph +</summary> +<pre class="cpp"><code>uint32_t square(uint32_t input) { + return input * input; +} + +// helper function to show stack depths +uint32_t squareHelper(uint32_t input) { + uint32_t x = 100 * 100; + return square(input); +} + +uint32_t factorial(uint32_t input) { + uint32_t acc = 1; + + while(input &gt; 1) { + acc *= input; + input -= 1; + } + return acc; +} + +int main() { + + uint32_t ub = std::sqrt(UINT32_MAX); + + for(uint32_t i = 0; i &lt; ub; ++i) { + for(uint32_t x = 0; x &lt; 100000; ++x) { + square(i); + } + + for(uint32_t x = 0; x &lt; 100000; ++x) { + squareHelper(i); + } + + for(uint32_t x = 0; x &lt; 20000; ++x) { + factorial(x % 10); + } + } +}</code></pre> +</details> +<p>The amount of time spent with the <code>main</code> function on the call stack is represented by the width of its bar.</p> +<p>Notice how there are multiple bars that say <code>square</code>. This is because we execute <code>square</code> at different call stack depths, and the width of each bar is proportional to the amount of time we spent with <code>square</code> on the call stack at a given stack depth.</p> +<p>Given how the widths are calculated, we can deduce the top-down width of a bar is how much time the program spent executing a specific function. In the case of <code>squareHelper</code> and <code>square</code>, we see we spent more time executing <code>squareHelper</code> than <code>square</code>, evidenced by the fact that from a top-down view, <code>squareHelper</code> is wider.</p> +<p>The astute among you might’ve noticed <code>square</code> and <code>squareHelper</code> seem to have been executed a few times outside of the main loop. This is due to how <code>perf</code> calculates the stack depth. The specifics of this are more in the weeds than we’ll discuss here, but it shouldn’t meaningfully impact the high level information that can be obtained from a flamegraph.</p> +<h2 id="perf">Perf</h2> +<p>I’ve been loose with my words in terms of the specific data being displayed by the flamegraph. The command I used to generate the data for the flamegraph doesn’t technically track the amount of time spent with a function on the stack at a given depth, we are instead tracking the number of on-CPU samples taken with the function on the top of the call stack. This distinction is important because things like blocking I/O will not be reflected in our timings.</p> +<p>This is the command I ran to generate the data:</p> +<pre class="bash"><code>perf record -g ./ex1</code></pre> +<p>The <code>perf record</code> command is used to run a command and record its performance profile, which is written to the <code>perf.data</code> file. By default, <code>perf record</code> will sample at 1000Hz while the program is on-CPU. The final option, <code>-g</code>, is used to specify we are interested in having the call stack included in our data.</p> +<p>To view aggregate information about the data in the <code>perf.data</code> file we can run:</p> +<pre class="bash"><code>perf report -g + ++ 100.00% 0.00% ex1 ex1 [.] _start ++ 100.00% 0.00% ex1 libc.so.6 [.] __libc_start_main ++ 100.00% 0.00% ex1 libc.so.6 [.] call_init (inlined) ++ 100.00% 0.00% ex1 libc.so.6 [.] 0x00007f6302a27781 ++ 99.79% 39.46% ex1 ex1 [.] main ++ 34.97% 19.94% ex1 ex1 [.] squareHelper(unsigned int) ++ 29.99% 19.90% ex1 ex1 [.] square(unsigned int) ++ 21.61% 20.69% ex1 ex1 [.] factorial(unsigned int) +...</code></pre> +<p>If you are interested in information about which functions are calling each other, and the associated on-CPU times, you can run this command:</p> +<pre class="bash"><code>perf report -g --stdio + +... + +|--88.87%--main +| | +| |--19.84%--squareHelper(unsigned int) +| | | +| | --4.99%--square(unsigned int) +| | +| |--19.55%--factorial(unsigned int) +| | +| --10.07%--square(unsigned int) +| +|--5.02%--squareHelper(unsigned int) +| +|--4.98%--square(unsigned int) +| +--1.13%--factorial(unsigned int) +...</code></pre> +<p>I often find the <code>--stdio</code> command difficult to read because the report can get wide for non-trivial programs, and horizontal navigation in the terminal is unergonomic.</p> +<p>The final command worth mentioning is:</p> +<pre><code>perf stat ./ex1 + + Performance counter stats for &#39;./ex1&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 132 page-faults:u # 4.1 faults/sec page_faults_per_second + 32,436.48 msec task-clock:u # nan CPUs CPUs_utilized + 356,775 branch-misses:u # 0.0 % branch_miss_rate (88.89%) + 63,706,083,988 branches:u # 1964.0 M/sec branch_frequency (88.89%) + 134,126,037,336 cpu-cycles:u # 4.1 GHz cycles_frequency (88.89%) + 300,951,617,557 instructions:u # 2.2 instructions insn_per_cycle (88.89%) + TopdownL1 # 4.4 % tma_backend_bound + # 61.4 % tma_bad_speculation (88.89%) + # 0.0 % tma_frontend_bound (77.78%) + # 34.2 % tma_retiring (88.89%) + + 32.440411262 seconds time elapsed + + 32.234190000 seconds user + 0.002939000 seconds sys</code></pre> +<p>Which provides high level information about the program’s entire execution.</p> +<p>This is the command I run most frequently when validating optimizations that target functions making up a sizeable percentage of the program’s CPU usage. If the program is uniformly expensive across many functions, <code>perf stat</code> is less useful than a full report / flamegraph.</p> +<h2 id="case-study">Case Study</h2> +<h3 id="amfora">Amfora</h3> +<p><a href="https://github.com/makew0rld/amfora">Amfora</a> is a fancy terminal-based Gemini protocol browser with syntax highlighting. Gemini is a text-based protocol which primarily distributes <a href="https://geminiprotocol.net/docs/gemtext.gmi">gemtext</a> documents. These documents use a similar syntax to markdown, but it’s slightly simpler. I chose to profile this program because software in the Gemini ecosystem is often simple to understand, making it great for educational use cases.</p> +<p>When I start profiling software I often find it useful to use an oracle to understand what’s possible. An oracle in this context is a program that performs comparable tasks with decent performance characteristics. I chose <a href="https://git.laack.co/foreseen/">Foreseen</a>, a Gemini protocol browser I developed, as the oracle because it fulfils both criteria.</p> +<p>I like to start by searching for worst case performance scenarios. Worst cases are often an amplification of an average case, so by improving worst case performance we can often improve the average case.</p> +<p>To start our quest to find worse case scenarios for Amfora, we’ll load a large text file from disk, killing the program once the first lines of text are drawn in the terminal. Since neither browser supports text streaming, they both have to load and parse the document and then send text to the terminal, making this is a fair comparison. To generate a baseline document I executed the following Python code:</p> +<pre class="python"><code>with open(&quot;lf.txt&quot;, &quot;w&quot;) as f: + for i in range(0,200000): + f.write(str(i) + &quot;\n&quot;)</code></pre> +<p>The resulting file was ~1.3MB in size. I then ran both programs killing each program with Ctrl-C at first draw.</p> +<pre><code>perf stat amfora lf.txt + +Performance counter stats for &#39;amfora lf.txt&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 2,742 page-faults:u # 9306.7 faults/sec page_faults_per_second + 294.63 msec task-clock:u # nan CPUs CPUs_utilized + 1,052,499 branch-misses:u # 0.2 % branch_miss_rate (90.02%) + 575,712,764 branches:u # 1954.1 M/sec branch_frequency (89.14%) + 955,922,330 cpu-cycles:u # 3.2 GHz cycles_frequency (87.42%) + 2,669,651,845 instructions:u # 2.8 instructions insn_per_cycle (88.36%) + TopdownL1 # 6.7 % tma_backend_bound + # 24.9 % tma_bad_speculation (90.11%) + # 16.4 % tma_frontend_bound (79.92%) + # 52.0 % tma_retiring (90.37%) + + 0.473990805 seconds time elapsed + + 0.272938000 seconds user + 0.021266000 seconds sys +</code></pre> +<pre><code>perf stat foreseen lf.txt + + Performance counter stats for &#39;foreseen lf.txt&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 8,936 page-faults:u # 104842.2 faults/sec page_faults_per_second + 85.23 msec task-clock:u # nan CPUs CPUs_utilized + 209,148 branch-misses:u # 0.1 % branch_miss_rate (88.30%) + 176,970,330 branches:u # 2076.3 M/sec branch_frequency (88.29%) + 248,919,966 cpu-cycles:u # 2.9 GHz cycles_frequency (88.29%) + 736,194,375 instructions:u # 3.0 instructions insn_per_cycle (88.99%) + TopdownL1 # 3.9 % tma_backend_bound + # 29.7 % tma_bad_speculation (89.50%) + # 21.2 % tma_frontend_bound (77.84%) + # 45.2 % tma_retiring (88.33%) + + 0.354029324 seconds time elapsed + + 0.066861000 seconds user + 0.021186000 seconds sys</code></pre> +<p>This is very unscientific, especially given I was watching to see when to kill the programs, but this is how I normally start profiling. We can’t conclude anything from these results as the timings are within margin of error.</p> +<p>Since Amfora can highlight code let’s see how that impacts performance. I pasted the Python code above at the top of the text file, encapsulated by gemtext preformatted text lines, and then renamed the file to <code>lf.gmi</code>. We then find:</p> +<pre><code>perf stat amfora lf.gmi + + Performance counter stats for &#39;amfora lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 900,174 page-faults:u # 12090.4 faults/sec page_faults_per_second + 74,453.35 msec task-clock:u # nan CPUs CPUs_utilized + 229,042,926 branch-misses:u # 0.4 % branch_miss_rate (88.94%) + 62,713,218,442 branches:u # 842.3 M/sec branch_frequency (88.85%) + 153,193,058,619 cpu-cycles:u # 2.1 GHz cycles_frequency (88.97%) + 306,159,421,078 instructions:u # 2.0 instructions insn_per_cycle (88.90%) + TopdownL1 # 0.6 % tma_backend_bound + # 97.8 % tma_bad_speculation (88.83%) + # -0.1 % tma_frontend_bound (77.86%) + # 1.7 % tma_retiring (88.91%) + + 28.983517177 seconds time elapsed + + 63.894262000 seconds user + 8.207151000 seconds sys</code></pre> +<pre><code>perf stat foreseen lf.gmi + + Performance counter stats for &#39;foreseen lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,551 page-faults:u # 90396.7 faults/sec page_faults_per_second + 72.47 msec task-clock:u # nan CPUs CPUs_utilized + 154,134 branch-misses:u # 0.1 % branch_miss_rate (88.98%) + 166,258,529 branches:u # 2294.2 M/sec branch_frequency (88.99%) + 232,523,911 cpu-cycles:u # 3.2 GHz cycles_frequency (88.96%) + 705,288,446 instructions:u # 3.0 instructions insn_per_cycle (88.93%) + TopdownL1 # 7.6 % tma_backend_bound + # 23.2 % tma_bad_speculation (88.96%) + # 18.8 % tma_frontend_bound (77.41%) + # 50.4 % tma_retiring (88.45%) + + 0.381682961 seconds time elapsed + + 0.058132000 seconds user + 0.015949000 seconds sys</code></pre> +<p>That’s a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let’s look at a flamegraph for this.</p> +<object data="/images/amfora-before.svg" type="image/svg+xml" width="100%"> +</object> +<p>So 68.34% of samples occurred while executing the GC. That’s a bit surprising as there shouldn’t be that many allocations occuring. We also see 25.47% of the samples occur when executing the RenderGemini function. Of this, most of it is spent on concatstring3, a function used to concatenate strings.</p> +<p>Looking in the RenderGemini function for string concatenations we find:</p> +<pre><code>rendered += fmt.Sprintf(&quot;[%s]&quot;, config.GetColorString(&quot;preformatted_text&quot;)) + + buf + fmt.Sprintf(&quot;[%s:%s:-]\r\n&quot;, config.GetColorString(&quot;regular_text&quot;), config.GetColorString(&quot;bg&quot;)) + +... + +rendered += buf + &quot;\r\n&quot; + +... + +rendered += ren + +... + +buf += strings.TrimSuffix(lines[i], &quot;\r&quot;) + &quot;\r\n&quot;</code></pre> +<p>Yeah, this isn’t great. Each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into <code>buf</code> until reaching a preformatted text switch line (“```” lines), we have to re-create the accumulator string 200,000 times to parse our file.</p> +<p>Fortunately, this is easy to solve by using a strings.Builder which doesn’t have this issue, and that’s <a href="https://github.com/makew0rld/amfora/pull/370/changes">what I did</a>. After this change, we can perform our profiling again to find:</p> +<pre><code>perf stat amfora lf.gmi + + Performance counter stats for &#39;amfora lf.gmi&#39;: + + 0 context-switches:u # 0.0 cs/sec cs_per_second + 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second + 6,039 page-faults:u # 4871.9 faults/sec page_faults_per_second + 1,239.55 msec task-clock:u # nan CPUs CPUs_utilized + 3,631,996 branch-misses:u # 0.1 % branch_miss_rate (89.02%) + 3,017,891,055 branches:u # 2434.7 M/sec branch_frequency (88.96%) + 4,654,531,574 cpu-cycles:u # 3.8 GHz cycles_frequency (88.93%) + 14,168,541,807 instructions:u # 3.1 instructions insn_per_cycle (88.81%) + TopdownL1 # 2.2 % tma_backend_bound + # 36.9 % tma_bad_speculation (88.80%) + # 18.5 % tma_frontend_bound (77.83%) + # 42.5 % tma_retiring (88.90%) + + 1.185772969 seconds time elapsed + + 1.193871000 seconds user + 0.039062000 seconds sys</code></pre> +<p>This is much better. We are now using ~33x less cycles, and it’s ~25x faster than before. It’s also only 3x slower than foreseen on this task.</p> +<object data="/images/amfora-after.svg" type="image/svg+xml" width="100%"> +</object> +<p>Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples.</p> +<p>We could continue identifying hotspots, but now there are 4 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller impact on overall performance, so I’m going to leave this as is.</p> +<h2 id="whats-next">What’s Next</h2> +<p>This post only discussed on-CPU profiling for program execution. There are a few additional metrics worth profiling that I’ll discuss in the future:</p> +<ol type="1"> +<li>Off-CPU profiling</li> +<li>Memory profiling</li> +<li>Build time profiling</li> +</ol>]]></description> +<pubDate>Sat, 17 Oct 2026 00:00:00 -0500</pubDate> +<guid>https://blog.laack.co/basic-software-profiling-cpu-usage.html</guid> +</item> +<item> <title><![CDATA[PyGame vs. Raylib]]></title> <link>https://blog.laack.co/pygame-vs-raylib.html</link> <description><![CDATA[<p>I implemented <a href="https://cp-algorithms.com/graph/mst_prim.html">Prim’s algorithm</a> to create a MST of a graph in Python. I also created a visualization of it, shown below.</p> diff --git a/posts/site/index.html b/posts/site/index.html @@ -11,6 +11,7 @@ <h1>Andrew's Blog</h1> <p><a href="feed.xml">RSS Feed</a></p> <ol> +<li><a href="basic-software-profiling-cpu-usage.html">basic-software-profiling-cpu-usage</a> - <em>2026/10/17</em></li> <li><a href="pygame-vs-raylib.html">pygame-vs-raylib</a> - <em>2026/09/20</em></li> <li><a href="easily-hostable-protocols-over-applications.html">easily-hostable-protocols-over-applications</a> - <em>2026/04/25</em></li> <li><a href="config-files-suck.html">config-files-suck</a> - <em>2025/11/19</em></li> diff --git a/posts/wip/basic-software-profiling-cpu-usage.md b/posts/wip/basic-software-profiling-cpu-usage.md @@ -1,346 +0,0 @@ -# Basic Software Profiling: CPU Usage - -Software today is much slower than it should be. I believe this is, in part, because most software developers do not understand performance. - -An easily explainable aspect of performance is profiling, and in this post I'll discuss the basics of if. I'll be focusing on compiled programs, but the core ideas may be applied to interpreted programs too. - -## Tools - -[Perf](https://en.wikipedia.org/wiki/Perf_%28Linux%29) is a suite of tools for program profiling on Linux. We will mainly use it to collect aggregate performance metrics and call stack information. Analogous tools exist for other Free operating systems, and many bespoke tools exist for interpreted programs. - -[Flamegraph](https://github.com/brendangregg/FlameGraph) is a suite of tools for generating flamegraphs, a type of graph designed for the visualization of performance data. I'll be using my [fork of flamegraph](https://git.laack.co/flamegraph) which doesn't add j***s***** to generated SVGs, but apart from that they function identically. - -## Reading a Flamegraph - -Look at the flamegraph and try to understand the performance characteristics of the program: - -<object data="/images/ex1-fg.svg" type="image/svg+xml" width="100%"></object> - -<details> -<summary>Source code for the program that generated this flamegraph</summary> - -```cpp -uint32_t square(uint32_t input) { - return input * input; -} - -// helper function to show stack depths -uint32_t squareHelper(uint32_t input) { - uint32_t x = 100 * 100; - return square(input); -} - -uint32_t factorial(uint32_t input) { - uint32_t acc = 1; - - while(input > 1) { - acc *= input; - input -= 1; - } - return acc; -} - -int main() { - - uint32_t ub = std::sqrt(UINT32_MAX); - - for(uint32_t i = 0; i < ub; ++i) { - for(uint32_t x = 0; x < 100000; ++x) { - square(i); - } - - for(uint32_t x = 0; x < 100000; ++x) { - squareHelper(i); - } - - for(uint32_t x = 0; x < 20000; ++x) { - factorial(x % 10); - } - } -} -``` - -</details> - -The amount of time spent with the `main` function on the call stack is represented by the width of its bar. - -Notice how there are multiple bars that say `square`. This is because we execute `square` at different call stack depths, and the width of each bar is proportional to the amount of time we spent with `square` on the call stack at a given stack depth. - -Given how the widths are calculated, we can deduce the top-down width of a bar is how much time the program spent executing a specific function. In the case of `squareHelper` and `square`, we see we spent more time executing `squareHelper` than `square`, evidenced by the fact that from a top-down view, `squareHelper` is wider. - -The astute among you might've noticed `square` and `squareHelper` seem to have been executed a few times outside of the main loop. This is due to how `perf` calculates the stack depth. The specifics of this are more in the weeds than we'll discuss here, but it shouldn't meaningfully impact the high level information that can be obtained from a flamegraph. - -## Perf - -I've been loose with my words in terms of the specific data being displayed by the flamegraph. The command I used to generate the data for the flamegraph doesn't technically track the amount of time spent with a function on the stack at a given depth, we are instead tracking the number of on-CPU samples taken with the function on the top of the call stack. This distinction is important because things like blocking I/O will not be reflected in our timings. - -This is the command I ran to generate the data: - -```bash -perf record -g ./ex1 -``` - -The `perf record` command is used to run a command and record its performance profile, which is written to the `perf.data` file. By default, `perf record` will sample at 1000Hz while the program is on-CPU. The final option, `-g`, is used to specify we are interested in having the call stack included in our data. - -To view aggregate information about the data in the `perf.data` file we can run: - -```bash -perf report -g - -+ 100.00% 0.00% ex1 ex1 [.] _start -+ 100.00% 0.00% ex1 libc.so.6 [.] __libc_start_main -+ 100.00% 0.00% ex1 libc.so.6 [.] call_init (inlined) -+ 100.00% 0.00% ex1 libc.so.6 [.] 0x00007f6302a27781 -+ 99.79% 39.46% ex1 ex1 [.] main -+ 34.97% 19.94% ex1 ex1 [.] squareHelper(unsigned int) -+ 29.99% 19.90% ex1 ex1 [.] square(unsigned int) -+ 21.61% 20.69% ex1 ex1 [.] factorial(unsigned int) -... -``` - -If you are interested in information about which functions are calling each other, and the associated on-CPU times, you can run this command: - -```bash -perf report -g --stdio - -... - -|--88.87%--main -| | -| |--19.84%--squareHelper(unsigned int) -| | | -| | --4.99%--square(unsigned int) -| | -| |--19.55%--factorial(unsigned int) -| | -| --10.07%--square(unsigned int) -| -|--5.02%--squareHelper(unsigned int) -| -|--4.98%--square(unsigned int) -| ---1.13%--factorial(unsigned int) -... -``` - -I often find the `--stdio` command difficult to read because the report can get wide for non-trivial programs, and horizontal navigation in the terminal is unergonomic. - -The final command worth mentioning is: - -``` -perf stat ./ex1 - - Performance counter stats for './ex1': - - 0 context-switches:u # 0.0 cs/sec cs_per_second - 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second - 132 page-faults:u # 4.1 faults/sec page_faults_per_second - 32,436.48 msec task-clock:u # nan CPUs CPUs_utilized - 356,775 branch-misses:u # 0.0 % branch_miss_rate (88.89%) - 63,706,083,988 branches:u # 1964.0 M/sec branch_frequency (88.89%) - 134,126,037,336 cpu-cycles:u # 4.1 GHz cycles_frequency (88.89%) - 300,951,617,557 instructions:u # 2.2 instructions insn_per_cycle (88.89%) - TopdownL1 # 4.4 % tma_backend_bound - # 61.4 % tma_bad_speculation (88.89%) - # 0.0 % tma_frontend_bound (77.78%) - # 34.2 % tma_retiring (88.89%) - - 32.440411262 seconds time elapsed - - 32.234190000 seconds user - 0.002939000 seconds sys -``` - -Which provides high level information about the program's entire execution. - -This is the command I run most frequently when validating optimizations that target functions making up a sizeable percentage of the program's CPU usage. If the program is uniformly expensive across many functions, `perf stat` is less useful than a full report / flamegraph. - -## Case Study - -### Amfora - -[Amfora](https://github.com/makew0rld/amfora) is a fancy terminal-based Gemini protocol browser with syntax highlighting. Gemini is a text-based protocol which primarily distributes [gemtext](https://geminiprotocol.net/docs/gemtext.gmi) documents. These documents use a similar syntax to markdown, but it's slightly simpler. I chose to profile this program because software in the Gemini ecosystem is often simple to understand, making it great for educational use cases. - -When I start profiling software I often find it useful to use an oracle to understand what's possible. An oracle in this context is a program that performs comparable tasks with decent performance characteristics. I chose [Foreseen](https://git.laack.co/foreseen/), a Gemini protocol browser I developed, as the oracle because it fulfils both criteria. - -I like to start by searching for worst case performance scenarios. Worst cases are often an amplification of an average case, so by improving worst case performance we can often improve the average case. - -To start our quest to find worse case scenarios for Amfora, we'll load a large text file from disk, killing the program once the first lines of text are drawn in the terminal. Since neither browser supports text streaming, they both have to load and parse the document and then send text to the terminal, making this is a fair comparison. To generate a baseline document I executed the following Python code: - -```python -with open("lf.txt", "w") as f: - for i in range(0,200000): - f.write(str(i) + "\n") -``` - -The resulting file was ~1.3MB in size. I then ran both programs killing each program with Ctrl-C at first draw. - -``` -perf stat amfora lf.txt - -Performance counter stats for 'amfora lf.txt': - - 0 context-switches:u # 0.0 cs/sec cs_per_second - 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second - 2,742 page-faults:u # 9306.7 faults/sec page_faults_per_second - 294.63 msec task-clock:u # nan CPUs CPUs_utilized - 1,052,499 branch-misses:u # 0.2 % branch_miss_rate (90.02%) - 575,712,764 branches:u # 1954.1 M/sec branch_frequency (89.14%) - 955,922,330 cpu-cycles:u # 3.2 GHz cycles_frequency (87.42%) - 2,669,651,845 instructions:u # 2.8 instructions insn_per_cycle (88.36%) - TopdownL1 # 6.7 % tma_backend_bound - # 24.9 % tma_bad_speculation (90.11%) - # 16.4 % tma_frontend_bound (79.92%) - # 52.0 % tma_retiring (90.37%) - - 0.473990805 seconds time elapsed - - 0.272938000 seconds user - 0.021266000 seconds sys - -``` - -``` -perf stat foreseen lf.txt - - Performance counter stats for 'foreseen lf.txt': - - 0 context-switches:u # 0.0 cs/sec cs_per_second - 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second - 8,936 page-faults:u # 104842.2 faults/sec page_faults_per_second - 85.23 msec task-clock:u # nan CPUs CPUs_utilized - 209,148 branch-misses:u # 0.1 % branch_miss_rate (88.30%) - 176,970,330 branches:u # 2076.3 M/sec branch_frequency (88.29%) - 248,919,966 cpu-cycles:u # 2.9 GHz cycles_frequency (88.29%) - 736,194,375 instructions:u # 3.0 instructions insn_per_cycle (88.99%) - TopdownL1 # 3.9 % tma_backend_bound - # 29.7 % tma_bad_speculation (89.50%) - # 21.2 % tma_frontend_bound (77.84%) - # 45.2 % tma_retiring (88.33%) - - 0.354029324 seconds time elapsed - - 0.066861000 seconds user - 0.021186000 seconds sys -``` - -This is very unscientific, especially given I was watching to see when to kill the programs, but this is how I normally start profiling. We can't conclude anything from these results as the timings are within margin of error. - -Since Amfora can highlight code let's see how that impacts performance. I pasted the Python code above at the top of the text file, encapsulated by gemtext preformatted text lines, and then renamed the file to `lf.gmi`. We then find: - -``` -perf stat amfora lf.gmi - - Performance counter stats for 'amfora lf.gmi': - - 0 context-switches:u # 0.0 cs/sec cs_per_second - 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second - 900,174 page-faults:u # 12090.4 faults/sec page_faults_per_second - 74,453.35 msec task-clock:u # nan CPUs CPUs_utilized - 229,042,926 branch-misses:u # 0.4 % branch_miss_rate (88.94%) - 62,713,218,442 branches:u # 842.3 M/sec branch_frequency (88.85%) - 153,193,058,619 cpu-cycles:u # 2.1 GHz cycles_frequency (88.97%) - 306,159,421,078 instructions:u # 2.0 instructions insn_per_cycle (88.90%) - TopdownL1 # 0.6 % tma_backend_bound - # 97.8 % tma_bad_speculation (88.83%) - # -0.1 % tma_frontend_bound (77.86%) - # 1.7 % tma_retiring (88.91%) - - 28.983517177 seconds time elapsed - - 63.894262000 seconds user - 8.207151000 seconds sys -``` - -``` -perf stat foreseen lf.gmi - - Performance counter stats for 'foreseen lf.gmi': - - 0 context-switches:u # 0.0 cs/sec cs_per_second - 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second - 6,551 page-faults:u # 90396.7 faults/sec page_faults_per_second - 72.47 msec task-clock:u # nan CPUs CPUs_utilized - 154,134 branch-misses:u # 0.1 % branch_miss_rate (88.98%) - 166,258,529 branches:u # 2294.2 M/sec branch_frequency (88.99%) - 232,523,911 cpu-cycles:u # 3.2 GHz cycles_frequency (88.96%) - 705,288,446 instructions:u # 3.0 instructions insn_per_cycle (88.93%) - TopdownL1 # 7.6 % tma_backend_bound - # 23.2 % tma_bad_speculation (88.96%) - # 18.8 % tma_frontend_bound (77.41%) - # 50.4 % tma_retiring (88.45%) - - 0.381682961 seconds time elapsed - - 0.058132000 seconds user - 0.015949000 seconds sys -``` - -That's a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let's look at a flamegraph for this. - -<object data="/images/amfora-before.svg" type="image/svg+xml" width="100%"></object> - -So 68.34% of samples occurred while executing the GC. That's a bit surprising as there shouldn't be that many allocations occuring. We also see 25.47% of the samples occur when executing the RenderGemini function. Of this, most of it is spent on concatstring3, a function used to concatenate strings. - -Looking in the RenderGemini function for string concatenations, we find: - -``` -rendered += fmt.Sprintf("[%s]", config.GetColorString("preformatted_text")) + - buf + fmt.Sprintf("[%s:%s:-]\r\n", config.GetColorString("regular_text"), config.GetColorString("bg")) - -... - -rendered += buf + "\r\n" - -... - -rendered += ren - -... - -buf += strings.TrimSuffix(lines[i], "\r") + "\r\n" -``` - -Yeah, this isn't great. Each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into `buf` until reaching a preformatted text switch line ("```" lines), we have to re-create the accumulator string 200,000 times to parse our file. - -Fortunately, this is easy to solve by using a strings.Builder which doesn't have this issue, and that's [what I did](https://github.com/makew0rld/amfora/pull/370/changes)*. After this change, we can perform our profiling again to find: - -``` -perf stat amfora lf.gmi - - Performance counter stats for 'amfora lf.gmi': - - 0 context-switches:u # 0.0 cs/sec cs_per_second - 0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second - 6,039 page-faults:u # 4871.9 faults/sec page_faults_per_second - 1,239.55 msec task-clock:u # nan CPUs CPUs_utilized - 3,631,996 branch-misses:u # 0.1 % branch_miss_rate (89.02%) - 3,017,891,055 branches:u # 2434.7 M/sec branch_frequency (88.96%) - 4,654,531,574 cpu-cycles:u # 3.8 GHz cycles_frequency (88.93%) - 14,168,541,807 instructions:u # 3.1 instructions insn_per_cycle (88.81%) - TopdownL1 # 2.2 % tma_backend_bound - # 36.9 % tma_bad_speculation (88.80%) - # 18.5 % tma_frontend_bound (77.83%) - # 42.5 % tma_retiring (88.90%) - - 1.185772969 seconds time elapsed - - 1.193871000 seconds user - 0.039062000 seconds sys -``` - -This is much better. We are now using ~33x less cycles, and it's ~25x faster than before. It's also only 3x slower than foreseen on this task. - -<object data="/images/amfora-after.svg" type="image/svg+xml" width="100%"></object> - -Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples. - -We could continue identifying hotspots, but now there are 4 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller impact on overall performance, so I'm going to leave this as is. - -## What's Next - -This post only discussed on-CPU profiling for program execution. There are a few additional metrics worth profiling that I'll discuss in the future: - -1. Off-CPU profiling -2. Memory profiling -3. Build time profiling