commit 7966dec9c0f0f93f3c63ea7b60133cb2aec8d888
parent c50febe00c377ba670e571a6c2867c61be20aa45
Author: Andrew Laack <andrew@laack.co>
Date: Sun, 11 Oct 2026 21:50:00 -0500
Iteration
Diffstat:
2 files changed, 55 insertions(+), 56 deletions(-)
diff --git a/posts/site_wip/basic-software-profiling-cpu-usage.html b/posts/site_wip/basic-software-profiling-cpu-usage.html
@@ -98,12 +98,8 @@
<link rel="stylesheet" href="style.css" />
</head>
<body>
-<p>TODOS:</p>
-<ul>
-<li>review the ending stuff</li>
-</ul>
<h1 id="basic-software-profiling-cpu-usage">Basic Software Profiling: CPU Usage</h1>
-<p>Collectively, we waste trillions of CPU cycles every day because software isn’t profiled frequently. I believe this is an educational problem.</p>
+<p>We waste trillions of CPU cycles every day because software isn’t profiled frequently. I believe this is an educational problem.</p>
<p>This post will explain how you can find the parts of a program hogging the most CPU time. The post will focus on compiled programs, but the core ideas can be applied to interpreted programs.</p>
<h2 id="tools">Tools</h2>
<p><a href="https://en.wikipedia.org/wiki/Perf_%28Linux%29">Perf</a> is a suite of tools for program profiling on Linux. We will mainly use it to collect aggregate performance metrics and call stack information. Analogous tools exist for other Free operating systems, and dedicated tools exist for many interpreted languages to collect call stack information.</p>
@@ -206,7 +202,7 @@ Source code for the program that generated this flamegraph
63,706,083,988 branches:u # 1964.0 M/sec branch_frequency (88.89%)
134,126,037,336 cpu-cycles:u # 4.1 GHz cycles_frequency (88.89%)
300,951,617,557 instructions:u # 2.2 instructions insn_per_cycle (88.89%)
- TopdownL1 # 4.4 % tma_backend_bound
+ TopdownL1 # 4.4 % tma_backend_bound
# 61.4 % tma_bad_speculation (88.89%)
# 0.0 % tma_frontend_bound (77.78%)
# 34.2 % tma_retiring (88.89%)
@@ -226,15 +222,15 @@ Source code for the program that generated this flamegraph
<p>Both of these questions are about decreasing work, not making the work faster. Making work faster can be complex, and it’s often easier to improve performance by decreasing wasted work and algorithmic improvements.</p>
<h2 id="case-study">Case Study</h2>
<h3 id="amfora">Amfora</h3>
-<p>Amfora is a fancy terminal-based Gemini protocol browser with syntax highlighting. When I start profiling software in a well-trodden domain, I find it useful to use an oracle to understand what’s possible. An oracle in this context is a program that performs comparable tasks with sensible performance.</p>
-<p>For Amfora I chose Foreseen (my Gemini protocol browser) as the oracle as it has adequate performance, and performs a sensible subset of the tasks performed by Amfora, notably without the inclusion of syntax highlighting or character addressable coloring.</p>
-<p>I normally start by testing out what I think will be the worst cases. I’ve found worst cases are often amplifications of average cases, so by improving worst case performance you can improve average case performance as well.</p>
-<p>Let’s evaluating the worst case for just rendering. We’ll load a large text file from disk into both browsers, ending execution once text is drawn. I used this code to generate the file:</p>
+<p><a href="https://github.com/makew0rld/amfora">Amfora</a> is a fancy terminal-based Gemini protocol browser with syntax highlighting. Gemini is a text-based protocol which primarily distributes <a href="https://geminiprotocol.net/docs/gemtext.gmi">gemtext</a> documents. These documents use a similar syntax to markdown, but a bit simpler. I chose this program to profile because software in the Gemini ecosystem is generally quite simple to understand, making it great for educational purposes.</p>
+<p>When I start profiling software I often find it useful to use an oracle to understand what’s possible. An oracle in this context is a program that performs comparable tasks with decent performance characteristics.</p>
+<p>I chose <a href="https://git.laack.co/foreseen/">Foreseen</a>, a Gemini protocol browser I developed, as the oracle as it has adequate performance, and performs a sensible subset of the tasks performed by Amfora.</p>
+<p>I normally start by searching for worst case performance scenarios. Worst cases are often an amplification of an average case, so by improving worst case performance you can often improve average case performance.</p>
+<p>Let’s look for the worst case for the task of rendering a textual document. We’ll load a large text file from disk using both browsers, killing the program once the first lines of text are drawn. Since neither browser supports text streaming, they will both have to load and parse the document, and then draw some text to the terminal. To generate the document I executed the following Python code:</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">"lf.txt"</span>, <span class="st">"w"</span>) <span class="im">as</span> f:</span>
<span id="cb9-2"><a href="#cb9-2" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>,<span class="dv">200000</span>):</span>
<span id="cb9-3"><a href="#cb9-3" aria-hidden="true" tabindex="-1"></a> f.write(<span class="bu">str</span>(i) <span class="op">+</span> <span class="st">"</span><span class="ch">\n</span><span class="st">"</span>)</span></code></pre></div>
-<p>The resulting file is ~1.3MB in size, less than the ~2MB max file size for Amfora, and the 5MB truncation size for foreseen.</p>
-<p>I then ran both programs killing each program with Ctrl-C at first draw of the text from the file:</p>
+<p>The resulting file was ~1.3MB in size. I then ran both programs killing each program with Ctrl-C at first draw of the text from the file:</p>
<pre><code>perf stat amfora lf.txt
Performance counter stats for 'amfora lf.txt':
@@ -243,14 +239,14 @@ Performance counter stats for 'amfora lf.txt':
0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second
2,742 page-faults:u # 9306.7 faults/sec page_faults_per_second
294.63 msec task-clock:u # nan CPUs CPUs_utilized
- 1,052,499 branch-misses:u # 0.2 % branch_miss_rate (90.02%)
- 575,712,764 branches:u # 1954.1 M/sec branch_frequency (89.14%)
- 955,922,330 cpu-cycles:u # 3.2 GHz cycles_frequency (87.42%)
+ 1,052,499 branch-misses:u # 0.2 % branch_miss_rate (90.02%)
+ 575,712,764 branches:u # 1954.1 M/sec branch_frequency (89.14%)
+ 955,922,330 cpu-cycles:u # 3.2 GHz cycles_frequency (87.42%)
2,669,651,845 instructions:u # 2.8 instructions insn_per_cycle (88.36%)
- TopdownL1 # 6.7 % tma_backend_bound
- # 24.9 % tma_bad_speculation (90.11%)
- # 16.4 % tma_frontend_bound (79.92%)
- # 52.0 % tma_retiring (90.37%)
+ TopdownL1 # 6.7 % tma_backend_bound
+ # 24.9 % tma_bad_speculation (90.11%)
+ # 16.4 % tma_frontend_bound (79.92%)
+ # 52.0 % tma_retiring (90.37%)
0.473990805 seconds time elapsed
@@ -265,21 +261,21 @@ Performance counter stats for 'amfora lf.txt':
0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second
8,936 page-faults:u # 104842.2 faults/sec page_faults_per_second
85.23 msec task-clock:u # nan CPUs CPUs_utilized
- 209,148 branch-misses:u # 0.1 % branch_miss_rate (88.30%)
- 176,970,330 branches:u # 2076.3 M/sec branch_frequency (88.29%)
- 248,919,966 cpu-cycles:u # 2.9 GHz cycles_frequency (88.29%)
+ 209,148 branch-misses:u # 0.1 % branch_miss_rate (88.30%)
+ 176,970,330 branches:u # 2076.3 M/sec branch_frequency (88.29%)
+ 248,919,966 cpu-cycles:u # 2.9 GHz cycles_frequency (88.29%)
736,194,375 instructions:u # 3.0 instructions insn_per_cycle (88.99%)
- TopdownL1 # 3.9 % tma_backend_bound
- # 29.7 % tma_bad_speculation (89.50%)
- # 21.2 % tma_frontend_bound (77.84%)
- # 45.2 % tma_retiring (88.33%)
+ TopdownL1 # 3.9 % tma_backend_bound
+ # 29.7 % tma_bad_speculation (89.50%)
+ # 21.2 % tma_frontend_bound (77.84%)
+ # 45.2 % tma_retiring (88.33%)
0.354029324 seconds time elapsed
0.066861000 seconds user
0.021186000 seconds sys</code></pre>
-<p>This is very unscientific, but that’s how I like to start profiling. Given how unscientific this methodology is, we can’t conclude anything from this as the timings are within margin of error. It might be worth looking into this further in the future if nothing obvious is found.</p>
-<p>Since Amfora can highlight code, let’s see how the performance changes when we add our python code block from above to the top of the file and rename the file to lf.gmi:</p>
+<p>This is very unscientific, especially given I was eyeballing when to kill the programs, but that’s how I normally start profiling. We can’t conclude anything from these results yet as the timings are within margin of error.</p>
+<p>Since Amfora can highlight code, let’s see how that impacts performance. I pasted the Python code block above at the top of the text file and then renamed the file to <code>lf.gmi</code>. We then find:</p>
<pre><code>perf stat foreseen lf.gmi
Performance counter stats for 'foreseen lf.gmi':
@@ -288,14 +284,14 @@ Performance counter stats for 'amfora lf.txt':
0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second
6,551 page-faults:u # 90396.7 faults/sec page_faults_per_second
72.47 msec task-clock:u # nan CPUs CPUs_utilized
- 154,134 branch-misses:u # 0.1 % branch_miss_rate (88.98%)
- 166,258,529 branches:u # 2294.2 M/sec branch_frequency (88.99%)
- 232,523,911 cpu-cycles:u # 3.2 GHz cycles_frequency (88.96%)
- 705,288,446 instructions:u # 3.0 instructions insn_per_cycle (88.93%)
- TopdownL1 # 7.6 % tma_backend_bound
- # 23.2 % tma_bad_speculation (88.96%)
- # 18.8 % tma_frontend_bound (77.41%)
- # 50.4 % tma_retiring (88.45%)
+ 154,134 branch-misses:u # 0.1 % branch_miss_rate (88.98%)
+ 166,258,529 branches:u # 2294.2 M/sec branch_frequency (88.99%)
+ 232,523,911 cpu-cycles:u # 3.2 GHz cycles_frequency (88.96%)
+ 705,288,446 instructions:u # 3.0 instructions insn_per_cycle (88.93%)
+ TopdownL1 # 7.6 % tma_backend_bound
+ # 23.2 % tma_bad_speculation (88.96%)
+ # 18.8 % tma_frontend_bound (77.41%)
+ # 50.4 % tma_retiring (88.45%)
0.381682961 seconds time elapsed
@@ -313,7 +309,7 @@ Performance counter stats for 'amfora lf.txt':
62,713,218,442 branches:u # 842.3 M/sec branch_frequency (88.85%)
153,193,058,619 cpu-cycles:u # 2.1 GHz cycles_frequency (88.97%)
306,159,421,078 instructions:u # 2.0 instructions insn_per_cycle (88.90%)
- TopdownL1 # 0.6 % tma_backend_bound
+ TopdownL1 # 0.6 % tma_backend_bound
# 97.8 % tma_bad_speculation (88.83%)
# -0.1 % tma_frontend_bound (77.86%)
# 1.7 % tma_retiring (88.91%)
@@ -322,7 +318,7 @@ Performance counter stats for 'amfora lf.txt':
63.894262000 seconds user
8.207151000 seconds sys</code></pre>
-<p>That’s a lot slower for Amfora. Let’s look at a flamegraph for this.</p>
+<p>That’s a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let’s look at a flamegraph for this.</p>
<object data="/images/amfora-before.svg" type="image/svg+xml" width="100%">
</object>
<p>So 68.34% samples occurred while executing the GC, that’s a bit surprising, even given how awful garbage collectors are. We also see that 25.47% of the time, most of the remaining time, is spent in the RenderGemini function. Of this, most of it is spent on concatstring3, a function under the hood that’s used to concatenate strings.</p>
@@ -341,9 +337,9 @@ rendered += ren
...
buf += strings.TrimSuffix(lines[i], "\r") + "\r\n"</code></pre>
-<p>Well, there’s our problem; each of these seemingly innocuous appends is actually very costly. Unlike sensible OO / procedural programming languages, Go strings are immutable so each of these operations will create an entirely new string each time, and since we accumulate into <code>buf</code> until reaching a preformatted text switch line (“```” lines), the file we were trying to open has to re-create the full string 200,000 times, since we only have the tail 200,000 lines in the file don’t have a format switcher.</p>
+<p>Well, there’s our problem; each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into <code>buf</code> until reaching a preformatted text switch line (“```” lines), the file we were trying to open has to re-create the full string 200,000 times, since the tail 200,000 lines in the file don’t have any format switch lines.</p>
<p>This is relatively trivial to solve by using a strings.Builder which doesn’t have this issue, and that’s <a href="https://github.com/makew0rld/amfora/pull/370/changes">what I did</a>*. After this change, we can perform our profiling again to find:</p>
-<p>(NOTE: The profiling I did to find this differed slightly from what’s shown here, hence the difference in numbers between what we see now, and what’s seen in the PR, but they’re similar enough)</p>
+<p>(NOTE: The numbers I referenced in the PR differ from what’s shown here, but they’re similar)</p>
<pre><code>perf stat amfora lf.gmi
Performance counter stats for 'amfora lf.gmi':
@@ -352,26 +348,27 @@ buf += strings.TrimSuffix(lines[i], "\r") + "\r\n"</code></p
0 cpu-migrations:u # 0.0 migrations/sec migrations_per_second
6,039 page-faults:u # 4871.9 faults/sec page_faults_per_second
1,239.55 msec task-clock:u # nan CPUs CPUs_utilized
- 3,631,996 branch-misses:u # 0.1 % branch_miss_rate (89.02%)
- 3,017,891,055 branches:u # 2434.7 M/sec branch_frequency (88.96%)
- 4,654,531,574 cpu-cycles:u # 3.8 GHz cycles_frequency (88.93%)
+ 3,631,996 branch-misses:u # 0.1 % branch_miss_rate (89.02%)
+ 3,017,891,055 branches:u # 2434.7 M/sec branch_frequency (88.96%)
+ 4,654,531,574 cpu-cycles:u # 3.8 GHz cycles_frequency (88.93%)
14,168,541,807 instructions:u # 3.1 instructions insn_per_cycle (88.81%)
- TopdownL1 # 2.2 % tma_backend_bound
- # 36.9 % tma_bad_speculation (88.80%)
- # 18.5 % tma_frontend_bound (77.83%)
- # 42.5 % tma_retiring (88.90%)
+ TopdownL1 # 2.2 % tma_backend_bound
+ # 36.9 % tma_bad_speculation (88.80%)
+ # 18.5 % tma_frontend_bound (77.83%)
+ # 42.5 % tma_retiring (88.90%)
1.185772969 seconds time elapsed
1.193871000 seconds user
0.039062000 seconds sys</code></pre>
-<p>Much better, that’s now using 32x less cycles, and is far faster than before.</p>
+<p>Much better, that’s now using 32x less cycles, and is far faster than before. It’s also only 3x slower than foreseen on this specific task. This tells me there are further optimizations that can be performed, but I believe we’ve fixed the most obvious issue.</p>
<object data="/images/amfora-after.svg" type="image/svg+xml" width="100%">
</object>
<p>Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples, much better.</p>
-<p>We can go deeper into this if we’d like, but there are now 3 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller amount of impact on perf, so we can leave this where it is.</p>
+<p>We can go deeper still on this, but there are now 3 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller amount of impact on performance, so we can leave this where it is. Additional optimizations would also require more investment in instrumentation to support profiling, but I believe most people don’t have to go that far since we’ve seen there are often low hanging fruit that are so obviously incorrect that we don’t have to get very scientific to show tangible improvements.</p>
+<p>(Relevant <a href="https://xkcd.com/2400/">XKCD</a>)</p>
<h2 id="whats-next">What’s Next</h2>
-<p>This post only focused on on-CPU time. There are two additional metrics worth profiling I’ll discuss in the future:</p>
+<p>This post only discussed on-CPU time profiling. There are two additional metric,s worth profiling I’ll discuss in the future:</p>
<ol type="1">
<li>Off CPU profiling</li>
<li>Memory profiling</li>
diff --git a/posts/wip/basic-software-profiling-cpu-usage.md b/posts/wip/basic-software-profiling-cpu-usage.md
@@ -291,7 +291,7 @@ perf stat amfora lf.gmi
8.207151000 seconds sys
```
-That's a lot slower for Amfora (TODO: How much?). Let's look at a flamegraph for this.
+That's a lot slower for Amfora, ~76x slower than Foreseen, and ~61x slower than it was with the normal text file. Let's look at a flamegraph for this.
<object data="/images/amfora-before.svg" type="image/svg+xml" width="100%"></object>
@@ -316,11 +316,11 @@ rendered += ren
buf += strings.TrimSuffix(lines[i], "\r") + "\r\n"
```
-Well, there's our problem; each of these seemingly innocuous appends is actually very costly. Go strings are immutable so each of these operations will create an entirely new string each time, and since we accumulate into `buf` until reaching a preformatted text switch line ("```" lines), the file we were trying to open has to re-create the full string 200,000 times, since we only have the tail 200,000 lines in the file don't have a format switcher.
+Well, there's our problem; each of these seemingly innocuous appends is very costly. Go strings are immutable so each append creates an entirely new string, and since we accumulate into `buf` until reaching a preformatted text switch line ("```" lines), the file we were trying to open has to re-create the full string 200,000 times, since the tail 200,000 lines in the file don't have any format switch lines.
This is relatively trivial to solve by using a strings.Builder which doesn't have this issue, and that's [what I did](https://github.com/makew0rld/amfora/pull/370/changes)*. After this change, we can perform our profiling again to find:
-(NOTE: The profiling I did to find this differed slightly from what's shown here, hence the difference in numbers between what we see now, and what's seen in the PR, but they're similar enough)
+(NOTE: The numbers I referenced in the PR differ from what's shown here, but they're similar)
```
perf stat amfora lf.gmi
@@ -346,17 +346,19 @@ perf stat amfora lf.gmi
0.039062000 seconds sys
```
-Much better, that's now using 32x less cycles, and is far faster than before.
+Much better, that's now using 32x less cycles, and is far faster than before. It's also only 3x slower than foreseen on this specific task. This tells me there are further optimizations that can be performed, but I believe we've fixed the most obvious issue.
<object data="/images/amfora-after.svg" type="image/svg+xml" width="100%"></object>
Looking again at the flamegraph, we see the RenderGemini function is now mostly performing word wrapping, and the GC is only utilizing the CPU during 16.48% of our samples, much better.
-We can go deeper into this if we'd like, but there are now 3 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller amount of impact on perf, so we can leave this where it is.
+We can go deeper still on this, but there are now 3 ~equally costly groupings, and as performance becomes more diffuse, optimizations will generally have a smaller amount of impact on performance, so we can leave this where it is. Additional optimizations would also require more investment in instrumentation to support profiling, but I believe most people don't have to go that far since we've seen there are often low hanging fruit that are so obviously incorrect that we don't have to get very scientific to show tangible improvements.
+
+(Relevant [XKCD](https://xkcd.com/2400/))
## What's Next
-This post only focused on on-CPU time. There are two additional metrics worth profiling I'll discuss in the future:
+This post only discussed on-CPU time profiling. There are two additional metric,s worth profiling I'll discuss in the future:
1. Off CPU profiling
2. Memory profiling