blog

Personal blog
git clone git://git.laack.co/blog.git
Log | Files | Refs

commit 3436b761624b874b214fcf29b80a24c84bf9d30e
parent 9828a2bbad8dcae7b94e96b8c5510489d7962fe3
Author: Andrew Laack <andrew@laack.co>
Date:   Thu,  1 Oct 2026 00:05:32 -0500

Browser rnd stuff

Diffstat:
Mposts/wip/a-gemini-browser.md | 8+++++++-
Dposts/wip/mapping-geminispace.md | 44--------------------------------------------
2 files changed, 7 insertions(+), 45 deletions(-)

diff --git a/posts/wip/a-gemini-browser.md b/posts/wip/a-gemini-browser.md @@ -713,4 +713,10 @@ Admittedly, c++ has some annoying copy stuff where you'll accidentally copy stuf --- -ok, so now that we're finally back to a 1 on 1, it's still not a fair fucking comparison. +alright, let's stick to our own lane; no reason to compare with utter dogshit software. + +--- + +perf record -g foreseen + +(normal usage for a few minutes) diff --git a/posts/wip/mapping-geminispace.md b/posts/wip/mapping-geminispace.md @@ -1,44 +0,0 @@ -# Mapping Geminispace - -Many others have mapped Geminispace for research purposes [???] and searching [???], but that doesn't mean I (you) shouldn't! - -## Cross Roads - -I started at a cross roads; do I download an existing crawl, or do it myself? Within the later category there are two approaches 1) make my own crawler 2) use an existing one. Since I'm primarily interested in the process of learning about Geminispace's composition, I chose the later because I rightfully intuited that it might teach me some stuff in the process. - -## Crawling - -The modern web is a nightmare to crawl. Some time ago I setup a crawler, indexer, and search engine that ran locally, but the process of setting up the crawler was a nightmare because I didn't want to do any JS rendering, and there are lots of adverserial sites on the web, trying to trap crawlers. In contrast, while respecting robots.txt files based on the 'researcher' pseudo-useragent stipulations, I didn't have any of these issues. - -The crawler worked as follows: - -1. Pick a site to crawl based on previous links in a sqlite database -2. Download the site - - I shelled out to gemget for this, stopping downloads if the download reached 1.0MB of data transferred -3. Parse links -4. Insert the links into the sqlite database - -This was much simpler than what I had to do on the web because it didn't make sense to download and parse in the same pass so I had a crawler that just crawled links, and a separate indexer that pulled out links from downloaded html. This was not necessary with gemtext because the parsing is much cheaper (computationally), and the number of pages to crawl was also much smaller so I wasn't exceptionally concerned about the overhead of doing both steps within the same execution context. - -I could've stuck to a single process crawling geminispace, but I wanted to go a bit quicker, so I spun up 5 processes that would grab 10 random links from the database, and then perform the steps to download and parse them. - -In the end, I was left with a sqlite database that linked urls that were requested with their associated UUID which was the flat-text file that gemget downloaded to my disk, and links between sites. - -### Learnings - -#### The Big Sites - -I've known for some time that tuxmachines has lots of information in geminispace, to the point that using tlgs.one to search for things is rather difficult due to the quantity of stuff they have. As such, after doing crawling for a few hours, I added them to a blacklist of domains to skip over. Along with them I blacklisted taz.de which appears to be a german syndication, and techrights.org. I knew about these sites prior to doing the crawling, except for taz.de. - -These were the sites that I blacklisted during crawling to make sure I could crawl sensibly, but what about the other big players that I didn't flag, but do host lots of content in geminispace? - -#### Clustering - -#### Most Interesting Places - -#### Aggregate Metrics - -- ~% of dead links - - I didn't have retries -- language distribution -- entropy