blog

Personal blog
git clone git://git.laack.co/blog.git
Log | Files | Refs

commit de1880a75ceb4c116e90f037ef34dbed1ec4e0a6
parent dda634f991e7eaf74c7feeffd8d86f7a38908140
Author: Andrew Laack <andrew@laack.co>
Date:   Tue, 29 Sep 2026 10:48:42 -0500

Started working on post about geminispace mapping

Diffstat:
M.gitignore | 1+
Aposts/wip/mapping-geminispace.md | 44++++++++++++++++++++++++++++++++++++++++++++
Mpython/geminispace/search.py | 17++++++++++++++++-
Mpython/search-engines/search.csv | 6++++++
4 files changed, 67 insertions(+), 1 deletion(-)

diff --git a/.gitignore b/.gitignore @@ -5,3 +5,4 @@ *.jpg python/youtube/urls/urls.txt data +robots diff --git a/posts/wip/mapping-geminispace.md b/posts/wip/mapping-geminispace.md @@ -0,0 +1,44 @@ +# Mapping Geminispace + +Many others have mapped Geminispace for research purposes [???] and searching [???], but that doesn't mean I (you) shouldn't! + +## Cross Roads + +I started at a cross roads; do I download an existing crawl, or do it myself? Within the later category there are two approaches 1) make my own crawler 2) use an existing one. Since I'm primarily interested in the process of learning about Geminispace's composition, I chose the later because I rightfully intuited that it might teach me some stuff in the process. + +## Crawling + +The modern web is a nightmare to crawl. Some time ago I setup a crawler, indexer, and search engine that ran locally, but the process of setting up the crawler was a nightmare because I didn't want to do any JS rendering, and there are lots of adverserial sites on the web, trying to trap crawlers. In contrast, while respecting robots.txt files based on the 'researcher' pseudo-useragent stipulations, I didn't have any of these issues. + +The crawler worked as follows: + +1. Pick a site to crawl based on previous links in a sqlite database +2. Download the site + - I shelled out to gemget for this, stopping downloads if the download reached 1.0MB of data transferred +3. Parse links +4. Insert the links into the sqlite database + +This was much simpler than what I had to do on the web because it didn't make sense to download and parse in the same pass so I had a crawler that just crawled links, and a separate indexer that pulled out links from downloaded html. This was not necessary with gemtext because the parsing is much cheaper (computationally), and the number of pages to crawl was also much smaller so I wasn't exceptionally concerned about the overhead of doing both steps within the same execution context. + +I could've stuck to a single process crawling geminispace, but I wanted to go a bit quicker, so I spun up 5 processes that would grab 10 random links from the database, and then perform the steps to download and parse them. + +In the end, I was left with a sqlite database that linked urls that were requested with their associated UUID which was the flat-text file that gemget downloaded to my disk, and links between sites. + +### Learnings + +#### The Big Sites + +I've known for some time that tuxmachines has lots of information in geminispace, to the point that using tlgs.one to search for things is rather difficult due to the quantity of stuff they have. As such, after doing crawling for a few hours, I added them to a blacklist of domains to skip over. Along with them I blacklisted taz.de which appears to be a german syndication, and techrights.org. I knew about these sites prior to doing the crawling, except for taz.de. + +These were the sites that I blacklisted during crawling to make sure I could crawl sensibly, but what about the other big players that I didn't flag, but do host lots of content in geminispace? + +#### Clustering + +#### Most Interesting Places + +#### Aggregate Metrics + +- ~% of dead links + - I didn't have retries +- language distribution +- entropy diff --git a/python/geminispace/search.py b/python/geminispace/search.py @@ -29,8 +29,22 @@ def parse_robots(text, agents=("*", "researcher")): rules.append(val) return rules + +# these are domains I noticed had lots of stuff. +# I indexed some of them, but added them once I noticed they were far too outsized. +BLACKLIST = {"taz.de", "tuxmachines.org", "techrights.org"} + +def is_blacklisted(host): + host = (host or "").lower().rstrip(".") + return any(host == b or host.endswith("." + b) for b in BLACKLIST) + def can_fetch(robot_dir, domain, target): os.makedirs(robot_dir, exist_ok=True) + + host = urllib.parse.urlparse(target).hostname + if is_blacklisted(host): + return False + pth = os.path.join(robot_dir, domain) if not os.path.exists(pth): subprocess.run(['timeout', '10', 'gemget', f"gemini://{domain}/robots.txt", @@ -56,6 +70,7 @@ def download(data_dir, site): domain = urllib.parse.urlparse(site).netloc if not can_fetch("robots/", domain, site): + print("Can't fetch.") return @@ -89,7 +104,7 @@ def grab_links(prior, gem_file_path): def init_db(filename): - conn = sqlite3.connect(filename, timeout=30) + conn = sqlite3.connect(filename, timeout=300) curs = conn.cursor() curs.execute(''' diff --git a/python/search-engines/search.csv b/python/search-engines/search.csv @@ -95,3 +95,9 @@ code completion model intelligence,ddg,1780963578.2915564,False,0,False,0,1,0 lock screen history,startpage,1789509690.6684983,False,0,False,1,-1,0 lock screen history,brave_search,1789509690.6684983,False,0,False,4,-1,0 lock screen history,ddg,1789509690.6684983,False,0,False,2,-1,0 +python local translation library site:github.com,startpage,1790607464.6173463,False,0,False,0,1,0 +python local translation library site:github.com,brave_search,1790607464.6173463,False,0,False,0,1,0 +python local translation library site:github.com,ddg,1790607464.6173463,False,0,False,0,1,0 +term for only getting a view of a slice of something,startpage,1790646401.3678308,True,60.0,True,1,7,9 +term for only getting a view of a slice of something,brave_search,1790646401.3678308,False,0,False,2,-1,10 +term for only getting a view of a slice of something,ddg,1790646401.3678308,False,0,False,2,3,9