mapping-geminispace.md (3045B)
1 # Mapping Geminispace 2 3 Many others have mapped Geminispace for research purposes [???] and searching [???], but that doesn't mean I (you) shouldn't! 4 5 ## Cross Roads 6 7 I started at a cross roads; do I download an existing crawl, or do it myself? Within the later category there are two approaches 1) make my own crawler 2) use an existing one. Since I'm primarily interested in the process of learning about Geminispace's composition, I chose the later because I rightfully intuited that it might teach me some stuff in the process. 8 9 ## Crawling 10 11 The modern web is a nightmare to crawl. Some time ago I setup a crawler, indexer, and search engine that ran locally, but the process of setting up the crawler was a nightmare because I didn't want to do any JS rendering, and there are lots of adverserial sites on the web, trying to trap crawlers. In contrast, while respecting robots.txt files based on the 'researcher' pseudo-useragent stipulations, I didn't have any of these issues. 12 13 The crawler worked as follows: 14 15 1. Pick a site to crawl based on previous links in a sqlite database 16 2. Download the site 17 - I shelled out to gemget for this, stopping downloads if the download reached 1.0MB of data transferred 18 3. Parse links 19 4. Insert the links into the sqlite database 20 21 This was much simpler than what I had to do on the web because it didn't make sense to download and parse in the same pass so I had a crawler that just crawled links, and a separate indexer that pulled out links from downloaded html. This was not necessary with gemtext because the parsing is much cheaper (computationally), and the number of pages to crawl was also much smaller so I wasn't exceptionally concerned about the overhead of doing both steps within the same execution context. 22 23 I could've stuck to a single process crawling geminispace, but I wanted to go a bit quicker, so I spun up 5 processes that would grab 10 random links from the database, and then perform the steps to download and parse them. 24 25 In the end, I was left with a sqlite database that linked urls that were requested with their associated UUID which was the flat-text file that gemget downloaded to my disk, and links between sites. 26 27 ### Learnings 28 29 #### The Big Sites 30 31 I've known for some time that tuxmachines has lots of information in geminispace, to the point that using tlgs.one to search for things is rather difficult due to the quantity of stuff they have. As such, after doing crawling for a few hours, I added them to a blacklist of domains to skip over. Along with them I blacklisted taz.de which appears to be a german syndication, and techrights.org. I knew about these sites prior to doing the crawling, except for taz.de. 32 33 These were the sites that I blacklisted during crawling to make sure I could crawl sensibly, but what about the other big players that I didn't flag, but do host lots of content in geminispace? 34 35 #### Clustering 36 37 #### Most Interesting Places 38 39 #### Aggregate Metrics 40 41 - ~% of dead links 42 - I didn't have retries 43 - language distribution 44 - entropy