blog

Personal blog
git clone git://git.laack.co/blog.git
Log | Files | Refs

mapping-geminispace.md (3045B)


      1 # Mapping Geminispace
      2 
      3 Many others have mapped Geminispace for research purposes [???] and searching [???], but that doesn't mean I (you) shouldn't!
      4 
      5 ## Cross Roads
      6 
      7 I started at a cross roads; do I download an existing crawl, or do it myself? Within the later category there are two approaches 1) make my own crawler 2) use an existing one. Since I'm primarily interested in the process of learning about Geminispace's composition, I chose the later because I rightfully intuited that it might teach me some stuff in the process. 
      8 
      9 ## Crawling
     10 
     11 The modern web is a nightmare to crawl. Some time ago I setup a crawler, indexer, and search engine that ran locally, but the process of setting up the crawler was a nightmare because I didn't want to do any JS rendering, and there are lots of adverserial sites on the web, trying to trap crawlers. In contrast, while respecting robots.txt files based on the 'researcher' pseudo-useragent stipulations, I didn't have any of these issues.
     12 
     13 The crawler worked as follows:
     14 
     15 1. Pick a site to crawl based on previous links in a sqlite database
     16 2. Download the site
     17     - I shelled out to gemget for this, stopping downloads if the download reached 1.0MB of data transferred
     18 3. Parse links
     19 4. Insert the links into the sqlite database
     20 
     21 This was much simpler than what I had to do on the web because it didn't make sense to download and parse in the same pass so I had a crawler that just crawled links, and a separate indexer that pulled out links from downloaded html. This was not necessary with gemtext because the parsing is much cheaper (computationally), and the number of pages to crawl was also much smaller so I wasn't exceptionally concerned about the overhead of doing both steps within the same execution context.
     22 
     23 I could've stuck to a single process crawling geminispace, but I wanted to go a bit quicker, so I spun up 5 processes that would grab 10 random links from the database, and then perform the steps to download and parse them.
     24 
     25 In the end, I was left with a sqlite database that linked urls that were requested with their associated UUID which was the flat-text file that gemget downloaded to my disk, and links between sites. 
     26 
     27 ### Learnings
     28 
     29 #### The Big Sites
     30 
     31 I've known for some time that tuxmachines has lots of information in geminispace, to the point that using tlgs.one to search for things is rather difficult due to the quantity of stuff they have. As such, after doing crawling for a few hours, I added them to a blacklist of domains to skip over. Along with them I blacklisted taz.de which appears to be a german syndication, and techrights.org. I knew about these sites prior to doing the crawling, except for taz.de.
     32 
     33 These were the sites that I blacklisted during crawling to make sure I could crawl sensibly, but what about the other big players that I didn't flag, but do host lots of content in geminispace?
     34 
     35 #### Clustering
     36 
     37 #### Most Interesting Places
     38 
     39 #### Aggregate Metrics
     40 
     41 - ~% of dead links
     42     - I didn't have retries
     43 - language distribution
     44 - entropy