gemini-search

commit 6f97d745e1e7212e61fed3ff65efa5a7a12e3bab
parent bdfe38a18bdcf7dc4ac5427c1d9f0273111f665f
Author: Andrew Laack <andrew@laack.co>
Date:   Wed,  6 May 2026 20:27:08 -0500

Comments for todo work after noticing some issues w/ db / fs consistency

Diffstat:
M
crawl/main.go
 | 
10
++++++++++

1 file changed, 10 insertions(+), 0 deletions(-)
diff --git a/crawl/main.go b/crawl/main.go
@@ -274,6 +274,16 @@ func worker(db *sql.DB, wg *sync.WaitGroup) {
 
 		forwardGeminiLinks := gemtextparser.ParseLinks(body, currentUrl)
 
+		// TODO: We should check to make sure the filename isn't too long before we get to this point
+		// otherwise we have an annoying db inconsistency. 
+		// I noticed this when performing indexing and it said "file name too long" which means the path
+		// was inserted into the db, but presumably nothing was written to disk because it'd've been too long
+		// also, there is a generally worrying incosistency that occurs when writing to disk and using sqlite
+		// without some sort of locking mechanism. 
+
+		// TODO: When the meta != text/gemini, what happens? I'm seeing a bunch of files that are empty; this 
+		// isn't really what i'd expect, I'd expect some short circuiting.
+
 		outputLocation, err := writeSiteOutput(OUTPUT_DIR, body, currentUrl)
 
 		if err != nil {