From 03d504e79ab221ed0d3f6acaa0cc69d697a75d9e Mon Sep 17 00:00:00 2001
From: Seirdy <seirdy@seirdy.one>
Date: Wed, 24 Jul 2024 14:23:35 -0400
Subject: [PATCH] New note: common crawl and search engines

---
 .../notes/common-crawl-and-search-engines.md    | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100644 content/notes/common-crawl-and-search-engines.md

diff --git a/content/notes/common-crawl-and-search-engines.md b/content/notes/common-crawl-and-search-engines.md
new file mode 100644
index 0000000..f004a2a
--- /dev/null
+++ b/content/notes/common-crawl-and-search-engines.md
@@ -0,0 +1,17 @@
+---
+title: "Common Crawl and search engines"
+date: 2024-07-24T14:23:35-04:00
+replyURI: "https://corteximplant.net/objects/994acaa1-dd50-44b0-bb50-43211521a779"
+replyTitle: "We really fucking need a community owned search engine index that fucks the corporate rules (but still respects personal websites' indexing preferences)"
+replyType: "SocialMediaPosting"
+replyAuthor: "Marta Sokolska"
+replyAuthorURI: "https://me.pionaiki.com/"
+#syndicatedCopies:
+#    - title: 'The Fediverse'
+#      url: ''
+---
+Common Crawl is the closest thing we have to an open index, though it doesn't meet your requirement of ignoring robots.txt for corporate websites while obeying it for personal sites. Unfortunately, being open and publicly available means that people use it to train LLMs. ([Google did this for initial versions of Bard]({{<relref "/notes/opting-out-of-llm-indexing.md">}})), so a lot of sites block its crawler. Most robots.txt guides for blocking GenAI crawlers include an entry for it now.
+
+[Common Crawl powers Alexandria Search and was the basis of Stract's initial index]({{<relref "/posts/search-engines-with-own-indexes.md#smaller-indexes-or-less-relevant-results">}}), both of which are upstart FOSS engines.
+
+A similar EU-focused project is [OpenWebSearch/Owler](https://openwebsearch.eu/common-goals-with-common-crawl/).