1
0
Fork 0
mirror of https://git.sr.ht/~seirdy/seirdy.one synced 2024-11-10 00:12:09 +00:00

Add some more docs to robots.txt

This commit is contained in:
Rohan Kumar 2024-03-20 21:34:55 -04:00
parent de3936943e
commit 247ec11dae
No known key found for this signature in database
GPG key ID: 1E892DB2A5F84479

View file

@ -88,4 +88,7 @@ Disallow: /
# I'm not familiar enough with Omgili to make a call here.
# In the long run, my embedded robots meta-tags and headers could cover gen-AI
# I don't block cohere-ai or Perplexitybot: they don't appear to actually scrape data for LLM training purposes. The crawling powers search engines with integrated pre-trained LLMs.
# TODO: investigate whether YouBot scrapes to train its own in-house LLM.
Sitemap: https://seirdy.one/sitemap.xml