fmaury/seirdy.one
1
0
Fork 0
mirror of https://git.sr.ht/~seirdy/seirdy.one synced 2024-11-23 12:52:10 +00:00
Code Issues Projects Releases Wiki Activity

Add some more docs to robots.txt

Browse source
This commit is contained in:
Rohan Kumar 2024-03-20 21:34:55 -04:00
parent de3936943e
commit 247ec11dae
No known key found for this signature in database
GPG key ID: 1E892DB2A5F84479
1 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
static/robots.txt
View file

@ -88,4 +88,7 @@ Disallow: /
# I'm not familiar enough with Omgili to make a call here. # I'm not familiar enough with Omgili to make a call here.
# In the long run, my embedded robots meta-tags and headers could cover gen-AI # In the long run, my embedded robots meta-tags and headers could cover gen-AI
# I don't block cohere-ai or Perplexitybot: they don't appear to actually scrape data for LLM training purposes. The crawling powers search engines with integrated pre-trained LLMs.
# TODO: investigate whether YouBot scrapes to train its own in-house LLM.
Sitemap: https://seirdy.one/sitemap.xml Sitemap: https://seirdy.one/sitemap.xml