mirror of
https://git.sr.ht/~seirdy/seirdy.one
synced 2024-11-10 00:12:09 +00:00
Add some more docs to robots.txt
This commit is contained in:
parent
de3936943e
commit
247ec11dae
1 changed files with 3 additions and 0 deletions
|
@ -88,4 +88,7 @@ Disallow: /
|
|||
# I'm not familiar enough with Omgili to make a call here.
|
||||
# In the long run, my embedded robots meta-tags and headers could cover gen-AI
|
||||
|
||||
# I don't block cohere-ai or Perplexitybot: they don't appear to actually scrape data for LLM training purposes. The crawling powers search engines with integrated pre-trained LLMs.
|
||||
# TODO: investigate whether YouBot scrapes to train its own in-house LLM.
|
||||
|
||||
Sitemap: https://seirdy.one/sitemap.xml
|
||||
|
|
Loading…
Reference in a new issue