fmaury/seirdy.one
1
0
Fork 0
mirror of https://git.sr.ht/~seirdy/seirdy.one synced 2024-11-14 09:42:09 +00:00
Code Issues Projects Releases Wiki Activity

Block webz GenAI scraping

Browse source
This commit is contained in:
Seirdy 2024-10-23 00:09:46 -04:00
parent 26484342a2
commit aea602e88a
No known key found for this signature in database
GPG key ID: 1E892DB2A5F84479
1 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
static/robots.txt
View file

@ -129,6 +129,10 @@ Disallow: /
User-agent: Cotoyogi
Disallow: /
# https://webz.io/bot.html
User-agent: Webzio-extended
Disallow: /
# I'm not blocking CCBot for now. It publishes a free index for anyone to use.
# Googe used this to train the initial version of Bard (now called Gemini).
# I allow CCBot since its index is also used for upstart/hobbyist search engines