fmaury/seirdy.one
1
0
Fork 0
mirror of https://git.sr.ht/~seirdy/seirdy.one synced 2024-11-27 14:12:09 +00:00
Code Issues Projects Releases Wiki Activity

Add another IP-violation crawler to robots.txt

Browse source
This commit is contained in:
Rohan Kumar 2023-07-24 23:43:13 -07:00
parent de6a37b092
commit c02e2a78a5
No known key found for this signature in database
GPG key ID: 1E892DB2A5F84479
1 changed files with 5 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
static/robots.txt
View file

@ -38,6 +38,11 @@ Disallow: /
User-agent: CheckMarkNetwork/1.0 (+https://www.checkmarknetwork.com/spider.html) User-agent: CheckMarkNetwork/1.0 (+https://www.checkmarknetwork.com/spider.html)
Disallow: / Disallow: /
# Stop trademark violations and affiliate non-compliance in paid search. Automatically monitor your partner and affiliates’ online marketing to protect yourself from harmful brand violations and regulatory risks. We regularly crawl websites on behalf of our clients to ensure content compliance with brand and regulatory guidelines. (https://www.brandverity.com/why-is-brandverity-visiting-me)
# --> fuck off.
User-agent: BrandVerity/1.0
Disallow: /
# Eat shit, OpenAI. # Eat shit, OpenAI.
User-agent: ChatGPT-User User-agent: ChatGPT-User
Disallow: / Disallow: /