fmaury/seirdy.one
1
0
Fork 0
mirror of https://git.sr.ht/~seirdy/seirdy.one synced 2024-11-23 12:52:10 +00:00
Code Issues Projects Releases Wiki Activity

More robots.txt exclusions

Browse source 
For shitty services that at least respect robots.txt
This commit is contained in:
Rohan Kumar 2023-07-24 15:33:02 -07:00
parent 48f1e3873e
commit 287a0a5dc0
No known key found for this signature in database
GPG key ID: 1E892DB2A5F84479
1 changed files with 11 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
static/robots.txt
View file

@ -3,10 +3,14 @@ Disallow: /noindex/
Disallow: /misc/
# I opt out of online advertising so malware that injects ads on my site won't get paid.
# You should do the same.
# You should do the same. my ads.txt file contains a standard placeholder to forbid any
# compliant ad networks from paying for ad placement on my domain.
User-Agent: Adsbot
Disallow: /
Allow: /ads.txt
Allow: /app-ads.txt
# The next three are borrowed from https://www.videolan.org/robots.txt
# > This robot collects content from the Internet for the sole purpose of # helping educational institutions prevent plagiarism. [...] we compare student papers against the content we find on the Internet to see if we # can find similarities. (http://www.turnitin.com/robot/crawlerinfo.html)
# --> fuck off.
@ -28,6 +32,12 @@ Disallow: /
User-Agent: BLEXBot
Disallow: /
# Providing Intellectual Property professionals with superior brand protection services by artfully merging the latest technology with expert analysis. (https://www.checkmarknetwork.com/spider.html/)
# "The Internet is just way to big to effectively police alone." (ACTUAL quote)
# --> fuck off.
User-agent: CheckMarkNetwork/1.0 (+https://www.checkmarknetwork.com/spider.html)
Disallow: /
# Eat shit, OpenAI.
User-agent: ChatGPT-User
Disallow: /