fmaury/seirdy.one
1
0
Fork 0
mirror of https://git.sr.ht/~seirdy/seirdy.one synced 2024-11-23 21:02:09 +00:00
Code Issues Projects Releases Wiki Activity

slightly re-org robots.txt

Browse source
This commit is contained in:
Rohan Kumar 2022-07-13 18:12:47 -07:00
parent 707e249c2f
commit 5c5f134c95
No known key found for this signature in database
GPG key ID: 1E892DB2A5F84479
1 changed files with 15 additions and 13 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
static/robots.txt
View file

@ -1,34 +1,36 @@
User-agent: * User-agent: *
Disallow: /noindex/ Disallow: /noindex/
Disallow: /misc/ Disallow: /misc/
Disallow: /webmentions/
# "This robot collects content from the Internet for the sole purpose of # helping educational institutions prevent plagiarism. [...] we compare # student papers against the content we find on the Internet to see if we # can find similarities." (http://www.turnitin.com/robot/crawlerinfo.html) # I opt out of online advertising so malware that injects ads on my site won't get paid.
# You should do the same.
User-Agent: Adsbot
Disallow: /
Allow: /ads.txt
# the UA should not be case-sensitive, but I gotta cover my bases.
User-Agent: AdsBot
Disallow: /
Allow: /ads.txt
# > This robot collects content from the Internet for the sole purpose of # helping educational institutions prevent plagiarism. [...] we compare student papers against the content we find on the Internet to see if we # can find similarities. (http://www.turnitin.com/robot/crawlerinfo.html)
# --> fuck off. # --> fuck off.
User-Agent: TurnitinBot User-Agent: TurnitinBot
Disallow: / Disallow: /
# "NameProtect engages in crawling activity in search of a wide range of # > NameProtect engages in crawling activity in search of a wide range of brand and other intellectual property violations that may be of interest to our clients. (http://www.nameprotect.com/botinfo.html)
# brand and other intellectual property violations that may be of interest
# to our clients." (http://www.nameprotect.com/botinfo.html)
# --> fuck off. # --> fuck off.
User-Agent: NPBot User-Agent: NPBot
Disallow: / Disallow: /
# "iThenticate is a new service we have developed to combat the piracy of intellectual property and ensure the originality of written work for# publishers, non-profit agencies, corporations, and newspapers." (http://www.slysearch.com/) # iThenticate is a new service we have developed to combat the piracy of intellectual property and ensure the originality of written work for# publishers, non-profit agencies, corporations, and newspapers. (http://www.slysearch.com/)
# --> fuck off. # --> fuck off.
User-Agent: SlySearch User-Agent: SlySearch
Disallow: / Disallow: /
# "BLEXBot assists internet marketers to get information on the link structure of sites and their interlinking on the web, to avoid any technical and possible legal issues and improve overall online experience." (http://webmeup-crawler.com/) # BLEXBot assists internet marketers to get information on the link structure of sites and their interlinking on the web, to avoid any technical and possible legal issues and improve overall online experience. (http://webmeup-crawler.com/)
# --> fuck off. # --> fuck off.
User-Agent: BLEXBot User-Agent: BLEXBot
Dissalow: /
User-Agent: Adsbot
Disallow: /
User-Agent: AdsBot
Disallow: / Disallow: /
Sitemap: https://seirdy.one/sitemap.xml Sitemap: https://seirdy.one/sitemap.xml