fmaury/seirdy.one
1
0
Fork 0
mirror of https://git.sr.ht/~seirdy/seirdy.one synced 2024-11-10 00:12:09 +00:00
Code Issues Projects Releases Wiki Activity

remove redundant robots.txt entry

Browse source
This commit is contained in:
Rohan Kumar 2022-07-24 11:32:16 -07:00
parent 64d97f7d06
commit c648324fe6
No known key found for this signature in database
GPG key ID: 1E892DB2A5F84479
1 changed files with 0 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
static/robots.txt
View file

@ -8,11 +8,6 @@ User-Agent: Adsbot
Disallow: / Disallow: /
Allow: /ads.txt Allow: /ads.txt
# the UA should not be case-sensitive, but I gotta cover my bases.
User-Agent: AdsBot
Disallow: /
Allow: /ads.txt
# > This robot collects content from the Internet for the sole purpose of # helping educational institutions prevent plagiarism. [...] we compare student papers against the content we find on the Internet to see if we # can find similarities. (http://www.turnitin.com/robot/crawlerinfo.html) # > This robot collects content from the Internet for the sole purpose of # helping educational institutions prevent plagiarism. [...] we compare student papers against the content we find on the Internet to see if we # can find similarities. (http://www.turnitin.com/robot/crawlerinfo.html)
# --> fuck off. # --> fuck off.
User-Agent: TurnitinBot User-Agent: TurnitinBot