mirror of
https://git.sr.ht/~seirdy/seirdy.one
synced 2024-11-10 00:12:09 +00:00
remove redundant robots.txt entry
This commit is contained in:
parent
64d97f7d06
commit
c648324fe6
1 changed files with 0 additions and 5 deletions
|
@ -8,11 +8,6 @@ User-Agent: Adsbot
|
|||
Disallow: /
|
||||
Allow: /ads.txt
|
||||
|
||||
# the UA should not be case-sensitive, but I gotta cover my bases.
|
||||
User-Agent: AdsBot
|
||||
Disallow: /
|
||||
Allow: /ads.txt
|
||||
|
||||
# > This robot collects content from the Internet for the sole purpose of # helping educational institutions prevent plagiarism. [...] we compare student papers against the content we find on the Internet to see if we # can find similarities. (http://www.turnitin.com/robot/crawlerinfo.html)
|
||||
# --> fuck off.
|
||||
User-Agent: TurnitinBot
|
||||
|
|
Loading…
Reference in a new issue