mirror of
https://git.sr.ht/~seirdy/seirdy.one
synced 2024-11-23 21:02:09 +00:00
remove redundant robots.txt entry
This commit is contained in:
parent
64d97f7d06
commit
c648324fe6
1 changed files with 0 additions and 5 deletions
|
@ -8,11 +8,6 @@ User-Agent: Adsbot
|
||||||
Disallow: /
|
Disallow: /
|
||||||
Allow: /ads.txt
|
Allow: /ads.txt
|
||||||
|
|
||||||
# the UA should not be case-sensitive, but I gotta cover my bases.
|
|
||||||
User-Agent: AdsBot
|
|
||||||
Disallow: /
|
|
||||||
Allow: /ads.txt
|
|
||||||
|
|
||||||
# > This robot collects content from the Internet for the sole purpose of # helping educational institutions prevent plagiarism. [...] we compare student papers against the content we find on the Internet to see if we # can find similarities. (http://www.turnitin.com/robot/crawlerinfo.html)
|
# > This robot collects content from the Internet for the sole purpose of # helping educational institutions prevent plagiarism. [...] we compare student papers against the content we find on the Internet to see if we # can find similarities. (http://www.turnitin.com/robot/crawlerinfo.html)
|
||||||
# --> fuck off.
|
# --> fuck off.
|
||||||
User-Agent: TurnitinBot
|
User-Agent: TurnitinBot
|
||||||
|
|
Loading…
Reference in a new issue