12 commits

Author	SHA1	Message	Date
Rohan Kumar	c02e2a78a5	Add another IP-violation crawler to robots.txt	2023-07-24 23:43:13 -07:00
Rohan Kumar	287a0a5dc0	More robots.txt exclusions ... For shitty services that at least respect robots.txt	2023-07-24 15:33:02 -07:00
Rohan Kumar	b858a31f40	Fuck off, OpenAI	2023-04-07 18:05:37 -07:00
Rohan Kumar	c648324fe6	remove redundant robots.txt entry	2022-07-24 11:32:16 -07:00
Rohan Kumar	5c5f134c95	slightly re-org robots.txt	2022-07-13 18:12:47 -07:00
Rohan Kumar	95a1685567	Kang VLC's robots.txt commentary	2022-06-12 21:52:28 -07:00
Rohan Kumar	3300d14f99	robots: disallow some toxic bs	2022-04-22 21:45:15 -07:00
Rohan Kumar	e03a35da82	Update robots.txt	2021-06-11 15:09:43 -07:00
Rohan Kumar	aa5c16bdf3	Update robots.txt	2021-01-23 12:47:50 -08:00
rohan kumar	638cb80ed3	Fix robots.txt	2020-12-15 23:14:09 -08:00
rohan kumar	243238be28	Fix robots.txt	2020-11-30 13:06:44 -08:00
rohan kumar	349ba15f38	Add robots.txt	2020-11-29 11:37:33 -08:00