18 commits

Author	SHA1	Message	Date
Rohan Kumar	247ec11dae	Add some more docs to robots.txt	2024-03-20 21:34:55 -04:00
Rohan Kumar	619c4ec3f6	minor robots.txt refactor + block facebookbot	2024-03-13 02:23:28 -04:00
Rohan Kumar	0e89f7f052	Update docs in robots.txt	2024-03-13 01:14:49 -04:00
Rohan Kumar	1cd7f2c106	add some AI scrapers to robots.txt	2024-03-12 23:53:58 -04:00
Rohan Kumar	4a11ca9f39	opt out of gen-ai training	2024-03-12 20:29:15 -04:00
Rohan Kumar	034c6301fc	Update robots.txt with OpenAI's new bot	2023-08-06 16:54:29 -07:00
Rohan Kumar	c02e2a78a5	Add another IP-violation crawler to robots.txt	2023-07-24 23:43:13 -07:00
Rohan Kumar	287a0a5dc0	More robots.txt exclusions ... For shitty services that at least respect robots.txt	2023-07-24 15:33:02 -07:00
Rohan Kumar	b858a31f40	Fuck off, OpenAI	2023-04-07 18:05:37 -07:00
Rohan Kumar	c648324fe6	remove redundant robots.txt entry	2022-07-24 11:32:16 -07:00
Rohan Kumar	5c5f134c95	slightly re-org robots.txt	2022-07-13 18:12:47 -07:00
Rohan Kumar	95a1685567	Kang VLC's robots.txt commentary	2022-06-12 21:52:28 -07:00
Rohan Kumar	3300d14f99	robots: disallow some toxic bs	2022-04-22 21:45:15 -07:00
Rohan Kumar	e03a35da82	Update robots.txt	2021-06-11 15:09:43 -07:00
Rohan Kumar	aa5c16bdf3	Update robots.txt	2021-01-23 12:47:50 -08:00
rohan kumar	638cb80ed3	Fix robots.txt	2020-12-15 23:14:09 -08:00
rohan kumar	243238be28	Fix robots.txt	2020-11-30 13:06:44 -08:00
rohan kumar	349ba15f38	Add robots.txt	2020-11-29 11:37:33 -08:00