felixfoertsch/tildes
1
0
Fork 0
mirror of https://gitlab.com/tildes/tildes.git synced 2026-04-16 06:18:34 +02:00
Code Issues Packages Projects Releases Wiki Activity

robots.txt: Block OpenAI GPTBot

Browse Source
This commit is contained in:
Deimos
2024-02-17 22:46:50 -07:00
parent d0d6b6d3dc
commit f355bc6a80
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
tildes/static/robots.txt
View File

@@ -30,6 +30,10 @@ Disallow: /
User-agent: dotbot
Disallow: /
# https://platform.openai.com/docs/gptbot
User-agent: GPTBot
Disallow: /
# Unknown/suspicious scraper - UA only contains a gmail address
User-agent: MauiBot
Disallow: /