Creepy crawlies
Abusive web crawlers consume more CPU than legitimate access on Git.
“we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.”
Konstantin Ryabitsev highlights the significant resource cost of abusive web crawlers compared to legitimate access on the Git repository. This trend raises concerns about the impact of scrapers on resource allocation, particularly for web services like Datasette.