The Arch Wiki (and possibly other Arch websites, but when I checked I wasn’t able to catch it) uses Anubis (https://anubis.techaro.lol/) to mitigate AI crawlers causing issues on their servers. Anubis works by making your computer do some computations before being let in. For users like you and me, it’d be a minor inconvenience, but for these AI crawlers, it’d cost them a lot of computation as they hit these sites a huge number of times.
Using Anubis is said to be a nuclear option, but as these AI crawlers don’t care about that, sites, especially FOSS ones, started using it.
The Arch Wiki (and possibly other Arch websites, but when I checked I wasn’t able to catch it) uses Anubis (https://anubis.techaro.lol/) to mitigate AI crawlers causing issues on their servers. Anubis works by making your computer do some computations before being let in. For users like you and me, it’d be a minor inconvenience, but for these AI crawlers, it’d cost them a lot of computation as they hit these sites a huge number of times.
Using Anubis is said to be a nuclear option, but as these AI crawlers don’t care about that, sites, especially FOSS ones, started using it.