Danbooru

Hitting Cloudflare challenge wall

Posted under Bugs & Features

Hi, I have the same problem as the user here topic #25514 I’m using a simple shell script based on curl which sends one API request to get the json metadata for the batch to be downloaded, then extracts URLs and downloads them. Nothing criminal, and I used it seldom, maybe once or twice per month (not even every month) to grab images by character or artist tag. This amounted to 400–600 images per run, thus 2–3 API requests plus requests for particular files to cdn. subdomain.

The script stopped working since circa November 2023, but I thought maybe this is temporary. But when I tried to download images later with it, the script still didn’t work, there was a cloudflare challenge HTML “Just a moment...” returned for the request of JSON metadata. Some time ago I tried to circumvent the cloudflare wall by forging user agent, referer, and was close to forging cookies too, but finally stumbled among the thread above (after several other threads like it, but older), where evazion explained in details how forging is not supposed to be done, and the “threat” levels, which Cloudflare uses.

I can say, that up until this day I haven’t been forging user agent, referer or anything else, yet I couldn’t use my grabber since last November because of Cloudflare. (By default, curl sets UA string to “curl/X.Y.Z”, so it definitely wasn’t empty.) I suppose that the deal is with me using Tor to access Danbooru, but otherwise I can’t access it, since the domain is blocked. From the browser I can access Danbooru as usual (via Tor indeed), but the grabber script won’t work no matter what.

So, my question is, can my access via API be restored? The script won’t be forging anything and will use “ennzen-danbooru-grabber/X.Y.Z” string for user agent from now on to make the requests distinctive.

kittey said:

Setting a user agent and passing your API key with every request should be enough.

The curl default UA might already be blocked, so setting a custom UA is required.

Changing the user agent was among the first things, which I tried. Nethier setting it to one space, to an empty string, to my username, “Danbooru user $username”, a string of random characters worked. It’s like the man in the linked topic said: until recently it “just worked”, then stopped. Cloudflare is tightening up the screws I suppose.

1