Posted under Bugs & Features
This has pretty much ruined my RSS reader, thumbnails now return a 403 while occasionally a few do just work. Opening the thumbnail URLs in a new tab does work fine. My rss reader (TTRSS) doesn't do the downloading of the images, it just leaves that to the browser. Thumbnails have no issue loading on danbooru itself.
Update: I'm now blocking anonymously hotlinked images. This is when you embed a Danbooru image on another site and you strip the referrer so we can't see where the image is embedded from. This is done using Referrer-Policy to strip the Referer header.
Hotlinking is allowed as long as I can tell which site the traffic is coming from. Anonymous hotlinking is blocked because there are a lot of shitty hentai sites that hotlink all our images and surround them with porn ads. This is effectively stealing our bandwidth while they profit off it. I'm not paying to run someone else's shitty porn site.
If you have a webapp that embeds Danbooru images, you should make sure it's not stripping the referrer. This could done either in a Referrer-Policy HTTP header, a <meta name="referrer" content="no-referrer"> tag, or on the <img> tag with <img referrerpolicy="no-referrer"/>. Stripping the referrer is usually not necessary because modern browsers only send the domain name anyway, not the full URL, so all we can see is which site is embedding the image, not the exact page on that site. For local webapps, all we'll see is that you're running it on localhost or maybe an internal IP like 192.168.0.1, which doesn't tell us much (and anyway, we already know your actual IP by virtue of you requesting the image).
Gyuutan said:
This has pretty much ruined my RSS reader, thumbnails now return a 403 while occasionally a few do just work. Opening the thumbnail URLs in a new tab does work fine. My rss reader (TTRSS) doesn't do the downloading of the images, it just leaves that to the browser. Thumbnails have no issue loading on danbooru itself.
For Tiny Tiny RSS, you can enable the "Cache media" option in the feed settings as a workaround. Otherwise you'd need a userscript to remove referrerpolicy="no-referrer" from the image tags, or get the maintainers to add an option to remove it.
soredeii said:
Lately, the donmai.moe website doesn't show me any images. Danbooru does work.
It works for me. Make sure you don't have any security or privacy browser extensions that could be removing the referrer header. Try disabling all your extensions, or see if it works in a different browser.
evazion said:
It works for me. Make sure you don't have any security or privacy browser extensions that could be removing the referrer header. Try disabling all your extensions, or see if it works in a different browser.
I tried disabling all extensions and tracking protection but in the end it was my firefox hardening: network.http.referer.XOriginPolicy must be set to 0 (as opposed to 1 or 2).
I guess I'll toggle it on and off or use danbooru instead (it does puzzle me why danbooru does work).
I use an iPhone app called Anime Boxes for browsing on my phone, and it’s worked great for years until today when I got slugged with the 403 issue. Any and all searches triggers the 403. The app allows me to browse the site with a more mobile-friendly interface and I can’t imagine it’s using any more bandwidth than using a browser to view the site directly would. Am I just boned going forward?
I use an iPhone app called Anime Boxes for browsing on my phone, and it’s worked great for years until today when I got slugged with the 403 issue. Any and all searches triggers the 403. The app allows me to browse the site with a more mobile-friendly interface and I can’t imagine it’s using any more bandwidth than using a browser to view the site directly would. Am I just boned going forward?
Fixed. I added a workaround for the app for now, but in the future it should really set a custom user agent so that I can distinguish it from other apps.
I tried disabling all extensions and tracking protection but in the end it was my firefox hardening: network.http.referer.XOriginPolicy must be set to 0 (as opposed to 1 or 2).
Setting XOriginPolicy will break most sites that have anti-hotlinking protection, including for example Pixiv. So if you choose to enable it you should expect some sites to be broken. Nowadays browsers default to sending minimal information in the referrer header (just the domain name of the site you're coming from), so removing it has minimal privacy benefit.
evazion said:
Fixed. I added a workaround for the app for now, but in the future it should really set a custom user agent so that I can distinguish it from other apps.
Thank you for the speedy fix attempt, but unfortunately it doesn’t seem to have worked. (I’m also not the developer of the app, so I don’t have any influence over its internal implementation, I’m afraid.)
evazion said:
If you have a webapp that embeds Danbooru images, you should make sure it's not stripping the referrer. This could done either in a Referrer-Policy HTTP header, a
<meta name="referrer" content="no-referrer">tag, or on the <img> tag with<img referrerpolicy="no-referrer"/>. Stripping the referrer is usually not necessary because modern browsers only send the domain name anyway, not the full URL, so all we can see is which site is embedding the image, not the exact page on that site. For local webapps, all we'll see is that you're running it on localhost or maybe an internal IP like 192.168.0.1, which doesn't tell us much (and anyway, we already know your actual IP by virtue of you requesting the image).
Thanks for that tip. I'm running a webapp on localhost and removing that meta tag from my html file fixed the issue. Seems like that tag is a default when creating a new html file within VS Code.
Just for the record, the app is simply reloading images automatically every minute. It's not scraping images by itself - I have to manually interact with pictures to download them.
radredish said:
Thank you for the speedy fix attempt, but unfortunately it doesn’t seem to have worked. (I’m also not the developer of the app, so I don’t have any influence over its internal implementation, I’m afraid.)
It works for me in Android but I don't have an iPhone so I can't test it there. I can't tell why the app is being blocked because it doesn't identify its traffic. You'd have to point the developer at this thread to get them to take a look.
The BooruSlideshow extension by Chirmaya is no longer usable with Danbooru. As far as I know, it fetches a maximum of 100 images on the initial search, then fetches a few more every so often just to continue the slideshow, so I believe it would be low-traffic. I posted an issue on the extension's GitHub, but it's no longer being actively developed so I doubt the dev would see it any time soon. If there is a potential workaround I could implement on the user side I would greatly appreciate it. Also would love an automated slideshow mode on Danbooru itself...
Smiley1 said:
The BooruSlideshow extension by Chirmaya is no longer usable with Danbooru. […] If there is a potential workaround I could implement on the user side I would greatly appreciate it.
Try editing the extension’s rules_header_referer.json in an attempt to set your user agent, as required? You can try to replace the file’s contents with the snippet below, but make sure to write your own name instead of “YourNameHere”. I don’t use the extension, so this might not help at all. It may of course break if you also use Gelbooru or other ’Boorus.
[
{
"id": 1,
"priority": 1,
"action":
{
"type": "modifyHeaders",
"requestHeaders":
[
{
"header": "User-Agent",
"operation": "set",
"value": "Danbooru user YourNameHere"
}
]
},
"condition":
{
"urlFilter": "https://*.gelbooru.com/*",
"resourceTypes": ["main_frame"]
}
}
]kittey said:
Try editing the extension’s rules_header_referer.json in an attempt to set your user agent, as required? You can try to replace the file’s contents with the snippet below, but make sure to write your own name instead of “YourNameHere”. I don’t use the extension, so this might not help at all. It may of course break if you also use Gelbooru or other ’Boorus.
Show
[ { "id": 1, "priority": 1, "action": { "type": "modifyHeaders", "requestHeaders": [ { "header": "User-Agent", "operation": "set", "value": "Danbooru user YourNameHere" } ] }, "condition": { "urlFilter": "https://*.gelbooru.com/*", "resourceTypes": ["main_frame"] } } ]
Close! The Referer is what has to be set instead of the User-Agent, so in that file the header should be Referer.
Also the urlFilter should be https://*.donmai.us/* and resourceTypes should be ["image"].
And make sure to enable this ruleset in manifest.json by setting enabled to true under rule_resources.
The rule is that if a request sends the Sec-Fetch-Site: cross-origin header, and it doesn't send the Referer header, then it will be blocked. Browsers normally send both when loading an image from a different domain. There are other rules based on the user agent, but they all boil down to "don't pretend to be a browser when you're not".
For reference, I'm blocking around 250k to 300k images per day that are anonymously hotlinked from other sites. I don't know where it's coming from, but a lot of it is from China, so presumably there's some Chinese website out there hotlinking all of our images. This is why anonymous hotlinking is being blocked.
Unfortunately, the proposed solutions didn't work. When looking at the request headers in the browser Inspect menu, it appears that the requests for each image do not pass referer even after editing rules_header_referer.json and manifest.json. What it does pass is a user-agent disguised as a browser, even if I change "header" to "User-Agent". I don't know much about browser extension or javascript development, but I'm guessing that the header setup is being overwritten somewhere within the javascript itself. Looking at the javascript myself, it seems like sites_manager.js contains a commented-out section for building request headers, but even after un-commenting it and where it gets called and editing the headers within the function, the referer is still not being passed and images still return 403. At this point, I'm not sure what to do.
Smiley1 said:
Unfortunately, the proposed solutions didn't work. When looking at the request headers in the browser Inspect menu, it appears that the requests for each image do not pass referer even after editing rules_header_referer.json and manifest.json. What it does pass is a user-agent disguised as a browser, even if I change "header" to "User-Agent". I don't know much about browser extension or javascript development, but I'm guessing that the header setup is being overwritten somewhere within the javascript itself. Looking at the javascript myself, it seems like sites_manager.js contains a commented-out section for building request headers, but even after un-commenting it and where it gets called and editing the headers within the function, the referer is still not being passed and images still return 403. At this point, I'm not sure what to do.
Have you also adjusted the requestFilter part? Replace the Gelboru URL with "<all_urls>" so it matches adjusts the headers for all URLs (to start with). If that works, you can then be more specific and make it so it only applies to danbooru.
radredish said:
Thank you for the speedy fix attempt, but unfortunately it doesn’t seem to have worked. (I’m also not the developer of the app, so I don’t have any influence over its internal implementation, I’m afraid.)
The iPhone version of the app tends to be *behind* the Android version, because it has to go through Apple's ridiculously painful process of approval. Because of this, I would expect it to "not work" on iPhone if the developer made a difference in the app since then to account for the changes to Danbooru's handling of bot traffic.