I think it'd be a good idea to add a compress/optimize option for PNG images when you upload.
It probably wouldn't be good for stuff originating from pixiv (I suspect it would result in a md5_mismatch), but it seems good when you upload from your computer. Especially when you're uploading high-quality PNGs from game_cgs.
I find myself optimizing them using tools such as OptiPNG and PNGOUT before uploading. And I think optimizing during the uploading process could simplify things more. After all, saving bandwidth is good, right?
I also think optimizing PNG compression is a good idea (I use pngcrush and AdvaceCOMP), but I don’t think it’s a good idea to recompress all PNG uploads on Danbooru.
Even if you only try a few known good parameter settings, PNG compression optimization is brute force number crunching, no matter how you look at it. A highres image will take several minutes to recompress even on a fast server. absurdres images probably take several tens of minutes and one incredibly_absurdres image could keep the server busy for an hour. Reducing the parameter search space reduces the time required, but also reduces effectivity.
I see two problems with this:
1. Server load: Performance is already an issue and even with severe limits on searches we’re still gettings timeouts from time to time. Pixiv source searches are pretty much entirely hopeless and subscription updates also tend to take their sweet time.
2. Upload feedback: Right now any image you upload will be available right away for tagging and duplicate detection. With recompression you’d have to wait several minutes and the site might find a duplicate after doing all the compression work. If the original image gets used temporarily after an upload and the compression is done in the background, you’d have to replace the image, hash and whatever, possibly leading to race conditions and you won’t be able to find the image by its original hash. So the problem with detecting duplicates stays unless you store the hashes of the original and recompressed images.
Users doing the optimization themselves before upload is probably the only feasible solution, but even that breaks the automatic duplicate detection.
Even if you only try a few known good parameter settings, PNG compression optimization is brute force number crunching, no matter how you look at it. A highres image will take several minutes to recompress even on a fast server. absurdres images probably take several tens of minutes and one incredibly_absurdres image could keep the server busy for an hour. Reducing the parameter search space reduces the time required, but also reduces effectivity.
I see two problems with this:
1. Server load: Performance is already an issue and even with severe limits on searches we’re still gettings timeouts from time to time. Pixiv source searches are pretty much entirely hopeless and subscription updates also tend to take their sweet time.
...
So I've tested some images with OptiPNG, and I believe that images that are highres or higher with an alpha channel that is not stripped will cause problems (expect it to last 2+ mins on default settings). It will help maybe if there were restrictions, like beyond a certain dimension don't run the optimizer.
I won't go into PNGOUT because while it does generate even smaller files, it's notorious for taking forever, especially on images with alpha channels.
Edit A way to manage server resources would be to place images in an "optimization queue" after uploading. Run only one instance of the compressor at a time, when an image is done compressing the next image in the queue gets optimized. You could search for pending optimization posts with "status:pendingopt" and would have a colored border, say, orange.
kittey said:
...store the hashes of the original and recompressed images.
Good idea. And an image with such hashes could be searched using "status:optimized".
