Sacriven said:
We're also deleting md5 mismatch pics now? This isn't even a sample.
Yarr, looks like the script messed up
EDIT: Fixed now
Updated
Posted under General
Sacriven said:
We're also deleting md5 mismatch pics now? This isn't even a sample.
Yarr, looks like the script messed up
EDIT: Fixed now
Updated
Alright, anyways, just so no one gets confused -- I'll elaborate more on this. BrokenEagle98's script probably made a false positive, and I work fast through image samples so I assume from the getgo that his is correct without double-checking visually sometimes.
But let me note, again, that Twitter used a different image sampling method than they do now. Let's take post #1642836 for example. It's a Twitter sample, going by the link itself, but not the one we currently know Twitter samples by. Quick comparison:
File: sunny_milk_touhou_drawn_by_spirytus_tarou__1413787eb145c38f7febee495cd95f3e.jpg
CRC-32: a0158e4e
MD4: 4bc0e7c8509bbc6aa801d20a37fdcb7e
MD5: 1413787eb145c38f7febee495cd95f3e
SHA-1: f49037244fcbf7482434b63d46281119f03ed898
Sample at the time he uploaded it. The sample from the current link has this information:
File: BjPxhHJCMAEiAhH.jpg-large
CRC-32: 30920fba
MD4: 2f9b492d501c814414ae5a17d1fa1cff
MD5: bd8d460b18a0bfcbfad2deb6db77e653
SHA-1: 8dd84e804f45d21d594faabe2661c05d0fe845b1
This also counts for posts that are sampled, but are still the same resolution as their parent. Let's take post #1949989 and its parent:
File: kawashiro_nitori_touhou_drawn_by_maturiuta_sorato__5bc14be330d85ca045ede0cdbb2c4f08.jpg
CRC-32: 8d1781b2
MD4: 7a4e271d175be5d05e4dc52f8ffdc559
MD5: 5bc14be330d85ca045ede0cdbb2c4f08
SHA-1: 8ad963a411a590004f48e3dec61258164dbf5f29
Which is a sample, but different from the current sample below.
Current sample from this link
File: B9bwU1rCMAAwsgb.jpg-large
CRC-32: a9aa3d65
MD4: f22d435cf8ee0080c223d0d7a7117015
MD5: 2cb244fa234f233e1a63490b450ff73a
SHA-1: a67ecead58dcb3d433412fa93ea9a5fde9ee3ad6
And of course, the original image.
post #2669391 (post I replaced with)
File: kawashiro_nitori_touhou_drawn_by_maturiuta_sorato__587f1d2cefe4821bfb484262f9676050.jpg
CRC-32: 45986f34
MD4: cdea1f9b60a1447ebb36fcafe199d186
MD5: 587f1d2cefe4821bfb484262f9676050
SHA-1: a050ea5d13319664196f44ce2e2d8ed61d1e5418
If you're confused, don't be. Let's just say that Twitter's kind of stupid? And although they've tried improving on their sampling algorithm it still produces a bunch of generally crappy results for lower resolution images at near-zero benefits for bandwidth and a lot more cost for storage on their end.
The mistake here was that BrokenEagle98's script derped, and I didn't make a thorough visual comparison (did not note her pasties were removed). But we still consider images that are md5 mismatches from Twitter to be samples (especially if they have more blatant artifacting) as long as they are visually identical. The last part is important, and I hope I made note of that in twitter sample. If not, I'll revise it real quick.
Pinging relevant users @Sacriven @chinatsu although I think latter got the message in the discord general chat.
For more examples, you can look here:
CodeKyuubi said:
So what you're saying is, the old original had a different md5 than the current original, so the bot detected the new original's md5 and got an md5 mismatch with the old original's md5?
Originals stay the same for Twitter samples. This image is still the same as when it was posted -- nothing has changed. But what did change was this image, the sample. Previously it had an MD5 of 5bc14be330d85ca045ede0cdbb2c4f08 but currently, it has an MD5 of 2cb244fa234f233e1a63490b450ff73a.
I can do a diff comparison later if desired, but let's just say that whatever they did when they switched over to the new sampling algorithm is kinda weird, just to keep things short.
Mikaeri said:
Originals stay the same for Twitter samples. This image is still the same as when it was posted -- nothing has changed. But what did change was this image, the sample. Previously it had an MD5 of 5bc14be330d85ca045ede0cdbb2c4f08 but currently, it has an MD5 of 2cb244fa234f233e1a63490b450ff73a.
I can do a diff comparison later if desired, but let's just say that whatever they did when they switched over to the new sampling algorithm is kinda weird, just to keep things short.
But if the original was the image on Danbooru like what Sacriven posted, why was there a false positive when it only applies to samples?
CodeKyuubi said:
But if the original was the image on Danbooru like what Sacriven posted, why was there a false positive when it only applies to samples?
He was comparing the wrong image, that's why. The comment was deleted, but he was trying to compare the second image (one without pasties) to the one with the first. Of course it'd result in an md5 mismatch, since they're both different images with noticeably different visual features (again, the pasties). But he reran the script, and found that there was a match so the comment was removed.
It was just a derp on his end. Also partly my fault for not making the visual comparison (as again, I work fast), but I told him to rerun the script on all currently tagged md5 mismatches on Twitter in case there are any more false positives like that. If there are, they'll be edited accordingly and we can move from that.
Updated
I just created the PSD available and PDF available wikis for how to handle these nonuploadable image filetypes. Should there also be a forum topic to point users to if they have questions, and if so, what should that topic be, or do we need to create a new topic...?
<MIGRATING TO OWN TOPIC>
Provence. said:
If that's the case then those users should be told to stop.
Maybe. They could just be replacing the posts with notes, which as of yet aren't able to be replaced because of current limitations with the new feature. I'll double-check some of them now that I'm awake.
That, or they're misconstruing what md5 mismatch actually means outside of a Twitter use case. There are md5 mismatches out there that are bonafide image samples (from user modification or sampled/cropped from a 3rd party website, but these should be rare and in between, and would necessitate a full image replacement without using the feature instead.
Updated
pool #12319 It's this pool, it's not really either of those things. It appears the user just flagged every old post of each of those images as sample with no regards to the actual contents of the images themselves.
pool #3324 and this pool, without going through the entire queue it might be even more pools, in fact.
edit: wait this is a bot. This makes it even worse.
Log said:
pool #12319 It's this pool, it's not really either of those things. It appears the user just flagged every old post of each of those images as sample with no regards to the actual contents of the images themselves.
pool #3324 and this pool, without going through the entire queue it might be even more pools, in fact.
edit: wait this is a bot. This makes it even worse.
I'm confused -- what's this about a bot? Also, I double checked for some of these unsourced images and some of them are sampled from yandere (post #750087, child of that is also probably a sample).
Let me get in contact with the user... sec.
Special thanks to @chinatsu for pointing out that some Tumblr links contain the MD5 hash, which was able to remove a lot of the MD5 mismatches. I noticed these types of links mostly from the faulty 68.media.tumblr.com. I wonder if the MD5 hash links were implemented to overcome that faultiness...? Regardless, I'll be adding a comment to issue #2938 documenting this new discovery.
Discovered that the GET response headers for Tumblr images have a value called "Etag" that most of the time has a match to the MD5 hash of the image. After incorporating that into the scanning script, the following is the details of the rescan of MD5 mismatch.
Bump. Continuing discussion from topic #14156, without derailing the thread.
@reiyasona @RaisingK Do you guys think it'd be possible to make it a regular job to replace samples on your machines? I've added a new row in topic #14119 so we can consider automation for sample replacements, so approvers won't have to do it by hand a lot of the time.
Also pinging @BrokenEagle98 for ideas. Some things:
EDIT: Part of me wonders if we could offload the scripts to DanbooruBot so everything's fairly persistent, but let's see.
Updated
Mikaeri said:
Do you guys think it'd be possible to make it a regular job to replace samples on your machines? I've added a new row in topic #14119 so we can consider automation for sample replacements, so approvers won't have to do it by hand a lot of the time.
I already do that with pixiv samples, which I can verify sample-ness for directly, and I'm relying on BrokenEagle98's comments to verify the tumblr samples for me. I'm considering stopping the tumblr replacements once the backlog is cleared, though; I don't like automating something like replacements unless I know how to (and am willing to make the effort to) automate verification myself.
