Deleting Duplicates

[Misrule]

For finding duplicate images, I use VisiPics. It's a great little program and will crunch through thousands of files identifying the ones that are identical. It can also be set to identify images that are similar and many other options.
And, once it has identified the duplicates, you can go through them and decide which one(s) to delete and which one to keep. All within the program. Thoroughly recommend it.

[DTravel]

I use SimilarImages, have been for many years. I'm not sure where you could get a copy since I think the programmer isn't working on it anymore. I think you can find info on it in the Useful Programs thread.

[Al Gebra]

Originally Posted by karloschaos

DupDetector

Highly recommended by me too.

[electile disfunction]

I have been using Anti-Dupl.net for quite some time now.

It makes and stores databases of your collection(s), finds similarities & differences, remembers false positives, is very fast, etc.
http://antidupl.sourceforge.net/english/index.html

e.d.

P.S.: It is free & open source, too.

[Artsey]

Just want to thank the people who suggested software. I have checked some out. Using dupe detector reduces about 10% of the task. They all still have a quality control problem in not being able to zoom in to compare the quality of two similar images. While they do help in the detection and locating many dupes I still have to use the method described earlier to directly compare the images for quality. I can't just assume a small image size is no good. A larger version may be just a resampled re-jpeg enlargement of the smaller image. I've found that many times. Likewise a larger file size is not assumed to be better. I've found large file size images that were merely due to re-jpeg of a previous jpeg or the result of heavy flaws like moire patterns and excessive sharpening. I've even foung images that were poor quality but the same image dimensions, name and file size as another better quality that I assume to be an original. As if someone posted degraded images intentionaly trying to mislead a sort leaving one with a poor quality image. Not images I got here but elsewhere.

If anyone knows of a comparison software that allows for zoom in that would help a lot.

[electile disfunction]

Originally Posted by Artsey

Just want to thank the people who suggested software. I have checked some out. Using dupe detector reduces about 10% of the task. They all still have a quality control problem in not being able to zoom in to compare the quality of two similar images. While they do help in the detection and locating many dupes I still have to use the method described earlier to directly compare the images for quality. I can't just assume a small image size is no good. A larger version may be just a resampled re-jpeg enlargement of the smaller image. I've found that many times. Likewise a larger file size is not assumed to be better. I've found large file size images that were merely due to re-jpeg of a previous jpeg or the result of heavy flaws like moire patterns and excessive sharpening. I've even foung images that were poor quality but the same image dimensions, name and file size as another better quality that I assume to be an original. As if someone posted degraded images intentionaly trying to mislead a sort leaving one with a poor quality image. Not images I got here but elsewhere.

If anyone knows of a comparison software that allows for zoom in that would help a lot.

Have you tried two (or more) dedicated, small, but excellent pieces of software used together, or do you want only a single program with compromises in seach and graphic quality for some reason?

Anti-Dupl is a stand-alone program but relies on whatever default viewing software you use for closer inspection. Example: When needed, I just double-click on the preview in AntiDupl and the full pic with all graphic data instantly opens in Irfanview (which you can close by pressing the [Esc] key just once!). This combination of A-D & Irfanview is extremely fast and powerful for finding the quality problems you outlined above. (99%+ of all the image files I download are poor, degraded, misleading images and I need to delete about 200 to 4000 of them every day!)

The programmer of A-D states outright that A-D is only for comparing thousands of images (even across many directories and and drives) and then listing possible duplicates for you to consider. Adding a redundant viewer software would take his time and effort away from creating good and fast comparison software.

Do you need something more?

e.d.

P.S.: I regularly search and compare graphic images individually as large as 140 MB, and neither A-D nor Irfanview has ever had problems with them.

[electile disfunction]

I have been considering your entire issue about finding and deleting duplicates. Mrs e.d. & I are in a very similar situation to you as we have hundreds of thousands of picture files, spread over many broad categories, collected over years.

I'll give you (and everyone reading) a brief outline of how we sort our files, and how we look for duplicate & similar pictures. This might give you a few ideas to get a good start at what you need and what will work for you!

In our collection we have: known duplicates which we want to keep for various reasons; many similar pictures--some to keep some to delete; pictures that are radically different but computers see as similar; and additions & deletions almost constantly. Files range in size from the smallest computer icons of 10s of bytes to deep space astronomy photographic tifs larger than 150 MB each. Picture subject and sort categories range from a small nephew making rude noises at his sister, to works of incredibly complex but sublime digital art--there is no "easy" way to sort our collection ... trust me, I've tried.

Tools:
You need some good but basic software to keep your sanity. All of it you can get and use for free, at some time you may wish to purchase some (and/or make donations to the creators).

a) Font--if you have any respect for your sanity, choose a well designed, sans serif, professional font that has all the letters and punctuation you use and (most importantly!!!) different glyphs for each of these characters:
1 i I L O 0 o F f $ S s T t W V U w v u
You need this to see thousands of file names easily and constantly. Use this font either universally on your computer or just set it up in your favourite file sorting program.
Every operating system comes with at least one (this is how programmers can see what they are typing): try Segoe UI, Trebuchet, or Tahoma and use them compare to others if you wish. (Note: Trebuchet is used by many hard-of-seeing people for all their computer and print needs.)

b) Graphic file viewer--it needs to load quickly and display accurately: jpg, gif, png, tif files and maybe others. It is best if you can have many instances of it running simultaneously (for visually comparing similar pictures). Irfanview is awesome, ACDsee used to do this (I haven't looked at it in a decade). The programs that come with Windows are mostly useless for this as they are slow, resource hogging, and try so hard to be "helpful" you don't know if what you are seeing is real or not most of the time.
You will likely want a program that allows you to quickly find the metadata and exif info of your picture files.

c1) A program to find duplicate computer files, regardless of file name; and
c2) A program to find similar picture files, that creates & reuses it's own databases.
There are hundreds of programs to find duplicate computer files spread of many drives. Find one you like; if it works with--or is part of--your favourite computer file sort & searcher (like Windows Explorer, etc.) so much the better. Before any visual-type search of new files, use this to find the obvious duplicates as programs are almost always infinitely faster that any graphic-search software.
Programs to find similar picture files are much more difficult to find--VEF member have been looking for and testing them for years with widely varying results. Many have been suggested for you in this thread already. I have found Anti-Dupl.net is very good and fast, it's also free of charge and reliable, and it works on our (3) computers without fussing with the OS, but YMMV.

d) (optional) Computer file renamer: you may not wish to fully rename any files but adding a couple words at the beginning of file names can infinitely cut down on your work. Examples: almost all of our visual artists have created self portraits--I use a renamer to append "Autoportrait_" to the beginning of those files' names ... Therefore it is easy to remove, the original file names remain intact, the files are easy to seach for and separate from the other works. Groups of pictures (from many different scanners or sources) will always be together, and any additional or new files can easily join the group with "copy and paste" commands.

e) (optional) Software to check csv files for known collections (if you need this you already know what it is). VEF posting Help Guides probably have suggestions for these, and they get reviewed/recommend regularly in other VEF threads.

Process:
1) Sort your collection of pics any way you wish. Our hierarchy usually includes these levels of folders (or more depending on many variables):
i. general subject (examples: our photos, animals, art, astronomy, hair styles, info.); then
ii. creator/artist/souce (examples: vacations, mammalia; Leonardo Da Vinci, NASA, Helmut Newton (photographer), wife's hair); then
iii. subsection(s) (examples: camping 2011, aquatic, architecture, solar system, photshoot name, long styles).
Also at thesubsection level we usually start including appropriate "miscellaneous" directories--examples: in nudes we sort whatever pictures remain by the models' surnames, but in astronomy we sort leftovers by wavelength and/or date.

2) Find any duplicate files. Keep or destroy then as necessary.
NEVER tell your programs to delete any file automatically unless YOU know exactly what you are doing!

3) Tell your "similar picture finder" software where it should look for and compare picture files. Sometimes I seach individual directories or drives, but usually (for nudes especially) I'll ask the computer to compare all new files to all known files on a regular basis--it will ignore mistakes it made in the past and add any new images to its databases as necessary.

4) You should be able to sort the "similar picture finder" results alphabetically and/or by directory and/or by % similarity and/or by type of difference so that your 'known & sorted' file names occur together and any new resultant names are easy-to-see beside them. Open any pictures you need to see for comparisons, tell your softwares of the results (Anti-Dupl continually deletes files, and updates filenames & directories if you change them, for example), and go on to the next results.

5) Once your "similar picture finder" has its initial database(s) and file of erroneous matches to ignore, all subsequent duplicate searches wil be much faster. (Right now, we can search 100,000 files over three hard drives in 3 or 4 minutes, where the initial database creation and search may have taken 20 minutes.)

6) Your intial review of the results found by your programs will take longer than any subsequent search, too but you will remove a lot of junk that first time. And you'll learn a lot, too!

That's the overview, I hope it gives you some ideas for what you want and need for your files. I will add corrections and other ideas as they occur to me.

Please feel free to send me a Private Message if you think I can help with something.

e.d.

[Artsey]

Want to thank everybody who contributed sorting ideas. That has increased my computer assisted dupe detection to about 20 to 30% of the dupes. Still, major differences in dupes prevents computer assistance from detecting them. Different color, cropping, scan pattern effects etc. causes it.

Working my files I have developed my criteria sort strategy to further speed up and improve the reliability of the dupe detection. My confidence is high that I get 100% of the dupes and with my newest strategies get the sort task to speeds that rival computer assisted detction. I have to time it sometime.

Anyway the additional strategies ar that after the innitial criteria sort that may reduce the size of a criteria, bikini/water/standing for example, to 150 to 200 images can be broken down into images with arms up leg up hat or head wear. Easy to see in thumbnail and geting a criteria down to 30 to 40 images to sort speeds it up hugely. It's a mathematical factoral thing. To explain; when looking at a folder with 200 bikini/water/standing images I look at the first image and scan the folder looking at 199 other images. Then the second image I have to scan 198 of them again. the third 197 etc. But if I get a clearly defined criteria, like bikini/water/standing/arms up, with 25 images I have reduced the scan time to an eighth of the time. Plus I easiliy recognize many images that don't have dupes so don't even have to scan the whole folder for many.

An example of the time demanded of the new strategy. I just sorted my miscilaneous movie star folder. It had 20,000 images. Deleted 15,000, spent about an hour a day for maybe a week and a half, and have huge confidence I got 100% of the unwanted dupes. In addition to the reduction of time demand it has reduced the fatigue and mental concentration demanded. It actually made it a moderately enjoyable task.

[Artsey]

Just thought I'd drop in to mention a new dupe deletion issue I recently discovered. This is a cross folder dupe issue. When I didn't have many images years ago I scanned my sorted folders to find, for instance pics of latex clad women to build up my latex folder. Now I have lots of latex images, and it's time to pare down that miscilaneous latex folder. Aside from sorting into latex/bikini or latex/swim or latex catsuit or latex strappy I move images that I used to build up the folder back to the sorted model folders and often find that I have a dupe in that folder. That's from the years ago build up of the latex folder from the model folder. So I can go ahead and delete those images from the latex folder. I have plenty of miscilaneous images in my latex foler these days, and I also often have the whole shoot of that model's latex set where I may have only had a couple back then. I arrange the latex folder by date reverse so new images are at the top, and the older images are at the bottom of the list. The older images at the bottom are the ones I was throwing into the latex folder to build it up way back then.