Discussion of the download / upload tools

[halvar]

Originally Posted by Al Gebra

It seems that some thumbnails are downloaded more than once. I got quite a lot of
"thumb_[file name]_1.gif/jpg" pics - all of them had different file names originally.

You are right. The reason for this is the retries. At the retry it cannot be determined if the thumb exists because of the original download attempt or because of two files with the same name.

Now that I am writing this: I could compare the files. Either a quick compare of the file size (exact number of bytes) or do a complete comparison byte for byte.

To get rid of the duplicates you already have: There are free tools out there that search for duplicate files. Filter the result for thumb_*_1.* and delete them.

And a minor request: Would it be possible to implement some kind of progress bar or the percentage of work done in your tool (xy % processed) or the like in the GUI? It seems that version 1.23 is working remarkably slower than prior versions - on my PC (Windows 10), at least - I filled only the mandatory fields, just one thread ID, no filter used. I simply wished to get an idea of how long it still takes until the download will be finished.

It would need two progress bars: one for the threads and one for the posts of the current thread in progress. The total number of files to download is not known, only the numbers of posts of the currently processed thread. And the duration is not predictable. It depends on how many timeouts you get. Since the tool is only a temporary thing I don't think I will add them.

You could look into the threads download folder to see how many pages are left with no existing page folder.

The new version is slower due to increased timeouts. (30 seconds instead of 20)

[halvar]

Originally Posted by effCup

Regarding sluggish image downloads from imagebam, I've seen (many times before, but again often yesterday & today), if one tries to load some images in a browser window there's a long delay with the only feedback being "transferring data from www.google-analytics.com".

google analytics links are not fetched by the tool, so the they should not be a problem.

Originally Posted by effCup

Probably it's not easy to do the re-checking/re-attempting while also controlling for possible duplicates--because the tool perhaps won't easily tell the difference between clashing filenames caused by duplicate downloads, as opposed to clashing filenames caused by duplicate filenames already linked/in the post. One could perhaps check filesize or maybe(?) some image properties like dimensions but...

I will do a byte for byte comparison. That should not take long (runtime wise) comparing to the time it takes to download.

[deezer]

Thanks a lot for implementing the thumbnail-download for images which gets a timeout.

About the duplication: better get the thumb twice, then nothing.

I will dl the new version and give it a try.

Thanks a lot for all your effort and for creating this great tool!

[deezer]

I tested now 1.23 and it works great.
I get all the images and the thumbs when there is an timeout.

[testuser4711]

Originally Posted by halvar

https://1fichier.com/?egemhhxsm6

• (deleted)
  

  2017-06-20 22:45:55 INFO: configuration value: rescue.request-timeout:30 
  2017-06-20 22:45:55 INFO: configuration value: rescue.connect-timeout:30 
  2017-06-20 22:45:55 INFO: configuration value: rescue.socket-timeout:30

Setting higher values results in slower processing because of waiting for connects that will never happen.
Setting lower values results in aborting downloads that would have worked but took too long.

These are the three timeout values the Apache HTTP components uses. The difference and meaning is not clear to me. I would tend to set a lower value to the connect timeout and a higher value to socket and request (20, 60, 60)

But I am not sure tweaking these values is efficient. I would rather loose one download that takes 60 seconds to complete than the 5 downloads I that would go through in the same time.

I think connect-timeout means the initial TCP handshake times out, request-timeout probably means you sent a http request, but it timed out and socket-timeout probaly means you are already reading data from the server, but the read times out.

About all of the timeouts/undownloadable pictures people seem to be getting when using the tool: You are totally at the "mercy" of the web servers that imagbam/imagebox run and how loaded they are at the moment of download. A web server might start serving a request and then gets starved for resources of one type or the other and simply abort the request. Or one might not be able to connect to the server full stop to begin with (that would be the connect-timeout mentioned above). I can very well imagine that problem to get worse as the end of june comes near as a) increasing numbers of people start downloading their files before they vanish, b) reliability is probably not foremost on the mind of the imagebam/box team right now, with employees looking for other jobs and routine maintanance on the backburner (in ten days all this will be moot anyway).

[effCup]

imgbox is currently unresponsive--thanks to deezer for analysis/spotting nature of problem.

Running halvar's tool over threads with imgbox links in them I think will produce "empty" dot files, i.e. containing no image filenames, so file size = 0.

You could simply decide to try them again later.

One way to "discover" where such files might lie is: on windows, use cmd to cd to the directory in which you've saved the folders output by halvar's tool, then run this (all one line):

forfiles /m .* /s /C "cmd /c if @fsize EQU 0 echo @path" > output.txt

It should produce a text file containing a listing of the paths of all such files, so you can re-check them later. Hopefully that may be less work than e.g. re-running halvar's tool?

edit: ha! Just after that imgbox seems to be back up & running.

Hmm, & the above also finds dot files of zero size that correspond to missing -bam files as well, which is less helpful.

[andw]

I've solved the difficulty experienced with the threaded versions of vef-image-rescue by moving from an "older" amd x4 computer to testing on a "newer" intel i5 computer (both win-7 64bit). Armed with that information I've "borrowed" an intel-based server and it's running four v1.23 cmd prompts at the moment......approx 1000 pages to go so hoping imagbam doesn't get flaky like imgbox.

[halvar]

Originally Posted by effCup

imgbox is currently unresponsive--thanks to deezer for analysis/spotting nature of problem.

Running halvar's tool over threads with imgbox links in them I think will produce "empty" dot files, i.e. containing no image filenames, so file size = 0.

You could simply decide to try them again later.

One way to "discover" where such files might lie is: on windows, use cmd to cd to the directory in which you've saved the folders output by halvar's tool, then run this (all one line):

forfiles /m .* /s /C "cmd /c if @fsize EQU 0 echo @path" > output.txt

It should produce a text file containing a listing of the paths of all such files, so you can re-check them later. Hopefully that may be less work than e.g. re-running halvar's tool?

edit: ha! Just after that imgbox seems to be back up & running.

Hmm, & the above also finds dot files of zero size that correspond to missing -bam files as well, which is less helpful.

You can also use the report to get an overview of missing files. It takes some time to generate the report and the resulting file is rather large ( up to 100MB or more). The report is a csv file containing:
threadId, threadTitle, pageNr, postCount, postNr, poster, thumbnailURL, imageUrl, imageFileName, imagefile size

imagefilename and imagefilesize are 'n/a' and -1 for failed downloads, e.g.

3164;Loni_Sanders_Classic_Pornstars;17;166;1481827;Curtis23;http://thumbnails39.imagebam.com/12068/857a3b120672517.jpg;http://www.imagebam.com/image/857a3b120672517;gfb42646.jpg;27399
3164;Loni_Sanders_Classic_Pornstars;17;166;1481827;Curtis23;http://thumbnails40.imagebam.com/12068/b63e56120672520.jpg;http://www.imagebam.com/image/b63e56120672520;n/a;-1

The report is generated from local files only, nothing is downloaded. It scans your whole storage folder. You can start the report using the button 'Create report of local files' on the second tab. The report generation cannot run in parallel with downloading. The report file is stored in the working folder.

[deezer]

Originally Posted by halvar

You can also use the report to get an overview of missing files. It takes some time to generate the report and the resulting file is rather large ( up to 100MB or more). The report is a csv file containing:
threadId, threadTitle, pageNr, postCount, postNr, poster, thumbnailURL, imageUrl, imageFileName, imagefile size

imagefilename and imagefilesize are 'n/a' and -1 for failed downloads

Wow, great.
I tried the report on a small batch that I downloaded and get an quick overview what is missing.
"n/a and -1" is also reported, if the image-download failed, even you saved the thumb, correct?
In my test, at least most times the thumbnail is saved for the reported (n/a; -1) files except one gallery where the images are deleted by the uploader.

[halvar]

Originally Posted by deezer

"n/a and -1" is also reported, if the image-download failed, even you saved the thumb, correct?

Yes, that is correct.