Discussion of the download / upload tools

[andw]

Haven't been able to run the threaded versions, seem to end in a race / lock condition. The log window shows one worker for a while then the other with an increasing pause between them swapping before the gui becomes unresponsive. Memory usage goes from ~120MB to ~350MB and CPU usage drops to zero.

Currently using 1.17 although I'll have to change the 1000 post limit soon.

[bugsbunny80]

Many thanks for your fabulous software halvar! Is there a way to allow the downloading of single posts instead of the entire thread? Or someone knows a software for Internet Explorer allowing to download galleries on imgbox?

[halvar]

Originally Posted by andw

Haven't been able to run the threaded versions, seem to end in a race / lock condition. The log window shows one worker for a while then the other with an increasing pause between them swapping before the gui becomes unresponsive. Memory usage goes from ~120MB to ~350MB and CPU usage drops to zero.

Currently using 1.17 although I'll have to change the 1000 post limit soon.

tricky...

The GUI should not become unresponsive. I found one probable cause for this. The updating of the log view is not thread safe. This I can fix.

I currently have problems downloading from Imagebam. I get Connect timeouts. Relevant lines fro the log:

vef-imagerescue.log.0:2017-06-19 23:46:54 INFO: image-page: [ForkJoinPool-1-worker-1] http://www.imagebam.com/image/43330256412819 
vef-imagerescue.log.0:2017-06-19 23:49:02 SEVERE: Error downloading 'http://www.imagebam.com/image/43330256412819' 
org.apache.http.conn.HttpHostConnectException: Connect to 5.imagebam.com:80 [5.imagebam.com/199.58.85.103] failed: Connection timed out (Connection timed out)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)

Worker 1 tries to download 43330256412819 at 23:46:54. A connect is initiated but three minutes later at 23:49:02 the connect is aborted because of a timeout. There are no log entries from worker 1 in this period. This could match your description.

My plan is

• Fix the probable cause of GUI freezing
• Experiment with timeouts. There is seemingly a default of 3 minutes which is definitely too long. I am thinking more of 10 seconds.
• Make the parsing for posts more resilient (deleted posts problem).

I had no problems with multi threading until I got this Connect Exceptions. But I must admit that I usually do not use the GUI.

addendum: after setting the timeout to 10 seconds both workers log connection timeouts every 10 seconds. I will try again in a few hours.

[halvar]

Originally Posted by bugsbunny80

Is there a way to allow the downloading of single posts instead of the entire thread? Or someone knows a software for Internet Explorer allowing to download galleries on imgbox?

There is currently no way to load only a single post. You could get one page only though. That would get you 10 posts. You could throw the posts you don't need away.

I do not know anything about Internet explorer, I don't even have it.

[highwayman274]

Originally Posted by bugsbunny80

Many thanks for your fabulous software halvar! Is there a way to allow the downloading of single posts instead of the entire thread? Or someone knows a software for Internet Explorer allowing to download galleries on imgbox?

bugsbunny80, and other Members who are trying to save their galleries at imagebam & imgbox, may find the free version of Bulk Image Downloader useful. The free version will only download 100 images at a time, but I have used it to download larger galleries by capturing them in two separate "runs".

Originally Posted by highwayman274

I am not very IT savvy, but was quickly able to get my Google Chrome browser to work with Bulk Image Down-loader.


I used this method to save my galleries at imagebam and imgbox . . . and have also used it to save galleries posted by others at VEF.

Just go to the gallery view, press the little blue button, and let it rip!

[Al Gebra]

Originally Posted by halvar

I currently have problems downloading from Imagebam. I get Connect timeouts. Relevant lines fro the log:

(...) 
vef-imagerescue.log.0:2017-06-19 23:49:02 SEVERE: Error downloading 'http://www.imagebam.com/image/43330256412819' 
org.apache.http.conn.HttpHostConnectException: Connect to 5.imagebam.com:80 [5.imagebam.com/199.58.85.103] failed: Connection timed out (Connection timed out)

Worker 1 tries to download 43330256412819 at 23:46:54. A connect is initiated but three minutes later at 23:49:02 the connect is aborted because of a timeout. There are no log entries from worker 1 in this period. This could match your description.

Is it possible to limit the number of retries to get "timed out" and "Not found" pics? It seems that they are lost anyway. If you try to get them "manually" the page is loading endlessly - to no avail. So it is a waste of time waiting for them.

[deezer]

If the timeout occurs, it seems that the rescue tool, doesn't save the thumb instead of the image.

[Al Gebra]

Originally Posted by deezer

If the timeout occurs, it seems that the rescue tool, doesn't save the thumb instead of the image.

Using vef-imagerescue-1.21 some (original .jpg) thumbs (on ImageBam) were saved as .gif files (better than nothing!) example (original full pics seem to be dead already). Not complaining, just saying!

halvar, I'm still thankful that you provided your tool and are still working on improving it.

BTW, I find it honourable that IB is still maintaining their servers considering their doom soon.
On the other hand I noticed that many files once hosted there are no longer available anyway.

[deezer]

Originally Posted by Al Gebra

Using vef-imagerescue-1.21 some (original .jpg) thumbs (on ImageBam) were saved as .gif files (better than nothing!) example (original full pics seem to be dead already). Not complaining, just saying!

Yes, usually thumbs are saved as gif or jpg, if the full image is dead, but if the timeout occurs, unfortunately not as far as I have seen.

[bugsbunny80]

Originally Posted by highwayman274

bugsbunny80, and other Members who are trying to save their galleries at imagebam & imgbox, may find the free version of Bulk Image Downloader useful. The free version will only download 100 images at a time, but I have used it to download larger galleries by capturing them in two separate "runs".

Thanks for the suggestion highwayman! However, I uses Bulk but can't dl imgbox galleries. Maybe my version is too old...