Discussion of the download / upload tools - Page 12

bugsbunny80 · June 20th, 2017, 01:53 AM

...anyway, not only the shutdown news is frustrating but to make the thing worse a lot of galleries, while perfectly viewable as preview, are unreachable. The curious thing is the more recent posts seems to be more affected than older ones.
Some examples...
http://vintage-erotica-forum.com/sho...4&postcount=31
http://vintage-erotica-forum.com/sho...8&postcount=41
http://vintage-erotica-forum.com/sho...7&postcount=31

halvar · June 20th, 2017, 03:14 AM

Quote:

Originally Posted by deezer

If the timeout occurs, it seems that the rescue tool, doesn't save the thumb instead of the image.

Yes timeouts are hard errors where it is assumed that nothing is working and the download is aborted. Unlike errors with a response from the server like '404 file not found'. These indicate that the server is online and will serve future requests. If the server cannot be reached it does not make sense to continue. (in most cases)

And the retry should to handle temporary glitches.

halvar · June 20th, 2017, 03:27 AM

Currently I cannot download from imagebam anymore. Not only imagerescue is affected, but also Imagehost grabber.

When downloading the test thread http://vintage-erotica-forum.com/t333370-p1-x.html with IHG, only a few images can be downloaded. The others take a long time, 10 retries and fail.

I hope that it's only temporary. At this failure rate it does not make sense to do mass downloads. I will wait for the evening and then try again.

halvar · June 20th, 2017, 03:30 AM

Quote:

Originally Posted by Al Gebra

Using vef-imagerescue-1.21 some (original .jpg) thumbs (on ImageBam) were saved as .gif files (better than nothing!) example (original full pics seem to be dead already)

If the download of the image fails it is tried to download the thumbs. Thumbs are sometimes gifs, I just save them as I get them.

halvar · June 20th, 2017, 03:35 AM

Quote:

Originally Posted by Al Gebra

Is it possible to limit the number of retries to get "timed out" and "Not found" pics? It seems that they are lost anyway. If you try to get them "manually" the page is loading endlessly - to no avail. So it is a waste of time waiting for them.

Recently I also noticed that retries mostly fail. But it is hard to say it is alway better to not retry. Maybe tomorrow we will have recoverable errors where the retry succeed...

halvar · June 20th, 2017, 06:17 AM

It seems Imagebam is running better now, I could download the thread t121430-Girls_In_Pearls_General_Modern_Topics.

But imgbox seems to run better, here I see the two workers alternating, as one wouls expect:

Code:

2017-06-20 08:16:52 INFO: [ForkJoinPool-1-worker-1]image-page: http://imgbox.com/adns1lhD
2017-06-20 08:16:53 INFO: [ForkJoinPool-1-worker-1]downloaded: https://i.imgbox.com/adns1lhD.jpg to storage/t229284-Chubby_But_Not_BBW_General_Modern_Topics/t229284-p4-Chubby_But_Not_BBW_General_Modern_Topics-post-33-2420539/20081031-IMG_2583.jpg
2017-06-20 08:16:53 INFO: [ForkJoinPool-1-worker-1]image-page: http://imgbox.com/adqMtAnb
2017-06-20 08:16:53 INFO: [ForkJoinPool-1-worker-0]downloaded: https://i.imgbox.com/abqpEesm.jpg to storage/t229284-Chubby_But_Not_BBW_General_Modern_Topics/t229284-p4-Chubby_But_Not_BBW_General_Modern_Topics-post-32-2420537/Laura_f07.JPG
2017-06-20 08:16:53 INFO: [ForkJoinPool-1-worker-0]image-page: http://imgbox.com/abj4TFol
2017-06-20 08:16:53 INFO: [ForkJoinPool-1-worker-1]downloaded: https://i.imgbox.com/adqMtAnb.jpg to storage/t229284-Chubby_But_Not_BBW_General_Modern_Topics/t229284-p4-Chubby_But_Not_BBW_General_Modern_Topics-post-33-2420539/20081031-IMG_2582.jpg
2017-06-20 08:16:53 INFO: [ForkJoinPool-1-worker-1]image-page: http://imgbox.com/abbwANK8

halvar · June 20th, 2017, 06:55 AM

https://1fichier.com/?vrti7pxval

network timeout is set to 20 seconds
Worker name is stated in log statements
Updating GUI log view is now thread safe
Deadlock detection: If the GUI freezes because of a deadlock the statement 'DEADLOCK DETECTED' and diagnostic information is written in the log. It won't appear in the gui because of the deadlock. If this happens, please send me the log (last 1000 lines should suffice). It may take two minutes until a deadlock is detected.
parsing pages is more resilient, e.g deleted posts. If a post is skipped '...Ignoring Post...' is written to the log.

It seems that the success rate for imagebam varies from thread to thread or post to post.

I would suggest removing the automatic retry if it does not provide any value.
You can alway do a manual retry by deleting the empty dot files and rerunning the thread.
I usually do this after a couple of hours.

**andw** · June 20th, 2017, 09:26 AM

Many thanks for the info and updates.

Using v1.22 what would be the command line arguments to filter by poster and thread range.

I can see the object line as:

public VEFImageRescue(int threadId, String username, String password, String storagePath, String filterPoster, int pageFrom, int pageUpto)

however when instantiated it's:

new VEFImageRescue(Integer.parseInt(args[0]), args[1], args[2], "storage", null, from, to).start();

which I think means the storagePath defaults to relative path "storage" and filterPoster is ignored and set to null?

Wish I was an object-oriented programmer.

**effCup** · June 20th, 2017, 09:28 AM

I understand the appeal of not wanting your tool to hang around waiting for images to download when they cannot. More troubling to me, however, is the lack of notice that I, as user, get of these issues.

Code:

2017-06-20 20:38:35 INFO: Usage without GUI: java -jar vef-imagerescue.jar thread username password 
2017-06-20 20:39:50 INFO: 2 threads specified: [272029, 269539] 

[...]

2017-06-20 20:39:58 INFO: Starting Thread: 269539 
2017-06-20 20:39:59 INFO: HTTP/1.1 200 OK 
2017-06-20 20:40:00 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-20 20:40:00 INFO: Download first thread page: http://vintage-erotica-forum.com/t269539-p1-x.html 
2017-06-20 20:40:01 INFO: HTTP/1.1 200 OK 
2017-06-20 20:40:04 INFO: Thread name: Carmen_new_content_Mystery_Corner 
2017-06-20 20:40:04 INFO: Download forum page: http://vintage-erotica-forum.com/t269539-p1-x.html to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner.html 
2017-06-20 20:40:05 INFO: NO DEADLOCK 
2017-06-20 20:40:10 INFO: End of Thread reached - no next page link found -  
2017-06-20 20:40:10 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/dcbf03331549696 
2017-06-20 20:41:35 INFO: NO DEADLOCK 
2017-06-20 20:42:15 INFO: [ForkJoinPool-1-worker-1]downloaded: http://111.imagebam.com/download/I0u15wnqG1tuhVUz-24pbQ/33155/331549696/IMG_20140607_0001.jpg to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner-post-1-2930106\IMG_20140607_0001.jpg 
2017-06-20 20:42:15 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/500b8d331549812 
2017-06-20 20:43:05 INFO: NO DEADLOCK 
2017-06-20 20:44:34 INFO: [ForkJoinPool-1-worker-1]downloaded: http://111.imagebam.com/download/uD2N4wHSk_20aWSQzEOR9w/33155/331549812/IMG_20140607_0002.jpg to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner-post-1-2930106\IMG_20140607_0002.jpg 
2017-06-20 20:44:34 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/42f53a331549922 
2017-06-20 20:44:35 INFO: NO DEADLOCK 
2017-06-20 20:46:05 INFO: NO DEADLOCK 
2017-06-20 20:46:10 INFO: [ForkJoinPool-1-worker-1]downloaded: http://110.imagebam.com/download/fXp0GvgUNKChBUEVxY0hXA/33155/331549922/IMG_20140607_0003.jpg to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner-post-1-2930106\IMG_20140607_0003.jpg 
2017-06-20 20:46:10 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/ae0526331550076 
2017-06-20 20:46:31 SEVERE: [ForkJoinPool-1-worker-1] Error downloading 'http://www.imagebam.com/image/ae0526331550076' 
java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:264)
	at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
	at java.util.zip.InflaterInputStream.fill(Unknown Source)
	at java.util.zip.InflaterInputStream.read(Unknown Source)
	at java.util.zip.GZIPInputStream.read(Unknown Source)
	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70)
	at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
	at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
	at sun.nio.cs.StreamDecoder.read(Unknown Source)
	at java.io.InputStreamReader.read(Unknown Source)
	at java.io.Reader.read(Unknown Source)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:225)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:306)
	at vef.imgrescue.ImageBam.extractImageURL(ImageBam.java:25)
	at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:56)
	at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$8(ImageLinkProcessor.java:121)
	at java.util.ArrayList.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:105)
	at vef.imgrescue.ImageLinkProcessor.lambda$null$1(ImageLinkProcessor.java:56)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.TreeMap$EntrySpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$2(ImageLinkProcessor.java:42)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
	at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

2017-06-20 20:46:31 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/62bac8331550167 
2017-06-20 20:47:28 INFO: 1 threads specified: [269539] 
2017-06-20 20:47:28 INFO: Starting Thread: 269539 
2017-06-20 20:47:29 INFO: HTTP/1.1 200 OK 
2017-06-20 20:47:29 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-20 20:47:29 INFO: Using existing target folder :D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner 
2017-06-20 20:47:29 INFO: Download forum page: http://vintage-erotica-forum.com/t269539-p1-x.html to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner.html 
2017-06-20 20:47:30 INFO: End of Thread reached - no next page link found -  
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]image-page: http://www.imagebam.com/image/dcbf03331549696 
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]http://www.imagebam.com/image/dcbf03331549696 is already processed 
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]image-page: http://www.imagebam.com/image/500b8d331549812 
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]http://www.imagebam.com/image/500b8d331549812 is already processed 
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]image-page: http://www.imagebam.com/image/42f53a331549922 
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]http://www.imagebam.com/image/42f53a331549922 is already processed 
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]image-page: http://www.imagebam.com/image/ae0526331550076 
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]http://www.imagebam.com/image/ae0526331550076 is already processed 
2017-06-20 20:47:30 INFO: [ForkJoinPool-1-worker-0]image-page: http://www.imagebam.com/image/62bac8331550167 
2017-06-20 20:47:35 INFO: NO DEADLOCK 
2017-06-20 20:47:36 INFO: [ForkJoinPool-1-worker-1]downloaded: http://111.imagebam.com/download/CQ9gJ1IxenONoaU6N_ix-Q/33156/331550167/IMG_20140607_0005.jpg to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner-post-1-2930106\IMG_20140607_0005.jpg 
2017-06-20 20:47:36 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/e45d0d331550323 
2017-06-20 20:48:32 INFO: [ForkJoinPool-1-worker-0]downloaded: http://111.imagebam.com/download/CQ9gJ1IxenONoaU6N_ix-Q/33156/331550167/IMG_20140607_0005.jpg to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner-post-1-2930106\IMG_20140607_0005.jpg 
2017-06-20 20:48:32 INFO: [ForkJoinPool-1-worker-0]image-page: http://www.imagebam.com/image/e45d0d331550323 
2017-06-20 20:49:05 INFO: NO DEADLOCK 
2017-06-20 20:49:33 INFO: [ForkJoinPool-1-worker-1]downloaded: http://109.imagebam.com/download/aq9YOoslW_bxMZaSYDr6cw/33156/331550323/IMG_20140607_0006.jpg to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner-post-1-2930106\IMG_20140607_0006.jpg 
2017-06-20 20:49:33 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/0775c5331550426 
2017-06-20 20:50:35 INFO: NO DEADLOCK 
2017-06-20 20:50:52 INFO: [ForkJoinPool-1-worker-1]downloaded: http://111.imagebam.com/download/1cKND9lGeZDXMhIeZOd-fA/33156/331550426/IMG_20140607_0007.jpg to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner-post-1-2930106\IMG_20140607_0007.jpg 
2017-06-20 20:50:52 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/dee561331550512 
2017-06-20 20:50:55 INFO: [ForkJoinPool-1-worker-0]downloaded: http://109.imagebam.com/download/aq9YOoslW_bxMZaSYDr6cw/33156/331550323/IMG_20140607_0006.jpg to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner-post-1-2930106\IMG_20140607_0006.jpg 
2017-06-20 20:50:55 INFO: [ForkJoinPool-1-worker-0]image-page: http://www.imagebam.com/image/0775c5331550426 
2017-06-20 20:50:55 INFO: [ForkJoinPool-1-worker-0]http://www.imagebam.com/image/0775c5331550426 is already processed 
2017-06-20 20:50:55 INFO: [ForkJoinPool-1-worker-0]image-page: http://www.imagebam.com/image/dee561331550512 
2017-06-20 20:51:12 SEVERE: [ForkJoinPool-1-worker-1] Error downloading 'http://www.imagebam.com/image/dee561331550512' 
java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:206)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:189)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
	at java.util.zip.InflaterInputStream.fill(Unknown Source)
	at java.util.zip.InflaterInputStream.read(Unknown Source)
	at java.util.zip.GZIPInputStream.read(Unknown Source)
	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70)
	at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
	at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
	at sun.nio.cs.StreamDecoder.read(Unknown Source)
	at java.io.InputStreamReader.read(Unknown Source)
	at java.io.Reader.read(Unknown Source)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:225)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:306)
	at vef.imgrescue.ImageBam.extractImageURL(ImageBam.java:25)
	at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:56)
	at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$8(ImageLinkProcessor.java:121)
	at java.util.ArrayList.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:105)
	at vef.imgrescue.ImageLinkProcessor.lambda$null$1(ImageLinkProcessor.java:56)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.TreeMap$EntrySpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$2(ImageLinkProcessor.java:42)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
	at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

2017-06-20 20:51:12 INFO: [ForkJoinPool-1-worker-1]image-page: RETRY http://www.imagebam.com/image/ae0526331550076 
2017-06-20 20:51:33 INFO: 1 threads specified: [269539] 
2017-06-20 20:51:33 INFO: Starting Thread: 269539 
2017-06-20 20:51:34 INFO: HTTP/1.1 200 OK 
2017-06-20 20:51:34 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-20 20:51:34 INFO: Using existing target folder :D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner 
2017-06-20 20:51:34 INFO: Download forum page: http://vintage-erotica-forum.com/t269539-p1-x.html to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner.html 
2017-06-20 20:51:35 INFO: End of Thread reached - no next page link found -  
2017-06-20 20:51:43 INFO: 1 threads specified: [269539] 
2017-06-20 20:51:43 INFO: Starting Thread: 269539 
2017-06-20 20:51:44 INFO: HTTP/1.1 200 OK 
2017-06-20 20:51:44 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-20 20:51:44 INFO: Using existing target folder :D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner 
2017-06-20 20:51:44 INFO: Download forum page: http://vintage-erotica-forum.com/t269539-p1-x.html to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner.html 
2017-06-20 20:51:44 INFO: End of Thread reached - no next page link found -  
2017-06-20 20:51:53 INFO: Usage without GUI: java -jar vef-imagerescue.jar thread username password 
2017-06-20 20:52:10 INFO: 1 threads specified: [269539] 
2017-06-20 20:52:10 INFO: Starting Thread: 269539 
2017-06-20 20:52:11 INFO: HTTP/1.1 200 OK 
2017-06-20 20:52:11 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-20 20:52:11 INFO: Using existing target folder :D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner 
2017-06-20 20:52:11 INFO: Download forum page: http://vintage-erotica-forum.com/t269539-p1-x.html to D:\saved from vef\t269539-Carmen_new_content_Mystery_Corner\t269539-p1-Carmen_new_content_Mystery_Corner.html 
2017-06-20 20:52:12 INFO: End of Thread reached - no next page link found -  
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/dcbf03331549696 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]http://www.imagebam.com/image/dcbf03331549696 is already processed 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/500b8d331549812 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]http://www.imagebam.com/image/500b8d331549812 is already processed 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/42f53a331549922 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]http://www.imagebam.com/image/42f53a331549922 is already processed 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/ae0526331550076 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]http://www.imagebam.com/image/ae0526331550076 is already processed 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/62bac8331550167 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]http://www.imagebam.com/image/62bac8331550167 is already processed 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/e45d0d331550323 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]http://www.imagebam.com/image/e45d0d331550323 is already processed 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/0775c5331550426 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]http://www.imagebam.com/image/0775c5331550426 is already processed 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/dee561331550512 
2017-06-20 20:52:12 INFO: [ForkJoinPool-1-worker-1]http://www.imagebam.com/image/dee561331550512 is already processed 
2017-06-20 20:52:12 INFO: FINISHED! 
2017-06-20 20:52:12 INFO: Finished Thread: 269539 
2017-06-20 20:53:24 INFO: NO DEADLOCK 
2017-06-20 20:54:54 INFO: NO DEADLOCK

Code:

2017-06-20 21:02:12 INFO: Starting Thread: 307745 
2017-06-20 21:02:13 INFO: HTTP/1.1 200 OK 
2017-06-20 21:02:13 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-20 21:02:13 INFO: Download first thread page: http://vintage-erotica-forum.com/t307745-p1-x.html 
2017-06-20 21:02:14 INFO: HTTP/1.1 200 OK 
2017-06-20 21:02:14 INFO: Thread name: Mystery_Followup_Who_is_Maija_in_Scandinavian_Gir 
2017-06-20 21:02:14 INFO: Download forum page: http://vintage-erotica-forum.com/t307745-p1-x.html to D:\saved from vef\t307745-Mystery_Followup_Who_is_Maija_in_Scandinavian_Gir\t307745-p1-Mystery_Followup_Who_is_Maija_in_Scandinavian_Gir.html 
2017-06-20 21:02:15 INFO: End of Thread reached - no next page link found -  
2017-06-20 21:02:15 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/8670c8281124113 
2017-06-20 21:02:19 INFO: [ForkJoinPool-1-worker-1]downloaded: http://106.imagebam.com/download/1xTN4AZCFgt2jMST-bizEw/28113/281124113/newdoc482%20-%20Copy%20%282%29.jpg to D:\saved from vef\t307745-Mystery_Followup_Who_is_Maija_in_Scandinavian_Gir\t307745-p1-Mystery_Followup_Who_is_Maija_in_Scandinavian_Gir-post-1-2984985\newdoc482 - Copy (2).jpg 
2017-06-20 21:02:19 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/bed0d4281124155 
2017-06-20 21:02:24 INFO: NO DEADLOCK 
2017-06-20 21:02:39 SEVERE: [ForkJoinPool-1-worker-1] Error downloading 'http://www.imagebam.com/image/bed0d4281124155' 
java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:264)
	at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
	at java.util.zip.InflaterInputStream.fill(Unknown Source)
	at java.util.zip.InflaterInputStream.read(Unknown Source)
	at java.util.zip.GZIPInputStream.read(Unknown Source)
	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70)
	at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
	at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
	at sun.nio.cs.StreamDecoder.read(Unknown Source)
	at java.io.InputStreamReader.read(Unknown Source)
	at java.io.Reader.read(Unknown Source)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:225)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:306)
	at vef.imgrescue.ImageBam.extractImageURL(ImageBam.java:25)
	at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:56)
	at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$8(ImageLinkProcessor.java:121)
	at java.util.ArrayList.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:105)
	at vef.imgrescue.ImageLinkProcessor.lambda$null$1(ImageLinkProcessor.java:56)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.TreeMap$EntrySpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$2(ImageLinkProcessor.java:42)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
	at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

2017-06-20 21:02:39 INFO: [ForkJoinPool-1-worker-1]image-page: http://www.imagebam.com/image/3a7762281124166 
2017-06-20 21:03:00 SEVERE: [ForkJoinPool-1-worker-1] Error downloading 'http://www.imagebam.com/image/3a7762281124166' 
java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:264)
	at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
	at java.util.zip.InflaterInputStream.fill(Unknown Source)
	at java.util.zip.InflaterInputStream.read(Unknown Source)
	at java.util.zip.GZIPInputStream.read(Unknown Source)
	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70)
	at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
	at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
	at sun.nio.cs.StreamDecoder.read(Unknown Source)
	at java.io.InputStreamReader.read(Unknown Source)
	at java.io.Reader.read(Unknown Source)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:225)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:306)
	at vef.imgrescue.ImageBam.extractImageURL(ImageBam.java:25)
	at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:56)
	at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$8(ImageLinkProcessor.java:121)
	at java.util.ArrayList.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:105)
	at vef.imgrescue.ImageLinkProcessor.lambda$null$1(ImageLinkProcessor.java:56)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.TreeMap$EntrySpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$2(ImageLinkProcessor.java:42)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
	at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

2017-06-20 21:03:00 INFO: [ForkJoinPool-1-worker-1]image-page: RETRY http://www.imagebam.com/image/bed0d4281124155 
2017-06-20 21:03:20 SEVERE: [ForkJoinPool-1-worker-1] Error downloading 'http://www.imagebam.com/image/bed0d4281124155' 
java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:206)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:189)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
	at java.util.zip.InflaterInputStream.fill(Unknown Source)
	at java.util.zip.InflaterInputStream.read(Unknown Source)
	at java.util.zip.GZIPInputStream.read(Unknown Source)
	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70)
	at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
	at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
	at sun.nio.cs.StreamDecoder.read(Unknown Source)
	at java.io.InputStreamReader.read(Unknown Source)
	at java.io.Reader.read(Unknown Source)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:225)
	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:306)
	at vef.imgrescue.ImageBam.extractImageURL(ImageBam.java:25)
	at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:56)
	at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$8(ImageLinkProcessor.java:121)
	at java.util.ArrayList.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:105)
	at vef.imgrescue.ImageLinkProcessor.lambda$null$3(ImageLinkProcessor.java:62)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.HashMap$EntrySpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$4(ImageLinkProcessor.java:62)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
	at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

2017-06-20 21:03:20 INFO: [ForkJoinPool-1-worker-1]image-page: RETRY http://www.imagebam.com/image/3a7762281124166 
2017-06-20 21:03:28 INFO: [ForkJoinPool-1-worker-1]downloaded: http://102.imagebam.com/download/I_k7Q-q_b-eCralKtZrYyQ/28113/281124166/newdoc482%20-%20Copy%20%284%29.jpg to D:\saved from vef\t307745-Mystery_Followup_Who_is_Maija_in_Scandinavian_Gir\t307745-p1-Mystery_Followup_Who_is_Maija_in_Scandinavian_Gir-post-1-2984985\newdoc482 - Copy (4).jpg 
2017-06-20 21:03:28 INFO: FINISHED! 
2017-06-20 21:03:28 INFO: Finished Thread: 307745 
2017-06-20 21:03:54 INFO: NO DEADLOCK 
2017-06-20 21:05:24 INFO: NO DEADLOCK 
2017-06-20 21:06:54 INFO: NO DEADLOCK 
2017-06-20 21:08:24 INFO: NO DEADLOCK 
2017-06-20 21:09:54 INFO: NO DEADLOCK 
2017-06-20 21:11:24 INFO: NO DEADLOCK 
2017-06-20 21:12:54 INFO: NO DEADLOCK 
2017-06-20 21:14:24 INFO: NO DEADLOCK 
2017-06-20 21:15:54 INFO: NO DEADLOCK 
2017-06-20 21:17:24 INFO: NO DEADLOCK

The post in the first thread above, all the images are downloadable, but they're all very (tediously) slow to doanload, whether in browser, IHG, or your tool. Your tool "bailed out" on three of the images. I was able to get them manually, albeit it was so slow that one could see the rows of pixels slowly filling down the browser window one by one.

Likewise with the second thread above. Your tool "bailed out" on the second of three images in that post.

What troubles me, and maybe it's only me with my problem with "sticky" images, is that before I could tell your tool was having a problem with a given thread because its download had not completed. Above they were completed, so the only warning I have is:

a) scrolling back through the output window to check for errors
or b) comparing the no. of images downloaded to disk vs. those shown in the browser window for the post.

Both are tedious manual "checks" to have to perform, compared to the previous version of your tool.

For someone like deezer who is working through the many MIR queries (mostly very small threads), he's been able to chuck 20-50 threads at your tool and come back after, say, a half hour. The risk with v.1.22, at least in conditions like mine, are that there will be far more comparatively "silent" errors (not literally, because they're in the output window, but maybe more difficult to spot there?) that we humans won't easily catch?

Maybe it is only me, in which case others can/should carry on with 1.22 whereas I perhaps should revert to 1.21?

**effCup** · June 20th, 2017, 10:07 AM

A further stray thought: @halvar, does your tool do anything to handle image filename clashes? I'm afraid that some posters don't give their scans unique filenames, so there's the potential for 2+ different scans to have the same filename.

I haven't noticed any such issues so maybe you've got it already sussed. I hadn't thought of the problem until just now.

June 20th, 2017, 03:27 AM	#113
halvar Blocked! Join Date: Jan 2008 Location: HH Posts: 1,963 Thanks: 115,040 Thanked 32,801 Times in 1,955 Posts	Imagebam troubles Currently I cannot download from imagebam anymore. Not only imagerescue is affected, but also Imagehost grabber. When downloading the test thread http://vintage-erotica-forum.com/t333370-p1-x.html with IHG, only a few images can be downloaded. The others take a long time, 10 retries and fail. I hope that it's only temporary. At this failure rate it does not make sense to do mass downloads. I will wait for the evening and then try again.

June 20th, 2017, 06:55 AM	#117
halvar Blocked! Join Date: Jan 2008 Location: HH Posts: 1,963 Thanks: 115,040 Thanked 32,801 Times in 1,955 Posts	version 1.22 of imagerescue released https://1fichier.com/?vrti7pxval network timeout is set to 20 seconds Worker name is stated in log statements Updating GUI log view is now thread safe Deadlock detection: If the GUI freezes because of a deadlock the statement 'DEADLOCK DETECTED' and diagnostic information is written in the log. It won't appear in the gui because of the deadlock. If this happens, please send me the log (last 1000 lines should suffice). It may take two minutes until a deadlock is detected. parsing pages is more resilient, e.g deleted posts. If a post is skipped '...Ignoring Post...' is written to the log. It seems that the success rate for imagebam varies from thread to thread or post to post. I would suggest removing the automatic retry if it does not provide any value. You can alway do a manual retry by deleting the empty dot files and rerunning the thread. I usually do this after a couple of hours.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

June 20th, 2017, 01:53 AM	#111
bugsbunny80 Junior Member Join Date: Dec 2007 Posts: 9 Thanks: 602 Thanked 95 Times in 7 Posts	...anyway, not only the shutdown news is frustrating but to make the thing worse a lot of galleries, while perfectly viewable as preview, are unreachable. The curious thing is the more recent posts seems to be more affected than older ones. Some examples... http://vintage-erotica-forum.com/sho...4&postcount=31 http://vintage-erotica-forum.com/sho...8&postcount=41 http://vintage-erotica-forum.com/sho...7&postcount=31

June 20th, 2017, 09:26 AM	#118
andw Super Moderator Join Date: Mar 2009 Posts: 3,867 Thanks: 1,042,853 Thanked 52,316 Times in 3,824 Posts	Many thanks for the info and updates. Using v1.22 what would be the command line arguments to filter by poster and thread range. I can see the object line as: public VEFImageRescue(int threadId, String username, String password, String storagePath, String filterPoster, int pageFrom, int pageUpto) however when instantiated it's: new VEFImageRescue(Integer.parseInt(args[0]), args[1], args[2], "storage", null, from, to).start(); which I think means the storagePath defaults to relative path "storage" and filterPoster is ignored and set to null? Wish I was an object-oriented programmer.

June 20th, 2017, 10:07 AM	#120
effCup *Vintage Idiot* Join Date: Feb 2012 Location: History Posts: 22,129 Thanks: 226,701 Thanked 356,690 Times in 21,624 Posts	A further stray thought: @halvar, does your tool do anything to handle image filename clashes? I'm afraid that some posters don't give their scans unique filenames, so there's the potential for 2+ different scans to have the same filename. I haven't noticed any such issues so maybe you've got it already sussed. I hadn't thought of the problem until just now.