Discussion of the download / upload tools

[effCup]

Also:

2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-0] http://www.imagebam.com/image/18faf0214326512 
2017-06-18 03:38:19 INFO: http://imgbox.com/aakZS1OT is already processed 
2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/aab1fIDL 
2017-06-18 03:38:19 INFO: http://imgbox.com/aab1fIDL is already processed 
2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/yRKJx2mj 
2017-06-18 03:38:19 INFO: http://imgbox.com/yRKJx2mj is already processed 
2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/omwroqtK 
2017-06-18 03:38:19 INFO: http://imgbox.com/omwroqtK is already processed 
2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/CpNqokEZ 
2017-06-18 03:38:19 INFO: http://imgbox.com/CpNqokEZ is already processed 
2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/O3RVsO8H 
2017-06-18 03:38:19 INFO: http://imgbox.com/O3RVsO8H is already processed 
2017-06-18 03:38:19 SEVERE: null 
java.lang.NullPointerException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at java.util.concurrent.ForkJoinTask.getThrowableException(Unknown Source)
	at java.util.concurrent.ForkJoinTask.reportException(Unknown Source)
	at java.util.concurrent.ForkJoinTask.join(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.processForumPages(ImageLinkProcessor.java:59)
	at vef.imgrescue.VEFImageRescue.start(VEFImageRescue.java:100)
	at vef.imgrescue.Gui.lambda$null$6(Gui.java:248)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at java.util.concurrent.ForkJoinTask.getThrowableException(Unknown Source)
	at java.util.concurrent.ForkJoinTask.reportException(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$2(ImageLinkProcessor.java:42)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
	at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException
	at vef.imgrescue.Gui$1.publish(Gui.java:51)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.info(Unknown Source)
	at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:38)
	at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$7(ImageLinkProcessor.java:119)
	at java.util.ArrayList.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:103)
	at vef.imgrescue.ImageLinkProcessor.lambda$null$1(ImageLinkProcessor.java:56)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.TreeMap$EntrySpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	... 4 more

& as before, closing the app. & re-running on that thread encountered no errors & finished the thread.

NB, I'm posting these errors for halvar not because they're necessarily very serious or frequent (they're not); just because he might be interested.

[halvar]

Originally Posted by effCup

First sighting of any Java errors?

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.rangeCheck(Unknown Source)
	at java.util.ArrayList.get(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$getPosts$9(ImageLinkProcessor.java:149)

I have seen this once and deezer reported the same.

This happens when the post nr cannot be determined. I am looking for the link to report a post '..report.php?p=nnn' to extract the post number. If the link is not found - bam!

It may be that the html file was corrupt. Because when you retry the last html-page is downloaded again. So it works the scond time.
Fetching information from arbitrary portions of the html is error prone. Sometimes a page is rendered differently just because the user has different settings and the thing breaks. Given more time one would select the elements to get the information from more carefully.

If this happens again, please send me the html files before you retry.

[halvar]

Originally Posted by effCup

Also:

Caused by: java.lang.NullPointerException
	at vef.imgrescue.Gui$1.publish(Gui.java:51)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)

This is really curious, I never thought this could happen. In this case the log text cannot be written in the log field because there is not text. There should always be text or at least an empty string. I will add a check to prevent this.

& as before, closing the app. & re-running on that thread encountered no errors & finished the thread.

NB, I'm posting these errors for halvar not because they're necessarily very serious or frequent (they're not); just because he might be interested.

Thanks for the reports, they help to improve the tool. We should think about setting up a bug tracker. The

[effCup]

Hmm, only noticed later that the "top" of that log snippet was missing/cropped. I'd simply copied all & pasted from the gui window. Let me know if you want to see more/the top/start of that instance, as I should be able to retrieve it from your log file?

----

edit: Hmm. Running the tool on this thread, it has stalled at this point:

2017-06-18 04:10:45 INFO: 2 threads specified: [279125, 276115] 
2017-06-18 04:10:45 INFO: Starting Thread: 279125 
2017-06-18 04:10:46 INFO: HTTP/1.1 200 OK 
2017-06-18 04:10:46 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-18 04:10:46 INFO: Download first thread page: http://vintage-erotica-forum.com/t279125-p1-x.html 
2017-06-18 04:10:47 INFO: HTTP/1.1 200 OK 
2017-06-18 04:10:47 INFO: Thread name: Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov 
2017-06-18 04:10:47 INFO: Download forum page: http://vintage-erotica-forum.com/t279125-p1-x.html to D:\saved from vef\t279125-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov\t279125-p1-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov.html 
2017-06-18 04:10:48 INFO: End of Thread reached - no next page link found -  
2017-06-18 04:10:48 INFO: image-page: [ForkJoinPool-1-worker-0] http://www.imagebam.com/image/9d10cc175732916 
2017-06-18 04:10:52 INFO: downloaded: http://60.imagebam.com/download/w3FvcZ3Piu_OFcrAmNX6Uw/17574/175732916/Hustler%20Magazine%20April%201982%20035.jpg to D:\saved from vef\t279125-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov\t279125-p1-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov-post-1-3115074\Hustler Magazine April 1982 035.jpg 
2017-06-18 04:10:52 INFO: image-page: [ForkJoinPool-1-worker-0] http://www.imagebam.com/image/2fb172175732953 
2017-06-18 04:10:53 SEVERE: download:  http://25.imagebam.com/download/rqE15VsmXWdS40oDBbf2CA/17574/175732953/Hustler%20Magazine%20April%201982%20036.jpg failed HTTP/1.1 404 Not Found 
2017-06-18 04:10:54 INFO: downloaded: http://thumbnails25.imagebam.com/17574/2fb172175732953.jpg to D:\saved from vef\t279125-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov\t279125-p1-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov-post-1-3115074\thumb_Hustler Magazine April 1982 036.jpg 
2017-06-18 04:10:54 INFO: image-page: [Thread-6] RETRY http://www.imagebam.com/image/2fb172175732953

That last line is the url of the second quoted image in that thread's first post.

If I look in the storage folder, the second image has been processed (as a thumb, because it's a dead image). This was not a re-run over a partially-completed thread.

Maybe it's just a coincidence, but I'd have thought the tool wouldn't stop at a point where it (seems it) should have already finished with that url/image file.

I'd expect (& have often seen) it instead stopping/pausing at the next thumb-link/url that has yet to be processed.

This thread should be "finished", but the tool has not moved on to the second thread in the queue--there's no folder been created for it.

(only now) Stopping the tool & re-starting it it does finish with that thread & move on to process the next one in the queue.

[halvar]

Originally Posted by effCup;4070862
If I look in the storage folder, the second image [I

has[/I] been processed (as a thumb, because it's a dead image). This was not a re-run over a partially-completed thread.

Maybe it's just a coincidence, but I'd have thought the tool wouldn't stop at a point where it (seems it) should have already finished with that url/image file.

I'd expect (& have often seen) it instead stopping/pausing at the next thumb-link/url that has yet to be processed.

This thread should be "finished", but the tool has not moved on to the second thread in the queue--there's no folder been created for it.

(only now) Stopping the tool & re-starting it it does finish with that thread & move on to process the next one in the queue.

There are two server calls between the first log statements 'image-page...' and 'downloaded ... to ...
If a download stalls you see the 'image-page...' but not the 'downloaded to'. How long do you usually wait before you act? I would give it at least 30 seconds. Usually it should be done in a second, but it sometimes just take longer. The timestamps in the log show how long it usually takes (time between 'image-page' and 'downloaded to'.

The word RETRY in the log statement means that this is a retry for a previously failed attempt. It could be that retries take longer than normal requests because of the reason of the fail.

The log is cropped in the because only the last part is relevant. If you want to check for a longer time period you can use the files. The log files are limited to a total of 50MB.

[deezer]

I warmly recommend to use halvar's tool. It's working great.

Not only for downloading and saving, but also for organize the work in a section!

Working in the MIR-section there are a thousands of threads, most of them very short. But we must be sure to get them all.
effCup compiled a list of the threads using halvar's tool. Copying 20 threads-numbers, paste them into the tool and let it do the work.

With the list effCup created, the work is absolutely well organized.
If I'm done with a batch I note this in the list, so nobody else will download this threads, which avoids double work.

I have downloaded now 85 threads of the ordinary queries and the tool (v1.16) works very fine.
Only once I faced minor problem.

When the tool finished with a batch, I take a quick look into the log, if a error occured.
If so, I check the thread manually again, but it happened only once until now.

I will store the threads in batches of around 500 MB at a filehost.
The re-hosting and editing of each thread can be done also after 30.06.2017!
But saving must be done before end of June.

The best thing is, while the tool is running I can work in the box and re-host the images and edit posts.

I strongly suggest, to use this tool. Not only for downloading, but also for organize the work in a section with the thread-generator!

The work in MIR is now organized, like effCup here and here and Al Gebra here suggested. And it's going extremely well.

[Al Gebra]

Originally Posted by Al Gebra

I admit that my request is not 1st priority, but maybe it can be granted with just a little effort. Is it possible to implement an option in vef-imagerescue that allows to download only a limited number of pages in a certain thread, say page 15 to 34?

A similar option is possible in ImageHost Grabber (Aktuellen Tread saugen ~ download current thread [or the like in English]). When you use that option the tool asks for the first page to d/l and then the last page. It even works with the individual "Number of Posts to Show Per Page" I have chosen in my VEF Control Panel (= 10 posts).

I just tested vef-imagerescue-1.19, and it did what I requested, thanks halvar!

On this occasion I noticed that many links to my pics I just recently uploaded to ImageBam are already dead, example (see also my following posts there). Only the thumbs (preview pics) are still visible/available. And vef-imagerescue-1.19 saved these at least.

[effCup]

Originally Posted by halvar

I am looking for the link to report a post '..report.php?p=nnn' to extract the post number. If the link is not found - bam!

It may be that the html file was corrupt.

If this happens again, please send me the html files before you retry.

2017-06-18 17:20:01 INFO: 2 threads specified: [268142, 268227] 
2017-06-18 17:20:01 INFO: Starting Thread: 268142 
2017-06-18 17:20:03 INFO: HTTP/1.1 200 OK 
2017-06-18 17:20:03 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-18 17:20:03 INFO: Using existing target folder :D:\saved from vef\t268142-Mystery_Followup_Vintage_blonde_in_ripped_dress_M 
2017-06-18 17:20:03 INFO: Download forum page: http://vintage-erotica-forum.com/t268142-p1-x.html to D:\saved from vef\t268142-Mystery_Followup_Vintage_blonde_in_ripped_dress_M\t268142-p1-Mystery_Followup_Vintage_blonde_in_ripped_dress_M.html 
2017-06-18 17:20:06 INFO: End of Thread reached - no next page link found -  
2017-06-18 17:20:06 SEVERE: Index: 0, Size: 0 
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.rangeCheck(Unknown Source)
	at java.util.ArrayList.get(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$getPosts$9(ImageLinkProcessor.java:149)
	at java.util.stream.Collectors.lambda$toMap$58(Unknown Source)
	at java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source)
	at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.collect(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.getPosts(ImageLinkProcessor.java:146)
	at vef.imgrescue.ImageLinkProcessor.processForumPages(ImageLinkProcessor.java:39)
	at vef.imgrescue.VEFImageRescue.start(VEFImageRescue.java:100)
	at vef.imgrescue.Gui.lambda$null$6(Gui.java:248)
	at java.lang.Thread.run(Unknown Source)

Saved files/folder output here, before any re-starts.

The html tags seem "whole", but you're right, the report button(s) are missing from that page's source.

the post no. appears in several places (in a complete page, I mean), I think all buttons:

reply buttons
showpost buttons
report buttons (what you're currently using)
edit buttons

Sorry, I haven't really studied the html structure.

[halvar]

Originally Posted by effCup

The html tags seem "whole", but you're right, the report button(s) are missing from that page's source.

the post no. appears in several places (in a complete page, I mean), I think all buttons:

reply buttons
showpost buttons
report buttons (what you're currently using)
edit buttons

Thanks for the file. It seems that the login failed or got lost. The file contains text 'Register on the forum now ...' that is shown if you are not logged in. And if are not logged in you cannot report posts so the report post links are missing.

In cases of wrong username/password this error should not occur.
If a login fails you should see log text like this and the process is aborted.

2017-06-18 09:35:48 INFO: Starting Thread: 718
2017-06-18 09:35:49 INFO: HTTP/1.1 200 OK
2017-06-18 09:35:49 SEVERE: IDstack cookie NOT found. NOT logged in to VEF. Aborting
2017-06-18 09:35:49 INFO: Finished Thread: 718

But it can also be that the login is lost after it initially succeeded. The login is kept in a session cookie. Sometimes these cookies get lost. This could be for various reasons and also happens sometimes in browsers.

I could change it to get the post number from the other links you mentioned. But I only want to do only really necessary changes now to avoid introducing new bugs. And besides I do not know if you really get the pages if you are not logged in.

[deezer]

Don't know, if you saw this problem yet:

An error occured with this post:
http://vintage-erotica-forum.com/showpost.php?p=3711213

2017-06-18 12:41:03 SCHWERWIEGEND: Error downloading 'http://www.imagebam.com/gallery/cpnazhtdpdpc3lht4u54c0icw6f3rp54' 
org.apache.http.client.ClientProtocolException
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at vef.imgrescue.AbstactImageHost.downloadFile(AbstactImageHost.java:89)
	at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:66)
	at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$7(ImageLinkProcessor.java:119)
	at java.util.ArrayList.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:103)
	at vef.imgrescue.ImageLinkProcessor.lambda$null$1(ImageLinkProcessor.java:56)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.TreeMap$EntrySpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinPool.helpComplete(Unknown Source)
	at java.util.concurrent.ForkJoinPool.awaitJoin(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$2(ImageLinkProcessor.java:42)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
	at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: org.apache.http.ProtocolException: Target host is not specified
	at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71)
	at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	... 29 more

The poster set a thumbnail, which directs not to the full image, but to an imagebam gallery:-)

[URL=http://www.imagebam.com/gallery/cpnazhtdpdpc3lht4u54c0icw6f3rp54][IMG]http://thumbnails116.imagebam.com/49580/eb8796495795513.jpg[/IMG][/URL]

I will dl the gallery now with IHG. If you need the html-file, please let me know.

But I assume, this isn't a bug in the tool. It's more a very similar situation as with direct linked galleries, which we yet discussed here.


Btw, the tool saved properly the image which is linked in the signature of Gladys Allova.