Discussion of the download / upload tools

[effCup]

Originally Posted by halvar

If you mean the link to the gallery: It is not that easy. I would have to check:

• is a linked page an image page or album page
• if it is an album find zip if available or parse page to get images

Parsing the page would add another level of indirection. I'm sorry, I don't think I can do this.

Don't worry. I think all of my such galleries are already saved/should be re-uppable. A few other members have made similar posts but not many, so it's not a big issue.

[temptester666]

Thanks, halvar. As you asked, I deleted the thread folder from the first run and ran the tool on the same thread again. This was for Caroline Laurie's thread (t=628), which I chose for testing because it is very short.

Again, a significant percentage of files were corrupted. On the file we are using as an example, Pres12024.jpg, here is the most recent result:



In addition, here's a new observation: Caroline Laurie's thread is short, and has only 9 posts. But, on both this run and the earlier run, the tool skipped posts 1-5 and captured posts 6-9. Here is the partial list of posts captured, both times:

$ ls ./t628-Caroline_Laurie_Classic_Pornstars/
t628-p1-Caroline_Laurie_Classic_Pornstars.html
t628-p1-Caroline_Laurie_Classic_Pornstars-post-6-1987433
t628-p1-Caroline_Laurie_Classic_Pornstars-post-7-2282935
t628-p1-Caroline_Laurie_Classic_Pornstars-post-8-3178094
t628-p1-Caroline_Laurie_Classic_Pornstars-post-9-3983329

Originally Posted by halvar

Do you have SEVERE log entries for the affected files? If data is only retrieved partially there should be an error message.

No. The corrupted files files do not have "SEVERE" tags in their log lines, or any notes of interest as far as I can tell. For files that did not download, the logs do have "SEVERE" error messages. But for corrupted files, which did download, but badly, there are no interesting notes. For the corrupted example file Pres12024.jpg there are two lines because the log represents two runs:

$ grep Pres12024 vef-imagerescue.log.0
2017-##-## 10:##:## INFO: download: http://117.imagebam.com/download/bXeKK5UqKyEVHUZwhuzMwA/53993/539926047/Pres12024.jpg to storage/t628-Caroline_Laurie_Classic_Pornstars/t628-p1-Caroline_Laurie_Classic_Pornstars-post-9-3983329/Pres12024.jpg 
2017-##-## 20:##:## INFO: download: http://117.imagebam.com/download/bXeKK5UqKyEVHUZwhuzMwA/53993/539926047/Pres12024.jpg to storage/t628-Caroline_Laurie_Classic_Pornstars/t628-p1-Caroline_Laurie_Classic_Pornstars-post-9-3983329/Pres12024.jpg

I guess the tool thinks they're ok, even though they're not.

Originally Posted by halvar

The java version should not be the problem.

We shouldn't discount the serious possibility that I might have installed java badly. It's obvious you're a competent Linux user, how do you install your JRE? Here is exactly how I set up my environment: I created a new OS (16.04), downloaded your tool, installed updates, and ran the tool from the terminal. It complained that java was not installed, offered installation options, and I ran...

sudo apt install default-jre

After that ImageRescue ran, but with the problems described here.

Originally Posted by halvar

Could you please delete a thread folder, try the thread again and send me the log?
What kind of computer do you have? A very fast one?

Ran it again, gladly. Relevant portions above, but sending you full logs in a private message.

The computer is a year old, and is very average. Moderately priced, has a reasonable 8G of RAM, isn't super fast but isn't super slow. Average new machine from a major manufacturer.

And finally, I had this happen in 1.14 and 1.16, both.

And I am still assuming it's my environment and not the programmer's fault!

Thanks again for all your hard work, halvar.

[effCup]

Originally Posted by temptester666

Caroline Laurie's thread is short, and has only 9 posts. But, on both this run and the earlier run, the tool skipped posts 1-5 and captured posts 6-9.

I'll answer the easy bit: halvar's tool only downloads/backs up images from 2 hosts: imagebam & imgbox. That's precisely what we want (for now).

Other hosts are not (this month) going belly up.

[temptester666]

Effie is absolutely right about the mistake I made here.

Is there a way to strike-through comments on VEF?

[palo5]

Originally Posted by rotobott

I just get this



Probably best for us not to waste any more time on this, due to my pitiful computer skills, hey when I was at school we used slide rules

That's nothing -- we used our fingers, but still put a man in space before you

[halvar]

get it here: https://1fichier.com/?b6kgv3l0nf

Changes to 1.16 (1.17 is skipped)

• Filter processed posts by posters user name. In the GUI a username can optionally specified. If specified only posts created by this user are processed (and posts where the user cannot be detected).
• Range of pages to be processed can be specified in the GUl. Just leave it to from 1 to 10000 to process all pages.
• Multi threading: I finally decided to use multi threading, currently two. This means that there are two workers who process posts in parallel. Somebody suggested this, I cannot remember who. This should speed things up a little.

But beware: If you specify a filter in th GUI and later run the command line on the same thread the command line will not filter and all pages and posts will be processed.

BCFC_1982 has written a short howto:

Originally Posted by BCFC_1982

Instructions based on Windows 10 OS.

[deezer]

Originally Posted by halvar

If you mean the link to the gallery: It is not that easy. I would have to check:

• is a linked page an image page or album page
• if it is an album find zip if available or parse page to get images

Parsing the page would add another level of indirection. I'm sorry, I don't think I can do this.

Thanks for checking, and as effie mentioned above, it's really not a thing that's quite important.

[effCup]

Originally Posted by halvar

• Multi threading: I finally decided to use multi threading, currently two. This means that there are two workers who process posts in parallel. Somebody suggested this, I cannot remember who. This should speed things up a little.

Excellent work, halvar.

A question: does the "stop" button stop both workers?

The reason I ask: difficult to be sure quite this early, but maybe some different behaviour?

In particular: when I (once again) encounter "sticky" images and pull the old "trick" of stopping tool, loading in browser, and restarting tool. Previously when I did that, the tool would start again, find the end of the thread, find the already processed local files, eventually return to the "sticky" image and with any luck it might un-stick for me, & progress.

With 1.18: after pressing re-start, the tool seems to progress through only to where it finds the end of the thread. I do not see it progressing through the already processed local files, etc. Only if I quit the tool and re-run it can I get it to go past the end of the thread stage & into checking the already processed local files, then finally only the remaining (sticky+) images.

----

Hmm, and now, testing with two threads, I get this error msg:

2017-06-17 22:31:30 INFO: 2 threads specified: [239942, 236884]
2017-06-17 22:31:34 SEVERE: process page from must be a numeric value

The fields for process pages have not been altered, i.e. they're still 1 & 100000.

If I remove the second thread id the tool processes as before. This time I left it "alone" and although it seemed to get stuck a few times, it also unstuck itself.

[halvar]

Originally Posted by effCup

Hmm, and now, testing with two threads, I get this error msg:

2017-06-17 22:31:30 INFO: 2 threads specified: [239942, 236884]
2017-06-17 22:31:34 SEVERE: process page from must be a numeric value

The fields for process pages have not been altered, i.e. they're still 1 & 100000.

If I remove the second thread id the tool processes as before. This time I left it "alone" and although it seemed to get stuck a few times, it also unstuck itself.

Shame on me: I made the old copy and paste error (copy a few lines of code and not apply all necessary adjustments).

The checks intended for the page-from and page-up- to fields (numeric without spaces) are applied to the thread list where spaces are allowed. Thus preventing the download of more than one thread.

I fixed this in version 1.19: https://1fichier.com/?us1qj6ncpk

another change in 1.19:
an additional check is done after an image download: If the number of downloaded bytes does not match the size the server initially communicated the download fails and is retried later. This could help analyze the problems temptester666 had. I downloaded over a thousand of files using this check and it has not kicked in yet.

Concerning Multi-Threading (parallel workers): Both workers should stop if you hit the stop button. But they do not stop immediately. The stop button sets a stop signal and the worker check the stop signal in their loop. They stop the loop if the signal is set.
So it may take a few seconds until they get the message.
I'm not really happy with this stopping and starting. There should be a Toggle-Button which makes the current state clearly visible.
The retries at the end are not multi-threaded.

[effCup]

1.19 is really good.

First sighting of any Java errors?

2017-06-18 02:30:02 INFO: 2 threads specified: [271651, 271445] 
2017-06-18 02:30:02 INFO: Starting Thread: 271651 
2017-06-18 02:30:04 INFO: HTTP/1.1 200 OK 
2017-06-18 02:30:04 INFO: IDstack cookie found. Successfully logged in to VEF 
2017-06-18 02:30:04 INFO: Download first thread page: http://vintage-erotica-forum.com/t271651-p1-x.html 
2017-06-18 02:30:05 INFO: HTTP/1.1 200 OK 
2017-06-18 02:30:05 INFO: Thread name: Mystery_Followup_Exciting_17_cover_dark_haired_gi 
2017-06-18 02:30:05 INFO: Download forum page: http://vintage-erotica-forum.com/t271651-p1-x.html to D:\saved from vef\t271651-Mystery_Followup_Exciting_17_cover_dark_haired_gi\t271651-p1-Mystery_Followup_Exciting_17_cover_dark_haired_gi.html 
2017-06-18 02:30:07 INFO: End of Thread reached - no next page link found -  
2017-06-18 02:30:07 SEVERE: Index: 0, Size: 0 
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.rangeCheck(Unknown Source)
	at java.util.ArrayList.get(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.lambda$getPosts$9(ImageLinkProcessor.java:149)
	at java.util.stream.Collectors.lambda$toMap$58(Unknown Source)
	at java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source)
	at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.collect(Unknown Source)
	at vef.imgrescue.ImageLinkProcessor.getPosts(ImageLinkProcessor.java:146)
	at vef.imgrescue.ImageLinkProcessor.processForumPages(ImageLinkProcessor.java:39)
	at vef.imgrescue.VEFImageRescue.start(VEFImageRescue.java:100)
	at vef.imgrescue.Gui.lambda$null$6(Gui.java:248)
	at java.lang.Thread.run(Unknown Source)

Not sure what the problem was.

I quit the app. & ran it again on the same two threads, & this time no complaints or problems.