Vintage Erotica Forums

Vintage Erotica Forums (http://vintage-erotica-forum.com/index.php)
-   Image Hosts (http://vintage-erotica-forum.com/forumdisplay.php?f=88)
-   -   Discussion of the download / upload tools (http://vintage-erotica-forum.com/showthread.php?t=333326)

effCup June 17th, 2017 04:20 AM

Quote:

Originally Posted by halvar (Post 4070335)
If you mean the link to the gallery: It is not that easy. I would have to check:
  • is a linked page an image page or album page
  • if it is an album find zip if available or parse page to get images
Parsing the page would add another level of indirection. I'm sorry, I don't think I can do this.

Don't worry. I think all of my such galleries are already saved/should be re-uppable. A few other members have made similar posts but not many, so it's not a big issue.

temptester666 June 17th, 2017 04:25 AM

Thanks, halvar. As you asked, I deleted the thread folder from the first run and ran the tool on the same thread again. This was for Caroline Laurie's thread (t=628), which I chose for testing because it is very short. ;)

Again, a significant percentage of files were corrupted. On the file we are using as an example, Pres12024.jpg, here is the most recent result:

http://www.hotflick.net/u/n/UGYRKCqo...w7KnlBr4dl.jpg

In addition, here's a new observation: Caroline Laurie's thread is short, and has only 9 posts. But, on both this run and the earlier run, the tool skipped posts 1-5 and captured posts 6-9. Here is the partial list of posts captured, both times:
Code:

$ ls ./t628-Caroline_Laurie_Classic_Pornstars/
t628-p1-Caroline_Laurie_Classic_Pornstars.html
t628-p1-Caroline_Laurie_Classic_Pornstars-post-6-1987433
t628-p1-Caroline_Laurie_Classic_Pornstars-post-7-2282935
t628-p1-Caroline_Laurie_Classic_Pornstars-post-8-3178094
t628-p1-Caroline_Laurie_Classic_Pornstars-post-9-3983329

Quote:

Originally Posted by halvar (Post 4070120)
Do you have SEVERE log entries for the affected files? If data is only retrieved partially there should be an error message.

No. The corrupted files files do not have "SEVERE" tags in their log lines, or any notes of interest as far as I can tell. For files that did not download, the logs do have "SEVERE" error messages. But for corrupted files, which did download, but badly, there are no interesting notes. For the corrupted example file Pres12024.jpg there are two lines because the log represents two runs:
Code:

$ grep Pres12024 vef-imagerescue.log.0
2017-##-## 10:##:## INFO: download: http://117.imagebam.com/download/bXeKK5UqKyEVHUZwhuzMwA/53993/539926047/Pres12024.jpg to storage/t628-Caroline_Laurie_Classic_Pornstars/t628-p1-Caroline_Laurie_Classic_Pornstars-post-9-3983329/Pres12024.jpg
2017-##-## 20:##:## INFO: download: http://117.imagebam.com/download/bXeKK5UqKyEVHUZwhuzMwA/53993/539926047/Pres12024.jpg to storage/t628-Caroline_Laurie_Classic_Pornstars/t628-p1-Caroline_Laurie_Classic_Pornstars-post-9-3983329/Pres12024.jpg

I guess the tool thinks they're ok, even though they're not.
Quote:

Originally Posted by halvar (Post 4070120)
The java version should not be the problem.

We shouldn't discount the serious possibility that I might have installed java badly. :o It's obvious you're a competent Linux user, how do you install your JRE? Here is exactly how I set up my environment: I created a new OS (16.04), downloaded your tool, installed updates, and ran the tool from the terminal. It complained that java was not installed, offered installation options, and I ran...
Code:

sudo apt install default-jre
After that ImageRescue ran, but with the problems described here.
Quote:

Originally Posted by halvar (Post 4070120)
Could you please delete a thread folder, try the thread again and send me the log?
What kind of computer do you have? A very fast one?

Ran it again, gladly. Relevant portions above, but sending you full logs in a private message.

The computer is a year old, and is very average. Moderately priced, has a reasonable 8G of RAM, isn't super fast but isn't super slow. Average new machine from a major manufacturer.

And finally, I had this happen in 1.14 and 1.16, both.

And I am still assuming it's my environment and not the programmer's fault!

Thanks again for all your hard work, halvar.

effCup June 17th, 2017 04:28 AM

Quote:

Originally Posted by temptester666 (Post 4070362)
Caroline Laurie's thread is short, and has only 9 posts. But, on both this run and the earlier run, the tool skipped posts 1-5 and captured posts 6-9.

I'll answer the easy bit: halvar's tool only downloads/backs up images from 2 hosts: imagebam & imgbox. That's precisely what we want (for now).:rolleyes:

Other hosts are not (this month) going belly up.:thumbsup:

temptester666 June 17th, 2017 05:44 AM

Effie is absolutely right about the mistake I made here.

Is there a way to strike-through comments on VEF?

palo5 June 17th, 2017 06:13 AM

Quote:

Originally Posted by rotobott (Post 4068369)
I just get this

https://t9.pixhost.to/thumbs/665/41406125_capturev.jpg

Probably best for us not to waste any more time on this, due to my pitiful computer skills, hey when I was at school we used slide rules :D

That's nothing -- we used our fingers, but still put a man in space before you :D:D:D

halvar June 17th, 2017 06:15 AM

version 1.18 of imagerescue released
 
get it here: https://1fichier.com/?b6kgv3l0nf

Changes to 1.16 (1.17 is skipped)
  • Filter processed posts by posters user name. In the GUI a username can optionally specified. If specified only posts created by this user are processed (and posts where the user cannot be detected).
  • Range of pages to be processed can be specified in the GUl. Just leave it to from 1 to 10000 to process all pages.
  • Multi threading: I finally decided to use multi threading, currently two. This means that there are two workers who process posts in parallel. Somebody suggested this, I cannot remember who. This should speed things up a little.

But beware: If you specify a filter in th GUI and later run the command line on the same thread the command line will not filter and all pages and posts will be processed.

BCFC_1982 has written a short howto:
Quote:

Originally Posted by BCFC_1982 (Post 4068285)
Instructions based on Windows 10 OS.


deezer June 17th, 2017 10:32 AM

Quote:

Originally Posted by halvar (Post 4070335)
If you mean the link to the gallery: It is not that easy. I would have to check:
  • is a linked page an image page or album page
  • if it is an album find zip if available or parse page to get images
Parsing the page would add another level of indirection. I'm sorry, I don't think I can do this.


Thanks for checking, and as effie mentioned above, it's really not a thing that's quite important.

effCup June 17th, 2017 10:33 AM

Quote:

Originally Posted by halvar (Post 4070426)
  • Multi threading: I finally decided to use multi threading, currently two. This means that there are two workers who process posts in parallel. Somebody suggested this, I cannot remember who. This should speed things up a little.

Excellent work, halvar.:cool::thumbsup:

A question: does the "stop" button stop both workers?:confused:

The reason I ask: difficult to be sure quite this early, but maybe some different behaviour?:confused:

In particular: when I (once again) encounter "sticky" images and pull the old "trick" of stopping tool, loading in browser, and restarting tool. Previously when I did that, the tool would start again, find the end of the thread, find the already processed local files, eventually return to the "sticky" image and with any luck it might un-stick for me, & progress.

With 1.18: after pressing re-start, the tool seems to progress through only to where it finds the end of the thread. I do not see it progressing through the already processed local files, etc.:confused: Only if I quit the tool and re-run it can I get it to go past the end of the thread stage & into checking the already processed local files, then finally only the remaining (sticky+) images.

----

Hmm, and now, testing with two threads, I get this error msg:

2017-06-17 22:31:30 INFO: 2 threads specified: [239942, 236884]
2017-06-17 22:31:34 SEVERE: process page from must be a numeric value

The fields for process pages have not been altered, i.e. they're still 1 & 100000.:confused:

If I remove the second thread id the tool processes as before. This time I left it "alone" and although it seemed to get stuck a few times, it also unstuck itself.

halvar June 17th, 2017 01:05 PM

vef-imagerescue 1.19 is available bugfix multiple threads ids
 
Quote:

Originally Posted by effCup (Post 4070612)
Hmm, and now, testing with two threads, I get this error msg:

2017-06-17 22:31:30 INFO: 2 threads specified: [239942, 236884]
2017-06-17 22:31:34 SEVERE: process page from must be a numeric value

The fields for process pages have not been altered, i.e. they're still 1 & 100000.:confused:

If I remove the second thread id the tool processes as before. This time I left it "alone" and although it seemed to get stuck a few times, it also unstuck itself.

Shame on me: I made the old copy and paste error (copy a few lines of code and not apply all necessary adjustments).

The checks intended for the page-from and page-up- to fields (numeric without spaces) are applied to the thread list where spaces are allowed. Thus preventing the download of more than one thread.

I fixed this in version 1.19: https://1fichier.com/?us1qj6ncpk

another change in 1.19:
an additional check is done after an image download: If the number of downloaded bytes does not match the size the server initially communicated the download fails and is retried later. This could help analyze the problems temptester666 had. I downloaded over a thousand of files using this check and it has not kicked in yet.

Concerning Multi-Threading (parallel workers): Both workers should stop if you hit the stop button. But they do not stop immediately. The stop button sets a stop signal and the worker check the stop signal in their loop. They stop the loop if the signal is set.
So it may take a few seconds until they get the message.
I'm not really happy with this stopping and starting. There should be a Toggle-Button which makes the current state clearly visible.
The retries at the end are not multi-threaded.

effCup June 17th, 2017 02:36 PM

1.19 is really good.:):thumbsup:

First sighting of any Java errors?

Code:

2017-06-18 02:30:02 INFO: 2 threads specified: [271651, 271445]
2017-06-18 02:30:02 INFO: Starting Thread: 271651
2017-06-18 02:30:04 INFO: HTTP/1.1 200 OK
2017-06-18 02:30:04 INFO: IDstack cookie found. Successfully logged in to VEF
2017-06-18 02:30:04 INFO: Download first thread page: http://vintage-erotica-forum.com/t271651-p1-x.html
2017-06-18 02:30:05 INFO: HTTP/1.1 200 OK
2017-06-18 02:30:05 INFO: Thread name: Mystery_Followup_Exciting_17_cover_dark_haired_gi
2017-06-18 02:30:05 INFO: Download forum page: http://vintage-erotica-forum.com/t271651-p1-x.html to D:\saved from vef\t271651-Mystery_Followup_Exciting_17_cover_dark_haired_gi\t271651-p1-Mystery_Followup_Exciting_17_cover_dark_haired_gi.html
2017-06-18 02:30:07 INFO: End of Thread reached - no next page link found - 
2017-06-18 02:30:07 SEVERE: Index: 0, Size: 0
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.rangeCheck(Unknown Source)
        at java.util.ArrayList.get(Unknown Source)
        at vef.imgrescue.ImageLinkProcessor.lambda$getPosts$9(ImageLinkProcessor.java:149)
        at java.util.stream.Collectors.lambda$toMap$58(Unknown Source)
        at java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source)
        at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.util.stream.ReferencePipeline.collect(Unknown Source)
        at vef.imgrescue.ImageLinkProcessor.getPosts(ImageLinkProcessor.java:146)
        at vef.imgrescue.ImageLinkProcessor.processForumPages(ImageLinkProcessor.java:39)
        at vef.imgrescue.VEFImageRescue.start(VEFImageRescue.java:100)
        at vef.imgrescue.Gui.lambda$null$6(Gui.java:248)
        at java.lang.Thread.run(Unknown Source)

Not sure what the problem was.:confused:

I quit the app. & ran it again on the same two threads, & this time no complaints or problems.:confused::cool::thumbsup:


All times are GMT. The time now is 03:26 AM.



vBulletin Optimisation provided by vB Optimise v2.6.1 (Pro) - vBulletin Mods & Addons Copyright © 2024 DragonByte Technologies Ltd.