June 17th, 2017, 03:40 PM | #81 |
Vintage Idiot
Join Date: Feb 2012
Location: History
Posts: 22,138
Thanks: 226,735
Thanked 356,789 Times in 21,632 Posts
|
Also:
Code:
2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-0] http://www.imagebam.com/image/18faf0214326512 2017-06-18 03:38:19 INFO: http://imgbox.com/aakZS1OT is already processed 2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/aab1fIDL 2017-06-18 03:38:19 INFO: http://imgbox.com/aab1fIDL is already processed 2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/yRKJx2mj 2017-06-18 03:38:19 INFO: http://imgbox.com/yRKJx2mj is already processed 2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/omwroqtK 2017-06-18 03:38:19 INFO: http://imgbox.com/omwroqtK is already processed 2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/CpNqokEZ 2017-06-18 03:38:19 INFO: http://imgbox.com/CpNqokEZ is already processed 2017-06-18 03:38:19 INFO: image-page: [ForkJoinPool-1-worker-1] http://imgbox.com/O3RVsO8H 2017-06-18 03:38:19 INFO: http://imgbox.com/O3RVsO8H is already processed 2017-06-18 03:38:19 SEVERE: null java.lang.NullPointerException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at java.util.concurrent.ForkJoinTask.getThrowableException(Unknown Source) at java.util.concurrent.ForkJoinTask.reportException(Unknown Source) at java.util.concurrent.ForkJoinTask.join(Unknown Source) at vef.imgrescue.ImageLinkProcessor.processForumPages(ImageLinkProcessor.java:59) at vef.imgrescue.VEFImageRescue.start(VEFImageRescue.java:100) at vef.imgrescue.Gui.lambda$null$6(Gui.java:248) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.NullPointerException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at java.util.concurrent.ForkJoinTask.getThrowableException(Unknown Source) at java.util.concurrent.ForkJoinTask.reportException(Unknown Source) at java.util.concurrent.ForkJoinTask.invoke(Unknown Source) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source) at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.util.stream.ReferencePipeline.forEach(Unknown Source) at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source) at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$2(ImageLinkProcessor.java:42) at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source) at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source) at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) Caused by: java.lang.NullPointerException at vef.imgrescue.Gui$1.publish(Gui.java:51) at java.util.logging.Logger.log(Unknown Source) at java.util.logging.Logger.doLog(Unknown Source) at java.util.logging.Logger.log(Unknown Source) at java.util.logging.Logger.info(Unknown Source) at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:38) at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$7(ImageLinkProcessor.java:119) at java.util.ArrayList.forEach(Unknown Source) at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:103) at vef.imgrescue.ImageLinkProcessor.lambda$null$1(ImageLinkProcessor.java:56) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) at java.util.TreeMap$EntrySpliterator.forEachRemaining(Unknown Source) at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source) at java.util.concurrent.CountedCompleter.exec(Unknown Source) ... 4 more NB, I'm posting these errors for halvar not because they're necessarily very serious or frequent (they're not); just because he might be interested. |
June 17th, 2017, 03:58 PM | #82 | |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
Quote:
This happens when the post nr cannot be determined. I am looking for the link to report a post '..report.php?p=nnn' to extract the post number. If the link is not found - bam! It may be that the html file was corrupt. Because when you retry the last html-page is downloaded again. So it works the scond time. Fetching information from arbitrary portions of the html is error prone. Sometimes a page is rendered differently just because the user has different settings and the thing breaks. Given more time one would select the elements to get the information from more carefully. If this happens again, please send me the html files before you retry. |
|
June 17th, 2017, 04:08 PM | #83 | ||
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
Quote:
Quote:
|
||
June 17th, 2017, 04:14 PM | #84 |
Vintage Idiot
Join Date: Feb 2012
Location: History
Posts: 22,138
Thanks: 226,735
Thanked 356,789 Times in 21,632 Posts
|
Hmm, only noticed later that the "top" of that log snippet was missing/cropped. I'd simply copied all & pasted from the gui window. Let me know if you want to see more/the top/start of that instance, as I should be able to retrieve it from your log file?
---- edit: Hmm. Running the tool on this thread, it has stalled at this point: Code:
2017-06-18 04:10:45 INFO: 2 threads specified: [279125, 276115] 2017-06-18 04:10:45 INFO: Starting Thread: 279125 2017-06-18 04:10:46 INFO: HTTP/1.1 200 OK 2017-06-18 04:10:46 INFO: IDstack cookie found. Successfully logged in to VEF 2017-06-18 04:10:46 INFO: Download first thread page: http://vintage-erotica-forum.com/t279125-p1-x.html 2017-06-18 04:10:47 INFO: HTTP/1.1 200 OK 2017-06-18 04:10:47 INFO: Thread name: Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov 2017-06-18 04:10:47 INFO: Download forum page: http://vintage-erotica-forum.com/t279125-p1-x.html to D:\saved from vef\t279125-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov\t279125-p1-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov.html 2017-06-18 04:10:48 INFO: End of Thread reached - no next page link found - 2017-06-18 04:10:48 INFO: image-page: [ForkJoinPool-1-worker-0] http://www.imagebam.com/image/9d10cc175732916 2017-06-18 04:10:52 INFO: downloaded: http://60.imagebam.com/download/w3FvcZ3Piu_OFcrAmNX6Uw/17574/175732916/Hustler%20Magazine%20April%201982%20035.jpg to D:\saved from vef\t279125-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov\t279125-p1-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov-post-1-3115074\Hustler Magazine April 1982 035.jpg 2017-06-18 04:10:52 INFO: image-page: [ForkJoinPool-1-worker-0] http://www.imagebam.com/image/2fb172175732953 2017-06-18 04:10:53 SEVERE: download: http://25.imagebam.com/download/rqE15VsmXWdS40oDBbf2CA/17574/175732953/Hustler%20Magazine%20April%201982%20036.jpg failed HTTP/1.1 404 Not Found 2017-06-18 04:10:54 INFO: downloaded: http://thumbnails25.imagebam.com/17574/2fb172175732953.jpg to D:\saved from vef\t279125-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov\t279125-p1-Mystery_Followup_Brunette_Feb_84_Velvet_Talks_Cov-post-1-3115074\thumb_Hustler Magazine April 1982 036.jpg 2017-06-18 04:10:54 INFO: image-page: [Thread-6] RETRY http://www.imagebam.com/image/2fb172175732953 If I look in the storage folder, the second image has been processed (as a thumb, because it's a dead image). This was not a re-run over a partially-completed thread. Maybe it's just a coincidence, but I'd have thought the tool wouldn't stop at a point where it (seems it) should have already finished with that url/image file. I'd expect (& have often seen) it instead stopping/pausing at the next thumb-link/url that has yet to be processed. This thread should be "finished", but the tool has not moved on to the second thread in the queue--there's no folder been created for it. (only now) Stopping the tool & re-starting it it does finish with that thread & move on to process the next one in the queue. Last edited by effCup; June 17th, 2017 at 04:28 PM.. |
June 17th, 2017, 05:04 PM | #85 | |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
Quote:
If a download stalls you see the 'image-page...' but not the 'downloaded to'. How long do you usually wait before you act? I would give it at least 30 seconds. Usually it should be done in a second, but it sometimes just take longer. The timestamps in the log show how long it usually takes (time between 'image-page' and 'downloaded to'. The word RETRY in the log statement means that this is a retry for a previously failed attempt. It could be that retries take longer than normal requests because of the reason of the fail. The log is cropped in the because only the last part is relevant. If you want to check for a longer time period you can use the files. The log files are limited to a total of 50MB. |
|
June 18th, 2017, 12:39 AM | #86 |
Sunny Mod
Join Date: Jan 2016
Posts: 5,514
Thanks: 48,509
Thanked 53,342 Times in 5,485 Posts
|
I warmly recommend to use halvar's tool. It's working great.
Not only for downloading and saving, but also for organize the work in a section! Working in the MIR-section there are a thousands of threads, most of them very short. But we must be sure to get them all. effCup compiled a list of the threads using halvar's tool. Copying 20 threads-numbers, paste them into the tool and let it do the work. With the list effCup created, the work is absolutely well organized. If I'm done with a batch I note this in the list, so nobody else will download this threads, which avoids double work. I have downloaded now 85 threads of the ordinary queries and the tool (v1.16) works very fine. Only once I faced minor problem. When the tool finished with a batch, I take a quick look into the log, if a error occured. If so, I check the thread manually again, but it happened only once until now. I will store the threads in batches of around 500 MB at a filehost. The re-hosting and editing of each thread can be done also after 30.06.2017! But saving must be done before end of June. The best thing is, while the tool is running I can work in the box and re-host the images and edit posts. I strongly suggest, to use this tool. Not only for downloading, but also for organize the work in a section with the thread-generator! The work in MIR is now organized, like effCup here and here and Al Gebra here suggested. And it's going extremely well.
__________________
. Last edited by deezer; June 18th, 2017 at 08:54 AM.. Reason: wording |
The Following 11 Users Say Thank You to deezer For This Useful Post: |
June 18th, 2017, 02:19 AM | #87 | |
Former Staff
Join Date: Jun 2007
Location: Germany
Posts: 11,875
Thanks: 19,210
Thanked 570,935 Times in 11,033 Posts
|
Quote:
On this occasion I noticed that many links to my pics I just recently uploaded to ImageBam are already dead, example (see also my following posts there). Only the thumbs (preview pics) are still visible/available. And vef-imagerescue-1.19 saved these at least.
__________________
m Please add source, post complete photo and scan sets - with indexes, if available, preserve genuine file names (that will help to ID sources and model names), thank, credit, and quote original posters. I'm afraid I haven't any time for reuploads. Don't send reports (or PMs) of dead files or requests! Once the files posted above are expired, please help each other, add the info I provided as well. To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. -> Underlined words in my posts are clickable. <-
|
|
The Following 14 Users Say Thank You to Al Gebra For This Useful Post: |
June 18th, 2017, 05:37 AM | #88 | |
Vintage Idiot
Join Date: Feb 2012
Location: History
Posts: 22,138
Thanks: 226,735
Thanked 356,789 Times in 21,632 Posts
|
Quote:
Code:
2017-06-18 17:20:01 INFO: 2 threads specified: [268142, 268227] 2017-06-18 17:20:01 INFO: Starting Thread: 268142 2017-06-18 17:20:03 INFO: HTTP/1.1 200 OK 2017-06-18 17:20:03 INFO: IDstack cookie found. Successfully logged in to VEF 2017-06-18 17:20:03 INFO: Using existing target folder :D:\saved from vef\t268142-Mystery_Followup_Vintage_blonde_in_ripped_dress_M 2017-06-18 17:20:03 INFO: Download forum page: http://vintage-erotica-forum.com/t268142-p1-x.html to D:\saved from vef\t268142-Mystery_Followup_Vintage_blonde_in_ripped_dress_M\t268142-p1-Mystery_Followup_Vintage_blonde_in_ripped_dress_M.html 2017-06-18 17:20:06 INFO: End of Thread reached - no next page link found - 2017-06-18 17:20:06 SEVERE: Index: 0, Size: 0 java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck(Unknown Source) at java.util.ArrayList.get(Unknown Source) at vef.imgrescue.ImageLinkProcessor.lambda$getPosts$9(ImageLinkProcessor.java:149) at java.util.stream.Collectors.lambda$toMap$58(Unknown Source) at java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source) at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.util.stream.ReferencePipeline.collect(Unknown Source) at vef.imgrescue.ImageLinkProcessor.getPosts(ImageLinkProcessor.java:146) at vef.imgrescue.ImageLinkProcessor.processForumPages(ImageLinkProcessor.java:39) at vef.imgrescue.VEFImageRescue.start(VEFImageRescue.java:100) at vef.imgrescue.Gui.lambda$null$6(Gui.java:248) at java.lang.Thread.run(Unknown Source) The html tags seem "whole", but you're right, the report button(s) are missing from that page's source. the post no. appears in several places (in a complete page, I mean), I think all buttons: reply buttons showpost buttons report buttons (what you're currently using) edit buttons Sorry, I haven't really studied the html structure. Last edited by effCup; June 18th, 2017 at 05:43 AM.. |
|
June 18th, 2017, 07:49 AM | #89 | |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
Quote:
In cases of wrong username/password this error should not occur. If a login fails you should see log text like this and the process is aborted. Code:
2017-06-18 09:35:48 INFO: Starting Thread: 718 2017-06-18 09:35:49 INFO: HTTP/1.1 200 OK 2017-06-18 09:35:49 SEVERE: IDstack cookie NOT found. NOT logged in to VEF. Aborting 2017-06-18 09:35:49 INFO: Finished Thread: 718 I could change it to get the post number from the other links you mentioned. But I only want to do only really necessary changes now to avoid introducing new bugs. And besides I do not know if you really get the pages if you are not logged in. |
|
June 18th, 2017, 11:26 AM | #90 |
Sunny Mod
Join Date: Jan 2016
Posts: 5,514
Thanks: 48,509
Thanked 53,342 Times in 5,485 Posts
|
Don't know, if you saw this problem yet:
An error occured with this post: http://vintage-erotica-forum.com/showpost.php?p=3711213 Code:
2017-06-18 12:41:03 SCHWERWIEGEND: Error downloading 'http://www.imagebam.com/gallery/cpnazhtdpdpc3lht4u54c0icw6f3rp54' org.apache.http.client.ClientProtocolException at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at vef.imgrescue.AbstactImageHost.downloadFile(AbstactImageHost.java:89) at vef.imgrescue.AbstactImageHost.download(AbstactImageHost.java:66) at vef.imgrescue.ImageLinkProcessor.lambda$downloadImages$7(ImageLinkProcessor.java:119) at java.util.ArrayList.forEach(Unknown Source) at vef.imgrescue.ImageLinkProcessor.downloadImages(ImageLinkProcessor.java:103) at vef.imgrescue.ImageLinkProcessor.lambda$null$1(ImageLinkProcessor.java:56) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) at java.util.TreeMap$EntrySpliterator.forEachRemaining(Unknown Source) at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source) at java.util.concurrent.CountedCompleter.exec(Unknown Source) at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.util.concurrent.ForkJoinPool.helpComplete(Unknown Source) at java.util.concurrent.ForkJoinPool.awaitJoin(Unknown Source) at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source) at java.util.concurrent.ForkJoinTask.invoke(Unknown Source) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source) at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.util.stream.ReferencePipeline.forEach(Unknown Source) at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source) at vef.imgrescue.ImageLinkProcessor.lambda$processForumPages$2(ImageLinkProcessor.java:42) at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source) at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source) at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) Caused by: org.apache.http.ProtocolException: Target host is not specified at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71) at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ... 29 more Code:
[URL=http://www.imagebam.com/gallery/cpnazhtdpdpc3lht4u54c0icw6f3rp54][IMG]http://thumbnails116.imagebam.com/49580/eb8796495795513.jpg[/IMG][/URL] But I assume, this isn't a bug in the tool. It's more a very similar situation as with direct linked galleries, which we yet discussed here. Btw, the tool saved properly the image which is linked in the signature of Gladys Allova.
__________________
. Last edited by deezer; June 18th, 2017 at 11:53 AM.. |
|
|