Discussion of the download / upload tools

[halvar]

Originally Posted by Pavlov49

This sounds easy, even for me.

1. Your tool - I'll call it "hVEFDL" (=halvar's VEFDownLoader) from now on - stores the images in a folder structure such like:
   Z:\_Pix\VEF-Rescue\
   t3146-Ginger_Lynn_Classic_Pornstars\
   t3146-p4-Ginger_Lynn_Classic_Pornstars-post-34-249738\
   96458_676211_354be8a7724_122_699lo.jpg
   ....
2. I assume ALL of us have their own downloads organized according to model's ID, magazines, movies, ...
3. We need to keep the .originalfilenameswithoutextension-files to let hVEFDL recognize during thread-updates, what files had previously been saved.
4. I want to keep with my original folder structure - but don't want to copy hVEFDL's files to my own folders, to avoid thousands of duplicates.
5. What if I kept the HTMLs, the .originalfilenameswithoutextension-files, and the thumbs in hVEFJDL's folder, but moved all pictures from there to my own folders? Would this work?

As long as you keep the html-files and dot-files you can move the images away.

One little information will get lost. While migrating to the database with the upcoming version the image file size is written in the database. If the file is not there anymore, it will write -2.

I personally would not do this. For me the downloaded files are a raw backup and I want to keep that untouched. Duplicate files are no problem for me, as hard disks are cheap. And I manage my files using MD5 hashes so I quickly can find out were duplicates are.

But that is a matter of personal preference.

[halvar]

Concerning the naming scheme of post folders: I created the first issue in the issue tracker:
https://git.teknik.io/halvar/forum-backup/issues/1

I created a git repository at teknik.io:
https://git.teknik.io/halvar/forum-backup

teknik.io is a github clone and provides source code repositories, issue tracker and so on.

Using this makes organizing the project much easier.

Since the whole project is available to the general public, I will keep it neutral. There will be no mentioning of this particular forum, erotica or porn. Keep that in mind if you create issues.
You have to register if you want to create an issue.

The current state is still work in progress. Documentation and refactoring will be done. So please don't judge my programming skills to harshly when looking at the code...

[halvar]

I released the first version of the new forum-backup tool.
https://1fichier.com/?6n0dl4rukx

The download is about 65 MB because a JRE (Java Runtime Environment) for Windows (32bit!) is included. This makes installation and usage easier.

If you already have a storage folder created by the image rescue tool it can be reused:

• Data from the dot-files will be migrated for the threads that are processed.
• Threads in the storage folder that are not in the treads.csv file are ignored.

forum-backup does not create dot-files any more, it keeps meta data in an embedded h2-database (http://www.h2database.com).
Please do not add too many threads at once to the threads.csv file. Start with one and check how it works.

A short howto can be found here:
https://git.teknik.io/halvar/forum-backup/wiki/Howto

Currently supported image hosts are:

• Hotflick
• ImageBam
• ImageUpper
• ImageVenue
• ImgBox
• PimpAndHost
• Pixhost
• PostImage

The project can be found here: https://git.teknik.io/halvar/forum-backup
The issue tracker is here: https://git.teknik.io/halvar/forum-backup/issues. Feel free to create new issues (you have to register to do this).

[halvar]

Originally Posted by Pavlov49

Does this work on Win 10-x64, too?
What if the old version was running successfully, thus JRE already been correctly working?
Will there be any kind of conflicts?

It should work. 32bit is the lowest common denominator.

I only have Win7 32bit at Hand for testing. Please let me know if it doesn't.

Maybe it is time for me to get Windows10. But I prefer GNU/Linux...

You also can modify the bat files to point to another JRE.

[halvar]

get it here: https://1fichier.com/?ebpooruxf6

short documentation is here: https://git.teknik.io/halvar/forum-backup/wiki/Howto
release notes are here: https://git.teknik.io/halvar/forum-b...i/ReleaseNotes
additional documentation is here: https://git.teknik.io/halvar/forum-backup/wiki

[deezer]

I didn't test the new forum-backup-tool until now. But tested it with v1.24.

Originally Posted by deezer

This post:
http://vintage-erotica-forum.com/showpost.php?p=3711213

...

Btw, the tool saved properly the image which is linked in the signature of Gladys Allova.

V1.24 saves also the images in the signatures of a post.
To avoid that I edited my Thread Display Options and unchecked the Show Signatures-checkbox.
Using this option while running the tool, the images in the signature were not saved.

[Denaniel]

Thanks again for this great tool, halvar!

I've backed up over 500 threads, mostly those in Classic Movies forum that I've posted in myself, just in case I have to restore the posts someday. :crossingfingers:

One little feature request: When I search the log for "unsupported image hoster", I can download many of the skipped images manually, but there is no easy way to figure out which threads those unsupported image links appeared in. I've tried checking the threads which appear in the log immediately before and after the "unsupported" URLs, but had no luck finding the correct threads.

Is there any way to add the thread number to the "unsupported image hoster" line(s) in the log?

Example excerpt from recent log file:

2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-3] downloaded: http://thumbnails26.imagebam.com/10709/37484a107089190.jpg to L:\scanned images\mag scans\storage\my threads\t137027-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M\t137027-p1-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M-post-1-1349053\thumb_01-00-16.jpg 
2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-3] image-page: http://www.imagebam.com/image/f99d16107089191 
2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-1] downloaded: http://img248.imagevenue.com/aAfkjfp01fo1i-24151/loc338/95888_VTS_05_1_123_338lo.JPG to L:\scanned images\mag scans\storage\my threads\t12259-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff\t12259-p1-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff-post-10-2688581\95888_VTS_05_1_123_338lo.JPG 
2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-1] image-page: http://img253.imagevenue.com/img.php?image=95915_VTS_05_2_123_83lo.JPG 
2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-2] downloaded: http://105.imagebam.com/download/tDZc7e2bY9h3qavGxEVq1A/46337/463366495/drt_idx.jpg to L:\scanned images\mag scans\storage\my threads\t112211-Dirty_Twins_1993_Classic_Movies\t112211-p1-Dirty_Twins_1993_Classic_Movies-post-6-3531474\drt_idx.jpg 
2017-07-21 16:53:05 INFO: Unsupported image hoster: http://depic.me/hwh47nrc9daw/vts_01_1.vob.jpg 
2017-07-21 16:53:05 INFO: Unsupported image hoster: http://depic.me/lrwwhgzy9f2d/vint.jpg 
2017-07-21 16:53:05 INFO: Unsupported image hoster: http://imagezilla.net/show/ittUw2R-Happening.1981.BDrip.1080p.x264.Den.mkv.jpg 
2017-07-21 16:53:05 INFO: Unsupported image hoster: http://www.stooorage.com/show/512/993925_mddbakalsm.jpg 
2017-07-21 16:53:05 INFO: Unsupported image hoster: http://www.stooorage.com/show/512/993926_mddbakalsm-e.jpg 
2017-07-21 16:53:05 INFO: Unsupported image hoster: http://jdownloader.org/download/offline 
2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-2] image-page: http://img276.imagevenue.com/img.php?image=95568_thumbs20100928200003_123_461lo.jpg 
2017-07-21 16:53:07 INFO: PROGRESS: Download images613/702 
2017-07-21 16:53:10 SEVERE: [ForkJoinPool-1-worker-3] download:  http://25.imagebam.com/download/lgTO7LT7A9DP603kLjOaCQ/10709/107089191/01-04-06.jpg failed HTTP/1.1 404 Not Found 
2017-07-21 16:53:12 INFO: [ForkJoinPool-1-worker-3] downloaded: http://thumbnails25.imagebam.com/10709/f99d16107089191.jpg to L:\scanned images\mag scans\storage\my threads\t137027-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M\t137027-p1-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M-post-1-1349053\thumb_01-04-06.jpg 
2017-07-21 16:53:12 INFO: [ForkJoinPool-1-worker-3] image-page: http://www.imagebam.com/image/2de7bd107089192 
2017-07-21 16:53:12 INFO: [ForkJoinPool-1-worker-1] downloaded: http://img253.imagevenue.com/aAfkjfp01fo1i-30177/loc83/95915_VTS_05_2_123_83lo.JPG to L:\scanned images\mag scans\storage\my threads\t12259-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff\t12259-p1-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff-post-10-2688581\95915_VTS_05_2_123_83lo.JPG 
2017-07-21 16:53:12 INFO: [ForkJoinPool-1-worker-1] image-page: http://img261.imagevenue.com/img.php?image=95915_VTS_06_1_123_27lo.JPG 
2017-07-21 16:53:15 INFO: [ForkJoinPool-1-worker-3] downloaded: http://33.imagebam.com/download/k2XuxOj6UPUsx6RqJh4EKA/10709/107089192/01-06-01.jpg to L:\scanned images\mag scans\storage\my threads\t137027-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M\t137027-p1-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M-post-1-1349053\01-06-01.jpg 
2017-07-21 16:53:16 INFO: [ForkJoinPool-1-worker-3] image-page: http://pimpandhost.com/image/show/id/5486279 
2017-07-21 16:53:17 INFO: PROGRESS: Download images616/702 
2017-07-21 16:53:19 INFO: [ForkJoinPool-1-worker-1] downloaded: http://img261.imagevenue.com/aAfkjfp01fo1i-4227/loc27/95915_VTS_06_1_123_27lo.JPG to L:\scanned images\mag scans\storage\my threads\t12259-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff\t12259-p1-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff-post-10-2688581\95915_VTS_06_1_123_27lo.JPG

The two depic.me links worked okay for me, but in this case I was lucky that I recognized the movie from the screencaps, which turned out to be "Expectations":

t116096-Expectations_aka_JOse_tout_1977_Classic_Movies

This thread was last processed four days earlier, so the "unsupported image hoster" lines in the log were clearly part of a retry, with no easy way to link the images to the thread where they belong.

[halvar]

Originally Posted by Denaniel

Is there any way to add the thread number to the "unsupported image hoster" line(s) in the log?

I would also like to add a short reason for download fails (unsupported host, 404, timeout, ...). I will try this.
https://git.teknik.io/halvar/forum-backup/issues/23

I assume you are using the old tool imagerescue? Then you can generate report csv file by clicking on the button "Create Report of local files". This parses all local html files and creates a csv file in your storage folder. This file contains a line for every image link found including thread id and image url. It may take some time for your 500 threads.
You can lookup your info there.

The new forum-backup does not have this yet. I am working on a web console with a report feature. If you know sql you could do your own queries too.

Here is a sample report:


I am planning to create a new release including this next weekend.

[Denaniel]

Originally Posted by halvar

I assume you are using the old tool imagerescue? Then you can generate report csv file by clicking on the button "Create Report of local files". This parses all local html files and creates a csv file in your storage folder. This file contains a line for every image link found including thread id and image url...

I tried that, using vef-imagerescue 1.26, but the csv file only lists the imagebam and imagebox links. It ignores all other imagehost links.

Update: I used forum-backup 0.0.2 batch file "create-report.bat" and the new report does list all the image links, including the unsupported image hosters. So that made it possible to find the correct threads for the mystery images. Cheers.

[effCup]

vef-imagerescue can generate a listing of threads that are within an input forum. Can similar functionality be added to forum-backup? Either, similar to before: a) generate a file listing that can later be used as an input for the app., or: b) simply feed the app. with the forum id, and it then processes all the threads found within that forum?

Such functionality would be particularly valuable for the model id request section where there's lots of small threads.