July 2nd, 2017, 03:11 PM | #151 | |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
Quote:
One little information will get lost. While migrating to the database with the upcoming version the image file size is written in the database. If the file is not there anymore, it will write -2. I personally would not do this. For me the downloaded files are a raw backup and I want to keep that untouched. Duplicate files are no problem for me, as hard disks are cheap. And I manage my files using MD5 hashes so I quickly can find out were duplicates are. But that is a matter of personal preference. |
|
July 2nd, 2017, 04:04 PM | #152 |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
forum-backup at teknik.io
Concerning the naming scheme of post folders: I created the first issue in the issue tracker:
https://git.teknik.io/halvar/forum-backup/issues/1 I created a git repository at teknik.io: https://git.teknik.io/halvar/forum-backup teknik.io is a github clone and provides source code repositories, issue tracker and so on. Using this makes organizing the project much easier. Since the whole project is available to the general public, I will keep it neutral. There will be no mentioning of this particular forum, erotica or porn. Keep that in mind if you create issues. You have to register if you want to create an issue. The current state is still work in progress. Documentation and refactoring will be done. So please don't judge my programming skills to harshly when looking at the code... |
July 7th, 2017, 01:41 PM | #153 |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
forum-backup 0.0.1 is released
I released the first version of the new forum-backup tool.
https://1fichier.com/?6n0dl4rukx The download is about 65 MB because a JRE (Java Runtime Environment) for Windows (32bit!) is included. This makes installation and usage easier. If you already have a storage folder created by the image rescue tool it can be reused:
forum-backup does not create dot-files any more, it keeps meta data in an embedded h2-database (http://www.h2database.com). Please do not add too many threads at once to the threads.csv file. Start with one and check how it works. A short howto can be found here: https://git.teknik.io/halvar/forum-backup/wiki/Howto Currently supported image hosts are:
The project can be found here: https://git.teknik.io/halvar/forum-backup The issue tracker is here: https://git.teknik.io/halvar/forum-backup/issues. Feel free to create new issues (you have to register to do this). |
July 7th, 2017, 06:44 PM | #154 | |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
Quote:
I only have Win7 32bit at Hand for testing. Please let me know if it doesn't. Maybe it is time for me to get Windows10. But I prefer GNU/Linux... You also can modify the bat files to point to another JRE. |
|
July 9th, 2017, 09:38 AM | #155 |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
forum-backup 0.0.2 is released
get it here: https://1fichier.com/?ebpooruxf6
short documentation is here: https://git.teknik.io/halvar/forum-backup/wiki/Howto release notes are here: https://git.teknik.io/halvar/forum-b...i/ReleaseNotes additional documentation is here: https://git.teknik.io/halvar/forum-backup/wiki |
July 21st, 2017, 10:19 PM | #156 | |
Sunny Mod
Join Date: Jan 2016
Posts: 5,512
Thanks: 48,485
Thanked 53,322 Times in 5,482 Posts
|
How to avoid saving images in signatures
I didn't test the new forum-backup-tool until now. But tested it with v1.24.
Quote:
To avoid that I edited my Thread Display Options and unchecked the Show Signatures-checkbox. Using this option while running the tool, the images in the signature were not saved.
__________________
. |
|
July 23rd, 2017, 01:16 AM | #157 |
Administrator
Join Date: Dec 2006
Location: Rocinante Ops Deck
Posts: 13,918
Thanks: 114,775
Thanked 425,825 Times in 13,444 Posts
|
Thanks again for this great tool, halvar!
I've backed up over 500 threads, mostly those in Classic Movies forum that I've posted in myself, just in case I have to restore the posts someday. :crossingfingers: One little feature request: When I search the log for "unsupported image hoster", I can download many of the skipped images manually, but there is no easy way to figure out which threads those unsupported image links appeared in. I've tried checking the threads which appear in the log immediately before and after the "unsupported" URLs, but had no luck finding the correct threads. Is there any way to add the thread number to the "unsupported image hoster" line(s) in the log? Example excerpt from recent log file: Code:
2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-3] downloaded: http://thumbnails26.imagebam.com/10709/37484a107089190.jpg to L:\scanned images\mag scans\storage\my threads\t137027-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M\t137027-p1-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M-post-1-1349053\thumb_01-00-16.jpg 2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-3] image-page: http://www.imagebam.com/image/f99d16107089191 2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-1] downloaded: http://img248.imagevenue.com/aAfkjfp01fo1i-24151/loc338/95888_VTS_05_1_123_338lo.JPG to L:\scanned images\mag scans\storage\my threads\t12259-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff\t12259-p1-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff-post-10-2688581\95888_VTS_05_1_123_338lo.JPG 2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-1] image-page: http://img253.imagevenue.com/img.php?image=95915_VTS_05_2_123_83lo.JPG 2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-2] downloaded: http://105.imagebam.com/download/tDZc7e2bY9h3qavGxEVq1A/46337/463366495/drt_idx.jpg to L:\scanned images\mag scans\storage\my threads\t112211-Dirty_Twins_1993_Classic_Movies\t112211-p1-Dirty_Twins_1993_Classic_Movies-post-6-3531474\drt_idx.jpg 2017-07-21 16:53:05 INFO: Unsupported image hoster: http://depic.me/hwh47nrc9daw/vts_01_1.vob.jpg 2017-07-21 16:53:05 INFO: Unsupported image hoster: http://depic.me/lrwwhgzy9f2d/vint.jpg 2017-07-21 16:53:05 INFO: Unsupported image hoster: http://imagezilla.net/show/ittUw2R-Happening.1981.BDrip.1080p.x264.Den.mkv.jpg 2017-07-21 16:53:05 INFO: Unsupported image hoster: http://www.stooorage.com/show/512/993925_mddbakalsm.jpg 2017-07-21 16:53:05 INFO: Unsupported image hoster: http://www.stooorage.com/show/512/993926_mddbakalsm-e.jpg 2017-07-21 16:53:05 INFO: Unsupported image hoster: http://jdownloader.org/download/offline 2017-07-21 16:53:05 INFO: [ForkJoinPool-1-worker-2] image-page: http://img276.imagevenue.com/img.php?image=95568_thumbs20100928200003_123_461lo.jpg 2017-07-21 16:53:07 INFO: PROGRESS: Download images613/702 2017-07-21 16:53:10 SEVERE: [ForkJoinPool-1-worker-3] download: http://25.imagebam.com/download/lgTO7LT7A9DP603kLjOaCQ/10709/107089191/01-04-06.jpg failed HTTP/1.1 404 Not Found 2017-07-21 16:53:12 INFO: [ForkJoinPool-1-worker-3] downloaded: http://thumbnails25.imagebam.com/10709/f99d16107089191.jpg to L:\scanned images\mag scans\storage\my threads\t137027-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M\t137027-p1-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M-post-1-1349053\thumb_01-04-06.jpg 2017-07-21 16:53:12 INFO: [ForkJoinPool-1-worker-3] image-page: http://www.imagebam.com/image/2de7bd107089192 2017-07-21 16:53:12 INFO: [ForkJoinPool-1-worker-1] downloaded: http://img253.imagevenue.com/aAfkjfp01fo1i-30177/loc83/95915_VTS_05_2_123_83lo.JPG to L:\scanned images\mag scans\storage\my threads\t12259-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff\t12259-p1-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff-post-10-2688581\95915_VTS_05_2_123_83lo.JPG 2017-07-21 16:53:12 INFO: [ForkJoinPool-1-worker-1] image-page: http://img261.imagevenue.com/img.php?image=95915_VTS_06_1_123_27lo.JPG 2017-07-21 16:53:15 INFO: [ForkJoinPool-1-worker-3] downloaded: http://33.imagebam.com/download/k2XuxOj6UPUsx6RqJh4EKA/10709/107089192/01-06-01.jpg to L:\scanned images\mag scans\storage\my threads\t137027-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M\t137027-p1-Anal_DP_Blonde_A_Connie_Peterson_Collection_ABA_M-post-1-1349053\01-06-01.jpg 2017-07-21 16:53:16 INFO: [ForkJoinPool-1-worker-3] image-page: http://pimpandhost.com/image/show/id/5486279 2017-07-21 16:53:17 INFO: PROGRESS: Download images616/702 2017-07-21 16:53:19 INFO: [ForkJoinPool-1-worker-1] downloaded: http://img261.imagevenue.com/aAfkjfp01fo1i-4227/loc27/95915_VTS_06_1_123_27lo.JPG to L:\scanned images\mag scans\storage\my threads\t12259-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff\t12259-p1-The_History_of_the_Blue_Movie_1970_LoopsOld_Stuff-post-10-2688581\95915_VTS_06_1_123_27lo.JPG t116096-Expectations_aka_JOse_tout_1977_Classic_Movies This thread was last processed four days earlier, so the "unsupported image hoster" lines in the log were clearly part of a retry, with no easy way to link the images to the thread where they belong.
__________________
Please PM me if any of my links are dead. Don't post your thanks, hit the "Thanks" button! To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. ... To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. ... To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. |
July 23rd, 2017, 06:10 AM | #158 | |
Blocked!
Join Date: Jan 2008
Location: HH
Posts: 1,963
Thanks: 115,040
Thanked 32,801 Times in 1,955 Posts
|
Quote:
https://git.teknik.io/halvar/forum-backup/issues/23 I assume you are using the old tool imagerescue? Then you can generate report csv file by clicking on the button "Create Report of local files". This parses all local html files and creates a csv file in your storage folder. This file contains a line for every image link found including thread id and image url. It may take some time for your 500 threads. You can lookup your info there. The new forum-backup does not have this yet. I am working on a web console with a report feature. If you know sql you could do your own queries too. Here is a sample report: I am planning to create a new release including this next weekend. |
|
July 23rd, 2017, 09:27 AM | #159 | |
Administrator
Join Date: Dec 2006
Location: Rocinante Ops Deck
Posts: 13,918
Thanks: 114,775
Thanked 425,825 Times in 13,444 Posts
|
Quote:
Update: I used forum-backup 0.0.2 batch file "create-report.bat" and the new report does list all the image links, including the unsupported image hosters. So that made it possible to find the correct threads for the mystery images. Cheers.
__________________
Please PM me if any of my links are dead. Don't post your thanks, hit the "Thanks" button! To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. ... To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. ... To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts. |
|
July 23rd, 2017, 09:46 AM | #160 |
Vintage Idiot
Join Date: Feb 2012
Location: History
Posts: 22,136
Thanks: 226,724
Thanked 356,768 Times in 21,631 Posts
|
vef-imagerescue can generate a listing of threads that are within an input forum. Can similar functionality be added to forum-backup? Either, similar to before: a) generate a file listing that can later be used as an input for the app., or: b) simply feed the app. with the forum id, and it then processes all the threads found within that forum?
Such functionality would be particularly valuable for the model id request section where there's lots of small threads. |
|
|