Discussion of the download / upload tools

[halvar]

https://1fichier.com/?g7rl5vpbd7

changes to 1.23:
bugfix: thumbnails are not downloaded twice (thumb_xxx.jpg and thumb_xxx_1.jpg)

[deezer]

One minor remark: I always check the csv-report after downloading a batch, for the images with "n/a; -1".

Only once, the tool missed the dl of an image, which I could grabb manually.

But now I've seen following:
There is a file-name given in the report and a file-size "0".
The tool saved an image with a file size 0, but not the thumb, which is available.
The image isn't available for dl.

This post:

http://vintage-erotica-forum.com/showpost.php?p=3293177

289896;solved_Monique_ATK_Super_wow_busty_blonde_unknown;1;1;3293177;bannator;http://thumbnails106.imagebam.com/41247/da03fa412465077.jpg;http://www.imagebam.com/image/da03fa412465077;superb-big-boobs-in-a-picture-jpg980.jpg;0

I assume, the tool gets a filename and a dl, even when it is size zero, so there is not reported : "n/a; -1", but "filename; 0".
So I will check in the future the file-size "0" in the reports, to dl the thumb manually.

[halvar]

Originally Posted by deezer

One minor remark: I always check the csv-report after downloading a batch, for the images with "n/a; -1".

Only once, the tool missed the dl of an image, which I could grabb manually.

But now I've seen following:
There is a file-name given in the report and a file-size "0".
The tool saved an image with a file size 0, but not the thumb, which is available.
The image isn't available for dl.

This post:

http://vintage-erotica-forum.com/showpost.php?p=3293177

289896;solved_Monique_ATK_Super_wow_busty_blonde_unknown;1;1;3293177;bannator;http://thumbnails106.imagebam.com/41247/da03fa412465077.jpg;http://www.imagebam.com/image/da03fa412465077;superb-big-boobs-in-a-picture-jpg980.jpg;0

I assume, the tool gets a filename and a dl, even when it is size zero, so there is not reported : "n/a; -1", but "filename; 0".
So I will check in the future the file-size "0" in the reports, to dl the thumb manually.

Thanks for the thorough checking.
The error detection is not 100% reliable, but should be more than 99.9%. There are many things in networking that can go wrong and sometimes something goes wrong without an error message.
So there still could be corrupt files with a size > 0. I thought about decoding each downloaded file to make sure the files are valid, but that would slow things down.

I regularly browse through many of the files and have not encountered a corrupt file.

[deezer]

Originally Posted by halvar

Thanks for the thorough checking.
The error detection is not 100% reliable, but should be more than 99.9%. There are many things in networking that can go wrong and sometimes something goes wrong without an error message.
So there still could be corrupt files with a size > 0. I thought about decoding each downloaded file to make sure the files are valid, but that would slow things down.

I regularly browse through many of the files and have not encountered a corrupt file.

As I've said, it's only a minor problem, better said: It's absolutely no problem!
I only wanted to bring this to your attention.

At the moment I completely rely at your tool.
Downloading, making the report and checking the -1 or 0 files. Btw, I've seen this "0" file only once.

But I'm absolutely happy with the tool as it is now!

[halvar]

https://1fichier.com/?6kam44b708

This release contains two functions to fix issues with old downloads (before 1.7 and before 1.15). The download functionality remains unchanged.

Fix missing imgbox filenames for downloads before version 1.15 (released on 06-16-2017):
Imgbox downloads were not set to the original filename. The filename was the cryptic hash (e.g. VAESDLAd). This fix scans through the whole storage folder and checks images downloaded from imgbox if they are still named after the hash. If that is the case, the imgbox web page for the image is fetched and the original filename is extracted. The image file is renamed and the new filename is written to the dot-file.
To start this process press the button "Fix Imgbox file names" on the second tab.

Add the postnumber to post folders (e.g. rename tnnn-...-post-1 to tnnn-...-post-1-nnnn) for downloads before version 1.7 (released on 06-11-2017):
Local html files in the storage are scanned for posts. Post count and post number are extracted. If the corresponding post folder's name does not contain the post number, the post folder is renamed. This process does not download anything, it only works on local files. The post number might be necessary for a possible (half-) automatic post update tool.
To start this process press the button "postnr for folders before 1.7" on the second tab.

If you have a large storage folder these processes take a long time. I recommend temporarily moving the affected thread folders in another storage folder and moving them back afterwards.

[halvar]

Originally Posted by Pavlov49

1. Can I remove all these from my HDD?
2. What happens to later downloads of the same sites, if this "scrap" was removed? Would all pix be downloaded again if the tool did not find the scrapped files?
3. May there be a chance to incorporate other hosters in later versions?
   We just have overcome a potential desaster with ImageBam and ImageBox. WHAT if the next would be ImageVenue, PixHost and all the other preferred ones?

You are right, if you delete the scrap files, the images will be downloaded again. So you should keep them. Using files is the easiest way to note which links and images are already processed. And they connect the link in the post with the image file. The name is taken from the link and the file contains the real image name. This connection would be necessary to create a post repair tool. I could use an embedded database, but then it would be harder for humans to extract information without tool support.

I am currently rebuilding the tool from scratch. It will support most of the hosters. The new version will hopefully be easier to use. I intend to distribute a Java Runtime Environment and a batch file with it. So you do not need to install Java.

The basic principle will stay the same. It will work per forum thread and save the files in the same directory structure.

But there will be no GUI, only command line and configuration files. I am running it nightly on my NAS using cron so there is no GUI.

Maybe the scrap files will be replaced with an embedded database. I am still thinking about it. As long as you have the scrap files the data can be imported to the database. And a database could make it more manageable if you have more than a million files.

Maybe the next version can be released next next weekend.

[halvar]

Originally Posted by Pavlov49

In fact I am scared about the missing GUI .
I am no command line guy, haven't ever been.

Don't worry it will be easy. Just create two textfiles:

one file for general configuration values, e.g.

baseurl=http://vintage-erotica-forum.com
username=<your vef username>
password=<your vef password>
storagepath=<your path where you store your downloads, like 'c:/vef/storage'>

and one file with your thread ids:

12345
6789
33333

And then double click a batch file to start. This is more convenient than entering the values in a GUI. At least that's the plan.

The general idea is to define a list of threads that you like and then start the backup periodically to get new posts.

Problems and limitations

• It doesn't catch post edits
• moved or deleted posts
• ...

[effCup]

Originally Posted by Pavlov49

a folder structure such like:
Z:\_Pix\VEF-Rescue\

t3146-Ginger_Lynn_Classic_Pornstars\

t3146-p4-Ginger_Lynn_Classic_Pornstars-post-34-249738\

96458_676211_354be8a7724_122_699lo.jpg

@halvar: can I suggest/request instead:

Z:\_Pix\VEF-Rescue\

t3146-Ginger_Lynn_Classic_Pornstars\

p4-post-34-249738\

96458_676211_354be8a7724_122_699lo.jpg

.... or some such?

[deezer]

Originally Posted by effCup

@halvar: can I suggest/request instead:

p4-post-34-249738\

Is there a need for the pagenr. ("p4") and the post nr. ("post-34")?

Pagenr. depends on the settings.
Page nr. and post nr. can change if other posts gets moved/deleted.

Wouldn't it be sufficient to have the postID ("p249738"), which doesn't change?

[effCup]

Originally Posted by deezer

Is there a need for the pagenr. ("p4") and the post nr. ("post-34")?

Pagenr. depends on the settings.
Page nr. and post nr. can change if other posts gets moved/deleted.

Wouldn't it be sufficient to have the postID ("p249738"), which doesn't change?

I agree, but when discussed here earlier there was the possible "human reassurance" factor. /shrug/ As you say, the others are not canonical.

edit: if we really wanted to save typing we might ask for: p=249738