Discussion of the download / upload tools

[halvar]

Originally Posted by effCup

Relatively minor but unfortunate issue: halvar's tool (1.13) for imgbox images, seems to (at least sometimes?) not get/use the image filenames? Example post here.

I'm afraid the original filename is not available on imgbox. At least I didn't find it.

Now that I think of it I could number the images in the order they appear in the post.

I think on 1st of July we will have a perfect tool....

I am currently planning

• a small release soon with the possibility to specify more than one thread id in the GUI. They will be processed one after another. This is a minor change that should not break anything.
• a larger release later which includes a report generation of downladed files as a csv file. The report could contain: thread-id, thread-titlepostNr, pageNr, thumbnailLink, ImageLink, sucess-or-fail, etc. This could be filtered and aggregated using excel. This would be useful for checking for missed downloads, errors, etc. And it coud be used to number files in the order of the post.
  The report would not download anything, just work on local files.

[effCup]

Originally Posted by halvar

I'm afraid the original filename is not available on imgbox. At least I didn't find it.

if one manually downloads those images I linked above imgbox names them correctly.

In the page source from imgbox, there is only one location where the image filename is stored, in the "title" attribute:

<div class="image-container">
	<img alt="Xnrwelb8" class="image-content" id="img" onclick="rs()" src="https://i.imgbox.com/XNRWELb8.jpg" title="639m.jpg" />
</div>

[effCup]

I hesitate to suggest this but: one improvement for halvar's tool would be multi-threading the downloads like image host grabber & similar, e.g. max. 5 images at once. At present when a single image is "sticky" it halts all the downloads.

Just floating the idea. Do not think/try if it would delay other stuff.

[halvar]

Originally Posted by effCup

if one manually downloads those images I linked above imgbox names them correctly.

In the page source from imgbox, there is only one location where the image filename is stored, in the "title" attribute:

<div class="image-container">
	<img alt="Xnrwelb8" class="image-content" id="img" onclick="rs()" src="https://i.imgbox.com/XNRWELb8.jpg" title="639m.jpg" />
</div>

You are right, shame on me I didn't see this.

I created a new Version 1.14: https://1fichier.com/?mnvu0sfpwq

• The GUI allows specification of multiple thread ids separated by space, e.g. '718 2900'. These threads are processed one after another.
• The filenames of imgbox Images is now preserved.

[halvar]

Originally Posted by effCup

I hesitate to suggest this but: one improvement for halvar's tool would be multi-threading the downloads like image host grabber & similar, e.g. max. 5 images as once. At present when a single image is "sticky" it halts all the downloads.

Just floating the idea. Do not think/try if it would delay other stuff.

It is a good idea, the problem is that multi-threading is a bit complicated and error prone. There are of course features in the newer Java-Versions like the Excutor Framework, Blocking-Queues and so on.
I thought of putting single download-jobs in a queue and then have multiple threads process them parallel.

I have implemented this kind of stuff in business-applications and it is not that hard to implement. But nonetheless tricky. Given the restricted amount of time I opted for the more safe, single threaded route.

I use the command line version and create scripts that download thread after thread. So I can run it unattended over night.

[effCup]

Yes. Multi-threading would also make it more difficult to manually find which images are being "sticky" and so need browser-viewing to "un-sticky" them. Suggestion withdrawn.

edit: maybe commenting too early but 1.14 seems perhaps more resilient re. downloading "sticky" images? Could be server loads/time of day, etc. /shrug/ Seems smoother; gets stuck but more often than before also un-sticks without needing intervention of a stop & restart.

[BCFC_1982]

Great stuff again hal. Is there a limit to the number of threads the new tool can process?

[halvar]

Originally Posted by BCFC_1982

Great stuff again hal. Is there a limit to the number of threads the new tool can process?

Theoretically no. But I do not know if there are memory leaks, so it could be getting slower and stop completely if too much threads are processed without restarting the program.
I was thinking about a larger input field, but feared that somebody might enter hundreds of threads. That could and should work, but I am not really sure.

I think you should be safe with 10 thread ids, but it may also depend on the thread size.

An alternative is to use the command line. Just create a batch file (or shell script if you have a real operating system ) with subsequent calls. Simple but it works. This also processes one after another and restarts the program ever time. So memory leaks are not an issue.

e.g

java -jar vef-imagerescue-1.14.jar 197832 user password
java -jar vef-imagerescue-1.14.jar 6702 user password
java -jar vef-imagerescue-1.14.jar 2690 user password
java -jar vef-imagerescue-1.14.jar 9853 user password
...

[halvar]

Originally Posted by effCup

Yes. Multi-threading would also make it more difficult to manually find which images are being "sticky" and so need browser-viewing to "un-sticky" them. Suggestion withdrawn.

edit: maybe commenting too early but 1.14 seems perhaps more resilient re. downloading "sticky" images? Could be server loads/time of day, etc. /shrug/ Seems smoother; gets stuck but more often than before also un-sticks without needing intervention of a stop & restart.

I also thinks it is because of different day times and server loads.
Stop and Restart shouldn't really help. Best let it finish first and then restart once.

[BCFC_1982]

Hi hal - great stuff again

Just a report mate:

Thread: 260294

2017-06-15 18:29:55 INFO: image-page: http://vintage-erotica-forum.com/sho...ghlight=farrah
2017-06-15 18:29:56 SEVERE: Error downloading 'http://vintage-erotica-forum.com/showthread.php?t=22941&highlight=farrah'
2017-06-15 18:29:56 SEVERE: Illegal char <?> at index 143: C:\vef-imagerescue-1.14\storage\t260294-Teazer_Magazine_Magazine_Scans\t260294-p1-Teazer_Magazine_Magazine_Scans-post-5-3137119\.showthread.php?t=22941&highlight=farrah

It seems to be picking up the thread link contained in the post?

Originally Posted by Mudmonster

I have acquired another magazine, and have started scanning it for your enjoyment!

Fantastically top heavy Farrah


Mae North



The rest of the mag to follow shortly. Thanks for looking!