Vintage Erotica Forums

Vintage Erotica Forums (http://vintage-erotica-forum.com/index.php)
-   Image Hosts (http://vintage-erotica-forum.com/forumdisplay.php?f=88)
-   -   Discussion of the download / upload tools (http://vintage-erotica-forum.com/showthread.php?t=333326)

halvar June 15th, 2017 12:49 PM

Quote:

Originally Posted by effCup (Post 4068886)
Relatively minor but unfortunate issue: halvar's tool (1.13) for imgbox images, seems to (at least sometimes?) not get/use the image filenames? Example post here.

I'm afraid the original filename is not available on imgbox. At least I didn't find it.

Now that I think of it I could number the images in the order they appear in the post.

I think on 1st of July we will have a perfect tool....

I am currently planning
  • a small release soon with the possibility to specify more than one thread id in the GUI. They will be processed one after another. This is a minor change that should not break anything.
  • a larger release later which includes a report generation of downladed files as a csv file. The report could contain: thread-id, thread-titlepostNr, pageNr, thumbnailLink, ImageLink, sucess-or-fail, etc. This could be filtered and aggregated using excel. This would be useful for checking for missed downloads, errors, etc. And it coud be used to number files in the order of the post.
    The report would not download anything, just work on local files.

effCup June 15th, 2017 01:09 PM

Quote:

Originally Posted by halvar (Post 4068932)
I'm afraid the original filename is not available on imgbox. At least I didn't find it.

if one manually downloads those images I linked above imgbox names them correctly.

In the page source from imgbox, there is only one location where the image filename is stored, in the "title" attribute:
Code:

<div class="image-container">
        <img alt="Xnrwelb8" class="image-content" id="img" onclick="rs()" src="https://i.imgbox.com/XNRWELb8.jpg" title="639m.jpg" />
</div>


effCup June 15th, 2017 01:55 PM

I hesitate to suggest this but: one improvement for halvar's tool would be multi-threading the downloads like image host grabber & similar, e.g. max. 5 images at once. At present when a single image is "sticky" it halts all the downloads.

Just floating the idea. Do not think/try if it would delay other stuff.

halvar June 15th, 2017 01:56 PM

new version of vef-imagerescue (1.14) : imgbox filenames are now preserved
 
Quote:

Originally Posted by effCup (Post 4068944)
if one manually downloads those images I linked above imgbox names them correctly.

In the page source from imgbox, there is only one location where the image filename is stored, in the "title" attribute:
Code:

<div class="image-container">
        <img alt="Xnrwelb8" class="image-content" id="img" onclick="rs()" src="https://i.imgbox.com/XNRWELb8.jpg" title="639m.jpg" />
</div>


You are right, shame on me I didn't see this.

I created a new Version 1.14: https://1fichier.com/?mnvu0sfpwq
  • The GUI allows specification of multiple thread ids separated by space, e.g. '718 2900'. These threads are processed one after another.
  • The filenames of imgbox Images is now preserved.

halvar June 15th, 2017 02:06 PM

Quote:

Originally Posted by effCup (Post 4068966)
I hesitate to suggest this but: one improvement for halvar's tool would be multi-threading the downloads like image host grabber & similar, e.g. max. 5 images as once. At present when a single image is "sticky" it halts all the downloads.

Just floating the idea. Do not think/try if it would delay other stuff.

It is a good idea, the problem is that multi-threading is a bit complicated and error prone. There are of course features in the newer Java-Versions like the Excutor Framework, Blocking-Queues and so on.
I thought of putting single download-jobs in a queue and then have multiple threads process them parallel.

I have implemented this kind of stuff in business-applications and it is not that hard to implement. But nonetheless tricky. Given the restricted amount of time I opted for the more safe, single threaded route.

I use the command line version and create scripts that download thread after thread. So I can run it unattended over night.

effCup June 15th, 2017 02:27 PM

Yes. Multi-threading would also make it more difficult to manually find which images are being "sticky" and so need browser-viewing to "un-sticky" them.;) Suggestion withdrawn.:)

edit: maybe commenting too early but 1.14 seems perhaps more resilient re. downloading "sticky" images? Could be server loads/time of day, etc. /shrug/ Seems smoother; gets stuck but more often than before also un-sticks without needing intervention of a stop & restart.
:cool::thumbsup:

BCFC_1982 June 15th, 2017 02:32 PM

Great stuff again hal. Is there a limit to the number of threads the new tool can process?

halvar June 15th, 2017 02:51 PM

Quote:

Originally Posted by BCFC_1982 (Post 4069005)
Great stuff again hal. Is there a limit to the number of threads the new tool can process?

Theoretically no. But I do not know if there are memory leaks, so it could be getting slower and stop completely if too much threads are processed without restarting the program.
I was thinking about a larger input field, but feared that somebody might enter hundreds of threads. That could and should work, but I am not really sure.

I think you should be safe with 10 thread ids, but it may also depend on the thread size.

An alternative is to use the command line. Just create a batch file (or shell script if you have a real operating system ;) ) with subsequent calls. Simple but it works. This also processes one after another and restarts the program ever time. So memory leaks are not an issue.

e.g
Code:

java -jar vef-imagerescue-1.14.jar 197832 user password
java -jar vef-imagerescue-1.14.jar 6702 user password
java -jar vef-imagerescue-1.14.jar 2690 user password
java -jar vef-imagerescue-1.14.jar 9853 user password
...


halvar June 15th, 2017 02:55 PM

Quote:

Originally Posted by effCup (Post 4068999)
Yes. Multi-threading would also make it more difficult to manually find which images are being "sticky" and so need browser-viewing to "un-sticky" them.;) Suggestion withdrawn.:)

edit: maybe commenting too early but 1.14 seems perhaps more resilient re. downloading "sticky" images? Could be server loads/time of day, etc. /shrug/ Seems smoother; gets stuck but more often than before also un-sticks without needing intervention of a stop & restart.
:cool::thumbsup:

I also thinks it is because of different day times and server loads.
Stop and Restart shouldn't really help. Best let it finish first and then restart once.

BCFC_1982 June 15th, 2017 06:06 PM

Hi hal - great stuff again :)

Just a report mate:

Thread: 260294

2017-06-15 18:29:55 INFO: image-page: http://vintage-erotica-forum.com/sho...ghlight=farrah
2017-06-15 18:29:56 SEVERE: Error downloading 'http://vintage-erotica-forum.com/showthread.php?t=22941&highlight=farrah'
2017-06-15 18:29:56 SEVERE: Illegal char <?> at index 143: C:\vef-imagerescue-1.14\storage\t260294-Teazer_Magazine_Magazine_Scans\t260294-p1-Teazer_Magazine_Magazine_Scans-post-5-3137119\.showthread.php?t=22941&highlight=farrah

It seems to be picking up the thread link contained in the post?



All times are GMT. The time now is 09:02 PM.



vBulletin Optimisation provided by vB Optimise v2.6.1 (Pro) - vBulletin Mods & Addons Copyright © 2024 DragonByte Technologies Ltd.