Discussion of the download / upload tools

[halvar]

Originally Posted by BCFC_1982

Thread: 260294

2017-06-15 18:29:55 INFO: image-page: http://vintage-erotica-forum.com/sho...ghlight=farrah
2017-06-15 18:29:56 SEVERE: Error downloading 'http://vintage-erotica-forum.com/showthread.php?t=22941&highlight=farrah'
2017-06-15 18:29:56 SEVERE: Illegal char <?> at index 143: C:\vef-imagerescue-1.14\storage\t260294-Teazer_Magazine_Magazine_Scans\t260294-p1-Teazer_Magazine_Magazine_Scans-post-5-3137119\.showthread.php?t=22941&highlight=farrah

It seems to be picking up the thread link contained in the post?

You are right. I scan for the links with embedded images. If the embedded image points to imagebam or imgbox it is assumed to be a thumbnail and the link pointing to the image.
e.g.

<a href="http://some-url"><img src="...imagebam..."></a>

It does not check if the href also points to imagebam or imgbox. So if some-url points anywhere else, anywhere else is downloaded. In your case that lead to an error because the file could not be saved because of the '?' in the filename.
I will add a check to the href-link also.

[BCFC_1982]

Only experienced it once. Other threads with links in seem to have escaped this behaviour

Thanks again for your hard work and patience. V1.14 is getting its share of love

[effCup]

Originally Posted by halvar

I could print a line for each thread of a forum, like
thread id, thread title, number of pages

Having thread title as well as id would be particularly good if (as at present) there are forums with lots of threads that still need to be "processed", but some threads in that same forum have already been (manually/etc.) processed.

That is: machines only want thread ids, but humans benefit from thread titles, if they need to "pick out"/exclude some thread ids from a list to be later processed.

Having number of pages may also be useful as a possible rough gauge of thread size (i.e. for consideration of capacity required for image backups), but that can also already be seen by members just using vef. /shrug/

The version of your tool that you release on 1 July will be the greatest ever.

Thanks so much for all your work already done.

[deezer]

I don't really understand all the techincal stuff you are talking about:-)
But I made a suggestion some days ago, how to organize, that most/all stuff will be saved:

Originally Posted by deezer

5. Organization of saving posts (of inactive members/or by thread)

This could be done in two alternive ways (mods decision):
a) a table with member-names, which posts should be saved
b) a table with threads, which should be saved

On top of the thread is the table.
- Members posting in following, which thread/member-posts they want to save.
- when a member posts, that he save a thread/Members-posts, the table will be edited, that everybody knows, which tasks are still to do.
- when a member saved a thread/members-post, he post a notice, so that the table will be edited, and this task marked as done.


Quoting my old post:

(I don't know if it is possible to set up such a list, because there are quite a lot of threads/inactive-members at the board. I think this table can't be set up manually. But that's something you know better.)

If you think organization by saving by threads is better, maybe this thread could be splitted into the various sections/subsections of the forum (Modern/Classic/Vintage/Mir)

Rethinking this, only option is listing by thread not by poster, because halvard's tool is working by thread.

Problem is, it is definitely not possible to update the table manually, marking all threads which has saved by whom.

Question to you techies: Do you know, how to generate such a table, maybe top of a thread in the rescue-forum.
And each member, who saved a thread, is posting the thread-number in this thread and the table in the first post gets updated, marking this thread as done?

Hope this is understandable:-)

[halvar]

get it from here: https://1fichier.com/?vj1t51ndwf

changes to 1.14:

• bugfix: if the enclosing link of a thumbnail points to another host than the thumbnail the link is now ignored. The link is not intended to be an image link.
  

  Originally Posted by BCFC_1982

  2017-06-15 18:29:55 INFO: image-page: http://vintage-erotica-forum.com/sho...ghlight=farrah
  2017-06-15 18:29:56 SEVERE: Error downloading 'http://vintage-erotica-forum.com/showthread.php?t=22941&highlight=farrah'
  2017-06-15 18:29:56 SEVERE: Illegal char <?> at index 143: C:\vef-imagerescue-1.14\storage\t260294-Teazer_Magazine_Magazine_Scans\t260294-p1-Teazer_Magazine_Magazine_Scans-post-5-3137119\.showthread.php?t=22941&highlight=farrah
  It seems to be picking up the thread link contained in the post?

• enhancement: create a csv file of threads for a forum. There is a new tab 'Create Thread List..." in the GUI. If you specify a forum id and a maximum number of pages then it fetches all threads for the forum. The threads are written in the log view and in a csv-file in the working directory. The threads are sorted like in the default view: descending by last post date.
  

  Originally Posted by effCup

  My problem/query: can you create/assemble such a queue of threads from a single vef forum? Do you understand the question I'm asking?

I have created a thread list for
Classic Pornstars: https://1fichier.com/?ho1inxdp93
Modern Models: https://1fichier.com/?duocxmunzy

Edit:
Concerning the csv:
The csv is separated by ';' and contains thread id, thread titel, number of pages
I just noticed a little bug: if the thread has 2 or 3 pages the page count is stated as Optional[2] or Optional[3]

[effCup]

@halvar: for your csv listing of threads in a forum, instead of listing the number of pages in each thread, would it be at all easy to instead list the number of posts (or "replies", whichever is easier)? That would be tremendously helpful for the model id request section, as we're using a non-default sort order for processing stuff (by number of replies, descending). I'm not asking you to mimic that sort order. If your listing showed no. of posts in each thread rather than no. of pages then I could easily re-sort the resulting csv to get precisely what I want. Then I can "trim off" the "top" rows that are already processed, and proceed.

Thanks.

[BCFC_1982]

Thank halvar. Not me.

WinRAR associates Jar files with it so when you double click it opens it in WinRAR.

Right click the jar file, click open with, then scroll down the programs looking for Java(TM) SE binary. That's the program you need to open the tool.

I know because I had exactly the same problem as that!

Hope this helps.

[effCup]

Originally Posted by halvar

Stop and Restart shouldn't really help. Best let it finish first and then restart once.

I agree, it shouldn't, but it does seem to. Note that it's not simply stop & restart--I'm loading the sticky image in a browser to try and un-sticky it, in between.

I perhaps should have mentioned before: this behaviour is far from new or confined to your tool. Plenty of times using image host grabber or just a browser window the downloading of an image can often become "stuck", and one has to try to re-trigger that download. That's been the case for months/years.

One factor--other than possible image host "flakiness" in their server arrays--may be ISP caching. No, not my browser cache on my computer but rather the large-scale caching of content by isps to "speed" the internet's response/latency. It is a bigger factor in some countries than in others. Much of vef's content may not be already cached by my ISP. /shrug/

I know from countless practical experience that simply leaving such downloads to "sort themselves out" seldom resolves the issue. I can leave your tool for half an hour & it sometimes may have downloaded only 2-3 images. That's no fault at all of your work.

No, I'm not using a slow dial-up connection or even copper broadband, I'm using fibre--65 Mbps theoretical max; ~34.31 Mbps practical test. Though note that fibre speeds are relatively inconsequential for small files such as jpgs; it's only on streaming or large file downloads that one really notices.

While writing this post I have had to stop, load image, & re-start your tool maybe 20+ times for a single thread of 6 posts (but 2 of them with lots of -bam/-box images). Untimed but it has taken several minutes to complete.

edit: hmm, no, 65Mbps is just the local modem/router's potential. My ISP "sells" this fibre as a 30Mbps package so speeds slightly above that are pretty reasonable. There's also 100Mbps packages, some have 200, etc.

[halvar]

download it here: https://1fichier.com/?ejvhsikaim

changes to 1.15:

• Added reply-count to the thread list. That was easier than to figure out the number of posts.
  

  Originally Posted by effCup

  for your csv listing of threads in a forum, instead of listing the number of pages in each thread, would it be at all easy to instead list the number of posts (or "replies", whichever is easier

• Added a report feature. It creates a csv file containing information on the downloaded items. It can be useful to check for correctness of downloads. The report only processes local files, it does not download anything. To start it press the button "Create report...". Creating the report may take some time, depending on the amount of files you have in your storage. An example report is here: https://1fichier.com/?4pzby8ycft
• the core downloading feature remains untouched so that should work as before

The report will the basis for fixing downloaded items:

• Renaming post folders without post number to contain the post number
• Renaming downloaded imgbox files that do not have the original file name

I will integrate these fixings in a future release.

[deezer]

Halvar, many thanks for your effort.
you created a great tool!
I don't use it at the moment for downloading by myself. But I use the files that BCFC downloaded for re-hosting. It's great!

May I ask you something which is probably only a point in the MIR-section.
Sometimes galleries, which are not posted elsewhere at VEF are direct linked in a MIR-post without a thumbnail-preview. In a V 1.9 the tool didn't download this galleries.
Does it download such direct linked galleries in V 1.14 or V 1.16?