How to index a thread?

[deepsepia]

We've got a great little thread that's been going for years

http://vintage-erotica-forum.com/t53...ns-merged.html

. . . and I was wondering how to go about indexing these entries. Manually one can search the thread -- most posters have been good about identifying artists. But it'd be really nice to have an A-Z index of artists, like the very useful work potterstoke did a while back
http://vintage-erotica-forum.com/sho...postcount=2061

. . . but with links to each of the artists

Trying to think of the simplest way to take a list of names and then generate a set of thread links for each of these names . . . any ideas?

[Denaniel]

To make an index, it helps to know a little about BBCode, but if you don't, I can show you the basics you'll need to create a simple index.

If you have an alphabetical list of names, you can easily link each name to a post, so that when you click on the name you are taken automatically to the post with that artist. But if there is more than one post by that artist, and they are not consecutive posts (posts on different pages), then you can add additional links after the name in the index, something like this:

-Link2-Link3-Link4- etc.

so that when you click on any one of those links, you are taken to a different post with that artist's work.


PostID Numbers

First, you need to be able to find the postID number for any post you want to link to. This is not the # in the upper right corner of a post, which is the post count #, and which may change if posts are later deleted or added. The postID number does not change, even if the post is moved to a different thread.

If you click on that little number sign (#) in the upper right corner of any post, it will open a new tab in your browser with that post. You can then get the postID number from the URL in your browser address bar. For example, if I click on the #1 in your post above, I get the following URL from the address bar:

http://vintage-erotica-forum.com/showpost.php?p=5580106&postcount=1

The postID is the number after php?p= and before the next ampersand (&), i.e.

5580106

BTW, if you hover your cursor over that little number sign, some browsers will show the URL and you can skip the step of opening a new tab, if you just remember the postID number in the URL.

To link a name in the list to the post above, I wrap the name with [post] tags, and add the postID like this:

[post=5580106]deepsepia[/post]

Normally, you wouldn't be able to see the [post] tags because the BBCode hides them, adds a hyperlink and underlines the name "deepsepia", but I added some special tags that reveal the BBCode and stop if from functioning inside the tags. They are called noparse tags, btw.

Here is what the same expression looks like without the noparse tags:

deepsepia

Now, if you click on the underlined name "deepsepia" above, it should take you to post #1 above this one.


Sample Index

I made a very short sample index for the first three artists in the Erotic/Pornographic art/prints and illustrations thread:

Namio Harukawa [post=60344]-Link-[/post]
Carlos Zefiro [post=111771]-Link-[/post] +3 following posts
Mihaly Zichy [post=62659]-Link-[/post] +2 following posts


Here's what it looks like without the noparse tags:

Namio Harukawa -Link-
Carlos Zefiro -Link- +3 following posts
Mihaly Zichy -Link- +2 following posts

Now, if you want to get fancy, you can add bold text and colors

[B][COLOR="black"]Namio Harukawa[/COLOR][/B] [post=60344][B][COLOR="red"]-Link-[/COLOR][/B][/post]
[B][COLOR="black"]Carlos Zefiro[/COLOR][/B] [post=111771][B][COLOR="red"]-Link-[/COLOR][/B][/post] [B][COLOR="green"]+3 following posts[/COLOR][/B]
[B][COLOR="black"]Mihaly Zichy[/COLOR][/B] [post=62659][B][COLOR="red"]-Link-[/COLOR][/B][/post] [B][COLOR="green"]+2 following posts[/COLOR][/B]

Namio Harukawa -Link-
Carlos Zefiro -Link- +3 following posts
Mihaly Zichy -Link- +2 following posts

Feel free to adapt it any way you like, of course.

Let me know if you have any questions.

[deepsepia]

Originally Posted by Denaniel

Let me know if you have any questions.

This is a terrific guide to style, a fantastic resource. Wow, many, many thanks. I've done a fair amount of BBcode, but you've got a lot of great ideas here that I haven't thought of before, most grateful!

There was a trickier bit that isn't BBcode, but rather is effectively scraping that I'm trying to figure out. We've got some 2800 posts and ideally what I'd like to be able to do is automate the indexing, that is to step through the posts from 1 to 2800, attaching the titles so that we don't have to do this all by hand.

Ideally, we'd be able to create a database of posts and then generate the index from those posts, if you see what I mean . . . is there a way to do that?

[Denaniel]

Originally Posted by deepsepia

This is a terrific guide to style, a fantastic resource. Wow, many, many thanks. I've done a fair amount of BBcode, but you've got a lot of great ideas here that I haven't thought of before, most grateful!

There was a trickier bit that isn't BBcode, but rather is effectively scraping that I'm trying to figure out. We've got some 2800 posts and ideally what I'd like to be able to do is automate the indexing, that is to step through the posts from 1 to 2800, attaching the titles so that we don't have to do this all by hand.

Ideally, we'd be able to create a database of posts and then generate the index from those posts, if you see what I mean . . . is there a way to do that?

I have no idea how to do that, but there are coders at VEF who might know how. I will send you a PM with a couple suggestions, but if anyone reading this knows how to do it, please speak up! Cheers.

[tedm04]

Originally Posted by deepsepia

There was a trickier bit that isn't BBcode, but rather is effectively scraping that I'm trying to figure out. We've got some 2800 posts and ideally what I'd like to be able to do is automate the indexing, that is to step through the posts from 1 to 2800, attaching the titles so that we don't have to do this all by hand.

Ideally, we'd be able to create a database of posts and then generate the index from those posts, if you see what I mean . . . is there a way to do that?

First, thank you for pursuing this idea, which would be a wonderfully helpful resource.

Is the project that you're contemplating restricted to indexing the titles?

I've noticed that lots of posters ignored the title field, notably Mac1, once the thread's mainstay, who we miss. See, e.g., http://vintage-erotica-forum.com/sho...postcount=1003. And while many of his early posts consisted of random-seeming assortments of unidentified images, his artist-specific posts are possibly the largest single set of such material in the thread. For instance, he put up nearly 40 posts of Becat images.

Although I don't, alas, have a clue how to do it, it might be possible to write a script that scraped the post body, or at least the first line or two of the body, for a specified list of artist names, but there might be an awful lot of manual clean-up. It might be less work to clean up the titles by hand, tedious, of course, but probably not requiring much mental agility. That in turn would require moderator powers and some protocols to avoid and recover from mistakes. (I, for one, am glad that there's nothing I can do that would screw up another member's posts.)

And maybe it doesn't make sense to think about indexing anything but the titles until you see how that works, the better so often being the enemy of the good.

[deepsepia]

Originally Posted by tedm04

First, thank you for pursuing this idea, which would be a wonderfully helpful resource.

Is the project that you're contemplating restricted to indexing the titles?

Yes, at first. I'm looking at Scrapy and "BeautifulSoap" -- two packages designed for web scraping. The first cut would be to simply grab the titles. That won't be perfect, obviously -- people spelled names differently, and we have a lot of titles with no names. But if I can get that, it would be a place to start, anyway.

Originally Posted by tedm04

I've noticed that lots of posters ignored the title field, notably Mac1, once the thread's mainstay, who we miss. See, e.g., http://vintage-erotica-forum.com/sho...postcount=1003. And while many of his early posts consisted of random-seeming assortments of unidentified images, his artist-specific posts are possibly the largest single set of such material in the thread. For instance, he put up nearly 40 posts of Becat images. .

mac1 had a bunch of random images. He also had quite a few archives with many more than the samples from thumbnail galleries, things that were on rapid share or megaupload or some similar defunct archives. I know that I once had them, but I also know that I don't know where they are now-- some drive in a box somewhere probably.

In any event, that's a much harder problem one that needs a lot of hand editing. Very generally, if we got a large pool of random images, one could go through them with Yandex image search and figure out most of them.

but as I say, first things first-- I know we do already have a lot of indexed posts, and we also have potterstroke's very good index to artists and their various 'nicks, see

http://vintage-erotica-forum.com/sho...postcount=2059

-- and subsequent posts. Its quite a helpful checklist.

[deepsepia]

So my first step is to see "how do we scrape a title out of a post", automatically

Looking at our friend Naga's post

http://vintage-erotica-forum.com/sho...postcount=2793

. . . I downloaded it with cUrl to see what I'd get

The key part is we can extract that title, which is Eros art Paris

I've listed just a bit of the code that cUrl generates, filled with table descriptions and other stuff, but what's important for our purposes is that the title text is there, I've edited a lot of it so you can see a bit of what it looks like

<td class="alt1" id="td_post_5569225" style="border-right: 1px solid #8EADAD">
	
		
		
			<!-- icon and title -->
			<div class="smallfont">
				<img class="inlineimg" src="images/icons/icon1.gif" alt="Default" border="0" />
				<strong>Eros art Paris</strong>
			</div>
			<hr size="1" style="color:#8EADAD" />
			<!-- / icon and title -->

So we know that we can both identify a post and scrape a title . . . all a bit of an experiment to learn how this works

One interesting thing is that there are two different "post IDs" -- there's a global post index, which is the "real" post ID, the way the database entry is looked up. There's then a numerical list of posts in a given thread eg, starting from post #1 -- but this isn't how the listing is actually accessed.

If that's confusing, look at

http://vintage-erotica-forum.com/showpost.php?p=5569225&postcount=2793

See that "showpost.php?p=5569225" -- that's the actual "post look up". But when you're in a thread, the number you actually see is the "postcount", in this case 2793 (and the next post will be 2794 and so on . . .)

[BCFC_1982]

If not already stressed, do not rely on the the postcount, that can change and if relied on, you be knackered Always use the Post ID

When you get to your actual index post, you may hit a limit of the number of characters on the post before VEF can’t cope, so be prepared to split the index over a certain number of posts. If including images, a max of 150 can be used per post (please correct me if wrong folks).

[deepsepia]

Originally Posted by BCFC_1982

If not already stressed, do not rely on the the postcount, that can change and if relied on, you be knackered Always use the Post ID

When you get to your actual index post, you may hit a limit of the number of characters on the post before VEF can’t cope, so be prepared to split the index over a certain number of posts. If including images, a max of 150 can be used per post (please correct me if wrong folks).

Thanks for these useful observations-- yes on postcount, definitely.