Managing PM's from forum sites

[aciurczak]

Hi all -

Looking for some suggestions on how to manage PM storage. I know, first option is just delete old ones to free up space and don't worry about them. But let's assume I want to archive them off into a usable format, same as any other email.

On forums that I manage myself, I use this add-on:

http://www.vbulletin.org/forum/showthread.php?t=91269

which allows a user to download their PM's in standard .mbox UNIX mail file format. Many mail programs, including Thunderbird, can import the .mbox files directly, and the messages can immediately be moved into any mail folder, just like any other mail message within the email client.

I'm not suggesting Andy install that mod, as it has limited utility, and in addition there are many forums I'm on where I'd like to manage my PM's, and they all aren't going to install it. Out of the box vBulletin allows you to download all of your PM's as XML, CSV, or TXT. What I would like to do is find some straightforward method to download all messages at once into a file (doesn't matter whether it's XML, CSV, or TXT, but XML is most likely to work), convert the file into some intermediate format that Thunderbird can then import. If I could find an appropriate XML --> .mbox converter for vbulletin PM files, that would likely do the trick.

How does anyone else manage their PM's? Thx in advance for any help or suggestions.

- Alex

 

[McGrude]

Delete the oldest ones.... Sorry that solution sucks for your needs.

 

[aciurczak]

I thought I sidestepped that one with my first comment, but no such luck.

Every once in awhile there's something interesting in my PM's that I'd like to keep, longer than the PM space feasibly allows. If it's in with the rest of my email, I can usually find it pretty quickly with standard search tools, even when looking years later. I could go through PM's 1 by 1 and wipe the chaff, but that seems time-consuming and inelegant, when they really take up so little space.

 

[McGrude]

I thought I sidestepped that one with my first comment, but no such luck.

Every once in awhile there's something interesting in my PM's that I'd like to keep, longer than the PM space feasibly allows. If it's in with the rest of my email, I can usually find it pretty quickly with standard search tools, even when looking years later. I could go through PM's 1 by 1 and wipe the chaff, but that seems time-consuming and inelegant, when they really take up so little space.

Yeah the mbox format kicks ass. It's nearly 30 years old and pretty much everything can either read it directly, or there is a conversion tool. If Andy would just give us the database server hostname and a db login and password we could slurp those messages down...

I don't know of any other solution that would not require scripting up a bunch of shell or perl scripts to do what you need (you can provide login information, including cookies to wget if shell or perl scripting is your thing...). Without support for a download-in-bulk in format X option on the bulletin board the only solution seems to be a manual PITA process.

 

[aciurczak]

It's not that complicated at all, there already is a download in bulk feature. It pulls all PM's into one of those three formats, and saves them all in a single file. In the XML file is all of the easily tagged fields you need. Basically: To, From, Subject, and Message Body. That's why the mbox hack is actually pretty simple, it does the same thing as the existing vbulletin functionality, just slightly tweaks the output format into a file that meets the .mbox format with those same 4 fields.

I just can't find a converter from the XML format --> mbox, and if I ever had the programming skills to whip up a Perl converter, I lost them a long while back.

 

[McGrude]

I may be able to hack something together, but my programming chops are oriented towards PHP right now, so without it running on the server, or you having access to running php on the command line I don't know how useful that would be.

The mbox format is really very simple, and if the XML is easily parsed (for me, that is. I despise XML in almost every application of it. I feel it should be more about data description rather than data transport, transmission or storage) I can probably make some hack that would work, but... It would be a hack.

 

[aciurczak]

If you have a sec, just go to your pm list, scroll to the bottom right, and download that XML file and take a peek at it. It's easy to read, but I know that doesn't mean it's automatically going to be easy to convert. Any hack that you could come up with would be incredibly useful, and it doesn't have to be slick. This is one of those things that people would only use 2 or 3 times per year at max, so it doesn't have to do anything other than work and I think people would be reasonably ecstatic.

 

[aciurczak]

Looking at the different formats, I'm starting to believe that converting from the .txt format may be even easier than the XML format. Here's an example what comes out in the txt file:

================================================================================
From : aciurczak
To : gwade
Date : 2004-11-24 22:09
Title : hey - can you click on chat?
--------------------------------------------------------------------------------
garrett - click on chat; looks like we both went bike shopping today...

================================================================================
From : aciurczak
To : gwade
Date : 2004-11-25 13:13
Title : sent you an email...
--------------------------------------------------------------------------------
garrett - sent you an email through this board; didn't know if PM or email was preferred for you... Trying to confirm whether you're riding fri, sat , or both. From this thread:

https://www.southbayriders.com/forums/showthread.php?t=9533

looks like 9:15 AM on saturday at the Page Mill park n ride wouldn't be a bad place to be, right?

================================================================================

Basically just parsing "From :", "To :", "Date :", "Title :", and then the message body, with ============= as the separator in between messages...

 

[McGrude]

The MBOX format looks like the following in it's most basic form :

From user@example.com  Mon Mar  9 23:44:28 2009
Date: Mon, 9 Mar 2009 23:44:27 -0700 (PDT)
From: McGrude <user@example.com>
To: user@example.com
Subject: test

test

Each message begins with From (note no colon) then the from address and date/time
Then Date:, From:, Subject: an empty line (well.. only a new line) then the message body begins. The next message is delimited by another line starting with "From " Note again, no colon, just a space.

If "From " begins a line of a message body it is replaced with ">From "

E.g.

From user@example.com  Mon Mar  9 23:47:26 2009
Date: Mon, 9 Mar 2009 23:47:25 -0700 (PDT)
From: McGrude <user@example.com>
To: user@example.com
Subject: test

>From

 

[aciurczak]

Found a free program that imports the XML file from vBulletin PM downloads, and provides its own viewer and navigator through the offline file. Works well enough, but it doesn't have an export feature into mbox or anything else, can only export individual messages as HTML, which doesn't help me get all of the PM's into the same archive as older email.

 

[Curt]

txt2mbox: an SBR utility by Curt

No warranty express or implied

#!/usr/bin/perl

use strict;
use warnings;
use POSIX;
use open IN => ':crlf';

sub usage {
    print STDERR "Usage: txt2mbox sbr-messages.txt\n";
    print STDERR "   Writes one output file per SBR folder.\n";
    exit 1;
}

&usage if ($#ARGV != 0);

my $infile = shift;
open(INFILE, "<:crlf", $infile) ||
    die "Could not open $infile: $!";

my $folder = '';
my $folder_boundary = '#' x 80;
my $header_boundary = '=' x 80;
my $message_boundary = '-' x 80;
my $pushback = '';
my $msgno = 0;

while (1) {
    if ($pushback) {
	$_ = $pushback;
	$pushback = '';
    } elsif ($_ = <INFILE>) {
	chomp;
    } else {
	last;
    }
    if (/^${folder_boundary}$/) {
	# Close current output file, if any
	if ($folder) {
	    close FOLDER;
	    $folder = '';
	}
	# Switch output file to new folder name
	$_ = <INFILE>; chomp;		# Get line with folder name
	die "Expected folder name"
	    if !/^Folder :\t(.*)$/;
	$folder = $1;
	print "Writing folder $folder\n";
	$_ = <INFILE>; chomp;		# Skip folder end boundary
	$_ = <INFILE>; chomp;		# Skip blank line
	open(FOLDER, ">$folder") ||
	    die "Could not open output file $folder: $!";
    } elsif (/^${header_boundary}$/) {
	die "Message not in folder" if (!$folder);
	# Read header lines From, To, Date, Title
	$_ = <INFILE>; chomp;
	die "Missing From"
	    if !/^From :\t(.*)$/;
	my $hdr_from = $1;
	$_ = <INFILE>; chomp;
	die "Missing To"
	    if !/^To :\t(.*)$/;
	my $hdr_to = $1;
	$_ = <INFILE>; chomp;
	die "Missing Date"
	    if !/^Date :\t(.*)$/;
	my $hdr_date = $1;
	$_ = <INFILE>; chomp;
	die "Missing Title"
	    if !/^Title :\t(.*)$/;
	my $hdr_title = $1;
	# Parse the date
	die "Invalid date format"
	    if ($hdr_date !~ /^(\d\d\d\d)-(\d\d)-(\d\d) (\d\d):(\d\d)$/);
	my $unix_time = mktime(0, $5, $4, $3, $2 - 1, $1 - 1900);
	my $t1str = strftime("%a %b %d %H:%M:%S %Y",
			     localtime($unix_time));
	my $t2str = strftime("%a, %d %b %Y %H:%M:%S %z (%Z)",
			     localtime($unix_time));
	printf "Writing message %d\n", ++$msgno;
	print FOLDER "From user\@southbayriders.com  $t1str\n";
	print FOLDER "Date: $t2str\n";
	print FOLDER "From: \"$hdr_from\" <user\@southbayriders.com>\n";
	print FOLDER "To: \"$hdr_to\" <user\@southbayriders.com>\n";
	print FOLDER "Subject: $hdr_title\n";
	print FOLDER "\n";
    } elsif (/^${message_boundary}$/) {
	die "Message not in folder" if (!$folder);
	while (<INFILE>) {
	    chomp;
	    if (/^(${header_boundary}|${folder_boundary})$/) {
		$pushback = $_;
		last;
	    }
	    if ($_ =~ /^From /) {
		print FOLDER ">$_\n";
	    } else {
		print FOLDER "$_\n";
	    }
	}
    } elsif ($folder) {		# Skips junk at beginning of file
	die "Expected folder or message";
    }
}

if ($folder) {
    close FOLDER;
    $folder = '';
}

close INFILE;

 

[aciurczak]

You rock hard, I don't care what they say.

Worked like a charm. For Windows users, here's all you need to get this to work.

1. download activeperl from here

It's a free implementation of Perl for windows, and allows you to run just about all Perl scripts with no muss, no fuss.

2. install activeperl (just run the downloaded file and click next until you're done, easy-peasy)

3. copy the text from Curt's message above, and save it as a text file named txt2mbox.txt. Once it is on your computer, make sure the txt file looks right. Then just change the file extension to .pl, so the full filename is txt2mbox.pl

4. go to SBR and download your PM's as TXT. Just right-click on the TXT link at the bottom right of the PM screen, and save the file to your computer. Name it whatever you want, the simpler the better. sbrpms.txt works fine.

5. copy both that txt2mbox.pl file and the sbrpms.txt file into the same directory, preferably a directory that doesn't require admin permissions. Vista screams if you try and do this right in the C:\ directory, so if you run into issues there rather than fighting it just use a different directory like c:\Users\your username\test

6. open a command prompt, by choosing Run from the start menu, and typing cmd and hitting enter

7. within that command prompt, navigate to that folder you chose earlier, by typing cd c:\Users\whatever_folder_you_used

8. once in that folder, type dir and then enter to confirm that the two files you need are in the directory.

9. type txt2mbox.pl sbrpms.txt and hit enter; if the files are both there and you didn't have any typos, Curt's script will create new files within that same folder. If you have no custom folders within your PM's here, the likely 2 folders you will get are the Sent and the Inbox, for PM's going out and coming in.

10. rename the new files to whatever you want, but add the extension .mbox to each of them

11. copy those folders into the mail folder that you use for thunderbird (make sure thunderbird is not running at the time), then boot up thunderbird. Magically, you'll see those new folders right there in your folder list, and they will contain all of your SBR PM's that were in the files. At that point you can either keep them there, or move them into whatever other mail folder(s) you choose.

Optional: Since the .mbox file structure is pretty standard, just about any other email program you use should be able to import them in some way or another, but at that point you'll have to do some research to figure out how.

Great work Curt, this is extremely useful, and will likely come in handy for me on a whole pile of vBulletin boards.

 

[mweitz]

Why not just have them emailed to you and archive them with your regular, indexed email?

Mark

 

[mnb]

I just DL them as one text file and stuff it in my Documents folder. If i need to search it, it's just one text file, so it's easily searchable.

I'm at 92% full right now, so I'll be doing that again soon.

 

[AceT100]

Why not just have them emailed to you and archive them with your regular, indexed email?

Mark

Thats what I do...after all, when you get the notification email, it has a copy of the message.

 

[mweitz]

Right, I'm just wondering if I'm missing something. My solution is simple and theirs is complex so there must be an extra benefit to it .

Thats what I do...after all, when you get the notification email, it has a copy of the message.

 

[AceT100]

Right, I'm just wondering if I'm missing something. My solution is simple and theirs is complex so there must be an extra benefit to it .

Maybe its the 'he who has the most add on's' wins?

I dunno....how important can a PM be?

CTRL & P works too

 

[Andy]

I wish I was important enough to receive PM's that would be worth keeping. Most of my PM's consist of words like "you're an ass" and that sort of stuff.

I just delete my PM's.

 

[SirNapsAlot]

Holy Crud! and here i was just doing copy/paste and sending them to my main e-mail

edit: of course there's only a couple like the link to manuals galore ect...

 

[aciurczak]

Why not just have them emailed to you and archive them with your regular, indexed email?

Mark

That would only work for incoming PM's, and you'd lose any outgoing PM's that you sent to others; there's never an email notification for those.