Where's Curmudge

[admin]

Hey Curmudge, where are you?

[jingle_jangle]

Right here, Peter. Had a spot of trouble getting in, but after a shower and comb-out, I'm rarin' for action again.

Here's my smiley quotient for life:



...There. I feel better and relieved.

Except I lost about 350 posts in the switch-over.

[gibsonlp]

Nobody lost any posts.
As I explained:

Discus didn't really count the posts, only incremented their number, so:
1. Posts that were gone forever (some posts were not kept in the archive and were not converted) are not counted (again: discus counted them too as it only incremented and didn't count)
2. If a thread was removed - the posts count for your posts on this thread are gone too (again: discues counted them too.........)

[jingle_jangle]

Whatever, Gil...that's a computer techie explanation that sort of goes over my head.

Thanks for your efforts.

[gibsonlp]

Actually this is not a computer thingy at all if you read it without thinking about computers
Nevertheless:

1. All available posts are here, I actually counted the amount of posts I recieved on html from Peter and they are all here.
2. You are in a rick-a-holic status anyway, why the hell do YOU care?

[admin]

It is entirely possible that the number of posts varies from the old to the new forum. This is especially true for "old timers" who have been around for years. Over the course of time, I have deleted a number of threads and the old discus software did not make allowances for this. My apology to those who lost posts in this way. We have, however, been able to save more that one-third of a million posts over the past seven years. So, this should give you some reading material when we are in read only mode. We will strive harder in the future.

[jingle_jangle]

Yeah, Gil...not caring is a hobby of mine.

Just asking.

One more thing and I'll be as happy as a pig in pigskin...please give us back out old-style email notifications. They make scanning and selecting threads a walk in the park. I'm begging you, please...

[gibsonlp]

I will take a look at it, this is not going to be simple.

[elysrand]

I concur with Paul! The old-style notifications with the text of the posts embedded was a dream. To date, the job you have done on all this is simply amazing, Gil, I am very impressed (and I am in the computer business myself, running a North American telecommunications carrier that also provides Internet services and hosting...)

[gibsonlp]

At last! someone who can understand the technical aspect of converting a forum from html to phpbb

Anyways - I will do my best to have this feature back. I will keep you posted on any development.

[elysrand]

Wow, thank GIl! I can just imagine how complex the scripts must have been that you wrote to convert the format and content of the stored archives into the proper file format to be read properly by PHPBB. It must have been far from trivial! Heck, you may have even written code and compiled it, not scripts, to do the data import and conversion...

[gibsonlp]

The basic parts were written in bash, the more complex ones (with some fancy regex, complex arrays, more "logic, sql etc...) were written in php.

What I did is:

1. I reverse engineered the entire structure of the discus html export in order to know exactly what to look for
2. I wrote a converter to convert the posts from the html archives to an interim mysql table, I extracted all the valuable data from the html files (post id, poster, ip, date, post text, subject (by going to the "last" page and looking at it) and poster parent id (thread id - by going to the "last" page and from there going to the first archive to get the first post ID...)
   I had to deal with 4 different html formats with different data inside, the old ones for example only had the poster name posted, not the username.
3. I then wrote a converter to build an interim user table containing all the users data from a csv export, some users had missing or malformed registration date so:
4. I wrote another script to search for each user's first post and user it as the registration date should it be earlier than the listed registration date.
5. I wrote a script to match users to their posts by matching real names (partial matches as well - requires some complex regex)
6. I then wrote more scripts to convert the posts from the interim table to phpbb, I had to reverse engineer phpbb a little bit to get to know what exactly it expects to have for each post, this script involved writing html to bbcode converter that will also put a UID for several bbcode tags, I also converted the old attachments links to the new format (needs to be done carefully in order to mess up with other links).
7. I moved all the previous attachments to their right place so they will be recognized.
8. and, at the end - I wrote another script to go over phpbb and fix the statistics (per forum, user posts count, etc...)

The code is pretty messy now and not up to any of my standards in terms of efficiency and overall stracture, however - it does the job. Currently there is no similar converter out there, I will probably make it look nicer and contribute it to the community so other people will be able to enjoy it. It's more of a toolkit than an actual converter, but nevertheless.

[elysrand]

They would love to see that over at phpbb.com! I am sure that DiscusWare would not be too happy at seeing anyone publish a ready migration tool or utility away from them to any open source equivalent product, but that's too bad because there is nothing wrong at all with publishing it or using it. Your job was made harder because Discus changed their formats over time, as you pointed out. You would be doing a great service to the open source community to contribute your work on these conversion tools