Upcoming vBulletin Upgrade on January 12th, 2017

Really meant to get this up sooner, but in testing the upgrade process I came across a few issues that needed to be documented and/or worked around. I will address the ones that affect non-admins below.

So, here’s the situation. Thanks to a very generous donation the board is getting upgraded to vBulletin 4 on the 12th. I will be scheduling a 6 hour windows for this. This is based on two things. First, while doing the test migrations as of when I’m writing this I can only get the migration down to 3 hours 28 minutes. Admittedly part of this is because I have been running the test directly on the same host so this board being active during the tests affects that. The other reason for the 6 hour window is I need to perform some OS related updates to fix a minor issue. I’ve done this particular upgrade on my other VPSes so I know how long it takes to do it and fix it. It takes just over an hour.

So why 6 hours? You know that they say about the best laid plans, right? Yeah, I’m giving myself plenty of room to role back things if necessary. The windows will be from 6pm to midnight. During that time, you can hop on the Discord server and keep updated on the progress.

So, with that out of the way first let me give you a list of advantages to this change:

  • Most of the features I had to hack the core code in SMF to support is either native to vB4 or can be implemented entirely via its hooks system.
  • Post tagging (Very useful for content and ratings in the story forums :) )
  • Finer grained controls of mod powers (Why is this important? Keep reading ;) )
  • Better control of moderation for new members. It's actually fairly safe to let users who haven't fully validated their accounts still login and have access. With SMF it's all or nothing.
  • Better Profiles.
  • Better signature controls
  • You can upload a sig pic directly just like you would an avatar or profile picture.
  • Separate avatars and profile pictures
  • Better privacy controls for your profiles.
  • A better PM system. No seriously, even without modifications it's a lot saner.
  • Social groups. Why is this a good thing? Every time someone asks about adding an RP forum it makes baby kangaroos cry. With social groups, you can make your own RPs as long as your usergroup has the correct permissions (most of you will btw)
  • Personal Blogs and the Articles system. The former may seem useless to most people, but the latter will be much more useful. It will make adding extra pages in general much easier.
  • Global Announcements. Very useful for getting information to all users, including those on Tapatalk. If I had a nickel for everyone who says they never visit anything but the story forums and missed important announcements… ;)
  • Profile photo albums. These will be limited, but you can use them for uploading small images you want to share but don't want to find another host for. Yes, this includes signaute pics as well if you want ot use more than one.
  • Better poll controls, which will make running future contests easier.
  • The ability to create forums where only the original poster, admins, and the mods of those forums can see posts. This is more for staff but it does let me create a forum where problems can be reported so they can be dealt with and not require email or PMs.
  • Edit posts are preserved in their original form. No more he said/she said BS with post editing.
  • Users can delete their own posts. In the past I refused to let this be enabled because with SMF it's a pain in the butt to restore deleted posts even if you have a forum properly setup for deleted posts to be moved to. You can remove posts, but an admin can always see them and restore them or remove them permanently. This is the one feature I [i]will[/i] turn off if it gets abused.
  • A proper functioning search. You seriously have no idea how broken the search has been for ages on here simply because of the way the search indexes are stored.
  • It's much easier for me to add profile fields for things like showing which contests you have entered/won.
  • The ability to show ads. Now before anyone freaks out, there will be very strict rules about these. The big one being that I will not be accepting ads from companies with bad reputation in the community. There will be more info about this when the new board goes live.
So, now that I've got most of the advantages, I have to discuss a few disadvantages.
  • [b]EVERYONE[/b] will have to create new passwords. This is how you will know the migration is done actually. The final step will be emailing everyone a new randomly generated password. As such, please make sure your email address is correct prior to the 12th at 3:00pm. It's not the end of the world if you don't. You'll just have to email me to get it fixed.
  • Broken quotes. Only partially so. It has to do with the difference in how the SMF and vB quote tags work. They'll show up as quoted posts, but they won't link to the parent post. Instead they'll have some weird text after the username. These will require manual fixing.
  • Smart quotes will be changed to "dumb" quotes. This can't be avoided because of an encoding issue with how they're stored in the DB. This will affect signatures, posts, and PMs equally. No matter what I do I simply cannot get them to convert properly so I went with the option of changing them to dumb quotes. Posts/PMs/etc made after the migration will be fine.
  • You may need to setup your avatars again. This is again caused by a quirk of how SMF stores the information about them in the database. Some will migrate fine, some won't. It's completely random which ones do and which ones don't.
  • If you received a PM from a user that has had their account deleted, sent one to a user that has since been deleted, or to/from a banned user those PMs will [i]not[/i] transfer. This is an issue with the importer script and would take way more effort than it is worth to fix. If you have PMs that match the previously mentioned criteria, please let me know. I can create a manual backup of your messages for you.
  • IF you have a different display name and username, you will need to email me at [email]registrations@abdlstoryforum.info[/email] and let me know which you want to keep. After the migration I will have to manually correct your username if you would prefer to use your Display Name. If for some reason you don't see this until after the migration, you can still email me and get it changed (later you'll be able to do these changes yourself)
  • As much as I hate doing this, you will need to provide a date of birth on your first login on the new board. You do [b]not[/b] have to add it to your profile. That is an entirely separate step. This is strictly a record keeping issue to allow for some future changes that will actually be a [i]good[/i] thing :)
  • Your post based user titles and rank images will not transfer over initially. This is a manual process due to the way SMF stores the post based groups and handles the rank images (all one record in SMF, the rank images an post based titles are entirely separate entities on vBulletin)
  • URLs will change. It will take me awhile to get all of them in place, but there will be redirects from the old URLs to the new ones. This is a semi-manual process to implement properly.
  • vBulletin will strictly enforce the limitations on the dimensions and file sizes of avatars added by URL.
  • Custom Titles are possible, but their is no equivalent of the Personal Text option. And no, I'm not adding it, sorry.
Now one thing that is both an advantage and a disadvantage: How many posts you need to get off of moderated posting is not a hard number. Generally speaking it [i]should[/i] happen in roughly the same number of posts as it takes currently. Now, remember I mentioned the ability to fine tune mod permissions? This is why that is important: Post migration I will be contacting some potential people to help with moderation. Mostly releasing posts of new members from the moderation queue. Kita and I have made a list of users that I will be contacting to see if they are interested. Please don't ask if your name is on the list. And to those who don't get asked, please understand that the initial list is based on several factors, including (but no limited to) length of time as a member and activity in the last 90 days. If you have any questions, feel free to ask them and I will do my best to answer them. Just remember I am still actively running the migration tests and updating the post migration cleanup script so it may take some time to get a response.

Upgrade Timeline

4:00pm CST - New User Registrations disabled to allow initial user import.
5:00pm CST - Import Users, Usergroups, and settings that can be transferred while SMF still active
6:00pm CST - Set SMF to completely disabled for all users. Begin importing PMs, Posts, Polls, and other information that cannot be transferred while SMF is in active use.
8:00pm CST (estimated) - Run cleanup script 1. Fixes quotes in posts and PMs to show up with correct attribution if its included. Also fixes SMF’s lists to use the vB version of the item tags (testing with some optimizations gets this step to 2 hours).
10:00pm CST (estimated) - Run cleanup script 2. Fixes smart quotes and manually imports the Bio field from SMF to vB as this is a custom field and can’t be manually transferred. Other custom fields are not imported.
11:00pm CST (estimated) - Run cleanup script 3. Fixes permissions, rebuilds all forum and user stats and the vB caches to allow posts and the member list to work properly.
11:30 CST (estimated) - Run cleanup script 4. Forces all user accounts to have a known password to trigger the automated reset tool (required because the PW stored in the db after import is literally the password hash from SMF. Trust me, you do not want to type these in :stuck_out_tongue: ). Turns vB to active then sends the password reset emails to all members but myself (my account gets a pass because it has to be manually made as part of the initial install process so already has a working password).
12:00am CST - Begin fixing the inevitable reported issues (me, cynical? nah…)

Re: Upcoming vBulletin Upgrade on January 12th, 2016

Oh, I forgot, since someone will ask this: No, there will not be native apps for iOS or Android. It’s not that I don’t want them, it’s that literally Apple and Google would not approve them because of the board’s content. I already contacted them and asked. That was part of what delayed me posting this.

Yes, I will be enabling Tapatalk for those who want it.

And because I just realized I forgot this: The reason for vB4 instead of vB5 is because there’s no migration path to 5 without going through 4 and the minor differences in the feature set don’t justify the extra time and work needed.

Re: Upcoming vBulletin Upgrade on January 12th, 2016

Will the ads have tracking capabilities. And will we get better accessibilty can’t tell you how many times an embarrassing situaton almost happened because of the hacks to my reading system for this board. Not that that is a criticism of you mind the jackwagons at SMF make it nearly impossible to have good accessibility :slight_smile:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

Believe me, you are preaching to the choir on accessibility. I gave up on trying to fix that fuster cluck a long time ago. :slight_smile:

For the ads, the only tracking is what is recorded in the access logs on the server. All scripts and images will be locally hosted for them, so unless you click through them the only info anyone who places an ad will have access to is the anonymized dump without IP addresses showing how often the ad was actively seen.

And there will be a special group you can be placed in to disable them completely if you need it. :slight_smile:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

fuster cluck that’s a new one, it’s so obvious i can’t believe i haven’t seen it before :smiley:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

On a Digimon fan forum I hosted years ago we actually had a censor replacement that changed the original phrase to that to get around a silly rule with an advertiser. :slight_smile:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

raises hand

If we can’t get a mobile app thingy, please tell me the mobile site will be better.
I browse this site on my phone a lot more than I should, but while the WAP2 site is perfectly usable, it’s also bare bones.

Re: Upcoming vBulletin Upgrade on January 12th, 2016

Well, Tapatalk will still be an option, not that I’m entirely thrilled about it… (Long story).

The mobile site is a vast improvement over the SMF one. :slight_smile:

Give me a few minutes and I’ll post a few screenshots of what various parts of the mobile site look like :slight_smile:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

…In the same way that Taco Bell is an option. :stuck_out_tongue:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

I did say I wasn’t thrilled about it :stuck_out_tongue:

Anyway, screenshots attached. Note the posts one shows a very good example of why the cleanup script is necessary. This is from the current test run just before the cleaner ran :slight_smile:

Also take into account this only addresses the forums themselves, not the other sections and how they look on mobile :slight_smile:

EDIT: Oh, and the grey boxes are private info that needed to be hidden, those don’t exist on the actual site :stuck_out_tongue:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

Hey, we pig out on the 12-for-13 deal regularly in this house LOL. Hard to feed 3 people for that cheap these days.

Re: Upcoming vBulletin Upgrade on January 12th, 2016

It’s hard to feed two for that cheap these days depending on your location. :slight_smile:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

I vaguely recall one or more rants about TT and something to do with security.
My personal gripe was that it simply ran like shit. It was basically unusable across several phones, and I’m not compelled to try it again.

Looks good.

I’m very interested in the tagging feature.
Will all (or most) users be able to tag other threads? Like, can I tag a thread by another user?
Can tagging permissions be granted/revoked in case the above is true, but abused?
Or will it just be the OP or mods can tag stuff?
And is this all something I’d already know about if I read up on vB or used any other forum at all? :stuck_out_tongue:

And most importantly:
Will it be possible to have notifications sent based on tags? For instance, I want to be notified of every new topic that receives the ‘OHT’ tag. ;D

On a semi-related note; what’s up with single quotation marks or apostrophes and -ahem- shit getting fucked?
I’ve seen many sites where it just drops all of them in favor of a question-mark-symbol (or some such,) I’ve seen all manner of other goofy looking strings of symbols generated; hell there’s even a few lingering instances of something along the lines of ;&amp or similar on the Playstation Store. Anyway, the most common screw ups that I’ve seen like that are nearly always where a single quotation mark or apostrophe would be; is there a reason for that one in particular, or do I just happen to stumble across it a bunch?

Key word there is feed.

Feed means ‘to give or provide food.’
Food means ‘something that nourishes or sustains.’

Taco Bell does none of that. :stuck_out_tongue:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

The SMF plugin is horribly neglected in general so it tends to have more issues. They run vB for their own forums so they have a good reason to keep that plugin updated so less issues on that front. Really it’s more the stupid shit like adding chatrooms to the client that I have issues with these days. They regularly thrown in new features that can only be turned off if you have a paid subscription as a forum administrator.

I’m very interested in the tagging feature.
Will all (or most) users be able to tag other threads? Like, can I tag a thread by another user?
Or will it just be the OP or mods can tag stuff?

You can tag others’ threads if you’re in a specific usergroup but you’ll be restricted on how many tags you can add. You have to be manually approved for that usergroup. Mods can always add tags in the forums they mod, and of course Kita and I are exempt from that restriction completely for obvious reasons :stuck_out_tongue:

Can tagging permissions be granted/revoked in case the above is true, but abused?
And is this all something I’d already know about if I read up on vB or used any other forum at all? :stuck_out_tongue:

From regular users tagging their own threads, no. BUT I should also point out that only mods and admins can create new tags. Regular users can only use already existing ones.

And how much reading up on vB would help is debatable in this case because it’s one of that features that is very dependent on how it is initially configured. :slight_smile:

And most importantly:
Will it be possible to have notifications sent based on tags? For instance, I want to be notified of every new topic that receives the ‘OHT’ tag. ;D

It’s possible to search on them, but sadly not use them for notifications. Well technically there is a way, but the required feature isn’t going to be enabled. :slight_smile:

On a semi-related note; what’s up with single quotation marks or apostrophes and -ahem- shit getting fucked?
I’ve seen many sites where it just drops all of them in favor of a question-mark-symbol (or some such,) I’ve seen all manner of other goofy looking strings of symbols generated; hell there’s even a few lingering instances of something along the lines of ;&amp or similar on the Playstation Store. Anyway, the most common screw ups that I’ve seen like that are nearly always where a single quotation mark or apostrophe would be; is there a reason for that one in particular, or do I just happen to stumble across it a bunch?

This is fun. ’ and " and & are the three you’ll see messed up the most and that’s because they all have special meanings in either HTML itself or in URLs. Those codes like " and & are HTML entities used to encode them. What should happen is they would show up as " and & (for the two I just gave) if they are stored and rendered properly. Some sites (and forums) stupidly change the & in the entities to &, so it loses its special meaning when rendered by your browser (in fact, most of the time if you check, you’ll actually see this instead in the source: ")

Usually if you see a ? or some other weird symbols it’s an encoding issue. And in the specific case I mentioned in the original post, it’s the smart quotes that cause it. They show up really weird, and of course depending on your browsers age and the OS, some Unicode emojis will do the same.

For smart quotes they really render oddly because the importer can’t fix the brain damaged way SMF stores them without some help. Unfortunately, it’s a royal pain in the ass to fix them properly, hence the changing them to their “dumb” versions.

Re: Upcoming vBulletin Upgrade on January 12th, 2016

Gulp I’m pretty sure I trust you and Kita, but birth dates are a little close to unique identifiers for comfort. If I’m remembering accurately, something like 80% of people in the US can be uniquely identified by three pieces of information: birth date, zip code, and sex. A little mental math suggests birth date and region alone gets you to about 33 candidates per million people in the region.

For me, birth date and IP geolocator gets you to something like one in $disconcertingly_small_number.

Re: Upcoming vBulletin Upgrade on January 12th, 2016

Well besides what I mentioned up there in the original post, not having birth dates on file is actually a bit of a legal issue. It’s not like I’m going to go out of my way to verify the dates entered are 100% accurate. That would require a level of self-hate I’m not physically capable of. And as I pointed out above, your birth date and age will only be in your profile if you go out of your way to make them available. Literally the first thing after the users are imported is all profile information is set to require you to be contacts before your information is visible.

And in case it wasn’t painfully obvious, I already implemented the in database encryption just like it is on here. Otherwise I’d have had to set aside an entire 24 hours to do the upgrade :slight_smile:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

I take it that it’s more important for you guys (Renko, Kita, etc) to have verification that we’re over 18, than it is that you know exactly when we were born.

Re: Upcoming vBulletin Upgrade on January 12th, 2016

I’m sure everything is encrypted, and what you said had, in fact, already occurred to me, though I wasn’t going to say it publicly. A significant reason I said what I did was as a warning about just how close to a unique identifier a birth date can be, since I think most here would rather not let their real identity be let out except, perhaps, to specific individuals carefully vetted first.

Re: Upcoming vBulletin Upgrade on January 12th, 2016

It’s really not that unique of an identifier unless you have the misfortune of being born on Feb. 29th, and even then, not that useful.

More importantly, 3 people have access to your IP information or will ever have it. Same with your birthdate unless you explicitly make it visible. Like I said, it’s set to be hidden by default, plus you would have to go out of your way to make the section of your profile it displays in visible to anyone other than yourself. I actually need to correct something I said though, you have to make someone a friend by default for most sections of your profile to be visible. All existing members will literally have to go out of their way to have any section of their profile visible to non-admins because even mods won’t be able to see the information unless you change the settings.

I’ve gone out of my way to ensure that users’ information is as protected as it can be unless they choose to make it more visible. Heck that’s why even I can’t run DB queries through the adminCP, because it would give a way to get the unencrypted data easily. :slight_smile:

Re: Upcoming vBulletin Upgrade on January 12th, 2016

So… I made an ID-10-T error. I forgot to add the actual timeline to explain what will be happening on the 12th. I will (or by the time see it most likely have) add this to the first post, but adding this here as well to make sure it’s seen:

Upgrade Timeline

4:00pm CST - New User Registrations disabled to allow initial user import.
5:00pm CST - Import Users, Usergroups, and settings that can be transferred while SMF still active
6:00pm CST - Set SMF to completely disabled for all users. Begin importing PMs, Posts, Polls, and other information that cannot be transferred while SMF is in active use.
8:00pm CST (estimated) - Run cleanup script 1. Fixes quotes in posts and PMs to show up with correct attribution if its included. Also fixes SMF’s lists to use the vB version of the item tags (testing with some optimizations gets this step to 2 hours).
10:00pm CST (estimated) - Run cleanup script 2. Fixes smart quotes and manually imports the Bio field from SMF to vB as this is a custom field and can’t be manually transferred. Other custom fields are not imported.
11:00pm CST (estimated) - Run cleanup script 3. Fixes permissions, rebuilds all forum and user stats and the vB caches to allow posts and the member list to work properly.
11:30 CST (estimated) - Run cleanup script 4. Forces all user accounts to have a known password to trigger the automated reset tool (required because the PW stored in the db after import is literally the password hash from SMF. Trust me, you do not want to type these in :P). Turns vB to active then sends the password reset emails to all members but myself (my account gets a pass because it has to be manually made as part of the initial install process so already has a working password).
12:00am CST - Begin fixing the inevitable reported issues (me, cynical? nah…)