A website update - UPDATE: front page active again -
04-18-2012, 12:56 PM
[A late edit: our front page is once again operational. Some links to the recruiting database - and recruiting data - are still being repaired.]
We have had a lengthy and difficult struggle trying to recover from a complicated database collapse. What really made it difficult is that to this day we not know what triggered the collapse. We know by now exactly what happened - but no one among several experts has been able to explain why it happened - or to suggest even one plausible explanation for some completely contradictory evidence we see.
The core mystery is this: three critical files in our database do not exist in any of our backups - even though our system was running without error at those times. Given what we know about those files, and what they do, we are certain it is impossible for our website and database to have run without them – but they are simply not there in the backups.
We engaged three highly competent firms to analyze the evidence with a primary goal of first recovering all our information, secondarily to understand how or why it happened - so as to prevent future re-occurrences. Unlike a medical doctor, we tried to "cure first, diagnose later."
We had help from the software firm that built our website and database, also from our server supplier who manages all the physical boxes, security measures, and all backups, and furthermore from a software firm that specializes in recovering, analyzing, and even reconstructing exctly those three critical files that were missing from our several backups.
None of the experts had ever seen this kind of event before, even the firm that specializes in repairing and recovering exactly these kinds of files. Our server supplier is completely baffled as to why essential files that had to be present in the past don’t now exist in the backups they made.
It took time – many days - to research all the options and ideas that various analysts came up with. To make it worse, day after day we thought we had a good lead that would solve the problem - only to have a succession of those fail. Each one was a bitter pill.
And this situation made it impossible for us to publish a forecast about when we would be back up and running. A forecast would have been just a guess and therefore unreliable - we just couldn't do that.
With the latest such failure, just two days ago, we finally decided to step back to our beginnings and rebuild the database from scratch. We fortunately have found many remnants of data in files that were not harmed - so we're using those today in our rebuild process.
"Crimson Tide" explained: Several readers have noticed links in our menus that went to 'Bama's crimsonconfidential content. Those links have now been deleted as we do our repair & re-building processes - but perhaps a word about why that happened would be of interest.
To rebuild all our data-tables, in all their complexity (and they have to be perfect to work), the only method available to us was to use a specialized "script" to reconstruct the entire database. The 'Bama question arose because the last time that script was used, it was used to build the 'Bama site - so had some ‘Bama text in it.
The choice we faced was to either shut down our whole website for a day or two while we cleaned out the ‘Bama content - or let it continue operating while we did the cleanup online. We chose the latter - and some of you noticed the consequences.
We're nearly done now. The 'Bama content has been cleared out and most of our Bear Insider pages and links have been restored – even as we continue to buff & polish the details.
There's still work to do - several of us are now working to rebuild data on recruits (starting with the current year’s class of course), and front page articles from our recent past. Readers will see those both coming up in the next few days.
I created this website almost 14 years ago, and have never suffered a body blow anywhere close to this. It has been agonizingly painful to see so much good work, by so many people, over so many years - be so jeopardized.
Thank you for your patience. You can be totally sure we have been and are busting butts to get back up and fully operational again. Thankfully that is now happening.
Last edited by GreyBear; 04-18-2012 at 05:58 PM.