That was no fun at all!
The entire server was offline due to a failed RAID SAS controller card!
The issue is resolved and we have recovered as best we could from the data corrupt.
In our “autopsy” of what went wrong, it was discovered that multiple issues were at fault. Some was hardware and lack of having spares on hand and some of backup related – not having the proper disaster recovery plan.
That is almost laughable because that is what I do for a living (I, being John M. Hoyt)… I simply did not put enough emphasis on my own project and that caused backups of the database for Palmetto Showcase to be for versions other than the backups of the plugins and apps.
What have I done to make sure this doesn’t happen again?
Well, first, we fixed the hardware – but we have a spare part coming in, just in case. And another Upstate Live Music member has offered up a spare server to be used as a fail-over.
We have an IMAGE of the full server, stored to an external media.
We have automated dumps of the SQL databases – all of them.
We have automated offsite backup of the Palmetto Showcase databases and plugins.
I *think* we are much better off than we were.
Not saying we cannot suffer another catastrophic loss, just that it would be much easier to recover from it.
Thanks for your patience!
-John