T-SQL Tuesday #88 – The Daily (Database-Related) WTF
Image may be NSFW.
Clik here to view.
For some not-so-strange and not-so-odd reason, this month’s topic grabbed my attention. Kennie Pontoppidan (Blog | Twitter) invited us to write about our own database-related horror stories inspired by the IT horror stories from http://thedailywtf.com. Database-related horror stories? WTF stories? Sweet! Count. Me. In! It seems like nearly every IT professional I talk with has their own (usually several) horror/WTF stories. Having been in IT for close to twenty years now and a DBA for over half those years, I am not without my own stories. Wondering which one to write about, I simultaneously chuckled and shuddered while reminiscing about the first one that came to mind. Yep! That’s the one.
What is T-SQL Tuesday? It’s a monthly blog party, so to speak. Someone comes up with a topic and we all write about it. You can learn more about it here.
Wait… They Did WHAT?
The incident in question occurred many years ago during a typical day for me as a DBA at a prior organization. Little did I know what was about to happen would haunt my mind for years to come. Since it’s been many years and I don’t have an idyllic memory, I’ll give you the gist with how I remember it.
Someone from our IT security group came to me asking me for help. The system that runs our security badges wasn’t working. She asked me to take a look at the database. Since this was definitely a production issue (well, that and I have a vested interest in the security system), I promptly took a look. Given the time frame, this was probably SQL Server 2005 Standard Edition. I checked the SQL error logs. I ran sp_who2. I made sure the database itself was accessible and so on. Nothing stood out so far. Then I checked the SQL Agent jobs. I noticed one job in particular had failed that day. Upon gazing into the job itself, I noted something out of place. I did a double-take because I couldn’t believe I was seeing this on a production system that someone actually paid money for.
Using my plucky powers of deduction, I asked her if the database server had been rebooted recently. Why, yes! It had been rebooted just moments before. Why? Why would that matter?
After running through a myriad of emotions, I then asked who wrote/developed/conjured this database because they were going on my “watch-out-for-these-vendors” list. Guess what. It’s not a good list to be on.
I then showed her what I found. The job failed because it couldn’t find a specific stored procedure that was apparently needed for the system to run. Can you guess which database that stored procedure was supposed to be in? … TEMPDB! I’ll pause a moment for that to sink in. Still with me? Yeah, I kid you not. I really couldn’t believe what I was seeing. So then I took time to explain to her that the TEMPDB database is just that… temporary. It gets rebuilt every time you restart the database service which includes rebooting the server. We don’t ever back it up because there is no reason to… it’s temporary. I realize not everyone knows these things about SQL Server and that’s okay. I just would have thought that anything with the word “temp” in it would be an indicator that it’s not a good place to permanently store important things. Maybe that’s just me?
Image may be NSFW.
Clik here to view.
Per her request, I sent her an email detailing what I had found. She forwarded it to the vendor and they fixed it. All was well once again. I never did find out why they put that procedure in TEMPDB. I’m really hoping it was an oops and that’s what I’m going with because it’s hard for me fathom the alternative.
This may not be much of a horror story in the traditional sense. To me, it was more of a WTF moment. It also made me wonder for quite some time – was that design intentional? If so, what else were they doing that may not be a good idea? What else is out there that I don’t know about that may cause problems? It opened my eyes actually. I realized from that moment on that if someone can do it, they probably will, even if it’s probably not a good idea. Heck, I’ve done it myself. That’s how we learn and grow… well, hopefully.
Filed under: TSQLTuesday Image may be NSFW.
Clik here to view.
Image may be NSFW.Clik here to view.
