Home > Blogs > Eric Shupps | The SharePoint Cowboy > Posts > How to Hang a SharePoint Backup Job Without a Rope
 
Eric Shupps eshupps The SharePoint Cowboy is the founder and President of BinaryWave, a leading provider of operational intelligence solutions for Microsoft SharePoint. Eric Shupps eshupps The SharePoint Cowboy has worked with SharePoint Products and Technologies since 2001 as a consultant, administrator, architect, developer and trainer. he is an advisory committee member of the Dallas/Ft. Worth SharePoint Community group and participating member of user groups throughout the United Kingdom. Eric Shupps eshupps The SharePoint Cowboy has authored numerous articles on SharePoint, speaks at user group meetings and conferences around the world, and publishes a popular SharePoint blog at http://www.sharepointcowboy.com. Presentations by Eric Shupps eshupps The SharePoint Cowboy Webinar - Migrating Legacy On Premise Solutions to SharePoint Online and Windows Azure featuring Eric Shupps Who are you and what do you want - Working with OAuth in SharePoint 2013 SharePoint is Talking Are You Listening? Eric Shupps SharePoint 2013 Performance Enhancements Taking Advantage of the SharePoint 2013 REST API Eric Shupps on Improving Performance with New Features in SharePoint 2013 SharePoint 2013 New and Improved Migrating Legacy On Premise Solutions to SharePoint Online and Windows Azure Eric Shupps Presents SharePoint 2013 Real World Help Desk App End to End Windows Azure Apps for SharePoint 2013 Eric Shupps Demonstrates Customizing the Visual Studio 2010 SharePoint Deployment Process Introduction to SharePoint Development SharePoint 2010 Unit and Integration Testing with Eric Shupps Building Enterprise Records Management Solutions for SharePoint 2010 Taming Information Chaos in SharePoint 2010 SharePoint 2010 Performance and Capacity Planning Best Practices Building Dynamic Applications with the SharePoint Client Object Model Articles by Eric Shupps eshupps The SharePoint Cowboy Eric Shupps' Ten Steps to Optimize SharePoint Performance Webcasts by Eric Shupps eshupps The SharePoint Cowboy Secrets of SharePoint Part 5: Configuring Microsoft Office SharePoint Server 2007 for Optimal Performance Creating End User SharePoint Solutions for Performance and Scalability SharePoint 2010 Performance Enhancements for Administrators by Eric Shupps Microsoft SharePoint Server 2010 for the ASP.NET Developer Eric Shupps on Following Best Practices and Avoiding Common Errors with Microsoft Office SharePoint Server 2007 Development Eric Shupps - SharePoint Performance and Capacity Planning Essentials Troubleshooting Common Performance Problems in SharePoint 2010 Videos by Eric Shupps eshupps The SharePoint Cowboy Channel 9 Interview with Eric Shupps SharePoint TechTalk with Eric Shupps - Different Views on Social Computing SharePoint Post-Deployment Planning and Management with Eric Shupps SmartTrack for SharePoint Feature Overview SmartTrack for SharePoint Podcasts by Eric Shupps eshupps The SharePoint Cowboy SharePoint Pod Show - Design for Performance (Eric Shupps) SharePoint Pod Show - Test Driven Development with Andrew Woodward and Eric Shupps eshupps The SharePoint Cowboy Run As Radio - Eric Shupps Improves SharePoint Performance
SmartTrack for SharePoint Eric Shupps Eric Alan Shupps eshupps @eshupps SharePoint Cowboy BinaryWave

 
​The SharePoint Cowboy


photo of  Eric Shupps
BinaryWave
611 S. Main St., Suite 400
Grapevine , TX , 76051 USA
Eric Shupps and Eric Alan Shupps with BinaryWave the BinaryWave Inc in BinaryWave Incorporated around SmartTrack beside SharePoint Monitoring through SharePoint alongside SharePoint Monitoring visiting @eshupps via eshupps near SharePoint performance and SharePoint management. The SharePoint cowboy eshupps BinaryWave and more on Operational Intelligence via Eric Alan Shupps SharePoint blog. SharePoint monitoring is a hot topic along with SharePoint Performance Measurement and SharePoint tips and tricks from Eric Shupps for SmartTrack. Another Eric Shupps on Technology and Eric Shupps on SharePoint with a new SharePoint Post from Eric Shupps. New BinaryWave post through BinaryWave Inc. and with another SharePoint blog we get to Eric Alan Shupps Blog about SharePoint development. That's Eric Shupps - BinaryWave or Eric Alan Shupps Fort Worth Grapevine Texas Dallas of BinaryWave talking about BinaryWave Operational Intelligence at the SharePoint Cowboy blog and on twitter as @eshupps. Of course SharePoint Administration is on topic for eshupps and another great post from Eric Shupps regarding BinaryWave SharePoint and SharePoint Maintenance.
February 14
How to Hang a SharePoint Backup Job Without a Rope

Here's a scenario that many seasoned consultants will be familiar with. A customer has a major problem - usually self-induced - that they have been trying to solve themselves for some period of time or hired some junior consultant to elevate from a minor annoyance to a complete catastrophe. So they call me in a panic to come bail them out (which, if they'd done in the first place, would have prevented the problem altogether, but they never seem to learn). So I hop on a plane, parachute in (figuratively if not literally), and start pounding keys until the mess is cleaned up.

Sometimes, when you do this sort of thing, you learn something valuable along the way; mostly, you just discover that someone, somewhere, did something really stupid, which was compounded by gross incompetence and exacerbated by complete hysteria. But every so often you come across a situation that was really nobody's fault and doesn't involve a malicious moron going out of their way to make your life more difficult. These are rare but they do happen.

In point of fact, I spent Valentine's Day Weekend in just such a situation. A customer for whom I built a 2003 portal years ago had been trying for some time to migrate to 2007 using internal staff and making limited progress. They finally had it mostly ready to go but they were struggling to move it from development to production. They had migrated most of their old content to a development server (or so they told me), designed their new master pages and navigation structure, installed a medium server farm, configured a nice new SSP, and then hit the wall.

Naturally, this is the part where I swoop in to save the day. What I found when I got there was a development environment that had never had a single patch applied to it, content that was partially migrated five months ago, a production farm on SP1 with a mixture of x86 and x64 servers, and a customer who didn't see any problem with having legacy OWA web parts on the home page of an Intranet that serves thousands of users. Oh, the joys of being a consultant.

The first step, of course, was to get all the servers straightened out. While I couldn't convince them to go with a pure x64 environment (which they are going to regret in, oh, I dunno, the first five minutes after it goes live), I did get free reign to patch everything up to current revs. So off I went on the never-ending merry-go-round of download/install/run config wizard/repeat. Since the migration plan called for backing up the dev content databases and restoring them to production, this meant we had to bring the dev servers up to the same level as production, lest we spend eternity in a purgatory of mismatched content databases. Since this ate up the remainder of the first day, the plan (which I honestly believe is a set of utopian ideals we humans delude ourselves with just to give us something to cling to when everything goes completely down the toilet) was to come in the next day, migrate the remaining content, do the backup/restore, and get out early enough to salvage something of the frustrated expectations of everyone's significant other, it being Valentine's Day and all (which was fine for them but didn't do me much good being two thousand miles from home).

Heh. What's that saying about the best laid plans of mice and men?

First, we discovered, after much mucking around in the 2003 database that I don't recommend anyone ever try even at home and that brought back very painful memories from the dark days of SP development, that there were, jeez, only like 800 document libraries and 200 lists that had been updated since the original content migration (a total of nearly 10,000 list items). Who would have thought that users might actually upload stuff in the last four months? There went lunch. After abandoning all hope of doing anything about it until after the cutover (did I mention the go live date was only 72 hours away?), we set about trying to backup and restore what content we did have. so at least the design team could finish their changes and get the thing up and running in production; maybe, with any luck, we'd make dinner before the crush of couples descended upon every halfway decent restaurant in the greater Washington, DC area.

Yeah, right...as if. You see my friends, the dev server, which had behaved itself so admirably by completing backup after backup in the months gone by, suddenly decided that no content database backup, however big or small, was to be performed that day. Configure the parameters, submit the job, it creates the basic set of .bak files and .xml goodies, backs up the config database, then - WHAM! Full stop. No errors, no explanation, no nothing. The timer job just hangs, with the last entry in the backup log consisting of a very helpful notice that SQL would check back with us in exactly 4.22 hours to see if we had received satisfactory customer service. WTF?

After wasting the next couple of hours searching for ghosts in the machine and trying every timer job trick I've ever come across, it finally dawned on me to check the SQL server itself. You see, I got to thinking what had changed that would stop the backups from working, which got me to double-checking that all the patches I ran the previous day had, indeed, installed correctly, right up to the December cumulative update. That's when it hit me - was it possible, just maybe, that something in one of the SharePoint patches was dependent upon a certain patch level of SQL 2005? After all, every patch we've received so far has managed to break something important (SPD workflows after SP1, anyone? How 'bout the lovely API changes in the Infrastructure Update? Oh, you wanted AAM's to work with reverse proxies after the August patches? How ridiculous.) so it stood to reason that someone had never bothered to tell us that some line of code they "fixed" needed another "fix" that someone on the SQL team thought was really, really important.

And that's when I found it. The development SQL database hadn't been patched at all - not ever. Not one tiny hotfix. And here we are at SQL 2008 with three service packs under our belt for the previous version. Naturally, this meant a big fat download and a ton of time watching a crawling status bar would be the highlight of the rest of my afternoon as we all know how quick and efficient SQL service pack installations are. So much for an early dinner.

After the patch, I was finally able to complete the backup set. Luckily, someone did have the presence of mind to patch the production database in advance, so I didn't have to repeat that little adventure. But, of course, the tale couldn't end there, as the restore process had it's own set of hurdles to put in my path. First, of course, is the fabulous requirement to use UNC mappings for backup and restore locations. This is because, whether you realize it or not, a good portion of the backup/restore work is actually done on the SQL server itself, which needs to be able to access the directory the backup files are in (which is why a local mapping on the WFE or App server would never work). But that inevitably leads to all sorts of permission issues, especially if some GENIUS decides to run the SQL service under the LOCAL SYSTEM account. Does this security principal have permissions to access network resources like the lovely UNC share we just created? Of course not! That would just be silly. So now I have to store the backup files in a local directory on my production SQL box (and share them for good measure so the WFE's can see them) which goes against just about every good security practice I can possibly think of.

I cringe when I have to remote into a production SQL box for anything but I especially hate it when a half-dozen people are standing around watching. Not because I'm going to break anything (although that's always a possibility) but because they will then think they can do it without breaking anything. That's just asking for bad things to happen. But be that as it may, I was finally able to complete the restore process and get the production machines up and going. The internal team still had thousands of list items to migrate but that was their job - my work was done and I could finally have a well-deserved drink (good thing the hotel bar has a decent selection of Scotch).

The moral of this story, of course, is keep those servers patched! You may not ever think about a dependency between patch levels on different pieces of your architecture but it's a very real issue, as I hope I've demonstrated adequately. Your SQL servers must be on at least SP2 (if you're running SQL 2005) if your MOSS level is SP1 or higher for everything to work right. And don't assume the network or server people are on top of it - chances are they think SharePoint is your problem and you should be responsible for every piece of that puzzle. And please follow best practices for installing everything - don't let the DBA's convince you that it's just peachy to run SQL as a local account (I guarantee it will bite you in the arse at some point, as I also discovered this week when I encountered a customer who installed a single-box solution and failed to document the 'sa' password for a SQL Express instance that was also running under LOCAL SYSTEM).

So ends my sordid tale for this fine holiday. And a Happy Valentine's Day to you. Now where's that Cupid guy with my heart-shaped box containing a dozen Padron Serie 1926's?

Comments

You lucky boy in my day...

You mean you actually had space on your SQL server to do a backup and you didn't have to wait for 6 hours to pull that 100Gb database backup across the network, you didn't have to have a monkey stood over you to type in the passwords the moment the 5 minute screen saver kicked in..
.... and you're not on a restraining order for threatening to kill all of the muppets involved and feed thier bodies to the dogs?
Man you had it easy...

Nice to see an article that tells it how it really is and illustrates how the ignorance of others costs you time and companies money. Good effort glad to see you achieved the end result though I suspect your SO didn't see it that way
System Account on 2/16/2009 9:56 AM

Add Comment

Items on this list require content approval. Your submission will not appear in public views until approved by someone with proper rights. More information on content approval.

Title


Body *


Comment Date *

Select a date from the calendar.
Enter the current date to prevent automated spambot comments.

Spam Prevention *


How many letters, not including spaces, does it take to spell "SharePoint Cowboy"?

Attachments

 
photo of Eric Shupps Eric Alan Shupps eshupps @eshupps SharePoint Cowboy BinaryWave Eric Shupps
BinaryWave
611 S. Main St., Suite 400
Grapevine , TX , 76051 USA

 
 
 
Eric Shupps Eric Alan Shupps eshupps @eshupps SharePoint Cowboy BinaryWave 

Eric Shupps LinkedIn Eric Shupps Twitter Eric Shupps Facebook Eric Shupps Google+
 
 

 


 


BinaryWave Eric Shupps eshupps The SharePoint Cowboy SharePoint monitoring SharePoint monitoring tool SharePoint metrics SharePoint administratrion SharePoint monitoring best practices SharePoint management SharePoint management tool SharePoint operations SharePoint operationsmanagement SharePoint administration SharePoint administration tool SharePoint SLA SharePoint service level agreement SharePoint operational intelligence SharePoint performance SharePoint performance monitoring SharePoint analytics SharePoint real-time SharePoint intelligence SharePoint ITIL SharePoint service operations SharePoint uptime SharePoint alerts SharePoint health SharePoint tools SharePoint metrics SharePoint diagnostics SharePoint SmartTrack SmartTrack Operational Intelligence


Copyright © 2013 BinaryWave, Inc. All rights reserved.
This site is brought to you by BinaryWave in cooperation with Eric Shupps Eric Alan Shupps eshupps @eshupps The SharePoint Cowboy. We hope you enjoy the SharePoint-related content on topics such as performance, monitoring, administration, operations, support, business intelligence and more for SharePoint 2010, SharePoint 2013 and Office 365 created by Eric Shupps The SharePoint Cowboy. We also hope you will visit our product pages to learn more about SmartTrack, Operational Analytics for SharePoint, SharePoint monitoring, and SharePoint administration, while also discovering great offers from our partners. Please visit the blog of Eric Alan Shupps, Twitter handle @eshupps, for more information on application development, the SharePoint community, SharePoint performance, and general technology topics. Eric Shupps Eric Alan Shupps eshupps @eshupps The SharePoint Cowboy is the founder and President of BinaryWave, a leading provider of operational support solutions for SharePoint. Eric Shupps Eric Alan Shupps eshupps @eshupps The SharePoint Cowboy has worked with SharePoint Products and Technologies since 2001 as a consultant, administrator, architect, developer and trainer. He is an advisory committee member of the Dallas/Ft. Worth SharePoint Community group and participating member of user groups throughout the United Kingdom. Eric Shupps Eric Alan Shupps eshupps @eshupps The SharePoint Cowboy has authored numerous articles on SharePoint, speaks at user group meetings and conferences around the world, and publishes a popular SharePoint blog at http://www.binarywave.com/blogs/eshupps.