Deployment will Eat Your Lunch

They generally tend to congregate in Starbucks, Brooks Brothers outlet stores, and marinas – the business-person who gazes longingly, nay desperately, at his comrades using an user-friendly uber-trendy iPhone while he keys away on his standard-issue company Blackberry. Our friend’s golf-buddy observes his plight, and raises his eyebrow in just subtle enough fashion to perfectly communicate the commentary, “Blackberry? How very 2003.” The tragic part is that this is not our unfortunate friend’s fault. He didn’t choose to organize his work life using the mobile equivalent of MySpace and LA Gear – his employer simply issued him this sidearm because that was the company standard, and nothing so gauche as an iPhone could ever be allowed to connect to the corporate network.

As a purveyor of fine enterprise applications to clients at different levels on the technical evolutionary ladder, this is the same tragic sense of longing I feel when I hear someone talk about their Automated Deployment Process. It’s not that I don’t want to configure my project with a TFS managed, MS-Build controlled process that downloads, compiles, tests, and deploys all application modules to my development environment on a nightly basis, and to my test and production environments with the click of a button – it’s that we don’t always control either the hardware, or the deployment processes to install applications on that hardware . Usually, the production environments of large corporate clients have strict deployment control processes that simply do not allow the application’s developers to play in the production playground, let alone an automated process coordinated through Visual Studio.

Why won’t these boorish server sentinels let us spread our wings and fly with the best the modern development world has to offer? The first instinct of most software developers is to neglect deployment in favor of the more interesting problems of architecture and code – our tendency is to pout. “Fine,” we say, “if you want me to do it your way, I’ll just forget about deployment, give you the source code, and let you figure it out when it’s time to go-live.” Unless they’ve been through a horror-show deployment themselves, our project managers won’t be much help, either. In my personal experience, project plans for new software routinely underestimate deployment effort by a factor of two or three. In this attitude lies danger, because if you neglect this process, your application’s deployment will eat your lunch. It is only after we’ve been burned by deployments that we learn to appreciate early planning.

Fortunately, we usually have a lot more control of the situation than we sometimes realize. There are standard best practices we can still implement no matter what restrictions IT places on our deployment. The key is planning and preparation.

For the Things You Do have Control Over, Get it Right

Just because you can’t automatically deploy your system to test doesn’t mean that you’re helpless as a kitten. The biggest step you can take into alleviating any deployment headache is implementing an automated build process. Does your client have a favorite build automation tool? Ask them for a download/build/package template at the beginning of the project that you can modify to fit your application. If your client doesn’t do automated tools (most don’t), use your own favorite tool, but make sure to leverage the client’s source control system. If your client uses SVN, use SVN. If they’re still on Visual Source Safe suck it up and install VSS. You haven’t experienced mundane, error prone, manual tasks, until you’ve tried to unbind your project from your current source control server, migrate it to a completely different product, re-bind the project to the new system, and reconfigure your automated build process (you are using an automated build process, right? I think we just mentioned that.)

Make sure you go into the deployment process with eyes wide open. Bring up deployment at the kickoff meeting. Chances are the people in the room won’t know the exact delivery format they want for an application deployment package, but they’ll know the people who do. When you’re building your deployment strategy you’ll need to know whether they want you to deliver an installer, provide a solution that builds its own binary, or runs a batch file that calls xcopy to transfer files to the web server retro style (don’t snicker, I have clients that have this exact policy).

Building your Deployment Process is a Development Task

One of the traps teams fall into is failing to establish a development server that they can use to develop and test the deployment process. Chances are your client knows the exact specifications and configurations of your web, application, and database servers. Ask the client to create a development version of each environment so you so can run your deployment processes before QA. If your client doesn’t give you a dev environment (and plenty won’t), don’t just accept it and revert to local development only – get the configuration settings from IT, build your own VM to match, and then use it.

The lack of a development environment is probably the biggest impediment to a smooth deployment that we bring on ourselves. Unless forced to use it, plenty of developers will get lazy and the only environment they’ll ever run their application on is their local IDE. That’s why it is essential for project managers to add a line item for development and deployment scripts as a development task. If you’re a developer, make sure your PM anticipates the time that you need to spend building the processes to deploy to production.

If your project plan doesn’t have deployment package tasks under the development umbrella, you didn’t do your job.

 

Get Your Hands off My QA Environment you Dang Dirty Developer

We’ve all been on the occasional project that has come up against a tight deadline and launched into “Just Get It Done” mode. (“Occasional” is the correct word, right? Hopefully this doesn’t describe all of your projects). In JGIDM, niceties like “error handling,” “good code”, and “personal hygiene” fall by the wayside as the delivery team desperately flees the dreaded “we’re going to be late” conversation with the LOB.

The first task to get the axe during JGIDM is the lunch break, and the second is a proper QA driven deployment process. Unless the person physically deploying your application to the production environment is a developer (which for Enterprise Applications is about as common as a meaningful budget for GUI design), then the development team should not be deploying to your QA environment. In most scenarios, you the developer will hand over a deployment package to IT containing some binaries, some resource files, perhaps a few database scripts, maybe an installer, and a document describing how to put the whole enchilada into production.

This package, and especially the document, will always be wrong the first time you write it.

Whoever follows the first draft of your Deployment Guide will arrive at this screen.

The reason your deployment doesn’t work the first time is that you have so much information about your application archived between your ears that it is quite literally impossible to regurgitate a clear and comprehensive set of instructions that installs your system without issue. You will almost certainly forget to mention the step that associates an existing database role with the user account that an integration point runs under, or that your IIS installation’s virtual directory needs to be set to use Windows Authentication – except for the link that serves as your FAQ page, that you need to turn on MSMQ in your app server, or any one of a dozen other tweaks that you forgot you set in your development environment.

If we can reasonably assume that the first person besides yourself who uses the installation guide is going to fail, the we need to be very careful about who you choose to see that initial “500 Internal Error” screen. If you choose your QA lead in your QA environment, she will reach over the cube wall and smack you upside the head when the entire system comes crashing down. You can mitigate this risk by wearing a helmet. To solve the problem, you can log into the QA environment, examine what isn’t working, fix your crappy deployment document, and let QA try again.

If you choose to let IT be the first group to follow these instructions while installing in the production environment, you will most likely get an angry email that copies your boss, your client contact, your LOB, the mayor, your high school computer science teacher, and Mr. T. The next two hours of your existence will involve lunch meetings with menus of humble pie. You will not have access to the production environment, so the resolution will also involve follow up emails to an IT department in a remote office – who is already suspect of your intelligence, competence, and worth as a human being – and pestering them for log files, screenshots of web server configuration wizards, and a lot of guessing. When you fix your crappy deployment document, you will have to answer more questions and explain why you are sure your deployment will work this time. If you accidentally left a component out of the deliverables, then you will have to submit a new version of the binaries through their production-change-control-we’re-really-serious-about-process review board that includes tickets to approve, emergency-production-deployment-variances, and all other incarnations of red tape that lead otherwise content software engineers to day-dream about parallel universes where they chose creative writing as a major back when they still had the chance.

Or worse, this could happen.

Is the Deployment Finished? Are You Sure?

Most enterprise level applications have a dozen moving pieces from website, to database, to import processes, to email queues, to MSMQ messaging. From the time you begin deployment, however, your management only wants you to answer one question: is it in production? How will you answer? Obviously, if you are still working on resolving DB permissions issues, or connectivity between your windows service and MSMQ, or are trying to convince corporate security that they need to upgrade their web server to ASP.NET 4.0, your only answer will be “Not yet, so please go back to your office and start ordering pizzas because I’m going to be here for a while.”

If, on the other hand, IT has successfully run all of your setup scripts without issue, can you automatically answer, “Yes, it’s done. Please tell the LOB to start hitting the site?” Maybe, but only if you’re the kind of person that enjoys base-jumping in a wing suit after work. The simple fact of the matter is that you won’t know that you deployment worked until you’ve used your application: all of it. If it’s a web app you’ll obviously need to hit the site, but there are other scenarios we care about. Does it have an email component? Trigger a system generated email. Is there an import process? Run an import file. Do you expose a web service? Write a test harness to consume the service. Do you have a reporting component? Run a report – then export it to PDF and Excel.

I’ve run my deployment scripts, now I’m going to share my application with the LOB!

Chances are you will not think of every test scenario on the day you deploy, which means that you’ll need to define your Deployment Validation Tests well in advance of your actual deployment. If you use test driven development or have a suite of automated tests, a subset of those tests may be all you need to validate your deployment. If you don’t automate your tests, that’s ok, too (I’m not one to get preachy about TDD), but you will need to define the minimum set of manual tests that will verify that every core component of the application is up and that they all can talk to each other. Remember, if your MSMQ server is up, your system is still broken until your app server can connect to and read from that queue.

If a project is a day at work, then deployment is the commute home. You hope it’ll go smoothly, but if you don’t plan for rush hour traffic, road-work delays, and accidents, you may be late and end up disappointing someone. If you know the risks, plan for the issues, and prepare beforehand, you’ll always be able to navigate the path from code to production.

Cross posted at the Computer Technology Solutions blog.