Setting up QA in an SCM Environment

A customer asks: "The upper management here is also trying to get a Quality Assurance Environment set up to use with our SCM tool. We do not have a lot of experience with QA environments, can you recommend a website, document or other information you might have in order for me to get started? I'm not looking for specific details, but would like road map of things to keep in mind when setting something like this up."

First of all, let me state that there are some SCM vendors that offer their own testing tools and there are other SCM vendors that recommend another vendor's "best of breed" testing tool. Each of those solutions has its pros and cons (I'll save that discussion for another blog-day!)

What all vendors provide, however, is the ability to manage your QA environment. Your first big question is this: how much testing do you need and who's going to be doing it? Is it the programmers? Do you have a separate QA staff? Will then end users get involved?

If it is just the programmers, then you probably just need one "System Test" environment, e.g. DEV to ST to PROD.

If the developers test first, then hand it off to QA, then you should have separate environments (and libraries) for each level, e.g. DEV to ST to QA to PROD. This is because the QA people will want very tight controls over when a series of changes can come into their world and when it can leave their world. They don't want to be shooting a moving target, right?

Finally, if you want the users to get involved in testing and approving their changes, you should set up a User Acceptance Testing level, e.g. DEV to ST to QA to UAT to PROD. Again, the idea is that you want the UAT environment to be very tightly controlled. Only one task's objects should typically be in UAT at a time.

For each environment you defined above, you will teach your SCM tool how to manage it by answering the following questions:

• What are the library name(s) for source, object and data? (You can have them all be in the same library if you wish).
• Will you maintain a full compliment of all database files at each level? (I recommend yes, albeit with a very small set of test data in each level. This gives many advantages to you that I can explain later if you'd like).
• Would you like to have just one data library, or multiples? (Hint: multiple data libraries sometimes come in handy if you have more than one tester, more than one application, etc).
• Will you maintain a full compliment of all objects at each level? (I recommend no. Keep each level empty except for the objects currently being tested at that level. Let you library list find items in the higher levels when you run your tests).
• How should you promote into this environment? (I recommend copy source, compile the object over the appropriate library list).
• Whom can execute the promotions at each level? (I recommend that Programmers can promote freely into the ST, but only QA can promote into QA and UAT).
• Would you like any one (or any group of people) to approve or sign off on the promotion before it can run? (I recommend yes! This is an easy way to get "electronic signatures" on file proving that you have followed your established procedures. It is also a fantastic way to end forever the classic complaint from the end user "this isn't what I was expecting"!)
• When a promotion is ready to be pulled in to the next environment, whom should be notified, how and what to say?
• When a promotion has successfully updated an environment, whom should be notified, how and what to say?
• Would you like to be able to recover (undo) a promotion into this environment? (I recommend yes to give you the ability to back out a promotion that really messed up your environment or that was promoted too soon.)
• When the promotion into the next environment is successful, would you like to remove the items from the lower environment? (I recommend yes for all items except the database files).

There are two "best of breed" testing tools that I have experience with. Original Software and SmartTest. Original's web site has a host of other great white papers that are worth reading to increase your base knowledge of the QA process.

Also, I'd highly recommend that you read this excellent white paper by Paul Conte, "Application Testing Basics: A Practical Guide to Improving Software Quality". You can find it here.