QA and Testing…Use a model office or test live….

This past couple of weeks has seen me overseeing some of our components as they go through QA and testing phases, be it early phases of a bunch of our products. I have also been keeping tabs on some of the testing of solutions we are providing for clients, and to be honest, I do feel we have a great approach to QA and testing. We always ensure components are tested in full in house, and then go through a number of testing iterations on either a model office environment or the actual live environment…

However, in the past month I have also seen a number of “testing phases” by organisations that simply are not realistic, and to be frank, mean you will need to repeat all that testing and work again…So what is the issue. Well, typically it is testing in environments that don’t quite match those that the final system will be implemented on…Now to me, that is mad and asking for trouble, and yet there are endless organisations out there that do this….

Making the testing phase easier…

The primary reason for cutting corners or not testing on exact replicas of live environments is admin and cost. To set up too exact environments will take time and effort, and for some organisations they just don’t want to invest either in this area of development…

Because of this, there are a number of options project managers take, some are simply to use whatever environment they have that is close to the live one for testing. Others are to utilise automated testing tools too much (to save on man power and to shorten the testing period) or finally, others use cloud computing to cut down on the admin and set costs, while maximising the potential users / tester access to the solution.

I have no real problem with any of these, however I would say, if you are in this situation, utilise what will be the live environment for your testing, if possible, and simply clean it down when you ready to go live….Whatever you do, don’t though rely on just that testing if there is any difference, no matter how small between to the two environments…Only today I have seen that an updated component on the server (different to that of the testing environment) caused what potentially could have been a big issue. However, luckily, the client in question took our advice and was testing the solution from scratch on the live environment…

Using the Cloud to test…

I have read that people are now looking to use cloud computing to carry out their QA and testing. Again this is fine, if your application and software is to be running in a cloud environment and on the servers provided by the same cloud provider. If not, it is still fine, as long as you then test in full again, on the “live environment”…

With cloud testing you have to remember, that you may have a complete change of underlying server components without actually being aware of it. You may also find that your cloud provider uses other cloud providers for certain components, making it even harder to be sure just what components are where (and what version). So this testing requires strict control and good relations with your cloud provider.

Unfortunately I have now seen software being tested in this way that is to be implemented in house, not up on the cloud. For me this is a massive no no….Why are they doing it? The answer is usually a simple one; it keeps the cost of testing environments down and ensures a wider range of users can gain quick access to the system to perform QA and user acceptance testing. Especially since what will be the live environment isn’t available (for whatever reason).

The basics of this are ok, however, when you get into the live environment you need to repeat all those tests, otherwise you leave yourself open to a raft of issues that will raise their ugly heads, not necessarily on go live day, but at any point…You see, any small difference can potentially have a massive impact, and testing and QA is all about minimising the risk of the system going wrong. If you don’t test on the correct platforms then that risk can only be increased, no matter how much testing you do…

Model office

When I first started in IT, I was told by a wise old man (he actually was old, and looking back, very wise, oh, and short tempered), “when testing, for God sake ensure everything is the same as it will be in live…..you can never test too much son….never…” He was a strong supporter of using a Model Office environment, and by model office environment, he meant an exact replica of the live environment…Right down to hardware used….He also believed that environment needed to be there long after go live was complete, and that any components that were to make their way to live, (even if OS related) should be tested on the model office in full first….Now I thought this goes without saying, however my first exposure to a project that required a lot of testing was for a very large bank. It soon became apparent that their testing environment wasn’t an exact replica of what would eventually be the live environment. There were subtle differences in hardware (which it turned out did come back to bite us), there were even some instances of components on the testing environment that were and would be newer than those on the live…

What has to be understood is, testing is as important as design and development phases. It cannot be scrimped on, and your testing environment should be there for the life time of your solution. If this isn’t possible, you need to come up with a way of making this possible, if not, your QA and testing will always leave you at greater risk come the day the system goes live….