Software testing
Published by peter June 25th, 2006 in java.A short discussion with my father today made me think about environments for software testing. One of my previous projects had a full OTAP environment for testing (Dutch acronym for development, system test, functional acceptance test and production). The current project features a testserver and database.
Eventhough it seems to be common practice to have isolated environments in which the application can be tested both cases as described above don’t even approach the final production environments:
- slower machines (less cpu’s and less memory, slower disks)
- less machines in the cluster
- far less concurrent users
- different configuration requirements (errors in configuration files are amongst the most common problems I’ve seen)
- different security setups (the current test machines allow outgoing HTTP requests whilst the production machines don’t)
- different JVM versions (the minor number, but still)
- test machines share the VM to other processes
The differences are so big that a application working for the full 100% on the test environment might not work at all in the production environment. So if the application is accepted as working on the test environment… what does that mean? Is it now safe to replace the old version of the production application by this new version? Who is responsible if a bug which only occurs in the production environment corrupts important data? What if the application suddenly exposes sensitive information when running in a cluster?
Is it actually possible to simulate the production environment in a scale which makes it possible to fully test the application, and is such a sollution still commercially viable?
A member of a highly respected consultancy organisation presented a story about “operational excellence”.
One of his “best practices” was to test outside office hours with full production load.
Our experience: We always deploy generic solutions for 100K users around the world, 7×24 available, very limited maintenance windows. In one particular case we really executed a comprehensive test plan, anonimized data captured from production, home made test drivers, thousands of euros. And guess what: no errors found. We then released into production: continuous stream of incidents, many weeks of instability. Conclusion: don’t test anymore, it is not worth the money!
The “best practice” may be fine for a specific business application with clearly defined small user group and predictable transaction load. But not for complex Internet oriented with unknown crowds of users.
We now work with a staging process, regression test set for reasonable level of functional testing, coexistence of old and new solutions, gradual deployment, small group of known and tolerant users first. And invest in backout procedures, monitoring during and after deployment, good alert mechanism to keep users and help desks informed about system outages, known problems and work arounds.
Very similar to transplanting a heart. You can’t test it either!