One of his “best practices” was to test outside office hours with full production load.

Our experience: We always deploy generic solutions for 100K users around the world, 7×24 available, very limited maintenance windows. In one particular case we really executed a comprehensive test plan, anonimized data captured from production, home made test drivers, thousands of euros. And guess what: no errors found. We then released into production: continuous stream of incidents, many weeks of instability. Conclusion: don’t test anymore, it is not worth the money!

The “best practice” may be fine for a specific business application with clearly defined small user group and predictable transaction load. But not for complex Internet oriented with unknown crowds of users.

We now work with a staging process, regression test set for reasonable level of functional testing, coexistence of old and new solutions, gradual deployment, small group of known and tolerant users first. And invest in backout procedures, monitoring during and after deployment, good alert mechanism to keep users and help desks informed about system outages, known problems and work arounds.

Very similar to transplanting a heart. You can’t test it either!