“There are two types of developers: those who test their software and those who will.” – Andy, circa 2015
It’s April Fool’s Day in the US, but I’m going to act like it’s Halloween. Software testing is no joke, and not testing should scare you.
In 1996 (yes kids, years used to begin with a “1”), in the sixth issue of Fast Company magazine, Charles Fishman wrote They Write the Right Stuff – an article about Lockheed Martin’s On-Board Shuttle Group and the work they did writing and maintaining software for NASA’s Space Shuttle program. Some interesting “nummers” from the article:
-
420,000 lines of code.
-
1 error in each of the previous (at the time of writing) versions.
-
The previous 11 versions of the software contained a total of 17 errors.
-
$35,000,000 / year budget.
I did the math: maintaining each line of code was $83.33/year. Plus 1/3rd of a penny. In 1996 money. According to the US Inflation Calculator, that’s equivalent to $124.12 per line-of-code per year today.
I mention the economics because I’m a consultant. I know how much I charge for my time. I have a pretty good idea what others charge for their time. I co-own Linchpin People, and we subcontract software developers for ourselves and other companies. I know what we pay software developers and how much we charge others for their services. At the risk of disillusioning those who aspire to co-own their own technical consulting firm, we have precisely 0 clients paying us $35,000,000 per year – even in 2015 money ($35 million in 1996 would be north of $52 million in 2015).
Why not? Because that’s a lot of money to pay for software. When lives, safety, and the pride of a nation are on the line, you’re paying for more than “just software.” The value of maintaining the Space Shuttle software was worth those costs. The cost/benefit analysis was sound.
What’s the value of your software?
Some Important Questions
As you ponder the “nummers,” you are likely thinking about the software in your enterprise. Or maybe just the programming-for-fun project you’re working on in your spare time. It may be prudent for you to spend some time thinking about the costs and benefits of your software. Here are some questions to get you started:
-
How important is your software?
-
What’s at stake if your software fails?
-
What are the risks?
-
Can your enterprise tolerate your software being offline for some period of time?
-
If so, for how long?
-
Are some times worse – riskier, higher stakes – than other times?
-
What’s your backup plan if your software dies and cannot be brought back?
-
What happens if you lose Production data?
-
Even all of it?
You may read that list and think, “Andy, you’re just trying to scare me.”
Guilty.
You durned right I’m trying to scare you. If fear is what it takes to wake you up to the realities of your situation, I’m not above scaring you. (In my mind, I just wrote, “I am an engineer.” But I digress…)
Process
If we were an agricultural society, it would be prudent to educate you about pests, soil care, and crop rotation. If we were manufacturers (and please understand, software is not manufactured…), it would be good to advise you to maintain a safe and healthy work environment for those who keep the machines running and keep your machinery in working order. But we make software.
What are the corollaries? One good way to mitigate risk is to possess and practice a process.
Before I go too far in this discussion about process, I want to reiterate my belief that people build processes to help people, processes are living and subject to maturity and change, therefore, people should always trump process. If you find a process harming a person’s life, liberty, or pursuit of happiness; you should re-examine the process. Processes should serve people, not the other way around. That’s ground rule #1 and it’s non-negotiable.
The Lockheed Martin team practiced a process. It is enumerated in the article (did you read the article yet? You should. Right now, if you haven’t already. It’ll only take a few minutes…):
-
The product is only as good as the plan for the product.
-
The best teamwork is a healthy rivalry.
-
The database is the software base.
-
Don’t just fix the mistakes — fix whatever permitted the mistake in the first place.
This list is rich. But you’ve read a lot already so I’m going to bring this post home by unpacking #4.
Mistakes Are Normal
Even at $124 per line of code per year, mistakes happen. If you believe you can pay enough people enough money to eliminate errors in software, you are mistaken (see what I did there?). And if one of the risks you identified in the Some Important Questions section above is, “People could get hurt,” you need to do all the engineering you can to see that people do not get hurt. But even if you do everything humanly possible to prevent and mitigate mistakes, you will never eliminate mistakes. So the very first step is to realize – and act like you realize – mistakes are going to happen.
Want to manufacture stress? Create a culture that does not tolerate mistakes. How? Rant, rave, yell, reprimand, and fire people when they fail.
Want to deliver great software? Create a culture that tolerates mistakes. Even better, create a culture where failing fast is celebrated. Why? People learn from mistakes. People who make more mistakes learn more. People who fail faster learn faster. People who learn faster make great software.
Mitigate the Negative Effects of Mistakes
Failing safely is the best way to mitigate the negative effects of mistakes, that’s why martial arts students first learn how to fall. Why? Because they’re going to fall! So are you. Identifying software errors quickly (failing fast) is a profitable practice. Testing software is the best way to identify software errors. There are some rules to software testing and the rules are important:
-
Test in a safe environment.
-
Test realistically.
-
Do not allow developers to test their own code.
Test in a Safe Environment
Set up a virtual machine. Something. Anything – except the Production server. There are conditions that need to be tested on the Production server. But, please, do not let “we need to test this in Production” be your first thought. Exhaust every other option first. Please.
Test Realistically
The reason you sometimes need to test software in Production is that there simply isn’t another server or environment in the enterprise that looks and acts like Production. There can be sound business and economic and timing reasons for such a condition. If you cannot test realistically in a safe environment, test as realistically as you can in a safe environment before testing in Production.
Do Not Allow Developers to Test Their Own Code
Do you want to read some stories of heartache? Some rants? Some helpful advice? Search for “Developers test their own code.” It will bum you out. And it may also help. Let me state up front that I develop software (SSIS is software even though contains “SQL Server” in its name). I miss things when I test my own code. Some of the links from the search above will contain stories about how and why this happens. Trust me. It happens.
Get someone other than the developer to test the code. Prepare a test plan – even if the plan is “run this script on this test server.” That’s a test plan. It’s good enough for someone who understands what the code is supposed to do, or for anyone capable of interpreting a green/red indicator.
Developers should conduct unit tests and functional tests. They should execute the code in a development environment to make sure it runs, does what they think it should do, and returns the expected results. Then someone else should run their code somewhere else to make sure all those things happen over there – in some location other than the environment where the software was built.
Conclusion
Remember: All software is tested; some, intentionally. What does that mean? That means that untested code is going to be tested in Production. Sometimes, the people testing your software in Production are your soon-to-be former largest customer. Think about it. Please. If this scares you, then good: I’ve accomplished what I set out to do.
:{>
Learn more:
Stairway to Biml
Linchpin People Blog: SSIS
Stairway to Integration Services
“There are two types of developers: those who test their software and those who will.”
"All software is tested; some, intentionally."
Those are the two best aphorisms I’ve heard this year. I always enjoy reading your posts on "Doing Software Right".
"Don’t just fix the mistakes — fix whatever permitted the mistake in the first place." This is so true. And on so many levels. So many times I see post-mortems that simply state, "preventable with more automated tests" yet the root cause is usually that neither the analyst nor the developer nor the QA guy really anticipated how the user would use the software. Automated tests won’t solve that…understanding your domain, your user, and your product will.