Profit-based TestingProfit-based Testing
Colleges, technicons, and universities do not generally teach software testing as part of Computer Science studies or as part of Information Systems curricula – and it shows!
Other disciplines, be they old or very new such as engineering and biotechnology respectively, are far more scientific and measured in their approach to problems and solutions than IT as a discipline with its business analysts, software designers, programmers, testers, and managers.
It is as if IT operates in a special dispensation that allows it to be sloppy. Business often pays for work estimates that have no real basis, and even then these are rarely met. If deadlines are met, this is often achieved by dropping a large percentage of the functions originally requested.
But hey… no one is measuring… and even if they are, almost everyone expects a losing result - so they dig deeper, wait longer, and adjust already low expectations downwards! For those who outsource, in-source, divorce, tomato sauce, right-size, downsize, up-skill, offshore, even these attempts backfire more often than not. It is time we considered the fundamentals.
Let us start with the basics and build up from there.
Innovation and service delivery drive business profits in competitive markets. Today, IT either drives or significantly supports service delivery. Software testing practice largely determines cost, quality, and schedule benefits during initial IT delivery and maintenance.
Testing practice determines IT outcome, which in turn determines business service delivery. Software testing improves the reliability, security, usability and functionality deliverables of the product.
The paradigm of the day is risk-based testing. Testers and project stakeholders must identify risks to the business and then address these risks by applying appropriate testing processes and techniques to mitigate each risk.
Although currently popular, this risky approach essentially makes everyone think of software testing as a grudge purchase similar to insurance (an unavoidable, unpleasant expense) – of course that’s until you need it!
We need a new view of testing in the IT industry! We need a new language for reporting testing benefits. Software testers must make the main goal of testing clear. Testing has exactly the same goal as the business itself, to help the company optimise profit (or value) - no matter how that is measured in a particular organisation.
Reducing risk is one thing. Reducing IT costs is another. Software testing has the big opportunity of optimising profit for the company while still achieving the first two.
The economic and strategic significance of software testing is the most misunderstood opportunity in IT today. CEOs, CIOs, project managers, IT stakeholders, and even test practitioners themselves have, on the whole, not yet discovered the real plot when it comes to software testing.
Profit equals total income minus total expenses.
As a simple equation:
Slightly expanding the equation we get:
It follows that one way to increase company profit is to reduce testing and development expenses. IT expenses are significant and can easily run up to ten or more percent of a company’s revenue. Any savings made here drop straight to the bottom line profitability of a company.
By way of using a fictitious business as an example, I will show you how to effect cost savings within your company/project by improving software testing. These cost savings will not only occur in testing, but also in the development, build and maintenance processes.
Our example business has revenues of 10,000,000 rand directly related to software web services. The IT budget is one million rand, of which development is allocated 700,000 rand and software testing 300,000 rand. Management has heard that improved software testing practices can reduce software testing costs as well as development costs and wishes to understand exactly how to achieve this.
The profit formula for the illustrative business case is therefore:
By which profit for the illustrative business can be calculated as:
View the bar chart below to see the relationship between revenue, testing and development expenses, and company profit.
Figure 1: Illustrative Business Case Study - Profit and its Components before IT Savings from Better Testing
A small amount of good test training can go a long way to reducing overall testing and therefore project and company IT costs.
Measured results from avionics software have shown Boundary Value Analysis (BVA) as a test technique to be 50 000 times more effective at detecting faults than merely conducting random tests (Stuart Reid, 1997).
In this case study, only a limited selection of testing interventions will be discussed. Many more opportunities to save costs are available, but it is beyond the scope of this article to detail them or even to mention them further. Actual savings are educated guesses. The point of this article is not to detail exact figures, but rather to show by order of magnitude reasoning, why testing should be given emphasis in a company or IT project situation.
Table 1: Cost Savings from Investing in Software Test Training
Table 1 shows saving from software test training investment.
Table 2: Cost Savings from Investing in Software Test Automation
Table 2 shows saving from test automation investment.
Testing is a discipline also intended for developers. It is not enough that only testers test – developers must also be involved in component and component integration testing. A developer should be involved with testing activities (which do not include debugging) for 20 to 30 percent of their work hours. Developers should plan, specify, and execute tests and record test results for later reporting.
Note that in Table 3, the testers experience cost savings although the developers invested in improved testing. This saving results from having to write fewer error logs, and because of fewer testing delays than an inferior product would produce.
Table 3: Cost Savings from Developer Test Training and White Box Tools
Testing is a discipline also intended for business analysts. It is not enough that only testers test – business analysts must also be involved in static testing and acceptance testing (user, contract, factory, and site acceptance). A business analyst should be involved with testing activities (which do not include requirements elicitation and specification) for 20 to 30 percent of their work hours. Business analysts should plan, specify, and execute tests and record test results for later reporting.
Table 4: Cost Savings from Business Analyst Training and Reviews (Static Testing)
Note that in Table 4, the testers experience savings, although it was the business analysts who improved their testing. This saving results from having to write fewer error logs and because of fewer testing delays than an inferior product would produce. The substantial improvement in the product specifications will assist test planning and reduce the number of test cycles required. Apart from savings from less testing, another benefit will be faster time to market for the company.
We now recalculate profit for our illustrative case study considering these savings (see Figure 2).
Profit, after savings, is now calculated to be:
Figure 2: Illustrative Business Case Study - Profit and its Components after IT Savings from Better Testing
This is an increase in overall project or company profit of 3.94 percent - and the company has now bought tools and the people have received training as a bonus.
This is all well and good; an investment in good testing principles leads to cost reductions and allows us to increase the bottom line profit. The great news is that this is not yet the end of the article. We shall now see the benefits of allowing testing to lead improvements on the revenue side.
Better usability web systems have been measured to improve revenues by as much as 40 percent.
By investing (for example) a further 70 000 rand in usability, revenues of the case study can conceivably improve by 40 percent (this can be accurately measured in a real case).
We can now see the revenue benefits of test led improvements. Usability, security, performance, maintainability, reliability, flexibility, and other quality attributes typically have the potential to create leading products and services. Tom Peters, in his book Re-Imagine, says that excellent design will be the competitive edge of business in the current century. The Apple® iPod® is a good example of compelling design generating strong revenues.
Profit, after test-led revenue improvements, is now:
Figure 3: Illustrative Business Case Study - Profit and its Components after Business Revenue Increases from Better Testing
This is an increase in overall project or company profit of 47.6 percent, and the company has now bought tools and the people have received training as a bonus. Do I need to point out that the extra revenue generated by test-led design (quality) improvements has completely eclipsed all of the IT development costs?
This should make us think!
Welcome to the new Millennium, welcome to a new breed of IT competition. We have to give testing its strategic position within our organisations.
Do not believe any testing benefits or claims that you do not measure! Measure to prove that there is in fact benefit.
Good testing practice can create additional profit by reducing test and development costs and by increasing revenues from product improvements and market delivery improvements (sometimes revenue is critically dependent on the ability to successfully deliver through a channel). Using testing to improve quality attributes like usability, flexibility, security, and performance can increase business revenues and therefore effectively dwarf IT expenditures.
Testing closes the final gap to the user experience.
Increasing revenues and decreasing costs is not impossible.Does profit-based testing address risks? Yes! Risks that translate to incidents have a cost associated with them, thus reducing revenues. Risk-based testing is a subset of profit-based testing.
The key difference with profit-based testing is the economic and business alignment of testing goals in everyone’s thinking. We don’t get excited about paying insurance (at least not on a daily basis). However, we all do get excited with making additional profit.
For non-profit organisations we could possibly accept the name value-based testing, but profit-based testing is likely to catch more attention in the boardroom.
Not considering the benefits and necessity of good testingis a breach of good governance and a dissipation of shareholder wealth.
Wayne Mallinson
info@testfocus.co.za
The reproduction, adaptation or broadcast, without permission, of any articles
or photographs published in this
publication, is forbidden and copyright
is expressly reserved for Test and Data Services (Pty) Ltd.
Opinions expressed by contributors are their own and do not necessarily express the opinions of the Publisher, Editor or members of the Editorial Panel.