Software testing at worst is a conformance activity executed at a late stage within a project, as the code is run, and failures are detected and recorded for repair. In such cases testing (even automated testing) will add little value, be very inefficient, and will fail to lift the system built past very mediocre quality levels.
At best, software testing will begin early in a project, be a catalyst for continuous improvement, provide all stakeholders with rapid feedback and feedforward, and elevate the project to:
Lean testing results from consolidating seventeen years of my experience in software testing (working together with software engineers, developers, project managers, business managers, and customers of software development projects) into a profound framework of 14 management principles used by the Toyota Corporation and documented by Jeffrey K. Liker1 in The Toyota Way.
Why would we want to do lean testing? Production of software can be likened to the production of cars; you will be able to identify situations where the management principles, discussed later in this article, can (and should) be applied to software production. Even if you are not interested, your management and shareholders might be! Toyota's annual profit for the financial year ending March 2003 was 8.13 billion dollars - exceeding the combined earnings of General Motors, Chrysler, and Ford.
Your project managers and developers will be interested to hear that Toyota has the fastest development process in the world. New vehicles are designed in 12 months or less; competitors typically require two to three years.
Your customers would be intrigued to know that although there is little to differentiate quality when cars are first delivered, after three years the gap in quality between Toyota and its competitors grows, and then later the gap balloons. In the 2003 annual issue of Consumer Reports, Toyota's cars had the least defects for the first three years of ownership; three times fewer defects than its US and European counterparts.
The Toyota lean production method has been successfully used in service industries and office environments, although not directly comparable to building cars. Lead time reductions in service organisations of up to 66% and higher, reduction in rework by 80%, and productivity increases of 29% have resulted, and many other achievements - best read in the context of documented case studies.
Real software engineering is not for the fainthearted. If you haven't got the hunger for improvement, the time, the tenacity, the vision, the faith... then rather don't kid yourself, mislead your colleagues and waste your company's resources - take early retirement, change careers - hand the reins to someone who has what it takes!
On the other hand, if you really want to make a positive contribution: read and consider these principles, and examples of how they apply to software engineering as a start to a more serious journey into lean testing and Software Engineering Operational Excellence.
Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals.
Toyota has some long-term beliefs that they adhere to. It is important for them to have a long-term goal that is bigger than making money. They believe in generating value for their customers, society, and the economy in which they work (this generation of value seems to attract money, and lots of it, but more as a side effect).
To achieve this philosophy or value, they firmly support continuous process improvement and respect for people.
This belief helps them to do the right thing - even if it is the tough thing, and impacts quarterly financials negatively. (Remember, respect for people - as examples: consult with that market segment; prototype to find the most usable design alternative; don't ship those bugs!)
This principle is not one of many items a company can choose from; in order to become lean, it is absolutely fundamental and foundational. Senior management must be committed to it; proper training must be given to employees, and a culture of sustaining improvement must become pervasive at all levels.
IT projects might be a culprit for breaking this long-term philosophy in many companies as, by definition, projects end. If a company's project managers are only measured against project goals (short-term) then lean thinking and achievement (requiring long-term foundations and results) may well be impossible. I've heard it said, "You get what you measure!" It is therefore the duty of senior management - the development director, CIO and CEO - to consider the strategic advantages of lean production, and then invest (time, money, and reputation) in this direction.
This is also a long-term journey. In this age of instant coffee and immediate news, it is difficult for us to muster the resolve, and to spend the time, to master something. 'Difficult' does not mean 'impossible', and I believe that a company's very survival, in the next 15 to 20 years, might hinge on starting to move in this key direction today!
Company boards should begin to invest in people, plant, and product: training, respect, and empowerment for people; tooling, hardware, and software environments (plant); and product quality from concept specification through to design optimisation - particularly in non-functional quality criteria such as security, maintainability, usability, and performance - and then technical integrity.
Create continuous process flow to bring problems to the surface.
Many companies' software processes are designed around batch models: one person or team completes a task and then hands it on to the next person or team. Think of a requirements specification being completed and then, only after approval, a design phase being entered, and then development of the code, etc. Batch models are the antithesis of lean thinking and suffer from many setbacks, the lack of which distinguishes lean production as a good strategic model.
In real terms, significant software problems may occur during an early phase of the software development life cycle, only to be discovered much later. For example defects in a specification may only be discovered three days/weeks/months later when a product is acceptance tested. By this time the cause and effect trail has gone cold, and the person who injected the defect (maybe a business analyst) learns nothing, and therefore will continue to enter those types of defects into software projects. Even worse, the cost of fixing the defect during the acceptance testing phase is approximately 30 to 70 times more expensive than during the requirements phase.
With lean thinking and flow it may be possible to approve individual or small groups of requirements, which may then be incrementally designed and coded, even on the same day. If defects occur (and they will) they will be immediately spotted, and sent back to the source for correction. The business analyst in this case would learn what was not acceptable.
If some design and testing members were involved during the requirements specification stage, the errors may have been avoided altogether. If the team was daily designing and coding requirements as the business analyst produced them, then the system would be very sensitive to defects anywhere in the pipeline. For this reason lean production is very difficult to buy into. Design, for example, would have to wait while business analysis improved a sub-standard specification. Coding would have to wait while designs were straightened out. Intuitively then, we might revert to batching: a month's worth of specification defects are collected, and corrected, without full team participation and learning.
When value flows are created, any problems can (will) stop the whole flow and value delivery. This forces everybody on the shop floor (in the project team) to learn to quickly deal effectively with defects. Very quickly people realise (often for the first time) that avoiding defects is better than removing defects: your team picks up quality principles. With batch processes, nobody takes ownership: nobody learns preventative measures, and the processes have huge efficiency potentials.
With lean production you will be shocked at the volume and serious nature of the problems that are flowing in the system. And as hamsters may be shocked to condition-required responses, your team will quickly learn to do things correctly or experience pain.
The discovery, discussion, responsibility, and repair of newly injected defects raises the participation, and ownership, of project members; many lean benefits will begin to be realised. Beware at this stage! Don't be fooled into thinking too simplistically, as the other twelve principles must also be applied for ultimate lean testing success!
As water is let out of a dam, the original flow of the river may be seen, and obstacles like rocks (or 42% rework inefficiencies - industry experience) may come to the surface. Lean production is good at exposing and significantly reducing or even eliminating types of waste. Toyota recognised eight forms of waste, easily targeted and reduced in their lean production systems:
Space precludes mentioning too many IT project parallels, but here are some that may ring a bell.
Resources are wasted when we engineer or test each requirement with the same effort, and fail to prioritise our efforts based on effort, risk, and cost benefit. In this case, because we attribute too much attention to less important requirements, we may make the double error of not spending sufficient effort on critical requirements.
Resources are wasted when tests done during one phase overlap excessively with tests done during another phase.
Resources are wasted where we do not consult with the users/market sufficiently at many incremental reviews, and the product falls short of their expectations.
Resources are wasted when we all wait for the last developer to complete a module so that the whole system can be tested, and errors reported back to development.
Resources are wasted when sixty percent of the product defects have been included in the specification, and only when the specification is completed do we decide to review it and remove these defects.
Resources are wasted when a test automator must wait while a test automation script, which is too long, must run for three minutes to repeat a defect that occurs near its completion.
Resources are wasted when we have to drive to the client in order to redo work or pacify the client.
Resources are wasted when a test manager brings the whole team to a meeting - when their representation alone would have been sufficient.
Resources are wasted when we have to walk to meeting after meeting to prioritise defects, and the fixing of incorrect designs, that could have been avoided.
Resources are wasted when we include too many features in a product, which are unnecessary to its sale or usefulness, and may even be detrimental to its usability or performance.
Resources are wasted when testers test every field of 300 screens and report similar errors throughout. Sampling 3 to 10 screens could have given the developers sufficient information to correct the defects.
Resources are wasted when we engineer non-functional qualities beyond where the client cares. A performance example is when we may spend days of effort to get a one-second response time - when the client only views a report once a day, and would be absolutely happy with a four-second response time.
Resources are wasted when we have too many developers during the design phase of a project who cannot be otherwise employed on an alternative project. Or, in general, too many of X resources during phase Y.
Resources are wasted when the whole production data set is installed to test the logic of a few functions.
Resources are wasted when we purchase too many test automation tool licenses, when we haven't been able to deliver measured value from a few of the same licences (even though we got a discount).
Resources are wasted when products are moved into production, and then backed out for further repair and testing, especially when this happens more than once. Think also of test and repair cycles being excessive.
Resources are wasted when defects are reported too frequently to developers/business analysts/system architects, given the fact that these persons would be interrupted too frequently to be productive.
Resources are wasted when we over-manage and overshadow people for want of good visible progress charts.
Resources are wasted when we have to allocate talented developers in large percentages to repair work, rather than to new product creation.
Resources are wasted when we don't do root cause analysis on defects to learn how to prevent them from reoccurring.
Resources are wasted when our applications fail (damaged data integrity, lose the service for a period, overpay clients, damage our reputation), and must be rebooted or taken back or extracted from punitive service level agreements.
Resources are wasted when employees are not given a working framework that will help them to take pride, ownership, and responsibility for their own defects and successes.
Resources are wasted when defects/designs/specifications/tests/processes are not examined with a mind, mandate, and culture to suggest and bring about improvements to them.
Resources are wasted when basics have not been communicated by means of training in key process and information areas.
In the next edition of Test Focus, I will start by elaborating on the third principle of Lean Production as it relates to software testing.