Where multiple teams are involved, new things must be motivated, synchronised, piloted, and rolled out to other teams upon success.Where multiple teams are involved, new things must be motivated, synchronised, piloted, and rolled out to other teams upon success.
A test manager on a small project isn’t necessarily prepared for managing numerous simultaneous projects where production volumes of testing are required. Testing does not scale linearly. With increasing production volumes, comes the requirement (and added benefits) of increased productivity.
Good communication is easy to maintain in a small team, in larger groups it is necessary to rely on more thorough planning, standardisation, fixed processes, templates, reports and frequent meetings to keep people on the same page, and get the work done timeously, within budget, without a drop in quality. In a small team it is easy to try something new. With multiple teams, new things must be motivated, synchronised, piloted, and rolled out to other teams upon success.
A larger size helps an organisation to reach critical mass and to specialise. Critical mass in testing terms might be when an organisation has enough work to keep 1 or more specialists busy full time.
Given that there are production volumes of testing in an organisation it should follow that the organisation has established standards and processes in development. Often these established processes existed before the advantages of testing were understood and the latest approaches to testing were known. The up-scaling of testing may force a change in the timing and way development are done, and will create new initiatives in testing. A formalised approach to large production volumes of testing will necessitate large (albeit positive) changes in the development environment.
Test Policy Document
There is good cause for companies with large testing volumes to create a test policy document. This document sums up the company’s testing philosophy. It’s approved by senior stakeholders and can be used to guide decisions and direction that testing must take. It states the company’s view on testing, helping individuals, projects and divisions to make decisions that are aligned to the policy. Providing that the policy is created with important stakeholder input, it can be very useful for aligning everyone to the testing imperatives of the organisation. (See Figure 1.)
Example Testing Policy Statements |
|
Figure 1: Example Testing Policy Statements
To experience success in companies with large production volumes of testing, one must measure testing parameters. Unmeasured productivity is nonsensical, for the essence of productivity is based on work accomplished/rand, hour or person. At the least, to see improvements and trends in productivity, measure the current work done, and rands, hours, people used to achieve this and compare this to the past.
Where project requirements are clear, accurate test effort estimation plays a key role in determining productivity. If we can accurately estimate test effort using manual testing in a routine way, we can change the way testing is done and calculate if it was more productive. An approach to team management that has worked well for me in delivering actionable test metrics has been to use Detailed Requirements Extraction and Management™ (DREAM).
DREAM bases test estimation on all the delivered requirements or a sampling of them. Once test effort is calculated it can be efficiently delegated, supervised, reviewed or audited by a single test manager. The estimation effort also forms the basis for test planning and the necessary monitoring and controlling keep testers on plan. Test status can be maintained and reported within minutes.
DREAM supports a documented, systematic, organised approach to testing – including a means for risk vs. effort prioritisation – and produces metrics that are quickly understood by stakeholders. Team assignments can be given and tracked to completion against requirements for test. This helps place specific accountability in terms of work to be completed and time-lines.
The progress of each team should be known as early as possible, as how projects use shared test environments, needs to be closely controlled to limit the tendency of the environments to become bottlenecks. Forewarned by accurate status reports is forearmed with alternative solutions, even if it is moving testers to shift work.
The detailed operations of applications need to be tested at system level, including interfaces to other systems. Testing work on large, complex systems can be divided functionally (or technical or other basis) and delegated to different testers or teams.
Checklists, recorded auditable test cases and results must be kept for recourse. This level of testing requires careful, informed test planning and design, guaranteed execution with reliable and traceable record keeping.
Testing should occupy a high, scientific position in organisations, with critical thinking pursuing a way to bring engineering discipline to the creativity of development. To leave the interpretation, quality, and schedule to chance without objective measurements and point of view of testing is naïve. Managing production volumes of testing stretches the management discipline and often is a case where test managers are given responsibility but not authority. In many businesses, they report to project managers and subjected to their budgets, deadlines and schedules.
Forge winning teams with support from project managers and mutual respect for each members’ contributions – the only way productivity, innovation, and differentiation can be sustainably. The culture that encourages teamwork, allows people to fail and try again, has mutual respect and is one where testing can fly. With tighter budgets and deadlines, more change, competition, regulations and more complex, integrated software challenges – only organisations with good communication from mutual trust, can succeed. System testing is the general responsibility of full time permanent or professional test practitioners.
I’ll explain the way acceptance testing differs from automation testing differs from system testing with this car analogy. System testing is checking nuts and bolts. User Acceptance Testing (UAT) is the user getting into the car and taking it for a drive on various roadways, motorways, rough, smooth, wet and dry to name some test scenarios.
Contract Acceptance Testing (CAT) is checking areas where users typically do not go e.g. measuring wheel alignment, crash testing etc.
Regulatory Acceptance Testing (RAT) might be where municipal authorities check that vehicle roadworthiness meets their legislated standards.
Factory Acceptance Testing (FAT) and Site Acceptance Testing (SAT) might be considered subsets of CAT. These would include car factory tests, such as waterproofing, vibration etc. SAT might be considered testing on specifically designed vehicle test tracks e.g. general obstacle courses for 4 x 4’s.
Operational Acceptance Testing (OAT) is a high level acceptance test – often the market acceptance and validation of a product – where the desired ‘business’ result is measured e.g.the new car was so popular, it increased our market share by 3% in 14 months.
Alpha and Beta testing can be considered as acceptance tests, although these are typically earlier and more limited phases of testing with ‘friendly’ users – to some extent used to quickly determine remaining faults before rolling out to larger audiences.
Acceptance testing is the responsibility of the business - a failed acceptance test is a business error. This controversial point is a necessary sentinel to business management everywhere and is easy to prove. If a software failure occurs, business will say “we should not have employed ‘these people’, this was a mistake!” Yes it was a mistake, your mistake – but often not because of ‘these people’.
Which safeguards and mechanisms did you put in place to assure success? I’m assuming you purchased the service, you must take responsibility for the result, and ought to have abandoned or fixed a failing project much earlier than acceptance testing!
Never take acceptance testing lightly. The specification and the tests form part of legal contract and a transfer of ownership occurs when software is accepted or signed off.Wayne Mallinson
info@testfocus.co.za
The reproduction, adaptation or broadcast, without permission, of any articles
or photographs published in this
publication, is forbidden and copyright
is expressly reserved for Test and Data Services (Pty) Ltd.
Opinions expressed by contributors are their own and do not necessarily express the opinions of the Publisher, Editor or members of the Editorial Panel.