Advanced Solutions International
Number of people on the test team: 7
Number of developers supported by the test team: 31
Best Practice Description :
Quality Assurance As a Career
The Quality Assurance Department at Advanced Solutions International is not a stepping stone to other departments or careers. ASI provides the infrastructure and training that will create a self-sustaining team of trained professionals.
Certification is the fundamental goal of this training. ASI requires every team member to obtain at least one certification and encourages employees to obtain multiple.
Currently team members have obtained Quality Assurance and Software Testing certifications from the American Society of Quality and the International Institute for Software Testing. While newer team members are working on completing their first certification, most of the team is working on their second and some team members are even pursuing their third.
The last link is ongoing training. Each team member's skill set is reviewed annually. This review is used to develop a training plan for that individual and insures they have the most current skills, processes and methodologies at their disposal.
Requirements
While the requirements begin with customer and market input and are refined by Product Management and Business Analysts, the quality assurance department gets involved as soon as the initial draft of the Software Requirements Specification is complete.
This SRS is posted to an internal community and all disciplines including QA post comments and questions to the community. This not only allows the tester an early preview of the upcoming features but gives them an opportunity to use their broad experience to help the Business Analysts create a more comprehensive specification.
The SRS is then used as a source document for both Use Case and Test Plan Development. Using the SRS, Test Plans are developed at a high level since exact implementation details are unknown at this point. Testers also match up requirements in the SRS to specific test cases in the Test Plan. Use cases further flesh out the test plan details, as each use case is translated into a series of tests cases that are used to validate the feature.
To further ensure high quality requirement, the Business Analyst and QA Teams have participated in several onsite IIST training events such as “Writing Testable Requirements” and “Discovering & Testing Requirements with Use Cases.” These events were designed to expose both departments to the challenges that other departments encounter when evaluating requirements. Both departments were able to compare their assumptions with those of the other department. In so doing, both teams gained a better understanding of what it means to develop requirements with the least amount of ambiguity.
Test Design
Test Design is based on a common philosophy and uses a standard template for test plans. Our philosophy is very simple: The evaluation of a product is a continuous and planned process. “Continuous” means that, with each release, we evaluate our effectiveness and make changes to improve our process. “Planned” means that all testing is done with a written test plan and never ad hoc.
All test plans and test cases are designed with the following goals in mind:
- Maximum coverage with minimal effort (extraneous tests cases are eliminated via reviews and logical analysis of test criteria).
- High risk areas are identified and additional detail or tests are created to insure those areas are fully tested.
- Integration Testing exit criteria are identified up front. These criteria must be met (passed) before the product is released to formal test.
Test Design also requires that each test case be written with sufficient detail so that any other member of the test team can execute the test. To accomplish this, each test follows the process outlined in the test plan template. This template insures that all required minimum data is available to execute the test. In addition, the template allows the test writer the latitude to include any other data necessary to understand and complete the test.
This test design philosophy is shared across the entire team. That means that regardless of who is writing a test plan, everyone can rely on the fact that the test design and its implementation are standard.
Test Execution
Test execution begins with the final review of the test pan. All departments participate in a formal test plan review. This review is occurs within the 2 weeks prior to the products release to formal test. This minimizes the impact of late changes in implementation.
Formal Testing follows the same workflow that our customers execute when installing the product, including client/server upgrades or installs and database upgrade paths.
Once all components are installed, tests are executed per the contents of the test plan:
- As tests are executed, the test plan is updated
- Pass – tester, date and status are entered
- Fail – tester, date, defect # and status are entered
- Defect issue is submitted
- Severity of the issue is included if applicable
- Limits testing
- Blocks testing
- Test plans are checked into a repository daily. Version control is maintained via the repository for the life of the test plan.
Test progress stats are posted to the production team weekly. In addition, each tester reviews the status of their test plan with the appropriate feature team on a weekly basis.
Test Management
In addition to the company wide Product Development Plan, the Quality Assurance Department also creates a Project Test Plan. This published plan includes:
- Process changes necessary due to limitations or restrictions inherent in the build. i.e. upgrading may not be possible in this build
- Test environment specific to each tester
- Process for returning failures
- Priorities – what gets tested first
As test plans are executed, testers annotate the state of the test cases:
- Untested Test case not run
- Failed Test case run and failed
- Blocked Unable to execute because of defect that blocks the test
- Hold Test in HOLD waiting on information
- FP Failed/Pass Test Case failed then passed (used to calculate failure rate )
- Passed All criteria successfully met for the test case
Test stats are compiled automatically for individual test plans as well as all the test plans for the release
- # pass, fail, block,% pass, % fail, % complete
Using these numbers, we extrapolate how many days are left until all test cases are run and passed. This date is matched to the planned released date to determine if we are on schedule. The product is not ready for release to the public until all test cases are executed and passed.
Using historical data we can estimate of the # of issues that the tester can be expected to execute in a day. Additionally, we factor in the failure rate as established for the testing to that point, to account for any retest.
Reviews/Feedback
ASI uses continuous improvement methods to determine the effectiveness of processes and test execution:
After a release, the team meets to review the objectives established in the Project Test Plan and determine if these objectives were met. In addition, success and failures are reviewed and process changes are implemented to take advantage of successes and to correct failures.
This meeting is conducted by all department managers and covers those areas that individual departments do not control. The object is to identify those high level successes and failures that need to be implemented or corrected prior to the next product release. Feed back from this session is returned to individual departments for inclusion in their internal processes as necessary.
- Post release defect reviews:
Every defect entered by an ASI customer is reviewed by QA. The purpose of this review is to determine why the defect was released to the field. Defects are categorized according to root cause:
- System Design- the issue is actually a design decision and not a ‘true defect'
- Incomplete testing- QA missed the bug, but should have caught it in testing.
- Regression- the bug made it to the customer because of incomplete regression testing.
- Side effect of new technology- upgrading a driver, library, etc. caused an unforeseen issue.
Reports are run to determine the defect trend for a release and how that trend compares to previous releases. When an omission in the test process is exposed, QA updates their QA Best Practice knowledge base. This entry will contain all of the relevant information as to what the defect is, how to correctly test that defect, and how to avoid similar situations in the future. The entry is linked to the source defect from the customer, providing the analyst with a complete history of the issue. Analysts are required to review this Best Practices knowledge base when developing test cases and prior to executing regression tests.
