Pitney Bowes Software

Number of people on the test team: 4

Number of developers supported by the test team: 13

Best Practice Description:

In 2005, I was tasked as team lead for QA to find a QA training course or certification to set as the baseline for our test engineers. We as a team did extensive research and chose the CSTP certification because of its education-based format. The courses were challenging, informative, and all three team members came away with ideas on how to improve not only our QA processes, but our engineering processes as a whole. It is now 2007 and our team oversaw the most successful yearly release of our software on record. In 2006, we lost nearly a month and a half of new feature development due to management of critical customer defect resolution. To date, we have had only two customer-reported defects requiring a patch. Each took less than two days to turn around and both issues were deemed as beyond normal usage of our software. I truly believe having our team CSTP certified is a very large reason for this. Here are some of the improvements we have implemented:

Requirements:

Requirements in our organization have always been vague at best. We were down to 17 individuals (from 70+) during the 2002 timeframe and did not have time to implement a clear development process. Our requirements were communicated via high bandwidth verbal communication because it was all that we had time for. With data provided to us in multiple courses about finding defects earlier in the development process, our QA department was able to successfully lobby product management to provide better requirements earlier in the release cycle. Those requirements are now being iterated through by not only QA, but Development as well. This practice has lead to less confusion at the end of the release cycle because we are clear as a team what the requirements are and what the software is expected to do.

Process Improvement

Since taking the CSTP courses, our team focus has shifted from finding defects to delivering quality software. This shift has caused our QA engineers to see themselves as valued members of an engineering team rather than low importance employees who have little effect on the organization as a whole. Our team is constantly examining not only our QA processes, but our overall release and development cycles. The shift in focus has earned respect within our organization for QA because we are leaders in improvement and innovation of better practices.

Metrics

Being responsible as Team Lead and now QA manager of our team, I had difficulties communicating with upper management. I tended to communicate to them as I would to a QA person and they did not get what I was saying. This led to difficulties justifying everything from new equipment, new test tools, and new employees, to communicating the overall health of our software. After one of our team members took the Risk Management and Metrics CSTP course, we as a team were able to implement a set of meaningful metrics. This has allowed me to communicate with upper management by giving them real data that they in turn can use. It has also shown weaknesses in our test processes that we have since been able to implement plans to improve.

Test Planning and Execution

Before 2005, our test plans were not as effective as they should have been. The test plans were much more about covering our group in case the release went poorly rather than defining what we would be doing and using it as a step to plan our upcoming testing efforts. We now use our test plans to help score the level of effort for our group, drive better functional specifications and business requirements, and solidify our development processes. The test plan has become a tool for QA to drive the development cycle process improvement. When our group is showing and putting forth the effort to document our plan, development and product management have agreed to do the same.

Functional Specifications

It has been a three year battle with development to implement a process that included functional specifications across the board. Starting with the upcoming yearly release, our development managers and teams have committed to not only providing QA with function specifications, but also including us the creation of this document to get our feedback. This marks an important step in establishing trust and team work amongst our engineering departments.

Automation

Our QA department has always been highly automated out of sheer necessity. With only one QA engineer to every 6 developers, we were forced to automate or sink. However, since becoming a team of CSTP's, we have improved our automation test strategies. The biggest improvement has been implementing data driven testing. Our products are API's and in the past our test cases were contained within source code. This was very problematic to maintain as well as challenging when bringing on new engineers and getting them up to speed quickly. Now a large number of test cases (10,000+) are contained within database tables. This means we can quickly update a test case without pushing the source code to all 25 test platforms and rebuilding simply to fix one test failure. Also, this has allowed me as the manager to lower my employee cost since no longer are we required to hire a development-capable QA engineer. We can now have room more traditional QA engineers who are easier to hire and are less expensive.

In closing, not only has the CSTP allowed our QA department to successfully implement software testing best practices, but we have gone beyond that to help implement software release best practices. These have led to better planning throughout the organization, increased new feature development, and higher quality of those new features. All of these positively affect our customer experience, leading to less customer support calls and improved customer satisfaction.

Overall, the CSTP certification has had a large impact on our test and development organization. We have now made it a requirement for all new test engineers to work towards the CSTP certification. Justification for the expense has not been difficult and my manager eagerly supported my request that CSTP certification become a requirement for all QA engineers. This is due to the obvious cost savings our organization has realized from taking the CSTP courses and implementing the ideas given.