Autodesk, Inc. (Desktop Test Automation Team)

Number of people on the test team: 11

Number of developers supported by the test team: 35

Best Practice Description :

Test Automation History and Background

The Test Automation (TA) team for this division was created six years ago and expanded over the last three years. It focuses on providing automation testing for two core products of Autodesk: AutoCAD Architecture and AutoCAD MEP. Both of these products are 3-dimensional modeling software with a complicated graphical user interface (GUI) and a large amount of features. They are dedicated to meeting design requirements of large-scale architecture and construction projects.

Test automation is an integral part of application testing and adheres to the product software development cycle, playing an active role to in helping to improve product quality. Each product release is partitioned into several stages with corresponding priorities. Each of these stages goes through a Design, Coding and Testing section which TA closely follows.

All automation script development is completed two months before product release to allow for System Test. In this final stage, the automation team will build up multiple testing environments on various operating systems and different load conditions to run tens of thousands of test cases.

Suggested Best Practices for Test Automation and Development:

The Best Practices that follow cover the working processes, tools and automation strategies that we have followed in testing and releasing a quality product. We have found them to be effective at making our scripts more reliable and valuable for catching regressions on a consistent basis.

I. Design:

Careful design of the work flow for development of automated test cases improves efficiency, reduces redundancy and avoids duplicating work while making the tasks reasonable and achievable.

a). Follow a Test Plan. A test plan will have specific steps on what the script should be validating and how the result should be obtained.
b). Scope the work. Meet with QA to determine which test cases should be automated. This is very important in determining the number of scripts to be developed as they can vary significantly in scale and complexity. The QA analyst is also most familiar with the feature and will be able to best determine what should be automated.
c). Feature learning stage. Complicated products require testers to have a related domain background; however TA engineers are often focused solely on computer science. To ensure the quality of the TA work, before writing scripts, assigned features are learned in depth through hands on testing, help tutorials and communication with senior QA analysts, Development engineers, etc.
d). Feature demo. After feature learning, the TA team has a visual demonstration of the functionality of the feature.

Benefits:

Providing detailed knowledge of the features to the TA engineers helped to improve the quality of the scripts. It also lessoned the burden on QA as the constant demand for feedback on expected behavior is lowered through the cycle.

II. Coding:

Writing the scripts should have the same push for software quality as the application being developed.

a). Keep development cycles small. Our TA team uses cycles consisting of two weeks and a maximum of 50 scripts per week. This allows the team to find design and implementation problems at a much earlier stage.
b). Set up guidelines and document them. White papers are good for new hires joining the team and for reference when confusing issues arise.
c). Define TA roles and responsibilities at the beginning of each cycle. These responsibilities are cycled through each phase of development so each team member can experience the role of Designer and Reviewer.

• The Designer is responsible for script development and maintenance.
• The Reviewer is to review the script and fill in a track list with his/her comments.
• The Project Coordinator is responsible for tracking project schedule and to check process execution status.

d). Review before submittal. A batch of scripts is developed, reviewed, and run successfully before final submission to the script database. Each script has both a Peer Review(by another TA engineer) and a Code review (by a senior engineer)

Benefits:

Reviewing scripts means poor coding habits are minimized, information is shared and maintenance overhead is reduced. Having team members take on multiple roles helps to develop their abilities in varied aspects.

III. Testing:

Both new feature development and defect fixing may affect existing features and result in regressions. As we generate nightly builds, a good process for running and analyzing automation tests is of vital importance.

a). Varying Regression Testing Frequency.

• A Sanity Test, run hourly: This test suite runs just a few scripts and failures are analyzed immediately to confirm the build is not a lemon.
• Nightly regression testing: A regression farm, consisting of a coordinator and close to thirty machines, is used to install the nightly build and run up to five thousand UI and API tests on all aspects of the application. TA engineers analyze the results and send a report out within 12 hours. Sreen captures of test case failures are used to get a snapshot of the application condition when the failure occurred. Data Sets generated by automation are saved on a local server and used for testing. Logging files with debug statements are also saved and used for analysis with each run.
• Weekly regression test: Some scripts are very repetitive and take too many hours to run nightly these are run on a weekly basis and include performance, load testing and visual acuity tests.

b). Track automation statistics data: By tracking data the following important information can be obtained.

• Test Case automation coverage
• Code coverage
• Ratio of automation development time against manual execution time
• Defect finding rate through automation

Benefit:

This data helps us measure our effectiveness and results as well as what needs further improvement. By differentiating the running frequency, we ensure that each script is “active” and is kept up-to-date. This also allows the team to adjust the types of tests, taking some out and adding others based on new feature development, feature stability and team workload. Through implementing an automatic installation and running process, execution and analysis is accelerated.

IV. Integrating Automation into new feature development

New feature development is an area that automation often has problems with due to the constant changes with the code. For our TA team we have integrated it successfully into the development process using the following methods.

a). Select a Pilot Project to determine major hurdles.
b). Only develop scripts for test cases Development or QA marked “ready to test”
c). Scripts should emphasize simplicity and easy validation. This makes them easy to run and maintain during this fluid stage.

Benefit:

By integrating test automation into new feature development, regression problems are found and fixed much earlier than before. When test automation reports regressions within 24 hours of a change, developers get time to review and fix them while code changes are still fresh in their minds. Meanwhile, QA analysts can spend more time in complex projects unsuitable for automation.

V. System Testing

Automated System Testing is the final stage in our product development cycle and lasts two months before the products release. We run various install scenarios on different operating systems. To ensure finishing large-scale testing on time, we implement the following strategies:

a). Run archived legacy scripts prior to system test. Any maintenance issues can then be dealt immediately rather than doing it while analyzing results.
b). Configure environments in advance. Make sure all test environments are set up with images ready to install.
c). Try to eliminate cascading effects of scripts. Scripts are categorized into suites based on the feature they're testing. At times it's inevitable that scripts affect each other. To avoid this, add a start up test that sets the environment up for testing and a cleanup test to reset for the next tests. Adjusting the running order of scripts in atest suite can also help, putting the susceptible suites in front of other stable ones, etc.
d). Account for different OS environments. Applications may have various UI or feature behavior on different Operating Systems. Scripts should be able to handle all expected situations.
e). Setup a testing matrix, track status and define clear responsibility. A spreadsheet can be used to track testing configurations and results in a matrix. If the matrix is too large, an All-pair methodology is used to remove less-risky configurations, to keep the workload reasonable. Color coding is used to track status for each team member running each configuration. A System Testing Coordinator is assigned to be responsible for setting up a testing schedule, tracking execution status and communicating with external resources.

Benefits:

Good preparation helps us improving testing efficiency; avoiding duplicated environment configuration work after system testing begins.

VI. Conclusion

Script development, running, analysis and defect reporting are integral parts of test automation. Successful test automation depends not only on tools but also on a standard testing process and the right test team roles, duties and skills. Hopefully our practices can benefit other organizations which are planning to involve or implement test automation to improve testing efficiency.