A test strategy describes the testing approach of the software development cycle. The purpose of a test strategy is to provide high-level objectives.
Software Test Strategy
4.0 Testing Approach
4.2 Functional and Usability Issues
5.0 Phases of Testing
5.1 Documentation Review
5.2 Unit, Module, and Integration Testing
5.3 System Testing
5.4 IBEE Pre – Production Testing
6.0 Test Management
6.1 Test Procedures
6.2 Test Report
6.3 Test Phase Acceptance
7.0 Risks, Issues, and Assumptions
7.4 Related Documents
ABCD (P) Limited has developed the Test Strategy for Web applications, particularly for eCommerce applications as per its Standards and got approved by IBEE Solutions (P) Ltd.
The objectives of this test strategy for the April 08 Release are as follows:
• to describe the overall testing for a framework to a Web Application ;
• to define the testing to be undertaken;
• to define the responsibilities of the various parties involved in testing;
• to ensure that all areas of testing requirements are addressed; and
• to ensure that the coverage of the testing is adequate to meet those acceptance criteria that are subject to testing.
The scope of this document is the testing of the changes to documentation, processes, hardware and software systems to deliver the business requirement solutions for the April 08 Release, to ensure that the software has been ported and functions correctly and that unchanged functionality is not impacted.
For the purposes of testing, the applications which are relevant are as follows:
4.0 Testing Approach
Web-based applications present new challenges both for developers and for testers. These challenges include:
Short release cycles;
Constantly changing technology;
A possible huge number of users during initial website launch;
Inability to control the user’s running environment;
24 hours availability of the website.
The quality of a website must be evident from the onset. Any difficulty whether in response time, accuracy of information, or ease of use – will compel the user to click to a competitor’s site. Such problems translate into lost users, lost sales, and poor company image. Testing priorities differ based on the type of website being tested. For content only sites, availability is the main concern, whereas interactive sites are concerned with the speed and reliability of interaction.
The environment in which a web application runs contains many components. The network administrator determines the actual configuration. The tester may need to understand a particular network setup in order to define some of the tests
1. User views the website through a browser connected to the internet.
2. The website software can execute on the user’s browser at host web server connected elsewhere on the internet, or on an application server. Where the software executes on how the developers designs the system.
3. A firewall, which is a combination of hardware and software exists to keep a network secure from intruders. The user can also install a firewall to protect his computer from external attacks.
4. A proxy server is software whose purpose is to be a sole connection between a private network and the internet. A proxy server performs many functions that include preventing certain files from entering or leaving the network as well as improving performance by catching data.
5. Many web applications use a database to store the data necessary to run the website.
By comparison a client server configuration consists of one or more user computers connected to a server via a local network or wide area network. Placement of the web application on the web causes the site to have the same issues. These issues includes
v Handling customer traffic;
v Customer value;
v Security of customer information;
v Customer service during the visit;
v Retention of the customer base.
Customer expectation of a site includes navigation that is easy to use and intuitive quick response times,24 hours availability of the site , reliability no crashes or long delays, and indication of progress when using the site. A customer who is quickly annoyed by these types’ issues while using the website may not return – resulting in lost revenue.
4.2 FUNCTIONAL AND USABILITY ISSUES
The first tests for a website should focus on the site’s intended behavior by assessing the following issues;
Ø Page content;
Executing these tests in an isolated and controlled test environment helps ensure that the core features of the website’s are correct; prior to accessing the site via the internet. After executing controlled environment tests, any new problems are most likely to be due to interactions from the live environment.
4.2.1 Functional Testing
Functional Testing making sure the features that most affect user Interactions work properly. These Include:
Contents and Images
4.2.2 Usability Testing
The key to usability testing is to study what a user actually does. An observer records the user’s action and reactions to determine what types of problems the user encounters and how he overcomes them.
The main steps to usability testing are to:
v Identify the website’s purpose
v Identify the intended users
v Define tests and conduct usability testing
v Analyze acquired information
4.2.3 Navigation Testing
Good navigation is an essential part of a web site, especially those that are complex and provide a lot of information. Assessing navigation is a major part of usability testing. Most users expect the following:
Easy and Quick access to the information they want;
Logical hierarchy of pages;
Conformation where they are at any point of time;
Facility to return to previous states or the home page;
Consistent look and layout of every page;
Key issues with navigation testing include:
Moving to and from pages;
Scrolling through pages;
Clicking on all images and their thumbnails to ensure they work;
Testing all links (both within and outside the web site) for validity and correctness
Ensuring no broken links exist
Viewing tables and forms to verify proper layout, this can vary under different browsers;
Verifying that windows with multiple frames are preceded as if each were a single page frame.
Measuring load time of every page;
Ensuring compatibility and consistent usage of buttons, keyboard shortcuts and mouse actions
4.2.4 Forms Testing
Websites that use forms need tests to ensure that each field works properly and that the form posts all data as intended by the designers. Testing of forms includes the following actions:
Using the tab key to verify that the form traverses fields in the proper order, both forwards and backwards;
Testing boundary values;
Checking that the form traps invalid data correctly, especially date and numeric formats;
Verify that the form updates information correctly.
4.2.5 Page Content Testing
Each web page must be tested for correct content from the user perspective. These tests fall into two categories: ensuring that each component functions correctly and ensuring that the content of each is correct. The first category of tests checks that:
All images graphics display correctly across various browsers;
All content is present per the requirements;
Page structures are consistent across browsers;
Critical pages maintain same content from version to version;
All parts of table or form are present and in the right place;
Links to relevant content inside and out of the site are accurate;
“Mouse over” text objects are correct;
Web pages visually appealing.
Configuration and Compatibility Testing
Reliability and Availability Testing
End- to- END transaction Testing
4.1.14 Database Testing
Database Testing is often an essential part of web testing, since many web sites typically provide some sort of search capability.
Key issues for database Testing include:
v Data Integrity;
v Data validity (input into database in proper form);
v Data manipulation and updates
The following checklist to formulate many administrative tests:
§ Understanding the database design
§ Understanding the security controls of the design; understand which user ids have read/write access to the database controls
§ Understanding the procedures of the database maintenance, updates and upgrades.
§ Ensure the backup and recovery procedures work as designed and don’t impact availability requirements.
§ Ensure that the database allows the maximum number of connections that the system is designed to handle.
§ Ensure that database operations have enough space and memory for the amount of data.
5. Phases of Testing
5.1 Documentation Review
All documentation identified as being impacted will be subject to a Product Review, which will be carried out in accordance with the Quality Manual.
5.2 Unit, Module/Component and Integration Testing
Unit, Module/Component and Integration testing will occur on the ecommerce B2C and B2B applications before the porting takes place. Testing will be conducted by IBEE Solutions as a part of the development phase for the defect fixes. No Unit, Module and Integration Testing will be carried out after the porting process, unless subsequent defects are identified which require fixing. Details of this testing can be found in IBEE Quality Plan (Reference 4).
5.3 System Testing
System Testing will be carried out at application level to ensure that the software functions consistently working or not. Test execution will take place by ABCD (P) LIMITED Testers. However, IBEE will be responsible for supporting and advising the ABCD (P) LIMITED Team throughout the testing process.
System Testing will consist of a set of tests to be undertaken by ABCD (P) LIMITED before providing IBEE with evidence that sufficient testing was successfully completed.
System Testing will consist of the following types of testing:
Benchmark Performance Testing; and
Performance testing should take place before Regression tests, but it is not strictly necessary for the tests to occur in that order.
Benchmark Performance Testing
Benchmarking will take place in order to ensure that there is an acceptable level of performance.
Formal Regression Testing
Formal Regression testing should be executed after all Performance testing is complete.
All Regression test scripts will be run on the ported software versions.
5.4 IBEE Pre – Production Testing
• The high risk business processes run successfully with production data;
• The unchanged functionality and interfaces are not adversely affected by the changes;
• The changes do not impact operational requirements;
• Any degradation in performance is commensurate with the functional change applied to the target system.
5.5.1 Responsibilities of ABCD (P) Limited
ABCD (P) Limited will develop a Release Test Strategy describing their overall test strategy for the Feb 08 Release. This will also contain further details on all phases of testing. IBEE Solutions will approve the Test Strategy, subject to a satisfactory review.
ABCD (P) LIMITED will define the scope and details of the tests in their Test Specification(s) and Test Materials (which include scripts, data and expected results). These documents will be reviewed by IBEE Change implementation Team (CIT) and Service Delivery in accordance with the Quality Manual (Reference 3).
ABCD (P) LIMITED will carry out System Tests for the applications ported to Windows Environment, and will be responsible for preparing, executing and managing these tests
ABCD (P) LIMITED will submit details of defects raised during System Testing phases and execution times from tuning and optimization activities to IBEE.
ABCD (P) LIMITED will be responsible for ensuring that the level of
Testing, after any defects have been fixed, is sufficient to provide full assurance that any defect has been resolved in every part of the software that it may affect. Testing defect fixes will require Unit, Module and Integration testing and may also require Regression and Performance Testing. The type and level of testing required will depend on the defect.
ABCD (P) LIMITED will determine the best course of action and consult with IBEE regarding these defect testing strategies.
In the respect of any reported defects, IBEE will manage the fixing and retesting to be as efficient as possible, in order to cause minimum disruption to the testing phase overall.
On completion of testing, ABCD (P) LIMITED will produce a Test Report summarizing the testing carried out and the results of those tests. ABCD (P) LIMITED will perform all System Tests and confirm that no outstanding severity 1 or severity 2 issues exist. If any severity 3 or 4 defects exist, then ABCD (P) LIMITED and IBEE will have to agree a strategy to resolve those defects.
This report will describe any defects found during testing and the steps taken to resolve them. The report will also confirm that Unit, Module and Integration testing carried out prior to the start of System Testing has been completed successfully.
The Test report that ABCD (P) LIMITED will produce will include evidence of successful Regression, Benchmarking and Tuning.
5.5.2 Responsibilities of IBEE Solutions (P) Limited
IBEE may witness the tests in a test environment provided and managed by ABCD (P) LIMITED. IBEE Corporate Assurance will review ABCD (P) LIMITED test materials. Furthermore, It will monitor any defects that arise, contribute to decision making on the strategy for resolving those defects and may witness the testing of defect solutions that are produced. IBEE Corporate Assurance will also audit test results produced from this phase of testing and other development review records.
IBEE will be responsible for developing and performing Pre Production Testing (PPT), from a revised IBEE test pack. IBEE will produce a test specification to be tested using test facilities provided by ABCD (P) LIMITED. This phase will occur after ABCD (P) LIMITED have completed their testing and handed over the tested build to IBEE. The IBEE PPT will use production data for the purposes of testing.
Although ABCD (P) LIMITED will not be executing specific deployment testing, IBEE will carry out full deployment testing in the process of running the extended PPT phase.
6. Test Management
6.1 Test Procedures
Defects identified by IBEE during product reviews (software testing and documentation reviews) will be notified to the product developer in accordance with the procedures specified in the Quality Manual.
In general, software defects raised during one phase of testing should be cleared by the start of the next phase. However, with the agreement of IBEE, severity 3 or severity 4 defects (those of low significance and low business impact) may be left outstanding, rather than delay any crucial part of the testing phase.
At the end of the project, any defects that have not been resolved will be handed over to IBEE Change Delivery (CIT).
6.2 Test Report
A Test Report will be produced as a record of the testing performed, summarizing the test results from each type of test carried out. The report will describe any defects found during testing, their severity level and the steps taken to resolve them, and to specify any deviations from the relevant test strategies and test specifications.
The following types of testing will be covered in the Test Report:
Benchmarking Performance Tests and Tuning.
Test logs will record all of the test details, such as date and time of test, step faults and anomalies. It will also record the decisions taken, which affect the test in any way. All test documentation, including: annotated System Test scripts, test logs, test Observation Reports (which may be defects), test results and the Test Report will be available for inspection by IBEE.
6.3 Test Phase Acceptance
ABCD (P) LIMITED will ensure overall that the scope of the testing supports the PID and Plan document (Reference 8), and that the acceptance criteria that are subject to testing have been met.
The scope of the audit of testing carried out by Corporate Assurance will be comparable with the audits of previous releases.
The Project team will review the generic acceptance criteria specified in the Delivery Acceptance Criteria (Reference 9) and prepare a statement of compliance for the Feb 08 Release for approval by the IBEE Solutions Programme Board. Note that the final judgment as to whether the acceptance criteria have been met will be decided by the IBEE Solutions Programme Board.
The timescales and dates for the individual test phases and all other scheduled tasks or milestones will be defined in the February 2011 Release PID and Plan (Reference 8).
7. Risks, Issues and Assumptions
The following risks have been identified during the development of the Test Strategy. The risk below will be added to the project risk register and managed through that mechanism:
1. This will be the first Release that uses the new approach to software testing. There is a potential risk that defects in the software will not be uncovered during testing and result in the software being issued with outstanding defects.
Mitigation: The new test approaches are currently being developed and tested to ensure all areas of functionality are sufficiently tested. This in conjunction with the fact that there are no functional changes in the Release should ensure that all defects are captured during testing.
No issues have been identified during the development of the Test Strategy. Any issues identified subsequent to this will be added to the project issue register and managed through that mechanism.
1. It is assumed that the “February 2011 Agreement” will not require Unit, Module and Integration testing.
System and Regression testing will detect any outstanding defects related to the Shipped Product on the Intranet Environment.
7.4 Related Documents
Reference 1: Change Delivery Guidelines for Testing Changes to B2C Systems (003ATB)
Reference 2: IBEE Solutions Software Quality Plan.
Reference 3: Change Delivery Release Acceptance Criteria (002RMR)
Reference 4: ELEXON PPT Test Specification and Procedures for Web Applications.
Reference 5: February 2011 Release PID and Plan.