qtp tool

QTP Tool

What is QTP Tool?
It is a Functional & Regression test tool, but it supports little bit Performance Testing, Compatibility Testing and Reliability Testing.

By inserting Transaction points (Start and End) we can measure Test Transaction time as well as part of the test transaction time, but only for one user, not like LoadRunner. LoadRunner support multiple concurrent users.

By executing same tests on different versions Operating systems and different versions of Browsers we can perform Compatibility Testing.

By repeating same operations with multiple sets of Test data (Data Driven Testing) we can cover Reliability Testing also.


QTP Product Information

QTP is I-tier Application (Stand-alone Application), it doesn’t have database, it stores its resources (Ex: Object Repositories, Function libraries, Recovery scenarios etc…) in File format on Computer Hard disk, otherwise we can store qtp resources in Quality Center Project.

QTP developed in .NET technology (VC++.NET), if we install QTP for first time in our machine, first it installs .NET framework, .NET framework is run-time for QTP Application.

QTP Version History
QTP Derived from Astra QuickTest and influenced by WinRunner. In 2001 Mercury Interactive Corporation take over Astra QuickTest.

Astra QuickTest Versions (1.0  to 5.0)

QTP Stable releases:

5.6 in 2002
6.5 in 2003
8.0 in 2004
8.2, 8.3 in 2005
9.0, 9.1 in 2006
9.2 in 2007
9.5 in 2008
10.00 in 2009
11.00 in 2010

In 2007 September QTP take over by HP
9.5 version onwards HP logo

QTP Supporting Environments

a) Application Environment

We have 3 types of Software Environments in the IT (Information Technology) Industry, they are:
1) CUI (Command user Interface) based Applications
2) GUI (Graphical User Interface) based Applications
3) Web based Applications

QTP Supports GUI and web Application Test Automation only, it doesn’t support CUI based applications, QTP is an object based test tool, in CUI environment no objects available.
Note: 95% Software Applications are either GUI based or Web based Applications only.

b) Operating System Environment
We have 2 major families of Operating systems available in the IT Industry

1) Microsoft Windows
2) UNIX and UNIX flavors like Linux, Solaris etc…
QTP Supports Windows Operating environment only, It doesn’t support  UNIX operating environment, but LoadRunner and Quality Center support Windows and UNIX environments.

QTP License

It has 2 types of license
a) Seat or Node Locked
b) Concurrent or Float 

QTP Add ins

Add in means additional component, by default QTP doesn’t support all types of Software Applications, In order to recognize Software Objects Add in, nothing but Technology compatibility files are required.

QTP is an Object based Test Tool, based on Front end Objects only it performs Test Operations, to recognize Front end objects Add ins are required.

We have two types of Add-ins Add ins available in QTP
---------
a) Internal Add-ins
---------------------
1) Standard Windows
2) Activex
3) Visual Basic
4) Web
Note: Standard Windows Add in is built in Add in, it is not visible Add in Manager.

b) External add-ins
-------------------
5) Java
6) SAP for GUI
7) SAP for Web
8) .NET for windows forms
9) .NET for Web forms
10) Delphi
11) Power Builder
12) Siebel
13) People soft
14) Oracle
15) Small talk
16) Web services
17) TE (Terminal Emulator, It is for  Main Frame Applications)
18) Stingray
19) WPF (Windows Presentation Foundation)
20) Silver Light (recently added, available in QTP 11.00 version only)

QTP IDE (Integrated Development Environment)

QTP has 3 types of features support for Test Automation:

a) Direct features (Features available in QTP tool main window)
Example: Recording, Object Repository, Object Spy, Checkpoints, output values, Step Generator, Data Table etc…

b) Integrated tools from HP
Test Bach Runner, Password encoder, Test Results deletion tool etc...

c) Integrated tools from others
SQL engine for database operations
VBScript engine for Scripting
MS Script Debugger for Debugging Tests (This we need to download and Install)

QTP Test Process

6 stages of Test Process available in QTP
a) Test Planning
b) Generating Basic Tests (Plain Navigation)
c) Enhancing tests 9inserting verification points, error handling, adding comments, parametrization etc… based on our test requirements.
d) Running & Debugging Tests
e) Analyzing Test Results
f) Reporting Defects

Creating Tests
In high level orientation, we have two types of test design methods available in QTP, in low level orientation 4 methods available.

a) Object Repository based Test Design
    1) Recording
    2) Manual Test Generation (Using Shared object repositories)

b) Descriptive Programming / Programmatic Descriptions
    1) Static Programming
    2) Dynamic Programming

Note: QTP supports Hybrid approach for creating tests, hybrid means combination of 2 or more approaches. In our project we can use Object Repository for some tests design and Descriptive program for some tests design.

Test Execution

Running individual tests

Batch Testing
    Using "Test Batch Runner" Tool
        Or
    Using AOM Script
        Or
    Using Quality Center Tool
        Or
    Using Driver script

QTP Framework

Framework, it is a systematic approach for automating Software Test process.
For More Visit: QTP Framework

QTP Certification

HP is conducting Certification programs for its Suite of Test tools.
For more details Visit:

QTP Certification

Quality Center Certification

LoadRunner Certification

-------------------------------------------***--------------------------------

waterfall model

 waterfall model

Introduction

This is most common and classic of life cycle models, also referred to as a linear sequential life cycle model. It is very simple to understand and use. In a Waterfall model each phase must be completed in its entirety before the next phase can begin. At end of each phase, a review takes place to determine if the project is on the right path or not?

In the waterfall approach, the whole process of software development is divided into separate phases. These phases in waterfall model are:

a.    Requirement specifications phase
b.    Software design
c.    Implementation
d.    Testing and maintenance

Note: Different variants of Waterfall Model available in the Industry.

Stages of the Waterfall Model

Requirement Analysis and Definition
All possible requirements of the system to be developed are captured in this phase. Requirements are a set of functions and constraints that the end user (who will be using the system) expects from the system. The requirements are gathered from the end user at the start of the software development phase. These requirements are analyzed for their validity, and the possibility of incorporating the requirements in the system to be developed is also studied. Finally, a requirement specification document is created which serves the purpose of guideline for the next phase of the model.

System and Software Design
Before starting the actual coding phase, it is highly important to understand the requirements of the end user and also have an idea of how should the end product looks like. The requirement specifications from the first phase are studied in this phase and a system design is prepared. System design helps in specifying hardware and system requirements and also helps in defining the overall system architecture. The system design specifications serve as an input for the next phase of the model.

Implementation and Unit Testing
On receiving system design documents, the work is divided in modules/units and actual coding is started. The system is first developed in small programs called units, which are integrated in the next phase. Each unit is developed and tested for its functionality; this is referred to as unit testing. Unit testing mainly verifies if the modules/units meet their specifications.

Integration and System Testing
As specified above, the system is first divided into units which are developed and tested for their functions. These units are integrated into a complete system during integration phase and tested to check if all modules/units coordinate with each other, and the system as a whole behaves as per the specifications. After successfully testing the software, it is delivered to the customer.

Operations & Maintenance
This phase of the waterfall model is virtually a never-ending phase. Generally, problems with the system developed (which are not found during the development life cycle) come up after its practical use starts, so the issues related to the system are solved after deployment of the system. Not all the problems come into picture directly but they arise from time to time and need to be solved; hence this process is referred to as maintenance.

Advantages of the Waterfall Model

o    Simple and easy to use
o    Easy to maintain due to rigidity of the model- each phase has specific deliverables and a review process
o    Phases are processed and completed one at a time.
o    Works well for similar projects where requirements are very well understood.
o    It is an Inexpensive model.

Disadvantages of the Waterfall Model

o    No working software is produced until late during the life cycle.
o    High amount of risk and uncertainty
o    Poor model for Complex and object-oriented projects.
o    Poor model for long and ongoing projects
o    Poor model, where requirements are at a moderate to high risk of changing.
o    Due to single stage of Testing No guaranty for Quality product.

Although there are a few disadvantages of the waterfall model, it continues to be one of the widely-used software development approach in the industry. This can be attributed to the fact, that the advantages of this model outweigh its disadvantages in certain kind of projects. And lastly, if the team does not have a lot of experience, this model perfectly serves the purpose.

Waterfall Model Vs. V Model

The main difference between waterfall model and V model is that in waterfall model, the testing activities are carried out after the development activities are over. On the other hand in V model, testing activities start with the first stage itself. In other words, waterfall model is a continuous process, while the V model is a simultaneous process. As compared to a software made using waterfall model, the number of defects in the software made using V model are less. This is due to the fact, that there are testing activities, which are carried out simultaneously in V model. Therefore, waterfall model is used, when the requirements of the user are fixed. If the requirements of the user are uncertain and keep changing, then V model is the better alternative. Also making changes in the software in waterfall model is a difficult task, and also proves to be a costly affair. The vice versa is true of the V model. At this stage, I would like to bring it to your notice, that any defects in the software cannot be determined, till the software reaches the testing phase. However, defects are noticed in the initial phases, due to which they can be corrected easily.

From the above discussion on waterfall model vs V model, it is clear that each of the models can be used depending on the system they are being developed for. Therefore, one can choose either waterfall model or V model taking the software and the user requirements into consideration. Often, for smaller systems, it is recommended that one use the waterfall model and for the bigger systems use the V model.

QTP Test Planning

Test Planning in Test Automation using QTP

Tasks in Planning Stage:


a) Get Technology/Environment (UI design technology and Database) Information from development team.

Note: QTP is an object based test tool, based on front end objects it performs test operations, In order to recognize Software objects, we need to load required Add-ins, In order to select Add-ins, we need to get Technology Info /Environment details from Development Team.

b) Select and load appropriate Add ins
Example:

Application Name: RocketLawyer (www.rocketlawyer.com)

Business Type: Legal Documentation web portal

Application Type: Internet, Public

Operating Environment:
---------------------
    User-side : Windows XP/Windows 7
    Server-side: Windows 2008 Server for Web server
             Windows 2008 Server for EDP server
             UNIX Server for Database server

Technical Environment:
---------------------
HTML, Activex for designing User Interface

HTML with Java script for Client-side validations

IIS as Web server

COM+ as Application Server

ASP.NET for Server side Container

VBScript for Server side Scripting

ADO.NET for connectivity in between Database server and Web server

VB.NET , C#.NET for Application Logic (Components developement)

SQL Server as Database Server

XML for Data Transportation
Note: Here I provided complete technology information for one example Software Application, but QTP Test Automation point of view, User Interface designed technology and Database information are sufficient.

----------------------------------
c) Analyzing the AUT in terms of Object identification
 
 How to analyze:
   By record and play back some end to end scenarios
   Using object spy
   Mapping objects in between object repository and AUT
  
Solutions:
 i) Tool level solutions
     object identification configuration
     low level recording
     virtual object configuration

  ii) Organization level solution
     Ask developement team to provide proper properties info for unrecognized objects

 iii) Vendor level solution
     rise ticket to vendor

d) Selecting areas or test cases for automation
 1. Tests that we have to execute on every build(sanity tests)
 
  Examples in Flight reservation Application:
   a) Login operation
   b) Insert Order
   c) Open Order

 2. Tests that we have to execute on every modified build(Regression Tests)

 3. Tests that we have to execute with multiple sets of test data(Data Driven tests)
     Examples in Flight reservation application:
       a) Login operation
       b) Insert Order
       c) Open Order
 
e) Tool settings configuration and globalize
 a) Files menu-> settings
 b) Tools menu-> options
 c) Tools menu-> object identification
Note: In any Software we can find two types of features One is Constant or Fixed features, another is Configurable features. We can use constant features as it is, but we need to configure Configurable features according to our requirements.

In QTP Application / Tool also we have these two types of features;

Example for Constant features:

-------------------------------

Recording

Object Repository

Data Table

Step Generator

Etc...

Example for Configurable features:

---------------------------------

Object Identification

Virtual Object

Test Settings

Etc...

QTP tool is One Tier / Stand- alone Application, in order use common settings from all machines, we need to globalize our tool setting configuration  after configuring required settings. To do this QTP has provided a seperate feature called "Generate Script".
f) Automation Framework Implementation (optional)

Software Test Metrics

Software Test Metrics

I) Introduction

When we can measure what we are speaking about and express it in numbers, we know something about it; but when we cannot measure, when we cannot express it in numbers, our knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but we have scarcely, in your thoughts, advanced to the stage of science.

Why we need Metrics?

“We cannot improve what we cannot measure.”
“We cannot control what we cannot measure”

AND TEST METRICS HELPS IN

o    Take decision for next phase of activities
o    Evidence of the claim or prediction
o    Understand the type of improvement required    
o    Take decision on process or technology change

II) Type of metrics

Base Metrics (Direct Measure)

Base metrics constitute the raw data gathered by a Test Analyst throughout the testing effort.  These metrics are used to provide project status reports to the Test Lead and Project Manager; they also feed into the formulas used to derive Calculated Metrics.

Ex: # of Test Cases, # of Test Cases Executed

Calculated Metrics (Indirect Measure)

Calculated Metrics convert the Base Metrics data into more useful information.  These types of metrics are generally the responsibility of the Test Lead and can be tracked at many different levels (by module, tester, or project).

Ex: % Complete, % Test Coverage

Base Metrics & Test Phases

o    # of Test Cases (Test Development Phase)   
o    # of Test Cases Executed (Test Execution Phase)   
o    # of Test Cases Passed (Test Execution Phase)        
o    # of Test Cases Failed (Test Execution Phase)        
o    # of Test Cases Under Investigation (Test Development Phase)    
o    # of Test Cases Blocked (Test dev/execution Phase)   
o    # of Test Cases Re-executed (Regression Phase)    
o    # of First Run Failures (Test Execution Phase)
o    Total Executions (Test Reporting Phase)
o    Total Passes (Test Reporting Phase)
o    Total Failures (Test Reporting Phase)
o    Test Case Execution Time ((Test Reporting Phase)
o    Test Execution Time (Test Reporting Phase

Calculated Metrics & Phases

The below metrics are created at Test Reporting Phase or Post test Analysis phase
o    % Complete   
o    % Defects Corrected
o    % Test Coverage   
o    % Rework
o    % Test Cases Passed   
o    % Test Effectiveness
o    % Test Cases Blocked   
o    % Test Efficiency
o    1st Run Fail Rate   
o    Defect Discovery Rate
o    Overall Fail Rate   

III) Crucial Web Based Testing Metrics

Test Plan coverage on Functionality

Total number of requirement v/s number of requirements covered through test scripts.
•    (No of requirements covered / total number of requirements) * 100
Define requirements at the time of Effort estimation

Example: Total number of requirements estimated are 46, total number of requirements tested 39; blocked 7…define what is the coverage?

Note: Define requirement clearly at project level

Test Case defect density

Total number of errors found in test scripts v/s developed and executed.
•    (Defective Test Scripts  /Total Test Scripts) * 100

Example: Total test script developed 1360, total test script executed 1280, total test script passed 1065, total test script failed 215
So, test case defect density is
               215 X 100
        ---------------------------- = 16.8%
            1280

This 16.8% value can also be called as test case efficiency %, which is depends upon total number of test cases which uncovered defects

Defect Slippage Ratio

Number of defects slipped (reported from production) v/s number of defects reported during execution.
•    Number of Defects Slipped / (Number of Defects Raised - Number of Defects Withdrawn)

Example: Customer filed defects are 21, total defect found while testing are 267, total number of invalid defects are 17

So, Slippage Ratio is
        [21/ (267-17)] X 100 = 8.4%

Requirement Volatility

Number of requirements agreed v/s number of requirements changed.
•    (Number of Requirements Added + Deleted + Modified) *100 / Number of Original Requirements
•    Ensure that the requirements are normalized or defined properly while estimating
Example: VSS 1.3 release had total 67 requirements initially, later they added another 7 new requirements and removed 3 from initial requirements and modified 11 requirements

So, requirement Volatility is
        (7 + 3 + 11) * 100/67 = 31.34%
Means almost 1/3 of the requirement changed after initial identification

Review Efficiency

The Review Efficiency is a metric that offers insight on the review quality and testing
Some organization also use this term as “Static Testing” efficiency and they are aiming to get min of 30% defects in static testing

Review efficiency=100*Total number of defects found by reviews/Total number of project defects

Example: A project found total 269 defects in different reviews, which were fixed and test team got 476 defects which were reported and valid

So, Review efficiency is [269/(269+476)] X 100 = 36.1%

Efficiency and Effectiveness of Processes

•    Effectiveness: Doing the right thing. It deals with meeting the desirable attributes that are expected by the customer.

•    Efficiency: Doing the thing right. It concerns the resources used for the service to be rendered

Metrics for Software Testing

•    Defect Removal Effectiveness

DRE= Defects removed during development phase   x100%
                   Defects latent in the product

Defects latent in the product = Defects removed during development
                                                 Phase+ defects found later by user


•    Efficiency of Testing Process (define size in KLoC or FP, Req.)

Testing Efficiency=  Size of Software Tested
                Resources used
----------------------------------------------------
Terms:

What is metric?
A measurement scale and the method used for measurement

What is measurement?
The process of assigning a number or category to an entity to describe an attribute of that entity.

What is measurement scale?
A scale that constrains the type of data analysis that can be performed on it.

What is measure?
The number or category assigned to an attribute of an entity by making a measurement.

Static Testing

Static Testing

Important Terms:

Static techniques and the test process

dynamic testing, static testing, static technique

Review process

entry criteria, formal review, informal review, inspection, metric, moderator/inspection leader, peer review, reviewer, scribe, technical review, walkthrough.

Static analysis by tools

Compiler, complexity, control flow, data flow, static analysis

I) Phases of a formal review

1) Planning	Selecting the personal, allocating roles, defining entry and exit criteria for more formal reviews etc.
2) Kick-off	Distributing documents, explaining the objectives, checking entry criteria etc.
3) Individual preparation	Work done by each of the participants on their own work before the review meeting, questions and comments etc.
4) Review meeting	Discussion or logging, make recommendations for handling the defects, or make decisions about the defects etc.
5) Rework	Fixing defects found, typically done by the author Fixing defects found, typically done by the author
6) Follow-up	Checking the defects have been addressed, gathering metrics and checking on exit criteria

II) Roles and responsibilities

Manager	Decides on execution of reviews, allocates time in projects schedules, and determines if the review objectives have been met
Moderator	Leads the review, including planning, running the meeting, follow-up after the meeting.
Author	The writer or person with chief responsibility of the document(s) to be reviewed.
Reviewers	Individuals with a specific technical or business background. Identify defects and describe findings.
Scribe (recorder)	Documents all the issues, problems

III) Types of review

Informal review	No formal process, pair programming or a technical lead reviewing designs and code. Main purpose: inexpensive way to get some benefit.
Walkthrough	Meeting led by the author, ‘scenarios, dry runs, peer group’, open-ended sessions. Main purpose: learning, gaining understanding, defect finding
Technical review	Documented, defined defect detection process, ideally led by trained moderator, may be performed as a peer review, pre meeting preparation, involved by peers and technical experts Main purpose: discuss, make decisions, find defects, solve technical problems and check conformance to specifications and standards
Inspection	Led by trained moderator (not the author), usually peer examination, defined roles, includes metrics, formal process, pre-meeting preparation, formal follow-up process Main purpose: find defects.

Note: walkthroughs, technical reviews and inspections can be performed within a peer group-colleague at the same organization level. This type of review is called a “peer review”.

IV) Success factors for reviews

• Each review has a clear predefined objective.
• The right people for the review objectives are involved.
• Defects found are welcomed, and expressed objectively.
• People issues and psychological aspects are dealt with (e.g. making it a positive experience for the author).
• Review techniques are applied that are suitable to the type and level of software work products and reviewers.
• Checklists or roles are used if appropriate to increase effectiveness of defect identification.
• Training is given in review techniques, especially the more formal techniques, such as inspection.
• Management supports a good review process (e.g. by incorporating adequate time for review activities in project schedules).
• There is an emphasis on learning and process improvement.

V) Cyclomatic Complexity

The number of independent paths through a program

Cyclomatic Complexity is defined as: L – N + 2P

L = the number of edges/links in a graph
N = the number of nodes in a graphs
P = the number of disconnected parts of the graph (connected components)

Alternatively one may calculate Cyclomatic Complexity using decision point rule
Decision points +1

Cyclomatic Complexity and Risk Evaluation
1 to 10a simple program, without very much risk
11 to 20 a complex program, moderate risk
21 to 50, a more complex program, high risk
> 50an un-testable program (very high risk)

Fundamentals of Testing

Fundamentals of Testing

Important Terms:

Why testing necessary?

bug, defect, error, failure, mistake, quality, risk, software, testing and exhaustive testing.

What is testing?

code, debugging, requirement, test basis, test case, test objective

Testing principles

Fundamental test process

conformation testing, exit criteria, incident, regression testing, test condition, test coverage, test data, test execution, test log, test plan, test strategy, test summary report and testware.

The psychology of testing

independence.

I) General testing principles

Principles

A number of testing principles have been suggested over the past 40 years and offer general guidelines common for all testing.

Principle 1 – Testing shows presence of defects

Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness.

Principle 2 – Exhaustive testing is impossible

Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, risk analysis and priorities should be used to focus testing efforts.

Principle 3 – Early testing

Testing activities should start as early as possible in the software or system development life cycle, and should be focused on defined objectives.

Principle 4 – Defect clustering

A small number of modules contain most of the defects discovered during pre-release testing, or are responsible for the most operational failures.

Principle 5 – Pesticide paradox

If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new defects. To overcome this “pesticide paradox”, the test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise different parts of the software or system to potentially find more defects.

Principle 6 – Testing is context dependent

Testing is done differently in different contexts. For example, safety-critical software is tested differently from an e-commerce site.

Principle 7 – Absence-of-errors fallacy

Finding and fixing defects does not help if the system built is unusable and does not fulfill the users’ needs and expectations.

II) Fundamental test process

1) Test planning and control

Test planning is the activity of verifying the mission of testing, defining the objectives of testing and the specification of test activities in order to meet the objectives and mission.

  It involves taking actions necessary to meet the mission and objectives of the project. In order to control testing, it should be monitored throughout the project. Test planning takes into account the feedback from monitoring and control activities.

2) Test analysis and design

Test analysis and design is the activity where general testing objectives are transformed into tangible test conditions and test cases.

Test analysis and design has the following major tasks:

Ø       Reviewing the test basis (such as requirements, architecture, design, interfaces).

Ø       Evaluating testability of the test basis and test objects.

Ø       Identifying and prioritizing test conditions based on analysis of test items, the specification, behaviour and structure.

Ø       Designing and prioritizing test cases.

Ø       Identifying necessary test data to support the test conditions and test cases.

Ø       Designing the test environment set-up and identifying any required infrastructure and tools.

3) Test implementation and execution

Ø       Developing, implementing and prioritizing test cases.

Ø       Developing and prioritizing test procedures, creating test data and, optionally, preparing test harnesses and writing automated test scripts.

Ø       Creating test suites from the test procedures for efficient test execution.

Ø       Verifying that the test environment has been set up correctly.

Ø       Executing test procedures either manually or by using test execution tools, according to the planned sequence.

Ø       Logging the outcome of test execution and recording the identities and versions of the software under test, test tools and testware.

Ø       Comparing actual results with expected results.

Ø       Reporting discrepancies as incidents and analyzing them in order to establish their cause (e.g. a defect in the code, in specified test data, in the test document, or a mistake in the way the test was executed).

Ø       Repeating test activities as a result of action taken for each discrepancy. For example, reexecution of a test that previously failed in order to confirm a fix (confirmation testing), execution of a corrected test and/or execution of tests in order to ensure that defects have not been introduced in unchanged areas of the software or that defect fixing did not uncover other defects (regression testing).

4) Evaluating exit criteria and reporting

Ø       Checking test logs against the exit criteria specified in test planning.

Ø       Assessing if more tests are needed or if the exit criteria specified should be changed.

Ø       Writing a test summary report for stakeholders.

5) Test closure activities

Ø       Checking which planned deliverables have been delivered, the closure of incident reports or raising of change records for any that remain open, and the documentation of the acceptance of the system.

Ø       Finalizing and archiving testware, the test environment and the test infrastructure for later reuse.

Ø       Handover of testware to the maintenance organization.

Ø       Analyzing lessons learned for future releases and projects, and the improvement of test maturity.

III) The psychology of testing

Ø       Tests designed by the person(s) who wrote the software under test (low level of independence).

Ø       Tests designed by another person(s) (e.g. from the development team).

Ø       Tests designed by a person(s) from a different organizational group (e.g. an independent test team) or test specialists (e.g. usability or performance test specialists).

Ø       Tests designed by a person(s) from a different organization or company (i.e. outsourcing or certification by an external body).

software life cycle

Testing throughout the software life cycle

Important Terms:

Software development models

COTS, interactive-incremental development model, validation, verification, V-model.

Test levels

Alfa testing, beta testing, component testing (also known as unit/module/program testing), driver, stub, field testing, functional requirement, non-functional requirement, integration, integration testing, robustness testing, system testing, test level, test-driven development, test environment, user acceptance testing.

Test types

Black box testing, code coverage, functional testing, interoperability testing, load testing, maintainability testing, performance testing, portability testing, reliability testing, security testing, specification based testing, stress testing, structural testing, usability testing, white box testing 

Maintenance testing

Impact analysis, maintenance testing.

i) Software development models

a) V-model (sequential development model)

Although variants of the V-model exist, a common type of V-model uses four test levels,

corresponding to the four development levels.

The four levels used in this syllabus are:

• component (unit) testing;
• integration testing;
• system testing;
• acceptance testing.

b) Iterative-incremental development models

Iterative-incremental development is the process of establishing requirements, designing, building and testing a system, done as a series of shorter development cycles. Examples are: prototyping, rapid application development (RAD), Rational Unified Process (RUP) and agile development models.

c) Testing within a life cycle model

In any life cycle model, there are several characteristics of good testing:

• For every development activity there is a corresponding testing activity.
• Each test level has test objectives specific to that level.
• The analysis and design of tests for a given test level should begin during the corresponding development activity.
• Testers should be involved in reviewing documents as soon as drafts are available in the development life cycle.

ii) Test levels

a) Component testing

Component testing searches for defects in, and verifies the functioning of, software (e.g. modules, programs, objects, classes, etc.) that are separately testable.

Component testing may include testing of functionality and specific non-functional characteristics, such as resource-behaviour (e.g. memory leaks) or robustness testing, as well as structural testing (e.g. branch coverage).

One approach to component testing is to prepare and automate test cases before coding. This is called a test-first approach or test-driven development.

b) Integration testing

Integration testing tests interfaces between components, interactions with different parts of a system, such as the operating system, file system, hardware, or interfaces between systems.

Component integration testing tests the interactions between software components and is done after component testing;

System integration testing tests the interactions between different systems and may be done after system testing.

Testing of specific non-functional characteristics (e.g. performance) may be included in integration testing.

c) System testing

System testing is concerned with the behaviour of a whole system/product as defined by the scope of a development project or programme.

In system testing, the test environment should correspond to the final target or production environment as much as possible in order to minimize the risk of environment-specific failures not being found in testing.

System testing may include tests based on risks and/or on requirements specifications, business processes, use cases, or other high level descriptions of system behaviour, interactions with the operating system, and system resources.

System testing should investigate both functional and non-functional requirements of the system.

d) Acceptance testing

Acceptance testing is often the responsibility of the customers or users of a system; other stakeholders may be involved as well.

The goal in acceptance testing is to establish confidence in the system, parts of the system or specific non-functional characteristics of the system

Contract and regulation acceptance testing

Contract acceptance testing is performed against a contract’s acceptance criteria for producing custom-developed software. Acceptance criteria should be defined when the contract is agreed. Regulation acceptance testing is performed against any regulations that must be adhered to, such as governmental, legal or safety regulations.

Alpha and beta (or field) testing

Alpha testing is performed at the developing organization’s site. Beta testing, or field testing, is performed by people at their own locations. Both are performed by potential customers, not the developers of the product.

iii) Test types

a) Testing of function (functional testing)

The functions that a system, subsystem or component are to perform may be described in work products such as a requirements specification, use cases, or a functional specification, or they may be undocumented. The functions are “what” the system does.

A type of functional testing, security testing, investigates the functions (e.g. a firewall) relating to detection of threats, such as viruses, from malicious outsiders. Another type of functional testing, interoperability testing, evaluates the capability of the software product to interact with one or more specified components or systems.

b) Testing of non-functional software characteristics (non-functional testing)

Non-functional testing includes, but is not limited to, performance testing, load testing, stress testing, usability testing, maintainability testing, reliability testing and portability testing. It is the testing of “how” the system works.

Non-functional testing may be performed at all test levels.

c) Testing of software structure/architecture (structural testing)

Structural (white-box) testing may be performed at all test levels. Structural techniques are best used after specification-based techniques, in order to help measure the thoroughness of testing through assessment of coverage of a type of structure.

Structural testing approaches can also be applied at system, system integration or acceptance testing levels (e.g. to business models or menu structures).

d) Testing related to changes (confirmation testing (retesting) and regression testing)

After a defect is detected and fixed, the software should be retested to confirm that the original defect has been successfully removed. This is called confirmation. Debugging (defect fixing) is a development activity, not a testing activity.

Regression testing is the repeated testing of an already tested program, after modification, to discover any defects introduced or uncovered as a result of the change(s). It is

performed when the software, or its environment, is changed.

Regression testing may be performed at all test levels, and applies to functional, non-functional and structural testing.

iv) Maintenance testing

Once deployed, a software system is often in service for years or decades. During this time the system and its environment are often corrected, changed or extended.

Modifications include planned enhancement changes (e.g. release-based), corrective and

emergency changes, and changes of environment,

Maintenance testing for migration (e.g. from one platform to another) should include operational tests of the new environment, as well as of the changed software.

Maintenance testing for the retirement of a system may include the testing of data migration or archiving if long data-retention periods are required.

Maintenance testing may be done at any or all test levels and for any or all test types.