[Crazy Guy.]

What is compatibility testing?

Compatibility Testing ensures that a product works as expected in any specified operating environment – Hardware, Software, any combination of platform, database, application server and browser.

Compatibility Testing process provides technical integration, Network configuration compatibility, functionality and stability testing of complementary, third party products

Different Web servers/Application servers and Databases needs to be considered for compatibility testing

Why Compatibility Testing?

To ensure that web application or web site works properly across different range of browsers and operating systems.

To ensure that certain coding such as Java Applets, Active X Controls, JavaScript, CSS, and HTML function properly across multiple configurations.

To avoid increased tech support costs, lost business opportunities, and unhappy users by the early Investment in compatibility testing.

To identify and remove the potential difficulties prior to installation of the Software at a customer site


Different types of Compatibility Testing


The different types of Compatibility testing includes

Browser Compatibility Testing

OS Compatibility Testing

System Platform (Hardware) Compatibility testing

Version Compatibility Testing

Database Compatibility Testing

Installation Compatibility Testing

Data sharing Compatibility Testing

Cross-platform Compatibility testing

Wireless Platform Compatibility testing


Browser Compatibility Testing:

What is Browser Compatibility Testing?


Browser compatibility Testing ensures that the Application behaves as expected with the browsers specified.


WHY IT IS IMPORTANT?

Testing a Web site, or Web-delivered application for compatibility with a range of the leading browsers plays a vital role in finding Browser compatibility issues.

Application is said to be reliable only if it’s behaviour is the same way across different browsers.

The content of a Web application should be displayed in the same way irrespective of the browser used failing which will lead to major Production errors.


Browser Compatibility Testing:

What Causes Browser Display Differences?

Different Browsers
Different Browser Versions
Different Computer Types
Different Screen Sizes
Different Font Sizes
HTML Errors
Browser Bugs

[Crazy Guy.]

1. Different Computer Types
The Macintosh is still used by 12% of computer users, and has a very loyal following among graphic designers and publishers. In theory, if you view your page on both a PC and a Mac using the same version of the same browser, it should display the same, right?
 
In practice that's rarely the case. There are three reasons for this:
 
Font Availability. When you tell your Web page to use a particular typeface, such as "Arial," you may not always get the font you want. Fonts are a computer resource, and not all computers have the same fonts as your computer. That's true even between different PCs, but it's especially true between the PC and the Mac. If the typeface of your page suddenly changes between these computers, you've probably used a font that isn't available on both computer types. 
Font Size. The Mac will generally render your typeface in a smaller pixel size than the PC will. That's especially true if you use the FONT tag to set your type size, since this tag uses abstract units to define size. You can avoid this problem is you use Cascading Style Sheets to set your font size in pixels. ac.
Internet Explorer. Microsoft outsources the development of Internet Explorer for the Mac, and so to a large extent this is a different browser from the PC version. In particular, the Mac version of Internet Explorer is prone to quirks and bugs that you won't see in the PC version. If you check your Web page under only one browser on the Mac, do so under Internet Explorer!

2. Different Font Sizes:
Most browsers allow users to customize their default font size. Many users who work on computers all day do this to reduce eye strain. As a result, user preferences may cause the typeface that you used to design your Web page to increase as much as 50% larger in a user's browser. This increase in font size can hurt many carefully-planned page designs.

3. Different Browsers
Your Web browser is a translation device. It takes a document written in the HTML language and translates it into a formatted Web page. The result of this translation is a little like giving two human translators a sentence written in French and asking them to translate it into English. Both will get the meaning across, but may not use the same words to do so.
 
The basic rules for translating HTML documents are established by the World Wide Web consortium, which publishes the official HTML standards. But there's considerable room for interpretation within those ground rules.
For example, the HTML standards say that the TABLE tag should support a CELLSPACING attribute to define the space between parts of the table. But standards don't define the default value for that attribute, so unless you explicitly define CELLSPACING when building your page, two browsers may use different amounts of white space in your table.
 
In addition, the HTML standards usually run ahead of what the browsers support. No browser as yet supports 100% of the HTML Version 5 standard, but some browsers come closer than others. Over the past few years Internet Explorer has done a much better job of this than Netscape Navigator, though Opera has done arguably the best job.
 
But since support for the latest HTML tags isn't universal, you could be building your pages with parts of the language that not all browsers understand. In that case the browser will ignore that part of your page it can't translate, and the way your page displays will be affected.
 
This problem is further complicated by browser-specific "HTML extensions." Back during the heyday of the Browser Wars, both Netscape and Microsoft tried to get a competitive edge by running ahead of the HTML standards, inventing their own tags and attributes.
 
The theory was that Web designers would fall in love with these attributes and put little buttons at the bottom of the page saying, "Best viewed with Netscape Navigator." So Internet users would rush to download that browser and, voila, Netscape would win the Browser Wars.
 
In practice no Internet user will download a new browser just to view your Web site, so these HTML extensions only created headaches for Web designers.
 
The best way to minimize these problems is to pay attention to browser compatibility when building your Web page. Avoid using HTML extensions and be careful about using cutting-edge features of the language that may not yet be supported by all the major browsers.
 
We also recommend you test your pages with HTML Toolbox and review its Browser Compatibility report.

[Crazy Guy.]

Browser Compatibility Testing-Considerations

The Various Browser settings that should be noted are
General settings
Connection settings
Security settings
Content settings
Programs settings
Other advanced settings like Multimedia option, Java VM options, printing options etc
Support to Active desktop, Encryption, Instant messaging and style sheets

All the different browsers and operating systems on the market should be covered to verify that the web site or web application functions properly across a wide range of platforms.
OS Compatibility Testing

What is OS Compatibility testing?

OS Compatibility Testing focuses on testing those application features dependent
on the operating system platform.


WHY IT IS IMPORTANT?


To avoid the situations where the Application works fine on one operating system, but closes abnormally or doesn’t work as expected on another operating system.

The Application programming interface, security parameters, principles of working with memory, etc; may change from one OS to the other

Moreover, an Application may work incorrectly on localized versions of the operating system

It is important to test the Application on different operating systems to get a reliable software product


OS Compatibility Testing - Considerations:

The following are the important Operating Systems to be considered:

-MAC OS 10.x, Mac OS 10.3 (Panther), Mac OS 10.2.1 and 10.2.8 (Jaguar), Mac OS

-Win 98, Win 2000, Win XP, NT 4.0

-Fireworks 4

-Microsoft Windows* 2003 Enterprise Edition, Microsoft Windows 2000 Advanced Server, Microsoft Windows XP Professional, Microsoft Windows NT* 4.0 Advanced Server, Windows 95, 98, NT, Me, 2000

-Novell Netware* 6.5, Novell Netware 5.1

-SuSE Professional Linux 9.0, SuSE Professional Linux 8.2, SuSE 9.1 Professional EM64T

-RedHat Linux 9.0

-SCO OpenServer 5.0.6a

-Linux

[Crazy Guy.]

OS Compatibility Testing

Test matrix for OS compatibility testing :

The test matrix is to show the variation in the Application’s features/behaviour
across different Operating Systems

Browser/OS Compatibility Testing:

Though Browser and OS Compatibility testing are discussed as separate topics, in reality the Application’s compatibility features are tested with Browser – OS combinations

We can arrive at the Test matrix for the Browser/OS Compatibility testing based on what
combinations the Customers require to be tested. If not specified, we can make a list of
Browsers and OS that the application would be tested for Compatibility and get the sign off from the client

Test matrix for Browser/OS compatibility testing :

The test matrix should list the Different Browser/OS considered and the features of the
Application to be tested across the combination

Hardware Compatibility Testing:

What is Hardware Compatibility Testing?

This concentrates on Application’s behavior in interacting with the required hardware components & drivers as specified

WHY IT IS IMPORTANT?

Interaction with Hardware is an inevitable feature that should be available with an Application. like print, copy to CD drive options

This testing is necessary to find whether any conflict arises while software’s interaction with Hardware components that are connected or those which will be connected as per the specification

This testing ensures that the interaction results in taking place of proper action or that no negative action takes place.

Example : While printing whether the font type or any special symbols used in the application is recognizable by the Printer


Hardware Compatibility Testing


The following are the example features that can be considered for Hardware Compatibility Test

Application’s stability when a mouse with more than three buttons is used

Application’s primary functionality and stability on a dual-processor computer

Application’s error handling capacity &stability when attempting to use devices that are not connected with the system

Verifying application’s behavior when the keyboard short-cut keys are used for various functionalities

The most common hardware components for which the interaction of the application need to be verified are keyboard, mouse, CD, Printer, scanner, MICR

Version Compatibility Testing:


What is version Compatibility Testing?

Version compatibility testing deals with checking whether a particular version of
software/OS is compatible with the previous or the future versions of the software/OS

There are two types of version compatibility.

1. Forward compatibility :
If an application works with the Future versions of the software, then it is forward compatible

2. Backward Compatibility :
If an application works with the previous versions of the software, then it is Backward Compatible

Exp: A text file in windows 98 is backward compatible till MS-DOS 1.0 as well as
forward compatible to Windows 2000

[Crazy Guy.]

Version Compatibility Testing

WHY IT IS IMPORTANT?


Introduction of new software versions is very common in every release which makes the Version compatibility testing mandate for the software under test

It facilitates error free data migration/update process across the different versions of
the DB

Database Compatibility Testing

What is Database Compatibility Testing?

This type of Compatibility testing concentrates the database’s interaction with Applications for executing queries & retrieving data, supported designers, OS & Browsers.

WHY IT IS IMPORTANT?

Search option is a significant feature in any application which requires querying and retrieving data from the Database. This becomes inevitable incase of Web Application
It helps to check the Database’s interaction with OS& designers, Input/output data standards
It is also vital to check Database’ s compatibility with new databases introduced or different version of the same database as with every release these changes are most likely to take place

The important features to be considered in Database Compatibility Testing are

Data generation & capture features to or from the same database or to a different database

Migration –Database’s ability to migrate data from other databases. Testers must ensure that all supported migration routes are verified against the database under test.

Upgrade- Database’ ability to migrate the data to a new version of the database. All supported Upgrade routes need to be validated

Import/Export feature –Database’s ability to import and Export data to different sources / Application with the specified standards & format

Installation Compatibility Testing

What is Installation Compatibility Testing?

Installation compatibility Testing uncovers any system-specific problems faced during Installation of the Application


WHY IT IS IMPORTANT?


All effort of rigorous testing in delivering a defect free application will go in vain if any problem is faced with the Installation of the Application

The Application’s compatibility with OS, Hardware & other supported applications should be properly examined for easier installation of the Application

Installation process being the first interaction of the user to the application should be made smooth and free of compatibility issues

The following are the considerations for Installation Compatibility Testing

Server Installation compatibility test

Error handing functionality
Uninstalling program functionality
Copying and Modifying system files like DLL and Registry files
Directories creation in proper locations
Works correctly across all OS and system architectures
The disk space available for Installation


Client Installation compatibility test

Client Installation over a variety of systems for reboot problems or browser-specific issues are evaluated in this test


Data Sharing Compatibility testing

What is Data sharing Compatibility testing?

Data sharing compatibility test ensures Application’s ability to share data with other applications

WHY IT IS IMPORTANT?

A significant & basic feature among applications is sharing of data which is most frequently used.

Data sharing compatibility is one of the standards that should be assured for the software under test.

A great compatible product is one which adheres to published standards and allows users to transfer data with other software as per the specification.

The following are the considerations for Data sharing compatibility test

File save and File load functions on different computers for which the data format of the files needs to be compatible with each other

File Export and file import features which requires compatibility of programs with their older versions

Cut, copy, paste being the most familiar method has to be given importance to assure whether the data is properly copied in out of Clipboard

DDE (Dynamic Data Exchange) and OLE(Object Linking and Embeddding) methods where transferring data happen between two applications automatically.

The different ways of Data flow between the Applications under test

[Crazy Guy.]

Cross Platform Compatibility Testing

What is Cross-platform Compatibility testing?

Cross-platform compatibility testing is performed to make sure that the Application is compatible with a variety of leading hardware and software packages. The testing will be carried out on an agreed range of machines that match or exceed the minimum specification incorporating a wide range of leading OEM brands, processors, graphic cards, sound cards and speeds as required

WHY IT IS IMPORTANT?

Web applications need to be accessed from various web browsers with
different operating systems
The application of Client- Server architecture requires simple and complex
exchange of data requests and server responses from any basic application


Cross Platform Compatibility Testing:


An application should be tested for its compatibility with the combination of Hardware
and Software platforms


Hardware platform can refer to Processors or Computers architecture
Most common architecture are x86 and x86-64


Software platform refers to Operating systems or the programming environment
Most common software platforms include

Microsoft Windows
GNU/Linux x86
Mac Os X PowerPC
Mac OS X x86
Java

Wireless Platform Compatibility Testing

This kind of testing of the application is done
on a variety of wireless connection
configurations such as,
1. Infrastructure
2. Standalone LAN
3. Peer to Peer (Adhoc)
The compatibility of the Wireless Applications
are tested across various
1. AP's
2. PC Cards
3. Laptops
4. Desktops (with PC Card Readers)
5. Operating systems
6. Utilities
7. Access points and Routers

[Crazy Guy.]

Compatibility Testing Process

Identifying the customer's requirements and historical compatibility issues

Determining possible test scenarios and coverage measurements. Equivalence partitioning technique can be used effectively to classify the configurations that need to be considered while testing

Creating hardware and software configurations, designing and running test cases against the created required hardware and software matrixes.

Reporting of used matrix, test cases, and found defects and issues together with analysis of possible causes and fixing suggestions


Equivalence Partitioning in Compatibility Testing:
Though it is not feasible to test whether various versions of platforms and software applications work properly with each other, it is important to consider every conceivable combination of hardware and software.

Equivalence partitioning is an approach where classes of inputs are categorized for product or function validation . That is to cluster , like components into classes so that the testing of any one member in a class is representative of all other members of the class. Thereby the components are grouped into a common class, testing with which will generate identical results.

Experience, research and knowledge of the technology used are the main sources of this Analysis



Test Automation for Compatibility Testing

When to go for Automation?

Though we cannot automate the entire Compatibility testing process we can go for Automation in the following situations

When the interaction/Impact of the system under test with other applications, OS, database, browsers are very high for each configuration like Database migration, Import/Export data format validation

When the same test need to be performed across numerous configurations - regression cycle of compatibility testing

When the functionality exists in system under test for which return on automating the compatibility testing is high over Manual testing

[Crazy Guy.]

Test Automation Tools for Compatibility Testing

Q – Engine:
This is an integrated tool for functional and performance testing of the web applications taking various compatibility features into consideration.

Some of the features are:

Multi platform & Browser support: Functional and performance of the application can be tested across Windows XP, 2000, and Linux platforms and across different Browsers like IE/Firefox/Mozilla browsers

Multi database support: Q-Engine facilitates to validate and verify the HTML properties with the data retrieved from different databases like Oracle, MySQL,SQL Server and JDBC- compliant database

It is also featured to Run the Web application against different servers and ports


Browser photo –This tool helps to find the display differences of the web pages taken on 24 different browser and computer configuration combinations


Microsoft Application compatibility toolkit – It has a very wide range of compatibility evaluators to collect and analyze the compatibility features of the application

Online Cross-Browser Screen Capturing, Screenshot Tools

Browser Cam – Provides screen captures of the Web pages loaded in multitude of various browsers, OS versions, screen resolutions and scren sizes

Browsershots – Provides full page screenshots with Mac OS X (Safari) and quite a few browsers for several versions of Linux

iCapture – Provides screenshots with OS X 10.4.8 and Safari 2.0.4.

IE Net Renderer – Facilitates to see how the website is rendered by IE 7,6, 5.5

Lixlpixel Screen Capture- Provides screenshots with the browsers, Safari 2.0, Internet Explorer 5.2.3, Mozilla 1.7.7

Online Browser Simulators :

Anybrowser.com – Using this site,the Website under test can be checked against various versions of browser and browser information

Webpage Backward Compatibility viewer – A software from Delorie software,to check the backward Compatibility of the web application

Compatibility Testing Methodology:

Test requirements gathering

Identify the different OS and browsers and their versions of product/products.

Identify the external interfaces, their role in the business and the technology on which they reside.

Identify the compatibility test coverage for functional and non-functional test requirements

Test strategizing:

Prepare a matrix of application functions against supported variants.
Identify types of tests to be run across multiple configurations (Functionality, Performance, and Security etc.)
Determine the planned test coverage based on criticality of business functions, together with cost and time considerations.
Optimize test effort through automation, and grouping and prioritization of multiple variants (Browser A with Database M on OS X etc.)
Identify the test coverage.
Decide on the Scope of testing.
Business objective of the product.
Technical requirements of the customer.
If 100% test coverage is not possible, identify critical business transactions and get customer approval on the same
Continue..

Test strategizing

Identify the OS/Browser combinations based on customer needs and prioritize based on frequency of use(number of hits ).
Categorize the test case based on frequency of occurrence in real life scenario. Execute all test cases on base browser (Combination which is widely used) and execute only highly used test cases on other identified combinations.

[Crazy Guy.]

Test plan preparation
For each functional and non-functional parameters
to be tested under compatibility testing,
Identify the test coverage of each of the parameters
Prioritize and order the business transactions based on their business implications on product
Data base resource tuning for each of parameter required
Plan to start compatibility testing with lowest version of the browser/ OS selected
Study the release notes for each version of browser / OS released. Schedule and plan to make changes to the test scripts based on the release details
The Test cases should be prepared to evaluate End–to-end functionality from High level for most features and from a lower level for any feature

Test Lab Setup:

Identify all the devices that are required for compatibility testing.
Identify the simulators, if required
Identify all the different OS (with minimum and maximum configuration requirement), other devices like modems etc the software will support
Connect all these devices flexibly so that one can use the same test bed for different test scenarios
Use test bed to reuse the lab very effectively
protect the test bed data to avoid device configuration time for successive verification

Test environment set up & execution:

In case of compatibility testing of OS, plan to set-up environment for every version of the OS / different types of OS as the sources for each version / type of OS may differ at system calls level
Do a sanity check to ensure the environment is completely ready for test execution phase
Execute testing using test plan, identify defects if any
Track defects to closure

Defect Analysis & Result reporting:

For each of the functional, compatibility and Non-functional parameter tested

Carry out causal analysis
Arrive at quantitative and qualitative feedback
Identify the strengths and weaknesses of the product with respect to each parameter
Provide appropriate recommendations
Submit the report to the concerned team and customer

[Crazy Guy.]

Challenges in Compatibility Testing

In web based applications, the user environment is undefined and unknown
Availability of a number of Hardware platforms, OS and Browsers among the User base
Complexity involved in the planning and execution of test cases across this environment
Lack of understanding of complete requirements
Unstable test environment.

Compatibility Testing – Business Perspective

Logo Testing:

Few Companies have a concept of Logo testing in which they provide tools which
can test the application for various compatibility requirements over their platforms.
If the tests are passed, they allow the Customers to use their Logo on their
applications, there by increasing the market value of the product.

Few such tools are,

Application verifier from Microsoft
Compatibility kit from Sun Micro Systems
‘Designed for Palm Products’ Logo program from Palmsource Inc.

[Crazy Guy.]

Compatibility Testing Best Practices

Build a matrix of application functions against applicable configurations

Logically group and prioritize test configurations based on business criticality and frequency of use

Prepare a checklist for Compatibility testing which will cover the important features of the application while performing compatibility testing

Prioritize the test cases based on the frequency of occurrence and execute more frequent test cases only

Start testing from the lowest version of the variants

Automate test environment setup to minimize errors and effort overheads.

Automate tests that will be executed across numerous configurations

Re-test failed tests and their fixes across all applicable configurations

Overall strategy for Good compatibility testing

Analyze the device


How does it work?
How will this effect the testing?
Which of its features does the program use ?

[Crazy Guy.]

Analyze each subsystem/system components


What are the interactions between each component with various other sub-systems?
What are the important interaction and dependent functionality areas between the components and the dependent areas to the systems/subsystems?
For all components document all the dependent areas and the impact of each feature in other components.

Analyze the market

Which of the devices must the program work with?
How to get those devices?

Interact more with marketing team to get this information and collect the critical test features for all the devices

Save time and improve efficiency

Is it possible to subdivide the entire devices into a group of devices?
How to automate and organize the test lab effectively?

Test only one device per group till the errors that are common to the group is removed. This is done with Equivalence Partitioning.
Then test each device in the group for improving the efficiency in such tests. Organize and share the test results for the next project to plan and test more effectively

User’s Installation base Analysis

This is to decide on the Installation base available with the User so that the compatibility testing can be done to get rid of the potential errors with such configurations.

This analyses the system under test against Browsers, hardware and internet connections and related applications on the User’s side

[Crazy Guy.]

Focus on the difference of

Operating system versions
Input/Output device drivers
Extensions running in the background
Applications running concurrently
Network /Online software
Firewall configuration

[Crazy Guy.]

Summary



Important types of Compatibility Testing
The definition and the importance of different types of compatibility testing
The features to be considered specific to the type of compatibility
Compatibility Testing process
Compatibility Test Automation
Compatibility Testing methodology
Challenges and Best practices
Strategy for Compatibility Testing

[Crazy Guy.]

References

Software testing by Ran Panton
Testing applications on the web by Hung Q. Nguyen

[Crazy Guy.]

PEPZ this thread ends here..

[07mca035]

Thanks for sharing.