Test Automation

Table of Contents

Opening Remarks

ADDITIONAL HOMEWORK ASSIGNMENT

Duration: Half day (3-4 hours of lecture time)

Video Training Resources

Overview

Automation Test Life Cycle

Methodology Comparison

Test Demands

Automation Test Tool Categories

Test Management Tool

Bug Tracking Tool

Automation Testing Tool

Performance Testing Tool

Cross-Browser Testing Tool

Integration/Interface Testing Tool

Unit Testing Tool

Mobile Testing Tool

Definitions – Software Terminology

Benefits and Advantages

Frameworks

Automation Test Frameworks

Linear Framework Build

Advantages using Linear Framework Build

Disadvantages using the Linear Framework Build

Data Driven Framework Build

Advantages using Data Driven Framework Build

Disadvantages using the Data Driven Framework Build

Keyword Driven Framework

Advantages using Keyword Driven Framework Build

Disadvantages using the Keyword Driven Framework Build

Modular Framework

Advantages using Modular Framework Build

Disadvantages using the Modular Framework Build

Opening Remarks – Section 1

This training module provides a general overview of the tools and practices used in automation testing. It is not a comprehensive course, but it covers many concepts you need to know to be informed about automation testing.

By the end of this module, you should have a solid understanding of some automation tools and practices, including the basic features and functions, and be able to apply them in real-world scenarios.

There will be lab sessions sprinkled throughout the training. In addition, the assessment session will provide hands-on experience and practical applications, helping you to reinforce your newfound knowledge and build proficient skills in automation testing. I don’t mean to sound like this is a marketing ploy or tactic. So, let me say this in another way. I want you to be successful in the career of testing software. Therefore, I want to provide you with the training that will meet that expectation. If you find that it is lacking, I need your feedback to improve and assure that you complete this course with what can make you successful.

ADDITIONAL HOMEWORK ASSIGNMENT:

• View the official Playwright Website (https://playwright.dev/python/)
• View the official Selenium Website (https://www.selenium.dev/)

Duration: Half day (3-4 hours of lecture time)

Video Training Resources

Overview

Test automation is the process of automating the manual testing processes we have been talking about. Test automation is necessary to improve the efficiency and speed of testing. In contrast, manual testing of test cases might be an experience of you running one test case per 5 minutes. The issue with this is you might take extra time to read through the test case instructions and you might need to find new data too. So, the average can easily go up to 15 minutes. With that rate you are looking at 4 test cases per hour and maybe 30 test cases per day. An application test project can have hundreds of test cases and needs several test analysts to meet the manual test demand.  Test automation is becoming increasingly important as software applications continue to become more complex and the need for faster and more accurate testing grows.

Automation Test Life Cycle

You might not be ready for this, but I am about to introduce to you yet another acronym – ATLC. Yes, automation testing is a process now called Automation Testing Life Cycle. It is gaining ground as more companies seriously endorse the commitment to automating most of their test requirements. This is not a simple matter for any company. It is not a simple matter for a test team or individual. There must be thought put into the planning of having ATLC put in place just like a company endorsing a Test Center of Expertise (TCOE). A separate budget is needed for the short term. And a budget is required for the long-term. It is a paradigm shift from depending on development project budgets to approve money for all testing efforts.

What I want to do presently is focus on the introduction of ATLC. Here is a diagram to illustrate the ATLC methodology.

ATLC Diagram

It has up to six stages associated with the process starting with scoping to determine the boundaries of the automation realm. In other words, how much of the testing and test scripts will be automated. Experienced test analyst will tell you that you cannot automate everything that needs testing. However, to make it worth the effort it must be 70% or better.

Now I should not fail to try and answer a question you might have already. How do ATLC and STLC relate? Does one cancel out the other? Are they a part of each other? What’s the right answer?

The Software Test Life Cycle (STLC) and the Automation Test Life Cycle (ATLC) are both methodologies used in software testing. One does not cancel out the other.

The STLC is more of a sequential process than an iterative process that consists of a set of well-defined phases such as requirement analysis, test planning, test design, test execution, and test closure. It is a comprehensive methodology that includes all testing activities, including manual testing and automation testing. The scope of STLC is broader than ATLC.

The ATLC methodology is a subset of the STLC methodology and is specifically focused on automation testing. It is an iterative process that involves designing, developing, and executing automated test scripts. The ATLC generally consists of five phases:

Test Planning/Scoping: In this phase, the test team determines the scope of automation, identifies the test scenarios to be automated, and selects the appropriate automation tool.

Test Design: In this phase, the test team designs the automation framework, develops test scripts, and prepares the test data.

Test Development: In this phase, the test team develops the test scripts using the selected automation tool.

Test Execution: In this phase, the test team executes the automated test scripts and logs defects.

Test Maintenance: In this phase, the test team maintains and updates the automated test scripts, and the automation framework.

In summary, the STLC is a comprehensive methodology that includes all testing activities, including manual testing and automation testing. The ATLC is a subset of the STLC that specifically focuses on automation testing. While the STLC is a sequential process, the ATLC is an iterative process that involves designing, developing, and executing automated test scripts.

Tool selection is considered a phase in the ATLC phases, but it is only necessary when no automation tools are currently in use at a company site. During the tool selection, the testing team evaluates and selects the automation tool(s) that best fit the needs. This involves identifying the testing requirements, selecting potential tools, evaluating them against the requirements, and then selecting the most appropriate tool(s).

If a company already has an automation tool in place, then a tool selection phase is not necessary. However, the testing team may still need to evaluate the current tool to ensure that it is still meeting the needs and to determine if any upgrades or changes are necessary.

Also, it’s worth noting that the tool selection process is not a one-time event. As testing needs change and new tools become available, it’s important for the test team to regularly review and evaluate their automation tools to ensure that they are still meeting the needs and that they are using the most appropriate tool(s) for their testing requirements.

Last thoughts as mentioned on the right of the slide. These are some items that are not always obvious.

• The Test project team is responsible for managing ATLC
• The cycle can begin from the Maintenance phase or the Test Planning phase
• Maintenance is a cycle too – where Planning is minimal

Methodology Comparison

As I mentioned before, STLC is broader in scope than ATLC. ATLC can run parallel with STLC. You can have an automation team working alongside of the manual test team. One team is working ATLC activities while the manual team is working the STLC activities. It is possible for there to be only one test team that is sharing in the work on both methodologies. STLC is the parent and ATLC is the child. One day there may be some kind of merge. But for now, they can coexist together.

Test Demands

So, test automation changes how we get test projects done. With manual testing it is common to depend on tools like Excel, Word, Notepad, and PowerPoint to accomplish the testing activities. Test automation employs development tools that automate test requirements, planning, suites, test cases, execution, test results, and bug tracking. Nothing I can think of is left out.  The STLC and SBLC processes are covered.

I must add a disclaimer. Manual testing is not diminishing like you might think. The demand for automation testing is on the rise but so is manual testing. You might ask how that is possible. One important truth is that manual testing is cheaper than automation testing for small projects. The future holds much promise for manual test analysts, provided you keep improving test skills and learning new technologies. On the other hand, automation testing demand is increasing because of artificial intelligence (AI)-enabled technologies. All the automation test tools I mentioned for test requirements, planning, suites, test cases, execution, test results, and bug tracking are going to have AI features. This robotic-like functionality will make automation testing activities even easier than they already are. Whether you choose manual or automation testing skills, you will have work available. I used to have very good mainframe programming and testing skills. I thought they would disappear in the year 2000. That is over 20 years ago. I have a colleague older than I am. He is still taking contracts needing mainframe skills only. So, you can redefine yourself, stay with one, or do both. It’s up to you. The work will be there.

I can see doing three small manual test projects in a year or doing one test automation project per year. As I mentioned, keep improving on test skills and learn new technologies.

Before I move onto categories of automation test tools, I should inform you that test demands are shifting. There have been long-time warnings about testing as early as possible during a development project. It has led to some trending actions that may become a strong hold in development and production environments. One of these actions is known as shift-left testing. The other is shift-right.

Shift left testing aims to execute quick, automated, repetitive tests to identify bugs and possible risks at critical phases of software development. The shift right approach monitors user behavior, usage, performance, and security metrics to verify software operability in the hands of its actual users. Shift-left is where a process like TDD (Test-Driven Development) is gaining notoriety. The shift-right testing is gaining ground with APM tools and browser tools. The APM I mentioned is the acronym for Application Performance Monitoring software tools. The browser tools are a part of the browser known as Network and Lighthouse under the development tools of the browser, especially Chrome.

Automation Test Tool Categories

There are many different tools available for test automation. Let me try to categorize them for you. The list is decent for now. I recommend you do further research when you have time to investigate more on this subject. What I have now for you is a useful list. It will take a few minutes to get through this, but I think it will be beneficial. I will talk about each tool category describing each in three areas: purpose, features, and examples. This is not an exhaustive list, but a good reference list.

Test Management Tool

Purpose:

• software used to manage a test project (automated or manual)
• software that manages a repository for test resources

Features:

• Requirements management, test planning, test case development, test execution, test results management, test metrics reporting
• Bug reporting and tracking

Examples:

• ALM, Jira, QA Touch, Testiny

Bug Tracking Tool

Purpose:

• Software used by test analysts and software development teams to report software bugs and problems.
• It provides a repository for bugs, issues, and change requirements

Features:

• Repository, customizable workflow, customizable data entry, reporting, integration support
• Some of these tools may have a Worklog and time tracking feature
•  

Examples:

• Zoho, GitHub/GitLab, ClickUp, any test management tool

Automation Testing Tool

Purpose:

• Software used to run tests for software under test and report on the results.
• It usually includes a language for test script development and management

Features:

• Record and playback, code development, debugging, execution

Examples:

• Selenium, Appium, Playwright, Cypress, Robot Framework, Python Pytest, Junit, NUnit
• Codeless: Testim, Telerik, Datadog

Performance Testing Tool

Purpose:

• Software used for the purpose of evaluating the performance of components of a particular web application under a particular workload. During this testing, system components are monitored to verify the stability, reliability, and responsiveness.
• There are APMs which mean Application Performance Monitoring tools provide support to performance testing projects.

Features:

• Repository for test resources, record and playback, script development, test execution, performance monitoring and reporting, performance analysis

Examples:

• LoadRunner, JMeter, Neoload, Artillary
• New Relic, AppDynamics, and Dynatrace

Cross-Browser Testing Tool

Purpose:

• Software that is a type of non-functional testing that lets you check whether your website works as intended
• It is intended for companies that support multiple browsers for their applications. I have worked with functional and non-functional test automation tools that support multi-browser usage. So, I have not researched how many of these tools are in demand.

Features:

• Simple user interface, testing without installing each browser

Examples:

• Perfecto, Testim

Integration/Interface Testing Tool

Purpose:

• software testing in which the different units, modules or components of a software application are tested as a combined entity
• This can also be software in different applications or web services. This is where API services come in and interface testing tools

Features:

• Interface mocks and simulators, HTTP, JMS, TCP/IP, REST, SOAP, XML, JSON, etc., message header and body assertions

Examples:

• Integration: Citrus, CircleCI
• Interface: SOAP UI, Postman, JMeter

Unit Testing Tool

Purpose:

• testing tools which are used to perform unit testing on a specific module of code.
• They operate inside a language IDE

Features:

• Test script development, debugging, data-driven testing, parallel testing

Examples:

• JUnit, TestNG, NUnit, PHPUnit, Pytest, etc.

Mobile Testing Tool

Purpose:

• Software that helps you automate testing of your Android and iOS Apps.

Features:

• Cross-platform testing, multi-language support, interfaces, reporting

Examples:

• Appium, Kobiton, Katalon

Definitions – Software Terminology

• ATLC – a multistage and iterative process of using automation tools to maintain test data, develop test scripts, execute tests, and analyze test results to improve software quality. Automated testing is also called test automation or automated QA testing.
• Test Automation – the process of using automation tools to maintain test data, develop test scripts, execute tests, and analyze test results to improve software quality. Automated testing is also called test automation or automated QA testing.
• Test Automation Framework – a set of components that facilitate developing tests, executing tests and comprehensive reporting of test results. The major components that implement a test automation framework successfully are testing tools, scripts, and procedures.
• Test Object – The component or system to be tested (ISQTB). A more granular definition for web applications is a page or element of a page that has test properties. For example, a text box is a test object which has test properties, such as input data.
• Test Object Model – an application targeted for automation testing.
• Test Object Technique – an action that can be applied to a test object.
• Test Scripts – one or more executable test cases that is interpretive or compiled code.

Benefits and Advantages

In general, automated testing saves significant test execution time. The primary disadvantages of automated testing are that they can cost more money investing in software and test automation analysts, take much effort to implement, and significant time for maintenance. These are not insurmountable challenges. Let’s come back to that. Here are some important benefits or advantages.

• Faster Feedback Cycle – the faster test results can be reviewed the shorter the development and testing cycles.
• Automation Testing tools can support testing in Parallel which can also be a timesaver.
• Test Script development is ideal for repeated test cycles like regression testing.
• Automation testing can easily be designed for easy data-driven testing.
• Achieving close to or maximum test coverage is possible with automation testing.
• 24X7 Test Execution is possible if needed.
• Automation testing can save time and money. However, Gartner reports that goal-setting and stakeholder buy-in are two main challenges you’ll face if you don’t build a solid business case first. In other words, it’s not hard to figure out that automation results in much faster processes than manual human-led processes. But management needs a business case to get funding. Using Automation tools even when they are free still costs money. Humans are still involved needing money. I will not go into details on building a business case, but Gartner or a business website can help with that. I had to do this some years ago to get my fortune 100 company (ExxonMobil) to agree to a Testing Center of Expertise back in the year of 2000.
• With automation testing you can vastly increase test coverage because you can develop many more test cases in a short timeframe.
• Automation Testing improves development quality due to quicker turnaround finding bugs and fixing them.
• Through having Automation testing in place, the next step is continuous testing.

Frameworks

The more you learn about automation testing, the more you need to be acclimated to test frameworks. Today you will hear the term “test framework” associated with automation tools and in-house developed automated test procedures. I like the fact that automation tools are evolving into frameworks. And I know that in-house developed automation testing must evolve into a framework. I took time a few years ago to do a study and included some results in my Robot Framework video course. You can find it on this site and the www.Guruface.com site.

Turning automation testing resources into a test framework is essential to success. I learned my lesson a few years ago when I wrote a business case at another company as a contractor. They were used to reassigning as many as 50 business analysts to work as acceptance testers for multiple development projects that took place every time an annual release upgrade came for SAP applications. They approved my business case, and I spent around 3 months building automation test scripts with a good product called UFT. Against my better judgment I had to turnover the resources to a novice level test analyst. That is not an indictment to you. But experience is necessary before you take the lead on test projects. Anyhow this analyst did not have competent level knowledge on UFT or testing. This was a recipe for disaster. The resources I developed were soon after retired. Two lessons. Build a good automation test suite framework that requires minimal maintenance. Build a team that has the necessary skills to keep the framework alive.

So, here is something to start you on the road to building test frameworks that sets a good foundation. I am using excerpts from my Robot Framework course here. This lesson in full was about a 25-minute lesson. I hope I have cut it down to no more than 15 minutes to give you a taste. The lesson is partially a case study I did with automation test tools and a web application that no longer exists from Micro Focus. I checked with them and since then they have replaced it but it requires installation on your own machine to use. I have used it for some other testing. In developing this course, I don’t have time to at present. I will plan on doing this in the future.

Automation Test Frameworks

Today there are multiple automation frameworks that are used in the software testing domain. It helps to have a broad understanding of some automation frameworks. So, let us review some frameworks.  The internet is loaded with other individual perspectives on these frameworks, and discussions on others I will not mention in this lesson. 

But I will cover the ones you see listed here.

Each of these frameworks used in test automation development comes with advantages and disadvantages. That is one main thing I want to share with you now. But also, I want to give you some visuals where appropriate to help solidify your thinking regarding these frameworks.

Linear Framework Build

Certain test engines are designed to use this framework. For example, a tool called Unified Functional Testing or UFT, which I have used extensively on test projects.  It is typical when UFT is used to record and playback. It generates code as you navigate through a target application. The result is VBScript-like code that exemplifies a linear step through interacting with an application. Here is an example of a test case generated in UFT for a desktop application.

I want to add some dialog here to expose you to a concept introduced to automation testing by the UFT product. It is a script development concept they call Extensibility. In the diagram you can see code class references “Window” and “Dialog”. And then you see object properties, such as “WinObject” and others. You will also see methods like “Type”. This is an example of their concept called Extensibility. I like to call it Object testing. I want to talk about this later in this training. It is an interesting concept for test automation. You will see more of this in this case study.

Advantages using Linear Framework Build

1. Test build is limited to the Record and Playback features of UFT
2. Scripts built with this approach are simple, but may not be reusable
3. UFT parameterization features can be used to extend the reuse of the scripts
4. Script design follows application navigation flow
5. UFT code features can be used to improve the flexibility of the scripts

Disadvantages using the Linear Framework Build

1. Minimal reusability without parameterization and code enhancements
2. Maintenance can be overly challenging
3. Application flow and entry data modifications may cause script obsolescence

Data Driven Framework Build

This automation framework build is designed to drive each test from data stored in internal and/or external file storage. I am using UFT as an example because it is a flexible tool for frameworks. Not all tools can claim that. Anyway, the UFT local and global data tables is a good starting point. But if the testers are not going to be the test script developers, then input files would be preferable. The input files can be Microsoft spreadsheets or access tables, SQL tables, or even XML and ASCI files. UFT can read these files just like it inputs its own data tables.

The references you see to “Set Database Variable” are examples of the UFT product. Other tools can do something similar. These are pointers to stored data in the script. But as mentioned, some tools can pull the data from external sources as well for data driving.

Advantages using Data Driven Framework Build

1. The test scripts reference input files and data fields. The data is not hard coded in the test code
2. The input files are not limited to a record count, and the data can be changed without affecting the test code
3. The test execution remains fast and driven by the data combinations

Disadvantages using the Data Driven Framework Build

1. Application subject matter experts are necessary to define the data combinations
2. When application data requirements change, the scripts may not be designed to handle the changes

Keyword Driven Framework

This framework design drives test scripts with keywords that identify some function to execute. It is considered among the best test frameworks. The test script can be defined with one or more keywords that identify a sequence of functions to process. This design usually requires an object repository used to map objects to all required keyword actions. In the Keyword Driven Framework, test scenarios are written in a Microsoft Excel spreadsheet. The Driver Action will read the scenario and perform test execution.

Notice the actions in column “C”. These are commands that can support the keyword in column “A”. Together the test script processing is applied to the objects listed in column “B”. And data requirements are handled by column “D”. This approach is efficient, but not ideal for all test requirements. This is one of the frameworks that are well-suited for object testing more than application function testing.

Advantages using Keyword Driven Framework Build

1. The tests are object focused. The code is divided into page and page element actions
2. Spreadsheets drive tests using data and action controls
3. Testers do not need programming skills

Disadvantages using the Keyword Driven Framework Build

1. Functional libraries and Object repositories can take significant time to setup and maintain
2. Keyword driven frameworks can be complex and challenging to design
3. Code Maintenance can be complex
4. Automation test developer always required for environment maintenance

Modular Framework

In the Modular Framework testers need to initially identify reusable code. This code is then automated. The code must be written as functions, and it is important to design these functions based on testing requirements.

Just in case you are not familiar with coding, the “Call” statements you see in the code are examples of code that is referencing some that is external to this code page. You see some calls that are repeated, such as for login and book Ticket. The code can be made more executable by inserting some conditional statements to improve the code execution flow. The important takeaway is that some automation tools support this kind of coding that is modular. Like modular programming, modular test frameworks help to organize complex code into modular and reusable code blocks.

Advantages using Modular Framework Build

1. Test scripts can be created faster, and then modified to call reusable code functions
2. Reusable functions reduce initial coding effort
3. If any modifications are required for the application, function changes need to be updated only in a single place

Disadvantages using the Modular Framework Build

1. Some data may remain hard-coded
2. Reusable code design may not be easy to identify
3. Functions may not be easy to maintain and may not remain reusable very long

This is the end of section 1. To see and read the second section of this article, click here.