Introduction
I have been conducting some test research recently. My interest is in determining if several test tools I see have great potential are not only competitive, but complementary. I say competitive because the test tools I use should broaden the scope of what applications I can test. But they need to be complementary because test tools that can be paired together extend the strength of a test framework designed for test automation. For example, I have used Excel, Python, and Robot Framework together for some web application testing in the past. The tools have some unique strengths and weaknesses for the testing world. But some powerful testing can be accomplished using Excel to manage test input requirements, Python to design test scripts, and Robot Framework to drive the test process and report test results.
Well, I have added two more test tools to the mix. I think Playwright and Selenium compete with and complement Excel, Python, and Robot Framework. My initial thought was that Playwright and Selenium weren’t both needed. I was wrong. I will use this article to explain my changed thinking.
Competitive test tools are those that offer similar functionality but differ in key aspects such as pricing, performance, ease of use, or target audience. In this sense, test tools can be seen as competitors to each other, as they aim to provide the same services to their users.
Complementary test tools, on the other hand, are those that offer unique and distinct features that are not provided by other tools. In this sense, test tools can complement each other by offering a broader range of services and covering more aspects of the testing process.
For example, a functional testing tool and a performance testing tool can be considered complementary, as they cover different aspects of testing and can be used together to provide a more comprehensive testing solution. In conclusion, test tools can be both competitive and complementary, and the relationship between them will depend on their specific features and functionalities. The test tools like Playwright, Robot Framework, Python, and Selenium, can be both competitive and complementary depending on the context in which they are used.
Selenium and Playwright are both web testing tools, but they are diverse and yet unique:
- The Architecture of the two tools can be competitive. Selenium is a suite of tools for automating web browsers. In contrast, Playwright is a single library for automating web browsers.
- In terms of Language support the two tools also competitive. Selenium supports multiple programming languages which include Java, Python, C#, and Ruby. Playwright supports JavaScript, Java, Python, and .NET. However, Playwright is able to generate test scripts in all of the languages it supports. Selenium depends on other test tools to extend its support.
- When it comes to ease of use, Playwright has a more streamlined API and is generally considered easier to use. Selenium can require more code to perform the same tasks, but its test speed makes up for its complexities.
- Test tool Performance is highly dependent on the test environment and target application under test (AUT). Playwright is considered faster than Selenium. But I recently did some testing that proved otherwise. I will share that information in this article. Playwright is designed specifically for modern web applications and makes use of new browser features. But that does not stop Selenium from upgrading to a better browser interface.
- AI functionality is the newest interest for the test world. Selenium and Playwright do not include any AI functionality.
Selenium and Playwright have their own strengths and weaknesses, and the choice between them will depend on the specific needs and requirements of a test project. When there is a need for cross-language support and the test team are familiar with Selenium, then it might be a good choice. If the target AUT is complex, ease of use and performance could lead to Playwright as a better choice.
Top of Form
Playwright and Selenium are both web testing tools that allow you to automate tests for web applications. While they have some similarities, they also have some key differences, such as ease of use and performance, that make them somewhat competitive.
Robot Framework and Python are both scripting languages and testing frameworks that can be used to create automated tests. In this sense, Robot Framework and Python can be seen as complementary, as you can use Robot Framework to create test cases that execute the Python scripts.
The test tools can be both competitive and complementary, and the choice of which one to use will depend on the specific test requirements. For example, Playwright can write good test scripts in various languages, Robot Framework and Python are good for writing tests in a keyword-driven approach, and Selenium is strong for cross-language support.
Test Lab Demonstration
As part of this article, I want to expose you to some of what I found out creating a sample set of tests. My target AUT was Salesforce. I choose it often now because it is a challenging application. I mentioned in another article that I like to start testing web applications through their login and logout capabilities. It can be revealing.
Test Environment Preparation
I have another article that goes into detail about adding Playwright to your test environment. Here I am updating you on Selenium. I had installed Selenium a few years ago. But recently I found some upgrades. So, here are two installs that I recommend to ensure your test environment is up-to-date.
| npm install selenium-webdriver |
| pip install webdriver-manager |
Creating a driver and using it is easy although the syntax is verbose compared to the alternatives but still straightforward. Web Driver Manager automates the management of browser drivers, thereby avoiding installing any driver manually. The Web Driver Manager checks the version of the browser installed on a machine and downloads the proper drivers into the local cache.
Once the Web Driver Manager is installed, test scripts need to reference and import the proper library and class. The example here is for Python. For other browsers or other languages check the internet through your preferred search engine.
| from webdriver_manager.chrome import ChromeDriverManager |
Python Test Script Development
My approach to this login and logout test was to create scripts written in Python using both Selenium and Playwright features. My resources were.
- https://docs.pytest.org/
- https://playwright.dev/python/
- https://selenium-python.readthedocs.io/getting-started.html
- https://pypi.org/project/playwright/
I also wanted to use a single Python program with a function to test with Playwright and a function to test with Selenium. It was my desire to design the tests as close to the same as possible and be able to test functionality, ease of development, and performance. The test functions in Python were also designed to be callable by Robot Framework and executable by PYTEST.
Python Test Script
| import os |
| import re |
| from dotenv import load_dotenv |
| from selenium import webdriver |
| from webdriver_manager.chrome import ChromeDriverManager |
| from selenium.webdriver.common.keys import Keys |
| from selenium.webdriver.common.by import By |
| from playwright.sync_api import Page, expect, sync_playwright |
| # Load Environment Variables and Populate local variables |
| load_dotenv() |
| my_id = os.environ.get(“uNAME”) |
| my_secret_key = os.environ.get(“pWORD”) |
| my_app = os.environ.get(“sAPP”) |
| tst_list = [] |
| # Playwright Test Function (old recording) |
| def test_salesforce_old(page: Page): |
| tst_list = [] |
| # Login to Salesforce |
| page.goto(my_app) |
| tst_list.append(“Salesforce App Launch passed”) |
| page.fill(“#username”, my_id) |
| tst_list.append(“Salesforce Username entry passed”) |
| page.fill(“#password”, my_secret_key) |
| tst_list.append(“Salesforce PSW entry passed”) |
| page.click(“#Login”) |
| tst_list.append(“Salesforce Login passed”) |
| # Ensure login is successful |
| #assert page.title() == “Home | Salesforce” |
| #expect(page).to_have_title(re.compile(r”.*Home | Salesforce”)) |
| expect(page).to_have_title(re.compile(r”.*Lightning Experience”)) |
| tst_list.append(“Salesforce Lightning confirmed”) |
| # Open Salesforce Profile Window |
| page.locator(“button:has-text(\”View profile\”)”).click() |
| tst_list.append(“Salesforce View Profile icon click passed”) |
| # Logout from Salesforce |
| #page.click(“#userNavLabel”) |
| #page.click(“#userNav-menuItems > a:nth-child(7)”) |
| page.locator(“text=Log Out”).click() |
| tst_list.append(“Salesforce Log out passed”) |
| return tst_list |
| # Playwright Test Function |
| def test_salesforce_new(): |
| with sync_playwright() as playwright: |
| browser = playwright.chromium.launch(headless=False) |
| context = browser.new_context() |
| page = context.new_page() |
| tst_list = [“Browser Launched”] |
| page.goto(my_app) |
| tst_list.append(“Salesforce App Launch passed”) |
| page.get_by_label(“Username”).click() |
| page.get_by_label(“Username”).fill(my_id) |
| tst_list.append(“Salesforce Username entry passed”) |
| page.get_by_label(“Password”).click() |
| page.get_by_label(“Password”).fill(my_secret_key) |
| tst_list.append(“Salesforce PSW entry passed”) |
| page.get_by_role(“button”, name=”Log In”).click() |
| tst_list.append(“Salesforce Login passed”) |
| page.goto(my_app+”lightning/setup/SetupOneHome/home”) |
| tst_list.append(“Salesforce Lightning confirmed”) |
| try: |
| page.get_by_role(“button”, name=”Close”).click() |
| tst_list.append(“Salesforce Popup detected”) |
| except: |
| print(“Popup not present”) |
| page.get_by_role(“button”, name=”View profile”).click() |
| tst_list.append(“Salesforce View Profile icon click passed”) |
| page.get_by_role(“link”, name=”Log Out”).click() |
| tst_list.append(“Salesforce Log out passed”) |
| # pytest -s –browser chromium –headed –tracing on pwtests/test_pwSalesforce0.py |
| return tst_list |
| # Selenium Salesforce Test Function |
| def test_salesforce_sel(): |
| tst_list = [] |
| #driver = webdriver.Chrome() |
| driver = webdriver.Chrome(ChromeDriverManager().install()) |
| driver.get(my_app) |
| tst_list.append(“Salesforce App Launch passed”) |
| driver.maximize_window() |
| elem1 = driver.find_element(By.ID, value=”username”) |
| elem1.clear() |
| elem1.send_keys(my_id) |
| tst_list.append(“Salesforce Username entry passed”) |
| elem1 = driver.find_element(By.ID, value=”password”) |
| elem1.clear() |
| elem1.send_keys(my_secret_key) |
| tst_list.append(“Salesforce PSW entry passed”) |
| driver.find_element_by_id(“Login”).click() |
| tst_list.append(“Salesforce Login passed”) |
| try: |
| alert = driver.switch_to.alert |
| tst_list.append(“Salesforce Popup detected”) |
| except: |
| x = “No alert detected” |
| else: |
| alert.dismiss() |
| try: |
| button = driver.find_element_by_name(‘Allow’) |
| tst_list.append(“Salesforce Popup detected”) |
| except: |
| x = “No popup detected” |
| else: |
| button.click() |
| try: |
| driver.find_element(“User”).click() |
| tst_list.append(“Salesforce User Element click passed”) |
| except: |
| x = “Element not detected” |
| try: |
| l = driver.find_element_by_xpath(“//span[@class=’uiImage’]”) |
| l.send_keys(Keys.RETURN) |
| tst_list.append(“Salesforce View Profile icon click passed”) |
| except: |
| x = “Element not detected” |
| #driver.find_element_by_link_text(“View profile”).click() |
| #ls = driver.find_elements_by_css_selector(“icon noicon”) |
| try: |
| l = driver.find_element_by_xpath(“//*[@id=’oneHeader’]/div[2]/span/div[2]/ul/li[9]/span/button”) |
| l.send_keys(Keys.RETURN) |
| tst_list.append(“Salesforce View Profile icon click passed”) |
| except: |
| x = “Element not detected” |
| #s = l.getAttribute(“title”) |
| #print(“Title attribute value is :” + s) |
| #l = ls[0] |
| try: |
| driver.find_element_by_link_text(“Log out”).click() |
| tst_list.append(“Salesforce Log out passed”) |
| except: |
| x = “Element not detected” |
| tst_list.append(“Salesforce Browser closed”) |
| driver.close() |
| return tst_list |
The Python test script contains a few things I want to mention. Although I think the code is large enough that time only permits highlighting. So here are a few highlights of the Python program.
- The first two functions are Playwright functions. They were recorded using the Playwright tool called Codegen. I left the first function because I wanted to show you how much the first and second test scripts are different. It seems the Playwright Coders were busy improving Playwright. But comparing Playwright to Selenium, it is easier to develop Playwright code than Selenium code.
- The first Playwright script still has the recorded DEF statement which I changed for the second script. Notice the difference between the two statements. If you intend to make the function callable this change is necessary. Remove the Page reference.
- def test_salesforce_old(page: Page):
- def test_salesforce_new():
- Removing the Page reference requires some additional changes to the code.
- Make sure your library reference looks like this:
| from playwright.sync_api import Page, expect, sync_playwright |
- Make sure your function has the following code:
-
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=False)
context = browser.new_context()
page = context.new_page()
- The code includes many status values appended to a TST_LIST table. The intent here is to build a table of status data to pass to the Robot Framework script for reporting. There is another variation to this technique. Don’t use the RETURN statement at the end of each function. Insert another function that responds to a request for the table and a separate function to request table clearance. You may see benefit in doing so.
- The Selenium function is decent until the test reaches detecting “View profile” icon and “Log out” link for ending the Salesforce session. This is where I find Selenium frustrating. I tried many XPATH and CSS strings with no success. One of the main problems is popup windows or alerts that display which prevent visibility of the logout image and link. I introduced coding to handle alert prompts, but I was not successful. My code depends on closing the browser.
- The Python test script functions can not only be called via a Robot Framework test script, but it can be processed using PYTEST. This allows being able to produce a trace zip file that can be used by Playwright’s Viewer tool. You can find more details in at least one of my tutorial videos.
Robot Framework Test Script Development
There is an issue I found with testing the Python scripts with Pytest. I do think that Pytest is a nice Python tool. It does enable capturing test results that can be viewed in a browser. It does have flexibility with how tests are conducted. You can,
- Control Python console output
- Control headless versus headed testing
- Specify whether you want the browser trace or not
- Specify which test functions should execute
However, the trace viewer tool does not control sensitive or private data display. It captures that data and displays it. So, a password is not handled with discretion. That is when I think about using a tool like Robot Framework to manage this. It doesn’t allow passing the screen flow data from Python to Robot Framework, but status messages is the next best thing.
I did discover that all is not lost. The trace zip that gets created by Playwright can be unzipped. Then there is a file labelled “trace.trace” that can be edited. Simply do a find and replace all. Find the password value or other private data and mask it with asterisks or whatever.
Here is the Robot Framework code that I used to execute the Python functions and produce a report.
Robot Framework Test Script
| *** Settings *** |
| Library test_pwSalesforce0.py |
| Library SeleniumLibrary |
| *** Variables *** |
| ${page} |
| @{test_status} |
| @{plist} |
| @{tlist} |
| #robot -d pwtests\results pwtests\test_pwSalesforce0.robot |
| *** Test Cases *** |
| Example Salesforce Playwright Test 1 |
| #Set Browser Timeout 30 s |
| @{test_status}= test salesforce new |
| FOR ${testl} IN @{test_status} |
| Log ${testl} |
| END |
| Example Salesforce Selenium Test 2 |
| #Set Browser Timeout 30 s |
| @{test_status}= test salesforce sel |
| FOR ${testo} IN @{test_status} |
| Log ${testo} |
| END |
The Robot Framework test script is simple. It does the following:
- Reference the Python program as a library that has some keyword functions
- It references the Selenium library for normal testing
- It calls the Python functions for execution and then displays the status data returned
- The FOR loops navigate through the list tables to output the status data in a list format.
Robot Framework Test Results
The final section of this article is a demonstration of portions of the reporting enabled by Robot Framework. The output is HTML. And it can be voluminous. So, I just want to draw your attention to the status output to give you some idea of its usefulness.
Test Results Main Page
Test Results Summary Page
Test Results Test 1 Page
Test Results Test 2 Page
Closing
While I believe that Playwright has brought a good alternative to only testing web with Selenium, there are three strong points in favor of Playwright. And there is one important strong point in favor of Selenium.
- SELENIUM – the results I see for testing login/logout in Salesforce indicates Selenium is faster than Playwright.
- PLAYWRIGHT – it is much easier to develop Playwright scripts and they are callable.
- PLAYWRIGHT – Playwright methods seem to be more reliable. It detects locators better than Selenium.
- PLAYWRIGHT – it provides good test results output. It includes screen shots, a trace viewer, and it can generate test result videos.
I hope you found reasons to take a closer look at these combinations of test tool software. I will be publishing more reports on my test findings. I will also be publishing more video tutorials on the subject. Please come to my website and locate my videos. Links are on my website.
If you desire to get more detailed tutorials regarding good use of Selenium, I highly recommend the Guru99 site tutorials.