Katalon Studio versus Selenium-based open source frameworks
Katalon Studio is a test automation solution that leverages Selenium’s core engine. Although it uses several Selenium’s functionality, it is not simply a Selenium wrapper. This document offers a detailed comparison of Katalon and Selenium-based open-source frameworks, focusing on key features such as target users, deployment, test management and scripting language support, performance, usability, and integration with other tools.
1. Target Users
Katalon Studio’s users could be the testers with limited technical knowledge. Katalon Studio hides all technical complexities behind the scene and provides friendly UI with the manual mode (user can drag-drop, select keywords and test objects to form test steps), but still keeps necessary weapons for more technically powerful users who are able to dig deeper into coding with the scripting mode that fully supports development conveniences like syntax highlighting, code suggestion and debugging.
Selenium and Appium are mainly automated testing libraries which are more suitable for testers who possess good programming skills. They will typically need to use an IDE like Eclipse or Visual Studio, to import Selenium & Appium libraries (and certainly their dependencies) into a development project, to install necessary unit testing frameworks like JUnit, NUnit and TestNG before starting to write a test case.
Let’s imagine an analogy in the automotive industry where many cars share the same engine while manufacturers design their cars’ interior and exterior differently, depending on their cars’ models.
Similarly, Katalon Studio is basically an all-in-one solution. But if you still prefer (and have necessary skills) to build everything from scratch on your own, you can pick up an engine like Selenium, Appium, etc. to build another automated testing solution like Katalon Studio. Besides Katalon Studio, there are a few test frameworks built on top of Selenium and Appium such as Robot Framework (http://robotframework.org)
2. Deployment & Getting Started
Most open source projects are hard to setup and deploy because they are contributed by many different developers and providers who submit their work into a central place, i.e., Github. You may have to build everything using the submitted source code. But even if frameworks are packaged into a ready-to-use stuff like Selenium and Appium, you still have to go through technical guidelines for their API and integrate them into with an IDE and a preferred programming language. Below are high-level steps required by Selenium, Appium and Katalon Studio to setup and run.
- Download and install Java SDK
- Download and setup test libraries: Selenium client API of your preferred language, Selenium Server, or a standalone package (a bundle of client and server in only one library)
- Download IEDriver and ChromeDriver, setup the environment & PATH variables
- Download and setup the development IDE and your preferred testing frameworks like JUnit, NUnit and TestNG
- Go through the Selenium API document in order to start to create a project and writing some code.
- Download and install the SDK your programming language either Java, Python, Ruby or C#
- Download and setup test libraries (Appium, Node.js, Android SDK, XCode if Mac and iOS), setup the environment & PATH variables
- Go through the Selenium API document in order to start to create a project and writing some code.
- Katalon Studio is built as a unified bundle which includes almost all necessary things like Java, Android SDK, Web drivers to drive browsers, and required dependencies. All you have to do is to download and install it on your computer. Minimum installation is required:
- As Appium is frequently updated, Katalon Studio does not bundle Appium. If you are testing mobile apps, you have to install Appium, Node.js (on Windows) and XCode (on Mac)
- You can quickly get familiar with Katalon Studio as it has friendly UI and sample projects. The detailed user documentation (docs.katalon.com) definitely helps you go through quickly from simple to advanced features.
3. Test management and scripting language support
- Management of test artifacts:
- Katalon Studio provides rich graphical UI with menus, tree views, tables, etc. to manage test cases, objects and data files. Its composer for scripting with full syntax highlight, code suggestion, debugging is ideal for testers who just want to drag-drop or have limited programming skills.
(Katalon Studio manual mode)
(Katalon Studio scripting mode)
(Katalon Studio debug mode)
- Selenium and Appium, when integrating with a dedicated development IDE like Eclipse and Visual Studio, offer intuitive environments for developers but unfamiliar and unnecessarily complex features for manual or non-technical testers. Managing tests is challenging for non-technical testers when using Selenium and Appium.
(The only development mode of Selenium + TestNG + Eclipse)
(Test cases, Test suites, test data are mainly organized in XML file and programming code)
- Scripting language support:
- Katalon Studio: Java/Groovy is the only option for now
- Selenium, Appium + an IDE: Many choices for scripting languages
Katalon Studio uses Groovy, a language built on top of Java, and has to load many libraries for parsing test data, test objects, logging. So, it may be a bit slower as compared to Java for long test cases with hundreds of steps.
5. Usability and Functionality
5.1 Object spy: Spying test objects is the most important but time-consuming activity in test automation.
- Katalon Studio supports object spy for Web (by using browser plugins for IE, Firefox, Chrome) and for Mobile apps (by using the screenshot simulator). Katalon Studio also supports high-lighting captured objects in AUT:
(Katalon Object Spy with object high-lighting support)
- Selenium has a Firefox add-on called Selenium IDE for recording objects on Web applications. Unfortunately, this add-on is no longer maintained and supported.
- Appium provides a companion mobile object spy tool which is useful, but it has no high-lighting feature.
5.2 Test data management and data driven
- Katalon Studio provides Data File object which can query data from external sources: CSV file, Excel file, Relational DB. Katalon Studio also supports data driven testing on a data file object. For example, when testing the login feature, you can specify 100 pre-defined accounts in an Excel spreadsheet or a database.
- Selenium and Appium: testers have to hard-code test data directly in the code. Any support for external data sources requires considerable programming effort.
5.3 Recording and test generation
Test recording is the best way to help users learn and quickly adapt to test automation. It helps an automation tester to do two primary jobs: capturing objects (during recording) and generating test steps with appropriate actions or keywords, saving at least 50-60% time of composing test cases.
- Katalon Studio has the capability to record and generate test scripts on both Web and mobile apps:
- Similar to Katalon Studio, Appium supports recording and generating test scripts for mobile apps.
- Selenium has Selenium IDE, now designated as obsolete, to record user actions and can export into JUnit/TestNG test cases, but object identification information is hard-coded directly into generated code without object management support:
(Test Recording with Selenium IDE)
(Test case generated by Selenium IDE)
5.4 Built-in Keywords
- Katalon Studio is released with pre-defined sets of commonly used keywords or actions, users will find them enough to start implementing most test cases, but if they have more advanced or specific needs they still have a way to create custom keywords for their projects. Custom keywords are an extension of built-in keywords. They can be used the same way as built-in keywords and can be shared between users.
- Selenium and Appium: Users have to build common and reusable actions on their own in forms of programming functions.
6. Reporting and Analytics
- Katalon Studio provides intuitive reports with friendly and readable messages and captured screenshots at the time of failure which can be exported into CSV, HTML and PDF files.
- Katalon Analytics visualizes test execution results in an actionable fashion that enables teams to quickly diagnose and have better insights of their automation tests.
(Katalon Analytics dashboard with visualized execution history)
- Selenium and Appium do not have reporting tools themselves, but based on the testing framework being used will have some simple report templates. Sophisticated reporting and analytics capabilities like what Katalon offers are probably among the most wanted features for Selenium and Appium.
(TestNG Report with programming stack-trace, no screenshot of failure)
- Katalon Studio can integrate with several test management systems and bug tracking systems including qTest (https://www.qasymphony.com) and JIRA to submit bugs, defects, test logs and test results. Katalon Studio can also integrate with Kobiton, a cloud-based mobile lab for cloud-based mobile testing (https://kobiton.com).
- Selenium and Appium: no integration, users have to build the integration capabilities themselves.
8. CI Integration
- Katalon Studio provides a command line interface, allowing CI Systems to call and trigger test execution.
- Selenium and Appium: users have to create and configure execution jobs (Ant tasks, maven goals, etc.) to trigger test execution.
9. Customer Support
- The Katalon team actively supports Katalon Studio’s users more regular than the Selenium and Appium communities, via its website and forum.
- Selenium and Appium are open source, and users may find online help somewhere from other users or the Selenium group but nothing is guaranteed that they will get help quickly.
10. Bugs Fixes, Release Schedule
- The Selenium and Appium groups do not have a clear release plan, and bugs reported do not usually get fixed quickly.
- Katalon Studio is released monthly (minor releases for hot fixes are often available within a few days to a week), and bugs are usually fixed within days. User feedback is highly appreciated and resolved quickly.
11. Solutions for several automated testing problems
- Selenium can only deal with HTML content inside a browser. Special and customized controls such as Windows popups and embedded objects (Flash, Flex, Media) will make Selenium fail to handle.
- Katalon Studio, on the other hand, provides capabilities to handle these Windows controls and recognize images.
- Katalon Studio allows starting test from any step with an already opened session which is very helpful for troubleshooting test.
- Katalon Studio overrides Firefox Driver to make new Firefox versions (47+) backward compatible with old Web Driver while the Selenium community is still struggling with the unstable Gecko driver.