Example implementation of automated software testing

The following example describes a recommended approach to the implementation of automated software testing using the GUI automation tool Selenium, as seen by the author, Bruno Bosshard.

I have most experience with Selenium using the Java stack (by far the most popular), and therefore the following implementation example applies to this technology, but it can also easily be adjusted to Microsoft .NET by replacing the tools appropriately, for example replace Eclipse with Visual Studio, JUnit with Nunit, Cucumber with Specflow, Jenkins/Hudson with Team Foundation Server / Visual Studio Team Services etc.

I love Java, because it has a large community base and because it is platform independent (particularly supporting both Microsoft Windows and Linux operating systems). However, Microsoft .NET has also become open source and might catch up (to Java) with regards to platform interoperability. For smaller teams, Microsoft is offering Visual Studio “Community Edition” for free and also offers free cloud services, such as Git source control, defect tracking, build tools, and other Application Lifecycle Management (ALM) tools through Visual Studio Team Services, its cloud version of Team Foundation Server. For modern web and cloud developers (particularly with JavaScript), Microsoft offers the excellent cross-platform, free and open-source Visual Studio Code.

In Java, the developers will most likely work with an Integrated Development Environment (IDE), such as Eclipse (or Android Studio for Android apps), and that should therefore be the starting point both for Unit testing by the developers and specialised test automation developers that work closely with the developers, as well as all subsequent higher level testing, such as System (Integration) Testing and above.

I would use JUnit as framework for creating data driven tests. The data for the data driven tests could come from Microsoft Excel sheets feeding into JUnit, so that these Excel sheets could also be maintained and changed by business experts (Subject Matter Experts). In Java, access to the Excel sheets could be provided by the Apache POI API. Alternative data sources could be flat files (using the Java IO and utility classes), or databases (using JDBC for database connectivity).

For JavaScript applications, I would use Jasmine as (unit) test framework and Karma as test runner to execute the JavaScript tests in Continuous Integration.

Behaviour Driven Development (BDD) extends traditional unit testing and integration testing with sensible automated acceptance testing. BDD formalises and builds on the principles of Test-Driven Development (TDD). TDD reverses the usual sequence of coding first and testing afterwards. In TDD, tests are written in advance of the code needed to satisfy them.

BDD facilitates clearer and unambiguous communication and continuous feedback between the business stakeholders and development and testing. Traditional unit testing is still the foundation of automated testing and is required for fast code checks. The focus of traditional unit testing is on the program code internals (such as statement, branch, and path coverage), while BDD ensures software quality from a business oriented perspective. Traditional unit testing will still have the highest number of test cases and coverage.

I would use Cucumber (or Specflow in .NET) and Gherkin for automated acceptance testing. What makes Cucumber stand out from the crowd of other tools is that it has been designed specifically to ensure that automated acceptance tests can easily be read — and written — by anyone on the team. This reveals the true value of acceptance tests as a communication and collaboration tool. The easy readability of Cucumber acceptance tests through its Gherkin syntax and structure that is optimised for human readability draws business stakeholders into the process, helping developers and testers explore and understand their requirements. BDD ensures that the software design meets the need of the actual code and leaves behind a suite of tests and up-to-date documentation to help preserve the integrity of the code.

The unit tests and integration tests, as well as the automated acceptance tests using BDD (which are still technically unit tests based on JUnit) would be integrated into Apache Maven as Java build management tool. Maven would be used to define project structures, dependencies, builds, and test management. An alternative to Maven would be Apache Ant as build management tool.

As the final step, I would use Jenkins or JetBrains TeamCity as Continuous Integration (CI) server. Jenkins is derived from the Hudson CI server, which is still often used in Eclipse development environments. It supports Software Configuration Management (SCM) and source control tools including Git, CVS, Subversion, Mercurial, Perforce, and ClearCase, and can execute Apache Ant and Apache Maven based projects, as well as arbitrary shell scripts and Microsoft Windows batch files. One of the important features of a CI server like Jenkins is that it automatically triggers the build, based on defined criteria. Jenkins can be deployed to set up an automated testing environment where Selenium WebDriver and other tests can run unattended based on a defined schedule, or every time changes are submitted to the SCM tool. Jenkins also provides the ability to display test results by reading the results files generated by Unit test frameworks (such as JUnit). It also archives these results, which can be used to generate various metrics over time. The results can also easily and inexpensively be sent as push notifications to mobile devices (such as smart phones) by posting the results using JSON to an automatically generated REST interface, for example on the Microsoft Azure Mobile Services.

All program code (including all unit and integration tests) should ideally be checked by code coverage analysis tools. The results of these tools should then be consolidated in a code quality management platform like SonarQube, which can be integrated with the Continuous Integration server (such as Jenkins or JetBrains TeamCity) to apply Continuous Inspection. JavaScript (Jasmine) code can achieve code coverage with open source tools like Istanbul.

For mobile testing, I would use Appium, because it allows testing of all three types of mobile applications: native, hybrid and mobile web. It executes tests on actual devices, emulators and simulators. Most importantly, Appium can execute the same Selenium program code on both Google Android and Apple iOS, without the need to recompile apps, or having access to the source code.

For testing Google Android apps only, I would also consider Google’s free Android Studio IDE, which allows JUnit based User Interface tests using emulators or real devices. Android Studio also has a host of very smart, performance monitoring tools that allow developers to fine tune projects and maximize their efficiency.

I would also consider the use of Groovy for easier and faster writing of automated tests. Groovy extends Java with scripting capabilities, but it is completely optional in Java projects. It mixes and matches perfectly with existing Java program code and libraries, and it compiles straight to Java bytecode. Java projects can contain anything from zero percent to one hundred percent Groovy, so it is perfectly possible to start very small and to extend the Groovy portion over time, without the need for rewriting any existing Java program code.

If the team uses Android Studio for Android app development, then I would consider using the Kotlin programming language instead of Groovy. Kotlin is fully supported in Android Studio and is very similar to Groovy, but Kotlin is statically typed and has at least two major advantages over Groovy. The first major advantage is that Kotlin saves a lot of error-handling code by getting rid of null pointer exceptions, which the compiler achieves by not allowing a null value to be assigned to any object reference. The second major advantage is that Kotlin favours composition over inheritance, which allows for much cleaner and more flexible (Agile) design decisions than traditional Java inheritance.

I believe that Selenium WebDriver is best used with Java, if the tests should be executed on multiple platforms. I have installed Selenium in the cloud in Microsoft Azure on an Ubuntu base server image with Lubuntu as lightweight LXDE desktop (instead of Unity) requiring just a few hundred kilobytes of memory per machine and a cloud SQL Server service (platform-as-a-service) as data storage. Although many web tests can be executed headless (without GUI desktop), it is often required to use a real desktop to verify correct browser behaviour, particularly with JavaScript. Such a cloud virtual machine, or even a cloud Jenkins or TeamCity server, can be fully installed in less than 1 hour. Cloud virtual machines perform and scale up (bigger machines) and out (more concurrent machines) well, with full automated backup and geo-redundancy. On most cloud services, it is possible to add more machines or discard unused machines within minutes, or even ramp up or down capacity automatically, based on automated performance criteria, such as the percentage of processor (CPU) usage. This allows buying just the capacity that is required at any point in time, as cloud services are usually billed per production hour. I have most experience with Microsoft Azure cloud services, but the same also applies to other vendors, such as Amazon Web Services (AWS), VMware, Rackspace, and others. External cloud service capacities can be connected (integrated) into company-internal networks via Virtual Private Networks (VPN’s), or alternatively, a company internal cloud service can be established.

Once that Selenium scripts run fine, they can also be executed in headless Linux machines using virtual display servers such as Xvfb, if remote access via remote desktop (VNC) is not an option, although remote desktop can easily be tunnelled through Secure Shell (SSH). The advantage of this virtual display server approach compared to headless browsers is that it still uses real web browsers (such as Google Chrome, or Mozilla Firefox and others) in the background, as opposed to robot (headless) browsers.

However, a much better approach for headless testing are specialised headless browsers, such as htmlunit, PhantomJS or Zombie.js. This approach does not require an X11 display server on the Linux machines, and it does also scale out much better to run multiple test instances simultaneously. It could even be used to run some limited performance testing, although specialised performance test tools that work on the protocol level, rather than the GUI level, such as Apache JMeter or SmartBear SoapUI (for SOAP or REST API testing), would be more suitable for heavy load testing in most instances.

Headless testing can also be achieved by using Google Chrome in headless mode, starting with version 58 in Linux and version 60 in Microsoft Windows. Although I am sure that Google Chrome will replace the specialised headless browsers like PhantomJS in the future, there are currently still serious performance problems with headless Google Chrome in Linux, which often lead to very long execution times.

I have both written and introduced test automation frameworks on multiple projects. I have developed and implemented a hybrid framework for automated testing based on the open source Open2Test framework. This framework allows keyword independent interchangeable use of multiple execution engines such as HP QuickTest Professional, or Selenium WebDriver. It is however keyword centred and therefore only suitable, if Subject Matter Experts or test analysts with limited test automation experience want to learn and write test scripts using the keywords of the framework. In my opinion, this is not a good choice for experienced test automation engineers, or complex environments that require a lot of customisation.

In most cases, a better choice in Java environments would be the data driven open source EasyTest framework, which extends the JUnit unit testing framework. EasyTest imports and exports test data from a variety of formats (Microsoft Excel, CSV text files, XML sources, and other data formats) and also has basic reporting capabilities to Adobe PDF format, Microsoft Excel, and HTML format. I would use the EasyTest framework as a basis and extend it with a general keyword library, application specific libraries, and page objects and test templates.

If the team practices Behaviour-Driven Development (BDD), then I would use the open source library Serenity BDD for reporting. Serenity BDD can be used both with Unit test tools (such as JUnit) stand-alone, as well as on top of specialised BDD tools, such as Cucumber/Gherkin. Serenity BDD automatically generates attractive, graphically illustrated, narrative reports that document what the application does and how it works (“living documentation”). Serenity BDD tells not only what tests were executed, but more importantly, what requirements have been covered and tested. Serenity BDD can be integrated with requirements (user stories etc.) stored in an external source (such as Atlassian JIRA), or just use a simple directory-based approach.

An example Serenity BDD report
An example Serenity BDD report

Leave a Reply

Your email address will not be published. Required fields are marked *