Why in-house Test Automation Framework Development Fails?

Automated Testing has become an essential activity within any modern software development practice. In agile methodology, software is released on a regular basis to stay ahead of competitors. To release software frequently and reliably, one must rely on an automated build system, which minimizes human error and speeds up the deployment process.

One step within the automated build system is to execute automated tests. The purpose of the automated tests is to gain confidence that what we have built is functionally correct. In an ideal build process, there are different types of tests which target the application at different levels.

At the lowest level, unit tests target the individual classes and functions within each class. Their purpose is to catch developer errors. Unit tests are often the responsibility of the developers who wrote the software. The next layer of tests, integration tests, target the interaction of various components, namely APIs and third-party applications. These tests aim to find integration issues. Integration tests are also mostly written by developers. At the last layer, we have end-to-end tests which target the system as a whole and mimic user scenarios to validate various business cases. End-to-end tests are often executed via a user interface and for this reason, are referred to as UI tests. UI tests are generally the responsibility of testers in the team.

This article focuses on the problems associated with UI test automation and the in-house test automation framework development.

QAs do not have development background

As mentioned, end-to-end testing is often seen as the responsibility of testers. Coupled with the need to release faster, then the obvious route is to automate these tests. Historically, testers were seen as not technical enough to take on development tasks. Testers were not required to code or to build a test framework.

Agile development methodology demands testers to be more technical and take on more technical tasks. At least they should be able to code and use a programming language to script automated tests. However, this puts huge pressure on testers. Due to their lack of development skills and background, most testers struggle to keep up with the pressure to automate tests quickly and efficiently.

This has a negative effect on development because the effort is now shifted from testing to struggling with code. Development is hard. You can learn a programming language syntax in a week but will take years to master. Developing a test automation framework from scratch is not an easy task by any stretch. Most testers are certainly not technically capable to take on the challenge.

Developers are not interested in UI test automation

Due to the technical challenges testers face, sometimes the responsibility of developing a test automation framework along with UI automated testing is put on the developers. Unfortunately, in some organizations, there is still a large divide between developers and testers. Developers do not enjoy testing and testers do not enjoy developing. Also, there is enough pressure on developers to develop the software and they do not have enough time to work on the tasks which are seen as tester's responsibility. Further, once developers learn how a specific open source product or framework works, they lose interest in mastering it. Developers would also well know that once an in-house framework is developed by them, it will be theirs forever to maintain and enhance in addition to the development tasks they were already responsible for.

Most of the time, management misunderstands the cost and effort involved in creating an in-house test automation framework. This is clearly not a simple task that most QAs can take and requires a separate development team to build and maintain.

Talent acquisition is hard and more expensive

This new hybrid role of a technical QA (testers who are good at testing as well as coding also called SDET which is an acronym for Software Development Engineers in Test) cannot easily be fulfilled. In some ways, it is like looking for a unicorn. What really differentiates testers from developers is the mindset. Good testers have the right mindset to test a product and find creative ways to find bugs. If we ask them to focus on automated testing, this takes away their time to explore the system effectively. Instead, they will be trying to automate test with their limited coding skills. It is extremely hard to have the two mindsets in one role, and for this reason, finding such people is extremely challenging. And when we do find one of these unicorns, they tend to be very expensive and when they leave the project, the entire in-house framework development goes into a jeopardy.

Summary

Building an in-house test automation framework proves to be very challenging and expensive and often takes many months to develop and maintain forever.

Platforms such as Subject7, alleviates these challenges. It makes testers focus on their main role of testing the product, while being able to turn their manual tests into robust end-to-end automated tests without the need to code and create as powerful of automated tests as what these unicorns can create with coding.

Open Source is Not Free

There is a misconception in the software industry that open source software is free. While it is true that many test automation tools (Selenium, Cucumber, Katalan, etc.) are freely available, these tools are not without cost to the end user. These tools merely provide mechanisms to drive browsers, organize test suites, and enable parallel execution of automated tests, among other uses. However, in order to use these open source products, one has to dedicate one, or sometimes several, developers to write code to automate the browser, build the test suite, and build an automation framework. Costs in developing test automation from open source tools include time spent in learning and creating frameworks from scratch. Additionally, open source test tools require learning extra languages that may fall outside of the main product. As the size of a test framework grows, more people are required to maintain the framework, causing distraction from the organization’s core goals as developers and testers spend time building out a bespoke testing product.

In addition to trying to get internal resources to learn to use open source tools, finding highly skilled automation developers and architects is a very expensive and time-consuming venture. Once these resources are on board, replacing them is not only difficult, but any attrition puts the automation framework creation project at significant risk, as the development of automated tests is typically performed alongside writing of production code. Time lost in this manner is difficult to make up by orders of magnitude. Further, skilled automation developers and architects have significantly more costs to the program than regular manual testers. This cost variance is in the range of additional $20-$40 on an hourly basis. Even with the extra $20/hour for a single skilled automation developer, the annual cost of this additional single resource to the development team is $20 over 2080 hours, or roughly $40,000 per year.

So, what should companies expect from their automation software? An enterprise-level test automation platform should provide the following capabilities out of the box to enable maximum productivity right away:

  • Should be able to deal intelligently with XPath: Some browser plugins provide facilities to generate XPath, which some automation testers use in their test scripts. These generated XPath’s are often absolute (i.e. very dependent on the existing structure of the page and hence fragile as pages change) and as a result not robust. This means automated tests will break easily, and require a lot of maintenance from sprint to sprint. Another issue for automation engineers is the need to create relative XPath’s.
  • No-coding: Testers should spend time testing, not writing code, and not using substandard record and playback tools.
  • End to end automation capabilities: An enterprise automation solution will offer solutions to enterprise testing problems out of the box, whether it's native mobile apps, testing REST APIs, working with databases, or even desktop, so that users don't have to learn dozens of technologies just to test their software.
  • Cross-browser execution analysis and robustness of execution within each browser: As an organization’s regression tests are executed over and over as part of their continuous integration paradigm, making sense of the information from each test execution is a daunting task. Making sense of that data over time can be overwhelming. A mature test automation platform should summarize the trends of execution of a test case over time, within each browser and across browsers. Testers can use this detailed summarization of each test case to make intelligent adjustments to the test suite, enabling informed testing that truly makes use of the wealth of information available from each case in every browser on every run of the suite.
  • Parallel execution out of the box: Enterprise products often have complex testing requirements and automation suites that take time to run. Parallel test execution allows for tests to be run concurrently, saving time while getting results.
  • Minimal configuration or no configuration: Configuration is a hassle, especially when running various tools over multiple browsers to perform different kinds of testing. Getting each person in the development and product teams set to run tests can be a virtual nightmare.
  • Continuous-Integration-ready: A scalable, enterprise test automation solution needs to be ready to run. Continuous integration and configuring the build system to run automated tests with every new build is a requirement of modern software development.
  • Video and screen-shot of test failures: An enterprise automation solution should offer video and screen-shots of test failures to enable ease in debugging and troubleshooting any issues.

Creating a robust automation platform with all of the mature, enterprise level features would require thousands of requirements, architecture, development, and testing hours and would be another code-base to maintain for a business. Subject7 test automation platform has all of the characteristics mentioned above, and is a true no-code platform. It has been leveraged at various federal and commercial projects at the enterprise level, and has matured over time by incorporating feedback from various clients.

A benefit of Subject7 is that software teams can use their existing staff to create and maintain automated tests as robust as those built by highly skilled developers with open source products such as Selenium. From a business perspective, the subscription costs for Subject7 are incredibly low compared to the costs of hiring developers, building a test solution from scratch, maintaining that solution, and trying to keep up with any changes in the open-source software being used. Even the least technical members of the project team can use the Subject7 solution and provide value, with a low barrier to entry and little or no configuration.