Successful automated acceptance tests 2


Share it!Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInEmail this to someoneShare on Reddit

The main role of acceptance tests is to confirm that the developed application delivers the business value expected by a customer. Usually, they are based on an agreed set of user stories or use cases. Acceptance tests are at least executed at the project finish. The good practice is to execute them constantly since the given functionality has been implemented. They work as a guardian automatically detecting any regression. Additionally, the current application state is clearly visible to all involved parties.

Dozens of years ago acceptance tests were usually executed manually. It was a very mundane and error prone activity. Such sessions tended to be quite expensive due to the heavy amount of tester time required. As a consequence they were executed rarely and defects were found late in the project stage (if found at all). Acceptance test automation solves most of those problems. Creation of repeatable automatic test suites allows us to execute them on a daily basis on Continuous Integration servers. Such acceptance test stage provides us, even several times per day, with an exact report about the application state and business value which it can, in the very specific moment, provide.

Unfortunately, proper creation of acceptance test suite is not an easy task. Such tests are often considered as very expensive in terms of maintenance and are skipped by many projects. However, strong discipline and following proper design can lower their costs significantly. For example, simply recording Selenium test might look appealing. The creation of such test suites which would cover most application paths is an easy task but leads to huge maintenance costs. To avoid such situations, 3 layer architecture should be introduced and used:

  • acceptance criteria layer – represents customer requirements. Can be specified as a use cases, stories following format “given… when…. then…” or any other suitable pattern.
  • test implementation layer – a list of steps (the best if using some specific DSL) which should be performed to prove that the specific customer requirement works. It can be considered as a list of actions which the tester should execute, like create user, create payment, etc. However, such code shouldn’t be aware of any UI elements,
  • application driver layer – implementation of interaction with application. Test implementation layer uses it to execute defined actions. Only this layer is aware of specific UI elements and is able to interact with them.

The above architecture allows us to implement specific interactions with application UI in only one place – application driver. Thus, whenever any UI element changes we should be able to adjust all tests by only modifying a piece of code responsible for accessing that element.

Acceptance tests layers

Even if following this approach there are many pitfalls which can lead to high maintenance efforts. The most common ones that are worth remembering are:

  • make sure that only the application driver layer is aware of specific UI elements. Additionally, make sure that for each element or piece of screen there is only one place in the code that interacts with it. Thanks to that there is only one place which needs to be adjusted in case UI element would change. It also facilitates optimization and refactoring activities,
  • tests should not depend on each other. There are many solutions to this problem like each test using a separate user or is performed in a separate transaction which is rollbacked at the end of it,
  • tests should verify its starting conditions. Thanks to that we will be able to clearly distinguish leftovers from other tests from other problems.
  • timeouts should be handled actively. Instead of hardcoding timeouts use loops which will, in short time periods, verify if the expected action has been performed,
  • mock external systems which your application depends upon if you are not able to use them on daily basis for testing purposes. It is advised to implement a specific test for such systems or services and test them separately from the application. However, it is a good idea to perform full integration tests on later stages, before releasing to production.

Another practice worth following is to involve almost everyone in the project into automated acceptance test development. It is crucial that testers, developers and system users/customers will actively participate in the process. Customers can assure that tests are really valuable and confirms business value, testers provide very valuable information regarding test automation and developers will feel responsible for ensuring that all tests pass most of the time.

As a consequence of such an approach, acceptance tests become an executable specification of an application. We can move even further and use one of the frameworks allowing us to automatically translate user stories into executable tests. Then we will be able to say that each user story is verified in an automated manner. Any change in the documentation would be reflected in tests almost without any work.

While talking about successful automated acceptance tests we cannot forget about performance topic. This kind of test usually takes a lot of time and as a result provides feedback not as frequent as we would like to. There are many strategies on how to approach the problem, the most common are:

  • parallelization – as hardware is usually much cheaper than developer work, simple parallelization quite often does the job,
  • refactoring the biggest bottlenecks – in case you are using the application driver pattern you should be able to identify which interactions take the most time and enhance them. If the pattern was applied correctly, a fix in one piece of code can hasten a lot of tests,
  • prepare some data upfront – if your tests spend a lot of time on data preparation and cleanup, you can prepare such data upfront.

Automated acceptance tests is a little bit of a controversial topic. In my opinion they are a great way to prove that the application works correctly. Developing them right is very important. It is easy to end up with thousands of almost  unmaintainable and unstable tests that are hated by developers and testers. On the other hand, if done properly from the very beginning, they can bring tremendous value to the project. Do not be afraid of them, just make sure that their maintenance effort will be low.

Share it!Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInEmail this to someoneShare on Reddit
  • Alex Klunniy

    One thing to add here is never let your acceptance test fail for more than one day. React immediately. If tests fail for continuous amount of time, noone will treat them seriously as a thing bringing the real value.
    It is great if acceptance tests is a part of CI process. Then when you come to work at the morning, first look at your test results. If any failed make appropriate changes to the code or tests.

    • http://continuousdev.com/ Piotr Oktaba

      Very good point! We should avoid the scenario when there is a set of all the time failing tests which the whole team ignores (because they are failing all the time…).