Skip to main content
Version: 2.29

Continuous Testing

Continuous testing (CT) is the act of running tests on a schedule. The source of the code base or the tests is not necessarily specified. It could be a static, standalone instance that has the ability to execute a Cycle Feature File or a playlist on a schedule.

Requirements for continuous testing​

  • The development team writes a Cycle scenario to validate each new feature, improvement, or bug fix they develop.
  • Data and operations planning is required to avoid contention for functions running simultaneously across agents.
  • A task scheduler runs the full set of Cycle Feature Files or playlists at a regular interval.

Benefits​

  • Fewer bugs get shipped to production as regressions are captured early by the automated tests.
  • Building a release is more reliable because integration issues have been solved early.
  • Less context switching, because developers are alerted as soon as they break the test build and can work on fixing it before they move to another task.
  • Testing costs are reduced because a CT server can run hundreds of tests in a matter of seconds.
  • The QA team spends less time testing and can focus on significant improvements to the quality culture.

Determining how many agents are needed​

These major factors determine how many agents your CT setup will require.

How long the test suite takes to run​

If the full test suite must run every night, and it takes 20 hours to run on a single agent, that is too long for daily execution. Add agents and divide up the test suite until the time it takes to run the full suite is suitable for daily execution.

How many different platforms are being targeted​

If tests must cover multiple Windows versions, Linux distributions, or other platforms, a separate agent is likely needed for each of those platforms.

Necessary frequency and timing of test suites​

Once Jenkins, TeamCity, or a similar tool is part of the code merge process, every merge runs through that pipeline. If there is a bottleneck there, more agents reduce it so that multiple people can get their tests done and get code merged more quickly.