Skip to main content

Continuous Testing

Continuous Testing (CT) is the act of running tests on a schedule. The source of the code base or the tests isn’t necessarily specified. It could be a static, standalone instance that has the ability to execute Cycle Feature or a Playlist on a schedule.

Requirements for Continuous Testing​

  • Development team writes a Cycle Scenario to validate each new feature, improvement, or bug fix they develop.
  • Data and operations planning is required to avoid contention for functions running simultaneously across agents.
  • A task scheduler runs the full set of Cycle Features or Playlists at a regular interval.

Benefits​

  • Fewer bugs get shipped to production as regressions are captured early by the automated tests.
  • Building a release is easy as all integration issues have been solved early.
  • Less context switching as developers are alerted as soon as they break the test build and can work on fixing it before they move to another task.
  • Testing costs are reduced drastically – a CT server can run hundreds of tests in the matter of seconds.
  • The QA team spends less time testing and can focus on significant improvements to the quality culture.

How many agents running Cycle are needed?​

These major factors will determine how many agents your CT setup will require.

  • How long the test suite takes to run:

    • If we need to run our tests every night, and the full test suite takes 20 hours to run on a single agent - that's too long to run daily. Add agents and divide up the test suite until the time it takes to run the full test suite is suitable for daily execution.

  • How many different platforms are being targeted:

    • If we need to test on Windows versions x,y,z - Linux flavor a,b,c - etc - we would likely need a separate agent for each of those platforms.

  • Necessary frequency/timing of test suites:

    • Once we use Jenkins/TeamCity as part of the code merge process, every time a developer needs to merge code, they need to run through Jenkins/TeamCity. If there is a bottleneck there, more agents will reduce it so that multiple people can get their tests done and get code merged more quickly.