How resource-aware scheduling improves levels of automation

Intelligent test scheduling and device management refers to automating the processes involved in:

• Choosing which test cases to execute and when
• Assigning the correct devices to each test case
• Ensuring the most optimal combination of test cases and devices are matched with each other
• Reducing or eliminating test case failures that occur when devices aren’t available
• Deciding which tests can be executed with little or no test engineer involvement, such as regression tests

Test scheduling and prioritization go hand-in-hand when automating device management: With scheduling, the focus is on making sure the right test cases are run at the most optimal times. This is especially important for running tests outside of working hours and hours-long test cases. Likewise, precise scheduling is a critical aspect of migration testing when there’s only a few hours to test a new system before deciding whether or not to switch over to the new one.

However, scheduling doesn’t just involve the tests themselves: It also involves assigning the right devices to their respective test cases. As projects require devices to have multiple criteria and use different combinations of them, optimizing both scheduling and device prioritization is a challenging objective to achieve.

Challenges in test scheduling and device prioritization

Let’s illustrate some problems that can arise in device management with a basic scenario where testers manually select devices for their test cases:

• 10 testers are working on a project
• They share a list of test cases in a spreadsheet
• Test assignment is haphazard and manual
• Users might choose a test because, for example they know it will be unlikely to fail or they find the type of test case more interesting

A major productivity and accuracy problem, however, is that different test cases may need different phones. For example, testing international calls requires a phone that is recognized as “roaming” or outside of the country. Alternately, a scenario may require a phone that has a specific subscription plan or tariff. Because phones cannot yet take on all characteristics or requirements outlined in a test project, most test teams have a limited number of phones that can be used for any given test case. This is especially the case for smaller organizations that need to strike a balance between purchasing the right amount of phones vs. how many people are running test cases during business hours.

For example, if all 10 testers from our scenario above are trying to execute a roaming test case, but there are only five phones that are designated as “roaming” phones, then at least five of those test cases will fail because of not being able to acquire the correct phone. It also means some testers are “blocked” from executing the test case as long as the roaming phones are in use by other testers.

Consequences from this approach include:

• Bad test data that misrepresents how often a test is run, fails and its reasons for failing: This arises from running the test case over and over, while trying to acquire a specific device
• Members of the test team are unable to use their time well, because they need to wait until someone who does have a “roaming” phone notifies them that they’re finished
• Delaying a project’s completion because of poor time and resource management

How resource-aware scheduling makes testers’ lives easier

To solve these prioritization problems, QiTASC developed sQedule, which is a resource-aware scheduler that organizes tests using Agents. The application evaluates and allocates phones and other devices, allowing for multiple test cases to be run in parallel by the Agents. For example, a list of all test cases are passed to the scheduling process, which executes the maximum amount of possible test cases by optimizing the test case properties and available phones. In doing so, it addresses the following questions:

• Which tests should be prioritized and when should they be executed?
• Can they be executed outside of regular business hours?
• How should tests be distributed among the test engineer team?

While test engineers’ technical and project know-how plays an important role in evaluating which test cases should be run and when, the amount of effort spent on these tasks can be greatly decreased by automating phone allocation and scheduling tests that don’t require human supervision. sQedule‘s automated approach to device allocation ensures that test case scheduling is optimized according to available resources, and is especially useful when running regression tests or executing tests that are very long and must be run outside of employee working hours.

Intelligent scheduling may also cover:

• Recognizing configuration switches or IoT devices
• Schedule constraints, such as time constraints where certain test cases are only run on even days or during the last quarter of an hour
• Test-repeat mechanisms, which repeat tests a certain number of times until passed
• Remote scheduling
• System configurations