The goal of test automation is to greatly improve a product’s quality in a shorter period of time by eliminating redundant and time-consuming testing activities that were previously done manually. When people think of automated tests, they might initially think that it entails only the test execution. While automating the actions performed in the test is key to improving testing efficiency, this only covers a small amount of what can potentially be automated: Ensuring high test coverage and levels of automation requires going beyond the test execution itself and thinking about other aspects of entire test projects that can be simplified, integrated and automated.
At QiTASC, we’ve extended the scope of what we automate to areas previously left to manual testing. Between 2018 and early 2019, we’ve introduced and improved the following activities, which complement test creation, management and execution provided by intaQt:
- sQedule intelligent resource management and test case scheduling
- Device management and allocation with the intaQt Mobile App and intaQt Studio Phone Plugin
- The Device Abstraction Layer (DAL) allows any Android device to be configured and quickly integrated into the testing process
Audio Analysis and Comparison
- Confirming that the correct audio announcement is played
- Checking that the audio quality is within an acceptable range
- Speech-to-text & speech recognition automation
- Centralized project reporting and bug tracking
- Remote test case execution on phones and tablets
Additionally, 2019 will see the introduction of several huge productivity-enhancing products, which we’ll be revealing in the next few months:
- disQover - A test case generation tool that automatically creates and updates entire suites of test cases using graphical models
- QiTASC PhoneHub - A modified USB hub that allows intaQt to control phone hardware including electrical currents, connected/disconnected states, and phone audio streaming capabilities via integrated soundcards
Which tests should be prioritized and when should they be executed? Can they be executed outside of regular business hours? Deciding which tests to execute and when, or how to distribute them among a team of test engineers is an important but costly and time-consuming project management activity. While test engineers’ technical and project know-how plays an important role in evaluating which test cases should be run and when, the amount of effort spent on these tasks can be greatly decreased by automating phone allocation and scheduling tests that don’t require human supervision.
To solve this prioritization problem, we created sQedule, which is a resource-aware scheduler that schedules tests on Agents. sQedule evaluates and allocates phones, allowing for multiple test cases to be run in parallel by intaQt Agents, which requests test cases from the sQedule server. A list of all test cases are passed to the sQedule process, and sQedule will execute the maximum amount of possible test cases by optimizing the test cases properties and available phones. At QiTASC, we have run several thousand end-to-end regression tests within a day using sQedule.
This automated approach to device allocation ensures that test case scheduling is optimized according to available resources, and is especially useful when running regression tests or executing tests that are very long and must be run outside of employee working hours.
In addition to sQedule’s automated resource management, the application also offers features for managing configurations and time constraints such as:
- Recognizing configuration switches or IoT devices
- Supporting time constraints, for example, to specify that certain test cases are only run on even days or during the last quart of an hour
- Test-repeat mechanisms, which repeat tests a certain number of times until passed
- Remote scheduling, allowing testers to schedule and assign tests from intaQt Studio even if they do not have a local intaQt
In all cases, sQedule executes tests as soon as possible, which helps manage devices and projects much more efficiently.
A common testing goal is making sure that apps and devices work across borders, whether in international locations or in the home network. Although remote testing has been around for several years, a common challenge has been solving hardware problems that arise when a phone is located hundreds or thousands of kilometers away from a user. Previously, troubleshooting devices might require sending a support request to someone in another location, who would then need to physically access the phone.
The intaQt Phone Plugin, which is one of intaQt Studio’s many productivity features, lets users interact with all devices in their project no matter where in the world they are. With a few clicks, users can check whether or not a phone is turned on and actually registered to a network, or close dialog boxes that unexpectedly show up.
Newsroom: Remote Apptest with intaQt
Although watching a device carry out all the steps of an automated test case may seem counterintuitive, it’s important to be able to check what a phone is doing, especially if something critical arises. The intaQt Phone Plugin, which we mentioned above, has a live streaming feature that shows a test cases phones in realtime and even lets the user hear any audio being streamed on the device.
Having access to live streams of devices is especially useful for identifying incorrect behaviors or figuring out of a step isn’t working as intended. The intaQt Phone Plugin allows users to interact with devices on the screen in front of them in the rare cases where manual interventions might be needed.
When working with multiple team members who all have access to the same pool of phones, we observed several problems, such as:
- Not being able to see which users are accessing phones
- Not seeing if a phone is available/recognized by intaQt
- Not having an easy way of checking phone and subscriber characteristics
- Not hearing what audio is being played
In addition to livestreaming devices and allowing remote interaction, the intaQt Phone Plugin provides an overview of all phones in a project. Clicking on a phone reveals its properties, including hardware and network-related characteristics. It also shows if the phone is in use, and by whom, as well as the name of the test case the phone is assigned to.
Finally, it was time to give the QiTASC Autotest app a big update. The new intaQt Mobile App has a faster performance that is complemented by a cleaner appearance. The app now displays the tester and test that is currently using the phone, which simplifies tracking devices. The Mobile App is installed on each phone used by intaQt, which allows phones to be automated by QiTASC’s software. The user interface gives users quick access to device information, and makes SIM card and MSISDN management much easier and more reliable. We’ve also optimized phone-to-server communications for the new app, meaning phone acquisition and property retrieval are now almost instant – phone properties are pushed to intaQt in real time.
Newsroom: Managing Phones with the New intaQt Mobile App
When testing Android phones, minor differences between models and Android versions makes it difficult to ensure that any and all models can be tested consistently for reasons such as:
- Every Android model supports multiple Android versions
- Android versions may have slight implementation variations in the underlying code
- Even phones running the same Android version may have differences due the custom firmware that different manufacturers provide
For example, an end user might want to test the same phone model, such as the Samsung Galaxy S8, with all the different supported Android versions – 7.0 (Nougat) to 9.0 (Pie) – to ensure that the behavior of the device remains the same. Or, an end user wants to ensure that all devices running the latest Android version behave in the same way. However, Android OS segmentation and can threaten the reliability of testing or the ability to properly integrate a phone with test automation software.
To make testing multiple models as convenient and accurate as possible, QiTASC is introducing the Device Abstraction Layer or DAL, which allows any Android device to be configured and quickly integrated into the testing process. This feature enables the end user to take control of the phone integration process. With DAL, end users are no longer dependent on either the QiTASC phone integration process or the popularity of a certain device, meaning any required Android phone model can be configured with DAL and quickly integrated in the testing processes.
QiTASC wants to enrich and empower the ecosystem that it is surrounded by and to further the notion of open-source code and projects. For this exact reason, the DAL implementation of any device that already exists will be uploaded to a public repository anyone can access, improve or use DAL as a guide for integrating any device that is not already supported. Through the power of the community, it is expected that all popular devices will have a DAL implementation within a reasonable amount of time.
When testing telecommunications networks, one of the most challenging use cases is checking that a customer hears the correct audio under the following circumstances:
- Navigating IVR systems
- Listening to audio announcements
- Receiving network operator messages
Test cases that include audio attachments often require users to manually replay the recordings and check that the correct file was played. These manual efforts are time consuming and can greatly decrease the benefits of an otherwise automated test case. To solve this productivity issue, QiTASC developed Audio Fingerprinting, which is one of the features of our Audio Plugin. Audio Fingerprinting automatically compares the audio generated in a test case against a set of reference files and based on the files’ “fingerprints” can determine if the correct audio was played. This is especially useful when testing long announcements and when announcements in foreign languages need to be checked.
Newsroom: Automated Announcement Verification with the intaQt Audio Plugin
Like Audio Fingerprinting, checking the sound quality of a mobile phone call has traditionally fallen outside the scope of automated testing. Manually judging audio quality is quite error prone because of the variety of factors that influence what a tester might hear and how they judge “acceptable” audio quality.
intaQt Speech Channel Monitoring is a robust feature that checks for predominant frequency, pauses and distortion. This type of testing is done by automatically playing audio reference files during the call, which represents sounds that might be transmitted between two phones during a call. The content and quality of these files are then compared against a set of requirements that define acceptable tolerance, pauses, frequencies, and acceptable deviations.
Newsroom: intaQt Audio Plugin
Speech-to-text and speech recognition automation is most commonly associated with audio transcription tools, such as those used to transcribe interviews. However, as smart home devices and virtual assistants like Amazon’s Alexa become more common, it’s becoming increasingly important to test that the software properly recognizes voice prompts.
intaQt Speech-to-Text functionality supports speech-to-text recognition of
.wav audio files. This feature integrates services provided by the Google Cloud Platform Speech-to-Text API. The Speech-to-Text step specifies an audio file and its language and checks that it is converted to text correctly.
A huge efficiency killer is tracking a project’s milestones, requirements and results. Many companies still use spreadsheets to keep track of their projects, which requires each tester to manually update fields such as
pass/fail, defect numbers, error messages and other characteristics. Tracking data this way is incredibly error prone as well as time consuming, and creates a lot of additional work for test managers who need to review all the information.
QiTASC conQlude was created specifically to eliminate this problem by automatically uploading reports and test artifacts from all users in a project. Every test report is uploaded to a centralized location and parses the most important metrics such as test result, tester, defect numbers, duration and test case ID. Log artifacts that are automatically attached to the report include:
adb logcatsfrom participating phones
- intaQt Phone Service logs for participating phones
- Automatic screenshots of the phones when test cases fail
- The intaQt log
- All the custom attachments that the test creates
conQlude prevents data entry errors, inaccuracies and stops human errors from occurring. It also has proven to save test managers hours of time spent reviewing test data with an integrated approval workflow feature. For customers who use other tools like ticket tracking systems (such as YouTrack and Jira) or project management systems (such as HPQC), conQlude integrates seamlessly with these frameworks, and can both parse information from these systems and export test data to configured databases.
Newsroom: Transparent Project Reporting with QiTASC conQlude
One of conQlude’s most useful test management features is its automatic error categorization, which enables fast analysis of test results. Once error categories are defined, they are automatically assigned to all test execution reports that match the error category. If a new error category is created in the middle of a project, these errors are applied retroactively to test cases that were executed before the category existed. This helps reduce the need to review and sort through old test cases when new defects arise or are recategorized.
Documentation: conQlude Administration
Looking back on the past year, we’re proud of how much the QiTASC team has advanced the scope of what we automate, how our suite of applications integrate with each other, and the improved levels of productivity and reliability that this holistic approach to testing and quality assurance provides. As we continue into 2019, we’re excited to announce additional tools that will make testing even more effective at improving product quality within the context of increasingly short deadlines.
We’ll be gradually introducing the following, which you’ll be able to read a lot more about soon:
Frequently, entire test projects need to have small adjustments made across all test cases. This can arise, for example, after a new requirement is added or when a bugfix has been introduced. Manually changing each test case can take days or even weeks, preventing users from effectively utilizing their time and energy. Although some automated solutions exist for this problem, many of these still only exist in scientific research environments. Other commercially-available products enable model-based test case generation, but are limited to specific use cases such as app testing.
QiTASC disQover is a model-based test case creation and management tool that lets users easily and instantly make changes to entire projects. Using graphical models and a templating framework, disQover helps the user quickly identify where a change needs to be made and adjust the necessary parameters. disQover then propagates the changes throughout the project in a matter of minutes. The application is use case and technology agnostic, meaning it is compatible with all the types of automated testing and devices that QiTASC supports, including:
- Telecommunications network testing
- UI Testing
- Backend and Protocol Testing
- Mobile phones, VoIP phones, desktop browser, tablets and virtual devices
Because the underlying templating framework is also language and file-type agnostic, disQover is not only capable of generating test cases but other files, for example, it can generate documentation files (
*.md) and other text files from flows.
In order to be as user-friendly as possible, disQover’s complexity and mathematical logic stays “under the hood”, while an intuitive graphical user interface guides users as they create and updated flow diagrams that represent the test project. disQover provides templates, which means models can be reused, tweaked for different projects or become the framework for any other similar projects. This means that as a project grows, or when new projects are being created, the set-up and maintenance time required to make all test cases accurate and up-to-date becomes shorter and shorter. This time saved from the initial process can now be allocated to valuable work such as fixing defects and focusing on project development.
Android smartphone batteries are in a
charging state as long as the phones are connected to USB. Because intaQt can only control phones via USB, phones need to be connected to USB at all times during testing. This causes phones to charge their batteries to 100% and keep them charged at all times, which is not healthy for rechargable batteries. Furthermore, having a phone connected via USB at all times does not reflect the typical smartphone user’s behavior. Additionally, in rare cases, a phone may lose its connection to USB and needs to be manually unplugged and replugged, which can be especially challenging when using remote phones.
The QiTASC PhoneHub is a modified USB hub that provides better control and battery life to Android phones being tested by intaQt. These modifications reduce the amount of configuration required and limit the necessity to manually plug and unplug devices throughout a test project. The modifications allow intaQt to:
- Limit the electric current that can be consumed over USB by a phone
- Virtually unplug and re-plug a phone from USB
- Control audio streams via integrated soundcards: Devices connected to the PhoneHub can also be connected via audio cable
- intaQt can “listen” to audio coming from the phone and “talk” into the microphone to reflect a two-way phone conversation
- Audio streams for up to three phones can be used for test cases without any configuration required
intaQt together with the PhoneHub mimics this behavior of a user’s experience with a phone being used throughout typical daily use. The PhoneHub supplies the phone with a limited amount of electric current as long as the battery level is high enough to guarantee full functionality. If the battery is drained below a certain level, intaQt tells the PhoneHub to deactivate the electric current limitation, which recharges the battery. As soon as a certain battery level is reached, intaQt tells the PhoneHub to activate the limitation again, meaning the phone gets its power from of the battery instead of via USB.
This approach to managing the electric current significantly extends the battery life. In our experience, a battery will last only about one year if it is connected via USB at all times. However, its life can be extended to more than five years if the battery is in fact “used”. Furthermore, by being able to virtually unplug and re-plug phones, testers no longer need to get up from their desks to adjust the phone. Even more importantly, if a remote phone loses connection, testers are no longer dependent on the availability of people at the remote phone’s location, which is especially important when phones are located in a different time zone.
2018 saw a lot of big changes in regards to what testing and QA activities that customers can automate, meaning QiTASC provides increasingly robust levels of test automation test coverage and that goes far beyond test execution alone. We’ve focused on some of the most time-consuming testing activities that have remained manual activities even as other aspects of testing have become automated.
Users now have the ability to automate:
- Audio content analysis, comparison and quality assessment
- Test case scheduling and execution
- Device management and allocation for both test engineers and automated Agents
- Test reporting and defect tracking
- Speech-to-Text verification
Meanwhile, we’ll be introducing tools that automate and manage:
- Test case creation using models that represent all of a project’s test requirements
- Device integration and configuration, allowing all Android devices to be tested by intaQt
- Battery and soundcard usage
Best of all, these different features and products all integrate with each other seamlessly, which means that QiTASC customers will be able to manage, execute and analyze all of their testing activities from a single suite of tools. This reduces the redundancy that arises from using multiple test tools at different phases of the development cycle. We’re looking forward to sharing more information about our new and forthcoming products and hope you are too!
Would you like to learn more about QiTASC’s automation projects can benefit your testing activities? Contact us at firstname.lastname@example.org or visit our Contact page to speak to someone at QiTASC or arrange a demo!