Key takeaways:
- Mission-critical industries involve complex systems and impose demanding expectations on them. Automated software testing is essential to validate functionality and performance with confidence.
- Software test automation streamlines all phases of white-box and black-box testing. It effectively shifts the entire process of software quality assurance left.
- Artificial intelligence (AI), computer vision, and digital twins are providing smart solutions on par with human cognitive abilities for many problems that were preventing full automation in the past.
Cockpit display software for a spaceship that will travel 250 million miles in deep spacefor about half a year to reach Mars. Software for heads-up displays of supersonic aircraft where a millisecond-late response may mean the difference between life and death. Software for a self-driving car’s console. These are the kinds of software used in mission-critical industries.
The expectations for testing such software are enormously stringent. In this blog post, we explore the unique requirements and challenges that mission-critical industries impose on vendor-side and client-side quality assurance (QA) teams. We explain how software test automation and recent advances ensure QA in these sectors.
What is software test automation?
Software test automation uses specialized tools to programmatically check different aspects of software, including its functionality, performance, scalability, reliability, availability, and user experience.
Unlike manual testing, the focus is on automating all aspects of a test plan, including test creation, generating suitable input test data, automatically running the test on target platforms, and validating test results. This achieves optimum software quality against the testing effort.
What are some challenges of software test automation in mission-critical industries?
Figure 1. A model of the cockpit display of the NASA Orion spacecraft
All the mission-critical industries like aerospace, defense, automotive, and health care share some unique expectations from software quality assurance, such as:
- Need for impeccable functional correctness: The bar is set high because operational failures can lead to severe injuries and fatalities. Additionally, failures can mean significant loss of business, contractual penalties, long-running legal repercussions, and damaged reputations. Such quality levels just can’t be achieved through purely manual effort.
- Long operational lifetimes: Software and hardware systems must function reliably for several years, often under harsh operating conditions.
- Stringent regulations and standards: Software and hardware systems must comply with strict government regulations and industry-specific standards regarding functionality, quality assurance, safety, and performance.
- Effective cybersecurity and resilience: Clients in these industries have various expectations regarding resilience against cyberattacks, data confidentiality, information integrity, user privacy, and high availability.
Apart from these general concerns, each industry imposes unique requirements on software testing and automated testing.
For example, in the aerospace, defense, and space sectors, some unique needs are below:
- Due to highly classified systems, teams work and test in silos. Gaps in testing are inevitable. Therefore, client-side end-to-end qualification and acceptance are crucial.
- For systems like aircraft heads-up displays, human-machine interface (HMI) metrics like response times and cognitive load are critical. HMI and user experience (UX) experts must be involved in the testing process.
- The Department of Defense’s procurement guidelines require automated software testing.
- Cybersecurity and operational resilience are high-priority requirements.
In health care , medical software must run in stressful environments where every second counts and the possibility of human errors is high. Such software must have excellent usability and robustness.
What are some key considerations for software test automation in mission-critical sectors?
Figure 2. A mode or digital twin of an electronic medical record (EMR) system
Mission-critical sectors require software test automation as a key component of the development process and product lifecycle. Below, we analyze different angles of this paradigm.
What is a test automation framework?
A test automation framework provides the building blocks required to automate software testing. Depending on the abstraction level that the framework is targeting, these blocks may include:
- source-code level constructs like assertions and mocks to test each code module in isolation
- integration-level capabilities like capturing application programming interface (API) requests and replaying them for API testing
- platform-specific capabilities, like looking up document object model (DOM) elements for testing web applications
- end-user point-of-view capabilities such as image recognition to identify user interface (UI) elements
How does software test automation influence different types of white-box testing?
Figure 3. Various types/steps of testing for mission-critical systems
In white-box testing, automation frameworks can directly access and exercise the software’s source code. That means most of this testing is done only on the vendor’s side.
The specific ways automation improves different testing types are covered below.
Unit testing
For automated unit testing, 100% test coverage and repeatability are the primary goals. Repeatability ensures that unit-level regression bugs in existing code are not introduced by new code changes.
A common source of software failures is code that cannot handle all probable combinations of input values and events. This is a particularly dangerous problem in aerospace and defense, where phenomena like cosmic rays can unexpectedly change bits in underlying electronic circuits.
So, a critical goal of automated unit testing is to generate many inputs to throw at every unit. These include realistic values automatically generated from real-world data and random values. This strengthens every unit and shields cross-unit code flows against unknown and unexpected inputs. Test automation frameworks should be good at such fuzzing.
Integration testing
Integration testing checks the interworking of software components and dependencies for functional correctness. Note that mission-critical sectors involve software components and complex hardware like avionics, spaceship instrument panels, and imaging machines.
Automated integration testing ensures that multiple components work together harmoniously with the hardware to provide correct functionality with high reliability and availability.
White-box regression testing
Older integration test suites become future regression tests whenever new features are added. Regression testing gives teams the confidence that their changes did not inadvertently break any existing functionality.
Automated regression testing is the bedrock on which software guarantees can be confidently issued in mission-critical industries. Clients must ensure that older features are working correctly alongside new ones. The only way to objectively prove this is by relying on automated regression test suites with high reusability.
API testing
Though API testing is part of integration testing, it deserves special mention because most software nowadays, even in mission-critical industries, extensively uses microservices and cloud services through APIs.
To implement its responsibilities, a module may be an API provider that exposes interfaces to other components, a client that consumes APIs of other internal or external components, or both.
Automated API testing checks both types of interactions as follows:
- API providers: API providers are particularly vulnerable to cyberattacks through malicious inputs that compromise system integrity, confidentiality, or user privacy. Automation testing helps them handle all combinations of input values from consumers.
- API clients: Complex systems can send a large variety of outputs and errors from their API endpoints. Automated tests generated based on an API’s documentation or specification ensure a client can handle all probable results and errors.
Cybersecurity testing
Software test automation can ensure that cybersecurity and resilience are embedded into every component through techniques such as:
- generating all probable combinations of well-formed and malformed input values
- checking the effects of malicious inputs
- triggering automated vulnerability analyses while running functional tests
How does software test automation influence different types of black-box testing?
Black-box testing verifies all aspects of software — functionality, performance, availability, reliability, cybersecurity, user experience, and more — from the perspective of end users and production use. This is done without using the source code in any way.
In mission-critical industries, noninvasive testing is far more rigorous and extensive. Both the vendor-side and client-side testing teams conduct tests independently. Even independent verification and validation (IV&V) testing uses third parties to qualify separately and sometimes certify the software.
The sections below describe the automation of different types of black-box testing.
Acceptance testing
Acceptance testing is conducted by clients to check that all aspects of the software conform to their requirements and expectations.
It’s generally done manually by subject matter experts through exploratory testing and end-to-end testing. However, some testing may be partially automated, especially steps that involve repetitive procedures, API interactions, comparing offerings from multiple vendors, or data in electronic formats meant for other systems.
Performance testing
Performance testing is another area where software test automation adds tremendous efficiency. The initial test environment, the steps, and the measurements in each run must be exactly the same.
Repeatability and consistency among runs are essential for reliable performance metrics. Only software test automation can ensure such consistent test setups. Using technologies like Docker, automation can exercise the software multiple times through a deterministic sequence of steps in a controlled environment to collect the required metrics.
User interface (UI) and UX testing
Figure 4. A Gherkin feature file describes application behavior using plain language
UX and UI testing are critical for cockpit displays, instrument panels, and medical devices. To optimize usability, they test HMI and usability metrics like task success rates, cognitive load metrics, response times, and time taken for tasks. Eye tracking is also used.
Techniques like behavior-driven development (BDD) and keyword-based tests are often used to automate such tests. For example, a Gherkin feature file describes a feature’s behavior using plain text that can be written by nontechnical domain experts at the requirements stage and then tracked throughout the software development cycle. There are even tools to visualize them and generate automated UI tests.
What are some advances in software test automation?
Modern techniques to streamline automated software testing include model-based automation and artificial intelligence (AI). We look at relevant advances using these.
Model-based automation using digital twins
Figure 5. A digital model of an automotive display with screens, actions, and flows
Although automated testing may be ideal for testing some critical scenarios, it may not be practical, especially for the complex systems used in mission-critical sectors. For example, the required hardware may not be ready for automated end-to-end testing.
For such problems, modern tools allow digital twins to be constructed. They are accurate data-driven simulated models of the real system in functionality, visual features, user experience, performance, and other relevant aspects.
For example, most end-user software consists of many screens, actions, and navigation flows. A digital twin can be constructed by capturing the screens and actions as a user navigates some of them manually. The digital twin can then generate these partial captures to create a complete navigation and interaction model of the system. Later, playback of every recorded flow is possible.
Such models also enable low-code automated checking by nontechnical end users, reducing the load on testers to write and maintain elaborate test scripts for every scenario.
AI and machine learning
Figure 6. Intelligent UI testing using AI and computer vision
Advances in AI and machine learning open up several possibilities like below:
- Using computer vision for UI testing: Image recognition and text recognition can be used to test UI elements and visual outcomes at higher abstraction levels than commonly used tools like Selenium allow. Instead of drilling down a DOM or UI component hierarchy, the test can simply specify that a certain UI element must be present at a specified location for a test to be successful. Describing expectations this way makes automated tests less brittle against UI changes and easier to maintain.
- Generating automated tests: Generative AI, such as large language models, can generate test cases and code based on software documentation and API specifications.
Software engineering methodologies like DevOps, DevSecOps, Agile, continuous testing, and continuous integration/continuous deployment (CI/CD) are increasingly being adopted even in mission-critical industries.
For example, software test automation tools can be integrated into common CI/CD pipelines as follows:
- Run anywhere: The system under test can be launched on all target platforms. Cross-platform applications can be launched on different operating systems like Windows, Linux, Android, or iOS. Web applications can be launched on multiple browsers to test cross-browser compatibility. Mobile applications can be launched on multiple device versions to check compatibility.
- Trigger test runs: Software like Jenkins can launch automated test runs after each code commit.
- Automate UI tests: Using AI and computer vision techniques like image recognition, UI testing can also be automated to a great extent.
Keysight software test automation solutions
Figure 7. Automotive HMI testing using Eggplant
Keysight Eggplant Test and Eggplant Performance are purpose-built for automated functional, full-stack, and performance testing in mission-critical industries.
Eggplant dramatically streamlines your testing cost and time:
- Set up time-consuming test cases 50% faster.
- Achieve 3x quicker test execution.
- Automate 75% of regression tests.
Our experts will work closely with your team members to train them and integrate Eggplant into your existing platforms and workflows. Contact us!