Medical sensors, smart vehicles, smartphones, fitness trackers, fire alarms, door locks, coffee machines, and refrigerators are just a few examples of devices that may compose an IoT ecosystem. Statista predicts that by 2025, the number of IoT-connected devices worldwide will reach 75 billion.
Depending on a product’s nature, complexity, and number of users, an IoT solution needs to maintain hundreds to hundreds of thousands of devices working simultaneously. Making sure each device is working properly and the system can handle the load are challenges for every IoT manufacturer and solution provider.
What does it take to test IoT solutions?
In some sense, IoT testing is no different than testing any web or desktop software application. To spot and reproduce an issue, you need to simulate the scenario in which it occurs on devices found within the IoT ecosystem. Given the variety of IoT devices that may be part of an ecosystem, IoT testing is more difficult than simply creating a script and running it as you would test a mobile app.
Although each type of testing is applicable to IoT solutions and comes in handy at different stages of the product life cycle, some deserve special attention when it comes to testing an IoT ecosystem. Among them are end-to-end solution testing, security testing, and performance testing. In this article, we focus on the latter.
To provide IoT product testing services, QA engineers usually spend hours planning and setting up the infrastructure. Then, it takes time to hone the testing process, report on the progress, and analyze the results. Of special concern is the process of building testing standards for end-to-end testing, as QA engineers need to develop auxiliary software solutions for load testing to emulate a large number of specific devices while preserving their essential characteristics.
Components of an IoT system that are subject to testing
A typical IoT system consists of the following components:
- End devices (sensors and other devices that are the things in IoT)
- Specialized IoT gateways, routers, or other devices serving as such (for example, smartphones)
- Data processing centers (may include centralized data storage and analytics systems)
- Additional software applications built on top of gathered data (for example, consumer mobile apps)
To make sure an IoT ecosystem works properly, we need to test all of its functional elements and the communication among them. This means that IoT testing covers:
- Functional end device firmware testing
- Communication scenarios between end and edge devices
- Functional testing of data processing centers, including testing of data gathering, aggregation, and analytics capabilities
- End-to-end software solution testing, including full-cycle user experience testing
Approaching IoT testing solutions, service providers should decide on the IoT tests to apply and prepare to combine different quality assurance types and scenarios.
How can you perform IoT testing?
IoT testing involves a full range of quality assurance services. The way engineers test the system depends on its current level of maturity, the assets available to perform testing, and the requirements formed by the product team.
Some aspects of an IoT system and some scenarios are still possible to cover with manual testing. But when it comes to evaluating a system’s performance, tests should be automated.
Manual QA for functional testing
Manual testing of an IoT solution can check if all functional requirements are satisfied.
Typical tasks of manual QA engineers on IoT projects include:
- Setting up functionality to test on real devices as well as emulators or simulators
- Running tests on both real devices and emulators or simulators
- Balancing tests on real devices with tests on emulators or simulators
Manual functional testing is still a must for IoT projects. It can be used both in the early stages of development and during further product development until the product matures. At the same time, the complex nature of IoT makes automation a necessity for the majority of solutions.
QA automation for performance testing
Automation is required when an IoT system scales. Scaling means an increasing number of devices generating an increasing amount of data. This process is usually accompanied by system degradation, various issues, and bottlenecks.
To find and reproduce issues related to system performance when devices produce and share large amounts of data, the following types of testing can be used:
- Volume testing. To conduct volume testing, we load the database with large amounts of data and watch how the system operates with it, i.e. aggregates, filters, and searches the data. This type of testing allows you to check the system for crashes and helps to spot any data loss. It’s responsible for checking and preserving data integrity.
- Load testing checks if the system can handle the given load. In terms of an IoT system, various scenarios can be covered depending on the test target. The load may be measured by the number of devices working simultaneously with the centralized data processing logic or by the number of end devices or packets a single gateway handles. The metrics for measuring load include response time and throughput rate.
- Stress testing measures how an IoT system performs when the expected load is exceeded. In performing stress tests, we aim to understand the breaking point of applications, firmware, or hardware resources and define the error rate. Stress testing also helps to detect code inefficiencies such as memory leaks or architectural limitations. With stress testing, QA engineers will understand what it takes for a system to recover from a crash.
- Spike testing verifies how an IoT system performs when the load is suddenly increased.
- Endurance testing checks if the system is able to remain stable and can handle the estimated workload for a long duration. Such tests are aimed at detecting how long the system and its components can operate in intense usage scenarios or without maintenance.
- Scalability testing measures system performance as the number of users and connected devices grows. It helps you understand the limits as traffic, data, and simultaneous operations increase and helps you predict if the system can handle certain loads. If the system breaks, there may be a need to rework its architecture or infrastructure.
- Capacity testing determines how many users and connected devices an IoT application can handle before either performance or stability becomes unacceptable. Here, our aim is to detect the throughput rate and measure the response time of the system when the number of users and connected devices grows.
Running these types of performance tests becomes crucial at some point in every IoT project’s life. At the same time, checking the performance of a complex IoT system is itself a challenge, as it involves deploying complex infrastructure and programming a number of simulators and virtual devices to mimic an IoT network.
Why we focus on performance testing
To understand the importance of performance testing for IoT projects, let’s compare what the QA process looks like without performance testing and with a complex approach to testing.
At Yalantis, we provide IoT testing services for projects of different scales and complexities. Among our clients are well-known automotive brands and consumer electronics companies, manufacturers of IoT devices, and vehicle sharing startups. Their experience has proved that performance testing is an essential component of the success of every IoT system.
Read also:How we developed an IoT application for smart home management
Embedding performance testing into the IoT development process requires a systematic approach. To make regular testing a part of your software development lifecycle (SDLC), we advocate implementing an IoT testing framework.
Internet of Things testing framework
An Internet of Things testing framework is a structure that consists of IoT testing tools, scripts, scenarios, rules, and templates needed for ensuring the quality of an IoT system.
An IoT testing framework contains guidelines that describe the process of testing performance with the help of dedicated tools. In a nutshell, implementing a performance testing framework helps mature IoT projects approach QA automation in a complex and systematic way.
Although a framework establishes a system for regular performance measurement and contributes to continuous and timely project delivery, there are cases when it’s optional and cases in which such an approach to quality assurance is a must. The latter include:
- Projects with average loads
- Projects with high peak loads (during specific hours, time-sensitive events, or other periods)
- Rapidly growing projects
- Projects with strict requirements for fault tolerance; critical systems
- Projects sensitive to response time (solutions where decision-making relies on real-time data)
Read also: Ensuring predictive maintenance for large IoT manufacturers
How to implement an Internet of Things testing framework
Quality assurance is an essential part of the SDLC. An IoT testing framework is usually implemented based on the following needs of an IoT service provider:
- Define the current load the system can handle
- Set up and measure the expected level of system performance
- Identify weak points and bottlenecks in the system
- Get human-readable reports
- Automate performance testing and conduct it regularly
A typical flow for making use of IoT testing services includes four consecutive steps.
- Collect business needs
- Create testing scenarios
- Run performance tests
- Based on the needs specified, address issues and decide on possible improvements
A typical team for performance testing an IoT project should include the following specialists:
- A business analyst who’s in charge of understanding the needs of the IoT business and assisting with defining the scope of usage scenarios as well as the context in which they are applicable, key metrics, and customer priorities
- A solution architect to set up and deploy IoT testing infrastructure that covers necessary scenarios
- A DevOps engineer to streamline the process of IoT development by dealing with complex system components under the hood (for example, implementing or improving container mechanisms and orchestration capabilities for more effective CI/CD)
- A performance analyst (QA automation engineer) who’s responsible for engineering emulators for performance testing, running the tests, and measuring target performance indicators
Depending on the project’s complexity and maturity, the team may be extended by involving a backend engineer, a project manager, and more performance analysts.
What tasks can an IoT testing framework solve?
“What our clients usually lack in their testing strategy is regular assurance and reporting. A performance testing framework can bridge these gaps and is easily aligned with the company’s business processes.”
Technically, a framework is a set of performance testing tools connected within a system to help IoT testing companies achieve the expected results, i.e. ensure stable performance of an IoT system in different scenarios.
With an IoT testing framework, QA engineers and IoT developers receive a common tool with the following capabilities:
- Setting up load profiles
- Ensuring test scalability as the system grows
- Providing real-time visual representations of test results
Let’s overview the structure of a typical Internet of Things testing framework.
IoT testing framework infrastructure
When deciding on the IoT testing infrastructure, a QA team starts by analyzing the given IoT system. Taking into account the specific requirements within the scope of performance testing, they decide on the tools to use and establish communication between them to receive the desired outcome, i.e. a detailed report of performance metrics. Although framework infrastructure differs from case to case, the aim is always to collect, analyze, and present data.
A typical process of IoT testing for performance measurement within the implemented framework includes the following steps:
- Data is collected from resources within the IoT system (devices, sensors).
- A third-party service (Telegraf) is used to collect server-side metrics like CPU temperature and load.
- Collected data is sent to a client-side app capable of analyzing and reporting performance-related metrics (for example, Grafana).
- An Internet of Things testing tool for performance measurement generates the load and collects the necessary metrics (JMeter).
- Analyzed data from both Telegraf and JMeter is retrieved and stored in a dedicated time-series database (InfluxDB).
- Data is presented using a data visualization tool (Grafana and built-in JMeter reports)
When is the right time to implement an IoT testing framework?
“Of course, when the product is at the MVP stage, deploying a full-scale performance testing infrastructure is not something IoT providers should invest in. When the product is mature enough, a need to standardize the approach to testing appears.”
“The optimal time to get started with performance testing is two to three months before the software goes to full-scale production (alpha). However, you should make it a part of your IoT development strategy and put it on your agenda when you’re approaching the development phase. At the same time, there is no good or bad time to implement a performance testing framework. You achieve different aims with it at different stages.”
Below, we highlight some real cases where implementing a QA testing framework helped our clients prevent and solve various performance issues at different stages of the product life cycle.
You may also be interested in our expert article about IoT analytics.
Testing IoT: use cases
For this section, we’ve selected stories of three different clients that benefited from ramping up their strategy by using IoT testing services. Let’s overview how implementing an IoT testing framework worked for them.
RAK Wireless
RAK Wireless is an enterprise company that provides a SaaS platform for remote fleet management. The product deals with the setup and management of large IoT networks, for which performance is crucial.
Initial context. We’ve been collaborating with RAK Wireless for two years. During this time, we’ve gradually moved from situational testing to implementing a complex performance testing approach.
When the product was in private beta, the number of system users was limited, and there was no need to get started with performance testing. In the course of development, moving to public beta and then to an alpha release, we focused on estimating the system load and understood that we were expecting an increase in the number of users and devices connected to the IoT network.
The solution’s main target customers are enterprises and managed service providers handling tens and hundreds of different IoT networks simultaneously. For such customers, performance is one of the key product requirements.
With more customers starting to use the product, the number of IoT devices was growing as well. To prevent any issues that could appear, the client considered changing the approach to IoT testing from ad hoc to a more complex one.
Challenges. The product handles the full IoT network setup process, from connecting end devices to onboarding gateways and monitoring full network operability.
- The first challenge was to carefully indicate the testing targets and plan test scenario coverage. We chose a combination of single-operation scenarios and complex usage flow scenarios to define the metrics to collect at each step.
- Once test coverage was planned, another challenge was to properly emulate a number of proprietary IoT devices of different types to make sure we were using optimal infrastructure resources for a given scenario. We ended up creating several different types of emulators. For some scenarios, a data-sending emulation was enough; others required emulating full device firmware to replicate entire virtual devices.
What we achieved with the performance testing framework. We aimed to stay ahead of the curve and detect performance issues before any product updates were released. Implementing the testing framework allowed the product team to take preventive actions and plan cloud infrastructure costs. Transparent reporting and optimal measurement points allowed us to quickly report detected issues to the proper engineering team in charge of testing firmware, cloud infrastructure, or web platform management.
Automating performance testing allowed us to achieve coherence and ensure regular compliance system checks. A long-term performance optimization strategy helped us achieve and preserve the desired response time, balance system load by verifying connectivity with device data endpoints, and ensure system reliability under normal and peak loads.
Toyota
Initial context. Our client is a member of Toyota Tsusho, a corporation focused on digital solutions development and a member of the Toyota Group.
One of the products we helped our client develop was an end-to-end B2B solution for fleet management. Vehicles were equipped with smart sensors that connected to a system to send telemetrics data using the TCP protocol.
Some of the features provided by the IoT solution included:
- Building and tracking routes for drivers
- Tracking vehicle exploitation details
- Providing real-time assistance to drivers along the route
- Providing ignition control
Challenges. Taking into account the scale of the business, the client needed to maintain a system that comprised about 1,000 end devices working flawlessly in real time.
Among the tasks we aimed to solve with the testing framework were:
- Ensuring the delay in system response to data received from devices didn’t exceed two seconds
- Preventing any breakdowns related to the performance of sensors tracking geolocation, vehicle mileage, and fuel consumption
- Establishing a reporting system that would allow for processing and storing up to 2.5 billion records over three months
What we achieved with the performance testing framework. We created a simulator to simultaneously run performance tests on the maximum number of end devices.
The framework included capacity testing and allowed results to be represented by providing useful reports that display a specified number of records.
Running volume tests helped us spot product infrastructure bottlenecks and suggest infrastructure improvements.
Miko 3
Initial context. Our client is a startup that produces hardware and software for kids, engaging them to learn and play with Miko, a custom AI-powered robot running on the Android operating system. We were dealing with the third version of the product, which quickly became extremely successful. The client was doing some manual and automated testing to detect and fix performance issues. But the lack of a systematic approach to quality assurance resulted in system breakdowns during periods of peak load.
Challenges. Issues with performance appeared after the company sold a number of robots during the winter holidays. When too many devices were connected to the network, customers started to report various issues that prevented their kids from playing with the robots. It became obvious that the system was not able to handle the load.
The client understood that they needed to adjust and expand their performance testing strategy, as the product was scaling and the load was only expected to grow further.
What we achieved with a performance testing framework. We aimed to establish a steady process of load testing for both the main use cases (such as voice recognition for Miko 3) and typical maintenance flows (such as over-the-air firmware updates).
Implementing the performance testing framework allowed us to build a system capable of collecting and analyzing detailed data about the system’s performance. Having this information, we managed to define the system's limitations and detect bottlenecks in the architecture design as well as plan architectural optimizations.
Such an approach allowed us to systematically predict system load by measuring response times and throughput rates and analyzing error messages. A stable reporting process helped to distribute responsibility among the IoT testing team and optimize development costs.
Wrapping up
In a mature multi-component IoT ecosystem, performance testing is as crucial and technically demanding as end-to-end IoT testing.
The efforts invested in setting up a comprehensive testing framework with the required IoT application testing tools will pay off in detecting performance issues and improving the user experience before you get negative feedback from customers.
Such an approach to quality assurance brings transparency to system resource use and bottlenecks, allows you to estimate the system’s effective load capacity and scalability potential, helps you optimize infrastructure and costs, and supports decisions about improving the solution architecture.
Want to approach IoT performance testing in a holistic way?
We’ll show you how
