New techniques to simplify analysis of large waveform databases

Many electronic devices and systems perform basic functions that must be performed flawlessly over long periods of time. For example, electronic power grids, communications systems, and implanted medical devices cannot handle the errors that occur even once out of millions of events. For obvious reasons, the ability to capture and isolate extremely rare anomalies is the main challenge to ensuring this level of reliability. Voltage monitoring is not effective in identifying subtle device or system problems because it is so effectively controlled that subtle differences are difficult to detect. In contrast, current waveforms contain richer information related to the operation of a device or system. However, since current waveforms can fluctuate rapidly over wide dynamic ranges, it is important to sample them at a high sampling rate to capture the full bandwidth. This can generate huge data files, since capturing data at a rate of 10MB / s over a 24-hour period creates a data file larger than 1 TB. Sifting through this huge database to locate anomalous events is clearly a daunting task. Until recently, there were no solutions that could meet the hardware requirements just described. Data recorders can capture large amounts of data, but they have a relatively low bandwidth and can easily miss components of the high-frequency signal. Conventional oscilloscopes are good at picking up high-bandwidth signals, but they have limited capacity to store data. Even high-performance oscilloscopes with large memory depths cannot capture data at a high sampling rate over time periods of hours or days. The oscilloscope current sensors also do not have enough dynamic range to capture both low-level and high-level currents. Finally, none of these hardware solutions support any efficient way to quickly analyze the data they collect and identify defects. This becomes a big data analysis problem. Machine learning is one solution to deal with these types of big data challenges. The primary technology we discovered was Deep Learning Neural Networks (DLNNs), which have been very successful in recognizing images and sound. Unfortunately, DLNN has proven to perform marginally well when applied to waveform database analysis as well as requires significant computing power. To analyze large waveform databases, Keysight researchers had to develop new machine learning technologies that were optimized for this purpose. Developed over a period of five years, this new solution includes clustering techniques, unsupervised machine learning, and proprietary database compression techniques. It can analyze terabyte waveform databases by size faster than traditional technologies while operating on a laptop computer on a platform. Figure 1: System architecture for long-term waveform analytics software. Figure 1 shows the system architecture for the long-term waveform analyzes software. It has three components, and we will discuss each component separately. The acquisition subsystem sorts out pre-received data in real time during the acquisition process. Real-time tagging is the most important module in the acquisition subsystem, as it pre-sorts incoming wave segments. Similar waveform segments are grouped together and registered as members of a tag. It is important to note that pre-screening does not have to be perfect; It just needs to contain enough information to enable post-processing analysis. The database subsystem consists of the tag database and the non-lost database. The tags database is a brief summary of our previously sorted waveform clips. Provides a quick overview of long-term registration. The non-lost database is a complete archive of the complete long-term waveform record. It allows quick query of waveform in any location of huge database by timing or waveform similarity. The tags database size ranges from one hundred to one in five hundred the size of the non-lost database. This configuration allows for great flexibility in terms of data management and analytics. The analysis subsystem has two modes of operation: quick assembly and detail assembly. Quick Compilation enables a quick overview of your entire database. Typical computing time is less than one second. However, because Fast Clustering uses pre-sorted tag information, its accuracy is limited by a tag similarity threshold. Detailed grouping provides more accurate analysis because it uses database information that is not missing. The traditional analysis software needed to recheck the non-missing database on many occasions that took many hours. With this solution, the user can enjoy fast-response interactive analyzes without re-checking the missing database. It is worth noting that this technology is not only new to the testing and measurement industry, but also to the AI ​​/ machine learning community. Keysight presented a paper on this new solution at the IEEE Big Data 2019 Conference (* 1). Researchers stated at the conference that they had never seen anything like the performance and capabilities of this solution. This technology is incorporated into a Keysight CX3300A dynamic current waveform analyzer as an available option. It combines high voltage and current measurement with long term waveform analyzes. The following example shows a commercial power line voltage monitored over a period of four days at a sample rate of 1 mA / s. The different types of waveforms are grouped by block with their combinations displayed in the collect panel. You can select one or more groups and move to their duplicates in the main launch window using the arrow keys. Although the database contained more than 18 million wave segments, marking the data allowed anomalies to be identified within seconds. For example, the below screen capture shows some major overvoltages after 2 days and 21 hours in the data log. While this case is interesting, it is fairly simple, so let’s take a look at a more challenging example. Figure 2: An overvoltage anomaly detected in a commercial power line voltage. IoT devices need to work for long hours, and any unpredictable current surges may cause an internal drop in infrared radiation and cause the system to malfunction. To verify the integrity of the device, we measured a 17-hour Bluetooth device supply current at a sampling rate of 10MSa / s. This creates a single terabyte database file. Although the normal peak current is around 25 mA, we have found very rare current spikes of up to 50 mA. These occurred only 17 times from more than 7 million recorded wave clips. Further analysis showed that in this device there are two types of asynchronous events. The 50mA spike is observed when these two events fall within a narrow timing window, and this only occurs once per 400,000 times. This type of detailed analysis can only be achieved by using the CX3300A dynamic current measurement capabilities in conjunction with the data catalog option / long-term waveform analyzes option. Figure 3: Large elevation waveforms occurring 17 times out of more than 7 million wave segments identified within 5 minutes on the IoT device. As modern hardware and systems continue to become more complex, the software tools used to evaluate them need to be improved to keep up with their pace. In situations where devices are used in mission critical systems, it is important to understand the behavior of waveforms over long periods of time. The software used to capture the data also needs to be able to help analyze the data. This article showed that through the use of new machine learning technologies developed by Keysight. It is possible to efficiently analyze large waveform databases and quickly identify anomalies in these databases. Reference
[1] Kabayashi, Goto, J-Ren, M. Ojihara, “Extending heterogeneous waveform combinations for long-term monitoring of signal acquisition, analysis, and interaction: Linking Big Data Analytics to Measurement Tool Use Pattern”, IEEE 2019 International Conference on Big Data, Los Angeles, California. United States of America. 2019, pp. 1794–1803. Written by Alan Wadsworth, Director of Business Development at Keysight Technologies for Precision and Power Products. And by Masaharu Goto, Principal Research Engineer at Keysight Technologies. .