Mitsubishi Electric's High-Performance Sensor Database Performs Fast Storage and Searches of Massive Data

Mitsubishi Electric Corporation announced today it has developed a high-performance sensor database that performs fast storage, searches and aggregation of massive data collected from sensors that are expected to be used widely in the IoT (Internet of Things) era.

The database enables sensor data to be used quickly for purposes such as maintaining roads, railways and other infrastructure, monitoring factories and managing energy use in buildings and homes. It has the capacity to process up to 100 trillion data items, such as three-dimensional measurements by laser sensors on/around roads totaling 200,000 kilometers in length, or sensor data accumulated in a factory over a three-year period using 100,000 sensors that take measurements every 100 milliseconds.

Compared to existing databases, such as commonly used relational databases, Mitsubishi Electric has determined that its high-performance sensor database reduces storage space, load time, and search and aggregation time each to a range of just one-tenth to one-thousandth of current levels. This was achieved by optimizing:

• data compression, to reduce input/output data traffic by selecting compression patterns from more than 700 combinations;
• data location, by arranging compressed data in storage blocks to reduce input/output time; and
• data processing units, by processing data in cache memory where possible to enhance parallel processing

Massive data-processing performance can be improved through hardware enhancement, such as parallel or distributed processing using many servers, in-memory processing using large amounts of memory, or fast storage devices using flash memory. All of these methods, however, require very expensive hardware. Remarkably, Mitsubishi Electric’s high-performance sensor database achieves fast processing using a single server with only one or two CPUs and 4GB of main memory.

The amount of sensor data in a database generally increases with long-term use. Some trial systems start with small amounts of data for verification prior to full-scale operation using larger data. Mitsubishi Electric’s high-performance sensor database allows incremental expansion of servers as the amount of data grows. Storage capacity and performance are enhanced in the absence of data migration or changes in application software.