When it comes to the Internet of Things (IoT), many developers think in terms of microcontrollers, system-on-chip boards, single-board computers, sensors, and various other electronic components. While devices are undoubtedly the foundation of IoT, the core value of a connected solution lies in the data generated by these devices.
The devices layer is only the tip of the iceberg with the underlying data platform that is below the watermark tackling the heavy lifting. One of the key pillars of a robust IoT data platform is Apache Kafka, an open source software designed to handle massive amounts of data ingestion. It acts as a gateway to the data processing pipeline powered in the data center by Apache Storm, Apache Spark, and Apache Hadoopclusters.
If you are a developer considering IoT as a career option, it is time for you to start investing in Apache Kafka. This article explores the role that Apache Kafka plays in deploying a scalable IoT solution.
Kafka: A High-Performance Ingestion Layer for Sensor Data
IoT devices comprise of a variety of sensors capable of generating multiple data points, which are collected at a high frequency. A simple thermostat may generate a few bytes of data per minute while a connected car or a wind turbine generates gigabytes of data in just a few seconds. These massive data sets are ingested into the data processing pipeline for storage, transformation, processing, querying, and analysis.
Each data set consists of multiple data points representing specific metrics. For example, a connected Heating, ventilation and air conditioning (HVAC) system would report ambient temperature, desired temperature, humidity, air quality, blower speed, load, and energy consumption metrics.
In a large shopping complex, these data points are frequently collected from hundreds of HVACs. Since these devices may not be powerful enough to run the full TCP networking stack, they use protocols like Z-Wave, and ZigBee to send the data to a central gateway that is capable of aggregating the data points and ingesting them into the system.
Read the entire article at The New Stack