In the first part of a series of articles, the author addressed the problem of tuning and merging messages and reducing their load in telemetry sensors.
This part is about message payload and its optimization.
There are many methods to serialize an object with Qt . In the first part of the article series, we used JSON . To do this, all information about the sensor is stored in QJsonObject , and QJsonDocument takes care of the flow of values in QByteArray.
QJsonObject jobject; jobject[ "SensorID" ] = m_id; jobject[ "AmbientTemperature" ] = m_ambientTemperature; jobject[ "ObjectTemperature" ] = m_objectTemperature; jobject[ "AccelerometerX" ] = m_accelerometerX; jobject[ "AccelerometerY" ] = m_accelerometerY; jobject[ "AccelerometerZ" ] = m_accelerometerZ; jobject[ "Altitude" ] = m_altitude; jobject[ "Light" ] = m_light; jobject[ "Humidity" ] = m_humidity; QJsonDocument doc( jobject ); return doc.toJson();
JSON has several advantages:
- text JSON is descriptive, making it human-readable;
- and the information is structured;
- the exchange of basic information is simple and clear;
- JSON allows you to expand messages with additional values;
- There are many solutions for parsing and parsing JSON in the cloud.
However, this approach has some limitations. First, creating a JSON message can be a heavy operation that takes many cycles. The benchmark of the second part of testing shows that serialization and deserialization of 10,000 messages takes about 263 ms. This may not sound like a significant number of messages, but in this context, time corresponds to energy. This can significantly affect the sensor, which is designed to be used without charging for a long time.
Another aspect is that the payload for the sensor update MQTT message is 346 bytes. Given that the sensor only sends eight takes and one limited line, this can be a potentially large computational overhead.
inside the comments of the previous post, it is recommended to use QJsonDocument : Compact (compact), which reduces the size of the payload, on average to 290 bytes
How can this be improved?
As most of you know, there is also binary JSON which can reduce readability, but all other aspects are still relevant. The main thing about running benchmarks is that a simple doc.toJson switch in doc.toBinaryData will double the speed of the test, reducing the benchmark iteration to 125 milliseconds.
After checking the load, the message size is 338 bytes, the difference is almost imperceptible. However, this may change in different scenarios, for example if more lines are added within the message.
Depending on the requirements, it is possible to add third-party projects and other features.
In case the project is !within the Qt world", the entire data stream is defined and does not change QDataStream , is a possible option.
We add support for this to the SensorInformation class, which requires the addition of two statements.
QDataStream &operator<<(QDataStream &, const SensorInformation &); QDataStream &operator>>(QDataStream &, SensorInformation &);
The implementation is quite simple, below is an example of serialization:
QDataStream &operator<<(QDataStream &out, const SensorInformation &item)
{
QDataStream::FloatingPointPrecision prev = out.floatingPointPrecision();
out.setFloatingPointPrecision(QDataStream::DoublePrecision);
out << item.m_id
<< item.m_ambientTemperature
<< item.m_objectTemperature
<< item.m_accelerometerX
<< item.m_accelerometerY
<< item.m_accelerometerZ
<< item.m_altitude
<< item.m_light
<< item.m_humidity;
out.setFloatingPointPrecision(prev);
return out;
}
When using QDataStream, the benchmark time was 26 milliseconds, which is almost 10 times faster than text JSON. Also, the average message size is only 84 bytes compared to 290. Therefore, if the above limits are acceptable, QDataStream is a good option for the task at hand.
If the project allows you to add additional third-party components, one of the most well-known serialization solutions is the Google Buffer Protocol (protobuf).
To add protobuf to our solution, we need to make a couple of changes. First, protobuf uses IDL to describe data structures or messages. Then, the development of the information sensor:
syntax = "proto2" ;
package serialtest;
message Sensor {
required string id = 1;
required double ambientTemperature = 2;
required double objectTemperature = 3;
required double accelerometerX = 4;
required double accelerometerY = 5;
required double accelerometerZ = 6;
required double altitude = 7;
required double light = 8;
required double humidity = 9;
}
To add a protobuf (protocol) code generator to a qmake project, you need to add an extra compiler step like this:
PROTO_FILE = sensor.proto
protoc.output = $${OUT_PWD}/${QMAKE_FILE_IN_BASE}.pb.cc
protoc.commands = $${PROTO_PATH}/bin/protoc -I=$$relative_path($${PWD}, $${OUT_PWD}) --cpp_out=. ${QMAKE_FILE_NAME}
protoc.variable_out = GENERATED_SOURCES
protoc.input = PROTO_FILE
QMAKE_EXTRA_COMPILERS += protoc
Then, in order to have a comparable benchmark in terms of object size, a generated structure is used as a member for the SensorInformationProto class that inherits QObject, as for the QDataStream and JSON example
class SensorInformationProto : public QObject
{
Q_OBJECT
Q_PROPERTY( double ambientTemperature READ ambientTemperature WRITE
setAmbientTemperature NOTIFY ambientTemperatureChanged)
[...]
public :
SensorInformationProto( const std::string &pr);
[...]
std::string serialize() const ;
[...]
private :
serialtest::Sensor m_protoInfo;
};
The proto Info serialization function is generated by the protocol, so the step of creating the payload to be passed is as follows:
std::string SensorInformationProto::serialize() const
{
std::string res;
m_protoInfo.SerializeToString(&res);
return res;
}
Note that compared to previous solutions, protobuf uses an STD string. This means you lose the capabilities of QString unless the string is stored as a byte array (manual conversion required). Again, this will slow down the whole process due to parsing.
From a performance standpoint, benchmark results look promising. A benchmark with 10,000 elements takes just 5ms with an average message size of 82 bytes.
As a conclusion, the following table shows the different approaches to serialization:
| | Payload size | Time (ms) |
| json (text) | 346 | 263 |
| json (binary) | 338 | 125 |
| QDataStream | 84 | 26 |
| Protobuf | 82 | 5 |
One promising alternative is CBOR, which is currently being implemented by Thiago Macieira for Qt 5.12. However, as the development process continues, it is still too early to include it in this article. From the discussions, the results look promising, with significant performance over JSON, and with all of its benefits.
The article showed different approaches to serializing data into MQTT message payloads. This can be done exclusively within Qt or with external solutions (such as protobuf). Integrating external solutions into Qt is easy.
It is worth noting that all the serializations that were done with the benchmark were done with a small amount of data in the message, and if the amount of data is larger in size, the results may differ, and different approaches may lead to better results.
