Google, Qt, iot, mqtt, telemetry

MQTT is an important standard for telemetry, especially in the IoT scenario.

The Qt Company is often approached by Qt customers and users to learn how to connect to various cloud providers and preferably keep the list of requirements short.

In this article, I would like to provide more information on how to create a connection simply using Qt, without any third party dependencies. The following cloud providers have been selected for this comparison:

• Amazon IoT Core
• Microsoft Azure IoT Hub
• Google Cloud IoT Core
• Alibaba Cloud IoT Platform

The final summary can be seen in the table below.

Foreword

Before going into details, I would like to focus your attention.

First, the focus is on connecting devices to the cloud. The ability to send and receive messages is the main purpose. This article will not talk about the services, features, or costs of cloud providers themselves when messages are in the cloud.

Also, the idea is to use only Qt and/or Qt MQTT to establish a connection. Most, if not all, vendors provide an SDK for either devices or monitoring applications (web and native). However, using these SDKs adds additional dependencies, resulting in higher storage and memory requirements.

The order in which providers are evaluated in this article is based on public use.

In touch

The very first steps for sending messages is to create a solution for each provider and then establish a TCP connection.

Amazon IoT Core

We assume that you have created an AWS account and an IoT Core service from the AWS console in a browser.

The control panel of this service looks like this:

The create button opens a wizard to help you set up your first device.

The only information required is the name of the device. All other elements can be left empty.

The service allows you to automatically create a certificate for subsequent connection.

Store the certificates (including the root CA) and keep them available for use by the application.

No policy is required at this time. We will deal with this at a later stage.

The last missing element that should start the implementation of the example is the name of the host to connect to. Please note that for MQTT you must use the account specific prefix. You can also find information on the AWS IOT Dashboard Settings page.

Using Qt MQTT a connection is established with these few lines:

const QString host = QStringLiteral("<your-endpoint>.amazonaws.com");

const QString rootCA = QStringLiteral("root-CA.crt");

const QString local = QStringLiteral("<device>.cert.pem");

const QString key = QStringLiteral("<device>.private.key");

QMqttClient client;

client.setKeepAlive(10000);

client.setHostname(host);

client.setPort(8883);

client.setClientId("basicPubSub");

QSslConfiguration conf;

conf.setCaCertificates(QSslCertificate::fromPath(rootCA));

conf.setLocalCertificateChain(QSslCertificate::fromPath(local));

QSslKey sslkey(readKey(key), QSsl::Rsa);
conf.setPrivateKey(sslkey);

client.connectToHostEncrypted(conf);

Several details are important for a successful connection:

• The keepalive value must be within a certain threshold. 10 seconds seems like a good indicator.
• Port 8883 is the standardized port for encrypted MQTT connections.
• ClientID must be basicPubSub . This is a valid identifier (ID) automatically generated during IoT Core instance creation.

Microsoft Azure IoT Hub

First, you need to create an account for the Azure portal. On the dashboard, you need to create a new resource “Iot Hub”.

Initially, the control panel can seem overwhelming and overwhelming as Microsoft brings many cloud services and features to the fore. Since the focus is on connecting the first device, the easiest way is to go to Shared access policies and create a new access policy with all rights enabled.

This is highly undesirable in a production environment for security reasons.

By selecting the policy you just created, you can copy the connection string.

Next, we'll use the Azure Device Explorer app. This app is perfect for testing. After launch, enter Connection string (Connection string) from the top in editing the connection test and click Update (Update) .

The management tab allows you to create new test devices specifying authentication via X509 or Security Keys. Security Keys is the pre-selected standard method we're aiming for.

Finally, Device Explorer allows you to create a SAS token, which will be required to configure the MQTT client. Token has the following form:

HostName=<yourIoTHub>.azure-devices.net;DeviceId=<yourDeviceName>;SharedAccessSignature=SharedAccessSignature sr==<yourIoTHub>.azure-devices.net%2F…..

Requires only this part for authentication:

SharedAccessSignature sr==<yourIoTHub>.azure-devices.net%2F…..

Azure IoT Hub also uses TLS to connect. To get the root CA (Root Certificate Authority), you can either clone the Azure IoT C SDK or get the DigiCert Baltimore Root Certificate manually. Neither the web interface nor Device Explorer provide it.

To establish a connection from a Qt application using Qt MQTT, the code is as follows:

const QString iotHubName = QStringLiteral("<yourIoTHub>");
const QString iotHubHostName = iotHubName + QStringLiteral(".azure-devices.net");
const QString deviceId = QStringLiteral("<yourDeviceName>");

QMqttClient client;
client.setPort(8883);
client.setHostname(iotHubHostName);
client.setClientId(deviceId);
client.setUsername(iotHubHostName + QStringLiteral("/") + deviceId + QStringLiteral("/?api-version=2018-06-30"));
client.setPassword(QLatin1String("SharedAccessSignature sr=<yourIoTHub>.azure-devices.net%2Fdevices…"));

auto caCerts = QSslCertificate::fromData(QByteArray(certificates));
QSslConfiguration sslConf;
sslConf.setCaCertificates(caCerts);

client.connectToHostEncryped(sslConf);

Google Cloud IoT Core

Once you've created an account for the Google Cloud Platform, the web interface provides a wizard to start your first project using Cloud IoT Core.

Once the project has been created, it may be difficult to find your registry. The registry stores all information about devices, communications, rules, etc.

As in Microsoft Azure, all available services are placed on the dashboard. You'll find the IoT Core item under Big Data on the left side.

After using Google Cloud Platform for a while, search will prove to be very helpful in getting to the landing page.

From the registry itself, you can now add new devices.

The interface asks you for keys/certificates for your device. But he does not have the ability to create something from the service itself. There is documentation on how to create them. And at the production stage, these steps are likely to be automated in a different way. However, there are additional steps required to get started, which can be tricky.

Once your device is in the registry, you can start with the client side implementation.

Unlike other providers, Google Cloud IoT Core does not use a device certificate when creating a connection. Instead, the private key is used to generate the password. The password itself must be generated as a JSON Web Token. While JSON Web Tokens are an open industry standard, this adds another dependency to your project. Something must be able to create these Tokens. Google provides sample code, but adaptation is required to include it in the application.

The client identifier (ID) for an MQTT connection consists of several parameters and has the following form:

projects/PROJECT_ID/locations/REGION/registries/REGISTRY_ID/devices/DEVICE_ID

Based on personal experience, case sensitivity should be considered. Everything but the Project ID retains the same capitalization as your project, registry, and device. However, the project ID will be stored in lowercase.

Considering all this, the simplest implementation for establishing a connection is as follows:

const QString rootCAPath = QStringLiteral("root_ca.pem");
const QString deviceKeyPath = QStringLiteral("rsa_private.pem");
const QString clientId = QStringLiteral("projects/PROJECT_ID/locations/REGION/registries/REGISTRY_ID/devices/DEVICE_ID");
const QString googleiotHostName = QStringLiteral("mqtt.googleapis.com");
const QString password = QByteArray(CreateJwt(deviceKeyPath, "<yourprojectID>", "RS256"););

QMqttClient client;
client.setKeepAlive(60);
client.setPort(8883);
client.setHostname(googleiotHostName);
client.setClientId(clientId);
client.setPassword(password);

QSslConfiguration sslConf;
sslConf.setCaCertificates(QSslCertificate::fromPath(rootCAPath));

client.connectToHostEncrypted(sslConf);

Alibaba Cloud IoT Platform

Alibaba Cloud IoT Platform is the only product with multiple options: Basic and Professional . At the time of this writing, the structure of this product appears to have undergone a change. But this does not affect the MQTT-related issues explored here.

After creating an account for Alibaba Cloud, the web panel allows you to create a new instance of the IoT Platform.

Once an instance is created, the wizard interface allows you to create a product and device.

Of these, you will need a couple of details to establish an MQTT connection.

• Product Key
• Product Secret
• Device Name
• Device Secret

Implementation requires a couple of additional steps. To get all MQTT specific properties, a client id, username and password are created by concatenation and signing.

This is enough to connect a QMqtt client instance.

iotx_dev_meta_info_t deviceInfo;
qstrcpy(deviceInfo.product_key, "<yourproductkey>");
qstrcpy(deviceInfo.product_secret, "<yourproductsecret>");
qstrcpy(deviceInfo.device_name, "<yourdeviceID>");
qstrcpy(deviceInfo.device_secret, "<yourdeviceSecret>");

iotx_sign_mqtt_t signInfo;
int32_t result = IOT_Sign_MQTT(IOTX_CLOUD_REGION_GERMANY, &deviceInfo, &signInfo);

QMqttClient client;
client.setKeepAlive(10000);
client.setHostname(QString::fromLocal8Bit(signInfo.hostname));
client.setPort(signInfo.port);
client.setClientId(QString::fromLocal8Bit(signInfo.clientid));
client.setUsername(QString::fromLocal8Bit(signInfo.username));
client.setPassword(QString::fromLocal8Bit(signInfo.password));

client.connectToHost();

Qt Company does not use QMqttClient::connectToHostEncrypted() like all other providers. Alibaba Cloud IoT Platform is the only provider that uses a non-TLS connection by default. It is documented that one can use one and also get the rootCA. However, the fact that this is possible is still surprising.

Standard output (limits)

So far, we have established an MQTT connection with each of the IoT providers. Each vendor uses a slightly different approach for device identification and authentication, but all of these services conform to the MQTT 3.1.1 standard.

However, developers need to be aware of certain limitations or variations of the standard in order to follow the next steps. This will be discussed next.

None of the providers have built-in support for quality-of-service (QoS) level 2. To some extent, this makes sense because the telemetry information does not require multiple steps to verify message delivery. Whether the message will be processed and validated is not of particular interest in this case, although the developer should be aware of this limitation.

To refresh our memory on terminology, let's briefly recap what saved messages and "will messages" are.

Saved posts are stored on the server for future subscribers to keep up-to-date on available topics. "Messages of will" are embedded in the connection request and will only be propagated in the event of an unexpected disconnect from the client.

Amazon IoT Core

The client ID is used to identify the device. If the second device uses the same ID during the connection attempt, then the first device will be disconnected without any notification. The second device will connect successfully, if your app code contains some sort of auto-reconnect, this may cause all devices with the same client ID to be unreachable.

Stored messages are not supported by AWS and attempting to send a message that has been left will result in the connection being closed.

AWS IoT Core supports "messages of will" within the allowed topics.

Microsoft Azure IoT Hub

The client ID is used to identify the device. The behavior of two devices with the same ID is the same as for Amazon IoT Core.

Saved messages are not supported in IoT Hub. However, the documentation says that Hub will internally add a flag to let the server know that the messages have been set as saved.

"Messages of will" are allowed and supported, subject to topic limitations, which will be discussed below.

Google Cloud IoT Core

This provider uses the client ID and password to successfully identify the device.

Messages marked as saved lose this option during delivery. According to the debug logs, they are sent as regular messages. No documentation found on whether it can work similar to Azure IoT Hub which redirects this request to an internal message queue.

"Will messages" doesn't seem to be supported. While it is possible to store a "message of will" in the connect statement, it is ignored in case of an occasional disconnect.

Alibaba Cloud IoT Platform

The triplet of client ID, username and password is used to identify the device in the product.

Both "hold flag" and "will messages" are ignored from the server side. A message with the specified retention is forwarded as a normal message and is lost after delivery. Will messages are ignored and not stored anywhere during the connection.

Available (custom) themes

MQTT uses a topic hierarchy to create a detailed context for messages. Topics are similar to the directory structure, ranging from generic to device specific. One example of a topic hierarchy would be
Sensors / Europe (Europe) / Germany (Germany) / Berlin (Berlin) / device_xyz / temperature (temperature).

Each IoT provider handles topics differently, so developers should be very careful with this section.

Amazon IoT Core

First, you need to check which themes can be used by default. On the toolbar, go to *Secure (Security)-> Policies (Policies) * and select the policy created by default. It should look like this:

AWS IoT Core defines policies in JSON format and you will find some of the previous details listed in this document. For example, the available client IDs are specified in the Connect resource. It also allows you to declare which topics are valid for publishing, subscribing, and receiving. You can have multiple policies, and they must be device-specific. As such, it allows for a granular security model where certain types of groups have different access rights.

Note that the theme description also allows wildcards. They should not be confused with wildcards in the MQTT standard. This means that you must use "*" instead of "#" to include all subtopics.

Once you've created a theme hierarchy based on your needs, the code itself is simple and looks like this:

client.publish(QStringLiteral("topic_1"), "{\"message\":\"Somecontent\"}", 1);
client.subscribe(QStringLiteral("topic_1"), 1);

Microsoft Azure IoT Hub

The IoT Hub simply acts as an interface to connect existing MQTT solutions to the Hub. The user is not allowed to specify any custom theme, nor is it allowed to enter a theme hierarchy.

The message can only be published in the following form:

const QString topic = QStringLiteral("devices/") + deviceId + QStringLiteral("/messages/events/");
client.publish(topic, "{id=123}", 1);

Subscriptions have similar restrictions.

client.subscribe(QStringLiteral("devices/") + deviceId + QStringLiteral("/messages/devicebound/#"), 1);

The subscription wildcard is used to retrieve additional information that the IoT Hub can add to the message. For example, it could be a message ID. To combine multiple properties, the subtopic itself is encoded as a URL. An example message sent from the IoT Hub includes a subject:

devices/TestDevice01/messages/devicebound/%24.mid=7493c5cc-d783-4ecd-8129-d3c87590b544&%24.to=%2Fdevices%2FTestDevice01%2Fmessages%2FdeviceBound&iothub-ack=full

Google Cloud IoT Core

By default, the MQTT client should use this topic for publishing:

/devices/<deviceID>/events

But it is also possible to add additional themes using Google Cloud Shell or other APIs.

In this case, a customCross theme was created. Additional topics are reflected as subtopics on the MQTT side, if you publish a message on this topic, it will look like this:

/devices/<deviceID>/events/customCross

For subscriptions, custom themes are not available, and there are only two available themes that a customer can subscribe to:

/devices/<deviceID>/commands/#
/devices/<deviceID>/config/

Config-messages are saved messages from the cloud. They will be sent every time the client connects to synchronize the device.

Alibaba Cloud IoT Platform

Topics can be easily managed in the "Topic Categories" tab on the product toolbar.

Each topic can be configured to receive, send, or bi-directional. In addition, a couple of additional themes are generated by default to help create a scalable framework.

Note that the subject always contains the device ID. This has implications for communication routes, as noted below.

Communication routes

Communication in the context of IoT can be divided into three categories:

1. Device to Cloud (D2C)
2. Cloud to Device (C2D)
3. Device to Device (D2D)

The first category is the most common. Devices provide status information, sensor data, or any other information. A conversation in the other direction occurs in the case of providing behavior instructions, controlling debug levels, or any general instructions.

As far as communication between devices is concerned, one needs to be more verbose in the definition in this context. A typical example comes from home automation. At a certain light intensity, the sensor distributes information, and the blinds react automatically by lowering. Here, all algorithms are processed on devices and cloud intelligence is not required. In addition, there is no need to create additional rules or filters in the cloud instance itself. Also, all tested providers can create an instance of a method that works in the cloud and then redirect the command to another device.

We already covered the D2C and C2D cases in the previous section. Once the topic hierarchy is defined, the client can publish these topics as well as subscribe to them.

To verify that the C2D connection is working, select the "Test" tab on the left side of the toolbar. The browser will show a minimal interface that allows you to send messages with the specified subject.

Also, the device-to-device (device-to-device) case is well handled by subscribing and publishing to a topic, as specified in the policy.

Microsoft Azure IoT Hub

You can send messages from your device to the cloud and vice versa. However, the user is not free to choose the theme.

For submitting, Device Explorer is a good utility, especially for testing properties of a property pack.

Device-to-device communication is not possible when using Azure IoT Hub.

Google Cloud IoT Core

It is possible to send messages from the device to the cloud, which provides additional granularity with subtopics for publication. Messages are accepted on two available topics, as described in the section above.

Because custom partitions still contain the device ID, it's not possible to use a Google Cloud IoT Core instance as a standard proxy for device-to-device (D2D) message propagation.

The device toolbar allows you to send a command as well as configuration from the cloud interface to the device itself.

Alibaba Cloud IoT Platform

Publishing and Subscribing can be done in a flexible manner using the IoT platform. (Sub)Topics can be generated to provide more structure.

To check if a message has been sent from the cloud to the device, Topic List on the device toolbar contains a dialog box.

Communication between devices is also possible. Topics for them cannot be freely defined, they must be exactly one level below.

/broadcast/<yourProductName>/

The topic at this sublevel can be chosen freely.

Recommendations and more

Amazon IoT Core

• Features of MQTT for AWS.
• Available default themes.
• Design Guide for Topic Hierarchies, also a very good reference for non-AWS projects.

Microsoft Azure IoT Hub

• Features of MQTT for Azure.

Google Cloud IoT Core

• Features of MQTT for Cloud IoT Core.
• Generation of keys for devices.
• Teams.

Alibaba Cloud IoT Platform

• Features of MQTT for IoT cloud platform.
• Connection guide.

Additional notes

MQTT version 5 (MQTT version 5) appears to be too young to be accepted by major vendors. This is very unfortunate given that the latest standard adds a couple of features that are especially useful in the IoT world. Shared subscriptions provide automatic task balancing, a new authentication command provides more flexibility in device registration, connection properties, and messages, making cloud connections more performant, easier to limit/configure, etc. But for now, we'll have to wait for adoption.

Again, I would like to emphasize that the developers did not consider any of the features that the aforementioned IoT solutions provide for processing messages after they are received. This is part of a completely different study.

In addition, the use of RPC by providers has not been considered. Some have hard-coded themes for RPC handling, like Google, distinguishing between commands and configuration. Alibaba even uses default themes to handle firmware update notifications via MQTT. TrendMicro has published a study on security issues in MQTT, and RPC is featured prominently there, a must-read for anyone setting up an MQTT architecture from scratch.

How can I check it myself?

The developers have created a sample application that allows you to connect to any of the listed cloud providers when the required information is available. The interface itself is quite simple:

Final words

For any broader IoT and cloud service provider, a telemetry-based application can be connected using MQTT (and Qt MQTT). Each of them has different options for connection details, including a standard that is fully available to developers.

The Qt Company developers are looking forward to the adoption of MQTT version 5. The AUTH command provides better integration of authentication methods, and other features such as theme aliases and properties bring additional use cases to the IoT world. In addition, a shared subscription is useful for creating relationships between data and devices. This last point may come in handy for cloud providers as their goal is to handle the load inside the cloud.