Data Parsing

May 14, 2024 · 3 minutes to read

Notice Fragment and Vector are data types added in release 7.3. You need to update your Gateway to 7.3 or higher to enable them.

Introduction

These features have been introduced to help manage Burst type data. This includes both FFT and Raw data. Due to the high amount of messages involved (about 200 for FFT and over 2000 for raw data), it can be beneficial to optimize their usage and reduce the likelihood of errors.

Fragment

This data type is intended to help you keep all the data when the nodes swap Gateways while they are sending the burst data to the Gateway. Usually this would cause an issue in your backend where only the first half of the data is received. Now the Gateways can be configured so they send the fragments as is no matter if they missed the beginning.

Vector

This data type is intended to reduce the amount of MQTT messages sent from the Gateway to the backend. Usually for raw data you will receive 2000 individual messages to backend in JSON (string) format. With this data type, you will receive a single message combining all the payload data that will significantly decrease the bandwidth usage. This message will have the data base64 encoded, that is to say there will be an extra step in decoding the data compared to the normal.

Legacy (Burst)

This is the old data type and the default setting. It works the same way as before with the exception of the 2 new fields described in the JSON data description called “BurstId” in the “BurstEnd” message and “DataSize” in the “BurstStart” message.

Additionally you can choose to enable base 64 encoding so the data will be in slightly smaller packages without increasing the total packet amount.

Enabling the features

You can enable the feature following the steps below:

SSH connection

You can enable the feature by creating a file called /opt/iotgw/conf.d/vector.conf" with the data:

[WpLogic3]
OutputFormat: legacy
ValuesFormat: b

You need to replace the default value “legacy” with “vector” or “fragment” depending on which feature you want to use. “ValuesFormat” affects only legacy messages. “b” is the default and “b64” will encode the data in base 64.

Treon Support

You still have questions? Our dedicated team of experts is happy to help you! Please contact Treon Support directly by e-mail.

Did you know? Treon offers Premium Support and Maintenance Packages for our customers. Get even more out of Treon and boost your sales - inquire now about features and prices!