Industrial Edge Computing: From Sensor to Cloud with Buffering and QoS

A well-designed industrial edge architecture guarantees that no plant data is lost even if the network goes down: the gateway captures signals right at the machine, stores them in a persistent local buffer (store-and-forward) and publishes them to the cloud at the MQTT QoS level each signal needs. This article follows that complete chain, from sensor to broker, with concrete criteria for buffering, quality of service, normalisation and hardware sizing.

Why plant data cannot depend on the network

In an office, a twenty-minute network outage is an annoyance. In a plant, it is twenty minutes of unrecorded production: counters that keep advancing, alarms nobody stores, batches that later cannot be traced. The underlying difference is that industrial data is a continuous stream tied to a physical process that does not stop because the WAN link fails. If the architecture sends each reading directly to the cloud, every connectivity outage becomes a permanent gap in the historian.

And outages are not a rare case. A 4G link in a metal-roofed hall, a router shared with office traffic, an operator maintenance window or a simple restart of the corporate firewall are enough to interrupt publishing for minutes or hours. The right question is not whether the network will fail, but what the system does while it fails. From that question comes the central principle of the industrial edge: capture and publication are independent processes. Capture always happens, next to the machine, with the reliability of the local network. Publication happens when the WAN allows, without hurry and without loss.

To ground the concepts we will use one example throughout the article: a machining plant with twelve CNC centres, a compressor with a variable-speed drive and a finishing line. The CNCs expose data over OPC UA, the compressor over Modbus TCP and the finishing line through digital signals hardwired to a remote I/O module. The plant wants a cloud historian and dashboards, but its link is a fibre line shared with the rest of the company. It is the typical scenario where edge ingestion makes the difference between a complete historian and one full of holes.

Reference architecture: from sensor to cloud

The complete chain has four links, and it pays to be clear about which responsibility each one carries:

The gateway is the critical link and the one that concentrates the design decisions of this guide. Downwards it speaks the machines' protocols; upwards it publishes over MQTT, the dominant protocol in IIoT precisely because it was born for unreliable networks: lightweight connections, a publish and subscribe model, and quality-of-service levels negotiated per message. How what the gateway publishes is structured (topic hierarchy, a common namespace for the whole plant) is covered in depth in the guide to the OT/IT unified namespace; here we focus on the data arriving complete and on time.

One security note before moving on: the gateway lives on the boundary between the OT network and the outside, and that position makes it a natural enforcement point for the segmentation principles of IEC 62443. The connection always goes from the inside out to the broker (the gateway initiates the TLS session), so no inbound port needs to be opened towards the plant network.

Local buffering: store-and-forward done properly

Store-and-forward means exactly what it says: the gateway stores each reading on local disk at the moment of capture and forwards it when there is connectivity. If the WAN is available, forwarding is almost immediate and the buffer barely fills. If the WAN goes down, the buffer grows; when it comes back, the gateway drains the queue in order and the cloud historian ends up complete, with every sample stamped with its original capture timestamp, not the arrival time.

Whether the buffering is real, and not a datasheet ornament, depends on four properties:

1. Persistence on disk. A RAM buffer disappears with a power cut, which is exactly the kind of incident that tends to coincide with network outages. The buffer must survive a gateway restart.
2. Timestamps at origin. Each sample is stamped when it is captured. If the historian stamps by arrival, everything that sat in the buffer appears piled up in the minute of reconnection and the time series become useless.
3. An explicit overflow policy. Every buffer is finite. You must decide in advance what happens when it fills: discard the oldest, discard the lowest priority or reduce the sampling rate. Any of the three is defensible; not having decided is not.
4. Controlled draining. When the network returns, the gateway must not dump hours of data at maximum speed: it would saturate the link and the broker. Forwarding runs at a limited rate, interleaved with real-time traffic, which takes priority.

In our machining plant, the twelve CNCs and the compressor generate around 400 signals sampled every 1 to 10 seconds. At an average size of 200 bytes per serialised message, a full weekend without connectivity (60 hours) means in the order of 4 to 5 GB of buffer. It is a five-minute calculation that decides how much storage the gateway needs; we come back to it in the sizing section.

MQTT QoS applied: which level to use and when

MQTT defines three quality-of-service levels per message, and the choice is not cosmetic: it determines what delivery guarantee exists between gateway and broker, and at what cost in traffic and state.

In industrial practice, QoS 1 is the workhorse. The possible duplication is resolved upstream with idempotency: if each message carries an origin timestamp and a signal identifier, inserting the same sample twice into the historian is a harmless operation. QoS 2 is reserved for messages with command or transaction semantics, and QoS 0 for streams where the frequency itself makes each individual sample redundant (a vibration sampled ten times per second for trending, for example).

Two protocol mechanisms complete the picture. The persistent session makes the broker remember subscriptions and retain pending QoS 1 and 2 messages while the client is disconnected, which complements (not replaces) the gateway's local buffer. And the last will lets the broker automatically publish a status message when a gateway disconnects abruptly: it is the standard way of knowing that a plant has gone silent without waiting to miss its data. On these conventions Sparkplug B is built, formalising the birth and death messages of each node (NBIRTH, NDEATH) and per-session state management. The detail of brokers, topologies and topic conventions is covered by the MQTT integration solution.

Normalisation at the edge: model before publishing

If the gateway merely copies raw registers, the plant's complexity travels intact to the cloud. Register 40012 of the compressor, the nodens=2;s=Channel1.Device3.Axis2.Load of a CNC and digital input 7 of the I/O module land in the historian as they are, and each consumer has to know what they mean, what unit they are in and with which scaling factor to read them. That knowledge then lives duplicated in every dashboard and every script, which is the worst possible place.

Normalising at the edge means the gateway translates once, at the source, from protocol coordinates to business coordinates. Concretely:

• Structured names: the signal is published as plant/machining/cnc-07/spindle-load, not as a register address. The naming hierarchy is the foundation of the unified namespace.
• Engineering units: the value travels in bar, degrees Celsius or kW, with the scale and offset already applied. A raw Modbus integer should never leave the plant network.
• Metadata and quality: each sample carries an origin timestamp and a quality flag (good, uncertain, source disconnected), so that a real zero and a zero caused by a read failure are distinguishable upstream.
• Deadband and downsampling: not every change deserves a message. Publishing by deadband (only when the value changes significantly) reduces traffic and buffer usage without losing useful information.

In the machining plant example, normalisation turns three protocols and three different vocabularies into a single coherent signal tree. When months later someone adds a thirteenth CNC from another manufacturer, the work consists of mapping its signals to the existing tree in the gateway, and no upstream consumer notices the change.

Sizing the edge gateway

Serious sizing starts with three numbers: how many signals, at what rate, and how long the buffer must survive without a network. With those, you size the three dimensions that matter:

Storage

The buffer calculation is direct: signals x rate x message size x desired autonomy, with a generous margin. For the 400 signals in the example, 5 GB covers a long weekend; a 64 GB industrial SSD gives ample autonomy and leaves room for the system and logs. The detail that gets forgotten: the physical medium. A consumer microSD card copes badly with the continuous writes of a constantly rotating buffer; for store-and-forward you want storage with wear levelling (an industrial SSD or suitably rated eMMC) and an operating temperature range matching the cabinet it will live in.

CPU and memory

The gateway's load is set not by average traffic but by the peaks: draining the buffer after a long outage, and start-up, when all the sources connect at once. Normalisation (scaling, deadband, serialisation) is light per message but multiplies with throughput. As an order-of-magnitude reference, an industrial-class gateway with a four-core CPU and 4 GB of RAM comfortably handles several thousand signals sampled at second intervals; below that, measure with a real test before committing.

Connectivity and redundancy

At least two network interfaces, to physically separate the OT network (towards the machines) from the WAN uplink. If the plant justifies it, a second WAN (typically a 4G/5G modem as backup for the fibre) drastically reduces the time the buffer spends growing. And one decision that is more architecture than hardware: in mid-size plants a gateway per area or per line usually performs better than a single central gateway, because it limits the blast radius of a failure and simplifies the sizing of each device.

Typical failures and how to avoid them

Edge projects that end badly tend to trip over a short and much-repeated list of mistakes:

• Buffer in volatile memory. The system works in every test and loses data on the first real power cut. The correct acceptance test is unplugging the gateway with the WAN down and verifying that not a single sample is missing.
• Arrival timestamps instead of origin timestamps. The historian looks complete until someone analyses an incident that happened during an outage and finds six hours of data compressed into one minute.
• An undisciplined clock. If the gateway does not synchronise via NTP (or PTP where needed), origin timestamps drift and correlating signals from two gateways becomes impossible. Time synchronisation is part of the design, not an operating system setting.
• Uniform QoS for convenience. Everything at QoS 0 loses data on every micro-outage; everything at QoS 2 doubles the traffic cost for no reason. Per-signal classification is half an hour of work done once.
• Draining without rate control. After a long outage, the gateway saturates the link forwarding the buffer and pulls down the real-time traffic of the whole plant. The forwarding rate limit must be configurable and configured.
• No monitoring of the gateway itself. The device that watches the plant also needs watching: buffer occupancy, status of source connections, last will towards the broker. A buffer growing while the WAN is healthy is a warning of a problem, not a curiosity.
• Deferred normalisation. Publishing raw with the idea of modelling later creates consumers coupled to registers and addresses. Undoing that coupling afterwards costs far more than normalising from day one.

Frequently asked questions about the industrial edge

If you are planning your plant's connectivity and want the historian not to depend on network luck, at Captia Connect we work on exactly this chain: from edge ingestion at the machine to MQTT integration with your data platform. And if the project goes beyond the edge, our industrial data platform shows where this chain fits within the complete architecture.