An industrial historian is a time-series database specialised in process telemetry: it stores every sample as a tuple of tag, value, timestamp and quality, sustains continuous writes from tens of thousands of signals and answers time-range queries over years of compressed history.
What it actually stores
A historian record has four fields and no more: tag, value, timestamp and quality. The tag identifies the signal; the value may be analogue, digital or, less often, a string; the timestamp should ideally be the one from the source rather than from arrival; and quality is the flag inherited from the OPC model (good, bad, uncertain, with substatuses) that says whether the number can be trusted. That fourth field is the most ignored and the most damaging when ignored: a bad-quality value is not the same as a gap, and treating it as good contaminates every average computed downstream.
Each tag carries a configuration record around it: description, engineering units, data type, range, interpolation mode (stepped for digital and state signals, linear for analogue ones) and its compression parameters. The write pattern is continuous insertion with virtually no updates: an archived value is almost never corrected. Late-arriving data, typical when an industrial gateway flushes its local buffer after a link outage, is accepted but is the expensive operation of the system, because it forces reopening an archive that was already closed and compressed.
Sampling rates, volume and retention
Sampling rate follows the physics of the process, not habit. Slow process variables such as kiln temperatures, tank levels or line pressures are amply covered at one second. Machine state signals and piece counters are recorded on change rather than on a period. Energy consumption is logged on quarter-hourly or hourly periods aligned with the billing interval, because instantaneous averages cannot be reconciled against an invoice. At the opposite end sit fast phenomena: a vibration waveform or an electrical harmonic is sampled in kilohertz and does not belong in a process historian; what gets archived there are the derived indicators (RMS values, band amplitudes) computed in the measuring device or in the edge computing layer.
Volume is calculated, not eyeballed. Ten thousand tags sampled every second are 864 million samples per day; the same set at one minute is 14.4 million. That factor of sixty is why sampling rate is argued signal by signal. Retention is organised in tiers: fine detail for as long as someone will actually look at it to diagnose, and lower-resolution aggregates for the years after. Before fixing the horizon it is worth asking which cycles must remain comparable: a product campaign, a full season, a change of raw material supplier.
Compression: deadband and swinging door
A historian does not keep every sample it receives, and that is both its greatest virtue and its greatest risk. Reduction happens in two stages. The first is the deadband or exception filter, applied at the collector: a value is only sent if it has moved more than a threshold from the last one sent. The second is compression proper in the archive, usually through the algorithm known as swinging door: instead of deciding sample by sample, the engine checks whether a straight line drawn between the last archived point and the current candidate keeps every intermediate sample within an allowed deviation; while the line covers them, nothing is archived, and as soon as one falls outside, the previous point is archived and a new segment opens. Two time guards sit on top: a maximum time that forces an archive even on a flat signal, and a minimum time that prevents bursts.
What is lost under that scheme deserves to be written down. First, the shape of the curve between two archived points does not exist: it is reconstructed by interpolation, and any variation that stayed inside the band is irreversibly gone. Second, short spikes smaller than the threshold were never recorded, which is exactly the detail wanted when a quality defect has to be explained. Third, the resulting series is irregular in time, so an arithmetic mean over stored points is biased towards the busiest stretches: the correct aggregate over a compressed series is the time-weighted mean. Fourth, thresholds usually stay at whatever default the installer set, applied uniformly to signals with very different dynamics. The practical rule is that the deadband is an engineering decision with no way back: what was not recorded cannot be recovered.
Why a conventional relational database is not enough
A generic three-column relational schema (tag, instant, value) works until it does not, and it stops working by accumulation. Every sample occupies a row with header and index overhead often larger than the useful data itself. Continuous insertion requires maintaining tree indexes that fragment and make writes progressively more expensive as the table grows. Age-based deletion, a daily chore in this domain, becomes a mass row operation. And the typical query, a list of tags between two instants aggregated by hour, scans billions of rows with no structure exploiting the fact that data arrives ordered in time.
On top of that sits what a generic engine cannot do and the domain needs daily: return an interpolated series at a fixed interval from values recorded by exception, distinguish stepped from linear interpolation according to signal type, compute time-weighted averages, totalise a flow or a power by integration, or apply different retention policies per resolution. The real boundary is therefore not “relational versus historian”: a relational engine with a time-series extension (time partitioning, columnar compression, continuous aggregates) covers the pattern well. The boundary is between a generic row-oriented schema and an engine that assumes time is the primary key.
Historian, SCADA database and data lakehouse
In a typical Spanish plant what turns up is not a historian but its substitutes: SCADA trends with a few weeks of retention, the internal logger inside a power analyser, the files one machine vendor exports, and a collection of spreadsheets with manual readings. Each island keeps its own horizon and its own naming, which is why the question that starts the project is almost never “we need a historian” but “why can we not compare this line’s consumption against last year’s”.
A SCADA database and a historian are alike in that both store trends, and differ in everything else. The SCADA store exists to give context to the operator screen: short retention, a tag subset sized for supervision, a proprietary schema coupled to the product, and a server that is critical control room equipment. A historian is a dedicated system, with years of horizon, domain-specific retrieval functions and the capacity to absorb queries without putting supervision at risk. Hence the usual pattern: the historian is fed from the SCADA, or directly from the PLCs, rather than replacing its function.
With a data lake or a data lakehouse the relationship is complementary. A historian enforces schema on write: every value arrives attached to a known tag, and that upfront commitment is what lets it answer in milliseconds the question operations actually asks: give me this variable between these two hours. A lakehouse does the opposite: it stores columnar files over cheap object storage, with a table format adding transactionality and versions, decides much of the schema on read, and is built to scan large volumes and join separate domains (production with quality, with energy, with the ERP) in queries that take seconds or minutes and disturb nobody. Process questions about a specific signal go to the historian; aggregate, cross-domain analytics go to the lakehouse. The sensible architecture feeds both from the same capture stream and avoids the classic mistake of turning the historian into a corporate warehouse or the lake into a trend viewer; the full division of roles is in the industrial data platform guide.
Getting data in and out towards the data layer
Ingestion sets the ceiling on quality. Data arrives through collectors from the SCADA, over OPC-UA against the controllers, or by subscribing to an MQTT broker, almost always via an industrial gateway that timestamps next to the machine and buffers locally if the link drops. The latency of that chain adds up the source poll cycle, the exception threshold and the publication period; for trends and shift indicators it is more than enough, and to correlate events at millisecond resolution the value must be stamped at source with clocks synchronised over NTP. None of these operations touches control logic: the historian lives on the information systems side and capture is passive, with load bounded by grouping signals and sizing the period.
Addressing is the tag name, and that is a problem. The real semantic model of a historian is its tag list, inherited from instrumentation and from the integrator’s judgement: cryptic names, inconsistent across equivalent lines, with no notion of work order, batch, product or shift. Adding that context is not the historian’s job but that of the data layer around it, and it is what turns a series of numbers into something that answers business questions. A Unified Namespace is one of the usual ways to settle that naming before the data is archived.
Retrieval is by tag and range. Queries go through a SQL interface, an API or the product SDK, and one decision silently changes the results: asking for values as archived or interpolated at a regular interval. The same shift can yield different averages depending on the mode, so retrieval and aggregation criteria are fixed before any indicator is computed, not after arguing why two reports disagree. On load, a bulk export competes with interactive queries: large extractions are scheduled off-peak or run against a replica, always with read-only credentials.
How it is used: queries, aggregates and models
The most immediate use is direct query: reconstructing the hours before a breakdown, comparing two campaigns of the same product, verifying with data whether a modification improved anything. Above that sits aggregate computation, which in process series has its own rules: time-weighted rather than arithmetic averages, maxima and minima with their instant, accumulated time in each state, and totalisation by integration when the signal is a flow or a power and what matters is volume or energy. Those aggregates are the raw material of plant indicators, OEE included, provided the context the historian does not store is added to them.
Feeding analytics or AI models takes one further step. A model needs a regular time grid with every signal aligned on it, so the compressed series must be resampled, deciding per signal whether interpolation is stepped or linear; it needs explicit handling of gaps and of bad-quality samples; and it needs event labels (failures, stoppages, defects, batches) that do not live in the historian but in the CMMS, the MES or the ERP. There is also a limit worth checking before investing in modelling: if the phenomenon to be anticipated is smaller in amplitude than the deadband the signal was archived with, no model will find it, because that variation is not in the data. With that caveat, history is the base on which anomaly detection and predictive maintenance are built.
Related terms
The historian sits downstream of capture: it is fed from the SCADA and the PLCs over OPC-UA or by subscribing to MQTT, and upstream it hands its series to the data lakehouse for cross-domain analytics. Building that path end to end is the scope of OT/IT integration, and exploiting the resulting history is the scope of operational analytics.