Operating IoT Systems Installed in the Field — Disconnections, State Mismatch, and Remote Management

In IoT system projects, the demo usually goes well. You put a device on a desk in the office, press a button in the app, and the light turns on. Getting this far is not difficult.

The problems start after that device is installed in a basement machine room, in a store in another city, or at a site with poor connectivity.

The app says the device is on, but it is actually off. You press a button, nothing happens, and three minutes later it suddenly responds. A device has been offline for days and nobody knew. Updating the firmware means travelling to the site.

This article summarizes what is needed when you design an IoT system with the operations phase in mind. It focuses less on how to integrate an individual device and more on how to manage tens to hundreds of devices scattered across locations.

---

1. How do you know a device is alive

This is the most basic part, and the one most often implemented incorrectly.

The last communication time alone is not enough

A common implementation updates a last-communication timestamp every time the device sends data, and marks the device offline once a certain amount of time has passed.

The problem with this approach is that it cannot tell a quiet device from a dead one. A device that only sends data when its state changes will be wrongly marked offline overnight, when nothing happened.

That is why you need a structure in which the device periodically sends an "I am alive" signal. The interval is set by weighing the characteristics of the device against the cost of communication.

Define the decision criteria explicitly

These are the values that go into the online/offline decision.

If the recovery condition is a single signal, the state flaps badly. In an unstable network, when online and offline alternate every few minutes, alerts pour in and people end up turning them off. Requiring several consecutive signals is more stable.

It is simpler to run the decision as a periodic job than to evaluate it every time a signal arrives. Below is an example that marks a device as disconnected after it misses three consecutive liveness intervals. Three intervals is an illustrative value; adjust it to the communication quality of the site.

-- Assumption: PostgreSQL 9.4 or later, a batch job that runs every minute
-- device(heartbeat_interval_sec, last_seen_at, status, status_changed_at)
UPDATE device
SET status = 'OFFLINE',
    status_changed_at = now()
WHERE status = 'ONLINE'
  AND last_seen_at < now() - make_interval(secs => heartbeat_interval_sec * 3);

Thanks to the status = 'ONLINE' condition, devices that are under maintenance or unregistered are left untouched. Alerts are handled separately from this update, and are sent only for devices where the configured time has passed since status_changed_at.

If you use MQTT, use keep alive and Last Will together

MQTT has mechanisms for checking connection liveness built into the protocol. Before building your own, it is better to understand exactly how they behave and use them.

• Keep Alive — The client must send a PINGREQ within this interval even when it has no packet to send, and the broker closes the connection if it receives no packet for 1.5 times the Keep Alive value. Setting it to 0 turns this behaviour off.(MQTT 3.1.1, 3.1.2.10) In MQTT 5.0 the broker can specify a value through Server Keep Alive in CONNACK, in which case the client must use that value instead of the one it sent.(MQTT 5.0, 3.2.2.3.14)
• Last Will — If you register a will message when connecting, the broker publishes it on your behalf when the connection is closed abnormally. If the client sends DISCONNECT and closes normally, the broker discards the will without publishing it.(MQTT 3.1.1, 3.1.2.5 / 3.14.4)
• Retained messages — When a message is published with the RETAIN flag, the broker stores the last message per topic and delivers it to later subscribers. Publishing a retained message with a zero-byte payload removes the stored message.(MQTT 3.1.1, 3.3.1.3)

Combining these three lets you express online status with a single status topic per device. The device leaves online on connection, and offline as its will, both retained on the same topic. A dashboard that connects later receives the last status as soon as it subscribes.

# Assumption: Python 3, paho-mqtt 2.x, MQTT 5.0 broker, TLS (8883)
import paho.mqtt.client as mqtt
from paho.mqtt.packettypes import PacketTypes
from paho.mqtt.properties import Properties

DEVICE_ID = "dev-0001"
STATUS_TOPIC = f"devices/{DEVICE_ID}/status"

def on_connect(client, userdata, flags, reason_code, properties):
    if reason_code.is_failure:
        return
    client.publish(STATUS_TOPIC, "online", qos=1, retain=True)
    if not flags.session_present:  # Resubscribe only when the broker has no session left
        client.subscribe(f"devices/{DEVICE_ID}/commands", qos=1)

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                     client_id=DEVICE_ID, protocol=mqtt.MQTTv5)
client.tls_set()

will_properties = Properties(PacketTypes.WILLMESSAGE)
will_properties.WillDelayInterval = 30        # No will is sent if the device reconnects within 30 seconds
client.will_set(STATUS_TOPIC, "offline", qos=1, retain=True,
                properties=will_properties)

connect_properties = Properties(PacketTypes.CONNECT)
connect_properties.SessionExpiryInterval = 3600  # Keep the session for 1 hour after disconnection
client.on_connect = on_connect
client.connect("broker.example.com", 8883, keepalive=60,
               clean_start=False, properties=connect_properties)
client.loop_forever()

These are the parts that differ by version.

The sources are sections 3.1.2.11.2, 3.1.3.2.2 and 3.3.2.3.3 and Appendix C of the MQTT 5.0 specification.(MQTT 5.0) The combination of Clean Start 1 and Session Expiry Interval 0 is equivalent to Clean Session 1 in 3.1.1. Will Delay Interval reduces, at the broker level, the problem described earlier ("if the recovery condition is a single signal, the state flaps"), so it is worth turning on if you can use 5.0.

These are symptoms commonly seen in the field, and their causes.

There is one limitation you need to be aware of. Keep Alive and the will only tell you whether the connection to the broker is alive. They cannot catch a state where the process keeps its connection open but sensor reading or the control loop has stopped. So even with MQTT, it is safer to keep a separate application-level liveness signal that carries values such as the last measurement time and uptime.

Build device replacement and reinstallation into your assumptions

Field devices fail and get replaced. If the system has no defined way to handle this, data is interrupted or history disappears with every replacement.

There are two concepts to distinguish.

• Physical device — The actual unit. When it is replaced, it becomes a different entity
• Logical location — A role such as "3rd floor meeting room lighting". It stays the same when the device changes

History and statistics must be accumulated against the logical location. If they are accumulated against the device, you lose sight of that location's past data the moment the device is replaced.

These are the things to decide for the replacement procedure.

• Will the existing device be deleted, or preserved in an inactive state
• What is the procedure for attaching the new device to the same location
• Is the time of replacement recorded
• Can data from before and after the replacement be viewed as one continuous series

We recommend deactivation over deletion. Deleting a device breaks the references in the past logs it left behind. When you look up the history later, entries end up shown as "unknown device".

Divide status into three or more states

Two states, online and offline, are not enough to capture what happens on site. These are the distinctions needed in practice.

• Normal — Communication works and responses are normal
• Slow response — Communication works but command processing is slow
• Communication lost — The signal has stopped
• Unregistered — Installed but not registered in the system
• Under maintenance — Intentionally excluded

The under maintenance state is especially useful. It prevents fault alerts from continuing to arrive when a device has been turned off on purpose for equipment replacement or construction work.

---

2. There is no guarantee that a command was delivered

Control commands cross a network, so they can be lost along the way. The design has to assume this.

Separate the request from the confirmation

Some implementations switch the screen to "on" the moment the button is pressed. This looks responsive to the user, but at that point nobody has confirmed that the device actually turned on.

Dividing the life of a command into stages lets you represent it accurately.

① Request accepted   — The system receives the user's action
② Sent               — The command is sent to the device or hub
③ Receipt confirmed  — The device reports that it received the command
④ State reflected    — The change in the device's actual state is confirmed

These stages should be visible on screen. Rather than showing completion at ①, it is more accurate to switch to the final state only after ④ has been confirmed. In between, show that the command is in progress.

Decide what happens when there is no response

You need to decide what to do if no confirmation arrives within a certain time.

• Will you retry, and how many times
• Is the command safe to retry — a toggle-style command reverts to the original state when retried
• How will the user be told about the final failure
• Are failed commands recorded

The risk of toggle commands is often overlooked. Sending "flip the current state" twice puts you back where you started. It is safer to design commands so that they specify the target state, such as "turn on" or "turn off". Then the result is the same no matter how many times the command is sent.

Distinguish what each delivery quality level actually guarantees

MQTT QoS is often loosely understood as "guaranteed delivery". What it actually guarantees is narrow.

The definitions follow Section 4.3 of the MQTT 3.1.1 specification.(MQTT 3.1.1, 4.3) If you use a managed broker, check what it supports first. AWS IoT Core, for example, does not support QoS 2.(AWS IoT Core MQTT)

There are three points that are easy to miss in the design.

QoS is a per-hop guarantee. PUBACK means the broker has taken over the message. It does not mean the device received it, nor that the device executed it. The ③ receipt confirmation and ④ state reflection stages described above have to be built yourself, by having the device send a response on a separate topic.

The only retransmission the specification requires is on reconnection. The sole retransmission condition the specification defines is that, when a client reconnects with its session preserved, unacknowledged QoS 1 and 2 messages must be sent again. Version 5.0 goes as far as prohibiting retransmission at any other time.(MQTT 3.1.1, 4.4, MQTT 5.0, 4.4) The protocol does not handle the case where the connection is alive but the device does not respond, so response timeouts and retries are managed in the application.

With QoS 1, the same command can arrive twice. So each command carries a unique identifier, and the device must not execute an identifier it has already processed. This is the same principle as the idempotency key used in server-to-server calls, which we covered in detail in Where External System Integrations Fail — Principles of Interface Design.

This is the minimal form of a command message and its handling on the device side.

{
  "id": "0b6f6f0e-3c1e-4d0a-9a53-6a1d0c1f7b21",
  "target": { "power": "on" },
  "expires_at": 1791874800
}

expires_at is a Unix time in seconds, and the value is an example.

# Assumption: Python 3. In practice, store seen in a file or flash so that it survives a reboot
import json
import time

seen = {}  # command id -> processing result

def handle_command(raw, apply, now=time.time):
    command = json.loads(raw)
    if command["id"] in seen:
        return seen[command["id"]]      # Duplicate: do not execute again, just return the previous result
    if now() > command["expires_at"]:
        result = {"id": command["id"], "status": "EXPIRED"}
    else:
        apply(command["target"])        # Apply the target state
        result = {"id": command["id"], "status": "APPLIED"}
    seen[command["id"]] = result
    return result

The device publishes the return value on a response topic, and the server applies it only once even if responses with the same id arrive several times. To keep the list of processed identifiers from growing without bound, delete entries starting with those whose validity period has passed.

Define how commands to offline devices are handled

You need to decide what happens when a user tries to control a device that is offline. There are three options.

If you choose hold and deliver, you must also define a validity period. It would be a problem if a "turn on the lights" pressed three days ago ran when the device came back. You need a rule that discards held commands after a certain time.

If you implement holding with MQTT, combine session persistence with a QoS 1 subscription. While the device is disconnected, the broker queues messages in the session and delivers them on reconnection. Where you apply the validity period differs by version.

• MQTT 5.0 — If you specify Message Expiry Interval when publishing, the broker does not deliver expired messages. How long the session itself is kept is set with Session Expiry Interval
• MQTT 3.1.1 — The specification has no concept of expiry, so put an expiry time in the payload as in the example above and check it on the device. The session retention period depends on the broker configuration. In AWS IoT Core the default expiry time of an MQTT 3 persistent session is 1 hour, and when the session expires the queued messages are discarded(AWS IoT Core MQTT)

Checking the expiry time on the device assumes that the device clock is correct. For devices whose clock is hard to trust, the 5.0 approach, where the broker handles expiry, is safer. The clock problem is covered again in Section 3.

Instead of queuing held commands in a message queue, another approach is to store just one "desired state" on the server. Even if the lights were switched on and off five times, all the device needs when it returns is the last value. This approach leads into the desired/reported model in the next section.

---

3. The state on screen drifts from the actual state

A considerable share of operational inquiries come from here. "The app says it is off, but it is actually on."

The ways state diverges are known

• Someone operates a physical switch on site
• The device acts on its own schedule or a remote control
• The device returns to its default state when power is restored after an outage
• The command was delivered but the device could not execute it
• The state change notification was lost in the network

Of these, physical switch operation and power recovery are the most common. Both are state changes that the system did not take part in.

Reconcile with the actual state periodically

If you rely only on the device reporting its state changes, you miss the paths above. You need a procedure that queries the actual state at regular intervals and compares it with what the system has recorded.

These are the items to decide.

• Synchronization interval — Shorter is more accurate, but increases traffic and load
• Reference on mismatch — Do you treat the device's actual state as correct, or put it back to the state the system intended
• Whether to record — Is the fact that a mismatch was found logged in itself

The reference on mismatch must be decided carefully. Restoring the state the system intended can mean turning back on something a person on site turned off. In most cases it is safer to accept the actual state and only leave a record.

Store the desired state and the reported state separately

The desired/reported model is a way of handling the screen-versus-device mismatch structurally. The server keeps one document per device and stores the state the system wants (desired) and the state the device last reported (reported) separately. The AWS IoT Device Shadow and the Azure IoT Hub device twin are built this way, and the same structure can be applied as it is to a system you build yourself.

{
  "state": {
    "desired":  { "power": "on",  "target_temp": 24 },
    "reported": { "power": "off", "target_temp": 24 },
    "delta":    { "power": "on" }
  },
  "version": 18
}

The above is a shortened example of the AWS shadow document format. The rules of operation are as follows.

• The app writes only desired, and the device writes only reported. The screen shows reported, and displays "applying" when the two values differ. The command stages separated in Section 2 fit into a single document
• The difference (delta) includes only the items in desired whose value differs from reported or that are missing from reported. Items that exist only in reported are not included(AWS IoT Device Shadow document structure)
• The document's version increases every time it is updated. If an update request includes a version, it is processed only when it matches the latest version, and the device can ignore messages with a version lower than the one it holds. This is the mechanism for filtering out messages that arrive out of order
• The Azure twin keeps a $version that only increases for desired and for reported separately, and an ETag for the document as a whole. The size is limited to 32 KB each for desired and reported(Azure IoT Hub device twins)

The reconnection procedure is the key. Azure IoT Hub does not keep desired change notifications for devices that were disconnected. The order the documentation recommends is therefore connect, subscribe to change notifications, then retrieve the full desired document, ignoring any notification whose version is lower than or equal to that of the retrieved document. If you retrieve before subscribing, you miss changes made in between. The AWS shadow returns the full document when an empty message is published to the /get topic, so implement a single retrieval right after connecting.(AWS IoT Device Shadow MQTT topics)

These are problems that often occur with this model.

The first row is the concrete form of the "reference on mismatch" problem described earlier. Leaving desired as it is means the system keeps restoring the state it intended, while aligning desired on manual operation means accepting the actual state. Decide which way to go item by item. For example, a cooling setpoint defined by a schedule is restored, while for lighting power the on-site operation is accepted.

History does not go into this document. The document holds only the last value, so measurements and control history that need to be viewed as a time series are sent as separate messages and stored separately.

Handle the power recovery scenario separately

When the power goes out, many devices restart at the same time. Two things need to be decided here.

• Device default state — When power comes back, does the device return to on or stay off. In many cases this can be specified in the device settings
• Behaviour after recovery — Do you restore the state from before the outage, or apply the schedule for the current time

Simultaneous restart is also connected to the power peak problem covered in The Order for Cutting a Building's Electricity Bill — Savings Do Not Start Without Measurement. Recovery, too, needs a configuration that brings devices up sequentially.

Buffer data on the terminal while disconnected, then send it

If measurements and processing results are thrown away while the connection is down, the graphs have holes after recovery and settlement or statistics no longer add up. The basic structure is for the terminal to write to local storage first and delete only what has been sent successfully.

-- Assumption: SQLite on the terminal
CREATE TABLE outbox (
  seq         INTEGER PRIMARY KEY AUTOINCREMENT,
  topic       TEXT    NOT NULL,
  payload     TEXT    NOT NULL,
  measured_at INTEGER,
  sent_at     INTEGER
);

-- To send: oldest first, a fixed number at a time
SELECT seq, topic, payload FROM outbox
WHERE sent_at IS NULL ORDER BY seq LIMIT 100;

-- Storage limit: keep only the most recent 50,000 rows (example value)
DELETE FROM outbox WHERE seq <= (SELECT max(seq) FROM outbox) - 50000;

The server must store a record only once even if the same record arrives twice. This is because the terminal sends the same record again when the transmission succeeded but it did not receive the response.

-- Assumption: PostgreSQL 9.5 or later, unique constraint on telemetry (device_id, seq)
INSERT INTO telemetry (device_id, seq, measured_at, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (device_id, seq) DO NOTHING;

These are the things to decide alongside.

• Storage limit and discard order — Storage is finite. Decide whether to discard the oldest first or the least important first when it fills up. Put processing results used for settlement and periodic sensor values in separate tables with separate limits
• Transmission rate right after recovery — Uploading several days of data at once is a load in itself. Limit the number of records sent at a time, and prioritize real-time data so that events happening now are not pushed back
• Scope of the sequence number — When a terminal is replaced or its storage is reset, the sequence number restarts from 1 and collides with past data. Send an identifier generated at installation along with it and include it in the unique key
• Write endurance — Flash storage has a limited number of write cycles, so consider writing in batches rather than on every measurement

The Zero Salon KIOSK that we developed was also built so that device control and the on-screen flow continue while the server connection is down, and the results are sent again once the connection is restored. Because communication quality differed from one installation site to another, this path was not exception handling but a path used routinely.

Do not trust the terminal clock as it is

To send buffered data later, "when was this value measured" has to be accurate. Terminal clocks, however, are wrong more often than you might expect. A terminal without a battery-backed clock starts from an initial value right after power-on, and measurement begins before time synchronization finishes. If it boots with the network down, synchronization does not happen at all.

• Store two times — Keep both the measurement time stamped by the terminal and the reception time recorded by the server. Flag records where the difference is abnormally large and treat them separately in statistics
• Before synchronization, record with sequence numbers and elapsed time — Record only the time elapsed since boot and the sequence number, then back-calculate the actual time from the moment synchronization completes
• Do not use time to determine order — Determine which value is the latest by version or sequence number. Time-based comparison breaks the moment the clock goes backwards
• Be careful with validity checks — A terminal whose clock is at its initial value judges expired commands to be valid. Do not execute commands that have a validity period until synchronization has been confirmed
• Certificate validation also depends on the clock — There are cases where the TLS certificate validity check fails and the terminal cannot connect at all. Arrange the startup order so that time synchronization finishes before connection

Having the server include the server time in the messages it sends is another option. The AWS shadow puts timestamps in its responses so that even a device without a clock can calculate how old each value is.(AWS IoT Device Shadow document structure)

---

4. Unnecessary communication accumulates

As the number of field devices grows, traffic increases quickly. Each exchange looks small, but together they become server load and communication cost.

Split polling intervals by purpose

There is no reason to fetch all data at the same interval.

Where possible, event delivery is more efficient than periodic queries. Sending only when a value changes cuts traffic considerably. As covered earlier, though, a periodic signal for liveness checks is still needed separately.

Turn off polling for features that are not used

This is a type of problem we have actually run into. Terminals at sites that did not use a particular feature kept calling the query related to that feature every 5 seconds. The terminal settings had no value indicating whether the feature was in use, so the structure made conditional branching impossible in the first place.

For a single terminal this is negligible, but it is a different story when hundreds of them do the same thing.

These are the things to check at the design stage.

• Do the terminal settings have a per-feature enabled flag
• Does the polling loop check that flag before starting
• When the setting changes, is it applied to the loop that is already running
• Can the loop be stopped when it is not needed

If the setting that turns a feature on or off does not reach the terminal, the requests themselves keep coming in no matter how much you block them on the server later.

Reduce the size of the data you send as well

The amount exchanged per call is a cost just as the number of calls is. For terminals on a cellular line in particular, it translates directly into charges.

• Send only changes — Send only the items that changed rather than the full state every time
• Send in batches — Collect a certain number of sensor values and send them at once rather than sending each one individually
• Remove unnecessary fields — There are cases where values the server does not use keep being sent
• Apply compression — If the data is large, consider compressing it before transmission

With batching, first check whether a delay is acceptable. A temperature value can be a few minutes late, but an anomaly detection signal has to be sent immediately. Divide data into what is sent immediately and what is sent in batches according to its nature.

Keep retries from piling up

When the server becomes unstable for a moment, every terminal retries at the same time. Load concentrates on a server that is trying to recover, and recovery is delayed.

• Use a scheme that increases the retry interval
• Mix a random value into the interval to spread out the timing
• Offset the initial connection time slightly for each terminal

If you only increase the interval without adding a random value, terminals that were disconnected at the same moment come back at the same time. Choosing the wait time at random between 0 and the cap is simple to implement and spreads the load well.(AWS Architecture Blog, Exponential Backoff And Jitter)

# Assumption: Python 3. base of 1 second and cap of 300 seconds are example values
import random

def retry_delay(attempt, base=1.0, cap=300.0):
    return random.uniform(0, min(cap, base * 2 ** attempt))

Always set a cap. Without one, a terminal that was disconnected for a long time does not attempt to reconnect for hours even after the server has recovered. Reset the attempt count to 0 once the connection succeeds. For cases where all terminals boot at the same time, such as power recovery, add the same kind of random delay to the first connection after boot.

---

5. You must be able to manage devices without going on site

When installation points are scattered, the cost of site visits becomes a large part of operating expenses.

Information that can be checked remotely

• Online status and last communication time of terminals and devices
• Installed software and firmware versions
• Recent error history
• Current state of configuration values
• Communication quality indicators

Version information must be included. When a problem occurs, it is impossible to identify the cause if you do not know which version the terminal is running.

Remote updates need safeguards

• Staged rollout — Apply to a subset first and expand if there are no problems
• Rollback — It must be possible to return to the previous version when a problem occurs
• Scheduled application time — Keep updates from interrupting service during operating hours
• Failure handling — A terminal that loses power or its connection during an update must not end up inoperable

The last item is the most important. If a terminal does not boot because an update failed, someone has to go on site after all. The usual approach is to preserve the previous version and switch over only after the new version has been confirmed to work normally.

Make the device recover from a failed update on its own

The standard way to preserve the previous version is an A/B layout. Storage holds two copies of the execution area, and the new version is written to the one that is not currently running. Even if power is lost during the write, the running area has not been touched, so the device boots as before.

① Download    — Write the new image to the inactive area
② Verify      — Check the signature and hash. Stop here on failure
③ Switch      — Mark the new area as the next boot target and restart
④ Self-check  — After booting, the new version checks server connection, peripheral communication, and so on
⑤ Commit      — If the checks pass, mark the new area as good
                If it restarts before commit, the bootloader returns to the previous area

This structure is already implemented on each platform.

• Android — A/B updates use two slots, and if the new OS fails to boot, the bootloader falls back to the previous slot. The update is committed only once it has passed the post-boot integrity check and been marked successful(Android A/B system updates)
• ESP32 (ESP-IDF) — Uses two OTA app areas and an otadata partition that records the boot target. With the rollback option (CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE) enabled, the new app is in a pending-verification state on its first boot, and if it restarts before esp_ota_mark_app_valid_cancel_rollback() is called, it returns to the previous app on the next boot(ESP-IDF OTA)

What you put into the ④ self-check is the core of the design. If you only check that it boots, a "version that turns on but cannot connect to the server" gets committed, and that terminal cannot receive the next update either. At a minimum, check that the server connection and the path for receiving the next update are alive. Conversely, if there are too many check items and they take too long, a power loss during the check leads to an unnecessary rollback.

Keep signature verification separate from the security of the download path. TLS only protects the transport, and does not prevent cases where the distribution server is compromised or the image is altered in storage. Sign with a private key at the build stage, and have the terminal install only after verifying with its embedded public key.

# Assumption: OpenSSL 3.x. Keep the private key only on the build server and put only the public key on the terminal
openssl ecparam -name prime256v1 -genkey -noout -out ota-signing.pem
openssl ec -in ota-signing.pem -pubout -out ota-public.pem

# Build server: sign
openssl dgst -sha256 -sign ota-signing.pem -out firmware.bin.sig firmware.bin

# Terminal or pre-release check: verify. A single differing byte gives Verification failure
openssl dgst -sha256 -verify ota-public.pem -signature firmware.bin.sig firmware.bin

Even with a valid signature, there remains the attack of getting an older version with a vulnerability installed again. ESP-IDF provides an option that compares the app's security version with the chip's eFuse value and blocks the installation and booting of lower versions. However, the number of times it can be raised is limited, so raise it only when there is a security fix, not on every release.(ESP-IDF OTA) Enabling this option also restricts legitimate rollback targets, so it has to be decided together with the rollback policy.

These are problems that often occur during a rollout.

In a staged rollout, define the criteria for moving to the next stage in numbers ahead of time. Unless you define them, for example "when the commit report rate of the first group is at or above the threshold, and restart counts and error reports have not increased compared with the previous version", the decision to expand ends up being made by feel every time. Deliberately include sites with poor connectivity and older hardware in the first group.

The Zero Salon KIOSK is equipment with more than 200 units installed across cafes, shopping arcades and public facilities, so from early in development we set "every unit runs the same software" and "updates happen without a visit" as conditions. Values that differ per unit were separated into configuration, and the build was kept as one. Once versions start to diverge between units, it becomes difficult to sort the problems in the table above by cause.

Reduce the situations that require a site visit

The goal of remote management design is, in the end, to reduce the situations that can only be resolved by going on site. Listing which situations trigger a visit gives you your design priorities.

The last row is the key. A hardware failure cannot be fixed remotely, but if you know in advance you can plan the visit. Collecting signals such as a rising trend in error frequency, slowing response times and an increasing number of restarts leaves room to act before the failure.

If you manage multiple sites, bundling several tasks into a single visit is another way to reduce operating costs. Non-urgent inspection items are collected and handled together during a visit. For this, the list of outstanding tasks for each terminal has to be kept in the system.

Configuration must also be changeable remotely

Separately from software updates, there are values that change during operation.

• Server address
• Polling interval
• Whether a feature is enabled
• Items displayed on screen
• Renewal of authentication credentials

If these values are fixed in the build, changing even a single value requires a deployment. Operations are easier when you separate configuration from code and have the configuration downloaded from the server.

---

6. Keep devices of different customers from mixing

If you manage multiple buildings or business sites as SaaS, isolation matters.

Verify ownership on every request

A structure in which a device can be controlled with only its identifier is dangerous. Anyone who finds out an identifier can operate another customer's device.

These are the things to verify every time a request is processed.

• Is this token valid
• Which organization does this token belong to
• Does the target device of the request belong to that organization
• Does it have permission to perform this action

The third check is missing quite often in practice. Authentication passes, but the ownership of the target device is not verified. The list view shows only your own devices, yet detail views or control need nothing more than an identifier.

Distinguish terminal authentication from user authentication

No person logs in on a field terminal. An authentication method for the terminal itself is needed.

• Issue an individual token to each terminal. If several terminals share a common key, all of them have to be replaced when one leaks
• Restrict the token's scope of permission to the minimum. Limit it to the spaces and functions that terminal is responsible for
• Create a revocation procedure. When a terminal is lost or replaced, it must be possible to invalidate only that token
• Define a renewal cycle

If you use MQTT, ownership checks must also be enforced through the broker's topic permissions. In a structure that puts the device identifier in the topic (devices/{device ID}/commands), if a terminal can subscribe to devices/#, a single authenticated terminal can see the commands and status of every device. Restrict terminal accounts to publishing and subscribing only on topics that contain their own identifier, and allow wildcard subscriptions only for server-side accounts.

Keep access records

You must be able to trace what happened when a problem occurs. Which token sent which command to which device has to be recorded. These records are used not only for security purposes but also for identifying the cause of failures.

---

7. Frequently asked questions

Can an open-source IoT platform be used in a commercial service

Technically it is possible, and it is in fact widely done. However, using it as it is and turning it into a service are different pieces of work.

These are the points to check.

• License — Check the conditions for commercial distribution
• Multi-tenancy — Most open-source hubs are designed on the assumption of a single site. Managing multiple customers in one system requires a separate layer
• Version management — If the upstream project updates frequently, compatibility verification becomes a burden
• Incident response — When a problem occurs, you have to solve it yourself. That is, there is no support contract

In practice, the common configuration is the hub handles on-site control and the service layer is built separately. The hub absorbs the device protocols, and service elements such as accounts, permissions, history and billing are handled above it.

How do you choose the communication method

It depends on the characteristics of the device and the installation environment.

What is often missed when choosing Wi-Fi is the number of terminals that can connect per AP. Installing dozens of sensors can exceed the capacity of existing office APs. Check with the person in charge of the network before installation.

How long should data be kept

Handling it differently by purpose is the realistic approach.

• Raw time series — It accumulates at short intervals, so volume grows quickly. Keep only summary values after a certain period
• Aggregated data — Hourly, daily and monthly summaries are not much of a burden to keep long term
• Control history — If there is a dispute or audit purpose, set the retention period separately

Without a retention policy, storage keeps growing, and when you later try to delete data it is hard to judge how far back it is safe to delete. It is better to put a periodic cleanup job in place from the start.

If the system stops, does building control stop too

It depends on the design. In a structure where all control goes through an external server, a server failure leads directly to loss of control.

In a structure where the on-site hub can execute rules locally, basic operation continues even if the external connection is lost. In this case the external system takes on monitoring, configuration and history collection.

For targets where an interruption directly affects work, such as building equipment, we recommend a structure that can operate locally. This configuration, however, additionally requires synchronization design so that the local rules and the server settings do not drift apart.

---

8. Checklist before you start

State management

• Does the device send a liveness signal periodically
• Are the online/offline decision criteria and the recovery condition defined
• Can an under-maintenance state be represented
• Is there an alert criterion for sustained offline status

Command delivery

• Is the stage-by-stage status of a command shown on screen
• Are commands designed to specify the target state
• Is there a retry policy for no response
• Is the handling of commands to offline devices decided
• Do held commands have a validity period
• Do commands have a unique identifier, and does the device filter out duplicates
• Have you checked the MQTT version, supported QoS and session expiry time of the broker you use

State synchronization

• Is the actual state queried and compared periodically
• Is it decided which side is the reference on mismatch
• Are the device default state and recovery behaviour on power recovery defined
• Is the full desired state fetched again on every reconnection
• Is data from the disconnected period kept on the terminal, and does the server store duplicates only once
• Are the terminal time and the server reception time stored together

Communication efficiency

• Are query intervals differentiated by type of data
• Can polling for unused features be turned off
• Are configuration changes applied to running terminals
• Are backoff and jitter applied to retries

Remote management

• Can version information be checked remotely
• Are staged rollout and rollback possible
• Does the terminal recover when an update fails
• Is the new version committed only after it passes the self-check
• Does the terminal verify the image signature before installing
• Is configuration separated from code and changeable remotely

Security

• Is the ownership of the target device verified on every request
• Is an individual token issued to each terminal
• Are token permissions limited to the required scope
• Can individual tokens be revoked
• Are access records kept

---

Closing

In IoT projects, the gap in difficulty between building and operating is large. Integrating a device and getting it to work finishes within the expected scope, but everything that happens during the years the device runs in the field falls to people as manual work unless it has been reflected in the design.

The key is not to assume that a device always responds. Designing with disconnection, divergence and missed commands as part of the normal path greatly reduces the operational burden.

We have built systems that integrate and control the devices and sensors of buildings and facilities, and that manage field terminals remotely as well. In B-Saver, the building energy saving solution we operate ourselves, every control action is recorded through to success or failure and shown together with the device's last communication time, a mechanism for answering a "the command did not work" inquiry with a single lookup. If you are preparing a build that takes operations after installation into account, feel free to contact us.

---

Related articles

• The Order for Cutting a Building's Electricity Bill — Savings Do Not Start Without Measurement
• Why Is the Meeting Room Air Conditioner Still Running? — Designing Occupancy-Based Space Control
• Where External System Integrations Fail — Principles of Interface Design

References

• OASIS, MQTT Version 3.1.1 — https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html
• OASIS, MQTT Version 5.0 — https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html
• AWS IoT Core, MQTT — https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html
• AWS IoT Core, Device Shadow service documents — https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-document.html
• AWS IoT Core, Device Shadow MQTT topics — https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-mqtt.html
• Microsoft Learn, Understand Azure IoT Hub device twins — https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-device-twins
• Android Open Source Project, A/B (seamless) system updates — https://source.android.com/docs/core/ota/ab
• Espressif, ESP-IDF Over The Air Updates (OTA) — https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/system/ota.html
• AWS Architecture Blog, Exponential Backoff And Jitter — https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• Eclipse Paho, Python Client documentation — https://eclipse.dev/paho/files/paho.mqtt.python/html/client.html