DADOS Release Package

The zip containing this README is the DADOS Release Package. It provides everything needed to run the full DADOS pipeline using Docker Compose, including DADOSFlow, Lightning+, Persistence, and the Admin service.

The default docker-compose.yml runs all services on a single machine. For multi-machine deployments, see Multi-Machine Deployment below.

Requirements

Docker (keep Docker Desktop up to date; older versions may fail at container creation)

Quick Start

See QUICKSTART.md for step-by-step instructions. The short version:

./load-images.sh
docker compose up -d

Then open http://localhost:8443 to complete initial setup and enter your licenses before using Lightning+.

Package Contents

DADOS_Release_Package/
├── docker-compose.yml          # Start/stop all services (single machine)
├── compose/                    # Individual service compose files (multi-machine use)
│   ├── admin.yml
│   ├── dadosflow.yml
│   ├── lightning.yml
│   └── persistence.yml
├── config/                     # Configuration files (edit before starting)
│   ├── dadosflow.toml          # DADOSFlow: MQTT brokers, ingress, and egress addresses
│   ├── lightning.toml          # Lightning+: ports, windows, auth (mounted into container)
│   ├── persistence.toml        # Persistence: IPC listen address and storage options
│   └── flow/                   # Per-ingress hierarchy spec files
│       └── enterprise-b.json   # Example hierarchy mapping
├── images/                     # Docker image archives (loaded by load-images.sh)
│   ├── admin.tar.gz
│   ├── dados-lightning-plus.tar
│   ├── dadosflow.tar.gz
│   └── persistence.tar.gz
├── load-images.sh              # Loads all Docker images into Docker
├── README.md                   # This file
└── README.html                 # Rendered version of this README

Persistent data (admin credentials, license keys, Parquet files, Lightning+ assets) is stored in Docker-managed named volumes — not in this directory. See Persistent Data (Docker Volumes) below.

Starting and Stopping

Load Docker images (first time only)

./load-images.sh

Start all services

docker compose up -d

Stop all services

docker compose down

Stopping does not delete containers, images, or persistent data. Run docker compose up -d again to restart.

Remove containers, images, and volumes

WARNING: this will reset everything except your master key. Including existing admin account and stored licences.

docker compose down --rmi all -v

Removes containers and images but leaves persistent data volumes intact. To also remove volumes, see Persistent Data (Docker Volumes).

Initial Setup

After starting, open the Admin panel to complete initial configuration and activate your licenses before services will operate:

http://localhost:8443

Once licensing is configured, Lightning+ is accessible at:

http://localhost:5051 — UI and API
http://localhost:5051/uns-viewer/ — UNS Viewer (browser-based live data inspector)
http://localhost:5051/flow/ — Flow Monitor (browser-based pipeline performance dashboard)

Services and Ports

Service	Container	Ports
Admin	`DADOS-Admin`	`8443`
DADOSFlow	`DADOS-Flow`	`9876` (TUI metrics), `9877` (Flow Monitor WebSocket), `9878` (Prometheus metrics)
Lightning+	`DADOS-Lightning`	`5051` (UI / API), `50052` (gRPC), `50054` (Arrow IPC)
Persistence	`DADOS-Persistence`	`50053` (Arrow IPC)

If any of these ports are already in use, container startup may fail.

Configuration

DADOSFlow — `config/dadosflow.toml`

This file controls DADOSFlow’s MQTT inputs (brokers and ingress) and where it sends data (Lightning+ and Persistence).

MQTT Brokers

Brokers are defined as named [[broker]] entries and referenced by ingress sources.

[[broker]]
name = "local"
host = "host.docker.internal"
port = 1883

[[broker]]
name = "enterprise-b"
host = "broker.enterprise-b.com"
port = 1883
username = "admin"
password = "secret"

MQTT Ingress

Each [[ingress]] entry subscribes to a broker under a namespace.

[[ingress]]
namespace = "demo"
broker = "local"
client_id = "dadosflow-client-1"
topic = "spBv1.0/#"

Optional ingress fields:

shared_count — number of shared subscription consumers (load balancing across multiple DADOSFlow instances)
hierarchy_spec_file — path to a JSON file (inside the container) that defines a hierarchy mapping for this ingress; place spec files in config/flow/ (mounted to /app/config/ in the container)

Namespaces

Each ingress belongs to a namespace. Namespaces partition data by source or tenant and are forwarded as metadata to Lightning+ and Persistence.

Multiple ingress entries can share the same namespace.
If namespace is omitted, DADOSFlow uses default.

Lightning+ Address

This tells DADOSFlow where to send Arrow data for live windowing.

[lightning]
addr = "host.docker.internal"   # hostname or IP of the Lightning+ host
port = 50054

In a single-machine setup this defaults to host.docker.internal. For multi-machine deployments, set addr to the IP or hostname of the machine running Lightning+.

Persistence Address

This tells DADOSFlow where to send Arrow data for Parquet persistence.

[persistence]
addr = "host.docker.internal"   # hostname or IP of the Persistence host
port = 50053
enabled = true

Set enabled = false to disable persistence entirely. For multi-machine deployments, set addr to the IP or hostname of the machine running Persistence.

Lightning+ — `config/lightning.toml`

Controls Lightning+ application ports, window-time settings, and authentication behaviour. The file is mounted read-only at /app/config/lightning.toml inside the DADOS-Lightning container, overriding the image default. Directory paths (vega, uns, flow, etc.) are set by the container entrypoint and do not need to be changed here.

should_authenticate = true
install_vss_plugin = false

windows = [
  { window_seconds = 60, description = "1-minute window", enabled = true },
]

[application]
host = "[::]"
http_port = 5051
flight_port = 50051
flight_sql_port = 50052
ipc_port = 50054

Admin-pushed config updates are persisted to this same file when using a licensed deployment.

Persistence — `config/persistence.toml`

Controls the IPC listen address and storage output options.

[ipc]
addr = "[::]"   # listen on all interfaces
port = 50053

[storage]
partitioning = "none"   # "none", "hourly", "daily", or "monthly"
# export_at_rows = 1000
# export_at_time_ms = 60000

Data is written to the dados-storage-data Docker volume as Parquet files (see Persistent Data (Docker Volumes)). To write to S3 or an S3-compatible store (e.g. MinIO), uncomment and configure the [aws] section:

[aws]
access_key_id = "your-access-key"
secret_access_key = "your-secret-key"
region = "us-east-1"
s3_bucket = "your-bucket"
s3_endpoint = "http://your-endpoint:9000"  # omit for AWS S3

Persistent Data (Docker Volumes)

All persistent data is stored in Docker-managed named volumes. These survive container restarts, image updates, and re-downloading the deployment package to a new location.

Volume	Service	Contents
`dados-admin-data`	`DADOS-Admin`	Encrypted license keys and user credentials
`dados-storage-data`	`DADOS-Persistence`	Parquet files written by Persistence
`dados-lightning-assets`	`DADOS-Lightning`	Lightning+ persistent assets

Master Encryption Key

The Admin service encrypts license keys and local admin user credentials with a master key. Sensor data, Parquet files, and operational telemetry are not encrypted by this key.

Without a key file, the compiled-in default dev key is used. This key may change between releases — updating to a new image version without setting your own key means licenses and admin accounts will need to be re-entered after the update.

The key is stored as a plain file and mounted into the container via Docker’s secrets mechanism — it is not visible via docker inspect.

Generate a key file (run once before first start):

mkdir -p ~/.config/dados && openssl rand -hex 32 > ~/.config/dados/admin_master_key && chmod 600 ~/.config/dados/admin_master_key

The key is stored in ~/.config/dados/ — outside the release folder — so it survives downloading and extracting a new release package. The docker-compose.yml is already configured to read from that location; no other changes are needed.

Rules: - Run the command above before first start. - Never replace or delete ~/.config/dados/admin_master_key after encrypted data exists — doing so requires re-entering all licenses and admin credentials. - Recovery is straightforward: the admin server bootstraps a new owner account automatically; licenses just need to be re-entered. - Key rotation is a planned upcoming feature — see Contact for support.

The .salt file: On first run the Admin service generates a random 16-byte salt and stores it as dados-admin-data:/app/data/.salt. This file must not be deleted — losing it has the same effect as changing the master key.

Inspecting volumes

docker volume ls                           # list all Docker volumes
docker volume inspect dados-admin-data     # show location and metadata for a volume

On Linux, volume data lives under /var/lib/docker/volumes/<name>/_data/. On macOS and Windows, Docker Desktop runs a Linux VM — use docker volume inspect to find the path, or use docker cp to extract files from a running or stopped container.

Accessing Parquet files in `dados-storage-data`

Persistence writes Parquet files into the dados-storage-data volume. To find where that volume is on disk:

Linux:

docker volume inspect dados-storage-data --format '{{ .Mountpoint }}'

This prints the full path directly, e.g. /var/lib/docker/volumes/dados-storage-data/_data.

macOS / Windows (Docker Desktop):

Docker Desktop runs containers inside a Linux VM, so the volume path from docker volume inspect is a path inside that VM — not directly accessible from your host filesystem. The easiest way to get files out is docker cp:

docker cp DADOS-Persistence:/app/storage/. ./parquet-export/

This copies the full contents of the storage directory to a local parquet-export/ folder. The container does not need to be running — docker cp works on stopped containers as well.

Removing volumes

docker compose down does not remove volumes. To remove all volumes for the stack:

docker compose down -v

To remove a single volume by name:

docker volume rm dados-admin-data
docker volume rm dados-storage-data
docker volume rm dados-lightning-assets

Warning: Removing a volume permanently deletes all data inside it, including license keys, user accounts, and stored Parquet files. Back up any data you need before removing.

Multi-Machine Deployment

The default docker-compose.yml runs all services on a single host. To distribute services across multiple machines:

1. Transfer the package to each machine

Copy the full package directory to each machine that will run part of the stack. Run ./load-images.sh on each machine to load the images.

2. Update service addresses in `config/dadosflow.toml`

Set the [lightning] and [persistence] addresses to the IPs or hostnames of the machines running those services:

[lightning]
addr = "192.168.1.10"   # machine running Lightning+
port = 50054

[persistence]
addr = "192.168.1.11"   # machine running Persistence
port = 50053

3. Update Admin host for each service

Each service must be able to reach the Admin service for licensing. Update the environment variables in the relevant compose file for each service:

Service compose file	Environment variable	Default
`compose/dadosflow.yml`	`DADOSFLOW_ADMIN_HOST`	`host.docker.internal`
`compose/lightning.yml`	`LIGHTNING_ADMIN_HOST`	`host.docker.internal`
`compose/persistence.yml`	`PERSISTENCE_ADMIN_HOST`	`host.docker.internal`

Set these to the IP or hostname of the machine running DADOS-Admin.

4. Use the individual compose files

The compose/ directory contains a separate compose file for each service. On each machine, start only the services that belong there:

# Example: run only DADOSFlow on this machine
docker compose -f compose/dadosflow.yml up -d

# Example: run only Lightning+ on this machine
docker compose -f compose/lightning.yml up -d

Each compose file in compose/ mounts paths relative to the parent directory (..), so it should be run from the package root directory.

5. Ensure ports are reachable

Make sure firewalls and network rules allow traffic on the relevant ports between machines:

8443 — Admin UI
8444 — Admin service API (all services → Admin)
50054 — Arrow IPC (DADOSFlow → Lightning+)
50053 — Arrow IPC (DADOSFlow → Persistence)
5051 — Lightning+ UI/API (clients → Lightning+)
9877 — Flow Monitor WebSocket (browser → DADOSFlow)
1883 — MQTT (DADOSFlow → your broker)

The Data Path

MQTT Source — An external MQTT broker publishes Sparkplug B messages.
DADOSFlow — Subscribes to the broker, decodes Sparkplug B messages into Arrow tables, and forwards them to Lightning+ and Persistence.
Lightning+ — Receives Arrow tables and maintains live lookback windows, accessible via the UI and API.
Persistence — Receives Arrow tables and writes them to Parquet files (or S3).

UNS Viewer

Browser-based live data inspector hosted by Lightning+:

http://localhost:5051/uns-viewer/

Displays live UNS data and trends. Namespaces are discovered at runtime and can be switched in the UI.

Flow Monitor

Browser-based pipeline performance dashboard hosted by Lightning+:

http://localhost:5051/flow/

Displays real-time DADOSFlow throughput, latency analysis, and system metrics. Connect it to the DADOSFlow metrics server (port 9877 on the DADOSFlow host) by entering the WebSocket address in the Flow Monitor settings overlay. Namespaces and the Lightning+ URL are also configurable there.

Key panels: - Throughput — live messages/sec and metrics/sec with sparkline history - System Metrics — process CPU, memory, network I/O, and uptime - Incoming Messages — live table of the latest sampled metric values - Latency Analysis — E2E, broker, DADOSFlow-internal, and flight latency heatmaps with P50–P99.99 percentiles - Pipeline Breakdown — per-stage latency as a stacked bar chart

All settings (WebSocket URL, Lightning+ URL, namespace, poll rate) are saved per browser in localStorage.

JSON Payload Support (Feature Flag)

By default, DADOSFlow expects Sparkplug B protobuf payloads. With the json-payload feature flag enabled, incoming JSON payloads are automatically converted to Sparkplug B on-the-fly.

Expected JSON format:

{
  "timestamp": 1706400000000,
  "metrics": [
    {"name": "temperature/celsius", "value": 72.5, "type": "Double"},
    {"name": "motor/running", "value": true, "type": "Boolean"},
    {"name": "controller/state", "value": "auto", "type": "String"}
  ]
}

Troubleshooting

“permission denied” running load-images.sh:

chmod +x load-images.sh && ./load-images.sh

A container won’t start / port already in use:

lsof -i :<port>

Stop the conflicting process or reconfigure the service to use a different port.

DADOS-Lightning is Killed / OOM: Docker Desktop allocates ~8 GiB of memory to containers by default. Lightning+ stores datasets in memory; if it exhausts available memory it crashes with Killed. Increase the limit in Docker Desktop → Settings → Resources.

DADOSFlow can’t reach Lightning+ or Persistence: Check that [lightning].addr and [persistence].addr in config/dadosflow.toml are correct, and that the target ports are reachable from the DADOSFlow host.

UNS Viewer page doesn’t load: Confirm DADOS-Lightning is running (docker compose ps) and port 5051 is not blocked.

Flow Monitor shows “Disconnected”: The Flow Monitor connects directly from your browser to the DADOSFlow metrics WebSocket on port 9877. Open the settings overlay (gear icon) and set the WebSocket URL to ws://<dadosflow-host>:9877. Confirm DADOSFlow is running and port 9877 is reachable from your browser.

Flow Monitor loads but no latency data appears: Latency panels require Lightning+ trend data. Confirm the Lightning+ URL in the Flow Monitor settings matches the running instance (default: http://localhost:5051) and that at least one ingress namespace has data flowing through it.

Network Access & HTTPS

All service UIs and APIs are available to other machines on the same LAN by default — no extra configuration needed. Native HTTPS support is coming in a future release. In the meantime, if HTTPS is required, place a reverse proxy such as Caddy or nginx in front of the relevant ports.

License

This software is provided under the DADOS Release License Agreement. Redistribution or production deployment is prohibited.

Contact

Ken Gardner — Founder & CEO — kg@dadostech.com
Luke Gardner — Co-Founder & CTO — luke@dadostech.com

Patent Pending

Patent Pending — U.S. Provisional Application No. 63/893,504