# lightning-client

Python client for Lightning. Use this package to connect to a Lightning
server and read time-series metric data into Python data frames
(`datafusion`, `pyarrow`, `pandas`, or `polars`).

Under the hood: rolling-window trend queries, derived statistics
(mean / std / min / max / rate-of-change / …), a UNS ("unified namespace")
catalog of every live metric, server-side SQL, and live streaming — all
delivered as Arrow IPC for low overhead.

## Install

Unpack the release ZIP, `cd` into it, and install with pip:

```bash
unzip DADOS_Python_Connector_<version>.zip
cd DADOS_Python_Connector
python -m pip install --find-links wheels lightning-client
```

The unpacked `DADOS_Python_Connector/` directory contains:

```
DADOS_Python_Connector/
  README.md          ← this file
  wheels/            ← one lightning-client wheel per supported platform
  examples/          ← runnable example scripts and a notebook
```

`wheels/` ships one wheel per supported platform (macOS arm64, macOS
x86_64, Linux x86_64 / manylinux2014). `--find-links wheels` tells pip to
treat that directory as a local wheel index; pip picks the wheel whose
platform tag matches your OS and architecture.

Python 3.10 or newer. The native extension is bundled inside the wheel —
no Rust toolchain is needed on the install host. `python -m pip` (rather
than a bare `pip`) ensures the install goes into the Python you'll
actually run.

Pip resolves the runtime dependencies (`pyarrow`, `datafusion`) from your
configured package source (public PyPI by default, or whatever index you
have configured), unless they are already installed or otherwise supplied
locally.

### Optional installs

```bash
python -m pip install --find-links wheels "lightning-client[pandas]"   # adds pandas output
python -m pip install --find-links wheels "lightning-client[polars]"   # adds polars output
python -m pip install --find-links wheels "lightning-client[all]"      # pandas + polars
```

For internal-mirror or air-gapped installs, see
[`examples/INSTALL_ADVANCED.md`](examples/INSTALL_ADVANCED.md).

### Verify the install

This check imports the package — it does not need a running Lightning
server:

```bash
python -c "import lightning_client; import lightning_client._native; print('ok')"
```

If it prints `ok`, the install succeeded.

The example scripts (e.g. `python examples/quickstart.py`) additionally
require a reachable Lightning service — see the next section.

## Quickstart

**Before you run this:**

- You need network access to a running Lightning server.
- **Finding your server address:** Lightning's API listens on port `9100`. The address
  you pass to `LightningClient.connect()` is always `<host>:9100` where `<host>` is the
  IP address or hostname of the machine running Lightning:
  - Same machine as this script: `127.0.0.1:9100`
  - Remote server or VM: `192.168.1.50:9100` (use that machine's IP or hostname)
  - Docker: port 9100 must be forwarded when the container is started
    (`docker run -p 9100:9100 …`); then connect to the Docker host's address
- Replace `127.0.0.1:9100` in the example below with your server's address.
- Replace `"DEMO"` with the name of a namespace that exists on that server.
- The window label `"5m"` is only an example. Call `ns.windows()` first
  to see the labels your server is actually configured with, then use
  one of those.

```python
from lightning_client import LightningClient

client = LightningClient.connect("127.0.0.1:9100")   # your server
ns     = client.namespace("DEMO")                    # your namespace

print(ns.windows())                                  # see configured window labels
w = ns.window("5m")                                  # pick one from that list

df, meta = w.data()                                  # bootstrap fetch
df, meta = w.data(metric_name="temperature")         # server-side filter
df, meta = w.data(since=meta)                        # incremental
```

`w.data()` returns `(df, meta)` — `df` is the rows, `meta` carries the
cursor and timing/coverage of the window. Pass `meta` (or
`meta.snapshot_id` as a bare int) back as `since=` to fetch only what's
new since the previous call.

## Output formats

Every query (`w.data()`, `w.derived()`, `ns.uns()`, `w.sql()`, etc.)
returns a DataFrame-like object. You choose which DataFrame library that
object belongs to **once, at connection time**, with the `df=` argument:

```python
client = LightningClient.connect("127.0.0.1:9100", df="pandas")
```

Internally the client always decodes responses as Arrow (zero-copy from
the wire); the `df` setting just selects how that Arrow data is handed
back to your code. There is no per-call conversion option — pick the
library you want to work in, then the rest of your code reads naturally
in that library's idioms.

| `df` value     | Returned object         | Required extra | Choose this if…                                                                |
| -------------- | ----------------------- | -------------- | ------------------------------------------------------------------------------ |
| `"datafusion"` | `datafusion.DataFrame`  | base install   | Default. You want a lazy, SQL-friendly frame with no extra dependencies.       |
| `"pyarrow"`    | `pyarrow.Table`         | base install   | You're already working in Arrow, or you want the rawest, lowest-overhead form. |
| `"pandas"`     | `pandas.DataFrame`      | `[pandas]`     | Your downstream code or notebooks expect pandas.                               |
| `"polars"`     | `polars.DataFrame`      | `[polars]`     | You want polars' lazy / multi-threaded engine for downstream work.             |

If you omit `df=`, you get `datafusion`. `"pyarrow"` is the lightest:
nothing converts past Arrow. `"pandas"` and `"polars"` each materialise
into their library's native frame and so require the matching extra.

## Common tasks

### Trend categories

Sparkplug B message classes — each returns `(df, meta)`:

```python
w = ns.window("5m")

df, meta = w.data(metric_name="temperature")    # DDATA / NDATA
df, meta = w.birth()                            # NBIRTH / DBIRTH
df, meta = w.death()                            # NDEATH / DDEATH
df, meta = w.cmd()                              # NCMD / DCMD
df, meta = w.state()                            # STATE
```

### Derived metrics

Rolling-window statistics — mean, std, min, max, rate of change, slope…

```python
df, meta = w.derived(metric_name="temperature")
```

### SQL against the window

```python
df = w.sql("""
    SELECT timestamp, metric_name, value_double
    FROM data
    WHERE metric_name = 'temperature' AND value_double > 80
    ORDER BY timestamp DESC
    LIMIT 1000
""")
```

### UNS — Unified Namespace

A live catalog: one row per active metric, with the latest value,
timestamp, broker, and any custom hierarchy columns your deployment
defines (e.g. `site`, `area`, `line`).

```python
df = ns.uns()
ns.keys()                       # hierarchy column names available here
ns.distinct("site")
ns.exists(site="PlantA", line="Line1")
```

## Examples

Every topic above has a runnable example in the unpacked `examples/`
directory:

| Script                     | Topic                                                       |
| -------------------------- | ----------------------------------------------------------- |
| `examples/quickstart.py`   | Connect → window → `data()`; all `meta` fields              |
| `examples/catalog.py`      | Windows, UNS keys/topics/metrics, schemas, SQL              |
| `examples/trend.py`        | Filters: keyword, `where=`, `MetricFilter` AND/OR           |
| `examples/uns.py`          | UNS snapshot, version gate, `uns_delta`, combined patterns  |
| `examples/stream.py`       | Live `stream()`, server-side filters, depth, throughput     |
| `examples/poll.py`         | Cursor-based `poll()` / `since=`, checkpoint and resume     |
| `examples/datasources.py`  | Discover and inspect server-side datasources                |
| `examples/query.ipynb`     | Notebook: live latency plot (needs `[notebook]` extra)      |

Run any of them from the unpacked `DADOS_Python_Connector/` directory
(after editing the server address / namespace at the top of the script):

```bash
python examples/quickstart.py
```

See [`examples/README.md`](examples/README.md) for the notebook setup.

## Existing integrations

New code should use `LightningClient` as shown above. The pre-native HTTP
clients are still available under `lightning_client.legacy` for existing
integrations and require the `[legacy]` extra:

```python
from lightning_client.legacy import LegacyLightningClient

legacy = LegacyLightningClient("http://127.0.0.1:8080")
```
