Skip to main content

Ingest Data

A LynxDB server accepts data from the lynxdb ingest and lynxdb import commands, the REST API, log shippers that speak the Elasticsearch bulk API, OTLP or Splunk HEC, and syslog. Every path ends in the same storage and is queried with the same PPL queries.

Each event goes to an index, main unless the request names another. An index comes into being with its first events. Its events are kept as JSON objects: the keys of a JSON event are its fields, and the members of a nested object are fields named by their path, such as http.status.

Prerequisites​

Start a server:

lynxdb server

See Server Mode for storage options. To analyze a file once without storing it, use pipe mode instead.

Ingest from the CLI​

lynxdb ingest sends a file or standard input to the server as raw log lines.

# A file
lynxdb ingest access.log

# Source metadata and a target index
lynxdb ingest access.log --source web-01 --sourcetype nginx
lynxdb ingest app.log --source api-server --index production

# Standard input
cat access.log | lynxdb ingest --source web-01
kubectl logs deploy/api --since=1h | lynxdb ingest --source k8s-api
docker logs myapp 2>&1 | lynxdb ingest --source docker-myapp

Lines are sent in batches of 5000; --batch-size changes that. The source and source type are stored in the fields source and sourcetype, and the index is the name after source= in a query:

lynxdb ingest OpenSSH_2k.log --source sshd-01 --sourcetype sshd --index auth-logs
lynxdb query 'source=auth-logs | where source = "sshd-01" | stats count() by sourcetype'
{"count()":2000,"sourcetype":"sshd"}

Ingest through the REST API​

  • POST /api/v1/ingest: NDJSON, one JSON object per line, stored as sent
  • POST /api/v1/ingest/raw: log text, one event per line

Both take the index from ?index= or the X-Index header. See the Ingest API for every option.

JSON events​

Each line is one event, and its keys are its fields:

curl -X POST 'localhost:3100/api/v1/ingest?index=app' \
--data-binary @- <<'EOF'
{"timestamp":"2026-03-02T09:00:01.000Z","level":"error","service":"checkout","http":{"method":"POST","status":502}}
{"timestamp":"2026-03-02T09:00:02.000Z","level":"info","service":"catalog","http":{"method":"GET","status":200}}
EOF
{"data":{"accepted":2,"failed":0}}

The event time comes from the first of _time, timestamp, @timestamp, time and ts that the event holds (storage.time_fields), otherwise from the time the event arrived. A line that is not a JSON object is kept as an event holding the text in _raw.

Raw lines​

echo '2026-11-17T08:00:00.105Z host=web-03 level=ERROR status=502 msg="GET /api/v1/users completed"' \
| curl -X POST localhost:3100/api/v1/ingest/raw -H 'X-Source: web-03' --data-binary @-

# A whole file into the index "web"
curl -X POST localhost:3100/api/v1/ingest/raw \
-H "Content-Type: text/plain" \
-H "X-Source: web-01" \
-H "X-Index: web" \
--data-binary @access.log

A JSON line passes as it is. Any other line becomes an event with the line in _raw, its leading timestamp in _time and its key=value pairs as fields. The X-Source, X-Source-Type and X-Host headers set source (default http), sourcetype (default raw) and host. See Field Extraction for what is extracted.

Structured import​

lynxdb import reads NDJSON, CSV and Elasticsearch bulk exports. The format is detected from the file extension and content, or set with --format.

# NDJSON
lynxdb import events.ndjson

# CSV with a header row
lynxdb import export.csv
lynxdb import data.csv --source web-01 --index nginx

# Elasticsearch bulk export
lynxdb import es_dump.json --format esbulk

# Check a file without importing it
lynxdb import events.json --dry-run

# Standard input
cat events.ndjson | lynxdb import - --format ndjson

For NDJSON and CSV, the event time is read from the first of these fields present: time, _time, timestamp, _timestamp, @timestamp, ts, datetime. The scalar values of each record become fields and the record itself is kept in _raw.

Log shippers​

LynxDB accepts the protocols of common log shippers, so an existing shipper only needs LynxDB's address:

ShipperProtocolLynxDB endpoint
FilebeatElasticsearch bulkhttp://lynxdb:3100
Fluent BitElasticsearch bulkhttp://lynxdb:3100
VectorElasticsearch bulkhttp://lynxdb:3100
OpenTelemetry CollectorOTLP/HTTP, OTLP/gRPChttp://lynxdb:4318; gRPC on the address set with --otlp-grpc-listen
Splunk HEC sendersHTTP Event Collectorhttp://lynxdb:3100/services/collector/event

Elasticsearch bulk API​

Tools that write to Elasticsearch with _bulk can point at LynxDB:

# filebeat.yml
output.elasticsearch:
hosts: ["http://lynxdb:3100"]
# vector.yaml
sinks:
lynxdb:
type: elasticsearch
inputs: [app_logs]
endpoints: ["http://lynxdb:3100"]
mode: bulk

Each document is stored as sent, in the LynxDB index named by the bulk request's _index. A document without one goes to the index in the URL (POST /<index>/_bulk, POST /_data_stream/<name>/_bulk), otherwise to main. Query it with source=<index>, or source=filebeat-* for a family of indexes; lynxdb indexes lists the names. POST /_bulk, POST /<index>/_bulk and /api/v1/es/_bulk all accept bulk requests. See the compatibility API.

OpenTelemetry (OTLP)​

The OTLP/HTTP receiver listens on 0.0.0.0:4318 by default. OTLP/gRPC is off until --otlp-grpc-listen (or ingest.otlp.grpc_listen) sets an address.

# otel-collector-config.yaml
exporters:
otlphttp/lynxdb:
endpoint: http://lynxdb:4318

Splunk HEC​

Senders that post to Splunk's HTTP Event Collector can use:

http://lynxdb:3100/services/collector/event

Native syslog receiver​

LynxDB receives syslog over UDP and TCP without an intermediary, which suits network devices and hosts running rsyslog or syslog-ng.

# UDP and TCP on port 5514 (the default when no port is given)
lynxdb server --syslog :5514

# The standard port 514 (needs root or CAP_NET_BIND_SERVICE)
lynxdb server --syslog :514

Or in the config file:

syslog:
udp: ":5514"
tcp: ":5514"

rsyslog​

# /etc/rsyslog.d/50-lynxdb.conf
# Over TCP
*.* @@lynxdb-host:5514

# Or over UDP
*.* @lynxdb-host:5514

syslog-ng​

destination d_lynxdb {
tcp("lynxdb-host" port(5514));
};
log { source(s_src); destination(d_lynxdb); };

TLS​

The TCP listener can use the server's TLS configuration:

lynxdb server --tls --syslog-tcp :6514 --syslog-tls
# /etc/rsyslog.d/50-lynxdb-tls.conf
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* @@lynxdb-host:6514

The receiver parses RFC 5424, RFC 3164 and raw messages, detecting the dialect per message. See the syslog configuration for framing, batching and connection limits.

Pipe mode (no server)​

To analyze logs without storing them, query them directly. The CLI loads the input into a database of its own, runs the query and exits:

cat app.log | lynxdb query 'source=main | stats count() by level'
lynxdb query --file '/var/log/nginx/*.log' 'source=main status>=500 | top 10 uri'

See Pipe Mode.

Check what arrived​

# Server overview: event counts, storage, indexes
lynxdb status

# Events, bytes and time range of each index
lynxdb indexes

# A few events
lynxdb query 'source=main | head 5'

# Events of the last 5 minutes
lynxdb query 'source=main | stats count()' --since 5m

# The field catalog
lynxdb fields

A query syncs the database before it runs, so it sees every event whose ingest request was answered.

Next steps​