Ingest Data
A LynxDB server accepts data from the lynxdb ingest and lynxdb import
commands, the REST API, log shippers that speak the Elasticsearch bulk API,
OTLP or Splunk HEC, and syslog. Every path ends in the same storage and is
queried with the same PPL queries.
Each event goes to an index, main unless the request names another. An
index comes into being with its first events. Its events are kept as JSON
objects: the keys of a JSON event are its fields, and the members of a nested
object are fields named by their path, such as http.status.
Prerequisites
Start a server:
lynxdb server
See Server Mode for storage options. To analyze a file once without storing it, use pipe mode instead.
Ingest from the CLI
lynxdb ingest sends a file or standard input to the
server as raw log lines.
# A file
lynxdb ingest access.log
# Source metadata and a target index
lynxdb ingest access.log --source web-01 --sourcetype nginx
lynxdb ingest app.log --source api-server --index production
# Standard input
cat access.log | lynxdb ingest --source web-01
kubectl logs deploy/api --since=1h | lynxdb ingest --source k8s-api
docker logs myapp 2>&1 | lynxdb ingest --source docker-myapp
Lines are sent in batches of 5000; --batch-size changes that. The source
and source type are stored in the fields source and sourcetype, and the
index is the name after source= in a query:
lynxdb ingest OpenSSH_2k.log --source sshd-01 --sourcetype sshd --index auth-logs
lynxdb query 'source=auth-logs | where source = "sshd-01" | stats count() by sourcetype'
{"count()":2000,"sourcetype":"sshd"}
Ingest through the REST API
POST /api/v1/ingest: NDJSON, one JSON object per line, stored as sentPOST /api/v1/ingest/raw: log text, one event per line
Both take the index from ?index= or the X-Index header. See the
Ingest API for every option.
JSON events
Each line is one event, and its keys are its fields:
curl -X POST 'localhost:3100/api/v1/ingest?index=app' \
--data-binary @- <<'EOF'
{"timestamp":"2026-03-02T09:00:01.000Z","level":"error","service":"checkout","http":{"method":"POST","status":502}}
{"timestamp":"2026-03-02T09:00:02.000Z","level":"info","service":"catalog","http":{"method":"GET","status":200}}
EOF
{"data":{"accepted":2,"failed":0}}
The event time comes from the first of _time, timestamp, @timestamp,
time and ts that the event holds (storage.time_fields), otherwise from
the time the event arrived. A line that is not a JSON object is kept as an
event holding the text in _raw.
Raw lines
echo '2026-11-17T08:00:00.105Z host=web-03 level=ERROR status=502 msg="GET /api/v1/users completed"' \
| curl -X POST localhost:3100/api/v1/ingest/raw -H 'X-Source: web-03' --data-binary @-
# A whole file into the index "web"
curl -X POST localhost:3100/api/v1/ingest/raw \
-H "Content-Type: text/plain" \
-H "X-Source: web-01" \
-H "X-Index: web" \
--data-binary @access.log
A JSON line passes as it is. Any other line becomes an event with the line
in _raw, its leading timestamp in _time and its key=value pairs as
fields. The X-Source, X-Source-Type and X-Host headers set source
(default http), sourcetype (default raw) and host. See
Field Extraction for what is extracted.
Structured import
lynxdb import reads NDJSON, CSV and Elasticsearch bulk
exports. The format is detected from the file extension and content, or set
with --format.
# NDJSON
lynxdb import events.ndjson
# CSV with a header row
lynxdb import export.csv
lynxdb import data.csv --source web-01 --index nginx
# Elasticsearch bulk export
lynxdb import es_dump.json --format esbulk
# Check a file without importing it
lynxdb import events.json --dry-run
# Standard input
cat events.ndjson | lynxdb import - --format ndjson
For NDJSON and CSV, the event time is read from the first of these fields
present: time, _time, timestamp, _timestamp, @timestamp, ts,
datetime. The scalar values of each record become fields and the record
itself is kept in _raw.
Log shippers
LynxDB accepts the protocols of common log shippers, so an existing shipper only needs LynxDB's address:
| Shipper | Protocol | LynxDB endpoint |
|---|---|---|
| Filebeat | Elasticsearch bulk | http://lynxdb:3100 |
| Fluent Bit | Elasticsearch bulk | http://lynxdb:3100 |
| Vector | Elasticsearch bulk | http://lynxdb:3100 |
| OpenTelemetry Collector | OTLP/HTTP, OTLP/gRPC | http://lynxdb:4318; gRPC on the address set with --otlp-grpc-listen |
| Splunk HEC senders | HTTP Event Collector | http://lynxdb:3100/services/collector/event |
Elasticsearch bulk API
Tools that write to Elasticsearch with _bulk can point at LynxDB:
# filebeat.yml
output.elasticsearch:
hosts: ["http://lynxdb:3100"]
# vector.yaml
sinks:
lynxdb:
type: elasticsearch
inputs: [app_logs]
endpoints: ["http://lynxdb:3100"]
mode: bulk
Each document is stored as sent, in the LynxDB index named by the bulk
request's _index. A document without one goes to the index in the URL
(POST /<index>/_bulk, POST /_data_stream/<name>/_bulk), otherwise to
main. Query it with source=<index>, or source=filebeat-* for a family
of indexes; lynxdb indexes lists the names. POST /_bulk,
POST /<index>/_bulk and /api/v1/es/_bulk all accept bulk requests. See the compatibility API.
OpenTelemetry (OTLP)
The OTLP/HTTP receiver listens on 0.0.0.0:4318 by default. OTLP/gRPC is
off until --otlp-grpc-listen (or ingest.otlp.grpc_listen) sets an
address.
# otel-collector-config.yaml
exporters:
otlphttp/lynxdb:
endpoint: http://lynxdb:4318
Splunk HEC
Senders that post to Splunk's HTTP Event Collector can use:
http://lynxdb:3100/services/collector/event
Native syslog receiver
LynxDB receives syslog over UDP and TCP without an intermediary, which suits network devices and hosts running rsyslog or syslog-ng.
# UDP and TCP on port 5514 (the default when no port is given)
lynxdb server --syslog :5514
# The standard port 514 (needs root or CAP_NET_BIND_SERVICE)
lynxdb server --syslog :514
Or in the config file:
syslog:
udp: ":5514"
tcp: ":5514"
rsyslog
# /etc/rsyslog.d/50-lynxdb.conf
# Over TCP
*.* @@lynxdb-host:5514
# Or over UDP
*.* @lynxdb-host:5514
syslog-ng
destination d_lynxdb {
tcp("lynxdb-host" port(5514));
};
log { source(s_src); destination(d_lynxdb); };
TLS
The TCP listener can use the server's TLS configuration:
lynxdb server --tls --syslog-tcp :6514 --syslog-tls
# /etc/rsyslog.d/50-lynxdb-tls.conf
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* @@lynxdb-host:6514
The receiver parses RFC 5424, RFC 3164 and raw messages, detecting the dialect per message. See the syslog configuration for framing, batching and connection limits.
Pipe mode (no server)
To analyze logs without storing them, query them directly. The CLI loads the input into a database of its own, runs the query and exits:
cat app.log | lynxdb query 'source=main | stats count() by level'
lynxdb query --file '/var/log/nginx/*.log' 'source=main status>=500 | top 10 uri'
See Pipe Mode.
Check what arrived
# Server overview: event counts, storage, indexes
lynxdb status
# Events, bytes and time range of each index
lynxdb indexes
# A few events
lynxdb query 'source=main | head 5'
# Events of the last 5 minutes
lynxdb query 'source=main | stats count()' --since 5m
# The field catalog
lynxdb fields
A query syncs the database before it runs, so it sees every event whose ingest request was answered.
Next steps
- Field Extraction: what ingestion extracts and how to extract more
- Your First Query: query the data you ingested
- Ingest API: request and response reference
- CLI:
ingest: every flag