Skip to main content

Ingest Data

LynxDB accepts data through multiple paths: the lynxdb ingest CLI command, the REST API, structured import, and drop-in compatibility endpoints for existing log pipelines. This guide covers each method with practical examples.

Prerequisites​

Start a LynxDB server (or use pipe mode for local-only workflows):

lynxdb server

See the server mode guide for persistent storage options.

Ingest from the CLI​

The lynxdb ingest command sends log files or stdin to a running server.

Ingest a file​

lynxdb ingest access.log

Ingest with metadata​

Tag events with --source and --sourcetype so you can filter on them later:

lynxdb ingest access.log --source web-01 --sourcetype nginx
lynxdb ingest app.log --source api-server --index production

Ingest from stdin​

Pipe any output directly into LynxDB:

cat access.log | lynxdb ingest --source web-01
kubectl logs deploy/api --since=1h | lynxdb ingest --source k8s-api
docker logs myapp 2>&1 | lynxdb ingest --source docker-myapp

Tune batch size​

For large files, increase the batch size to reduce HTTP round-trips:

lynxdb ingest huge.log --batch-size 10000

The default batch size is 5000 lines per request.

Ingest via the REST API​

Use the structured ingest endpoint for event payload arrays, and the raw ingest endpoint for newline-delimited logs:

Send structured events​

curl -X POST localhost:3100/api/v1/ingest \
  -H "Content-Type: application/json" \
  -d '[
    {
      "event": "user login",
      "source": "auth-api",
      "fields": {
        "user_id": 42,
        "ip": "10.0.1.5",
        "level": "info"
      }
    }
  ]'

Send a structured batch​

curl -X POST localhost:3100/api/v1/ingest \
  -H "Content-Type: application/json" \
  -d '[
    {
      "event": "request started",
      "source": "api",
      "fields": {"path": "/api/users", "level": "info"}
    },
    {
      "event": "connection refused",
      "source": "api",
      "fields": {"service": "redis", "level": "error"}
    }
  ]'

Send raw text​

For unstructured log lines, post the raw text:

echo '192.168.1.1 - - [14/Feb/2026:14:23:01 +0000] "GET /api HTTP/1.1" 200 1234' \
  | curl -X POST localhost:3100/api/v1/ingest/raw --data-binary @-

Or send an entire file:

curl -X POST localhost:3100/api/v1/ingest/raw \
  -H "Content-Type: text/plain" \
  --data-binary @access.log

Structured import​

The lynxdb import command handles structured formats (NDJSON, CSV, Elasticsearch bulk exports) and preserves field types and timestamps. ndjson and csv are normalized into LynxDB's structured event envelope; esbulk uses the Elasticsearch-compatible bulk API.

Import NDJSON​

lynxdb import events.json
lynxdb import events.ndjson

Import CSV​

lynxdb import splunk_export.csv
lynxdb import data.csv --source web-01 --index nginx

Import Elasticsearch bulk export​

lynxdb import es_dump.json --format esbulk

Validate before importing​

Use --dry-run to check the file without writing any data:

lynxdb import events.json --dry-run

Timestamp auto-detection​

LynxDB automatically detects timestamps from these commonly used field names:

  • _timestamp
  • timestamp
  • @timestamp
  • time
  • ts
  • datetime

If none of these fields are present, LynxDB assigns the server receive time. You do not need to configure timestamp parsing.

Drop-in log shipper compatibility​

LynxDB accepts traffic from common production log shippers with native protocols:

ToolProtocolDefault LynxDB endpoint
FilebeatElasticsearch bulkhttp://lynxdb:3100
Fluent BitElasticsearch bulkhttp://lynxdb:3100
VectorElasticsearch bulkhttp://lynxdb:3100
OpenTelemetry CollectorOTLP HTTP/gRPChttp://lynxdb:4318, lynxdb:4317
Splunk HECHTTP Event Collectorhttp://lynxdb:3100/services/collector/event

The fastest way to get a known-good config is:

lynxdb shippers config filebeat --remote http://lynxdb:3100
lynxdb shippers config fluent-bit --remote http://lynxdb:3100
lynxdb shippers config vector --remote http://lynxdb:3100
lynxdb shippers config otelcol --remote http://lynxdb:4318
lynxdb shippers config splunk-hec --remote http://lynxdb:3100

After traffic starts, use:

lynxdb shippers
lynxdb doctor shippers

Compatibility endpoints​

LynxDB provides compatibility endpoints so you can migrate existing log pipelines without changing your shipper configuration.

Filebeat / Logstash / Vector (Elasticsearch _bulk API)​

Point any tool that speaks the Elasticsearch _bulk protocol at LynxDB:

# filebeat.yml
output.elasticsearch:
  hosts: ["http://lynxdb:3100"]
# vector.toml
[sinks.lynxdb]
type = "elasticsearch"
endpoints = ["http://lynxdb:3100"]

The _index field from the bulk request is mapped to the _source tag in LynxDB. Prefer the unprefixed ES-compatible endpoint for drop-in shippers; /api/v1/es/_bulk and /api/v1/ingest/bulk remain aliases. See the compatibility API reference for details.

OpenTelemetry Collector (OTLP)​

Send logs from an OpenTelemetry Collector using the OTLP/HTTP exporter:

# otel-collector-config.yaml
exporters:
  otlp_http:
    endpoint: http://lynxdb:4318
    encoding: json

Splunk HEC (HTTP Event Collector)​

If you have existing Splunk forwarders, point them at the HEC-compatible endpoint:

http://lynxdb:3100/services/collector/event

Native syslog receiver​

LynxDB can receive logs directly over syslog (UDP and TCP) without any intermediary. This is useful for network devices, servers running rsyslog or syslog-ng, and any system that speaks the syslog protocol.

Enable the syslog receiver​

# Enable on the standard syslog port (requires root or CAP_NET_BIND_SERVICE)
lynxdb server --syslog :514

# Or use a non-privileged port
lynxdb server --syslog :5514

Or in your config file:

syslog:
  udp: ":514"
  tcp: ":514"

Configure rsyslog​

# /etc/rsyslog.d/50-lynxdb.conf
# Forward everything over TCP
*.* @@lynxdb-host:514

# Or over UDP
*.* @lynxdb-host:514

Configure syslog-ng​

destination d_lynxdb {
  tcp("lynxdb-host" port(514));
};
log { source(s_src); destination(d_lynxdb); };

With TLS​

For secure transport, enable TLS on the syslog TCP listener:

lynxdb server --tls --syslog-tcp :6514 --syslog-tls

Configure rsyslog with TLS​

# /etc/rsyslog.d/50-lynxdb-tls.conf
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* @@lynxdb-host:6514

The syslog receiver supports RFC 5424, RFC 3164, and raw messages with automatic dialect detection. See the syslog configuration reference for all available options including framing, batch tuning, and connection limits.

Pipe mode (no server)​

You do not need a running server to analyze logs. LynxDB can ingest data into an ephemeral in-memory engine and query it in one step:

cat app.log | lynxdb query '| stats count by level'
lynxdb query --file '/var/log/nginx/*.log' '| where status>=500 | top 10 uri'

Data is not persisted. The engine starts, ingests, queries, prints results, and exits. See the pipe mode guide for more details.

Monitoring ingestion​

After ingesting data, verify it landed correctly:

# Check server stats
lynxdb status

# Count recently ingested events
lynxdb count --since 5m

# Peek at a sample of events
lynxdb sample 5

# See all discovered fields
lynxdb fields

See the lynxdb status and lynxdb fields commands for more options.

Next steps​