Skip to main content

Ingest Settings

The ingest section controls the HTTP ingest endpoints and the shipper-compatible receivers. Every endpoint appends events to the store of the index it names; Storage Settings describe how they are written.

Endpoints​

EndpointFormatStored as
POST /api/v1/ingestNDJSON, one event per lineEach line unchanged
POST /api/v1/ingest/rawLog linesJSON object lines unchanged; any other line as an event with _raw, its timestamp and its key=value fields
POST /api/v1/ingest/hec, POST /services/collector, /services/collector/event, /services/collector/rawSplunk HECOne flat JSON object per event
POST /api/v1/es/_bulk, POST /_bulk, POST /{index}/_bulkElasticsearch bulk APIEach document line unchanged
POST /api/v1/otlp/v1/logs on the API port, POST /v1/logs on ingest.otlp.http_listenOTLP/HTTP logsOne flat JSON object per log record
ingest.otlp.grpc_listenOTLP/gRPC logsOne flat JSON object per log record

See Ingest API and Compatibility API for payload formats.

Request Limits​

KeyFlagEnv varDefaultDescription
ingest.max_body_sizenoneLYNXDB_INGEST_MAX_BODY_SIZE100mbMaximum request body on the API port, as sent. For POST /api/v1/ingest and /api/v1/ingest/raw it is the only limit, as sent and after decoding. Minimum 1kb.
ingest.limits.max_compressed_body_bytes--ingest-max-compressed-body-bytesLYNXDB_INGEST_LIMITS_MAX_COMPRESSED_BODY_BYTES32mbMaximum body bytes on the wire, compressed or not, on the Elasticsearch, Splunk HEC and OTLP routes. Requests over it get 413.
ingest.limits.max_decompressed_body_bytes--ingest-max-decompressed-body-bytesLYNXDB_INGEST_LIMITS_MAX_DECOMPRESSED_BODY_BYTES256mbMaximum body after gzip, zstd or snappy decoding on the same routes. Must be at least the compressed limit.
ingest.max_line_bytesnoneLYNXDB_INGEST_MAX_LINE_BYTES1048576Longest single line of a Splunk HEC or Elasticsearch bulk body. Minimum 1024.
ingest.max_batch_sizenoneLYNXDB_INGEST_MAX_BATCH_SIZE1000Events per internal batch while a Splunk HEC request is read. It does not cap the events in a request.

POST /api/v1/ingest and /api/v1/ingest/raw are bounded by ingest.max_body_size alone, compressed or not. The Elasticsearch, Splunk HEC and OTLP routes are bounded by the two ingest.limits settings, and by max_body_size as sent, so with the defaults their wire limit is 32 MB. A 413 names the setting that applies, for example compressed body exceeds limit 33554432 set by ingest.limits.max_compressed_body_bytes. The body limits need a restart; max_line_bytes and max_batch_size take effect on reload.

Parse Mode​

KeyEnv varDefaultDescription
ingest.modeLYNXDB_INGEST_MODEempty (full)How Splunk HEC, OTLP and syslog events are parsed before they are written. full extracts JSON and key=value fields from the event text. lightweight extracts only _time, host, source, sourcetype, level and index, and leaves the rest in _raw for query-time extraction with rex, parse or spath.

Elasticsearch Compatibility​

KeyFlagEnv varDefaultDescription
ingest.es_compat.enabled--ingest-es-enabledLYNXDB_INGEST_ES_COMPAT_ENABLEDtrueAccept Elasticsearch bulk requests. When off, they get 503 with Retry-After: 5; the handshake endpoints that Filebeat, Fluent Bit and Vector call at startup keep answering. Reloadable.
ingest.es_compat.advertised_version--ingest-es-versionLYNXDB_INGEST_ES_COMPAT_ADVERTISED_VERSION8.15.0Version reported to shippers, in X.Y.Z form. Needs a restart.
ingest.es_compat.cluster_namenoneLYNXDB_INGEST_ES_COMPAT_CLUSTER_NAMElynxdbCluster name in the handshake. Must not be empty when compatibility is on. Needs a restart.

Splunk HEC​

KeyEnv varDefaultDescription
ingest.splunk_hec.enabledLYNXDB_INGEST_SPLUNK_HEC_ENABLEDtrueServe the /services/collector routes. When off they answer 404.
ingest.splunk_hec.require_tokenLYNXDB_INGEST_SPLUNK_HEC_REQUIRE_TOKENfalseWith false, the /services/collector routes accept any non-empty Authorization: Splunk <token> header. With true, the token must be on a configured list, and the server configures no tokens, so every request is rejected.

POST /api/v1/ingest/hec sits under /api/ and uses API key authentication when auth is on.

OTLP​

KeyFlagEnv varDefaultDescription
ingest.otlp.http_listen--otlp-http-listenLYNXDB_INGEST_OTLP_HTTP_LISTEN0.0.0.0:4318Separate OTLP/HTTP listener (POST /v1/logs). Empty disables it.
ingest.otlp.grpc_listen--otlp-grpc-listenLYNXDB_INGEST_OTLP_GRPC_LISTENemptyOTLP/gRPC listener. Empty disables it.
ingest.otlp.grpc_max_recv_bytes--otlp-grpc-max-recv-bytesLYNXDB_INGEST_OTLP_GRPC_MAX_RECV_BYTES16mbLargest gRPC message. Minimum 1mb.

The OTLP listeners serve plain HTTP and gRPC without API key checks, and the HTTP listener binds all interfaces by default. On a shared network, restrict port 4318 with a firewall, bind it to a private address, or set http_listen: "" and send OTLP to /api/v1/otlp/v1/logs on the API port, which uses the server's TLS and auth.

Complete Example​

ingest:
max_body_size: "100mb"
max_line_bytes: 1048576
mode: "full"
limits:
max_compressed_body_bytes: "64mb"
max_decompressed_body_bytes: "512mb"
es_compat:
enabled: true
advertised_version: "8.15.0"
otlp:
http_listen: "127.0.0.1:4318"
grpc_listen: ""