Skip to main content

Syslog Receiver

The syslog section configures a native receiver for syslog over UDP (RFC 5426), TCP (RFC 6587) and TCP with TLS. Messages are parsed as RFC 5424, RFC 3164 or raw text and written to the index syslog.index, one flat JSON object per message.

The receiver is off until syslog.udp or syslog.tcp has an address.

Quick Start​

# UDP and TCP on port 5514
lynxdb server --syslog :5514

# UDP only
lynxdb server --syslog-udp :5514

# TCP with TLS on port 6514 (needs server TLS)
lynxdb server --tls --syslog-tcp :6514 --syslog-tls
syslog:
udp: ":5514"
tcp: ":5514"

Point a sender at the receiver:

# rsyslog: @@ forwards over TCP, a single @ over UDP
*.* @@lynxdb-host:5514

# syslog-ng
destination d_lynxdb { network("lynxdb-host" port(5514) transport("tcp")); };

Ports below 1024 (such as 514) need root or the CAP_NET_BIND_SERVICE capability.

Check what arrived:

lynxdb query 'source=main sourcetype="syslog:rfc5424" | stats count() by host, app_name'
lynxdb query 'source=main severity <= 3 | stats count() by host, severity_label'

Listeners​

KeyFlagEnv varDefaultDescription
syslog.udp--syslog-udp, --syslogLYNXDB_SYSLOG_UDPemptyUDP address (host:port). One message per datagram.
syslog.tcp--syslog-tcp, --syslogLYNXDB_SYSLOG_TCPemptyTCP address (host:port).
syslog.tls--syslog-tlsLYNXDB_SYSLOG_TLSfalseWrap the TCP listener with the server's TLS configuration.

The flags accept :port, host:port, a bare port or a bare host; a missing port becomes 5514, or 6514 for TCP when --syslog-tls is set. syslog.tls needs server TLS (--tls, tls.enabled, or a certificate and key); without it the receiver fails to start.

Parser​

KeyFlagEnv varDefault
syslog.parser--syslog-parserLYNXDB_SYSLOG_PARSERauto
ValueBehavior
autoPer message: RFC 5424 when a version digit follows the <PRI> header, RFC 3164 for other <PRI> messages, raw otherwise.
rfc5424Parse every message as RFC 5424.
rfc3164Parse every message as RFC 3164 (BSD syslog).
rawNo parsing; the message is stored in _raw.

A message that fails to parse is stored as a raw event and counted in lynxdb_syslog_parse_errors_total.

Extracted Fields​

FieldRFC 5424RFC 3164Description
facility, facility_labelyesyesFacility code (0 to 23) and name (kern, user, daemon, local0 to local7, ...)
severity, severity_labelyesyesSeverity (0 to 7) and name (emerg, alert, crit, err, warning, notice, info, debug)
levelyesyesSame as severity_label
hostyesyesHostname from the header
app_nameyesyesAPP-NAME (RFC 5424) or tag (RFC 3164)
procidyesyesProcess ID
msgidyesnoMSGID
messageyesyesMessage text
sd_<SD-ID>_<param>yesnoStructured data parameters

Each event also keeps the message in _raw, and the key=value pairs of the message text become fields too. The event time is the header timestamp, or the receive time when the header has none. The sourcetype field is <syslog.sourcetype>:<dialect>: syslog:rfc5424, syslog:rfc3164 or syslog:raw.

TCP Framing​

KeyEnv varDefaultValues
syslog.framingLYNXDB_SYSLOG_FRAMINGautoauto, octet-counting, non-transparent
syslog.trailerLYNXDB_SYSLOG_TRAILERautoauto, lf, nul, crlf

With auto framing, a connection whose first byte is a digit uses octet counting (<length> <message>); any other connection uses non-transparent framing. With auto trailer, the first LF or NUL on the connection sets the trailer; a CR before LF is stripped. Framing does not apply to UDP.

Timestamps, Host and Routing​

KeyEnv varDefaultDescription
syslog.default_timezoneLYNXDB_SYSLOG_DEFAULT_TIMEZONELocalTime zone for RFC 3164 timestamps, which carry no zone or year. Local or an IANA name such as UTC or Europe/Berlin.
syslog.default_hostnameLYNXDB_SYSLOG_DEFAULT_HOSTNAMEemptyHost for messages whose hostname is missing or -.
syslog.indexLYNXDB_SYSLOG_INDEX (flag --syslog-index)mainTarget index.
syslog.sourcetypeLYNXDB_SYSLOG_SOURCETYPEsyslogBase sourcetype; the dialect is appended.
syslog.use_peer_as_sourceLYNXDB_SYSLOG_USE_PEER_AS_SOURCEtrueSet the source field to udp://<peer> or tcp://<peer>.

Limits and Batching​

KeyEnv varDefaultDescription
syslog.max_message_bytesLYNXDB_SYSLOG_MAX_MESSAGE_BYTES65536Larger messages are dropped. At least 1024.
syslog.udp_read_bufferLYNXDB_SYSLOG_UDP_READ_BUFFER2mbUDP socket receive buffer. Raise it on busy UDP receivers to avoid kernel drops.
syslog.tcp_idle_timeoutLYNXDB_SYSLOG_TCP_IDLE_TIMEOUT5mClose TCP connections idle this long.
syslog.tcp_max_connectionsLYNXDB_SYSLOG_TCP_MAX_CONNECTIONS1000Further connections are closed immediately.
syslog.batch_sizeLYNXDB_SYSLOG_BATCH_SIZE1000Events per batch handed to ingest.
syslog.batch_timeoutLYNXDB_SYSLOG_BATCH_TIMEOUT200msHand over a partial batch after this long.

Reload​

syslog.index, syslog.sourcetype, syslog.default_timezone, syslog.default_hostname, syslog.batch_size and syslog.batch_timeout take effect on reload. Listener addresses, TLS, parser, framing, trailer, buffer and connection limits need a restart.

Metrics​

MetricTypeLabels
lynxdb_syslog_messages_received_totalcountertransport, dialect
lynxdb_syslog_messages_dropped_totalcountertransport, reason (toolarge, conn_limit, backpressure)
lynxdb_syslog_active_connectionsgaugetransport="tcp"
lynxdb_syslog_parse_errors_totalcounterdialect

Complete Example​

tls:
cert_file: /etc/lynxdb/tls/server.crt
key_file: /etc/lynxdb/tls/server.key

syslog:
udp: ":5514"
tcp: ":6514"
tls: true
parser: auto
default_timezone: "UTC"
index: "main"
sourcetype: "syslog"
max_message_bytes: 131072
udp_read_buffer: "8mb"
tcp_max_connections: 5000
batch_size: 1000
batch_timeout: "200ms"