Skip to main content

PPL-to-IR compilation

ppl.Compile (pkg/ppl/compile.go) parses a query and walks its pipeline from the source command on. It keeps the current IR node and its output schema; each command reads that schema, checks its fields and functions, and wraps the node in one or more IR operators. The result is validated with ir.Validate before the optimizer sees it. lynxdb query --explain prints the logical plan this produces.

Source commands​

PPLIR
source=mainScan(main) with the open event schema (_time, _raw, then any column)
source=a, logs-*Scan(a, logs-*); * names are patterns
search termsFilter over the scan with Search predicates and comparisons
earliest=, latest=Scan.Time, kept symbolic (now-1h) until execution
source=[ ... ]the subsearch's plan as the source
describe mainSourceSchema(main)
makeresults count=2Compute(_time = now()) over Values(rows=2)
multisearch [..] [..]Union[by_time]
union a, bUnion

Search terms lower as follows:

TermIR predicate
errorSEARCH_WORD(_raw, "error")
conn*SEARCH_PREFIX(_raw, "conn")
*err*, us?rSEARCH_GLOB(_raw, ...)
"connection refused", api/v1/usersSEARCH_PHRASE(_raw, ...)
level=ERRORlevel = "ERROR"
status>=500status >= 500
host=web-*like(host, "web-%", TRUE)
NOT user=bobNOT coalesce(user = "bob", FALSE) (two-valued)
level IN (WARN, ERROR)(level = "WARN") OR (level = "ERROR")

A plan renders a free-text search with _raw as its field; it searches every string field of the event, of which _raw holds the text of a line that is not JSON.

Commands​

PPLIR
where cFilter(c)
fields a, b, fields - aProject(a, b), Project(drop a); a stored member such as http.path is a column of its own
fields p.x where p is an object computed by an earlier commandCompute(__path_0 = p.x), Rename(__path_0 -> p.x), then Project(p.x): the member becomes a column named by its path
rename a as bRenameColumns(a -> b)
eval x = e, y = fCompute(x = e, y = f)
replace, convert, nomvCompute with wildcard_replace, conversion functions (convert_num, dur2sec, ...), mvjoin
fillnull with v in fFillNull(f = v)
stats aggs by gAggregate(aggs BY g), then Project to put group columns first
stats ... by span(_time, 5m)a group key time_bucket(_time, 5, "m")
eventstatsWindow[partition]
streamstatsWindow[running] with a row bound or current-row exclusion
trendline sma(n, f)Window[running rows=n](moving_avg(f)) over Filter(NOT is_null(f))
sort kSort with explicit NULL placement
head n from mLimit(n, offset=m)
reverseReverse
dedup n kDedup(k, count=n)
top n f by gAggregate(count BY g, f), Sort, Dedup(g, count=n), Project
rare n fAggregate, ascending Sort, Limit(n), Project
bin f span=wCompute(f = time_bucket(...)) or a numeric range label
timechart, chartAggregate per bucket and series; series beyond limit are ranked by an IN subquery and renamed OTHER; then Sort
parse, grok, rexExtract[regex]; grok patterns expand to RE2 first
rex mode=sed "s/a/b/g"Compute(f = regexp_replace(f, "a", "b"))
regex f = 're'Filter(regexp_match(f, "re"))
spath path=pCompute(out = json_extract(in, "$.p"))
spath without pathCompute(in = parse_json(in))
patternsPatterns[simple, rows] or Patterns[simple, aggregate]
expand, mvexpandUnnest
flattenFlattenObject
mvcombineCombineValues
joinJoin[kind](keys) cond max qualify=l,r
lookup idx k append fleft Join with max=1 against Project(k, f) of the index, then Compute(f = coalesce(f, ...))
append [ ... ], appendpipe [ ... ]Union
appendcol [ ... ]ZipColumns
addtotals, addcoltotalsTotals
transposeTranspose
xyseries x y in (v1, v2) dAggregate by x with one first(d) FILTER (WHERE y = v) per value, then Sort

Expressions​

PPLIR
field, @timestampFieldRef; @timestamp becomes _time
literalLiteral with its IR type
+ - * / %, comparisons, AND OR XOR NOTBinary, Unary
IN (...), BETWEENInList, two comparisons
LIKE, ILIKElike(s, p, FALSE)
REGEXPregexp_match
CONTAINScontains(lower(s), lower(t))
case(...), if(...), nullif(...)Case
cast(x as t)Cast, or date/time/json calls
isnull(x)is_null(x)
match(f, 'a b')Search predicates on f
function callCall of the IR registry function the catalog maps the name to
aggregationAggCall inside Aggregate or Window; count(eval(c)) sets the call's filter
lambdaLambda with LambdaParams
x IN [ ... ], EXISTS [ ... ], [ ... ]Subquery in mode IN, EXISTS or scalar
INTERVAL n unit, DATE '...'duration or calendar literals, timestamp literals
earliest('-1d', t)t >= RelativeTime(now-1d)

Names of output columns​

An aggregation without as is named after its source text with spaces normalized: count(), percentile(response_time, 99), span(_time,1h). top and rare add count (and percent). patterns adds patterns_field (and pattern_count, sample_logs in aggregation mode). timechart keeps _time as the bucket column.

Tests​

Each command has a compile golden under pkg/ppl/testdata/compile: a .ppl query and the .ir plan it must produce. pkg/query checks a corpus of query plans (testdata/corpus_plans.sum), and pkg/database/execute_test.go runs queries end to end over a database.