Structured Native Language (SNL)

Structured Native Language (SNL) is the native query language of shadeDB. It was designed specifically for the shadeDB engine, providing a deterministic, structured, and lightweight syntax for interacting with database partitions. Rather than adopting a general-purpose relational query language, SNL is tightly integrated with the engine's internal architecture, allowing queries to map directly onto shadeDB's execution pipeline.

Every SNL query is composed of explicit execution stages separated by semicolons (;). Each statement has a clearly defined purpose, eliminating ambiguous execution paths and allowing the engine to process queries in a predictable top-down order.

Design Goals

SNL was created around a simple philosophy: make the query language match the database engine rather than forcing the engine to adapt to a general-purpose language.

Because SNL is purpose-built for shadeDB, its syntax reflects the engine's native execution model. Operations such as fetching, updating, recycling related records, freezing data, and projecting fields are expressed directly as engine instructions rather than being translated through multiple abstraction layers.

Why SNL?

• Engine-Native Design
  SNL was designed alongside shadeDB itself, allowing the query language and execution engine to evolve together.
• Deterministic Execution
  Queries execute in the order they are written. The engine does not rewrite queries or perform hidden execution plan transformations.
• Explicit Syntax
  Every operation is declared directly using dedicated clauses and execution modes, making query behaviour predictable and easy to reason about.
• Fully Typed, Everywhere
  Both the field being referenced and the value being matched or assigned are always wrapped in an explicit type - str<username>::str<john> - Document bodies are the one exception: they are plain JSON strings.
• Relationship Traversal
  Features such as RECYCLE, TARGET(), and RETRIEVE() allow related partitions to be traversed without requiring multiple client-side requests.
• Minimal Query Grammar
  A compact grammar reduces parser complexity while keeping queries readable and consistent across every supported operation.

Core Principles

Query Philosophy

SNL follows an explicit execution model. Each clause contributes a specific step to the query pipeline, and the engine processes those steps sequentially from top to bottom. There are no implicit joins, hidden mutations, or automatic query restructuring.

This approach makes query behaviour predictable, simplifies debugging, and allows developers to understand exactly how their queries are executed.

Query Lifecycle

1. 1. Receive Query
   The engine receives the SNL request from an official client or API endpoint.
2. 2. Parse & Validate
   The parser validates the query grammar, clauses, delimiters, and type wrappers.
3. 3. Authorize
   The caller's permissions are verified before any partition is accessed.
4. 4. Execute
   The requested operation is performed using the supplied clauses and execution modes.
5. 5. Shape Response
   Projection, filtering, ordering, and pagination clauses are applied before constructing the final response.
6. 6. Return Result
   The engine returns a standardized response containing the operation status, the query result (or error identifier), and the database execution latency.

Formal Language Grammar

SNL follows a deterministic, non-backtracking grammar designed for sequential parsing. Every query is processed from top to bottom using explicit delimiters and well-defined clauses, allowing the engine to validate syntax before execution begins.

The language specification below is presented using Extended Backus–Naur Form (EBNF). It describes the general structure of valid SNL queries. Individual execution modes may support additional clauses or impose further constraints, as documented throughout this reference.

Query            ::= PartitionClause? Mode ";" Statement*

PartitionClause  ::= "partition" "#" Identifier "#" ";"

Mode             ::= "FETCH"
                  |  "WHERE"
                  |  "INSERT"
                  |  "UPDATE"
                  |  "STRICT"
                  |  "OVERWRITE"
                  |  "STRICT-OVERWRITE"
                  |  "DELETE"
                  |  "FREEZE"
                  |  "UNFREEZE"
                  |  "RECYCLE"

Statement        ::= Condition ";"
                  |  CompoundCondition ";"
                  |  Document ";"
                  |  GetClause ";"
                  |  ExcludeClause ";"
                  |  UpdateClause ";"
                  |  StrictClause ";"
                  |  VerifyClause ";"
                  |  TargetClause ";"
                  |  RetrieveClause ";"
                  |  UseClause ";"
                  |  OrderClause ";"
                  |  Pagination ";"

Condition        ::= TypedField "::" TypedValue
CompoundCondition ::= Condition ( ( "||" | "&&" ) Condition )*   /* WHERE only */

Document         ::= JsonObject   /* plain JSON string, loaded with json.loads; no type wrappers */

TypedField       ::= Type "<" Identifier ">"

TypedValue       ::= Type "<" Literal ">"
                  |  VecLiteral

VecLiteral       ::= "vec<" TypedValue ("+" TypedValue)* ">"

Assignment       ::= TypedField "=" TypedValue

AssignmentList   ::= Assignment ("," Assignment)*

GetClause        ::= "GET(" FieldList ")"

ExcludeClause    ::= "EXCLUDE(" FieldList ")"

UpdateClause     ::= "UPDATE(" AssignmentList ")"

StrictClause     ::= "STRICT("
                        AssignmentList
                     ")"

VerifyClause     ::= "VERIFY("
                        AssignmentList
                     ")"

TargetClause     ::= "TARGET("
                        Field ","
                        Field
                     ")"

RetrieveClause   ::= "RETRIEVE("
                        Field ","
                        Field
                     ")"

UseClause        ::= "USE("
                        ClauseList
                     ")"

OrderClause      ::= "ORDER("
                        "ASCEND"
                     |  "DESCEND"
                     |  "RANDOM"
                     ")"

Pagination       ::= "START" "?" Integer
                  |  "LIMIT" "?" Integer
                  |  "START" "?" Integer ";" "LIMIT" "?" Integer

Type             ::= "int"
                  |  "float"
                  |  "bool"
                  |  "bytes"
                  |  "str"
                  |  "vec"
                  |  "any"

Comment          ::= "/" CommentText "/"

Lexical Rules

Every SNL query must conform to the language's lexical rules before execution. Queries that violate these rules are rejected during parsing and do not reach the execution pipeline.

Keyword Case

SNL keywords are case-insensitive. The following statements are therefore equivalent:

FETCH;

fetch;

Fetch;

Field names, however, are matched exactly as they exist within the database schema. Applications should therefore preserve the original field casing defined by the stored documents.

Comments

SNL supports inline comments so queries can be annotated without affecting execution. A comment begins and ends with a single unescaped forward slash (/). Everything between the two slashes is ignored by the parser, exactly as if it were whitespace.

FETCH;
/ match the demo account /
str<username>::str<john>;

Comments may appear on their own line between statements, or at the end of a line following a statement's terminating semicolon. They may not appear inside a type wrapper - a comment always closes the nearest unescaped /, so placing one inside <...> would split the literal value unexpectedly.

Escaping a Literal Slash

Because an unescaped / always opens or closes a comment, a literal forward slash that must appear inside a comment's text is escaped as \/:

/ paths look like a\/b\/c, not a comment inside a comment /

A literal / that appears inside a typed value (for example, a URL stored in a str<>) does not need escaping - comment parsing only applies outside type wrappers. See Reserved Characters & Escaping for the complete, context-by-context escaping rules.

Example - Annotated Query

/ locate the active admin account /
WHERE;
str<username>::str<john>;

/ only return public-safe fields /
GET(username, bio, profile_pic);

/ newest first, first page of 25 /
ORDER(DESCEND);
START ? 0; LIMIT ? 25;

Query Anatomy

Every SNL query follows a predictable top-down structure. A query begins with an optional Partition Clause, followed by an Execution Mode, followed by one or more clauses that refine how the operation should execute. Each statement is terminated with a semicolon (;), allowing the parser to process the query sequentially without ambiguity.

Unlike languages that rely on complex nested syntax, SNL uses independent execution clauses. Each clause has a single responsibility, making queries easy to read, validate, and execute.

General Query Structure

partition #accounts#;

FETCH;
str<username>::str<john>;

VERIFY(str<password>=str<secret>);

GET(username,email,age);

ORDER(ASCEND);

START ? 0, LIMIT ? 25;

Reading the query from top to bottom:

1. Optionally target a specific partition.
2. Select an execution mode.
3. Specify the record matching condition.
4. Apply optional behavioural clauses.
5. Project or filter returned fields.
6. Sort the result set if required.
7. Apply pagination.
8. Return the final response.

1. Partition Clause (Optional)

A query may open by targeting a specific partition with partition #name#;. When omitted, the query runs against the connection's default partition. See Partition Targeting for details.

2. Execution Mode

Every SNL query specifies an execution mode. The execution mode determines the primary action that the database should perform.

3. Record Selection

Most execution modes require a record selection statement immediately after the execution mode. This statement identifies which record or records the operation should target. Both the field and the value are typed:

str<username>::str<john>;

Record selection uses explicit type wrappers on both sides, allowing the engine to compare values without implicit type conversions and without quoting.

4. Behaviour Clauses

Behaviour clauses modify how the selected execution mode operates. Their availability depends on the chosen execution mode.

5. Projection Clauses

Projection clauses determine which fields appear in the final response without modifying the stored record.

6. Result Processing

After records have been located and processed, SNL can optionally sort and paginate the result set before returning it to the client.

Typical Execution Order

Regardless of execution mode, SNL evaluates queries in a deterministic sequence. The parser does not reorder clauses or generate hidden execution plans.

1. Validate query syntax.
2. Resolve the target partition, if specified.
3. Identify the execution mode.
4. Locate matching records.
5. Execute behavioural clauses.
6. Apply projection clauses.
7. Sort the result set.
8. Apply pagination.
9. Construct the response.

Partition Targeting

A shadeDB connection can host more than one named partition. By default, a query runs against the connection's default partition, but any query may explicitly target a different partition by opening with a Partition Clause - written as the very first statement, before the execution mode.

partition #product#;

FETCH;
str<sku>::str<sku-40231>;

The query above targets the product partition and fetches the record whose sku field equals "sku-40231", regardless of which partition the connection would otherwise default to.

Syntax

The partition name is wrapped between two literal # characters:

partition #partition_name#;

The keyword partition is case-insensitive like every other SNL keyword. The partition name itself is matched exactly against the partitions configured for the current database instance.

Behaviour

• Optional - omitting it runs the query against the default partition.
• Must be the first statement in the query, before the execution mode.
• Applies to the entire query that follows it, including any RECYCLE secondary stage that does not itself specify a partition.
• An unknown partition name causes the query to fail validation before execution begins, returning ERR_PARTITION_NOT_EXIST.

Maintenance & Ingestion Control

shadeDB separates ingestion control from partition maintenance. The instance-level control pauses the arrival of new records without interrupting normal work on existing data. The partition-level control is a hard lock that stops every mutation within one partition.

Control Model

How the Controls Interact

These controls are additive rather than hierarchical. Instance ingestion control applies to the entire instance, while partition maintenance adds a hard lock to its own partition. Enabling one does not cancel, weaken, or override the other.

Mutation Matrix

The following matrix is the authoritative behaviour for maintenance controls. A query is checked against the instance state first and the targeted partition state independently before the mutation reaches storage.

What Counts as “New Data”?

Instance ingestion control is concerned with new records entering the database. It therefore blocks INSERT;, but it does not turn existing-record operations into read-only operations. Updates may continue to change existing records, including fields introduced by an update, because the operation is still modifying an existing record rather than creating a new record.

Blocked Responses

Maintenance checks occur after parsing and authorization, but before storage mutation. Clients should branch on the stable error identifier rather than on the human-readable message.

{
    "status": "failed",
    "message": "ERR_INGESTION_PAUSED",
    "db_latency": "0.12ms"
}

A mutation rejected by partition maintenance returns ERR_MAINTENANCE_ACTIVE.

Operational Behaviour

• Neither control is an SNL clause; both are configured outside the query language.
• Reads (FETCH, *, RECYCLE) continue normally under both controls.
• Pause ingestion applies to every partition and blocks only new-record creation.
• Maintenance applies only to its selected partition and blocks every mutation there.
• Both controls may be active at the same time; their effects are cumulative by scope.
• Instance pause clears automatically when the instance next goes inactive and sleeps.
• Partition maintenance remains active until an operator explicitly disables it.
• Toggling either control requires dashboard access and password confirmation; SNL clients cannot change maintenance state through a connection token.

Connecting to Your Partition Cluster

Every shadeDB partition exposes a dedicated connection endpoint that applications use to communicate with the cluster. To establish a connection, three values are required:

• Partition Endpoint - The HTTPS endpoint assigned to your partition.
• Connection Token - Authenticates the client making requests.
• Cluster Token - Identifies the target cluster and authorizes access.

These credentials are available from the shadeDB dashboard after creating a partition. Treat both tokens as confidential credentials and never expose them in public repositories or client-side applications.

Install the Official Python Client

The official shadeDB Python client is distributed through PyPI. Install it before connecting to your partition:

pip install shadedb-api

Official package: https://pypi.org/project/shadedb-api/

Step 1 - Obtain Your Connection Details

From the shadeDB dashboard, open the target partition and copy the following values:

• Partition Endpoint
• Connection Token
• Cluster Token

One-Time Connection

To connect without saving the configuration locally, execute:

shadedb-api CONNECTION_ENDPOINT CONNECTION_TOKEN CLUSTER_TOKEN

Example:

shadedb-api http://host.com/connect/sdb7f585246acf1480b9caac05f811c3 
8da846a452f13d1d4cf4ddfac656a115 
8c2ff6b8c408f68

This establishes a temporary connection for the current session only. The credentials must be supplied again the next time the client starts.

Persistent Initialization

To save the connection information locally, initialize the client once using:

shadedb-api-init CONNECTION_ENDPOINT CONNECTION_TOKEN CLUSTER_TOKEN

Example:

shadedb-api-init http://host.com/connect/sdb7f585246acf1480b9caac05f811c3 
8da846a452f13d1d4cf4ddfac656a115 
8c2ff6b8c408f68

The initialization command stores the connection configuration securely for subsequent sessions.

Future Sessions

Once initialization has completed successfully, reconnecting is as simple as executing:

shadedb-api

The client automatically loads the previously saved endpoint and authentication tokens, allowing immediate access to the configured partition cluster.

Not using Python? See Building Your Own Client.

Building Your Own Client

shadedb-api is the only packaged client today. If you use another language, or want the client inside your own service, talk to the partition endpoint directly. It is one JSON POST.

The request

Send a JSON body to the Partition Endpoint with these fields:

• token - your Cluster Token
• cntoken - your Connection Token
• SNL - the query string

The response

Responses are JSON. Failures also return HTTP 200, so check the status field: failed means the query did not run, and message says why. A missing field returns "Missing cluster token", "Missing connection token" or "Missing query". HTTPS works on every plan; TCP and UDP are available on paid plans.

Python client

Standard library only. Replace the placeholders with your own values.

import json
import urllib.request


class ShadeClient:
    """Minimal shadeDB client. Keep your tokens on the server."""

    def __init__(self, endpoint, cluster_token, connection_token, timeout=10):
        self.endpoint = endpoint
        self.cluster_token = cluster_token
        self.connection_token = connection_token
        self.timeout = timeout

    def query(self, snl):
        body = json.dumps({
            "token": self.cluster_token,
            "cntoken": self.connection_token,
            "SNL": snl,
        }).encode()
        req = urllib.request.Request(
            self.endpoint,
            data=body,
            headers={"Content-Type": "application/json"},
            method="POST",
        )
        with urllib.request.urlopen(req, timeout=self.timeout) as res:
            data = json.loads(res.read())
        if isinstance(data, dict) and data.get("status") == "failed":
            raise RuntimeError(data.get("message", "query failed"))
        return data


db = ShadeClient("https://HOST/connect/INSTANCE_ID", "YOUR_CLUSTER_TOKEN", "YOUR_CONNECTION_TOKEN")
print(db.query("EXIST;\nstr<username>::str<shade>;"))

JavaScript client

For Node 18+, Deno or Bun, on the server side.

// Node 18+, Deno or Bun. Run it on your server, never in a public web page.
class ShadeClient {
  constructor(endpoint, clusterToken, connectionToken) {
    this.endpoint = endpoint;
    this.clusterToken = clusterToken;
    this.connectionToken = connectionToken;
  }

  async query(snl) {
    const res = await fetch(this.endpoint, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        token: this.clusterToken,
        cntoken: this.connectionToken,
        SNL: snl,
      }),
    });
    const data = await res.json();
    if (data.status === "failed") throw new Error(data.message);
    return data;
  }
}

const db = new ShadeClient("https://HOST/connect/INSTANCE_ID", "YOUR_CLUSTER_TOKEN", "YOUR_CONNECTION_TOKEN");
console.log(await db.query("EXIST;\nstr<username>::str<shade>;"));

Raw request

curl -X POST "https://HOST/connect/INSTANCE_ID" \
  -H "Content-Type: application/json" \
  -d '{"token": "YOUR_CLUSTER_TOKEN", "cntoken": "YOUR_CONNECTION_TOKEN", "SNL": "YOUR_QUERY"}'

Dashboard Overview

The shadeDB dashboard is where a database instance is provisioned and managed day to day - connection credentials, partitions, the cache pool, and clusters (access tokens). It is distinct from SNL itself: nothing on the dashboard is a query language feature, and none of it is reachable through a connected client. It is reached by signing in to the shadeDB web console and opening an instance.

Instance Metrics

The sticky header bar at the top of an instance's dashboard page summarizes its current state at a glance:

A small dot next to the shadeDB logo reflects whether the dashboard's live event stream is currently connected - it turns solid once the stream is established and reconnects automatically if the connection drops. See Notifications.

Connection Status & Danger Banners

The instance name is shown alongside a connection-status badge and a "Last updated" timestamp for the page's own data. If the instance's subscription has lapsed, an expired banner appears; if instance ingestion is paused or the currently selected partition is under maintenance, a corresponding banner appears as well. See Maintenance & Ingestion Control.

Connection Endpoint & API Connection Token

The Connection Endpoint panel shows the HTTPS endpoint official clients connect to, with a one-click copy button. Below it, the API connection token is hidden by default - use "Show connection token" to reveal it, then copy it as needed. See Connecting to Your Partition Cluster for how these values are used together with a Cluster Token to establish a connection.

Managing Partitions

Partitions are created and configured from the dashboard; SNL only ever selects one with Partition Targeting. Each partition gets its own storage path and disk, identified by its name - there is no separate partition id.

Creating a Partition

"+ New" opens a form for two things:

• Partition name - must be unique on the instance; this is how the partition is later targeted from SNL. The name is permanent once the partition is created and cannot be changed afterward.
• Unique integrity constraints - the fields that must hold a unique value per record in this partition. id is always included and cannot be removed; additional fields (e.g. email, uuid) can be added as tags before creating the partition.

Partition Status at a Glance

A dropdown at the top of the panel switches which partition's details the sidebar is currently showing; this only changes what the dashboard displays and has no effect on clusters, which stay instance-wide.

Editing a Partition

"Edit partition" allows adding or removing unique fields.

Danger Zone

• Maintenance - places this partition under a hard mutation lock; see Maintenance & Ingestion Control.
• Delete partition - permanently deletes the partition's disk, metadata, and unique indexes. Clusters are instance-wide and are not deleted, but any data stored on that partition is gone.

A brand-new instance with no partitions shows "Your database instance is still asleep" - at least one partition must be created before the instance can accept any records.

Cache Pool

The cache pool is shadeDB's in-memory layer for frequently accessed records. It is instance-wide - unlike partitions, it is not scoped to a single partition - and its purpose is purely to speed up reads; it holds no data that isn't also persisted through the normal write path described in Pipeline Mechanics.

Status Thresholds

Reported Stats

• Used - percentage of the pool's allocated size currently occupied.
• Density - number of keys currently cached.
• Memory - the actual memory consumed, shown in KB or MB.

The cache pool can be purged on demand from Instance Settings - this forces the next queries to read from disk until the cache repopulates.

Clusters & Access Tokens

A "cluster" on the dashboard is a named access token: a secret credential with its own permission set, optional query limit, and optional expiry date. This is the Cluster Token referenced in Connecting to Your Partition Cluster - it, along with the connection endpoint and connection token, is what an application uses to authenticate against the instance.

Creating a Cluster

"+ Cluster" opens a form with:

• Token name / alias - a human-readable label for the credential.
• Permissions - one or more of READ, WRITE, UPDATE, DELETE.
• Query limit - an optional cap on queries for this token; immutable once set, defaulting to unlimited if left blank.
• Expiry - an optional date after which the token stops working; defaults to Never.

On creation, the full secret key is shown exactly once, alongside the name, query limit, and expiry that were set. Copy it immediately - the dialog states it will not be shown in full again once closed.

Working With an Existing Cluster

• Show/Hide secret key - reveals the token's secret on its card, with a copy button.
• Edit config - lets you rename the token and change its permissions. Query limit and expiry are shown but disabled - both are immutable after creation.
• Terminate - permanently deletes the token after password confirmation. Any client using it stops connecting immediately.
• Search - the search box above the cluster grid filters by name as you type.

Each cluster card also shows live-ish stats - current QPS, average throughput, replica count, and its assigned permissions - so token behaviour can be sanity checked without leaving the dashboard.

Instance Settings

"Configure" on the bottom dock opens instance-level settings - separate from the per-partition settings in Managing Partitions.

Profile

The instance's display name can be renamed here; this is cosmetic and does not change the instance's connection endpoint or any tokens.

Danger Zone

Every action below requires password confirmation before it takes effect:

• Pause ingestion - pauses new-record ingestion across every partition until the instance next goes inactive and sleeps. Existing-record mutations remain available. See Maintenance & Ingestion Control for the full behaviour and its distinction from partition-level maintenance.
• Clear cache pool - purges the instance's cache pool immediately, forcing the next queries to read from disk until the cache repopulates.
• Delete instance - irreversibly deletes the instance itself: every partition, every stored record, and every cluster token. There is no SNL-level or dashboard-level way to undo this.

Notifications

The bell icon in the bottom-right corner surfaces events related to this instance - for example, changes to a partition's state - pushed to the dashboard in real time rather than requiring a page refresh.

• An unread count badge appears on the bell when new events arrive while the panel is closed.
• Opening the panel lists events newest-first; "Load more" pages through older ones.
• Each unread event has a "Mark as read" action.
• The small status dot next to the shadeDB logo in the header reflects whether this real-time connection is currently active; it reconnects automatically if interrupted.

The Wildcard (*) Clause

The *; clause performs a wildcard search across an entire database partition. Unlike FETCH;, which evaluates records against a specific field-value condition, the wildcard ignores record schemas and matching conditions, treating every record in the partition as a potential result.

Because wildcard searches are schema-independent, records are not required to share the same fields or document structure. Every visible record is considered during execution regardless of its individual schema.

*;

The query above performs a full-partition scan and returns every visible record stored within the database partition.

Supported Clauses

Wildcard queries support the same result-shaping clauses used by FETCH;.

Example

*;

GET(username,email);

ORDER(DESCEND);

START ? 0, LIMIT ? 50;

This query searches every record in the partition, projects only the username and email fields where available, sorts the results in descending order, and returns the first fifty records.

FETCH Mode

FETCH; is the primary retrieval mode in SNL. It searches the database for records matching the supplied condition and establishes the working dataset for the remainder of the query. By default, FETCH returns the matching records without modifying them.

FETCH also serves as the execution context for several optional clauses. After the requested records have been located, additional clauses may update, validate, verify, or shape the returned result before execution completes.

Basic Syntax

FETCH;
str<username>::str<john>;

The above query searches for every record whose username field equals "john".

Execution Flow

1. Parse the FETCH statement.
2. Locate all matching records.
3. Execute any supplied FETCH clause.
4. Return the final result.

Supported FETCH Clauses

FETCH supports several optional clauses that extend its behavior. Each clause executes after matching records have been found.

Examples

Basic Retrieval

FETCH;
str<username>::str<john>;

Returns every record whose username is john.

Updating a Retrieved Record

FETCH;
str<username>::str<john>;
UPDATE(str<age>=int<19>);

Finds the matching record and updates its age field to 19.

Verifying Credentials

FETCH;
str<username>::str<john>;
VERIFY(str<password>=str<input_password>);

Compares the supplied password with the stored password and returns only true or false.

Returning Selected Fields

FETCH;
str<username>::str<john>;
GET(username,bio,profile_pic);

Returns only the specified fields from the matching record.

Excluding Sensitive Fields

FETCH;
str<username>::str<john>;
EXCLUDE(password,auth_token);

Returns the matching record while omitting the specified fields.

WHERE Mode

WHERE; is SNL's compound-filter retrieval mode. It evaluates one or more typed conditions against each record in the partition and returns every record that satisfies them. Where FETCH; matches a single field-value pair, WHERE; lets you combine several conditions in one statement using explicit || (OR) and && (AND) operators.

Conditions are evaluated sequentially against record fields, including fields that are not indexed. This makes WHERE; the right tool for searching by attributes such as age, status, or category, rather than by a unique identifier.

Basic Syntax

WHERE;
int<age>::int<18>||str<gender>::str<male>;

The query above returns every record whose age is 18 or whose gender is "male". Each condition is a standard typed pair, exactly like the record selection used by FETCH;.

Condition Operators

Execution Flow

1. Parse the WHERE; statement and its compound condition.
2. Evaluate each typed condition against every record in the partition.
3. Combine the results using the || or && operator.
4. Apply projection using GET() or EXCLUDE(), if present.
5. Apply ordering.
6. Apply pagination.
7. Return the final result.

Ordering is applied before pagination, so LIMIT always counts from the sorted result set, never from an arbitrary subset of matches.

Supported Clauses

WHERE; supports the same result-shaping clauses as FETCH; and the wildcard.

VERIFY() and atomic UPDATE() are FETCH sub-clauses and are not available in WHERE; queries.

Examples

OR Filter with Projection and Pagination

WHERE;
int<age>::int<18>||str<gender>::str<male>;

GET(username,bio,id);

ORDER(DESCEND);

START ? 0, LIMIT ? 500;

Returns records where the age is 18 or the gender is male, projecting only username, bio, and id, sorted in descending order, and returning the first five hundred matches.

Random Sample

WHERE;
int<age>::int<18>||str<gender>::str<male>;

ORDER(RANDOM);

LIMIT ? 500;

Shuffles the matching records and returns up to five hundred of them, useful for sampling or shuffled feeds.

AND Filter

WHERE;
int<age>::int<47>&&str<gender>::str<male>;

GET(username,bio,profile_pic);

ORDER(RANDOM);

START ? 0, LIMIT ? 50;

Returns only records where the age is 47 and the gender is male, projecting three fields in random order and returning at most fifty records.

WHERE vs FETCH vs Wildcard

Behaviour

• Requires at least one typed condition immediately after WHERE;.
• Both the field and value of every condition are typed.
• Conditions are joined with || (OR) or && (AND).
• Fields that are not indexed can be filtered.
• Supports GET(), EXCLUDE(), ORDER(), and pagination.
• Does not modify any stored data.

EXIST Mode

EXIST; checks whether a record already exists. It returns True if a matching record is found and False otherwise. Use it to verify beforehand, for example before creating a record that must not be duplicated.

Basic Syntax

EXIST;
str<username>::str<shade>;

The query above returns True if a record whose username field equals "shade" exists, and False if it does not.

Execution Flow

1. Parse the EXIST statement.
2. Look for a record matching the supplied condition.
3. Return True if one is found, otherwise False.

Behaviour

• The result is only True or False. No record fields are returned.
• EXIST does not change any record.
• Use FETCH when you need the record itself.

Result

Example

Check before creating a record

EXIST;
str<username>::str<shade>;

If this returns False, the username is free and the record can be created with INSERT.

INSERT Mode

INSERT; creates a new record in the targeted partition. The document body is a plain JSON string with no type wrappers; the engine loads it with json.loads. Every insert is appended to the Write-Ahead Log (WAL) before it is synchronized with persistent storage, so inserted records are recoverable after an unexpected crash.

Basic Syntax

INSERT;
{
    "username": "john",
    "email": "john@example.com"
};

Inserting a Vector Field

Arrays are written as native JSON arrays inside the document body:

INSERT;
{
    "username": "john",
    "tags": ["beta", "verified", 3]
};

Here tags is stored as a three-element vector containing two strings and an integer. See Type System Wrappers for the full vec<> reference.

Possible Responses

UPDATE Mode

UPDATE; modifies existing records by updating only the fields provided in the update payload. Unlike OVERWRITE;, UPDATE performs a partial modification, preserving every field that is not explicitly changed.

The engine searches for all records matching the supplied query and applies the specified changes to each matching record. If multiple records satisfy the query, each record is updated independently.

UPDATE;
str<username>::str<john>;

{
    "email": "new@example.com"
};

The query above searches for every record whose username is "john" and updates only the email field. Every other field remains unchanged.

Execution Flow

1. Parse the UPDATE statement.
2. Locate all matching records.
3. Apply the supplied update payload.
4. Write the modified records back to storage.
5. Return the updated records.

Modes

Default Mode

In its default mode, UPDATE applies every field contained in the supplied payload. If a field does not already exist in the record, it is added to the schema.

Strict update: STRICT;

STRICT;
str<username>::str<john>;
{
    "email": "new@example.com",
    "role": "admin"
};

STRICT; replaces UPDATE; with strict schema protection (unknown fields such as role are discarded).

STRICT; protects the existing schema by preventing unknown fields from being introduced. During execution, any field that does not already exist in the target record is silently discarded, while recognised fields are updated normally.

Multi-Field Update

The atomic UPDATE() sub-clause (used from within FETCH) accepts multiple typed assignments in one call (the top-level UPDATE; body is plain JSON instead):

UPDATE(str<username>=str<shade>,str<age>=int<22>);

Behaviour Summary

• Updates only the specified fields.
• Leaves all unspecified fields unchanged.
• Can update one or multiple matching records.
• Supports STRICT; mode for schema protection.
• Use OVERWRITE; when replacing an entire record instead of modifying individual fields.

OVERWRITE Mode

OVERWRITE; replaces the entire contents of every record matching the supplied query with a new document. Unlike UPDATE;, which modifies individual fields, OVERWRITE discards the existing document and stores only the fields present in the supplied payload.

Any field omitted from the new document is permanently removed from the record. For this reason, OVERWRITE should be used when the complete record structure is known and intended to replace the existing one.

OVERWRITE;
str<username>::str<john>;

{
    "username": "john",
    "email": "new@example.com"
};

The query above searches for every record whose username is "john" and replaces the existing record with the supplied document. Any previous fields not included in the new document are removed.

Execution Flow

1. Parse the OVERWRITE statement.
2. Locate every matching record.
3. Discard the existing document.
4. Store the supplied document as the new record.
5. Return the overwritten record(s).

Modes

Default Mode

In its default mode, OVERWRITE replaces the existing record with the supplied document exactly as provided. Fields omitted from the payload are removed, while new fields are introduced into the record.

Strict overwrite: STRICT-OVERWRITE;

STRICT-OVERWRITE; preserves the existing schema during an overwrite operation. Any field in the supplied document that does not already exist within the target record is silently discarded before the overwrite is completed. Existing fields continue to be replaced normally.

STRICT-OVERWRITE;

str<username>::str<john>;

{
    "username": "john",
    "email": "new@example.com",
    "role": "admin"
};

If the existing record does not contain the role field, it is discarded before the overwrite is applied. The resulting record contains only recognised fields, preventing unintended schema expansion.

Behaviour Summary

• Replaces the entire contents of every matching record.
• Fields omitted from the payload are removed.
• Can overwrite one or multiple matching records.
• Supports STRICT-OVERWRITE; to prevent schema expansion.
• Use UPDATE; when only specific fields need to be modified.

DELETE Mode

DELETE; permanently removes a record from the database using a unique field. To prevent accidental data loss, DELETE only accepts queries against fields declared as unique. Queries targeting non-unique fields are rejected and no records are removed.

This design guarantees that every DELETE operation targets at most one record, eliminating the risk of unintentionally deleting multiple records that share the same value.

DELETE;
str<username>::str<john>;

Assuming username is defined as a unique field, the query above permanently removes the matching record from the database.

Execution Flow

1. Parse the DELETE statement.
2. Verify that the queried field is unique.
3. Locate the matching record.
4. Remove the record from storage.
5. Update indexes and metadata.
6. Return the operation result.

Behaviour

• Only unique fields may be used as DELETE conditions.
• At most one record can be deleted per operation.
• Queries against non-unique fields are rejected.
• If no matching record exists, nothing is deleted.
• Deleted records cannot be recovered through SNL.

Example

DELETE;

str<id>::int<42>;

If id is a unique field, the matching record is permanently removed.

FREEZE & UNFREEZE Modes

FREEZE; and UNFREEZE; provide a reversible way to hide records without permanently deleting them. A frozen record remains physically stored in the database but becomes inaccessible to normal database operations until it is explicitly unfrozen.

Once frozen, a record is excluded from standard retrieval operations and cannot be modified or deleted. This allows records to be temporarily archived, quarantined, or disabled while preserving their original contents.

FREEZE Syntax

FREEZE;
str<id>::int<56>;

The query above freezes the record whose unique id is 56. The record remains stored but is hidden from subsequent queries and protected against modification or deletion.

UNFREEZE Syntax

UNFREEZE;
str<id>::int<56>;

This restores the previously frozen record, making it visible to queries and allowing normal database operations to resume.

Execution Flow

FREEZE

1. Validate the supplied unique key.
2. Locate the matching record.
3. Mark the record as frozen.
4. Hide the record from normal queries.
5. Prevent future updates and deletions until unfrozen.

UNFREEZE

1. Validate the supplied unique key.
2. Locate the frozen record.
3. Verify the original freeze identifier.
4. Remove the frozen state.
5. Restore normal database access.

Behaviour

• Frozen records remain physically stored.
• Frozen records are hidden from normal FETCH operations.
• Frozen records cannot be updated.
• Frozen records cannot be overwritten.
• Frozen records cannot be deleted.
• UNFREEZE completely restores the original record.

RECYCLE Mode

RECYCLE; is SNL's multi-stage execution engine. Rather than requiring applications to perform multiple database requests and manually combine their results, RECYCLE executes an initial query, extracts one or more values from its result, and immediately reuses those values to perform a secondary query against another database partition.

RECYCLE is designed for traversing relationships between partitions, making it ideal for retrieving related data such as users and their messages, customers and their orders, blog posts and their comments, or any other linked datasets-all within a single SNL request.

RECYCLE;

FETCH;
str<id>::int<my_id>;

TARGET(message_composed_by_userid, user_id);
USE(FETCH/WHERE);

GET(message, created_at);

ORDER(ASCEND);
START ? 0, LIMIT ? 100;

In this example, the engine first retrieves the user identified by my_id. The value stored in the user's user_id field is then recycled into another partition, where it is matched against the message_composed_by_userid field. Only the message and created_at fields are returned from each recycled record.

Execution Flow

1. Execute the initial query.
2. Retrieve the matching record.
3. Extract the required field values.
4. Map those values through TARGET() or RETRIEVE().
5. Execute the secondary query specified by USE().
6. Apply response projection using GET() or EXCLUDE(), if present.
7. Apply ordering and pagination.
8. Return the final result.

Execution Modes

RECYCLE supports two execution modes depending on how the final response should be structured.

Conjoined Mode

Conjoined Mode is enabled through TARGET(). During execution, the parent record obtained from the initial query is merged into every matching secondary record, producing a unified dataset containing information from both partitions.

TARGET(message_composed_by_userid, user_id);

This mode is particularly useful when every related record should include information about its parent without requiring another database request.

Retrieve-Only Mode

Retrieve-Only Mode is enabled through RETRIEVE(). The extracted values are used solely for filtering the secondary partition. The original parent record is omitted from the final response, returning only the related records.

RETRIEVE(message_composed_by_userid, user_id);

This mode is useful when only the destination records are required and embedding the parent record would unnecessarily increase the response size.

Supported Clauses

Advantages

• Performs relationship traversal in a single SNL request.
• Eliminates multiple client-side database queries.
• Reduces network latency by executing multi-stage operations inside the engine.
• Supports both conjoined and retrieve-only execution modes.
• Supports response projection using GET() and EXCLUDE().
• Supports ordering and efficient pagination.

The GET Clause

GET(...) is a projection clause that limits the fields returned by a query. Instead of returning the complete record, GET extracts only the specified attributes, reducing response size and preventing unnecessary data from being transmitted.

GET is particularly useful when applications require only a small subset of a record, such as displaying usernames, profile pictures, or email addresses, without retrieving the remaining fields.

FETCH;
str<username>::str<john>;

GET(
    username,
    email
);

The query above searches for the record whose username is "john" and returns only the username and email fields.

How GET Works

1. Execute the parent query.
2. Locate the matching record or records.
3. Extract only the fields listed inside GET().
4. Discard every other field from the response.
5. Return the projected result.

Behaviour

• Returns only the requested fields.
• Preserves the original record stored in the database.
• Reduces response payload size.
• Can return one or multiple fields.
• Can be combined with any FETCH operation that returns records.

Example

FETCH;

str<id>::int<12>;

GET(
    username,
    profile_pic,
    bio,
    verified
);

Only the four requested fields are included in the response. All remaining attributes, such as passwords, authentication tokens, timestamps, and other stored values, are omitted.

STRICT Modes

"Strict" behaviour in SNL comes in three forms. Two are top-level execution modes that protect a record's schema during a write; the third is a FETCH sub-clause that validates fields before continuing.

STRICT; (strict update)

STRICT;
str<username>::str<john>;
{
    "email": "new@example.com",
    "role": "admin"
};

STRICT; replaces the former UPDATE; STRICT; pair. Fields that already exist (email) are updated; fields that do not (role) are discarded. No error is returned.

STRICT-OVERWRITE; (strict overwrite)

STRICT-OVERWRITE;
str<username>::str<john>;
{
    "username": "john",
    "email": "new@example.com",
    "role": "admin"
};

STRICT-OVERWRITE; replaces the former OVERWRITE; STRICT; pair. The record is replaced, but any field not already in its schema is dropped from the new document first.

STRICT(...) in FETCH

FETCH;
str<username>::str<john>;

STRICT(str<age>=int<18>);

Retrieves the record and verifies the age field exists before continuing. If it is missing, execution stops with a validation error. The record itself is never modified.

Why Use STRICT?

• Protects the database schema from unintended expansion.
• Prevents typos from creating unexpected fields.
• Makes schema-safe writes a single, explicit mode instead of a modifier.

The TARGET Clause

TARGET(...) is a RECYCLE clause that establishes the relationship between the primary query and a secondary database partition. It maps a field from the secondary partition to a value extracted from the initial record, allowing the engine to locate related records automatically.

Unlike RETRIEVE(), TARGET performs a conjoined execution. After the related records have been found, the fields from the initial record are merged into every matching secondary record, producing a unified result.

RECYCLE;

FETCH;
str<id>::int<my_id>;

TARGET(message_composed_by_userid, user_id);
USE(FETCH/WHERE);

START ? 0, LIMIT ? 100;

In the example above, the initial query retrieves a user using my_id. The value stored in the user's user_id field is then mapped to the message_composed_by_userid field in the secondary partition, allowing the engine to retrieve every message composed by that user.

How TARGET Works

1. Execute the initial FETCH query.
2. Extract the mapped value from the returned record.
3. Match that value against the target field specified in TARGET().
4. Retrieve every matching record from the secondary partition.
5. Merge the original parent record into each matching result.
6. Return the conjoined dataset.

Syntax

TARGET accepts two bare field-name arguments (these are not type-wrapped - they name fields, not values):

• First argument - The field in the secondary partition that should be searched.
• Second argument - The field extracted from the initial record whose value will be recycled.

For example:

TARGET(message_composed_by_userid, user_id);

Here, user_id is obtained from the first query, while message_composed_by_userid belongs to the secondary partition.

Behaviour

• Available only within RECYCLE;.
• Automatically maps values between related partitions.
• Merges the parent record into every matching secondary record.
• Supports one-to-one and one-to-many relationships.
• Can be combined with USE(), ORDER(), and START ? x, LIMIT ? y;.

Example Use Cases

• Retrieve every message written by a user.
• Retrieve all comments belonging to a post.
• Retrieve every order placed by a customer.
• Retrieve every session belonging to an account.
• Retrieve child records while embedding their parent information.

The RETRIEVE Clause

RETRIEVE(...) is a RECYCLE clause used to establish relationships between database partitions without merging the parent record into the final response. It extracts a value from the initial query, maps it to a field in the secondary partition, and returns only the matching secondary records.

Unlike TARGET(), RETRIEVE performs a filter-only operation. The initial record is used solely as a source of values for the secondary query and is omitted entirely from the returned payload.

RECYCLE;

FETCH;
str<id>::int<my_id>;

RETRIEVE(follower_id, id);
USE(FETCH/WHERE);

START ? 0, LIMIT ? 100;

In the example above, the engine first retrieves the record identified by my_id. The value of its id field is then recycled into the secondary partition, where it is matched against the follower_id field. Only the matching follower records are returned.

How RETRIEVE Works

1. Execute the initial FETCH query.
2. Extract the mapped value from the returned record.
3. Match that value against the destination field specified in RETRIEVE().
4. Retrieve every matching record from the secondary partition.
5. Return only the secondary records.

Syntax

RETRIEVE accepts two bare field-name arguments:

• First argument - The field in the secondary partition that will be searched.
• Second argument - The field from the initial record whose value will be recycled.

For example:

RETRIEVE(follower_id, id);

Here, the value of id obtained from the first query is matched against the follower_id field in the secondary partition.

Behaviour

• Available only within RECYCLE;.
• Maps values between related database partitions.
• Returns only the secondary records.
• Does not merge the parent record into the response.
• Supports one-to-one and one-to-many relationships.
• Can be combined with USE(), ORDER(), and START ? x, LIMIT ? y;.

Example Use Cases

• Retrieve a user's followers.
• Retrieve all comments belonging to a post.
• Retrieve every order placed by a customer.
• Retrieve every file owned by a workspace.
• Return related records without duplicating parent information.

The USE Clause

USE(...) controls which clauses from the initial FETCH; statement are inherited and executed during the secondary phase of a RECYCLE; operation. Rather than automatically applying every compatible clause, USE allows developers to explicitly specify which query behaviour should continue into the recycled execution.

This provides fine-grained control over the secondary query, ensuring that only the desired filtering and projection logic is carried forward while ignoring unnecessary clauses.

USE(
    FETCH,
    WHERE
);

The example above instructs the RECYCLE engine to inherit only the FETCH and WHERE clauses during the secondary execution. Any compatible clauses not listed inside USE() are ignored for that recycle stage.

How USE Works

1. Execute the initial FETCH; query.
2. Recycle the required values using TARGET() or RETRIEVE().
3. Read the clauses listed inside USE().
4. Apply only those clauses to the secondary query.
5. Continue execution and return the recycled result.

Behaviour

• Available only within RECYCLE;.
• Controls which compatible clauses are inherited by the secondary query.
• Allows multiple clauses to be specified.
• Improves control over recycled query execution.
• Prevents unnecessary clauses from affecting the secondary query.

Example

RECYCLE;

FETCH;
str<id>::int<my_id>;

TARGET(message_composed_by_userid, user_id);

USE(
    FETCH,
    WHERE
);

ORDER(ASCEND);
START ? 0, LIMIT ? 100;

In this example, only the FETCH and WHERE clauses are inherited during the recycled execution. The engine ignores any other compatible clauses that are not explicitly listed inside USE().

The EXCLUDE Clause

EXCLUDE(...) is a response projection clause that removes specified fields from the returned result while leaving every other field intact. Unlike GET(), which explicitly selects the fields to return, EXCLUDE() returns the complete record except for the fields listed inside the clause.

EXCLUDE is commonly used to omit sensitive or unnecessary information such as passwords, authentication tokens, API keys, internal identifiers, or other metadata that should not be exposed to the client.

FETCH;
str<username>::str<john>;

EXCLUDE(
    password,
    api_key
);

The query above retrieves the record whose username is "john" and returns every field except password and api_key.

How EXCLUDE Works

1. Execute the parent query.
2. Retrieve the matching record or records.
3. Remove every field listed inside EXCLUDE() from the response.
4. Return the remaining fields.

Behaviour

• Returns all fields except those explicitly excluded.
• Supports excluding one or multiple fields.
• Does not modify the stored record.
• Reduces response size by omitting unnecessary data.
• Can be used with any FETCH or RECYCLE operation that returns records.

Example

FETCH;

str<id>::int<15>;

EXCLUDE(
    password,
    auth_token,
    recovery_key,
    internal_id
);

Every field belonging to the matched record is returned except password, auth_token, recovery_key, and internal_id.

GET() vs EXCLUDE()

The ORDER Clause

ORDER(...) controls the sequence in which matching records are returned. Rather than sorting by a specific field, ORDER determines whether the result set should be returned in ascending, descending, or randomized order.

ORDER is commonly used alongside START and LIMIT to control pagination and the presentation of query results.

Ascending Order

ORDER(ASCEND); returns matching records in ascending order.

ORDER(ASCEND);

Descending Order

ORDER(DESCEND); returns matching records in descending order.

ORDER(DESCEND);

Random Order

ORDER(RANDOM); randomizes the returned result set. Each execution may produce a different ordering, making it useful for recommendations, featured content, random sampling, or shuffled feeds.

ORDER(RANDOM);

Behaviour

• Supports ASCEND, DESCEND, and RANDOM.
• Can be used with FETCH and RECYCLE queries.
• Works seamlessly with START and LIMIT for pagination.
• RANDOM produces a different ordering on successive executions.
• Affects only the order in which records are returned.

Examples

Ascending

WHERE;

ORDER(ASCEND);

START ? 0, LIMIT ? 20;

Descending

WHERE;

ORDER(DESCEND);

START ? 0, LIMIT ? 20;

Random

WHERE;

ORDER(RANDOM);

LIMIT ? 10;

The START & LIMIT Clauses

START and LIMIT control pagination within SNL queries. They determine where the returned result set begins and how many records are returned. Each keyword is followed by ? and then its integer value. The clauses may be used independently or combined into a single statement.

These clauses are commonly used with FETCH and RECYCLE queries to efficiently browse large collections of records.

Combined Syntax

START ? 20, LIMIT ? 50;

The query above skips the first 20 matching records and returns at most the next 50.

START Only

START may be used on its own to specify the zero-based offset from which matching records should begin to be returned.

START ? 100;

LIMIT Only

LIMIT may also be used independently to restrict the maximum number of records returned, beginning from the default starting offset of zero.

LIMIT ? 25;

Behaviour

• START specifies the first matching record to return.
• LIMIT specifies the maximum number of returned records.
• Both keywords require a ? before their integer argument.
• The clauses may be used individually or together.
• When combined, the syntax is START ? x; LIMIT ? y;.
• Supported by both FETCH and RECYCLE.
• Can be combined with ORDER() for predictable pagination.
• Neither clause modifies the underlying database records.

Examples

Type System Wrappers

SNL is a strongly typed query language. Every field reference and every value supplied to the engine is explicitly wrapped with its data type, eliminating the need for runtime type guessing, or implicit conversions. (Document bodies are the exception: they are plain JSON.) This allows the engine to parse queries predictably while preserving the original data type throughout execution.

Unlike traditional query languages that infer types from raw literals or rely on quoting to distinguish strings, SNL requires each field and each value to declare its intended type using a wrapper. This removes ambiguity and ensures comparisons, updates, and validations are performed against values of the correct type.

Supported Wrappers

Examples

str<age>::int<25>;

str<price>::float<19.99>;

str<verified>::bool<true>;

str<username>::str<john>;

str<avatar>::bytes<binary_data>;

str<shade>::any<shade>;

Typed Fields - Everything Is Typed

SNL v2 removed the old bare-field condition syntax. Both sides of a typed pair are now wrapped - the field name on the left, the value on the right - joined by :: at the top level, or by = inside a sub-clause argument list such as UPDATE(), STRICT(), or VERIFY():

/ top-level condition, uses :: /
str<username>::str<john>;

/ sub-clause assignment, uses = /
UPDATE(str<username>=str<shade>, str<age>=int<22>);

Field names are, in practice, always wrapped as str<...> since field names are string identifiers - the grammar allows any type wrapper on the field side, but str<> is the expected convention.

Vectors (vec<>)

vec<...> stores an ordered array whose elements may each carry a different type. Elements are separated with + instead of ,, since , is already reserved for separating typed pairs and clause arguments elsewhere in the grammar:

str<mixed_tags>::vec<int<77>+str<hello>+any<world>>;

This declares mixed_tags as a three-element vector: an integer (77), a string ("hello"), and an untyped value ("world"). A vector element may itself be another vec<>, allowing nested arrays.

The any<> Type

any<...> stores or matches a value without enforcing a fixed type. It is useful for fields whose type genuinely varies between records, or where the caller does not need - or does not know - the underlying type ahead of time. Values stored with any<> are preserved exactly as submitted.

Document Bodies (JSON Strings)

The { ... } body of INSERT;, UPDATE;, STRICT;, OVERWRITE; and STRICT-OVERWRITE; is a plain JSON string. It uses no type wrappers; the engine loads it with json.loads, so values keep their native JSON types (string, number, boolean, array, null).

INSERT;\1,
    "verified": true,
    "tags": ["beta", "verified", 3]
};

Type wrappers still apply to conditions (str<username>::str<john>) and to sub-clause assignments (str<age>=int<19>).

Whitespace Rule

Whitespace outside a <...> wrapper is insignificant and is trimmed by the parser - extra spaces around ::, =, ,, or clause parentheses never change how a query is interpreted. Whitespace inside a wrapper is preserved verbatim, since it is part of the literal value:

str<bio> :: str<Software Engineer> ;
/ identical to: str<bio>::str<Software Engineer>; /
/ the space between "Software" and "Engineer" is preserved either way /

Why Wrappers?

• Eliminates implicit type conversion.
• Prevents ambiguous comparisons.
• Ensures consistent query execution.
• Preserves the original data type throughout the query lifecycle.
• Reduces parser complexity by making every value's type explicit.
• Document bodies are the exception: plain JSON strings, loaded with json.loads.

Behaviour

• Every field reference and every value is wrapped using its corresponding type.
• The wrapper becomes part of the SNL syntax and is interpreted by the query engine.
• Typed pairs are supported across FETCH, INSERT, UPDATE, OVERWRITE, DELETE, FREEZE, UNFREEZE, and other compatible SNL operations.
• Top-level pairs follow the syntax Type<field>::Type<value>; sub-clause assignments follow Type<field>=Type<value>.

Reserved Characters & Escaping

SNL reuses a small set of punctuation characters as structural delimiters - :: separates a typed field from its value, + separates vector elements, / opens and closes a comment, and so on. Which characters are reserved depends on where you are in the query: the characters that matter inside a GET() argument list are not the same as the ones that matter inside a top-level typed pair.

Any reserved character that must appear literally inside a value is escaped with a preceding backslash. Escaping is only needed for the characters listed for that specific context - everything else can be written as-is.

Top-Level Typed-Pair & Partition Level

Inside a top-level Type<field>::Type<value> pair, or inside a partition name, the following characters are reserved and must be escaped if they need to appear literally:

str<label>::str<9\:30 AM \| morning slot>;

Sub-Clause Argument Level

Inside a clause's argument list - GET(), EXCLUDE(), UPDATE(), STRICT(), VERIFY(), TARGET(), RETRIEVE(), USE() - the following characters are reserved:

UPDATE(str<note>=str<see item \(a\)\, then \(b\)>);

Vector Level

Inside a vec<...> literal, + separates elements and must be escaped to appear literally within an element's value:

str<formula>::vec<str<5\+5=10>+int<10>>;

Assignment Level

Inside an AssignmentList (the body of UPDATE(), STRICT(), and VERIFY()), = separates the typed field from its typed value:

UPDATE(str<equation>=str<a \= b + c>);

Comment Level

Inside a / comment /, only a literal slash needs escaping:

Quick Reference

Execution Pipeline

Every SNL query passes through a deterministic execution pipeline before a response is returned. Each stage performs a specific responsibility, ensuring queries are executed consistently while maintaining security, integrity, and predictable performance.

Depending on the operation being performed, certain stages may be skipped. For example, read-only queries do not require write locks or persistence, whereas write operations continue through the full commit process.

Execution Stages

1. 1. Parsing
   The SNL parser reads the query sequentially, separating statements using semicolon (;) delimiters and stripping comments and insignificant whitespace. Clauses, typed fields, typed values, and an optional partition clause are tokenized and validated before execution begins.
2. 2. Partition Resolution
   If a leading partition #name#; clause is present, the engine resolves it to a concrete partition before proceeding. Otherwise, the connection's default partition is used.
3. 3. Authorization Validation
   The engine verifies that the current security context has permission to access the requested database partition and perform the requested operation. Queries that fail authorization terminate immediately.
4. 4. Query Planning
   Based on the parsed clauses, the engine determines the execution strategy, including the query mode, compatible sub-clauses, pagination, ordering, and any recycle operations that must be performed.
5. 5. Index Resolution
   Indexed fields are consulted to locate candidate records efficiently. Whenever possible, the engine avoids scanning unnecessary records by using available indexes to narrow the search space.
6. 6. Record Evaluation
   Candidate records are evaluated against the supplied conditions. Operations such as STRICT(), VERIFY(), and filtering clauses are applied before records are accepted or rejected.
7. 7. Transaction Isolation
   Write operations acquire the necessary locks to ensure transactional consistency while \1 If the targeted record is already locked, the operation returns ERR_TRY_AGAIN. Read operations bypass write locking, allowing high-throughput concurrent reads.
8. 8. Operation Execution
   The requested SNL operation is performed. Depending on the query, this may involve fetching records, updating existing values, overwriting documents, deleting records, freezing entries, or executing multi-stage RECYCLE queries.
9. 9. Response Projection
   Clauses such as GET(), EXCLUDE(), ORDER(), START, and LIMIT are applied to shape the final response returned to the client.
10. 10. Persistence & Commit
    Standard write operations are first appended to the Write-Ahead Log (WAL) to guarantee recoverability before being synchronized with persistent storage.
11. 11. Response Generation
    The engine serializes the final result and returns it to the client. Depending on the executed operation, the response may contain records, status values, Boolean results, or confirmation that the operation completed successfully.

Pipeline Characteristics

• Deterministic execution from parsing to response.
• Explicit partition resolution before any data access occurs.
• Permission validation before any data access occurs.
• Index-assisted record discovery for efficient lookups.
• Transaction isolation for write operations.
• Response shaping through projection, ordering, and pagination clauses.
• Crash recovery through Write-Ahead Logging for standard write operations.

Performance Optimization

SNL is designed to execute efficiently under high concurrency, but query structure still plays an important role in achieving optimal throughput and low latency. Choosing the appropriate clauses and execution modes can significantly reduce CPU usage, memory allocation, disk activity, and network transfer sizes.

The following recommendations help maximize performance while minimizing resource consumption across both small and large datasets.

Optimization Best Practices

• Use RETRIEVE instead of TARGET whenever parent data is unnecessary.
  TARGET() merges the parent record into every matching recycled result, requiring additional memory allocations and larger response payloads. If only the related records are required, prefer RETRIEVE().
• Project only required fields with GET().
  Returning fewer fields reduces serialization time, memory usage, and network bandwidth. Whenever possible, request only the fields your application actually needs.
• Hide unnecessary fields using EXCLUDE().
  Large attributes such as binary objects, lengthy descriptions, or internal metadata can unnecessarily increase response sizes. Excluding them results in smaller payloads and faster transfers.
• Paginate large result sets.
  Rather than retrieving every matching record in a single request, combine START ? x and LIMIT ? y to return manageable batches of data.
• Combine ORDER() with pagination.
  Using ORDER() together with START and LIMIT produces stable and predictable pagination across multiple requests.
• Prefer VERIFY() for credential validation.
  When checking passwords, API keys, or authentication tokens, use VERIFY() instead of fetching the value and comparing it inside application code. This reduces response payloads and simplifies validation.
• Use STRICT() when schema validation is required.
  Allowing the engine to validate fields during execution helps avoid invalid updates and prevents unnecessary retry operations.
• Use RECYCLE for related data retrieval.
  Instead of issuing multiple client-side queries and manually joining the results, use RECYCLE to perform relationship traversal within a single engine execution.
• Target the right partition up front.
  When a query only needs one partition, open with partition #name#; rather than relying on the default, so the engine can route the request without a redundant lookup.
• Use unique fields whenever possible.
  Queries against unique identifiers generally require less work than searches across non-unique values, making them ideal for record lookups and mutation operations.

General Recommendations

Security Matrix

Security is integrated into every stage of the SNL execution pipeline. Before a query is allowed to access any database partition, shadeDB validates the client's authorization context, ensuring that only permitted operations can proceed. Unauthorized requests are rejected before any records, indexes, or storage blocks are examined.

Beyond access control, SNL's grammar is intentionally restrictive. Every query follows a deterministic syntax, allowing the parser to validate structure before execution and reject malformed or invalid statements.

Security Features

• Partition-Level Authorization - Every query is validated against the caller's permissions before any data is accessed, including the partition named by an explicit partition #name#; clause.
• Deterministic Parsing - Queries are parsed using a fixed grammar with explicit delimiters, reducing ambiguity during execution.
• Strong Type Validation - Explicit type wrappers such as int<>, float<>, bool<>, str<>, bytes<>, and vec<> ensure values are interpreted as their intended types; any<> is the sole intentional exception and should be used only where dynamic typing is actually required.
• STRICT() Validation - Schema validation can be enforced during supported operations to reject invalid or unexpected fields.
• VERIFY() - Allows credential verification without returning the stored value, reducing unnecessary exposure of sensitive fields.
• Projection Controls - GET() and EXCLUDE() allow applications to expose only the data required by the client.
• Controlled Mutations - Operations such as DELETE, FREEZE, and UNFREEZE are constrained by their execution rules, helping prevent unintended modifications.

Injection Defense

Security Workflow

1. The query is parsed and validated.
2. The target partition is resolved, if one was specified.
3. User permissions are verified for the target partition.
4. The query structure and compatible clauses are validated.
5. Optional schema validation is performed when STRICT() is used.
6. The authorized operation is executed.
7. The response is filtered through projection clauses such as GET() or EXCLUDE() before being returned.

Error Catalog

Every SNL query returns a consistent response structure, regardless of whether the operation succeeds or fails. When an error occurs, the engine returns an error identifier through the message field while preserving the normal response format. This allows applications to handle successful queries and failed operations using the same response parser.

Errors can occur during parsing, authorization, schema validation, query execution, transaction processing, or storage operations. Each error identifier is stable and intended for programmatic handling, making it preferable to compare error codes rather than relying on human-readable text.

Successful Response

{
    "status": "success",
    "message": [
        {
            "username": "john",
            "age": 19
        }
    ],
    "db_latency": "0.41ms"
}

For successful operations, message contains the query result. The returned value depends on the executed operation and may contain records, Boolean values, status confirmations, or other valid SNL responses.

Error Response

{
    "status": "failed",
    "message": "ERR_SCHEMA_MISMATCH",
    "db_latency": "0.17ms"
}

When execution fails, message contains the error identifier instead of a query result. Applications should first inspect status. If the status indicates failure, the value contained in message should be interpreted as an SNL error code.

Core Error Reference

Handling ERR_TRY_AGAIN

INSERT;, UPDATE; / STRICT;, and OVERWRITE; / STRICT-OVERWRITE; may return ERR_TRY_AGAIN. It only means the targeted record is currently locked or in use by another operation. The query itself is valid.

{
    "status": "failed",
    "message": "ERR_TRY_AGAIN",
    "db_latency": "0.09ms"
}

• Do not change or fix the query; resend it unchanged.
• Wait briefly between attempts, and cap the number of retries in your application.
• Treat it as a transient state, not as a failure of the query or of the partition.

Error Categories

Debugging Checklist

1. Verify the status field.
2. If the status indicates failure, inspect the value of message.
3. Compare the returned error identifier with the Error Reference table.
4. Review the query for syntax, clause compatibility, and correct type wrappers on both fields and values.
5. Confirm any reserved characters inside literal values were escaped per Reserved Characters & Escaping.
6. If STRICT() is used, ensure every referenced field exists within the record schema.
7. Confirm that the executing client has permission to access the target partition.

Production Examples

This section demonstrates complete SNL queries commonly used in production environments. Each example combines multiple clauses to illustrate how different parts of the language work together during execution.

The examples below assume that the executing client already has permission to access the target database instance.

Example 1 - Fetch Active Users

partition #accounts#;

WHERE;
str<status>::str<active>;

GET(
    username,
    email,
    profile_pic
);

EXCLUDE(
    auth_token
);

ORDER(DESCEND);

START ? 0; LIMIT ? 25;

This query targets the accounts partition, retrieves active users, returns only the requested public fields, removes the authentication token from the response, sorts the result in descending order, and returns the first twenty-five matching records.

Typical Response

{
    "status": "success",
    "message": [
        {
            "username": "john",
            "email": "john@example.com",
            "profile_pic": "/images/john.png"
        }
    ],
    "db_latency": "0.34ms"
}

Example 2 - Verify Credentials

FETCH;
str<username>::str<john>;

VERIFY(
    str<password>=str<my_password>
);

Rather than returning the stored password, this query compares the supplied password against the stored value and returns a Boolean result indicating whether the verification succeeded.

Example 3 - Atomic Update

FETCH;
str<username>::str<john>;

UPDATE(
    str<bio>=str<Software Engineer>
);

Updates only the specified fields while preserving every other attribute within the record.

Example 4 - Strict Update

STRICT;
str<username>::str<john>;

{
    "bio": "Software Engineer"
};

Updates bio only if it already exists in the record. Unknown fields in the payload are silently discarded, so the schema cannot grow by accident.

Example 5 - RECYCLE Relationship Query

RECYCLE;

FETCH;
str<id>::int<my_id>;

RETRIEVE(
    message_composed_by_userid,
    user_id
);

USE(WHERE);

GET(
    message,
    created_at
);

ORDER(DESCEND);

START ? 0; LIMIT ? 50;

Retrieves a user's messages using a single SNL request. Only message data is returned because RETRIEVE() performs filtering without merging the parent record.

Example 6 - Insert With a Vector Field

INSERT;
{
    "username": "john",
    "permissions": ["read", "write", "pending_review"]
};

Stores permissions as a three-element vector mixing three strings.

Example 7 - Freeze a Record

FREEZE;

str<id>::int<42>;

Prevents the matching record from participating in normal queries and protects it from modification until it is explicitly unfrozen.

Example 8 - Delete a Record

DELETE;

str<id>::int<42>;

Deletes the matching record using its unique field. Since DELETE only operates on unique fields, accidental removal of multiple records is prevented by design.

Example 9 - Handling an Ingestion Pause

When instance-wide ingestion is paused, new records cannot enter any partition. Existing records can still be updated, deleted, frozen, or unfrozen. Applications should treat ERR_INGESTION_PAUSED as an operational state rather than a malformed query.

INSERT;
{
    "username": "new-user",
    "status": "pending"
};

{
    "status": "failed",
    "message": "ERR_INGESTION_PAUSED",
    "db_latency": "0.12ms"
}

Example 10 - Partition Hard Lock

If the accounts partition is under maintenance, the same client's reads continue to work, but every mutation against accounts is rejected. This is the appropriate control for schema work, controlled migrations, or any operation that requires the partition to remain unchanged.