Quantum Walk Pipeline¶

The quantum walk pipeline is the core of Leliel's re-ranking engine. It evolves a wavefunction over the factor graph using a Chebyshev approximation of the lensing Hamiltonian, measures the Born-rule probability distribution, and uses that distribution to re-rank query results.

All walk functions are in julia/LelielStore/src/ and julia/LelielCore/src/. The pipeline is fully wired to the live query path via list_records and query_with_context.

Lensing Hamiltonian¶

The walk is governed by the lensing Hamiltonian:

H = L + V(mass)

Where L is the graph Laplacian (D - A, degree minus adjacency) and V is the diagonal mass matrix with V[v, v] = mass(v). High-mass nodes have elevated potential energy. When the wavefunction evolves under exp(-i H dt), Born-rule probability concentrates at low-mass nodes and away from high-mass nodes.

The Schwarzschild threshold M_s is derived from the Fiedler eigenvalue:

M_s = lambda_2(L) * log(1/epsilon) / (G * K)

This is a physical derivation, not a configurable constant.

Chebyshev evolution¶

Time evolution exp(-i H dt) is approximated by a Bessel-weighted Chebyshev expansion:

U(dt) = sum_{k=0}^{K} c_k T_k(H)
  where c_0 = J_0(dt), c_k = 2(-i)^k J_k(dt)

J_k are Bessel functions of the first kind. At K = 30 terms the approximation error is below machine precision (~1e-15) for typical lensing Hamiltonians. Cost is O(K * N^2) versus O(N^3) for full eigendecomposition: approximately 33x cheaper at N=1000, K=30.

The Chebyshev evolver is in julia/LelielCore/src/Evolution.jl.

Walk modes¶

The WALK_MODE configuration variable selects the evolution chain used by both list_records and query_with_context:

Mode	Description
`combined`	Full pipeline: imaginary-time Chebyshev ground-state seed + unitary Chebyshev walk + ensemble Born measurement. Default and recommended mode.
`ground`	Imaginary-time Chebyshev seed only; returns the ground state distribution without further unitary evolution.
`unitary`	Uniform or context-seed initial state + unitary Chebyshev walk + ensemble Born. No imaginary-time step.
`grover`	Unitary walk with oracle amplification. Causal cone projection is applied before each Grover iteration so oracle amplitude is concentrated only inside the timelike subspace of the seed. Useful when strong locality is required; slower than `combined` on large graphs.

The :combined pipeline:

Imaginary-time seed: evolve an initial uniform state under exp(-H * tau) for several steps (LelielCore ImaginaryTime.jl). This concentrates amplitude at the ground state of H, which is a useful prior over the entire graph before context is applied.
Context injection: override the seed amplitude at the context node indices with the caller-supplied context_ids. The resulting state mixes the ground-state prior with the explicit context.
Unitary walk: evolve under exp(-i H dt) for K Chebyshev steps. Amplitude diffuses from the context nodes through the graph, bending around high-mass nodes.
Ensemble Born measurement: repeat the unitary walk from K random phase-rotated initial states and average the Born-rule probability distributions. This reduces phase-cancellation artefacts. Default: 5 shots.
Rank and return: sort records by averaged Born probability descending; return the top K.

Semantic psi (cold-start seeding)¶

When extra_data is present on ingested records, field-value pairs are indexed into per-record IDF feature vectors at ingest time. The IDF weight for pair (field, value) is:

IDF(field, value) = log(N / count(field, value))

Where N is the total record count and count is the number of records containing that field-value pair. The feature vector is normalised to unit length so dot product gives cosine similarity.

At query time, semantic_focused_psi computes the cosine similarity between each candidate record and the context records, then blends that similarity into the walk seed:

psi_seed = (1 - PSI_ALPHA) * psi_focused + PSI_ALPHA * psi_semantic

Where PSI_ALPHA = 0.3 and psi_focused is the standard context-seeded state. This provides a cold-start bias toward semantically similar records before utility mass has accumulated from traversal history.

Field weights¶

Before the cosine dot product, each IDF feature vector is scaled by caller-declared per-field weights and renormalized to unit length:

weighted(field, value) = user_weight[field] * IDF(field, value)
similarity(i, j) = dot(normalize(weighted_i), normalize(weighted_j))

This lets callers declare which fields carry semantic signal for their domain. A field with user_weight = 5.0 contributes 5x more to similarity than one with weight 1.0. After renormalization the result is still a unit vector, so the dot product remains a cosine similarity bounded in [-1, 1]. Fields absent from the weight map default to weight 1.0 (no scaling). An empty weight map applies no scaling.

Field weights are set at runtime via PUT /api/v1/config/field-weights and apply immediately to all subsequent queries without re-ingesting records. See the API reference for the endpoint details.

The semantic index is rebuilt during warm_start! after restart. The semantic psi function is in julia/LelielStore/src/Pipeline.jl.

Cross-cluster bridge injection¶

Before the walk runs, the pipeline performs a BFS on the snapshot to count connected components. When more than one component exists, the graph is structurally disconnected and amplitude seeded in one component cannot reach records in another.

In that case, the pipeline selects the lowest-mass (gem) node in each component and injects one unit-weight bridge edge between every pair of component gems:

bridge weight = 1.0  (structural connectivity, not semantic affinity)

The bridge weight is fixed at 1.0 rather than cosine similarity because cross-cluster cosine similarity is near zero by definition: records in different clusters differ on the very fields that distinguish the clusters. A cosine-weight bridge would be a near- dead edge the walk Hamiltonian cannot traverse.

Bridge injection runs first, before Ricci bypass and wormhole injection, so that the subsequently injected edges are not themselves misclassified as bottlenecks.

Ricci bottleneck bypass¶

The Ollivier-Ricci curvature of an edge (u, v) measures local graph geometry:

kappa(u, v) = 1 - W1(mu_u, mu_v)

Where W1 is the Wasserstein-1 distance between uniform distributions over the neighbourhoods of u and v. Negative kappa identifies structural bottleneck edges: edges whose endpoints sit inside dense subgraphs with few shared neighbours, so the walk concentrates on the bottleneck instead of spreading across the graph.

Computing Ricci curvature requires all-pairs BFS, which is O(E * n * (n + E)) and not affordable at walk time. It is instead computed by the analysis worker alongside SLEM and Fiedler, and the results are cached in the store as a list of (bottleneck_edge, bypass_u, bypass_v) tuples. bypass_u and bypass_v are the lowest-mass nodes in the respective neighbourhoods of the bottleneck endpoints.

At walk time, the cached list is read and a bypass edge is injected between each (bypass_u, bypass_v) pair in the snapshot. Bypass edges carry unit weight. The walk can then route amplitude through the bypass rather than concentrating on the bottleneck.

The Ricci cache is refreshed each analysis worker cycle. It is empty until the first cycle completes, at which point bypass injection activates automatically.

Wormhole edges (ER=EPR)¶

The store caches the Fiedler eigenvector v2 of the graph Laplacian after each ingest. At query time, inject_wormhole_edges! adds latent edges to the walk snapshot between any two high-mass nodes (both above M_s) whose Fiedler components are proximate:

|v2[i] - v2[j]| < WORMHOLE_ALPHA * std(v2)
  where WORMHOLE_ALPHA = 0.2

Wormhole edges are injected only into the walk snapshot. The live graph topology is unchanged. The walk can traverse these latent shortcuts but ingest co-occurrence edges are not affected.

This implements the ER=EPR principle: black hole nodes that are structurally equivalent in the Laplacian embedding share a latent entanglement channel. The wormhole edge represents this channel as a traversable shortcut in the walk.

Quantum eraser¶

After each walk, the eraser penalises nodes the walk assigned near-zero Born probability:

threshold: QS_ERASER_AMP_THRESHOLD = 1e-8
penalty:   QS_ERASER_ETA = 0.01 per node
cap:       QS_ERASER_MAX_PER_QUERY = 20 nodes per walk

The eraser mass penalty accumulates across queries, reinforcing the walk's suppression signal over time. A node the walk consistently assigns near-zero amplitude receives a growing eraser penalty without requiring explicit REINFORCE calls from callers.

Concurrency model¶

The walk uses an Option 3 + 4 SLEM-gated concurrency model:

When SLEM <= QS_SLEM_SERIAL_THRESHOLD (0.9): Concurrent path. Multiple walkers hold the topology read lock simultaneously. Each walker snapshots the FactorGraph (O(n+E)), releases the read lock, runs the Chebyshev walk lock-free on the snapshot, then writes REINFORCE and eraser deltas under the mass lock only. The mass lock serialises mass writes but does not block other walkers.

When SLEM > 0.9: Serial path. Exclusive write lock. One walker at a time operates on the live graph. Used when the graph is near critical slowing down and topology changes during concurrent walks could destabilise the walk result.

SLEM is updated by get_mesh_health (called by the analysis worker each cycle) and stored in store.slem. The transition between paths is automatic.

Substring tokenizer¶

Raw text strings (query text, source_id fragments, and extra_data string values) are tokenized using an amplitude-decaying substring tokenizer before the walk seed is built.

The tokenizer generates all substrings of length L from the input, where:

L_min = max(3, floor(sqrt(len(input))))
L_max = min(len(input), adaptive_ceiling)

The adaptive ceiling is derived from the Shannon entropy of the token-length distribution in the current corpus. Higher corpus entropy (more diverse token lengths) raises L_max to capture longer structure; lower entropy tightens it.

Each substring is weighted by a decay function:

weight(L) = base_amplitude * exp(-decay_rate * (L - L_min))

Longer substrings receive lower weight. This reflects the fact that rare long substrings are high-IDF but brittle: a one-character difference in the corpus produces a full miss. Shorter substrings are less specific but more robust. The decay rate is fixed at QS_SUBSTRING_DECAY.

A Count-Min Sketch (CMS) pre-filter discards substrings whose corpus frequency falls below QS_SUBSTRING_MIN_SUPPORT. This prevents hapax substrings from acquiring disproportionate IDF weight and dominating the query state.

The tokenizer runs in julia/LelielStore/src/Pipeline.jl and feeds into the IDF-weighted query state construction for both text_query and the walk seed for query_with_context.

Hyperedge Laplacian¶

The Hamiltonian includes a hyperedge Laplacian term that encodes categorical diversity:

H = L_graph + alpha * L_H + V(mass)

Where L_H is the hyperedge Laplacian and alpha = QS_HYPEREDGE_ALPHA = 0.3.

Hyperedge construction: at ingest time, every (field, value) pair in a record's extra_data becomes a hyperedge containing all records that share that pair. For example, all records with {"region": "us-east"} belong to the (region, us-east) hyperedge.

Hyperedge Laplacian: for the incidence matrix H_sc (n x E, columns normalised to unit length):

L_H = I - H_sc * H_sc^T

This penalises nodes that share many categorical memberships with their neighbours. A record that is identical across all categorical fields contributes no discriminating signal; the hyperedge penalty suppresses it relative to records that are categorically distinct.

L_H is never explicitly materialised. At walk time, hv_product! applies the additive term via two matrix-vector products:

alpha * (v - H_sc * (H_sc^T * v))

Cost is O(E * k) per Chebyshev step where k is the mean hyperedge size, well within the Chebyshev budget. The hyperedge columns are built in _build_hyperedge_columns in Pipeline.jl and attached to the graph snapshot at the start of each walk.

BP seed psi¶

query_with_context supports a multi-context seeding mode that uses Belief Propagation (BP) to combine multiple context records into a single coherent walk seed.

When context_ids contains more than one record, the standard seed sums unit amplitude at each context node. The BP seed mode instead runs one round of BP message passing across the context subgraph to compute a joint probability over context nodes weighted by their marginal agreement. The resulting marginals form the initial amplitude vector:

psi_0[v] = sqrt(bp_marginal(v))  for v in context_ids
psi_0[v] = 0                     otherwise (before cross-cluster bridge or wormhole injection)

This provides a more faithful prior when context records disagree (e.g., one high-mass and one low-mass record in context) rather than giving equal amplitude to all context nodes regardless of their individual certainty.

The BP seed is in julia/LelielStore/src/Pipeline.jl as _bp_seed_psi.

Causal cone projection¶

The causal cone projection is applied at query time in :grover walk mode to enforce spacetime locality during Grover oracle iterations.

BFS causal cone: starting from the context seed nodes, a BFS explores the factor graph to BFS depth d (default: 3). Nodes reached within d hops are classified as:

Timelike (BFS distance < d): inside the cone
Lightlike (BFS distance == d): on the cone surface

Nodes unreachable within d hops are spacelike.

Projection: before each Grover oracle iteration, the walk state psi_t is projected:

psi_t[v] = 0    for all v in spacelike set
psi_t renormalized to unit norm

This prevents the oracle from amplifying spacelike records. Without this projection, Grover amplitude leaks to disconnected or distant nodes because the oracle marks nodes by type (not position) and the unitary walk spreads amplitude globally.

Walk ordering (:grover mode only):

1. Build causal cone from seed context
2. Zero spacelike amplitude in psi_t
3. Renormalise psi_t
4. Apply Grover oracle iteration
5. Repeat for K_grover iterations

In :combined mode, no causal projection is applied; amplitude is free to diffuse to all graph nodes.

The projection is implemented as _apply_causal_projection! in Pipeline.jl.

QFI convergence signal¶

The walk engine tracks the Quantum Fisher Information (QFI) of the Born-rule amplitude distribution after each walk. QFI measures how informative the distribution is about a perturbation parameter (here, the mass field):

F_Q = 4 * sum_v |dpsi/dparam|^2

A high QFI indicates the walk result is sensitive to the mass field, meaning REINFORCE signals are being incorporated effectively. A low QFI indicates the distribution is near-uniform and the walk result carries little information above the topology prior.

GET /api/v1/gravity/julia/quantum-speed-limit returns tau_min_lower and tau_min_upper, which are inversely related to the energy spread Delta_E of the Hamiltonian. Large Delta_E (wide energy spread from large mass variance) implies faster-than-average state changes and a lower QSL bound.

SLEM and spectral health¶

SLEM (Second Largest Eigenvalue Modulus) measures how close the walk weight matrix is to the identity:

SLEM = 0: instant mixing; optimal
SLEM = 1: no mixing; critical slowing down

spectral_gap = 1 - SLEM. A larger gap means faster walk convergence.

SLEM > 0.9 is a red-flag state: mixing time diverges, small perturbations produce large state changes. Monitor via GET /api/v1/gravity/mesh.