How it works
Streaming read-ahead, batches of 20, parallel requests over persistent connections, and a per-session cache. Every row still goes to the API.
jev() is not an index lookup. On a base table or view it reads ahead, packs rows into small parallel API requests, and caches the answers for the rest of the session. Every row that reaches jev() is still sent to TypeSafe.
- Read-ahead. The first call for a table and condition starts a read-ahead that streams the table in physical order (TID range scans for tables,
OFFSETpages for views). Memory stays constant whatever the table size. - Batch. Rows go out
jev.batch_size(20) at a time as one shared state:{"condition": ..., "rows": [...]}with one yes/no Noul question per row. Jev scores every question over that state in one request, which amortises the ~270-token request overhead: about 175 input tokens per row in batches of 20, against about 435 for a row on its own. - Parallel requests. Up to 2 ×
jev.concurrency(32) requests are in flight over persistent HTTPS connections. Every row is answered as soon as its batch returns, so aLIMITstops the read-ahead after the in-flight window, and rows that cheaper predicates reject beforejev()runs (WHERE age > 60 AND jev(...)) are skipped rather than judged. - Session cache. Answers are keyed by row content in the backend session (PL/Python
GD). Re-running, changing the threshold, or sorting withjev_prob()does not call the API again.
Rows from a subquery or CTE (anonymous record) cannot be read ahead. They are judged one request at a time. Put jev() on a base table or a view when you can. Combining with SQL shows the view pattern.
Why 20 rows per request
Jev has to find rows[i] by position in the array, and that gets unreliable in long arrays. Against ground truth from structured columns (job title, EU membership, a phrase in a free-text field; 400 rows each), batches of 1–20 rows were 100 % correct, batches of 40 were 92–98 % and batches of 80 were 77–94 %. Wider rows (1,000 characters) made no difference at 20. Batches of 20 cost 4 % more tokens than batches of 40 and are just as fast, because a request's latency barely depends on its size. So jev.batch_size defaults to 20; leave it there.
Measured
On a 2,000-row table from Europe (~190 ms to the API): first run ≈ 3.5 s in 100 requests, ≈ 296k input tokens, ≈ $0.012; second run ≈ 50 ms from the cache; LIMIT 3 on a new condition ≈ 0.6 s. A new condition in a session that still holds its pooled connections (idle for less than jev.keepalive) takes ≈ 2.3 s: the first request on each fresh connection is the slow one.
Cost is input tokens × $0.042 per million (jev-1.13 list price; output tokens are free), roughly 175 tokens per row. jev_stats() reports the running total as estimated_cost_usd. Cost scales with rows and row width, not with how clever the SQL looks.