← older entries

Phase 238 — fix TIME whole-column reads (a third real ISM bug in the same family as Phase 237, found by profiling not guessing)

Direct follow-up ask: "Improve the performance of the TIME whole-column access" (the one MAIN metric Phase 237 explicitly left untouched and flagged as a candidate — 9-12x slower than C++ depending on the run, versus every other MAIN metric being at parity or faster).

Root cause, found via Profile.@profile on the real MS: IncrementalStMan.getcolumn's run-length fill loop (v = _ism_decode(ism, c, ...); out[r] = v for every row in a stored run) was live-measured at ~245 MiB / ~10 M allocations for a 9.8M-row Float64 column whose output array alone needs ~78.5 MiB — ~25 bytes of pure overhead per ROW, not per stored value (only ~30K of the 9.8M rows are actual distinct-value decodes; the rest are plain fill-loop writes of an already-decoded value). Root cause: _ism_decode's general form returns Bool | String | T | Array{T,N} depending on the runtime ColumnDesc it's given — its return type is fundamentally not inferrable from the call site, so getcolumn's own v = _ism_decode(...) is Any-typed, and every out[r] = v write in the fill loop — not just the decode itself — re-boxes/dispatches dynamically. (_ism_decode's scalar branches also each built a wasteful Vector{T}(undef, 1) just to return vals[1] — a smaller, separate waste in the same function, fixed alongside: both the Bool and general-numeric scalar cases now load the value directly via the Phase 237 _ld primitive with no intermediate array.)

Fix — the same type-instability barrier as Phase 237's getcell fix, applied one level up: a new _ism_getcolumn_scalar!(out, ism, colnr, isbool, nrow, ncol, ::Type{D}) where D (the on-disk element type) is a where {D} type parameter, not a runtime ColumnDesc field — getcolumn computes D = juliatype(c.type) once and calls this specialized function, which the compiler compiles fully type-stably per concrete D (Float64 for TIME, Int32 for FIELD_ID, …), eliminating the boxing entirely from the per-row fill loop. Covers every scalar ISM column (kind === :scalar && c.type != TpString — i.e. every column in _MAIN_ISM, the real common case); the non-scalar (array-valued / string) path is untouched, still going through the general _ism_decode.

Live-verified on the real MS: column(t,"TIME")[:] 0.155s / 245 MiB → 0.0085s / 75 MiB (18x faster, allocation now essentially just the output array). Re-running the full C++ comparison: TIME whole-column 0.05x of C++ (20x FASTER), was 9-12x slower — now the single best MAIN metric, alongside DATA/FLAG whole-column. Every MAIN-table metric is now at parity or faster than C++ (worst case 0.93x for cached-column DATA per-cell); the MAIN-table geometric-mean ratio improved from 0.369x to 0.197x (≈5x faster than C++ on average across every measured MAIN operation). Subtables unaffected (unchanged from Phase 237's numbers).

New test/ism_writer_tests.jl testset ("ISM getcolumn — whole-column allocation regression"): a synthetic 40,000-row ISM column with a realistic run length (changes every ~250 rows, matching the real MS's TIME column's ~325-row average — a short run length was tried first and found to make the bound meaningless, since _ism_colindex's own row-number/offset array construction is legitimate, unavoidable overhead that scales with the number of distinct-value entries, not with nrow, and a too-short run length inflates that non-bug cost disproportionately relative to the output array); an @allocated regression guard (< 1.5x the theoretical-minimum output-array size, comfortably under the ~4x-of-minimum the old per-row-boxing bug gave on the real MS's TIME column).

Full suite green: 5342 baseline + 3 new = 5345/5345. README/memory updated, merge on the user's word.

Phase 239 — fix the SPECTRAL_WINDOW.CHAN_FREQ subtable outlier (and centralise the underlying byte-reader bug fix, found in three other files while looking for "any others")

Direct follow-up ask: "Improve the subtable outlier performance, i.e., the SPECTRAL_WINDOW.CHAN_FREQ performance, and any others that might be identified" — the one subtable column, out of every one benchmarked since Phase 45's comparison, that stayed slower than C++ (5.08x) while every other subtable column was already faster.

Root cause: the exact same allocating byte-read pattern Phases 68/234/237/238 had already found and fixed three times — independently, in three different files, never centralised. CHAN_FREQ is an SSM-indirect (ragged/VariableShape) array column, read via arrayfile.jl's af_read and standard.jl's getcell/locate machinery — both used reinterpret(T, ::Vector{UInt8}) (via a view) for every scalar/array element read, the identical slow path this package's own tiled.jl (_rd_run!/_rd_bits!, Phase 68/234) and incremental.jl (_ld, Phase 237) had already replaced with a pinned-pointer unsafe_load primitive — just never carried over to standard.jl or arrayfile.jl.

Fix — centralised, not a fourth independent patch: extracted _hostconv/_ld/_rd_run!/_rd_bits! out of tiled.jl/ incremental.jl into a new shared datamanagers/bytes.jl, included first among the datamanagers/*.jl files (so every other data-manager file — standard.jl, tiled.jl, incremental.jl, arrayfile.jl — can use it, resolving the include-order problem that was presumably why each file grew its own independent copy of the same primitive in the first place). Rewired every site: standard.jl's _i32/_i64/ _be_i32 (used throughout getcell/locate/bucket-header parsing) and _read_elems (the fixed-shape-array getcell decode); arrayfile.jl's _af_get, open_arrayfile's version read, and af_read's three element-type branches — the Bool branch switched to the bulk _rd_bits! primitive, and the Complex branch was found to need no special case at all once using _rd_run! (a Julia Complex{T}'s in-memory layout — two contiguous T fields, real then imaginary — already matches casacore's own interleaved on-disk storage, and _hostconv already swaps each component; the same bulk primitive Phase 234-238 already proved correct for tiled DATA, ComplexF32, columns handles it directly, merging what used to be three branches into two).

Live-verified on the real MS: column(spw,"CHAN_FREQ")[:] 0.38x of C++ (was 5.08x — from 10x slower to ~2.6x faster). Every subtable column across ANTENNA/FIELD/SPECTRAL_WINDOW/POLARIZATION/FEED in the benchmark is now faster than C++ — no remaining subtable outlier (subtable geometric-mean ratio 0.286x → 0.215x). The MAIN table's own SSM-bound scalar columns share _i32/_be_i32/ _read_elems too, so this incidentally tightens those further (already faster than C++ before this phase, geometric-mean ratio essentially unchanged at 0.2x — within run-to-run noise). The one remaining ">1x" line in the benchmark, table/subtable open (metadata parsing, ~400 µs absolute — a fundamentally different operation from a column data read) was not part of this ask and is left as a documented, unaddressed asymmetry.

New test/indirect_tests.jl testset ("SSM indirect array — whole-column allocation regression"): a synthetic 2000-row ragged (non-uniform-cell-shape — _infer_shape's VariableShape() branch, confirmed via columndesc(...).shape isa VariableShape so the test can't silently drift onto the already-fast fixed-shape :direct path instead) array column with CHAN_FREQ-sized (63-65-element) cells; an @allocated regression guard for both getcolumn (whole-column) and getcell (per-cell), each with a documented, empirically-calibrated bound (a Vector{Any}-of-individually-allocated-cells column has real, unavoidable per-cell array-header overhead on top of its raw payload — not 1x, but nowhere near the several-hundred-bytes-per-element the old reinterpret/view pattern added). _HAVE_CASACORE-gated cross-check. The pre-existing "StManArrayFile codec round-trip" testset (both endians, Double/Int/Complex/Bool) already fully covers af_read's rewritten branches for correctness — no separate correctness test needed for the Complex merge.

Full suite green: 5345 baseline + 9 new = 5354/5354. README/memory updated, merge on the user's word.

Phase 240 — StandardStMan.getcolumn bulk read for SSM-indirect (:indarr/:indstr) columns

Direct follow-up ask: "Implement the SSM-indirect columns bulk read" — the architectural gap documented since Phase 16 and left open through Phases 237-239's byte-reader fixes: StandardStMan.getcolumn's :indarr (variable-shape numeric/Bool array) and :indstr (variable-shape string array) branches had no bulk path at all, just [getcell(ssm, ssmcol, c, r, 1) for r in 1:nrow] — every one of nrow calls redid locate (an O(log nbucket) binary search via bucket_of) from scratch. Phase 239 sped up the byte-reading primitives those calls use but never touched this per-row loop structure — the reason POINTING.DIRECTION/TARGET/ENCODER/ POINTING_OFFSET and every SYSPOWER array column stayed 4.3x-7.4x slower than real casacore C++ even after Phase 239, at 925K-2.4M rows.

Fix: walk each column's buckets directly — the same _foreach_bucket sequential-iteration trick the Bool/direct-array branch already used — instead of calling getcell (and its own fresh locate) per row. Within one bucket, a row's reference sits at a fixed byte offset from the bucket's own base (SSM_INDARR_REF = 8 bytes for :indarr's Int64 file offset, SSM_STRING_REF = 12 bytes for :indstr's string-bucket triple), so each row now costs one O(1) offset computation instead of a fresh O(log nbucket) search — live- verified on the real MS: POINTING.DIRECTION 4.34x slower than C++ → 0.63x (now faster than C++); SYSPOWER.REQUANTIZER_GAIN 7.43x → 0.42x; every column in the original "worst 10" list now reads faster than C++ (0.37x-0.63x), correctness spot-checked against Casacore.jl across 200 rows per column.

A real correctness wrinkle caught by the existing test suite, not guessed: the old per-row comprehension [getcell(...) for r in 1:nrow], thanks to Julia's collect-for-Generator type-widening (start from the first real element's type, typejoin in any later element that doesn't fit), happened to return a concretely-typed Vector{Matrix{ComplexF32}} for a column like FEED.POL_RESPONSE (every cell 2-D) — not Vector{Any}. Column's getindex(::Colon) returns the data manager's getcolumn result completely unwrapped, so that concrete eltype is exactly what eltype(getcolumn(...)), Tables.jl schemas, and the Phase-34 precision-narrowing path (Matrix{ComplexF16} after precision=:half) all depend on. A plain Vector{Any} from the new bucket-walking loop broke three pre-existing precision_tests.jl assertions immediately. Fixed by returning identity.(out) instead of the raw Vector{Any} — the same Vector{Any}-narrowing idiom already used throughout taql/ (groupby.jl, join.jl, query.jl) for precisely this purpose.

New test/indirect_tests.jl testset ("SSM indirect getcolumn — bulk bucket-walk correctness"): a synthetic 3000-row table (spans 3 of our own writer's 1024-row SSM buckets) with both a ragged numeric (:indarr) and a ragged string (:indstr) column, including an undefined (never-put, empty-array) cell in each; asserts getcolumn/getcell agree and match the source values at every bucket boundary (1024, 2048) and the undefined rows, plus a _HAVE_CASACORE cross-check.

Full suite green: 5354 baseline + 54 new = 5408/5408. README/memory updated, merge on the user's word.

Phase 241 — POINTING.NAME: bulk read for scalar String columns on StandardStMan

Direct follow-up ask: "Improve the performance of POINTING.NAME" — the one real subtable outlier left after Phase 240's re-survey (20.7x slower than casacore C++ at 925,645 rows; every other benchmarked subtable column was within ~1.8x, most of those timer noise).

Root cause: StandardStMan.getcolumn's TpString branch was still [getcell(...) for r in 1:nrow] — a fresh locate (O(log nbucket) search) plus a byte-slice copy per inline string, per row — the same per-row-getcell shape Phase 240 fixed for the indirect-array kinds.

Fix: walk the column's buckets directly and build each inline string with unsafe_string straight from the pinned buffer; strings longer than the inline limit still go through the string-bucket reader. Fixed-length (maxlength > 0) and array-shaped string columns keep the old path (unchanged behaviour/errors).

Live-verified on the real MS: POINTING.NAME 3.05 s → 0.023 s (135x), now 0.23x of C++ (faster); values identical to Casacore.jl. New test/indirect_tests.jl testset covers empty / inline / long (string-bucket) values across three SSM buckets, plus a _HAVE_CASACORE cross-check (the real column is all-empty, so it can't exercise those paths itself).

Phase 242 — taql() SELECT: ORDER BY / LIMIT without a WHERE, FROM, DISTINCT

Resumed the src/taql/ sweep at commands.jl's taql() string dispatcher. Its SELECT parser only understood cols [WHERE c], so SELECT A ORDER BY A (no WHERE) swallowed ORDER BY A into the column list and errored ("computed SELECT column … needs an AS alias"), even though the docstring promises ORDER BY; FROM t, LIMIT and DISTINCT were unsupported (… WHERE c LIMIT 2 errored inside the expression parser). Live-probed against real tableCommand.

Now SELECT [DISTINCT] cols [FROM t] [WHERE c] [ORDER BY k] [LIMIT n] [(INTO|GIVING) 'path'], all cross-checked live. LIMIT semantics match real TaQL SELECT (verified, and different from what a naive reading suggests): LIMIT 0 = no limit; LIMIT -k = all but the last k rows (first nrow - k, not the last k); applied after ORDER BY / DISTINCT. DISTINCT de-duplicates on the selected columns in first-occurrence order. New helper _select_rows (row-subset of a RefTable / GroupedTable result). New testset (23 assertions, incl. a real-TaQL cross-check of eight forms).

Phase 243 — IN [lo:hi[:step]] range elements

Swept a 57-form batch of WHERE expressions (comparisons, strings, LIKE/regex, booleans, arithmetic, functions, IN/NOT IN) against real TaQL. Everything agreed except TaQL's range elements in an IN list (A IN [1:5], [1:5:2], [2:], mixed [1,3:5,9]), which errored ("expected ']'") — listed as a non-goal since Phase 43. Live probing showed the real semantics differ from an interval: lo:hi[:step] is a discrete lattice lo, lo+step, … up to hi (default step 1; [lo:] unbounded above), matched by equality — B IN [1:2.5] over Double 0, .5, …, 4.5 matches only 1 and 2, not 1.5 or 2.5. Descending, zero- and negative-step ranges are errors (as in real TaQL); [:hi] stays unsupported (real TaQL rejects it too). New TQLRangeSet element + _tql_in, used by both the row (_tqleval) and group (_geval) evaluators. New testset with a real-TaQL cross-check of nine forms.

Phase 244 — T/F bool literals, string +, hex integer literals

A 60-form batch of literal / string-operation WHERE expressions against real TaQL agreed except three forms real TaQL accepts and TaQL-lite rejected: T / F boolean literals (G = T, A > 2 AND T), + as string concatenation (S + 'x' = 'abcx'), and 0x.. hex integers (A > 0x3). All three now work and are cross-checked live. Documented divergences: a column literally named T/F wins over the literal (real TaQL lets the literal win); real TaQL rejects a bare WHERE F, FALSE and 5., which stay permissive here; the 5L integer suffix is not supported. New testset (real-TaQL cross-check of eight forms).

Phase 245 — floor %///, substr/replace/bool/string, UTC mjd()

Batch-probed 83 numeric/string function and operator expressions value-by-value against real TaQL (then a 67-form detail pass). Fixed: % is floor-mod (sign of the divisor, x % 0 == x) and // is floor division with a Double result (-5 // 2 == -3.0, x // 0 == Inf) — both were truncating (Phase 24 had assumed DIVIDETRUNC); new substr / substring (0-based, negative start counts from the end, clamped), replace (literal replace-all, not regex), bool / boolean, string / str (C %g floats incl. inf, plain integers, fixed-width "True " / "False", optional printf format second argument); the no-argument mjd() / datetime() / date() / time() used local time instead of UTC (off by the UTC offset, 4 h here). Printf (stdlib) added as a dependency. rowid() remains unsupported. New testset with a real-TaQL cross-check of sixteen forms.

Phase 246 — ORDER BY: expression keys, leading direction, DESC tie order

groupby was probed first (58 g*/HAVING/WHERE/multi-key forms vs real GROUP BY): all matched except numeric aggregates over Bool, which real TaQL rejects and we accept — left as is. A 32-form ORDER BY probe then found three real gaps: (1) sort keys must be full expressions (ORDER BY A+B, abs(A), upper(S), A>3) — only bare columns worked; (2) a leading global direction (ORDER BY DESC A, B) is the default for keys without their own ASC/DESC; (3) when every key is descending the result is the reversed ascending sort, so fully-tied rows come out in reverse row order (mixed directions keep ties in row order). All three match real TaQL now (32/32 incl. ties). New testset with a 16-form real-TaQL cross-check.

Phase 247 — write commands: float→integer coercion, INSERT … [(cols)] SELECT … FROM name

110 UPDATE / DELETE / INSERT forms were applied to twin copies of a table — real TaQL vs taql() — and every column compared (46 forms clean, then 46 edge forms, then 18 coercion probes). Real gaps found and fixed: writing a floating value into an integer column errored (UPDATE t SET A = B, SET A = A / 2, INSERT … VALUES (2.9)); real TaQL truncates toward zero — now matched for both UPDATE and INSERT; out-of-range values saturate at the type's limits and NaN → 0 by our convention (real casacore's cast there is undefined behaviour that differs by architecture — ARM64 saturates, x86-64 gives typemin — so only in-range conversions are cross-checked live; found when CI on x86-64 failed the original cross-check). And INSERT INTO t [(cols)] SELECT … FROM <name|'path'>: a target column list was rejected, and a bare FROM name (the target itself, real TaQL's INSERT INTO t SELECT … FROM t) was unsupported. Deliberately not copied: real TaQL's adjacent-literal 'it''s' → its and VALUES (A) → default, and it rejects Bool↔numeric writes that we allow. New testset (cross-checks 13 forms against twin tables).

Phase 248 — mscal.baseline bare <N/>N are lengths; spw ^step

115 mscal.baseline / field / spw / uvdist selection specs compared row-for-row against real derivedmscal on the sample MS (real casacore throws on an empty selection where TaQL-lite returns 0 rows, so those were compared as "real errors ⇔ ours empty"; every uvdist form errors in this build's real casacore, so those were not comparable). Everything agreed except two forms: (1) a bare <N / >N / <=N / >=N in mscal.baseline with no & and no unit is a baseline length in metres (>3000 ≡ >3000m, >3 matches every baseline) — we treated it as an antenna-id comparison (a unit-less a~b stays an antenna-id range); (2) spw('0:^2'), a channel stride with no range, errored. Both fixed. New testset with a 13-spec real cross-check.

Phase 249 — real-casacore value-first meas.* calling forms

A 98-form probe of the meas.* TaQL functions against real casacore's meas.* UDFs found them essentially incomparable at first — real uses a different, value-first convention (meas.b1950([ra,dec] [, 'SRC' [, epoch [, pos]]]), meas.doppler('TO', value [, 'FROM']), meas.last(epoch, pos)), while Phase 97's puts the source frame first with scalar lon/lat. Both are now accepted (real's first argument is an array expression, so the two can't collide). Behaviour matches real casacore where compared: longitudes in (−π, π], meas.last returned as seconds of the sidereal day, epoch/position needed when the source frame (e.g. 'AZEL' → J2000) needs them, and an extra unused position argument tolerated. Epochs are plain MJD days and positions plain metres or an observatory name (real also takes 60454d / [x m, …] quantities — real TaQL's d/h units have a time/angle duality we don't reproduce). Not supported: the SUPERGAL frame. New testset.

Phase 250 — axis-collapse array reductions (sums, means, mins, …)

The "s"-suffixed reductions that collapse chosen axes of an array cell — flagged as an unimplemented family since Phase 186 — are now implemented: sums / products / means (avgs) / mins / maxs / medians / variances / stddevs / samplevariances / samplestddevs / avdevs / rmss / sumsqrs / anys / alls / ntrues / nfalses and fractiles(arr, frac, axes…). Semantics probed live (62 forms, all match real TaQL): axes are 1-based, given as a scalar, an array, or several arguments; the collapsed axes are dropped from the result shape (sums(V, 1) on a 3×4 cell → a 4-vector); axes beyond the array's rank are ignored; a full collapse returns a 1-element vector; axis 0 / negative / duplicate / non-integer axes are errors; variances / stddevs are population (sample* = n−1); medians / fractiles never average (lower-middle element, like gmedian). New testset with a 17-form real-TaQL cross-check.

Phase 251 — array-reshaping functions (transpose, reversearray, array, resize, diagonals, …)

Implements the array-reshaping family flagged since Phase 190, live-probed against real TaQL (68 forms, all match): transpose (reverses all axes), reversearray(arr[, axes…]) (1-based axes, each occurrence toggles so [1,1] is the identity; axes beyond the rank are ignored and, if none remain, all axes are reversed; axis 0 is an error), flatten / arrayflatten, array(v, shape…) (a scalar fills; an array is cycled or truncated column-major into the shape; the shape is one array or separate integers, not mixed), resize(arr, shape) (elements keep their index positions, cropped or zero-padded; the shape's rank may differ from the array's), diagonals(arr[, 1]) / diagonal (the first two axes must be equal-sized → shape (n, rest…); other first-axes unsupported), nullarray, isdefined / isnull, and a parse-time iscolumn('NAME') (a table-level test resolved against the table's columns). Still not implemented: iskeyword, regex/pattern/sqlpattern. New testset with an 18-form real-TaQL cross-check.

Phase 252 — group functions growid, gaggr/gstack, ghist

The group functions Phase 26 listed as non-goals, live-probed against real GROUP BY (33 forms, all match): growid() — the group's 0-based row ids (original table rows, also under WHERE); gaggr(x) / gstack(x) — collect the group's values (scalars → a vector; arrays are stacked along a new last axis, all the same shape); ghist(x, nbins, lo, hi) (alias ghistogram) — nbins + 2 integer counts: an underflow bin (x < lo), nbins equal left-closed bins, an overflow bin (x >= hi); nbins/lo/hi must be numeric literals. They compose with the scalar/array functions (sum(gaggr(X)), nelements(growid()), growid()[1]). New testset with an 8-form real cross-check.

Phase 253 — regex / pattern / sqlpattern values, glob {a,b}, keyword access, iskeyword, rowid

The last open TaQL function items, live-probed against real TaQL (40 + 26 + 25 + 10 forms):

• regex('..'), pattern('..'), sqlpattern('..') build a pattern value compared with == / != (a full-string match, usable on either side, over an expression or a per-row column argument: S == regex(Q)). regex is a regular expression, pattern a shell glob, sqlpattern a LIKE pattern. Real TaQL rejects ~ regex(..) / IN [regex(..)], so those stay errors; a constant invalid pattern (regex('[')) errors at parse time. (Real TaQL throws an unexplained "Slicer error" for a few forms — e.g. S == regex('a') — which TaQL-lite answers sensibly instead.)
• Glob {a,b} alternation (~ p/ab{c,d}/, pattern('{ab,xa}*')) — each alternative is itself a glob; a{b}c (one alternative) and a{}c (an empty one) work; , and } are literal outside braces; nested / unbalanced {, an unterminated [ and a trailing \ are errors, as in real TaQL. Bad regex / glob / LIKE patterns now raise ArgumentError rather than a raw PCRE error.
• Keyword access: ::NAME (table keyword), COL::NAME (column keyword) and .field into a Record keyword (D::MEASINFO.type) in any expression; names are case-sensitive, an array keyword is an array cell (::KWS[1]), a missing keyword or a whole-Record value errors. iskeyword('NAME') / iskeyword('COL::NAME[.field]') is the table-level test (true for a Record, false for a missing table/column/field). Resolved once per table, like mscal.* (works in query / groupby / update! / delete! / VirtualTaQLColumn); :: is only lexed outside [...], so V[1::2] keeps its step syntax.
• rowid() — the 0-based row id, rownumber() - 1 over the queried table (a WHERE / ORDER BY keeps the original row, a sub-select renumbers).

Two new testsets (139 assertions) with real-TaQL cross-checks.

Phase 254 — SUPERGAL direction frame

SUPERGAL (supergalactic coordinates) joins the direction frames, closing the last open item from the TaQL/measures sweep. It is casacore's fixed rotation off GALACTIC (MeasTable::galToSupergal = Rz(-90°)·Ry(-83.68°)·Rz(-47.37°)), so measconvert reaches it from every other direction frame (and meas.* / MEASINFO accept it). Sanity: the supergalactic pole is galactic (l, b) = (47.37°, 6.32°) and its origin (137.37°, 0°); cross-checked against the CASA measures oracle alongside the other frames.

Phase 255 — taql() SELECT LIMIT … OFFSET, OFFSET, and LIMIT a:b:s ranges

SELECT … LIMIT n OFFSET m, OFFSET m [LIMIT n] and the 0-based half-open range LIMIT a:b[:s] (every part optional) — live-probed against real TaQL (48 forms, all match). n == 0 is no limit, n < 0 gives nrow + n rows from the start row; a negative offset or range bound counts from the end; b == 0 means the end and b is clipped; an offset / start past the end, an empty range, step ≤ 0, and a range combined with OFFSET are errors. Applied after ORDER BY / DISTINCT. New testset with a 19-form real-TaQL cross-check.

Phase 256 — taql() SELECT with aggregates, GROUP BY, HAVING

taql()'s SELECT now takes [GROUP BY k, …] [HAVING cond] and g* aggregates (routed through groupby), live-probed against real TaQL (26 forms, all match sorted): aggregates without GROUP BY are one group over the whole table (SELECT gsum(K) AS X FROM t); GROUP BY takes columns or an expression (GROUP BY G+H, materialised as a hidden key); ORDER BY (output names), LIMIT / OFFSET apply to the grouped result. Behaviour fix: a non-aggregate, non-key select expression now takes the group's last row, as real TaQL does (it was the first row; mscal.* / meas.<frame>('COL') group values follow suit). Divergence: an empty single-group aggregate (WHERE K>100) is a 0-row result here, an (unexplained) error in real TaQL. CALC / COUNT commands cannot be cross-checked (Casacore.jl only wraps table results) and stay unimplemented.

Phase 257 — taql() SELECT sub-queries and table aliases

taql() SELECT gains FROM (SELECT …) (nested), x [NOT] IN (SELECT col …), [NOT] EXISTS (SELECT …), a column-less SELECT FROM t …, and FROM t [AS] a with a.COL qualifiers — live-probed against real TaQL (34 forms, all match). Sub-queries run first and are substituted (IN → a literal list, EXISTS → TRUE/FALSE); a FROM (…) result becomes the queried table. Divergence: a positive EXISTS / IN of an empty sub-query errors in real TaQL and simply matches no rows here. Only SELECT (not UPDATE / DELETE) takes sub-queries.

Phase 258 — sub-queries and aliases in taql() UPDATE / DELETE

Phase 257's sub-queries now also work in the WHERE of taql() UPDATE / DELETE (x [NOT] IN (SELECT …), [NOT] EXISTS (SELECT …); the inner query sees the table before the write), and UPDATE t [AS] a SET … / DELETE FROM t [AS] a aliases (with a.COL qualifiers) are accepted — checked on twin tables against real TaQL (9 forms, all match).

Phase 259 — taql() SELECT … JOIN

taql(target, cmd, others...) runs SELECT … FROM $1 a JOIN $2 b ON a.K == b.K ($1 is target, $2… the extra tables; columns are a.COL / b.COL; ON … IN … and the reversed order also work), live-probed against real TaQL (11 forms + every column type). Real TaQL's JOIN is a left join with type sentinels for unmatched left rows — Int → typemax(Int64), Float → NaN, Complex → NaN+NaN·im, Bool → false, String → "none" — reproduced exactly; WHERE / ORDER BY / LIMIT / aggregates compose. One condition only (real TaQL rejects AND and comma joins, as does this), and the right key must be unique.

Phase 260 — more taql() JOIN forms: chained, rowid(), duplicate keys

Live-probed against real TaQL (20 forms, all match): chained joins (… JOIN $2 b ON a.K == b.K JOIN $3 c ON b.N == c.N, each matched against the joined table so far), an index lookup ON a.K == b.rowid() (the left value is the 0-based right row), a.rowid() / b.rowid() as columns (0-based source row; sentinel when unmatched), = as well as ==, and a duplicate right key now matches its first row (was an error). Divergence: real TaQL returns NaN for the reversed index form ON b.rowid() == a.K; here it is the same lookup.

Phase 261 — taql() SELECT odds and ends

Probing ~45 more SELECT forms against real TaQL turned up four small gaps, now closed: SELECT ALL … (the default), the one-word ORDERBY, a grouped SELECT's HAVING naming a select alias (HAVING Y > 10), and an ORDER BY that is an expression over the group (ORDER BY G*-1, evaluated as a hidden column and dropped). Everything else probed already matched (LIKE, ~ p/…/, IN ranges, BETWEEN, %, rownr(), DISTINCT, multi-key ORDER BY, …). Real TaQL rejects ORDER BY gsum(K) and NOT G==3 (NOT binds tighter than == there); both are accepted here.

Phase 262 — ISM per-cell lookup is a binary search

A bucket-size benchmark (scratch build, 600 000 rows) showed getcell on a fast-changing IncrementalStMan column degrading with bucket size (A1: 26 ms → 3 429 ms for 50 000 random cells from the default to a 64 MiB bucket) — the entry lookup inside a bucket scanned the sorted row-number list linearly. It is now a binary search: flat at ~27 ms for every bucket size (124× faster at 64 MiB; no change at the default size). The same benchmark (and a tile-size one for DATA / FLAG) found no read benefit from larger buckets or tiles, so the writer defaults are unchanged. New regression test.

Phase 263 — TaQL-lite complex functions and result types

A ~130-form numeric-expression probe against real TaQL (checking values and result types) found: complex(re, im) missing; C ** 2 / pow(C, 2) on a Complex a MethodError (a Phase 184 regression — its Real-only _tql_pow); real / imag / conj of an Int and median / fractile returning an Int where real TaQL returns a Double; and scalar fractile(x, frac) missing. All fixed; every other probed form (arithmetic, integer division / modulo, bit ops, comparisons, reductions, complex abs/arg/norm/exp/log/trig, array complex functions) already matched. Real TaQL rejects NOT K>2 (NOT binds tighter than >), accepted here.

Phase 264 — fixed-shape string arrays; TaQL-lite string arrays and bool(string)

• Storage bug: a fixed-shape string array column (S S [SHAPE=[2]]) crashed write_table (a BoundsError — the column was declared Direct) and a casacore-written one could not be read ("string arrays not yet supported"). Casacore stores it indirect (option FixedShape, not Direct): each cell is one 12-byte ref to a string-bucket blob of just the elements, no shape header. Both directions now work, verified against real casacore (written by casacore → read here, and written here → read by casacore).
• TaQL-lite: a ~90-form string/date probe against real TaQL found the string functions (upper / lower / capitalize / trim / ltrim / rtrim / substr / replace / sreverse / strlength / string) and IN [...] need to be elementwise over a string array cell (they failed or collapsed to one scalar), and bool('...') must follow real TaQL's string rule (false for "", "0", "f", "false", "n", "no", trimmed and case-insensitive; it was always true). Still open: a constant expression inside an IN [...] list (S IN ['a'+'bc']). (A column named T collides with the T bool literal in real TaQL, which errors on it.)

Phase 265 — column type × shape × manager matrix (a real read bug)

Writing a matrix — 12 element types × {scalar, fixed 1-D / 2-D, variable} × {Standard, Incremental, Tiled*StMan}, in both directions against real casacore — found:

• Reading a casacore-created fixed-shape array column was garbage in StandardStMan / IncrementalStMan. A fixed-shape numeric or Bool array created by casacore ([SHAPE=[3]]) has option FixedShape but not Direct, so it is stored indirect (an offset into table.f<n>i); we read every fixed-shape array as direct (inline), returning offsets as data. Now a fixed-shape array is direct only when the column's Direct option says so (our own writer always sets it for numeric arrays), otherwise it is read through the array file (strings: the Phase 264 string blob).
• write_table could not write UInt8 (uChar), Int16 (Short), UInt16 (uShort) or UInt32 (uInt) columns — added. (Int8 / casacore Char is not a table column type.)
• A fixed-shape String array bound to IncrementalStMan crashed the writer — now goes through the array file.

After the fixes every combination round-trips in both directions (casacore-created → ours and ours → casacore; Casacore.jl cannot read a variable-shape tiled column, so those are ours-only). New test/type_matrix_tests.jl.

Phase 266 — empty and undefined: zero-row tables, never-written cells, empty selections

Probing zero-row tables and never-written cells — casacore-created → ours, ours → casacore, and a real casacore adding a row to a table of ours — found a family of bugs:

• copyms(ms, dst; rows = 1:0) / copytable of an empty query crashed (BoundsError in the tiled writers, eltype(storeddata[1]) for a virtual engine, "rowsPerBlock must be positive" for Dysco). Every manager now writes a zero-row table the way casacore does (header + dummy cube, no data): TiledShape/Column/CellStMan, all the virtual engines (including the stored / scale / offset companions), storage = :multifile. A Dysco file has no zero-row form, so an empty selection copies that column as a plain one. A copied empty sample MS validates, opens in casacore, and can be grown again by edit.
• A real casacore bus-errored adding a row to a zero-row IncrementalStMan table of ours. casacore's reader assumes every column has an entry at bucket-relative row 0 (the Phase 161 invariant); our zero-row bucket had none. It now holds casacore's default value there — byte-identical to a casacore-written file except the cache-size word.
• VariableShape now carries its number of axes (VariableShape(ndim); 0 = unknown, VariableShape() still works). A zero-row column has no cell to read its ndim from, and a tiled hypercube needs it (its header says ndim + 1) — it used to be forgotten and rewritten as 2, which casacore rejected ("mismatch in nrdim") after e.g. removerows! of every row. write_table infers it from the element type (Vector{Matrix{T}}[]); a TiledColumnStMan needs a declared fixed shape, so an empty one from data alone is a clear error. An empty column of element type Any (a zero-row groupby result) now says so instead of blaming Unitful.
• A never-written cell of a variable-shape tiled column read as an error; casacore reads an empty array. So a column of them — FLAG_CATEGORY and WEIGHT_SPECTRUM in the real ALMA MS, "defined but never written" — was unreadable (and dropped by copyms). It now reads as (0, 0, 0) / (0, 0) empties like casacore, in getcell, getcolumn and TiledCellStMan. Our tiled writers write an empty (zero-extent) cell as an undefined one — a gap in the row map on the dummy cube — so it round-trips.
• edit defining an undefined tiled cell, or changing a cell's shape, errored ("no stored cube to write into") because it took the in-place byte-patch path; it now falls back to the regenerating path (a same-shape edit still patches in place). Appended cells of a variable-shape column with nothing to copy a shape from keep the column's ndim.

New test/empty_undefined_tests.jl (zero-row casacore tables, empty copies of every manager/engine, growing from empty by our edit and by a real casacore INSERT for 112 type × shape × manager combinations, the zero-row ISM file, undefined tiled cells).

Phase 267 — keyword sets and table info against real casacore

Round-tripping table / column keywords (every value type, arrays, nested records) and table.info through CASA's casatools in both directions found:

• Bool array keywords were wrong in both directions. casacore bit-packs a Bool array in AipsIO (LSB first, ceil(n/8) bytes); we read and wrote one byte per element. A casacore-written Bool array keyword read as garbage, and one of ours (e.g. from copytable of such a table) made casacore refuse the whole table (AipsIO::getend: part of object not read). Fixed in the AipsIO array reader and the record writer.
• write_table(...; keywords) could not write numeric-array or nested-record keywords (copyms always could). It now takes arrays of every numeric type / Bool / String, and nested Dicts / Records.
• An undefined variable-shape SSM / ISM cell read as a 1-D empty whatever the column's ndim; it is now an empty array with the column's number of axes (as in Phase 266's tiled case), so a Matrix column's undefined cell is (0, 0).
• table.info's readme lost its trailing newline through a copy (casacore ends the readme with a newline; we did not write it).

Table keywords, column keywords (units, MEASINFO, hypercube sets), column comments and options, table.info, and the data-manager layout of a copy of a casacore-written table now compare equal to the original as casacore sees them. New test/keywords_tests.jl.

Phase 268 — byte order, and a memory-safety hole in the raw byte loads

Writing and reading big-endian tables (casacore's endianformat='big') across every element type × {scalar, fixed, variable} × manager, in both directions, found:

• write_table(...; endian = :big) produced an unreadable table. The storage-manager files were big-endian but table.dat always said "little-endian" — so neither our reader nor casacore could open it. edit of a big-endian table did the same on every rewrite (it kept the file byte order but flipped the flag). table.dat now carries the real byte order (both writers).
• Reading a casacore ISM table with a Direct fixed-shape String array crashed the whole process (SIGBUS). casacore stores that as [uInt total][uInt len, chars]… inline in the bucket; we assumed a string array in ISM is always indirect, took inline bytes for a file offset, and the raw-pointer load read outside the array. The reader now decodes the inline layout (and the writer can produce it), and — the real hole — every raw byte load (_ld, _rd_run!, _rd_bits!) now checks its range and raises a BoundsError, so a corrupt file or a layout we misread is an exception, not a crash.
• addrows! on a fixed-shape String array column failed (zero(String)); a rewritten string-array column no longer inherits a Direct option it is not written with.

Big-endian SSM / ISM / TiledShape / TiledColumn tables from casacore read identically to their little-endian twins, and our big-endian writes read back in casacore. New test/endian_tests.jl.

Phase 269 — cost of the Phase 268 bounds checks (investigation, no code change)

_ld / _rd_run! / _rd_bits! gained a range check in Phase 268. A/B on the real 9.8M-row ALMA MS (the C++-comparison survey, two runs each with the check on and with _chk made a no-op): UVW whole-column 20.5 vs 20.5 ms, TIME 0.83 vs 0.89 ms, DATA per-cell (50K rows) 54 vs 54 ms, FLAG per-cell 45–50 vs 46–50 ms — run-to-run noise (several ms on the tiny scalar columns, on both sides) is larger than any difference. The checks stay; the read paths are as fast as before and, against casacore C++, unchanged (TIME 0.12×, UVW 0.45×, DATA 0.93–0.99×).

Phase 270 — strings and casacore-edited tables (a sweep that found no bug)

Probed against real casacore and kept as regression tests (test/robustness_tests.jl, no source change):

• Non-ASCII, long and empty strings — héllo, 日本語, an emoji, "", embedded newline / tab, a 5,000-character and a 160,000-byte string — round-trip through every string layout (scalar, fixed and variable arrays; StandardStMan and IncrementalStMan; both byte orders) and read identically in casacore; so do UTF-8 column names, table keywords, nested-record keys and the readme. (An embedded NUL is truncated by Casacore.jl's own C-string conversion; ours keeps it.)
• Tables fragmented by real casacore — SSM and ISM tables that casacore built by inserting, updating and deleting thousands of rows (bucket splits, free lists, ISM run breaks), and tiled tables that grew by INSERT and hold hypercubes of several shapes — read identically in both readers and survive our own edit (remove / add / overwrite rows), which casacore then reads back.

Phase 271 — a MeasurementSet we write is a valid MS to real casacore

Building a reference MS with CASA's simulator (casatools.simulator) and opening ours with its ms tool found two reasons casacore rejected or choked on our tables:

• create_ms output was "not a valid MS". None of its columns carried the QuantumUnits and MEASINFO keywords casacore's MSTableImpl requires (TIME epoch/UTC, UVW uvw/ITRF, the *_DIR / POINTING directions J2000, antenna / feed positions ITRF, the spectral-window frequencies with their MEAS_FREQ_REF variable reference and code table, and the unit of every other standard column). They are now stamped from the schema, and every column we write that the simulator-made MS also has carries identical units and measure frames (cross-checked in the tests). casatools.ms opens it and getdata returns the right shapes.
• Any casacore that LOCKS one of our tables threw "another process changed the number of columns". The sync blob we wrote into table.lock used the short form (nrcolumn = -1); casacore's TableSyncData::read leaves nrcolumn unset for it, and PlainTable::lock then compares that garbage with the column count. The ms tool's getdata and tb.lock both hit it (Phase 13 only checked that the table opens). The blob is now the full form TableSyncData::write produces: the real column count, a table change counter and one data-manager change counter per manager (all set to the modify counter, i.e. "everything changed" — a full resync), for write_table and after every edit.

Interleaving casatools writes (putcol, addrows) and our edit on the same table now works in both directions. New test/msvalid_tests.jl.

Phase 272 — real casacore tools operating on our MS (a sweep that found no bug)

With Phase 271's valid MS, CASA's own tools now run on ours, and we read what they write — kept as regression tests in test/msvalid_tests.jl (no source change):

• every column of every table of a casatools.simulator-written MS (MAIN, 13 subtables incl. SOURCE, measures and all) reads identically in ours and in Casacore.jl;
• ms.split reads a copyms of the sample and writes a new MS (tiled DATA / FLAG, ISM scalars, every subtable): ours reads it back, it validates clean, and every MAIN column equals the original; ms.range, ms.msselect and tb.getcol work on ours;
• tb.putcol (DATA, FLAG, TIME) and putcell write straight into our tiled and IncrementalStMan files and we read the new values.

Phase 273 — locking against a live casacore process

Running our edit while a real casacore process (CASA's table tool) holds or has open the same table — using signal files to order the two — found:

• Our edit waits for a casacore lock and then sees its data. A tb.lock(True) + putcell + unlock in casatools, with our edit started meanwhile: ours blocks until the unlock, then reads casacore's flushed value and edits on top of it (no lost update).
• The sync counters now move the way casacore's do. Phase 271's full-form blob reused the modify counter for the table and data-manager change counters, which can coincide with a small counter casacore remembered (then it would see "unchanged"). They are now the previous blob's counters + 1, and the table counter moves only when the table's structure changed (a column added or dropped) — a data change bumps just the per-manager counters, exactly like casacore's own writers.
• Known limitation (documented, not changed): a casacore process that already has the table open re-syncs new rows / row counts after its next lock, but keeps reading the OLD cell values of a rewritten column until it reopens the table — our writers replace storage-manager files atomically (a new inode; that protects our own mmap readers and lock-free readers), and casacore keeps its open file handles. A real casacore writer, by contrast, patches in place and is seen immediately by an open casacore reader.

newer entries →