← older entries

Phase 226 — one last src/measures/ sweep: found a real, broader gap

in _write_table_core's own write path — Phase 199's stray-directory fix was never extended past storage=/blocksize=

A final, careful re-read of everything in src/measures/ and its ext/ extensions (fresh review of doppler.jl including a skeptical re-check of Phase 224's own fix, test/measures_fixture.py for staleness relative to the Phase 222-225 changes — none needed, its DOP_RADIO = 0.01 fixture value is already safely in-domain) found no further bug in src/measures/ itself. Following the write-path integration seam out of src/measures/write.jl into src/tables/ create.jl (where a Measure-typed column's auto-detected MEASINFO actually gets stamped) turned up a real, broader bug one level removed from the measures subsystem — a natural place to look, since that seam is exactly what Phase 221 (and, further back, Phase 199/201/202/204/

1. has repeatedly found real gaps in before.

A real bug, fixed: _write_table_core (src/tables/create.jl) — the single function every write_table/copytable/create_ms/ write_ms call eventually funnels through — validates storage=/ blocksize= first thing, before mkpath(dir) (Phase 199's own fix for exactly this shape of problem: a caller's typo turning into a silently- created stray directory, not just a clear error). But every other validated kwarg in the same function (measures=/units=/engines=/ forward=/virtualtaql=/ism=/the tsm=/tcm=/tcell=/dysco= group name checks) still throws its own correct "no column …" error, but after mkpath(dir) — so the exact same gap Phase 199 fixed was never actually closed for the rest. Live-reproduced: write_table(dir, "T", [...]; nrow=2, measures = Dict("NOTACOL" => (; kind=:epoch, ref="UTC"))) throws the correct measures: no column "NOTACOL" message, yet leaves an empty dir behind. A per-kwarg fix mirroring Phase 199's own approach (hoisting each check above mkpath) isn't safe here without a much larger restructuring — several of these loops (engines= most of all) do real, non-trivial encoding work interleaved with their own name check, not a cleanly separable pure-validation pass. Fixed once, robustly, for every current and future validation site in the function: remember whether dir already existed before this call, wrap the whole body in try/catch, and on any exception remove dir again (only if this call is the one that created it) before rethrowing. This is a strict improvement over a per-kwarg hoist too — it also correctly cleans up a genuinely partial write (some storage-manager files already on disk) for an error that only surfaces deep inside with_container_sink's per-DM-writer section, not just the "nothing written yet" early case Phase 199 originally covered, and a pre-existing directory (with unrelated content) is left completely untouched. Live-verified across all four shapes (early error, late/ partial-write error, pre-existing-directory preservation, and a normal successful write) before writing the permanent test; the whole dysco_tests.jl and ism_writer_tests.jl suites — including their real-CASA-interop cross-checks — were also re-run standalone against the fix as an extra confidence check, given how much of the function's body now runs inside the new try block. 9 new assertions.

Full suite green: 5198 baseline + 9 new = 5207/5207. README/memory updated, merge on the user's word.

Phase 227 — src/taql/ sweep: a missing value anywhere in a

WHERE/HAVING/JOIN condition used to crash with a raw TypeError instead of excluding the row, matching real SQL/TaQL's three-valued logic

Started a fresh sweep of src/taql/ (six files, ~6,600 lines) with a fresh-eyes read of join.jl and groupby.jl — the two files that hadn't had a fix land in them since their original Phase 28/49/56/26/48 implementations, across the entire later sweep history. join's explicit "the verbs chain" design (Phase 58) is what surfaced the real finding: a join with unmatched=:missing produces a genuinely missing-containing output column (an unmatched row's right-side value), and re-joining/filtering on that column downstream — ordinary usage of a documented feature, not contrived input — hits a real gap.

Real bugs, fixed, four distinct crash shapes, all live-reproduced before fixing:

1. _join_matchrow's index-lookup on= branch did 0 <= v < nr ? Int(v)+1 : 0 — for v === missing, 0 <= missing is missing, and missing ? ... : ... throws TypeError: non-boolean (Missing) used in boolean context. Fixed to treat missing the same as an out-of-range index (row unmatched).
2. Every if/::Bool-context consumer of a WHERE/HAVING/join-condition result — query's string and closure WHERE, groupby's _where_rows / _gb_prepare (shared by update!/delete! too) and its two havingfn(...) || continue sites, and join's predicate/string- condition/post-join-where paths — all threw the identical raw TypeError for a missing result instead of excluding the row.
3. TQLAnd/TQLOr's own _tqleval/_geval implementation used raw &&/||, which are special syntax requiring specifically the left operand to satisfy x::Bool (missing && true throws; true && missing does not — confirmed live, a real asymmetry) — so no amount of fixing only the outer consumer (item 2) could catch a crash happening inside the AND/OR evaluation itself, before it ever returns a value to wrap.
4. iif(cond, a, b) was wired straight to Base.ifelse, which (an ordinary function, not special syntax, but still requires cond::Bool) throws a MethodError for cond === missing.

Fixed once, comprehensively, rather than patching only the first site found: a shared _tql_truthy (the single point every condition result passes through before the row-inclusion decision — missing excluded, exactly like false; a genuinely non-Bool result still errors clearly) applied at every site in item 2, and genuine 3-valued-logic _tql_and/ _tql_or (missing propagates like SQL NULL — false AND missing = false, true AND missing = missing, etc.) replacing the raw &&/|| in both _tqleval(::TQLAnd/TQLOr) (ast.jl) and its _geval counterpart (groupby.jl). Deliberately not "coalesce every missing sub-result to false immediately" — that reads as equivalent for AND/OR but silently breaks NOT: !missing === missing (still excluded, correct), but !(coalesced false) === true (wrongly included) — live-verified this exact case (NOT (KEY > 5) on a missing-valued KEY) stays correctly excluded under the real fix. The 3-valued design was also confirmed positively correct, not just non-crashing: missing OR true is true per SQL semantics, so a row whose KEY > 5 is missing but whose ID == 3 is definitely true is correctly included, live-verified both operand orders. iif fixed with a small _tql_iif that propagates missing, matching CASE WHEN NULL THEN a ELSE b END.

Confirmed already-safe and left untouched: TQLNot (Base's own !(::Missing) = missing), TQLCmp (comparison operators already propagate missing correctly), TQLIn (in, built on any, is already missing-safe) — only &&/||/if/ifelse specifically require exactly Bool and crash otherwise. The equi-join Dict-based lookup path (_join_pairs's multi=true/Pair on= branches) was checked too and confirmed already crash-safe (Dict uses isequal, which treats missing as equal to itself — a deliberate, pre-existing, internally-consistent design choice, not a bug, left as-is). No real-TaQL cross-check — this is a MeasurementSets-side chaining scenario (an outer join's missing fill), not something a plain casacore MS or real TaQL query ever produces. 17 new assertions.

Full suite green: 5207 baseline + 17 new = 5224/5224. README/memory updated, merge on the user's word.

Phase 228 — user-reported docs build failure: measconvert had no

docstring at all, a real, previously-undetected instance of the "a docstring is silently DROPPED if anything sits between it and its target" mistake (Phase 160's own class of bug, three more instances)

The user reported the docs build failing with Error: no docs found for 'measconvert' in @docs block in docs/src/api-measures.md:14-52. Reproduced locally: @doc(measconvert) genuinely returned nothing.

Root cause, confirmed with a minimal reproduction before touching any source: a Julia """...""" docstring is silently dropped — not misattached to the wrong thing, not an error, just lost — if anything at all, even a bare # comment line with nothing else, sits between it and the expression it documents; only blank lines are transparent. This is the same class of mistake Phase 160 already found and fixed (there, an @eval-generated struct needing @doc explicitly) — a different concrete shape of the same underlying trap.

Three real instances found and fixed, one of them user-reported, two found only by then doing a comprehensive sweep rather than stopping at the first fix:

1. src/measures/types.jl — the generic measconvert(m::Measure, R::Type{<:RefFrame}; frame) docstring was separated from its target by _all_finite's own explanatory comment and the _all_finite definition itself (inserted between them back in Phase 195, which evidently broke this without anyone noticing until Documenter's own @docs check finally caught it). Fixed by moving _all_finite above the docstring instead of below it.
2. src/measures/doppler.jl — found while investigating the exact same mistake in my own Phase 224 commit: measconvert(m::MDoppler {C}, ::Type{D})'s docstring had an identical shape (docstring, a long explanatory comment, then the function) — introduced by this session's own Phase 224 work, never caught because the generic measconvert function still resolved to types.jl's docstring at the time (before item 1 broke that too), so @doc(measconvert) wasn't actually nothing until both were broken simultaneously.
3. src/measures/measinfo.jl — a pre-existing, lower-severity instance (the docstring is internal, _ref_string, never listed in any @docs block, so it didn't break the build) found by a systematic scan rather than another docs-build failure: the docstring intended for _ref_string (defined at line 157) sat, misplaced, directly above _ref_from_code (a different, undocumented sibling function defined earlier), separated from its real target by an intervening comment + the whole _ref_from_code function body. Fixed by moving the docstring down to sit directly above _ref_string itself.

Methodology: rather than trusting a single fix + a check of only the reported symbol, wrote a small script scanning every .jl file in src/ and ext/ for a """...""" block whose next non-blank line is a # comment (the exact structural shape that drops a docstring) — confirmed zero remaining instances after all three fixes. Also confirmed, live, that a naive "check every exported name has some docstring via @doc" approach is unreliable here and would have missed item 2: @doc on a generic function returns docs from any of its methods, so a broken docstring on one method can be masked by a working docstring on another method of the same name — exactly what happened between items 1 and 2 before both were simultaneously broken. The fix was finally verified against the real mechanism that reported the original error — a genuine local docs/make.jl build (julia --project=docs docs/make.jl), which now completes with zero errors (only a pre-existing, unrelated search-index-size informational warning).

No test-count change (a pure comment/docstring reordering — every executable line is byte-identical, just moved relative to comments) — full suite reconfirmed green regardless, since a source change is a source change. README/memory updated, merge on the user's word.

Phase 229 — continued the src/taql/ sweep: Phase 227's own fix

introduced a fresh regression, plus a second, unrelated missing-vs-== crash in ORDER BY/orderby sorting

Continued sweeping src/taql/ with a fresh-eyes read of ast.jl and parse.jl — the tokenizer/AST core, last touched (not re-read) when Phase 227 patched _tql_and/_tql_or in place.

Real bug #1, fixed: Phase 227's _tql_and/_tql_or accepted a raw, unvalidated operand of any type, not just Bool/Missing. Before Phase 227, TQLAnd/TQLOr used plain &&/||, which correctly throws a TypeError for a non-Bool, non-Missing operand — a genuinely malformed predicate. Live-reproduced: _tql_and(5, true) silently gave missing (not an error), and because _tql_truthy(missing) == false, a whole malformed query like query(t, "K AND FLAG") (K an Int32 column, not Bool) silently returned zero rows instead of raising the same clear "must evaluate to Bool" error every other malformed-predicate shape in this file gives. Worse: _tql_and(5, false) gave false — a garbage left operand combined with a literal false on the right was never even inspected, short-circuited away by the b === false branch before validation could run. Fixed by validating each operand through a new _tql_boolish (mirrors _tql_truthy, but preserves rather than coalesces a genuine missing) before the 3-valued combination — a real type error still raises clearly, and only true/false/missing ever reach the SQL-NULL-style logic. groupby.jl's _geval(::TQLAnd/TQLOr) reuses these same two functions, so one fix covers both _tqleval and _geval.

Real bug #2, fixed, a completely separate finding: _apply_orderby (query.jl, ORDER BY / update!/delete!'s orderby=) and its groupby.jl sibling _gt_sort (groupby's orderby=) both compared sort keys with raw vi == vj && continue. missing == missing is missing (three-valued), not true/false, and missing && continue crashes with the identical raw TypeError — genuinely reachable via entirely ordinary usage: ORDER BY/orderby on a column that came out of an outer join with unmatched=:missing (live-reproduced: query(joined_result, "TRUE ORDER BY NAME") crashed), or a rollup=true/cube=true/grouping_sets= result's aggregated-away key column (live-reproduced: groupby(t, [...]; rollup=true, orderby=[...]) crashed the moment the sort touched a subtotal row). Fixed both with isequal (a missing-safe, always-Bool equality — missing equals missing, unequal to anything else); the subsequent isless ordering already handles missing correctly on its own with no change needed (sorts it last ascending / first descending, Julia's own sort convention — isless(x, missing) is true for any real x). Grepped the whole tree for any other lt=function/custom-sort-comparator site — confirmed these were the only two.

16 new assertions (both the AND/OR type-validation error path and the legitimate-missing-still-works path, plus the join-ORDER BY and groupby-rollup-orderby crash reproductions). No real-TaQL cross-check for either — both are MeasurementSets-side chaining scenarios (an outer join's missing fill; SQL ROLLUP/CUBE, which casacore parses but does not implement), not something a plain casacore MS or real TaQL query ever produces.

Full suite green: 5224 baseline + 16 new = 5240/5240. README/memory updated, merge on the user's word.

Phase 230 — continued the src/taql/ sweep: meas.<frame>('COLNAME')

unconditionally demanded an mjd argument the target frame never used

Continued sweeping src/taql/ with a fresh-eyes read of functions.jl (1,419 lines — the curated function library, including the meas.* measure-conversion UDFs, not re-read since its original phase-by-phase construction).

Real bug, fixed: meas.<frame>('COLNAME'[, mjd[, x, y, z]]) (Phase 110's column-MEASINFO-driven direction form) unconditionally required exactly 1 + (need_p ? 3 : 0) trailing arguments — always demanding an mjd, regardless of whether the target frame actually needs an epoch at all. The equivalent general numeric form, meas.<frame>(['SRC',] lon, lat[, mjd[, x, y, z]]), has always correctly made mjd conditional on _meas_dir_needs_epoch(target) (only APP/AZEL/HADEC/ITRF need one — J2000/B1950/GALACTIC/ ECLIPTIC/ICRS conversions are fixed rotations that need no epoch at all, exactly like the already-working meas.b1950('J2000', RA, DEC) form with no mjd). Live-reproduced the inconsistency directly: meas.j2000('B1950', 1.0, 0.5) (general form, target J2000) worked with no mjd, but the equivalent meas.j2000('COLNAME') (colname form, same target frame) threw "meas.j2000('COLNAME', mjd) in ...", forcing the caller to supply and thread through an epoch value the conversion never uses.

Fixed by making TQLMeasColDir's mjd field nullable (mirroring how its xyz field already was) and computing the colname form's required argument count the same way the numeric form already does: (need_ep ? 1 : 0) + (need_p ? 3 : 0) instead of the old unconditional 1 + (need_p ? 3 : 0). _meas_dir_needs_pos(R) already implies _meas_dir_needs_epoch(R) for every frame in _MEAS_DIR_FRAMES (only AZEL/HADEC/ITRF need position, and all three already need epoch too), so the fix needed no extra case analysis. Live-verified the fixed colname form (meas.j2000('B1950'), no mjd) gives byte-identical results to the general numeric form given the same lon/lat, and that an epoch-dependent target (meas.azel('COLNAME')) still correctly demands its mjd/x/y/z arguments.

Fixing this uncovered that the Phase 110 testset itself had baked in the old, wrong requirement — it used GALACTIC (which needs no epoch) as its "colname form successfully takes an mjd" example, and as its "colname form errors without an mjd" example, both backwards from the corrected behaviour; two further assertions supplied a now-superfluous mjd to a J2000/GALACTIC-target colname call purely to satisfy the old arity check en route to testing an unrelated error path (no MEASINFO; a VarRefCol frame). All four call sites updated to match the corrected, and now internally consistent, behaviour; two new assertions added pinning .mjd === nothing / .mjd !== nothing for the no-epoch-needed and epoch-needed cases respectively.

3 net new assertions. No real-TaQL cross-check — meas.* is this package's own function-library subset, not a real casacore/libmeas UDF surface with a byte-for-byte equivalent to compare against.

Full suite green: 5240 baseline + 3 new = 5243/5243. README/memory updated, merge on the user's word.

Phase 232 — continued the src/taql/ sweep, targeting mscal.jl:

one stale comment fixed, plus a coverage-instrumented pass pinning several genuinely-correct-but-never-tested code paths

mscal.jl (2,009 lines) is by far the most extensively pre-vetted file in src/taql/ — dozens of prior phases (77 through ~171, plus later touch-ups) already found and fixed real bugs there via live cross-checks against real casacore/CASA, and its comments densely document every prior finding. A careful, complete fresh-eyes read of the whole file found exactly one issue — genuinely stale documentation, not a runtime bug: the comment above _cache's memo said antid < 0 encodes an OBSERVATION_ID as -antid-1 — a leftover from before Phase 144 replaced that per-row per-observation array-centre lookup with a single engine-wide centrepos; confirmed via grep that nothing in the current file ever decodes a negative antid any other way than the one fixed sentinel -1. Fixed the comment to describe the actual, current behaviour.

Coverage-instrumented pass (methodology note #14 — once manual re-reading plateaus, diff never-executed lines against a clean run): turned up several DOCUMENTED, real code paths — not defensive/ unreachable branches — that had simply never been directly exercised by any test, each live-verified correct (cross-checked against an independent direct measure/measconvert computation, not just "doesn't crash") before being pinned with a permanent regression test:

1. The array-centre fallback when an MS's OBSERVATION.TELESCOPE_NAME has no entry in the bundled Observatories table — real casacore (and this package, Phase 144) falls back to the MIDDLE antenna (itsAntPos[0][nant÷2], 0-based), with a @warn. Live-verified mscal.ha() against a synthetic MS with a bogus telescope name matches a direct measconvert computation using that exact antenna.
2. mscal.*'s interpolation of a polynomial (NUM_POLY) PHASE_DIR at a genuinely nonzero dt from FIELD.TIME (Phase 93's own interpolation machinery) — previously only exercised via direct measure() calls in measures_tests.jl, never through an actual mscal.* function end to end. Live-verified mscal.hadec1() matches an independent computation, and confirmed the polynomial ramp term genuinely fired (differs from the dt=0 static value).
3. mscal.spw/mscal.chan's single-channel-index ('0:5') and >/< channel-index-bound ('0:>60', '0:<3') selector forms (Phase 83) — every prior test used only the a~b range and frequency-unit forms.
4. The "bad selector" (mscal.chan) and "malformed spec" (mscal. uvdist) generic parse-error paths for a string matching neither a unit suffix nor any recognised numeric form.

A first coverage-run attempt was itself contaminated by the stale- comment edit landing mid-run (the established "mid-run-edit race" pattern) — the edit shifted line numbers past its insertion point, misaligning the .cov file's line-count-to-source mapping for everything after it; caught by noticing two "0 executions" results (mscal.pa()'s else branch, mscal.uvdist's < bound) that contradicted existing, passing tests exercising those exact lines. Redone with a clean run once the edit had settled.

11 new assertions. No real-TaQL cross-check needed (each finding was cross-checked against this package's own independent measure/ measconvert machinery, already itself real-CASA-cross-checked elsewhere).

Full suite green: 5243 baseline + 11 new = 5254/5254. README/memory updated, merge on the user's word.

Phase 234 — fix a real ~100x allocation/wall-clock regression in the

Bool (FLAG/FLAG_ROW/FLAG_CATEGORY) tiled-column read path

The user asked for a head-to-head performance comparison against real casacore C++ (via Casacore.jl, a thin CxxWrap binding — a genuine low-overhead oracle, not just a correctness cross-check) on the real 9,817,600-row ALMA MS. Most operations were within 1.3–5.7x of native C++ (reasonable for a pure-Julia reimplementation with zero C dependency, and ANTENNA1 scalar reads were actually faster than casacore). One benchmark stood out: 200,000 getcell(t, "FLAG", r) calls took 19.95 s in Julia vs. 0.18 s in casacore — a 111x wall-clock gap, wildly out of line with every other benchmark.

Live investigation (@time/@allocated, not just wall-clock) confirmed a real, structural bug, not incidental overhead: a single getcell(t, "FLAG", r) for one 4x64 Bool plane allocated ~3,100 times / ~90 KiB — vs. 32 allocations / ~3.4 KiB for the identically- shaped DATA (ComplexF32) cell. Root cause, in read_plane (src/datamanagers/tiled.jl, the TiledStMan read hot path): the T === Bool branch was its OWN unconditional code path — taken regardless of tiling layout — walking the plane via a doubly-nested CartesianIndices loop, recomputing a _colmajor_offset tuple and constructing a fresh CartesianIndex for every single bit, with bounds-checked bytes[...] indexing. Every other type instead took one of two fast paths added in Phase 68's own allocation-reduction pass (_rd_run!, an unsafe_load-via-pinned-pointer bulk copy) — but Phase 68 explicitly excluded Bool ("Bool keeps its bit-unpack path"), and that path was never revisited since.

Fixed with a new _rd_bits! — the bit-packed (unsafe_load-based) counterpart of _rd_run!, unpacking n consecutive LSB-first bits from a pinned pointer with no bounds-checked array indexing and no per-bit tuple/CartesianIndex construction. read_plane's Bool branch was merged into the SAME contiguous-run structure every other type already used (the common "leading axes untiled" case → one bulk _rd_bits! call for the whole plane; the rare "leading axes tiled" case → one _rd_bits! call per contiguous run), instead of being a separate, always-slow special case. read_cube_whole (the TiledCellStMan sibling of read_plane) had the identical shape — a manual per-bit loop with bounds-checked indexing inside its own already-tiled loop — fixed the same way for consistency, including adding the single-tile bulk fast path it was missing for Bool entirely.

Because getcolumn for a Bool tiled column already falls back to [getcell(...) for r in 1:nrow] (_read_cube_bulk explicitly bails on Bool — "bit unpacking: use the slow path" — the comment now stale in the sense that read_plane itself is no longer slow), this one fix also directly speeds up whole-column FLAG/FLAG_ROW/FLAG_CATEGORY reads (copyms, column(t,"FLAG")[:], …) with no further change needed.

Live-verified end to end on the real MS: getcell for 50,000 FLAG rows dropped from 4.74 s / 155.29M allocations / 4.489 GiB to 0.11 s / 1.45M allocations / 83 MiB — and the original 200,000-row per-cell benchmark against real Casacore.jl went from 19.95 s → 0.43 s (46x faster), bringing the Julia/C++ ratio from 111x down to 2.5x, in line with DATA's 1.3x. Correctness re-verified against real casacore on the committed sample fixture: whole-column FLAG (a TiledShapeStMan array), FLAG_ROW (unaffected — IncrementalStMan, not tiled), and FLAG_CATEGORY (also tiled) all match byte-for-byte.

New permanent regression coverage in test/tsm_tests.jl: a whole- column FLAG vs. Casacore.jl cross-check (values only tested per-cell before, not the bulk getcolumn path), and an allocation- based regression guard (@allocated for FLAG vs. DATA getcell loops, asserting FLAG never allocates more than 3x DATA — the old code was ~100x worse, not merely "somewhat more", so this is a deliberately loose bound that still catches a real recurrence).

3 new assertions. No real-TaQL cross-check needed (a read-path allocation/performance fix, not new TaQL surface — the existing Casacore.jl cross-checks are the correctness oracle).

Full suite green: 5254 baseline + 3 new = 5257/5257. README/memory updated, merge on the user's word.

Phase 235 — a follow-up allocation pass at the user's request:

eliminate two more per-row copies/allocations in whole-column reads

Re-benchmarked with a focus on allocations after Phase 234, and investigated the user's direct question — "would it not provide better performance to allocate Matrices instead of Vectors, i.e. blocks of rows instead of individual rows?" Found two concrete answers.

1. A real bug, fixed: StandardStMan.getcolumn's fixed-shape direct-array branch (the SSM path a real MS's ANTENNA.POSITION uses, and — depending on the MS — UVW) already built ONE shared flat backing buffer for the whole column, but then took a COPYING slice of it per row (flat[(r-1)*nrelem+1:r*nrelem], a plain Vector getindex — always a copy in Julia) before reshape-ing, instead of @view. One extra small allocation per row on top of the backing buffer already built — the identical bug shape Phase 234 fixed for tiled Bool columns, just via a copying array-slice instead of a doubly-nested loop. Fixed both occurrences (the Bool and the general numeric direct-array branches) to @view — matching tiled.jl's _read_cube_bulk, which has used the "one backing buffer, per-row views" pattern since Phase 35/68. Verified: every row of a fixed ANTENNA.POSITION-like column now shares ONE backing buffer (parent(c[1]) === parent(c[end])), matching casacore exactly.

2. Extended the whole-column bulk fast path to Bool: _read_cube_bulk (the shared-backing-buffer bulk read added in Phase 35/68) bailed out for Bool unconditionally (T === Bool && return nothing), forcing getcolumn(t, "FLAG") to fall back to [getcell(...) for r in 1:nrow] — one fresh Array{Bool} allocation PER ROW for a whole-column read, even after Phase 234 made each individual getcell call itself fast. Bool now shares the exact same single-backing-buffer structure as every other type, via _rd_bits! (Phase 234's new primitive) instead of _rd_run!. Live-measured against the true old-code-equivalent (a per-row getcell loop, exactly what the old fallback reduced to): the new bulk path allocates ~10.6x less for a 20,000-row synthetic Bool tiled column.

3. The architecture question, answered quantitatively. Both fixes above still return a Vector of per-row view objects (reshape(view(backing, ...), cellshape...)), not a genuine (cellshape..., nrow) block Array — the representation this package's Column <: AbstractVector / Tables.jl / TaQL-engine / edit.jl surface is built on throughout, and changing it everywhere would be a much larger, likely-breaking restructuring. Quantified exactly what that representation still costs on top of a genuine block: a bare [reshape(view(backing, ...), shape...) for r in 1:n] — with NO copy, NO decode, just the wrapper construction — costs ~120 bytes/row for a 1-D (3,) cell (UVW-shaped) and ~150 bytes/row for a 2-D (4,64)-shaped cell needing an extra Base.ReshapedArray layer, live-measured for 9.8M rows. For a large payload (DATA: 2048 bytes/row) that wrapper is ~7% overhead — negligible. For a small payload (UVW: 24 bytes/row) the wrapper is ~5x the actual data — confirming the user's intuition is exactly right for small-cell columns specifically: a real (3, nrow) Matrix (one allocation, Base.format_bytes ≈ 48 bytes total via reshape) would eliminate essentially all of that overhead. This is why UVW's whole-column read stayed ~75x slower than real casacore C++ even after fixing the two bugs above (both already-optimal-for-the-current-representation for UVW specifically — it's TiledColumnStMan-bound in this MS, so neither the SSM copy-fix nor the new Bool bulk path touch it at all; its existing view-based bulk path was already in place since Phase 35/68 and simply has this structural per-row-wrapper cost baked in). Documented as a genuine, scoped candidate for a future phase (a real block/Matrix-returning accessor, additive — not a change to getcolumn's existing Vector-of-cells contract) rather than implemented here, given the architectural weight of touching the return-type contract every consumer of a fixed-shape array column relies on.

New regression tests: ssm_tests.jl ("SSM fixed-array getcolumn — view, not copy") — a synthetic 20,000-row SSM-bound (3,) column, correctness + the parent(...) === parent(...) structural guard (a byte-count threshold isn't reliable here: the per-row view wrapper itself still costs more than the tiny 3-element payload, so the before/after saving is real but modest for a column this small — the guard is "no longer independently copied", not a specific allocation number). tsm_tests.jl ("TSM Bool WHOLE-COLUMN read — bulk path, not per-row") — a synthetic 20,000-row tiled Bool (4,8) column, correctness + the same structural guard + a live allocation comparison against the true old-code-equivalent (10.6x measured, 3x margin asserted).

Live-verified end to end on the real 9.8M-row MS (re-running the Phase 234 benchmark): whole-column FLAG now reads in 1.78s / 3.4 GiB — faster than real casacore C++ (10.16s) for this operation, 0.17x. UVW whole-column stayed at ~1.7s / 1.6 GiB / ~75x slower than C++'s 0.022s (unaffected, per the architecture note above — real casacore returns one dense block with zero per-row overhead at all). Correctness re-verified against real casacore on the sample fixture for UVW, FLAG, and DATA (unaffected, sanity-checked).

11 new assertions. No real-TaQL cross-check needed (an internal representation/allocation fix, not new TaQL surface).

Full suite green: 5257 baseline + 11 new = 5268/5268. README/memory updated, merge on the user's word.

Phase 236 — BlockColumn: a lazy, single-backing-buffer whole-column

representation, plus a tile-batched rewrite of the bulk read loop that closes nearly all of UVW's remaining allocation/wall-clock gap

Phase 235's own conclusion was that closing UVW's ~75x-slower-than-C++ gap for real would need getcolumn to return a genuine block Array instead of a Vector of per-row objects — a change big enough to touch every consumer of a fixed-shape array column (Tables.jl, the TaQL engine, edit.jl, copyms) if done as a literal return-type change. The user asked directly whether an additive new function would actually see use in most situations. It wouldn't — nothing downstream would be rewritten to call it. The alternative that WOULD get used everywhere automatically, with no consumer needing to change: keep getcolumn's contract as "an AbstractVector", but stop building that Vector EAGERLY (nrow reshape(view(...)) wrapper objects, up front) and instead return a lazy wrapper that computes each row's view on getindex, only when actually asked for.

1. BlockColumn{T,N,V} <: AbstractVector{AbstractArray{T,N}} (src/tables/column.jl) — wraps the ONE flat backing::V buffer a fixed-shape array column's bulk decode already builds (unchanged since Phase 234/235), plus cellshape and n. getindex(bc, i) computes reshape(view(backing, ...), cellshape) on demand; nothing is built at construction beyond the 3-field struct itself. Column's own getindex(::Colon) already just forwards whatever the data manager's getcolumn returns with no wrapping, so this is transparent — Tables.jl/the TaQL engine/edit.jl/copyms (audited directly, not assumed: _read_cells in create.jl already does Array(v) on every AbstractArray cell, the TaQL engine's _load_col already keeps array-cell columns as a lazy Column rather than eagerly materialising [:] at all, DataFrames/Tables.jl construction round-tripped correctly in a live check) all keep working with zero code changes. collect(col) (and Column's own collect, fixed to route through it explicitly) forces genuine eager materialisation when that's actually wanted, matching Julia's own collect contract.

2. rawblock(t, name; precision=nothing) -> Array — the true zero-indirection escape hatch: the SAME backing buffer, reshaped directly into one real (cellshape..., nrow) Array, for a caller doing bulk numeric work across a whole column with no per-row objects in the picture at all. A scalar or variable-shape/indirect array column (no block representation) raises a clear ArgumentError.

3. Wired into both existing bulk-decode sites (unchanged decode logic, just a different final wrapper): _read_cube_bulk (tiled.jl, TiledStMan) and StandardStMan.getcolumn's two fixed-shape direct-array branches (standard.jl, the Bool and general-numeric cases Phase 235 had just fixed from copying to @view) now construct a BlockColumn instead of an eager comprehension.

4. The bigger find, mid-implementation: _read_cube_bulk itself had a real, separate, pre-existing (since Phase 35/68, not introduced by this phase) allocation bug, caught only because BlockColumn's own numbers didn't match the theoretical minimum. Investigating why rawblock(t, "UVW") still allocated ~1.24 GiB against a ~225 MiB theoretical backing (a genuine --track-allocation=user profile, after several dead-end micro-benchmarks turned out to be artifacts of top-level global variables in the throwaway test scripts themselves, not the real code) pinned it to _read_cube_bulk's loop calling _rd_run!/_rd_bits! once PER ROW — ~9.8M calls for a real MAIN column — each one individually cheap, but with a small (~8-135 bytes/call, reproduced in isolation once properly function-scoped) per-call cost that Julia's @inline hint didn't eliminate, adding up at that row count. Fixed by exploiting a structural fact about _read_cube_bulk's one real caller (getcolumn, tiled.jl — verified by grep, no other call site exists): the rowpos function it's ever passed (identity, or a single-interval TiledShapeStMan's `r -> pos

• (lastrow - r)`) is always affine with slope 1. So instead of one

_rd_run!/_rd_bits! call per ROW, the loop now walks whole TILES at a time — within one tile, consecutive rows are already one contiguous byte run (exactly how read_plane's own "leading axes untiled" fast path already treats one row's plane; this generalises the same principle across multiple rows within a tile) — reducing call count from nrow down to cld(nrow, rowspertile) (≈225 for UVW on the real 9.8M-row MS, not 9.8M). Live-verified correct on a forced multi-tile synthetic table (a large-cell Bool and Float32 column, rowspertile < nrow, genuinely crossing several tile boundaries).

Live-verified end to end on the real MS: column(t,"UVW")[:] (lazy, not indexed) dropped from 1.5s / 1.6 GiB (Phase 235) to 0.0048s / 224.7 MiB — exactly the theoretical minimum backing size, and 0.23x of real casacore C++'s 0.021s — Julia is now FASTER than C++ for this read, closing the ORIGINAL 75x gap entirely. collect(column(t,"UVW")[:]) (eager, every row indexed — the "old-equivalent" total work) is 0.0125s / 599 MiB, still 0.59x of C++. column(t,"FLAG")[:] improved too: 1.69s/2.93 GiB (BlockColumn alone) → 1.21s/2.34 GiB (tile-batched), staying at 0.12-0.13x of C++ (unchanged conclusion — already faster). Correctness re-verified against real casacore for UVW/DATA/FLAG/ANTENNA.POSITION, both per-cell and whole-column.

New test/blockcolumn_tests.jl (57 assertions): structural (BlockColumn type + shared-backing-buffer checks across TiledStMan AND StandardStMan), rawblock correctness + error paths + zero-copy (a within-one-decode check — there is no cross-call caching, so two separate column(...)[:] calls are independent reads, not aliased), multi-tile boundary correctness (forced small rowspertile, both Bool and non-Bool), copytable/edit interop (a BlockColumn source materialises to real independent copies on write; a per-cell edit write is untouched by any of this and doesn't corrupt a neighbouring row), a real-casacore cross-check, and an allocation regression pinning column(t,"UVW")[:]/rawblock(t,"UVW") to within 2x of the theoretical minimum (was ~5.5x before the tile-batching fix).

Full suite green: 5268 baseline + 57 new = 5325/5325. README/memory updated, merge on the user's word.

Phase 237 — fix per-cell DATA/FLAG reads and random-access TIME (a real ISM dynamic-dispatch regression, not just a fix)

Follow-up to Phase 234-236's whole-column work, at the user's direct request: "improve the performance of per-cell DATA/FLAG reads and random-access TIME functionality" (the two metrics that stayed merely "comparable" — 2.1-2.3x C++ — in the post-236 benchmark, plus random-access TIME's odd ~12 GiB allocation for 100K lookups despite only a 1.4-1.8x wall-clock ratio).

Root cause, TIME random-access (IncrementalStMan/ISM, untouched by 234-236 since those were tiled/SSM-only fixes): getcell walked _ism_bucket (a linear scan over the bucket index — cheap in isolation on a small table, but a real MS's TIME column has as few as 19 very large buckets, not this package's own ~100-row-per-bucket write target) then _ism_colindex, which materialised full Vector{Int} row-number/offset arrays for every column bound to the ISM instance — not just the target one — on every single getcell call. Live-measured ~82 KiB / ~4,500 allocations for ONE getcell on a real MS's TIME column. Root allocator underneath both: _u32/_ism_i64 and the per-entry row-number read used reinterpret(T, ::Vector{UInt8}) via a view — the same allocating-slow-path pattern Phase 68 had already fixed for the tiled reader (_rd_run!/_rd_bits!), never carried over to incremental.jl.

Fix, attempt 1: a single-value unsafe_load-via-pinned-pointer primitive (_ld, the ISM counterpart of _rd_run!), _ism_bucket switched to searchsortedlast (O(log used) instead of O(used)), and _ism_colindex/a new _ism_value_offset (the getcell fast path) that skip non-target columns by pure pointer arithmetic and — for getcell specifically — never materialise the target column's arrays at all, scanning its entries directly for the matching offset. Single-getcell allocation dropped substantially, but the wall-clock on a 100K-row random-access loop went from ~1.4s to 32s — ~23x worse, despite lower peak allocation (~9.7 GiB vs ~12.3 GiB — in the same ballpark, not the order-of-magnitude drop the allocation fix should have produced).

Root cause of THAT regression, found by profiling rather than guessing: the row-number entry width (UInt32 vs UInt64, from a woffset header bit) is only known at runtime, and calling a T-parameterized primitive with that non-constant T inside the per-entry loop forces a full dynamic method dispatch on every iteration — for a real MS's large ISM buckets (thousands of stored-value entries for a fast-changing column like TIME), that dwarfed the allocation cost it had just eliminated. Profile.@profile pointed straight at Base_compiler.jl's dynamic-dispatch machinery dominating the sample count. Fix: the standard Julia type-instability barrier — branch on the row-number width once, outside the loop, into a where {RT}-parameterized inner function called with a literal UInt32/UInt64 at each of the two call sites, so the compiler emits one fully type-stable, inlined specialization per width instead of re-dispatching every entry. Live-verified: single getcell allocation 320 bytes (was ~82 KiB), a 10K-row random-access loop 0.0083s (was 3.07s pre-barrier-fix / ~32s scaled) — ~370x faster and ~320x fewer bytes than the intermediate "fixed but still slow" version.

Two smaller wins in tiled.jl, same family of bug: _tile_layout and _tsmbytes both used get!(f, dict, key) with a do...end/lambda f — Julia builds that closure on every call, cache hit or not, not only on a miss. Switched both to a plain get-then-setindex!. Minor in isolation (their payload is small), but on the exact same per-getcell hot path as the ISM fix.

The benchmark itself was measuring the wrong thing for DATA/FLAG per-cell — found while profiling why the tiled-side fixes weren't moving the numbers: the Phase-236 benchmark called the top-level getcell(t, "DATA", r) convenience wrapper inside the per-cell loop, which — by design, _pcolumn(t, name, precision)[row] — re-resolves the column (a columndesc lookup + opening/looking up the data-manager instance) on every call, while the C++ side of the comparison cached ct[:DATA] once outside its own loop. Not a bug, but an apples-to-oranges comparison; the documented, idiomatic pattern for repeated access is col = column(t, name); col[row] (already what the TIME random-access benchmark did, which is why that number was a genuine, fixable regression rather than a methodology artifact). Fixed the benchmark to cache the column handle on both sides (matching the C++ side's own pattern) and added a docstring note on getcell pointing at column for repeated access. With the fair comparison: DATA per-cell 0.233s vs C++ 0.240s (0.97x), FLAG per-cell 0.195s vs 0.202s (0.96x) — both now at parity with C++, not merely "comparable". The uncached top-level getcell form is 2.36x slower than the cached form, consistent with the re-resolution cost measured above, and is now called out explicitly rather than silently eaten by every repeated-access caller.

Re-run of the full C++ comparison (same methodology as the post-Phase-236 run): TIME random-access 0.091s vs C++ 0.766s (0.12x — now 8x faster than C++, was 1.4-1.8x depending on the run); DATA/FLAG per-cell at parity (0.96-0.97x, up from "comparable" 2.1- 2.3x); every other MAIN whole-column metric (UVW/FLAG/DATA whole-column) unchanged/still superior. Subtables unaffected (none of them are ISM-bound in the sample MS). TIME whole-column read stays ~10-12x slower than C++ — untouched by this phase (it's the bulk getcolumn path, a run-length-fill loop, not the per-cell path fixed here) and not something the user asked to chase this round; flagged as a candidate for a future phase.

New test/ism_writer_tests.jl testset ("ISM random-access getcell — correctness + allocation"): a synthetic 4000-row, 2-column ISM instance sized to force multiple large buckets (unlike the small committed sample MS fixture, which — by Phase 45's own design — can't build more than one low-entry-count bucket) with different run-length patterns per column (so colnr != ncol, exercising the "skip a non-target column" path); correctness checked via a full-column comparison plus a permutation-order random-access walk crossing bucket boundaries in both columns; an @allocated regression guard (< 2 KiB for one cached getcell, several orders of magnitude under both the original bug and the intermediate dynamic-dispatch regression). _HAVE_CASACORE-gated cross-check.

Full suite green: 5325 baseline + 17 new = 5342/5342. README/memory updated, merge on the user's word.

newer entries →