Phase 423 — DataFrame(t), query-result columns and direction conversions
A benchmark of the interop paths found three more cell-at-a-time loops. Copying a lazy column (copy(column(t, "B")), which is what DataFrame(t) does for every column) went through getindex per cell; a scalar Column, MappedColumn and ConcatColumn now copy with the bulk read. DataFrame(t) for a 1M-row, 4-column table went 572 → 37 ms. Indexing a column by a dense vector of rows (column(t, "B")[rows], which is how a query result's columns are read) read one cell per row; a selection of at least 4096 rows covering at least 1/8 of the column now reads the column once and indexes it (sparse selections keep the per-cell reads, so picking ten rows of a billion-row column stays cheap); bounds errors are unchanged. Direction conversions (measconvert to AZEL, AZELGEO, HADEC, APP, ITRF and back) called SOFA's one-shot atco13 / atci13 / atic13 / atoi13, which rebuild the whole astrometry context — the precession-nutation series, Earth ephemeris and site rotation, 64 KiB and ~50 µs — for every single direction. The context depends only on the frame, so it is now built once and cached (apco13 / apci13 / apio13, a small locked table that is cleared when full) and each direction costs the quick atciq / atioq / aticq / atoiq step: 20 000 J2000 → AZEL conversions 971 → 144 ms (1286 → 117 MiB). The split routines are the one-shot ones taken apart, and a test checks the results are bit-identical to the one-shot calls on 40 frames (more than the cache holds, interleaved) in both directions; every existing CASA cross-check passes unchanged. The mscal.* astronomy functions, which convert a direction per row, speed up with it.
Phase 424 — create_ms and copyms of a tiled MS are 5–7× faster
Timing the MS-level operations on a 20000-row, 64-channel, 4-correlation MS found create_ms at 922 ms / 901 MiB and copyms at 1020 ms / 1 GiB for ~66 MB of visibility data, while reading the same DATA column takes 2 ms. The profile put 80% of it in _pack_planes!, which copies a row's plane of every column of a shared hypercube (DATA, FLAG, WEIGHT_SPECTRUM) into the tile buffer: the element type was a runtime value (juliatype(types[k])), so every one of the ~100 million elements was a dynamic dispatch with a boxed conversion. The element type is now a function-barrier parameter (_pack_plane!), and a little-endian plane that already is a Vector{T} is a single memory copy (the element loop remains for big-endian or non-vector planes, and Bool planes keep their bit loop). create_ms 922 → 127 ms (901 → 199 MiB), copyms of 20000 rows 1020 → 195 ms (1026 → 324 MiB), copyms of 10000 rows 609 → 222 ms. A test packs planes of every element type in both byte orders, with and without an offset inside the tile, and compares the bytes with the element loop; an allocation guard bounds the marginal bytes per row of create_ms and copyms (6.5× and 11.7× the payload, 17.5× and 22.7× before). Also looked at and left alone: measconvert still allocates ~5.5 KB per direction, all inside SOFA.jl's atciq / atioq.
Phase 424 (addendum) — a numeric column read was 8× too allocation-heavy on Julia 1.10
The x86-64 Docker run on Julia 1.10 (the oldest supported version) tripped four of the allocation guards of Phases 417, 418 and 423 although the same guards pass on Julia 1.13, and a probe showed the cause: column(t, "B")[:] of a Float64 StandardStMan column allocated 64 bytes per element on 1.10 (8 on 1.13). The whole-column numeric reader walked each bucket with a closure over the runtime element type and indexed a reinterpreted view value by value, an allocating path on 1.10 that 1.13's compiler hides (the Phase 68 lesson again). It is now _ssm_read_numeric, with the element types as dispatch parameters, reading each bucket's run with _rd_run!: 64 → 8 B/element on 1.10, and copy(column) / dense selections / update! / WHERE queries on scalar StandardStMan columns with it. All four guards pass on both versions.
Phase 425 — reading a String cell allocates the String and nothing else
A scalar String cell of a StandardStMan column was copied twice on its way out: the inline reader took a byte-slice copy (ssm.data[off+1:off+len]) and built the String from that, and a value held in a string bucket went through an IOBuffer, a byte vector and then the String. Both now unsafe_string straight from the data (bounds-checked) — a value that spans several string buckets, the rare case, keeps the buffered reader. Per cell: 64 → 32 B for a short value and 384 → 64 B for a 100-byte one, 106 → 84 ns. A test round-trips strings from empty through inline, string-bucket and bucket-spanning (up to 20 000 bytes) lengths, with multi-byte UTF-8, in both byte orders, through column(...)[i], [:] and getcell, and bounds the allocation of one cell. Looked at and left alone: copyms of a tiled MS still allocates ~12× its payload, but an allocation profile shows it is the necessary copies of each cell (Array(v) per cell) plus the tile buffers, and removerows! of a 20000-row tiled MS (206 ms, 4× the data) is a genuine rewrite of the hypercube file.
Phase 426 — getcell(t, name, i) is 13× faster, and a numeric IncrementalStMan cell 22×
getcell(t, "TIME", i) in a loop — the most natural way to read cells — cost 476 ns and 496 B per call against 22 ns for a cached column(t, "TIME")[i], because every call rebuilt the Column: a description lookup, the data-manager lookup under the lock, a linear search for the column's position in its manager, and a constructor whose type parameter is only known at run time (442 ns, 480 B for the column call alone). The Column at the table's own precision is immutable and tied to the table's data managers, so a plain Table now keeps it (Table.colcache, dropped together with the data-manager cache by _evict_dms! and resync; a call with an explicit precision= is built fresh as before): getcell 476 → 37 ns. A MeasurementSet's getcell(ms, name, i) and column(ms, name) had cost another ~400 ns in the slurping args...; kw... forwarder — they have explicit methods now (1800 → 71 ns on the sample MS). The cell read itself: a numeric scalar of an IncrementalStMan column, with the element type known from Column{T}, goes straight to the typed load after the bucket lookup, like the StandardStMan one (Phase 421): 368 → 17 ns on the sample MAIN TIME. A test reads rows around bucket and run boundaries of every numeric type (including complex, big-endian) and compares with the source, checks the cache is the very object kept with the table, is bypassed by precision=, dropped by eviction and separate for a table opened at another precision, and bounds the allocation of a getcell loop.
Phase 427 — a tiled cell read is 2–3.5× faster, and a narrowed one decodes in place
Reading one cell of a tiled column (column(t, "DATA")[i]) took 1.76 µs and 4.1 KB for a 2 KB (4, 64) cell, and 0.97 µs for a FLAG plane, although the bytes are one contiguous run. read_plane computed everything from runtime values — the element type from the column description and the number of axes from the hypercube — so the array allocation, the tile arithmetic and the copy were dynamic dispatches with boxed arguments. It is now one dynamic call into _read_plane, specialised on the element type and the rank (Val{nd}): DATA at full precision 1763 → 493 ns, FLAG 974 → 392 ns, WEIGHT_SPECTRUM 462 → 414 ns. A narrowed read (the MAIN default is ComplexF16 for DATA, or precision = Float16 / BFloat16) used to read the whole ComplexF32 cell and convert it afterwards; the tiled getcell now takes astype and decodes straight into the narrow type, as the whole-column path has since Phase 35: 3758 → 1806 B per DATA cell (the conversion itself, ~1.5 ns per element, is the remaining time). A test compares narrowed and converted full reads for DATA, WEIGHT_SPECTRUM and FLAG with :half, Float16 and BFloat16 over random rows and bounds the allocation of a cell loop.
Phase 428 — getcell on a RefTable / ConcatTable goes straight to the table it points at
Phase 426 made getcell(t, name, i) cheap for a plain Table by keeping its Column objects; a query result (RefTable) or a concatenation (ConcatTable) still built a MappedColumn — or a ConcatColumn over the column of every part — on each call. getcell now has methods for both: a RefTable maps the name and the row and asks its parent, a ConcatTable finds the part with the same searchsortedlast the concatenated column uses (so empty parts are skipped) and asks that part, so the cell is read through the parent's cached column. getcell(rt, "TIME", i) 267 → 152 ns; the errors are unchanged (KeyError for an unknown column, BoundsError for a row outside the table) and the precision= keyword is passed through. A test compares every cell of a renamed-and-projected query result, a chained selection and a four-part concatenation (with an empty part in the middle and at the end) with column(...)[row], and checks the errors and the allocation of a loop.
Phase 429 — the caches of Phases 417–428 under concurrent use (one real bug fixed)
The performance phases added several caches — the per-table Column objects, the SOFA astrometry context, the compiled groupby aggregate arguments — so a stress test now exercises them from 16 tasks at once (julia -t 8): getcells of freshly opened columns racing with _evict_dms!, direction conversions over 40 distinct frames (more than the 16-entry astrometry cache holds, so it is cleared under the other tasks' feet), and concurrent groupbys with compiled aggregate arguments, every result compared with the serial one. It found a real bug in the Phase 417 code: the compiled aggregate arguments lived in a global WeakKeyDict keyed by the query's column dictionary, and under concurrent queries groupby threw KeyError: key Dict{String, AbstractVector}(…) not found (3 runs out of 3). The closure is now kept on the aggregate node itself (TQLAggr.compiled, valid for the column dictionary it was compiled against and recompiled for another one), so there is no global state and no hashing of a dictionary of million-element columns: as a side effect groupby … gsum(B*C) over 1M rows went 115 → 65 ms and gmean(sqrt(B)+C) 159 → 89 ms. The other caches passed (6 runs out of 6). The stress test runs as a child process in robustness_tests.jl (skipped under Rosetta emulation like its predecessors), next to a test that the node's cached closure follows a different column dictionary.
Phase 430 — a Performance page in the documentation
After fourteen phases of throughput work the practical advice was scattered over changelog entries, so the documentation gains a Performance page (docs/src/performance.md, between the Guide and the API reference): how to read cells and columns (bulk [:], a kept lazy Column, rawblock, dense row selections, the precision default), what the query engine compiles and why the string forms beat closures, why an edit session should do all its changes at once and what removerows! / copyms cost, how to convert many directions under one MeasFrame, and that the caches are thread safe — with the measured numbers for each. Its code examples are part of the documentation-examples test, so they cannot rot, and the Documenter build is still warning-free.
Phase 431 — corruption fuzz of the fast read paths (clean)
Phases 421–428 added fast paths that read through unsafe_load / unsafe_string / unsafe_copyto! behind an explicit bounds check (typed numeric cell loads, direct String reads, bulk gathers, cached columns, rawblock), so the Phase 376 corruption fuzz was extended to drive exactly those on every column of every corrupted table — col[i] for all rows, dense and reversed row selections, copy, the table-level getcell and rawblock — and a new deterministic sswords mode sets every 4-byte word of a StandardStMan file holding inline, bucket-held and bucket-spanning strings (plus numeric and complex columns) to extreme values. Results: 18 random runs of 150 corrupted tables (mixed, big-endian, MultiFile, engines, Dysco, RefTable) with three seeds each, 3 × 1312 corrupted ISM words and 2000 corrupted SSM words, with no crash, hang, out-of-memory kill or slow read — the bounds checks hold. The extended child and the sswords case are now part of corrupt_tests.jl.
Phase 432 — mscal.* converts only the geometry a function asks for
An mscal.* query computed, for every distinct (antenna, field, TIME), the hour angle / declination, the azimuth / elevation, the pole's azimuth / elevation, the ITRF direction and the sidereal time — each with its own SOFA astrometry context — whichever function was used, so mscal.last1() paid for three direction conversions it never read. The per-key geometry is now a small object whose fields are converted the first time a function asks for them, and the memo is typed (it was Dict{…,Any}). On the 600-row sample MS (per query, before → after): mscal.el1() 8.97 → 3.02 ms, mscal.ha1() 6.96 → 4.5, mscal.azel1() 7.18 → 2.97, mscal.pa1() 6.8 → 4.86, mscal.last1() 6.75 → 1.78; mscal.itrf() allocates 0.71 MiB where mscal.pa1() takes 5.85 (it asks for the pole and the AZEL as well). Results are identical (the real-derivedmscal cross-checks of Phases 77/280/301 pass unchanged); a guard in taql_mscal_tests.jl pins the relative allocations.
Phase 433 — SELECT DISTINCT keys rows by typed values
A benchmark of the grouping and string paths on 1M rows (groupby 50–130 ms, string == / LIKE 26–56 ms, IN 28 ms, BETWEEN 1.6 ms) found one outlier: taql(t, "SELECT DISTINCT K") took 327 ms, because every row built a Tuple of boxed values into a Set{Any} and read the columns through lazy views. The result columns are now materialised and the first row of each distinct tuple is found by _distinct_rows: one column uses a Set of its own element type, several go through a function barrier so the key is a tuple of concrete types. 1M rows: one column 327 → 4.4 ms, two columns (K, S) 292 → 35 ms; the row choice (first row of each tuple, NaN / -0.0 / array cells included) is unchanged and checked against the old definition on random tables.
Phase 434 — compiled IN [...] over a short literal list
The compiled WHERE closure for x IN [a, b, c] called the generic _tql_in per row, which scanned the list for range elements every time and compared against a Vector{Any}. A list of at most 16 plain scalars is now searched as a tuple (x in (a, b, c), unrolled, nothing boxed), with array cells and range lists keeping the generic path. 1M rows: K IN [1,2,3] 28 → 1.1 ms. A test compares the compiled and interpreted results over integer, float (NaN, -0.0), string, NOT IN, range, dynamic-element, array-cell and 18-element lists.
Phase 435 — a few cell edits of numeric StandardStMan columns patch the file in place
An edit session that overwrote even one cell of a StandardStMan column regenerated the whole data-manager instance (every column of it, strings included): 1000 t[:B][i] = v on a 1M-row table took 382 ms. When the changes are only cells of numeric scalar columns held by an uncontained StandardStMan file, no rows or columns are added or removed, and fewer than one row in eight changed, flush now compares the new values with the stored ones and writes just the differing cells at their file offsets (ssm_patch_scalars!); anything else keeps the regenerating, journaled commit. 1000 cells of 1M rows: 382 → 123 ms (most of the rest is materialising the overridden column). A side effect is that the file keeps its inode, so a casacore process that has the table open sees the new values without reopening (the Phase 273 limitation, for these columns). The cells are written one by one like in-place tile edits, so a kill mid-flush can leave some new and some old values, never a torn cell. The test patches little- and big-endian tables of Int32 / Float64 / UInt8 columns beside a string column, checks the inode, the values through both readers, and that many changes and no-op writes still work.
Phase 436 — ORDER BY a string column with few distinct values
Sorting 1M rows by a String column compared strings pairwise inside sortperm (ORDER BY S 81 ms, ORDER BY S, K 214 ms). When a key column has at least 10 000 rows and at most one distinct value per eight rows (names, modes, labels), _sorted_rows now dictionary-encodes it (_string_ranks: first-seen ids, the distinct values sorted once, then an integer rank per row) and sorts the integers; the permutation is the same because equal strings get equal ranks and the sort is stable. Mostly-distinct columns keep sorting the strings (the dictionary would not pay). 1M rows: ORDER BY S 81 → 33 ms, ORDER BY S, K 214 → 77 ms. A test compares the result with a direct string sort for ascending / descending / mixed keys, empty and non-ASCII strings, row subsets, and a mostly-distinct column.
Phase 437 — the edit session remembers which cells it set
After Phase 435 a one-cell edit of a 1M-row table still took about 110 ms, because the patch plan compared the whole overridden column (a million boxed values) with the stored one to find what changed. EditTable now keeps, per column, the indices of the cells set one by one (touched; a whole-column assignment, addrows! and removerows! mark the column "unknown" again), and the plan and the "did this column change?" check look at those cells alone. 1M rows, StandardStMan numeric column: one cell 130 → 9.9 ms, 1000 cells 145 → 11.7 ms (ISM columns still regenerate: ~0.3 s). A randomized test applies 24 mixes of single cells (some set back to the stored value), whole-column writes, string cells, removed and appended rows, with and without an ISM column, and compares the table with a plain model.
Phase 438 — update! no longer boxes the SET target columns
update! turned every SET target column into a Vector{Any} before doing anything, so on a 1M-row table even an update matching no rows read a million boxed values (138 ms) and a selective one paid it too. A concrete scalar vector is already mutable and takes the coerced values as they are, so only lazy array columns are boxed now (they must be, for the slice and mask writes). 1M rows: UPDATE .. WHERE matching nothing 138 → 2.7 ms, SET K = K+1 WHERE K == 3 178 → 26 ms, SET B = B+1 WHERE K < 5 291 → 135 ms. The sequential SET semantics (a later item sees an earlier item's write) are unchanged and tested with an Int16 target fed by a Double expression; the real-TaQL UPDATE cross-checks pass.
Phase 439 — the edit session keeps scalar columns typed
insert! of 10 rows into a 1M-row table took 570 ms: the first cell set on a column pulled the whole column into the session as a Vector{Any}, and the StandardStMan writer then stored it through its generic, bounds-checked per-value path (30% of the flush) after unboxing a million values. _materialize! now keeps a scalar column as the typed vector it reads (Vector{Int32}, Vector{Float64}, Vector{String}, …; EditTable.override is an AbstractVector dictionary), so the writers' memcpy path applies and nothing is boxed; array columns are still boxed cell by cell. 1M rows: insert! of 10 rows 571 → 373 ms. Values that do not fit the column are still refused where they are assigned. A test checks that the overridden columns are typed (Int32, Float64, String, Bool) and that appended and removed rows still give the right table; the engine, tiled, ISM, Dysco, RefTable, writer and type-matrix suites pass unchanged.
Phase 440 — COL[...] widens the selection, not the whole cell
Query expressions read columns as Int64 / Double / ComplexF64 like real TaQL, and DATA[1,1] widened the whole ComplexF32 visibility cell (2 KiB → 4 KiB, a fresh array per row) only to pick one element out of it. An index whose base is a column now indexes the cell as stored and widens what the subscript selects (the elements are converted one by one either way, so values and result types are identical; a masked selection V[cond] widens its data). 100 000 visibility rows of 4×64: real(DATA[1,1]) > 0.9 140 → 115 ms and 648 → 250 MiB, abs(sum(DATA[:,1])) > 3 169 → 144 ms and 681 → 292 MiB. A test compares the result of 13 index forms (scalars, slices, end, masked selections, arithmetic and functions over them) on complex, float and Int16 array cells with widening the cell first, values and element types.
Phase 441 — query loops stream tiled array columns in blocks
A query / groupby / update! over an array column (DATA, FLAG) kept the column lazy and read one storage-manager cell per row (~1.1 µs and one allocation each); materialising the column with [:] would have been fast but holds the whole column in memory, which a real MAIN cannot afford. _load_col now wraps a tiled array column (single hypercube, no narrowing, at least 64 rows) in a ChunkedColumn: when the loop is walking forward (the next row is at most 8 rows after the previous one) the rows from there are decoded into one buffer by the new getrows (at most 4096 rows and 8 MiB, in the same single pass [:] uses) and handed out as views; any other access — random, backwards, sparse — is the old cell-by-cell read, so a selective row list never decodes blocks it does not use. 100 000 rows of 4×64 visibilities: any(FLAG) 31 → 9 ms, ntrue(FLAG) > 130 40 → 16 ms, real(DATA[1,1]) > 0.9 115 → 73 ms, abs(sum(DATA[:,1])) > 3 144 → 102 ms; mean(abs(DATA)) is unchanged (181 ms) because widening and abs dominate it. Tests read a 9000-row column in both byte orders sequentially (across every block boundary), with skipped rows, in random order and backwards, compare with the per-cell reads for complex and Bool columns, and check that queries over a chunked column select the same rows as the materialised column.
Phase 442 — corruption fuzz of the block reader (clean)
Phase 441's ChunkedColumn / getrows read a tiled file through a block decode sized from the hypercube header, so the Phase 376 corruption fuzz was extended to it: every table read now also runs query(t, "nelements(<col>) >= 0") on each column (the query loop's lazy / chunked read), and a new deterministic tsmwords mode writes a 200-row table with a shared complex / Bool hypercube (more than the 64 rows below which a column is not chunked) and sets every 4-byte word of its tiled header file (cube shape, tile shape, row map, file records) to five extreme values, reading and querying after each: 600 corrupted headers, of which about two thirds make the table fail to open or read with an ordinary error and the rest read normally — no crash, hang or runaway allocation. The block reader's bounds checks hold; no code change besides the fuzz (corrupt_fuzz_child.jl, corrupt_tests.jl).
Phase 443 — flagging a few rows patches the bits in place
Phase 435's in-place cell patch skipped Bool columns, but a bit-packed StandardStMan Bool column is exactly what flagging touches (FLAG_ROW). ssm_patch_scalars! now sets or clears single bits (least significant bit first, one bit per row from the start of each bucket's column area, the layout the readers and the writer use) and the plan accepts Bool columns. 1M rows, FLAG_ROW next to an Int32 and a String column: 100 flagged rows through edit 352 → 9.8 ms, update! … SET FLAG_ROW = TRUE WHERE rownumber() < 500 AND K == 3 335 → 3.9 ms. The test toggles bits at the first and last rows, around byte boundaries and at random in 3000- and 20000-row tables of both byte orders, checks the file keeps its inode, and compares the table with a model through our reader and Casacore.jl.
Phase 444 — crash test of the in-place cell patch
Phases 435 and 443 write single cells straight into a StandardStMan file, outside the journaled commit group of Phase 398, so the crash matrix did not cover them. ssm_patch_scalars! now calls the commit hook after each patched cell (:patch), and a new test in crash_tests.jl "kills" an edit of 14 rows × (Int32, Float64, Bool cells) after every one of its 42 patch events: each time the table reads back, every cell is either the old or the new value (each cell is one write of at most 8 bytes, so none is torn), the rows that were not edited and the string column are untouched, and the table can be edited again (no lock is left behind). A kill after the patch but before the sync blob is written only leaves the lock file's modification counter one behind, which makes readers re-read, never read stale data.
Phase 446 — a faster test suite on CI
The last passing CI runs took 64 min (Julia 1.10), 86 min (pre) and 136 min (1.12), and one run was cancelled at the 150-minute limit. Timing the suite file by file (41 min locally) shows it is dominated by the TaQL tests: taql_query_tests.jl 694 s, taql_command_tests.jl 550 s and taql_fuzz_tests.jl 445 s, which are about a hundred testsets of 15–100 s each, most of them seeded random fuzz against real TaQL; nothing else is above 2.5 minutes. Julia 1.12 under x86-64 emulation cannot reproduce the CI slowness here (it crashes with a segmentation fault in the emulator), and the CI logs are not readable from outside, so three measures were taken that do not weaken any assertion: (1) MS_TEST_FUZZ_SCALE (default 1, so a local Pkg.test() is unchanged) scales the iteration counts of the heaviest real-TaQL fuzz loops through _fz in test/taql_helpers.jl; at 0.35 the whole suite takes 35 min instead of 41.5, with the same seeds and the same checks on the iterations that run; (2) CI.yml now runs code coverage (it makes the tests about 1.5× slower) and the full fuzz counts only on Julia 1.10, and the 1.12 and pre jobs run at scale 0.35 without coverage; (3) runtests.jl prints TEST FILE <name> <seconds> after each test file, so the next slow job can be read off its log. Whether the 1.12 job now fits comfortably can only be seen on the next CI run.
Phase 447 — coverage sweep of the Phase 437–444 code
A coverage-instrumented run of the edit, block-column, perf-guard, crash, SSM and tiled tests over the code added in Phases 437–444 (touched-cell tracking, the in-place SSM patch plan, ChunkedColumn / getrows) found exactly two branches no test executed: the patch plan's full-column comparison (a whole-column assignment t[:B][:] = new that differs from the stored column in only a few cells, so the quick look at the first rows does not rule it out) and getrows on a TiledColumnStMan column (the layout of the MS UVW: one hypercube, rows addressed directly instead of through a row map). Both turned out correct and are now pinned: the assignment test patches 3 changed cells in place (same inode) and regenerates for 900 changed ones with the same table either way, and the block-reader test chunks a 9000-row TiledColumnStMan column sequentially and in random order. No source change.
Phase 448 — thread stress of the Phase 437–444 code (clean)
The block reader of tiled array columns (ChunkedColumn / getrows) and the edit session's touched-cell tracking and in-place cell patches were added after the last thread stress (Phase 429), so they were run under many tasks at once. A new child process test (threads_block_child.jl, 12 tasks per round, julia -t 4 / 8 / 16) shares one table with a shared DATA / FLAG hypercube, a TiledColumnStMan column and scalar columns (9000 rows, more than one block) between tasks that run block-streamed queries, per-cell reads and whole-column reads, while other tasks edit private copies of it (patching Float64 and Bool cells in place) and read the result back: every query selects the same rows, every cell is the same and every edited table equals a model, in all runs. No race was found and no code changed; the test is hooked into robustness_tests.jl. A performance probe first (array-cell queries 0.5–2 µs per row; copytable, removerows! and one-cell edits of 100 000 tiled rows 40–220 ms) found nothing worth changing.
Phase 449 — random edit sessions against a model (clean)
Phases 437–444 changed how an edit session tracks and writes cells (touched-cell indices, in-place StandardStMan patches, typed columns kept in the session), and the earlier structural-edit fuzz (Phase 422) predates them. A new seeded test runs random sessions on tables of 50 to 30 000 rows (one to many buckets, so the in-place patch fires in some sessions and the regenerating path in others), both byte orders, StandardStMan and IncrementalStMan scalars (Int32, Float64, Bool, String, Float32), a shared tiled hypercube or a TiledColumnStMan next to ragged StandardStMan arrays: random cell sets, whole-column sets, addrows! and removerows!, one to three sessions in a row, the table compared with a Julia model after each session (225 checks). Nothing differed. The only surprise was in my first model: an appended ragged cell is zeros of the previous row's shape, the documented addrows! default (casacore leaves it undefined), not an empty array. A first attempt at the same phase, trimming the copies in the masked-array query path (DATA[!FLAG], 2 µs per row), made no measurable difference and was dropped. No source change.
Phase 450 — Julia against C++ again; slices of a tiled array column are read in blocks
The Julia-vs-casacore comparison (Casacore.jl, a thin CxxWrap binding over libcasacore, on the real 9 817 600-row ALMA MS, best of 3, Apple silicon) was repeated; geometric mean of the ratios Julia / C++ is 0.32 (about three times faster). Whole columns: TIME 0.9 ms vs 3.8 ms (0.25), ANTENNA1 1.5 vs 3.5 (0.44), FLAG_ROW 0.34 vs 0.46 (0.74), UVW 8.7 vs 18.3 (0.47), FLAG 275 ms vs 9.7 s (0.03); per cell (100 000 reads): sequential DATA 97 vs 109 ms (0.89), FLAG 28 vs 71 (0.40), UVW 33 vs 115 (0.29), random TIME 13 vs 775 ms (0.02); subtable columns POINTING.DIRECTION 498 vs 888 ms (0.56), POINTING.NAME 13 vs 83 (0.16), SYSPOWER.SWITCHED_DIFF 1.24 vs 2.78 s (0.45) and the small ones (CHAN_FREQ, POSITION, PHASE_DIR, SOURCE.NAME) 0.2–0.5. Random DATA cells (117 ms vs 26 s) mostly measure casacore's tile cache re-reading whole tiles, so it is not a like-for-like number. Table open is the one line where C++ looks faster (110 µs vs about 1 µs), but Casacore.jl hands back an already-open handle, so that is not comparable either (Phase 263). The only real loss was a slice of consecutive rows of a tiled array column, column(t, "DATA")[1:20000] (21.7 ms vs 17.8 ms): it was one cell read per row. A run of at least 64 rows is now decoded a block at a time with the block reader of Phase 441 and each cell copied out as an owned array (the same values, types and errors as the cell-by-cell read): 20 000 DATA rows 18.6 → 6.3 ms, FLAG 5.1 → 2.6, UVW 7.1 → 2.2. A test compares slices of 1 to 9000 rows (around the 64-row threshold and the 4096-row block size, narrowed and full precision, both byte orders, shared hypercube and TiledColumnStMan) with the cell reads, checks bounds errors and that the cells are owned, and a table with several hypercubes (where the block reader declines) falls back to cell reads.
Phase 451 — a StandardStMan file grew with the square of the row count
A tall-table check (5M and 20M rows of scalar columns: queries, groupby, copytable, update!, insert!) showed everything scaling linearly in time and about 100–600 bytes per row in memory, except that the first read of a 20M-row Float64 column allocated 3.1 GiB: the StandardStMan file our writer had produced was 2.9 GiB for 320 MB of data. The writer gave every bucket the size of the largest thing that has to fit in one bucket, and the index (8 bytes per data bucket) counted: once the index was longer than a data bucket (more than about 2·1024² rows of two Float64 columns) every bucket became as large as the index, so the file grew with the square of the row count — 4M rows 117 MiB for 61 MiB of data, 12M rows 1 GiB for 183 MiB, and every read of such a file paid for it. The index is now chained over as many index buckets as it needs, as casacore's own writer does (SSMBase::writeIndex; our reader and casacore's already follow the chain), and the bucket size comes from the data alone. The files are linear again (12M rows: 183 MiB) and small tables are unchanged except that an index longer than the data bucket no longer enlarges it. A test writes a 300 000-row table whose index needs three or more index buckets in both byte orders, reads it back (whole columns, 200 random cells), compares it with Casacore.jl, edits it through the same writer (edit and addrows!), and bounds the file size of a 4.5M-row two-Float64 table; casacore also reads a 6M-row table written this way. The other numbers of the tall check (20M rows): query K == 3 AND B > 0.5 0.37 s, sqrt(B)+C > 1 0.58 s, groupby … gmean 1.7 s, ORDER BY on the 1/100 selection 0.17 s, copytable of a selection 1.9 s, no superlinear behaviour; an insert! of 10 rows rewrites the table (5 s at 20M rows, 340 bytes per row, mostly the run-length encoder of the IncrementalStMan file on random data).
Phase 452 — an IncrementalStMan file holds what it needs, not the worst case
Following Phase 451, bytes per row of every writer were measured at 50 000, 200 000 and 800 000 rows (StandardStMan ints, floats, bools, short and long strings, fixed and ragged arrays, IncrementalStMan, shared tiled hypercube, TiledColumnStMan, MultiFile): all flat, so no other quadratic file, but the IncrementalStMan writer wasted most of its space on columns made of long runs, the case that format exists for. It fixed the rows per bucket from the worst case (every value of every row changes, 28 bytes a row for two columns) and wrote every bucket at that size, so a TIME that changes every 300 rows took as much space as random numbers: two run-length columns of 800 000 rows were 22 MB, and a whole-file read paid for it. The writer now adds rows to a bucket until its data and index part would not fit (_ism_bucket_starts; the ISMIndex already records each bucket's first row, so buckets of different sizes need nothing else, and casacore's own files have them), keeping the bucket size and the entry at row 0 of every bucket per column. Runs: 28 → 0.45 bytes per row; random values: unchanged (12). A test writes four run-length columns (Float64, Int32, String, Bool) of 400 000 rows in both byte orders, checks the file is under 4 bytes a row with buckets of different sizes, reads whole columns and 300 random cells, compares with Casacore.jl, lets real casacore UPDATE rows of the little-endian file and reads the result, and edits it with edit and addrows!. Three older tests forced several buckets with a few thousand rows and now use enough rows (and a column that changes on every other row) for that. The ISM, tiled, writer, edit, engine, crash, corruption-fuzz, type-matrix and zero-row suites pass (4059 checks).
Phase 453 — StandardStMan and IncrementalStMan files are mapped, not read
Opening the first cell of a 20M-row table read its whole 610 MB StandardStMan file into memory (35–75 ms and 611 MB for one getcell, 108 ms for a fresh query, 176 ms for a fresh column) — the tiled files had been memory-mapped since Phase 3 but these two were read. They are mapped read-only now (_dmfile_mmap, a contiguous container file maps too): the first cell costs 0.2 ms and 1 MB, a fresh query K == 3 26 ms and a fresh column of 20M doubles 27 ms. The one writer into the image, the in-place cell patch of Phases 435 and 443, now writes the cell to the file only (the bit is read back from the file for Bool); the mapping is shared, so a table that is already open sees the patched cell, and a table replaced by a structural edit (a new file, atomic rename) keeps reading the file it opened, as the tiled files always did. The corruption fuzz, crash matrix, edit, type-matrix, writer, container and Dysco suites pass unchanged (6322 checks). A test checks the first-touch allocation of a 24 MB table (2.6 MB), that an open table sees a patched cell, and that a table replaced by an edit still reads its old image.
Phase 454 — Linux check of the mapped files (clean) and a missing import
Phase 453 changed how two data managers hold their files, and file mapping, replacement by rename and truncation behave differently across operating systems, so before anything is pushed the tests that touch it were run on x86-64 Linux (the julia:1.10 container): metadata, StandardStMan, tiled, IncrementalStMan, API, precision, allocation guards, writer, indirect, ISM writer, multi-column tiled, type matrix, reference tables, engines, edit, locking, big tables, the crash matrix, Dysco, zero-row, corruption fuzz, keywords, byte order and robustness (including the threaded tests): 7601 checks passed. The one error was a test of mine: the Phase 451 test in ssm_tests.jl used Random without importing it (it passed in the full run only because an earlier file had loaded it); the file now has using Random. One caveat of mapping that is now documented here: a mapped table file that is truncated in place by another program while the table is open would make a later read fault (the tiled files always had this; our writers and casacore's never shrink a file in place, they replace it, which a mapped reader survives).
Phase 455 — a coverage run of the whole suite
The full suite (21 730 checks, 41 minutes with coverage) executes 98.4% of the 11 104 code lines of src/ and ext/ (the unexecuted ones are mostly the pre-TableRecord keyword decoder, error handlers and defensive guards). The reachable ones were probed one by one: nothing was wrong, but one was a missing feature and the rest are now tests. Feature: taql("CREATE TABLE … DMINFO [TYPE=\"TiledShapeStMan\" | \"TiledColumnStMan\" | \"TiledCellStMan\", …]) is accepted as real TaQL accepts it (the columns default to every array column; scalar and unknown columns are errors; the tile shape in SPEC is ours to choose); the three managers and shapes match a table made by real TaQL, and casacore reads ours (except TiledCellStMan, which casacore itself fails to read through Casacore.jl with "calcCacheSize: invalid arguments", for its own tables too). Tests added: an edit session that adds a column and then rows (the in-session read of an appended row is the default cell), adds and removes a column in one session, refuses a non-standard column name without data, re-encodes ScaledArray and autoscaled CompressFloat columns when a cell changes and removes their stored / scale / offset companions with the virtual column; addcolumn! of a Unitful column; ALTER TABLE … ADD COLUMN with SHAPE and NDIM against real TaQL (same shape and type); the error forms of CREATE TABLE; $VAR / ${VAR} in a stored table path; renaming a column named by a Hypercolumn_* declaration; float literals beyond the double range; validate of an MS with a broken subtable; resync of an MS with opened subtables; the KeyError of an unknown subtable. No source change besides the CREATE TABLE managers.
Phase 456 — TaQL-lite against real TaQL on the real MS; array subscripts are compiled
Ten WHERE queries on the 9 817 600 rows of the real MS (scalar columns, IN, BETWEEN, abs(…), FLAG_ROW, an array column) were timed against real TaQL (tableCommand, best of 3): the selected rows agree in every case and ours is 3–30 times faster on scalar conditions (ANTENNA1==0 AND ANTENNA2==5 18 vs 186 ms, FIELD_ID==1 5 vs 172 ms, ANTENNA1 != ANTENNA2 40 vs 260 ms; geometric mean 0.06), and 0.8 on sumsqr(UVW) > 1e6 (1.8 vs 2.3 s). The same condition written with subscripts, UVW[1]*UVW[1]+UVW[2]*UVW[2]+UVW[3]*UVW[3] > 1e6, took 25–32 s (2.5–3 µs and 3 KB per row): an index node was not compiled, so every row looked the column up in a dictionary, rebuilt the subscript closure and index tuple, and each of the five operators went through the widening and MethodError net of _bcast. Now a COL[…] whose subscripts do not depend on the row is compiled (_compile(::TQLIndex): the column is resolved once and an in-range positive integer subscript reads the element after one dynamic dispatch on the cell type, because a chunked column's element type is abstract; everything else is the same _tql_do_index as before), and _bcast of two plain Float64 / Int64 values just applies the operator (they need no widening or shape check). The indexed condition takes 5.4 s (real TaQL 11.7 s), UVW[1] > 100 1.3 s (2.0 s), 0.43–0.75 of real TaQL's time. A test on 5000-row tables (above the compile threshold) compares 12 forms — scalar and 2-D subscripts, end, negative and 0 subscripts, slices, Int16 and complex cells, OR, masked and ragged cells and the error for an out-of-range subscript — with references computed from the raw arrays, and the compiled closure with the interpreter row by row. The TaQL query, command, fuzz-against-real-TaQL, mscal and allocation-guard files pass (7466 checks).
Phase 457 — reading single cells of a Dysco column
A scale check of Dysco-compressed tables (24,000 rows of 4x64 visibilities, 300 baselines per block) showed whole-column reads, writes and copies scaling linearly, but a single column(t, "DATA")[i] taking 400 us: a cell read decodes the whole block it lives in (every baseline of an integration), and a loop over rows decoded it again for each row, so the cost grew with the table's baseline count. DyscoStMan now keeps the last decoded block of each column (replaced as a whole, so tasks reading at once are safe; blocks bigger than 64 MiB are not kept). Cells are copies, so a caller can change one without affecting later reads. 2,000 consecutive cells: 800 ms -> 3 ms. Whole-column reads, writes (47 us/row) and copytable are unchanged. Test in dysco_tests.jl.
Phase 458 — per-cell cost of every column kind; Dysco cache under threads (clean)
Read times per cell of one 100,000-row table with a column of each kind (Int32, Float64, String, Bool, ISM, fixed and tiled arrays, TiledColumnStMan, ragged, complex SSM, CompressFloat engine, string array): sequential, random, whole-column and slice reads are all 0.001-0.8 us/cell once compiled; no outlier. The Phase 457 block cache read by 32 tasks x 300 random cells x 30 rounds at julia -t 4 and -t 16: no mismatch. No code change.
Phase 459 — random edit sessions on engine and Dysco columns (clean)
Phase 449's model-checked edit sessions only covered plain storage managers. The same fuzz (cell sets, addrows!, removerows!, one to three sessions, both byte orders, 24-150 rows) now runs on a table whose array column is a ScaledArrayEngine, CompressFloat, BitFlagsEngine or a Dysco column (RowNorm, 16 bits), against a model: exact for the lossless cases, within quantisation for the others. 80 random tables, no difference. Test in edit_tests.jl.
Phase 460 — real casacore reads the engine tables our edit sessions rewrite (clean)
After every random edit session of Phase 459 on a CompressFloat or BitFlagsEngine table, Casacore.jl reads the rewritten table (a copy at a fresh path — casacore keeps an opened table's files, so a second session at the same path in one process would read the old ones) and agrees with the model: 40 random tables, no difference. Dysco has no casacore decoder in Casacore.jl and is checked against the model only. The check is part of the Phase 459 test in edit_tests.jl.
Phase 461 — the Dysco writer is 3-4x faster
Profiling the writer (8,400 rows of 4x64 visibilities, 10-bit data): 47 us/row, 26 KiB allocated per row. The block fill read every element of every cell through an untyped Vector{Any} element (a dynamic dispatch per value); it now goes through a typed function barrier (_dysco_fill_block!). 12-17 us/row without dither and 20-23 with (AF/RF/Row normalisation alike), 10 KiB/row. The files written are byte-identical (checked for the three normalisations). A ragged column given to dysco= is now an ArgumentError naming the column (it was a MethodError on the shape). Tests in dysco_tests.jl.
Phase 462 — writer survey: fixed-shape array cells 23x faster, engine encode 95x
Writing 50,000 rows of each column kind (us/row): every writer was at 0.03-0.9 except three. A fixed-shape array column in StandardStMan took 2.56 (it wrote the cell element by element from an untyped Vector{Any} entry); the cell is now copied by a typed method with one memcpy on a little-endian host (0.11 us/row, bytes identical, other byte orders and strided cells keep the element writer). The ScaledArray / ScaledComplex / BitFlags encoders took the stored type as a bare ::Type argument, which Julia does not specialise on: 26 ns instead of 2.5 us for a 4x8 cell, whole-column writes 3.56 -> 0.87 and 1.57 -> 0.76 us/row. Reads of engine columns were already fine. Tests in ssm_tests.jl (fast path equals element writer for 9 element types, both byte orders, dense and strided cells).
Phase 463 — assigning a whole tiled column in an edit session is 70x faster
A survey of edit-session operations (1,000 cell sets, a whole-column set, addrows!, removerows!) on 50,000-row tables of every column kind found one outlier: t[:A][:] = values on a tiled array column took 3.2 s. The flush wrote the cells one at a time and each write opened the tile file, mapped it, wrote and synced it. Cells are now written through one mapping per hypercube with one sync (tsm_setcells!), and a cell whose leading axes are untiled (the usual layout) is one contiguous run (_wr_run!, one memcpy on a little-endian host). Whole column 3.2 s -> 45 ms, 1,000 scattered cells 43 -> 2 ms; the other column kinds were already between 1 and 100 ms. Test in tsm_multicol_tests.jl (6 element types, both byte orders, a shared hypercube, a plane of another element type, a wrong-shape write leaves the file alone). A copytable survey of the same kinds found nothing (0.03-2 us/row).
Phase 464 — query-engine cost survey on array columns (nothing to fix)
Thirty-three queries over 100,000 rows with 4x16 float, complex and Bool tiled array cells, a ragged column, strings and an ISM key. Scalar conditions cost 0.00-0.07 us/row, string functions 0.03-0.07, array reductions on a cell 0.15-0.5 (mean(A) 0.23 = 3.5 ns per element), sort-based ones (median) about 1, the masked mean(A[F]) 1.4, ORDER BY 0.03. One ratio stands out and is left open: gmean(mean(A)) grouped by a key that is not in row order costs 0.84 us/row against 0.23 for the same expression in a WHERE, because each group's rows are spread over the table and the block reader decodes a block per access; grouping by a key in row order, or evaluating the argument for all rows in row order first, would remove it. No code change.