Phase 274 — our edits of tables real casacore last modified: a differential fuzz found edit corrupting them
Sweeps of casatasks workflows on our MSs (clearcal, flagdata clip, statwt, concat, virtualconcat, mstransform incl. multi-MS partitions, calibrater addcorr/addmodel), tb.addrows / removerows / addcols / removecols / renamecol on our tables, and wide (300 columns), tall (2 million rows) and heavy-cell ((1000, 1000) Float64) tables in both directions all found nothing (real virtualconcat and MMS outputs — genuine ConcatTables — read identically to their parts; casacore refuses removing rows of a tiled table or dropping an ISM column, our tables and its own alike). A differential fuzz — random putcell / addrows / removerows applied alternately by us and by casatools, compared with a Julia model after every step — found a real bug:
editof a table casacore had last modified could corrupt it. A regenerated StandardStMan file carries our bucket geometry, but thetable.datblock that describes it (the per-column offsets within a bucket) was only rewritten when rows were added. After casacore'sremoverows(its own geometry stays intable.dat) an edit of any column of a shared StandardStMan instance left every string of that instance unreadable — aBoundsErrorin our reader, and casacore itself misread the file too.editnow rewritestable.datwhenever a regenerated block differs from the one on disk. (For tables we wrote ourselves an unchanged row count gives an identical block, which is why nothing before saw it.) Regression test: casatools removes rows then we edit, plus a deterministic ours/casacore ping-pong ofputcell/addrows/removerowsover every storage kind (test/msvalid_tests.jl).
Phase 275 — structural edits alternating with casacore (sweep, no bug found)
Phase 274's differential fuzz extended with column operations: random addcolumn! / removecolumn! by us and addcols / removecols / renamecol by casatools, mixed with cell / row edits, on little- and big-endian tables (16 seeds × 16 rounds, checked against a Julia model and against Casacore.jl after every step): all clean. Kept as one deterministic run in test/msvalid_tests.jl.
Phase 276 — shared tiled hypercube edited by casacore and us (fuzz sweep, no bug found)
The ours ↔ casatools fuzz on a MAIN-like table: DATA / FLAG / WEIGHT_SPECTRUM in one TiledShapeStMan plus an ISM and an SSM column, random putcell / addrows by either side, both byte orders (12 seeds × 14 rounds, casacore side read back through casatools since Casacore.jl cannot read variable-shape tiled columns): all clean. Kept as one deterministic run in test/msvalid_tests.jl.
Phase 277 — TaQL-lite random-expression differential fuzz vs real TaQL (no divergence found)
A generator of random expressions compared with real tableCommand: 2700 WHERE clauses by matched row set (arithmetic, comparisons, AND/OR/NOT, LIKE/ILIKE, IN, BETWEEN, ~ glob patterns with {a,b}, string functions) and 600 numeric expressions by computed value (rounding, %, //, **, pow, iif, min/max, trig, exp, fmod, int): zero mismatches (the first round's apparent mismatches were the generator's — an unparenthesised NOT S == '', which real TaQL rejects because NOT binds tighter than ==, and real TaQL erroring on an empty result). Kept as a seeded 250-expression guard in test/taql_fuzz_tests.jl.
Phase 278 — TaQL-lite write commands: random UPDATE / DELETE / INSERT sequences vs real TaQL (no divergence in TaQL-lite)
Random command sequences (UPDATE ... SET col = expr [WHERE], SET FA[k] = expr, DELETE, INSERT ... VALUES) applied to twin tables — ours through taql, real through tableCommand — with every column compared after each command: 80 seeds × 12 commands, no divergence. Two things the fuzz turned up that are not ours: (1) real casacore drops set Bool bits when it deletes rows from a bit-packed Bool column (INSERT ... T then a later DELETE reads F; reproduced on a table casacore created itself and read identically by Casacore.jl), so the Bool column is not compared after a DELETE; (2) F is a reserved word (False) in real TaQL, so a column named F cannot be referenced there (ours lets a column of that name win). Also -0.0 % 4 is -0.0 in real TaQL, 0.0 here (signed zeros compared equal). A seeded 6-seed guard is in test/taql_fuzz_tests.jl.
Phase 279 — mscal.baseline: autocorrelation rows (found by a random-spec fuzz vs real derivedmscal)
A random-spec fuzz of mscal.baseline / field / spw / uvdist against real derivedmscal, on a randomised copy of the sample MS (the sample's own rows never have ANTENNA1 == ANTENNA2, so autocorrelations had never been exercised), found one real divergence: a bare antenna list ('15', 'ea1*', '1~10', '8~19,2~17', '<9', …) selects cross-correlations touching the set only, and a leading ! negates that, so a negated list keeps every autocorrelation — including the listed antennas' own. We included the listed antennas' autocorrelations in the positive form (and dropped them in the negated one). & (cross only), && (both) and &&& (auto only) already agreed. mscal.feed keeps the old rule (there FEED1 == FEED2 is the normal case). 700 random specs afterwards: no mismatch. Also learned: real casacore's grammars are stricter than ours — mscal.field('!0'), a ! after a comma or & in a baseline spec, and &<9 are parse errors in real and accepted here (a benign extension, like >=). Regression test: test/taql_mscal_tests.jl.
Phase 280 — J2000 → ITRF direction: casacore's is the apparent place (found by comparing mscal value functions on varied rows)
A value fuzz of the mscal.* functions against real derivedmscal on a randomised copy of the sample MS (TIME spread over a day, random antennas and fields — the sample's own rows are nearly one instant) showed ha / hadec / azel / pa / last agree to a few arc-seconds, but mscal.itrf() was a constant ~17″ off and delay() ~18× worse than it should be. casatools' me.measure(dir, 'ITRF') shows the same offset from us, so the divergence was the direction conversion itself: casacore's ITRF direction is the apparent place (annual aberration and light deflection) rotated to the terrestrial frame, whereas ours was the plain GCRS → ITRS rotation of the geometric direction. (It is the ~13″ "EOP / aberration model difference" that Phase 92 saw in the EarthMagneticMachine geometry and accepted.)
MDirectionconversion to/fromITRFnow goes through the apparent place (atci13/atic13+ Earth rotation angle and polar motion). Agreement with casatools: 8.5e-5 → 1.5e-6 rad;mscal.itrf()1e-4 → 4.5e-6 rad;mscal.delay()2.2e-9 → 1.2e-10 s (all vs real derivedmscal). Conversions of an ITRF direction into AZEL / HADEC / APP are consistent with that now too.MBaseline/MuvW(and the phase centre used as the uvw pole) keep the plain rotation for ITRF ↔ celestial frames: live-compared withmscal.uvwj2000(), the aberrated route is further from casacore's baselines (1.1 m vs 0.66 m on ≤ 36 km baselines). A residual of ≲ 7e-5 of the baseline (a few arc-seconds of rotation) between ouruvwj2000()and casacore's remains — its MBaseline route applies something we do not reproduce (not chased). Thedelay()↔uvw_j2000()w-component relation test therefore loosens to the aberration size (2e-4).
Phase 281 — TaQL-lite reads MAIN at full precision; mscal.stokes rescale for the pseudo magnitudes
A random mscal.stokes fuzz (480 specs: I/Q/U/V/IQUV/LIN/CIRC/single correlations/the five pseudo types, ± rescale, on DATA / FLAG / WEIGHT) against real derivedmscal, on a copy of the sample MS with random data and both a circular and an edited-to-linear POLARIZATION CORR_TYPE, found two things:
- Query expressions computed from MAIN's visibility columns were ~1e-3 (relative) off real TaQL.
readtableon a Measurement Set defaults MAIN'sFloat32/ComplexF32columns to half precision (Phase 34), and the TaQL engine inherited that, somean(DATA),abs(DATA[1,1]) > x,mscal.stokes(DATA, ...)and friends were evaluated onComplexF16values. TaQL-lite now reads its columns at full precision (precision = :full, asedit/copymsalready do). The half-precision default forms[:DATA]/column(t, "DATA")is unchanged; per-cell reads stay lazy, so the cost is decoding, not memory. mscal.stokes(..., 'Ptotal' | 'Plinear', true)was 2× real — the pseudo types were built from the unscaled I/Q/U/V; withrescalethey use the rescaled ones (the angle / fraction types were scale-invariant, which is why only the magnitudes showed it).
After both: 480 random specs, no mismatch. Regression test in test/taql_mscal_tests.jl.
Phase 282 — measures: UTC on a leap-second day (found by a random epoch/position/direction fuzz vs casatools)
250 random (epoch 2000–2030, ITRF position anywhere, direction) cases converted J2000 → B1950 / GALACTIC / ECLIPTIC / SUPERGAL / APP / AZEL / AZELGEO / HADEC / ITRF / ICRS and compared with casatools me.measure: everything agrees to ≲ 1″ (most far better) except one case 6″ off — 2012-07-01, the day after the 2012-06-30 leap second (0.45 s of hour angle), and a few 2–4″ cases in 2027+ (beyond the IERS data: casacore's table and EarthOrientation.jl's predictions differ; not ours).
Cause: an MEpoch{UTC} carries casacore's UTC MJD (every day 86400 s long — an MS TIME / 86400; the leap second is not represented), but SOFA's UTC "quasi-JD" counts the fraction of a leap-second day out of 86401 s, so on such a day the two disagree by up to a second and the UT1 − UTC we derived ran away by 0.25 s per quarter-day (casacore: constant). The MJD is now converted to/from SOFA's convention at the SOFA boundary (_utc_sofa / _utc_from_sofa); on the leap day TAI − UTC is a constant 34 s (35 s after), UT1 − UTC continuous, and 300 random epochs (40 % on leap-second days) match casatools' TAI / TT / TDB / UT1 to ≲ 5e-4 s. Regression test in test/measures_tests.jl.
Phase 283 — random-frame conversions vs casatools: frequency / radial velocity / reverse direction routes (sweep, no bug found)
Two more random fuzzes of measconvert against casatools me.measure:
- Frequency and radial velocity between the velocity frames, 200 random epochs / ITRF positions / source directions, 14 frame pairs: the constant-velocity hops (LSRK, BARY, LSRD, GALACTO, LGROUP, CMB) agree to ≲ 1e-9 m/s; hops through the Earth's motion (TOPO, GEO) to < 0.9 m/s (3e-9 of c) — the ephemeris floor, uncorrelated with the diurnal or orbital line-of-sight speed, the input velocity, or the epoch (Phase 141 had already looked at it).
- Direction conversions from every frame (
B1950,GALACTIC,ECLIPTIC,SUPERGAL,APP,AZEL,AZELGEO,HADEC,ITRF,ICRS) toJ2000/GALACTIC/AZEL/APP(150 random cases): all within 1.5″ (the EOP floor) — after the Phase 280 ITRF and Phase 282 leap-second fixes — exceptB1950→AZEL/APP(up to 5.5″). That one is casacore's: its directB1950→APPdiffers by the same 5″ from its ownB1950→J2000→APP(checked in casatools), i.e. a route inconsistency on their side; ours composes through ICRS.JTRUE/JMEAN/BMEANare not frames here.
A fixed-seed 136-assertion guard is in test/measures_tests.jl.
Phase 284 — TaQL-lite GROUP BY: random queries vs real TaQL (one divergence: one-row sample variance)
700 random SELECT keys, g*(expr)… [WHERE] GROUP BY keys [HAVING] queries (one or two key columns incl. String / Bool keys, one to three g* aggregates — gcount / gsum / gmean / gmin / gmax / gvariance / gstddev / grms / gmedian / gsamplevariance / gsamplestddev / gfirst / glast / gproduct — over random expressions, random WHERE / HAVING) compared with real TaQL by group key and value. One divergence: the sample (n−1) variance / stddev of a one-row group is 0.0 in real TaQL (Julia's var / std gave NaN) — gsamplevariance / gsamplestddev and the per-element gsamplevariances / gsamplestddevs now return 0.0 there. (The running / boxed sample variants keep their own behaviour: real TaQL throws on a window of fewer than two.)
Not ours: real casacore's gmax of an all-negative group returns 2.2e-308 (its running maximum starts at DBL_MIN, the smallest positive double), so gmax(D) on such a group differs and HAVING gmax(D) > 0 keeps groups that should be dropped — an upstream bug, not reproduced (the guard skips it). Kept as a seeded 60-query guard in test/taql_fuzz_tests.jl.
CI fix — Phase 281's real-casacore Stokes cross-check on Julia 1.10
CI (Julia 1.10 only) failed the Phase 281 mscal.stokes cross-check against real derivedmscal: 12 of its 16 assertions. Reproduced locally with Julia 1.10.12 (juliaup): real casacore prints Error: no BLAS/LAPACK library loaded for cgetrf_() / invert of singular matrix attempted and returns unconverted garbage — its StokesConverter inverts its conversion matrix with the LP64 LAPACK symbol cgetrf_, which Julia 1.10's libblastrampoline does not forward (1.10 ships only the ILP64 …64_ OpenBLAS; 1.12 / pre do), so casacore's own conversion is broken there. Not our bug: the test now probes real casacore's Stokes I first and skips the derivedmscal comparison (with an @info) when it disagrees, and on a real mismatch prints the expression and worst element. The data was also given a dominant Stokes I (co-polar terms) so the PFtotal / PFlinear ratios stay well-conditioned whatever the RNG stream. (Locally: the whole taql_mscal_tests.jl passes on Julia 1.10.12 and 1.13.)
Phase 285 — TaQL-lite array expressions: random queries vs real TaQL (ten divergences fixed)
A seeded random generator of expressions over array cells (3×4 Float64 / Int / Bool cells: arithmetic, comparisons, AND / OR / NOT / iif on arrays, the axis-collapse reductions sums … fractiles, transpose / reversearray / resize, the running* / boxed* windows, slices and masked arrays via arr[boolexpr] / replacemasked) compared value by value with real TaQL (SELECT expr AS R). Fixed:
AND/ORof Bool arrays raised "an AND/OR operand must evaluate to Bool" — a Phase 229 regression (the Bool/Missing-only validation forgot arrays). They combine elementwise again (a scalar broadcasts; a masked operand keeps the unioned mask).iiftakes an array condition (elementwise); theifelseMethodErroris gone.- Two array operands must have the same shape (real TaQL: "array shapes mismatch"); we silently broadcast (
A + sums(A,1)"worked"). running*/boxed*widths: a scalar width applies to the first axis only (we applied it to every axis —boxedsum(A,2)on a 3×4 cell is a 2×4 result, not 2×2); an array is padded to the rank (running: 0, boxed: 1) or truncated; a boxed width ≤ 0 means the whole axis; a negative running half-width is an error.- Masked arrays (
A[A>5],marray):running*/boxed*reduce the unmasked elements and return a masked array (masked where the window / bin holds none, and at therunning*edges); the axis-collapse functions do the same (sums(A[A>5],1)masked where a column is fully masked;anys/alls/ntrues/nfalsesstay unmasked,allsover nothing isfalse);transpose/reversearray/resizecarry the mask (resizepads with unmasked zeros);flattenkeeps only the unmasked elements; integer / slice subscripts of a masked array select data and mask (an all-integer one gives the bare element, otherwise the rank is kept) and a Bool subscript replaces the mask (it does not combine). - Fully masked scalar reductions are
0(real TaQL:mean(A[A>0])= 0.0,stddev= 0.0,product= 0.0, …; we gave NaN / an error /1forproduct);any/all/ntruekeep their empty values; scalarfractileof a masked array works. resize(arr, shape)takes exactly one integer-array shape (resize(A, 2, 2)andresize(A, 3)are errors in real TaQL; we accepted them).
Benign leniencies left (real errors, we accept): arithmetic on Bool operands, a bare : slice (A[:,1]), (expr)[i] on a parenthesised expression. Real casacore segfaults on iif with a masked branch (the generator avoids it). The old Phase 108 assertions that baked in the scalar-width-on-every-axis rule were corrected. Kept as a seeded 150-expression guard in test/taql_fuzz_tests.jl plus a verified-forms testset in test/taql_query_tests.jl.
Phase 286 — TaQL-lite GROUP BY: array-cell aggregate arguments vs real TaQL
Continuing Phase 285's array-expression sweep into the aggregate position: gmean(mean(A)), gsum(sums(A,1)[2]), gvariance(min(A[A>0.0]) + D), and similar — a g* aggregate whose argument is itself an array-cell expression rather than a bare scalar column. 2000 random SELECT K, g*(expr) AS X FROM t [WHERE …] GROUP BY K queries across four seeds found no divergence from real TaQL beyond the already-known upstream gmax-of-an-all-negative-group bug (Phase 284) — the array expression engine and the group-reduction machinery compose correctly. Kept as a seeded 300-query guard in test/taql_fuzz_tests.jl.
Phase 287 — TaQL-lite UPDATE on array-cell columns vs real TaQL (two divergences fixed)
Differential fuzz of UPDATE ... SET on array-cell columns (whole-array RHS, subscript slices, boolean masks, the masked (D, M) = expr[cond] pair form) against real TaQL, extending Phase 278's write-command fuzz and Phase 285's array-expression sweep. 1000 random queries found two real bugs:
- A scalar Bool subscript (
FA[B]whereBis a per-row scalar Bool column, not an array) was silently accepted as an integer index via Julia's ownInt(::Bool)—FA[false]gaveFA[0](aBoundsError),FA[true]gave the wrong element (FA[1]). Real TaQL requires a Bool subscript to be shaped like the array ("… must be an array"); now a clearArgumentError. update!skipped structural validation entirely whenWHEREmatched zero rows — every SET item is evaluated lazily, only against the matched rows, so a structurally invalid SET target/RHS on anUPDATEwith an empty match set silently "succeeded" with no error, while real TaQL type-checks the whole SET list once, independent of how many rows actually match. Fixed by validating every spec once against row 1 (into a throwaway copy, never persisted) whenever no row is matched.
Kept as a seeded 200-query guard in test/taql_fuzz_tests.jl.
Phase 288 — a real MMS bug: MeasurementSet's keyword-subtable concatenation only ever saw the first part
Swept two previously-unswept source files, src/schema.jl (the standard-schema validate machinery) and src/measurementset.jl (the high-level MeasurementSet API, including its MMS/ConcatTable subtable path). validate's keyword-mismatch, missing-keyword, and missing-subtable branches were completely untested (only "missing column" / "wrong type" had coverage) — confirmed all three correct and closed the gap with permanent tests; stdcolumns also gained direct coverage.
The real find, in subtable(ms::MeasurementSet, name)'s MMS branch (Phase 14's own "optional MMS fidelity" feature — reading a keyword subtable, such as POINTING or SYSCAL, that a ConcatTable's subtabnames marks as per-part rather than shared, by concatenating every part's own copy instead of just parts[1]'s): the loop for p in data.parts, (kw, pth) in subtables(p); kw == name && (push!(subs, readtable(pth)); break); end used break inside a combined multi-generator for statement — live-verified this exits the entire loop in Julia, not just the inner generator the way break does inside genuinely nested for blocks. So the very first part whose subtable list happened to contain a match for name ended the whole search, and every later part's contribution to the concatenated subtable was silently dropped — an MMS opened via MeasurementSet and accessed through ms.POINTING (or any other subtabnames entry) only ever saw one constituent SubMS's rows, not the true union. Live-reproduced with a 3-part synthetic MMS (write_concattable(...; subtabnames=["POINTING"])) before fixing: nrow(subtable(ms,"POINTING")) gave 1 (only the first part) instead of the correct 3. Fixed by switching to genuinely nested for loops, where break only exits the innermost one; re-verified the same fixture now gives 3, and that the ordinary (non-subtabnames, parts[1]-only) ANTENNA path and the subtable cache are both unaffected. New permanent regression test in test/reftable_tests.jl.
Phase 289 — src/tables/interface.jl sweep: a confusing raw error on an unknown row column
Swept the Tables.jl integration layer (src/tables/interface.jl) — fully live-tested (no external oracle needed: correctness is against Julia's own Tables.jl contract) across every AbstractTable kind (Table, RefTable, ConcatTable, GroupedTable, MeasurementSet): schema resolution, column and row access by both name and index, whole-table iteration (incl. nested loops over the same table object — each for r in t builds its own fresh Tables.rows(t) state, confirmed independent), an empty (zero-row) table, and Tables.columntable round-tripping. All correct.
Fixed: Tables.getcolumn(row::CTDSRow, nm::Symbol) for an unknown column name gave a raw, confusing ArgumentError("invalid index: nothing of type Nothing") — findfirst returning nothing fed straight into p.cols[nothing] with no check — instead of a clear message. column(t, name) / columndesc(t, name) already raise a plain KeyError for exactly this mistake; Tables.getcolumn on a row took a separate code path that never got the same guard. Fixed to raise the matching KeyError. New tests in test/tables_tests.jl, including row (not just column) access coverage for RefTable / GroupedTable / ConcatTable, which had none before this phase.
Phase 290 — src/tables/record.jl + src/tables/writer.jl sweep: no new bug, one real finding documented
Fresh read + cross-check of the Record/TableRecord/keyword-set decoder (record.jl) and the table.dat metadata writer (writer.jl) against real casacore source (tables/Tables/TableRecordRep.cc, casa/Containers/RecordRep.cc). Every write/read pair checked (nested TpRecord fields, SubTable values, _write_aipsarray's Bool bit-packing, the storage=:multifile/:multihdf5 ColumnSet write branch, the dead "ScalarRecord..." classname branch mirroring a real casacore column kind this package has no construction path for) is either already solidly covered (keywords_tests.jl's Phase 267 byte-exact 0x8D 0x05 Bool-array check + real-casacore cross-check; a direct live exercise of storage=:multifile confirmed its write path is genuinely covered elsewhere, just not in this phase's initial coverage subset) or confirmed genuinely unreachable through our own writer.
One real, previously-unexplained finding, documented rather than "fixed": read_keyset's old-style-format decoder (ScalarKeywordSet/ArrayKeywordSet, flagged since Phase 210 as untestable — no real fixture on this machine ever uses the pre-TableRecord format) leaves rec.rectype at its Record() default (RECORD_VARIABLE) for the Scalar/Array cases. Traced this against real casacore's own TableRecordRep::getRecord + TableRecord::getRecord: only the TableKeywordSet branch ever assigns the by-reference recordType there — for ScalarKeywordSet/ArrayKeywordSet, real casacore's own Int type; local is genuinely read uninitialized (no assignment on that code path at all). There is no well-defined upstream value to match; our default is a deliberate, now-documented choice facing real casacore's own undefined behavior, not a divergence to chase.
Phase 291 — resync of an in-memory (unpersisted) RefTable/ConcatTable gave a confusing error
Swept src/tables/resync.jl (the is_stale/resync coherent-re-read machinery — foundational to concurrent-access correctness, not the direct subject of the Phase 207-209 locking sweep, which focused on io/lock.jl). Also re-checked src/datamanagers/datamanager.jl (the DM name→type registries — confirmed no ambiguity between the pattern dict's five prefixes and the exact-name dict), src/datamanagers/bytes.jl (the shared low-level byte-reading primitives — confirmed the bit-unpack n == 0 early-return already correctly guards the one out-of-bounds risk in that function, and the fixed-vs-undefined-shape fallback in _empty_cell's callers is intentional), and src/constants.jl (the LSRK/LSRD/LGROUP/CMB velocity vectors independently cross-checked byte-for-byte against real casacore measures/Measures/MeasTable.cc source — exact matches). No further bug in any of those four.
Fixed: resync(t::Union{RefTable,ConcatTable}) for a stale but never-persisted RefTable/ ConcatTable (e.g. query()'s own result, path == "" by Phase 22's design; a hand-built ConcatTable can be path == "" too) silently called readtable(""), which threw the confusing ArgumentError: not a table directory: — live-reproduced. Worse, this happened after the parent's data-manager cache had already been evicted, a wasted side effect on a call that was always going to fail. Fixed to check for an empty path first and raise a clear, actionable error naming the actual problem; is_stale itself is unaffected and still correctly detects the underlying parent's change. New test in test/reftable_tests.jl, including a regression check that a genuinely persisted RefTable still resyncs correctly.
Phase 292 — src/tables/typeenum.jl + src/tables/units.jl sweep: no new bug, one real coverage gap closed
Fresh read + cross-check of the CasaType enum and its Julia-type mapping against real casacore source (casa/Utilities/DataType.h) — every one of the 31 enum ordinals (0-30) verified to match casacore's own declaration order exactly, and the TpChar => Int8 read-side mapping confirmed consistent with the already-documented (Phase 265) write-side exclusion ("Int8/Char is not a real casacore table column type — casacore aborts"). The Phase 34-36 half-precision narrowing machinery (_narrowtype/_narrows/_narrowvalue) re-checked and confirmed correct. No bug.
A coverage-instrumented pass then found a genuine, previously-completely-untested code path in src/tables/units.jl: columnunit/qcolumn's own "load Unitful" fallback (the varargs stubs, overridden by ext/UnitfulExt.jl once loaded) had never been exercised by any test — units_tests.jl itself imports Unitful/UnitfulAngles/UnitfulAstro at its own top before a single @test runs, and once the extension loads for a Julia process it stays loaded, so the fallback genuinely cannot be reached in-process. Live-verified the fallback gives the correct, actionable error message via a fresh child process that never imports Unitful — the exact same shape of gap, and the exact same lock_tests.jl _JULIA/_PROJ cross-process fix, Phase 225 already used for the analogous EarthOrientationExt "SOFA loaded, EarthOrientation not loaded" fallback. New test in test/units_tests.jl.
Phase 293 — src/tables/refedit.jl + src/tables/concatedit.jl sweep: two real bugs, both live-reproduced
First dedicated fresh-read sweep of the RefTable/ConcatTable in-place-edit views (Phases 125-130) — both files re-read in full and cross-checked against their own documented casacore provenance (RefColumn::put, ConcatColumn::put, RefTable::addColumn/removeColumn, ConcatTable::addColumn). Found two real bugs, both confirmed live before fixing.
Bug 1 — a missing flush method. Neither RefEditTable nor ConcatEditTable had a Base.flush method at all — only the do-block form (edit(f, rt)/edit(f, ct)) ever committed anything, by duplicating the commit logic inline instead of calling a public flush. So the documented non-do-block idiom t = edit(path); ...; flush(t) — which works for a plain EditTable and is even implied by these two types' own docstrings — raised a raw MethodError for t = edit(rt::RefTable) / t = edit(ct::ConcatTable). Live-reproduced: the write itself had already landed correctly (t["COL"][i] = v writes straight through to the parent/part), but there was no public way to commit it short of reaching into the private t.parent/t.parts fields — and since Phase 207 holds the write lock for the whole session (not just inside flush), hitting this MethodError left the table's lock stuck for the rest of the process with no way to release it. Fixed with Base.flush(t::RefEditTable) (delegates to flush(t.parent), returns t) and Base.flush(t::ConcatEditTable) (flushes every part, returns t) — both idempotent, matching Base.flush(::EditTable)'s own convention; the do-block forms now call these instead of duplicating the logic inline.
Bug 2 — ConcatEditTable.addcolumn! mutated parts as it looped, not after validating all of them. Both the no-data and with-data forms looped for p in t.parts; addcolumn!(p, name; ...) — so if name already existed on a later part but not an earlier one (a real possible MMS shape: one SPW's MAIN table already carries a column the others don't), the call correctly threw "column … already exists" from the later part's own _check_new_col, but the earlier part was left with a silently-committed pending add in its own EditTable.addcols. The error message implied nothing had happened; in fact a later, unrelated flush (even one reached via completely different code) would write a spurious, wrongly-sliced column to that one part only. Live- reproduced with a two-part ConcatTable where only part 2 already had column "B": the failed addcolumn!(ce, "B", ...) left ce.parts[1].addcols non-empty, and a subsequent flush gave part 1 a "B" column with the wrong data while part 2's genuine "B" was untouched. Fixed by validating every part's _check_new_col (and _check_kind) up front, before mutating any of them — the same validate-before-mutate shape already established in Phases 199/202/204/205 for exactly this class of "throws late, having already mutated something" bug.
New tests in test/edit_tests.jl for both, including the real-casacore cross-check confirming the untouched columns really are untouched on disk.
Phase 294 — src/tables/edit.jl sweep: a real bug in addcolumn!'s type= override
Full fresh read of EditTable/EditColumn (only ever patched piecemeal before — Phase 201's kind= validation, Phase 207's locking, Phase 210's removecolumn! gap — never read start to finish in one pass). The engine/Dysco companion-column cleanup in removecolumn!, the row-map shift arithmetic in removerows!, the fast-vs-regen dispatch in Base.flush/_tiled_fast_ok, and the per-sequence-group writer dispatch in _flush_regen were all re-checked against their own documented invariants and confirmed correct — including a hand-verified cross-check that removecolumn!'s hardcoded 9-entry engine-companion-keyword list is exactly the full set src/datamanagers/virtual.jl can ever write (_BaseMappedArrayEngine_Name + {ScaledArrayEngine,ScaledComplexData,CompressFloat,CompressComplex}_{Scale,Offset}Name, the last shared by CompressComplexSD), not a stale subset.
Found and fixed: _addcol_desc — the shared helper behind addcolumn!(t, name, data; ...) for both a plain EditTable and a RefEditTable (Phase 126) — only ran its Measure/Unitful.Quantity auto-flatten if type === nothing. So addcolumn!(t, name, measure_or_quantity_data; type=SomeCasaType) silently succeeded at add time with vals left as the raw, unflattened Vector{MEpoch{...}}/ Vector{Quantity} — no validation, no error — and the failure only surfaced deep inside a later flush → _flush_regen/_flush_fast → write_standardstman/write_incrementalstman, as a bare MethodError: no method matching Float64(::MEpoch{UTC}) that names none of the real cause. Live- reproduced before fixing. There was no legitimate use for the old guard either: addcolumn! has no units=/measures= kwarg of its own, so this auto-detection is the only way to get a MEASINFO/QuantumUnits keyword onto an added column at all — nobody could have been relying on type= suppressing it on purpose. Fixed by always running the flatten; type=/shape= now override the result of it (exactly as documented) instead of skipping it. A plain (non-Measure/Quantity) data + type= is unaffected (_measure_column_spec/ _quantity_column_spec both return nothing for it, same as before) — verified live for both the fixed case and this regression case, through both addcolumn!(::EditTable, ...) and addcolumn!(::RefEditTable, ...) (which shares the same fix for free).
New tests in test/measures_tests.jl. Also investigated a related permissive-but-harmless behaviour (a double removecolumn! on a plain EditTable succeeds silently, where RefEditTable's own removecolumn! throws) and a remove-then-addcolumn! of the same name in one session (works correctly, a deliberate "replace a column" idiom) — both confirmed not to cause any real corruption or misleading state, left as-is.
Phase 295 — src/tables/create.jl sweep: write_ms/create_ms left a stray directory behind on a mid-write failure
Full fresh read of the write path (write_table/_write_table_core, copytable/_copy_table/ _copy_table_cols, reference_copy, write_ms/copyms, create_ms, the _casatype_of/ _normalize_desc/_infer_shape/_stamp_measinfo helpers) — only ever touched piecemeal before by validation-focused phases (199, 202, 204, 205, the array-literal hazard in 210), never read start to finish in one pass.
Found and fixed: _write_table_core's own Phase-226 cleanup only ever removes a directory it created (_dir_preexisted || rm(dir; ...)) — correct for its own callers (write_table, copytable, reference_copy, none of which mkpath anything themselves before delegating to it). But write_ms and create_ms are different: both mkpath(dir) themselves, then make several subsequent _write_table_core/_copy_table calls into dir and its subdirectories (one per MS subtable, then MAIN, for write_ms; twelve subtables then MAIN, for create_ms). By the time either of those later calls reaches _write_table_core, dir already exists (they made it), so its own _dir_preexisted check correctly declines to remove it on failure — and neither write_ms nor create_ms had any top-level cleanup of their own. A genuine failure partway through (MAIN, for write_ms; any table after the first, for create_ms) left a stray, partially-written directory sitting at the caller's requested path, with no cleanup and no indication anything was left behind — the exact "claims to have failed but silently created state anyway" shape Phase 226 fixed for _write_table_core itself, never propagated up to its two top-level multi-table orchestrators.
Live-reproduced for both: create_ms(dir; nrow=-1) (a public, deterministic failure partway through synthesising the standard subtables) left dir behind with several subtables already written; write_ms(dst, ms) after corrupting a source public keyword's value (leaving its declared type alone, so MAIN's own write — not a per-column read, which _copy_table_cols already tolerates with a warn-and-skip — hard-fails deep inside write_table_files → write_record) left dst behind with every subtable already written. Fixed by wrapping each function's whole body (after its own ispath(dir) && error(...) guard, which already establishes dir did not exist when the call started, so there's no "was it already there" case to preserve, unlike _write_table_core's other callers) in a try/catch that unconditionally removes dir on any exception. reference_copy, write_table, and copytable were confirmed to need no change — none of them mkpath anything before delegating to _write_table_core, so its existing cleanup already covers them correctly.
New tests in test/writer_tests.jl, both live-reproduced failure modes plus a regression check that a valid call is unaffected.
Phase 296 — ext/UnitfulExt.jl sweep: a latent (currently unreachable) inconsistency fixed for consistency
First dedicated fresh read of the Unitful weak-dependency extension — only ever touched piecemeal before by Phases 65 (creation), 70 (write path), 160 (a docs @ref fix), and 197 (whose real fixes landed in the core src/tables/units.jl file, not here). Several candidate leads investigated and confirmed correct: PseudoUnits' klambda @unit scale factor (1000) independently verified to tie back to lambda's own dimensionless base exactly (1 klambda == 1000 lambda, not assumed from the macro call alone); _ms_ustring's m/s^2 compound-unit round-trip verified end to end through a real write_table → readtable → columnunit cycle (exercises the Phase-197 digit-implicit-exponent rule inside a compound, not just an atomic, unit string); qcolumn's array-cell broadcasting (vals .* u over a Vector{Array}) confirmed to rely on an already-tested, working Unitful idiom, not a latent bug; the UNITS_NO_JULIA_COUNTERPART note-lookup in _ms_uparse's error path confirmed to key correctly off the original (pre-normalisation) unit string for the :unsupported-kind entries that actually reach it.
Found and fixed (a real inconsistency, confirmed unreachable via its only current caller, fixed anyway for consistency): _tql_write_strip(x::Quantity, u) — the update! SET-RHS unit-stripping function — takes a u that can be nothing / a Unitful.Units / a Tuple of them (a genuinely mixed-unit column, per columnunit's own documented return shape), but only guarded the nothing case; its two siblings in the same file, _tql_unit_attach and qcolumn, both already guard the Tuple case with a clear "column has a mixed unit" error. Traced (not reproduced as a live, user-visible bug) exactly why: _tql_write_strip's only caller, update! (src/taql/commands.jl), always adds every SET target column to _tql_cols's needed set unconditionally, and _tql_cols unit-attaches every column in needed via _tql_unit_attach the moment any spec anywhere uses a quantity literal — so a mixed-unit SET target is already caught by _tql_unit_attach's own guard while columns are being loaded, before a single row's _tql_write_strip call ever runs. Live-verified this interception genuinely happens (a real mixed-unit column + update! with a quantity literal throws _tql_unit_attach's message, never reaches _tql_write_strip's missing branch). Added the matching guard anyway — cheap, matches its two siblings exactly, and removes a latent trap for a future caller or refactor that reaches this function some other way.
New tests in test/units_tests.jl, including a direct pin of the new guard (bypassing update!'s own interception, since that's what currently makes it otherwise untestable end to end) plus the two independently-verified correct behaviours (klambda scaling, m/s^2 round-trip).
Phase 297 — ext/HDF5Ext.jl sweep: container_read could silently return uninitialised memory
First dedicated fresh read of the MultiHDF5 (table.mfh5) weak-dependency extension — only ever implemented piecemeal before across Phases 20 (read), 21 (write), and 37 (the weak-dep split), never investigated on its own. Several details re-checked and confirmed correct or already-documented, non-new limitations: _open_multihdf5's filter for an empty virtual-file name (a defensive placeholder-slot convention borrowed from MultiFile's own format, structurally inert for MultiHDF5 since a named HDF5 group can't correspond to an empty name anyway); the block-axis-order assumption in container_read/_finalize_multihdf5 (d[b, :], an already-documented, still-unverifiable-on-this- machine caveat — no HDF5-enabled real casacore build exists here, per Phases 20/21's own notes); the writer's "build every dataset directly at its final size in one write" approach (verified against casacore's own doAddFile/extend/put to be bit-for-bit equivalent from any reader's perspective).
Found and fixed a real bug: container_read allocated out = Vector{UInt8}(undef, fsize) up front (fsize from the container's own header-attribute claim) and only explicitly assigned however many bytes its block-reading loop actually found in the dataset. If the dataset's real block count didn't cover the claimed fsize — a corrupted or truncated table.mfh5, or a genuine mismatch between this reader's block-axis-order assumption and however a real casacore build might someday write one (exactly the scenario the file's own "flip to d[:, b]" comment already anticipated) — the loop silently exhausted its blocks with done < fsize, and the function returned out with its tail bytes still raw, uninitialised heap memory, fed straight into whichever storage-manager reader opened that virtual file next. Live-reproduced with a hand-built file whose header claims 1000 bytes for a virtual file whose dataset genuinely has only 10: container_read returned a 1000-byte vector whose bytes 11-1000 were real garbage, no error, no warning. Fixed by checking done >= fsize after the loop and raising a clear, actionable error naming the exact byte-count mismatch instead of ever returning a partially-uninitialised buffer.
New tests in test/container_tests.jl: the live-reproduced malformed case now errors cleanly, and a regression check that an exactly-matching, well-formed file is completely unaffected.
Phase 298 — ext/SOFAExt.jl sweep: one confirmed-dead branch removed, two coverage gaps closed
First dedicated fresh read of the whole reference-frame conversion extension in one pass — it had never had a single "sweep this file" phase of its own, only ~40 individual bug-fix phases spread across epoch/direction/frequency/radial-velocity/uvw/baseline/Earth-magnetic-field conversions since Phase 66. A full read plus a coverage-instrumented run (against measures_tests.jl + taql_mscal_tests.jl in a scratch environment) found no new live bug, but did turn up one genuinely dead branch and two real-but-previously-untested reachable paths.
Dead branch removed: _dir_to_icrs's own _is_body(A) && return _body_dir_icrs(A, frame, false) case. Confirmed via grep that every call site of _dir_to_icrs — its own recursive calls for SUPERGAL/AZELSW/AZELSWGEO, and the one external call inside _mconv(::MDirection,...) — only ever passes a non-body direction: _mconv already branches on _is_body(A) before ever calling _dir_to_icrs, routing every body-frame direction (SUN/MOON/planets) through _body_dir_icrs directly with the right topo flag for the target frame. This branch dated from before that _mconv restructuring (Phase 76) and had been unreachable ever since; removed with a comment recording why.
Two coverage gaps closed (both live-verified correct before pinning with permanent tests, not assumed): (1) the 8 "$(nameof(X)) is not supported" fallback error(...)s at the end of each conversion family (epoch/direction/frequency/radial-velocity, source-side and target-side) — reachable whenever a MEASINFO names a frame this package parses but doesn't convert (OtherRef{S}); confirmed each one raises a clean, actionable error naming the real cause rather than crashing some other way (a MethodError, an infinite recursion, …). (2) _pole_dir's R === ITRF branch (the phase-centre resolution inside a MuvW conversion, for the case where frame.direction is itself stored in ITRF — rare in practice since a real MS phase centre is always J2000-ish, but a real, reachable state); checked against an independent from-scratch computation (0.0″ separation) and a forward/backward round trip through a third frame before being pinned as a permanent black-box test via the public measconvert/MuvW API.
New tests in test/measures_tests.jl.
Phase 299 — IGRF earthfield random fuzz vs casatools (investigation only, no bug found)
The IGRF-14 spherical-harmonic field synthesis (_earthfield_itrf, a verbatim port of casacore EarthField::calcField) and EarthMagneticMachine's line-of-sight geometry had only ever been cross-checked against real casatools at one fixed site and epoch (Phase 91's ALMA point, reused by Phase 66's shared fixture). Following the Phases 269–287 pattern — where a single deterministic cross-check repeatedly missed bugs a broader random fuzz caught — this phase spreads the same oracle across 14 random global sites (uniform on the sphere, 0–3000 m height) and epochs (2000–2030, the full span the bundled IGRF-14 model supports) in one casatools process (CASA startup dominates the cost, so looping inside one script call keeps a broad sweep affordable).
No new bug found: the field magnitude/components at each random site match casatools' IGRF-12 output within the already-established model-generation tolerance (~5% + 250 nT — IGRF-12 vs IGRF-14 is a real, documented, expected difference, not a bug), and the ITRF→J2000 rotation is confirmed correct (magnitude-preserving, and each rotated component matches casacore's own rotated value) at every one of the 14 sites/epochs, not just the one previously tested. A legitimate, valuable investigation-only result — the port and the rotation both hold up globally, not just at the one location anyone had ever actually pointed a real oracle at.
New testset in test/measures_tests.jl (112 cross-check assertions, fixed-seed MersenneTwister(299) for reproducibility).