← older entries

live-verifying them: V[boolexpr]'s mask polarity was backwards

Continuing Phase 190's own "Remaining names" list at the user's direction: negatemask(arr) / replacemasked(arr, val) / replaceunmasked(arr, val) — casacore's TEFMASKneg/TEFMASKrepl (ExprFuncNodeArray.cc:210-272). negatemask flips a masked array's mask (an unmasked input becomes FULLY masked, not an error — casacore's own !arr.hasMask() branch). replacemasked/ replaceunmasked replace the elements where the mask is True/False respectively with a scalar or same-shape array, preserving the original mask; on an unmasked input, replaceunmasked replaces EVERYTHING (an unmasked array is "all unmasked") while replacemasked is a no-op. nullarray() — a genuinely absent-array sentinel (MArray<Bool>()) with no clean mapping onto this package's always-a-concrete-array TQLMArray design — stays deliberately deferred, per Phase 190's own note.

While live-verifying these three against real casacore, found the actual bug was upstream of them, in V[boolexpr] itself (_tql_do_index, src/taql/ast.jl, dating to Phase 60): this package computed the masked array's mask as !boolexpr, but real casacore's mask is boolexpr DIRECTLY — no negation. Live-verified beyond doubt: arraymask(A[A>2]) for A=1:5 is [F,F,T,T,T] in real casacore (masked exactly where the condition holds), so mean(A[A>2]) == 1.5 (the mean of [1,2], the elements where the condition is FALSE) — not 4.0 (the mean of [3,4,5]) as this package's inverted mask produced. This is not a cosmetic detail: it silently inverted the result of every V[cond] masked-selection expression since Phase 60 — mean/ sum/min/max/variance/stddev/median/any/all/ntrue/ nfalse/arraymask/arraydata over a masked selection, the masked g*/gs* group aggregates (Phase 62), and the faithful SET (D, M) = V[cond] update (Phase 59) all inherited the inversion. marray(d, m) (mask given explicitly, never through V[cond]) was unaffected and already correct. Fixed by dropping the negation: TQLMArray(collect(arr), BitArray(collect(m))) instead of BitArray(.!m).

A second, smaller bug found in the same live-verification pass: nelements()/count() on a masked array is mask-AGNOSTIC in real casacore — nelements(A[A>3]) on an 8-element A is 8, the total array size, not the unmasked-element count this package's _tql_nelem computed (count(!, x.mask)). Fixed to length(x.data).

Both fixes verified together against real casacore across mean/ sum/min/max/variance/stddev/median/any/all/ntrue/ nfalse/nelements/count/arraymask on a masked selection — every one now matches exactly. Updated the stale hand-computed expectations in the pre-existing "TaQL-lite — masked arrays (TQLMArray) unit", "TaQL-lite query — masked-array expressions", "groupby — masked g* aggregates" (test/taql_query_tests.jl), and the SELECT ... AS (val, mask) / (col, maskcol) update-pair tests (test/taql_command_tests.jl) that had baked in the old, backwards convention. New testset "Phase 190 continuation — masked-array natives: negatemask/replacemasked/ replaceunmasked" (29 assertions, unit + real-TaQL cross-check). Standalone taql_query_tests.jl and taql_command_tests.jl both green in full, end to end, with no other regressions.

Phase 192 — a genuine GitHub Actions CI failure, sitting undetected

on main since PR #56: int()/integer() on a NaN/±Inf argument

is architecture-dependent undefined behavior in real casacore

A user-reported CI failure (Phase 184 — int()/integer(), real-TaQL cross-check) turned out to be real and already present on main — a genuine gap in this project's own workflow: the "merged" step never checks GitHub Actions' own CI status, only a local Pkg.test() run, so a real CI failure on the merge commit itself (and on the phase184-sweep branch that introduced it) had been sitting undetected since PR #56. Confirmed via GitHub's public REST API (reachable unauthenticated for a public repo, no gh CLI needed): the main branch's own CI run at a2aeba9 fails this exact test on all three Julia versions (1.10 / 1.12 / pre), Linux x64.

Root cause, confirmed with a direct compiled-C++ probe on real x86-64 hardware (Docker, --platform=linux/amd64) before touching any code: real casacore's int()/integer() (intFUNC) is a bare C++ static_cast<Int64>(double) — for a NaN or out-of-range argument this is genuinely UNDEFINED BEHAVIOR, and the two architectures this package has actually been tested on implement it differently:

• ARM64 (FCVTZS, what every earlier "live-verified against real casacore" claim in this file was tested on, on an Apple Silicon Mac): saturates piecewise — int(0.0/0.0) == 0, int(1.0/0.0) == typemax(Int64), int(-1.0/0.0) == typemin(Int64).
• x86-64 (CVTTSD2SI, what GitHub Actions CI — and the overwhelming majority of real casacore/CASA deployments — actually run on): gives the SAME "integer indefinite" sentinel, typemin(Int64), for EVERY one of NaN / +Inf / -Inf / any out-of-range value, uniformly. Verified two ways: a standalone C++ program compiled with g++ on x86-64 Linux ((int64_t)(0.0/0.0)==(int64_t)(1.0/0.0)==(int64_t)(-1.0/0.0)== INT64_MIN), and a live Casacore.jl/tableCommand run on the same architecture — int(sqrt(-1.0)), int(1.0/0.0), int(-1.0/0.0), int(0.0/0.0) all give -9223372036854775808 in real casacore on x86-64, where this package's ARM64-derived _tql_int gave 0, typemax(Int64), typemin(Int64) (matched by coincidence), and 0 respectively — three of the four genuinely diverge.

There is no single portable "real casacore" ground truth for a NaN/±Inf argument to int(), so chasing bit-for-bit equality with whatever one CPU's raw undefined behavior happens to produce is the wrong target. _tql_int keeps its existing well-defined, documented, architecture-independent saturating convention unchanged (NaN→0, +Inf→typemax(Int64), -Inf→typemin(Int64)) — a real, useful, designed behavior, not an attempt to replicate either CPU's garbage. The fix is to the test's methodology, not the implementation: the real-TaQL cross-check now only asserts exact equality against live casacore for the well-defined, in-range values (int(1e18), integer(±2.9), which have no UB on any platform), and checks the NaN/±Inf cases against this package's own documented sentinel values directly instead of against an architecture-dependent live oracle. _tql_int's comment records both architectures' actual behavior (with the compiled-C++ evidence) so a future sweep doesn't re-discover this by re-breaking it.

Re-verified end to end on real x86-64 Linux (Docker) after the fix: the full Casacore.jl cross-check suite (141 assertions) plus the targeted int()/integer() testset (7 assertions) both pass — the fix genuinely resolves the CI failure, not just a local ARM64 rationalization. Standalone taql_query_tests.jl unaffected on ARM64 (still 0 failures, full file).

Phase 193 — the Phase 192 finding recurs in a different corner:

every date/time formatting function threw on a NaN/±Inf argument;

real casacore's own NaN handling turns out to be self-inconsistent

Investigated the array-reshaping function family (array/transpose/ reversearray/diagonal/resize/flatten) from Phase 190's own "remaining names" list, and found it genuinely out of scope for a single phase: casacore's own argument-parsing machinery for these (getOrder/getReverseAxes/getDiagonalArg/getAlternate) threads a C-order-vs-Fortran-order STYLE toggle and a 0/1-based origin_p through every one of them, plumbing this package's TaQL-lite engine has no concept of at all — a real, substantial future feature, not a quick add. Flagged for a dedicated phase; not implemented here.

Redirected to the standing methodology note this session's own Phase 192 finding just added: "a live cross-check verified on only one architecture can itself be wrong — check whether a formula's raw numeric conversion is architecture-dependent UB before chasing bit-for- bit equality." Searched src/taql/functions.jl for the SAME risk shape (a round(Int, ...)/Int64(...) call fed directly by a column/expression value, not just general float math) and found the entire date/time formatting family shares it: _tql_dt_of (the shared choke point for year/month/day/week/weekday/dow/cdate/ cmonth/cdow/cweekday/ctod/cdatetime) plus _tql_hms/ _tql_dms/_tql_time_of_day_str (used by hms/dms/hdms/ctime) all did a raw round(Int, ...) with no NaN/Inf guard — live-verified on this ARM64 Mac: hms(0.0/0.0) threw InexactError: Int64(NaN), and likewise for every one of the twelve functions above.

Live-verifying real casacore's own NaN behavior for a fix, following the exact Phase 192 discipline, immediately surfaced why chasing it bit-for-bit is the wrong target here too — but for a NEW reason this time: real casacore's own answer is self-INCONSISTENT, not just architecture-dependent. cdate(0.0/0.0) == "17-Nov-1858" — exactly the MJD epoch (MJD 0) — while year(0.0/0.0) == -4712 and month(0.0/0.0) == 1 don't correspond to that date (or to each other) at all, and hms/dms/ctime embed a literal "nan" substring inside an otherwise fixed-width numeric field (hms(0.0/0.0) == "00h00m000nan", dms(1.0/0.0) == "+***d00m000nan" — the "***" is a genuine, DEFINED "field overflow" sentinel in MVTime::print, but the trailing "000nan" is not). Different funcName branches clearly hit different, mutually-contradictory undefined-behavior manifestations within the SAME casacore build — not a single "real" oracle value to replicate at all, even setting the Phase 192 cross-architecture question aside entirely.

Fixed by giving this package its own well-defined, portable, self-CONSISTENT convention instead of chasing any of that: a non-finite argument to any of the twelve _tql_dt_of-based functions, or to hms/dms/ctime/hdms, degrades to the MJD epoch itself (1858-11-17T00:00:00.000 / an all-zero "00h00m00.000" / "+000d00m00.000" / "00:00:00.000") — crash-free, predictable, and internally consistent (unlike real casacore's own answer for the same input). mjd()/date()/time() already propagated a non-finite value cleanly with no round in their path and needed no change — confirmed unchanged. New testset "Phase 193 — date/time functions don't throw on a non-finite argument" (66 assertions, all twelve _tql_dt_of-based functions plus hms/dms/ctime/hdms plus the three already-fine pass-through functions, across NaN/+Inf/-Inf/sqrt(-1)). Standalone taql_query_tests.jl green in full, no regressions.

Phase 194 — the same NaN-crash finding, a third corner:

mscal.time()'s default-row TIME

Continuing the sweep for the risk shape Phase 192 flagged: found one more site with it in src/taql/mscal.jl — _mssel_time (the mscal.time('spec') MSSelection-lite time-range predicate, Phase 94) built its "default row" calendar via round(Int, tm[r0] * 1000) with no NaN/Inf guard, where tm[r0] is the default row's own TIME column value. Live-verified reachable: a synthetic/malformed table whose default row's TIME is 0.0/0.0 makes mscal.time(...) throw a raw InexactError for the WHOLE predicate (every row, not just the bad one) — real MS TIME data essentially never hits this, but this package's own writers let a user store an arbitrary Float64 including NaN, so it's a real, reachable path, not a hypothetical. A second call site in the same file, _mstime_incl_hi's round(Int, lo_secs * 1000), is fed only by a parsed literal from the query STRING itself (never raw column data) and was confirmed unreachable with a non-finite value — left unchanged.

Fixed with the same convention as Phase 193: a non-finite default-row TIME degrades the default calendar to the MJD epoch instead of throwing (the per-row predicate comparisons themselves were already safe — a NaN row just naturally fails every >=/<=/abs(x-c)<=dT comparison instead of matching, no separate fix needed there). New testset "TaQL-lite — mscal.time() doesn't throw on a non-finite default-row TIME (Phase 194)" (3 assertions). Standalone taql_query_tests.jl + taql_mscal_tests.jl both green together, no regressions.

Phase 195 — the same root-cause SHAPE, a fourth corner:

measconvert crashed deep inside SOFA.jl on a non-finite measure/

frame value; fixed with a clear early error instead of a silent

sentinel, since a wrong NUMBER is more dangerous than a wrong string

Continuing the sweep beyond src/taql/: grepped the whole tree for the same risk shape (a raw numeric cast fed by a value that could be NaN/Inf) and found two more sites. src/datamanagers/dysco.jl's _dysco_encode_symbol/_dysco_encode_symbol_dither already guard non-finite input explicitly (Phase 19's own work) — confirmed correct, no change; its WEIGHT-column quantizer (a separate function, UInt32(floor(weight*scale+0.5))) has no such guard and WOULD crash on a NaN weight, but writing a NaN into a Dysco-compressed WEIGHT column is a narrow, synthetic-data-only path — flagged, not fixed this phase.

The real, reachable finding: ext/EarthOrientationExt.jl's _eop_lookup called its own _datetime (the same unguarded round(Int, mjd*MSEC_PER_DAY) pattern as Phases 193/194) OUTSIDE the function's existing try/catch "no IERS coverage" fallback, so a non-finite epoch's crash escaped that fallback entirely instead of hitting it. Moved the call inside the try — cheap, always-correct defense in depth.

But live-verifying measconvert end to end (measconvert(MDirection, AZEL; frame=MeasFrame(epoch=MEpoch{UTC}(NaN), position=...))) found the actual crash site is upstream of _eop_lookup entirely: SOFA.jl own utctai→jd2cal raises AssertionError: Day is out of range. from deep inside the UTC→TAI epoch-scale conversion, many stack frames below the measconvert call a user actually wrote — a genuinely confusing place to learn "your TIME value is NaN." Different fix from Phase 193/194's "degrade to a defined sentinel," and deliberately so: hms(NaN) returning an all-zero STRING is harmless (obviously a placeholder), but measconvert returning a physically-meaningless number that looks like a real answer would be actively misleading — this package's own numbers get consumed by real calculations, not just displayed. So the fix is a clear, early ArgumentError instead: a new _all_finite(x) helper (checks every Float64 field of a measure struct via fieldnames — works uniformly across MEpoch/MDirection/ MPosition/MFrequency/MRadialVelocity/MBaseline/MuvW/ MEarthMagnetic/MDoppler with no per-type method needed) is checked at measconvert's own entry point against both the measure being converted AND every non-nothing field of frame (epoch/ position/direction), before any SOFA call happens. New testset "measures — measconvert rejects a non-finite measure/frame (Phase 195)" (15 assertions: a bad value in the measure itself, a bad value in each frame field, finite inputs still convert normally, and _eop_lookup's own defense-in-depth checked directly). Standalone measures_tests.jl (153 assertions total, incl. the 79-assertion casatools oracle cross-check) and taql_query_tests.jl + taql_mscal_tests.jl together all green, no regressions.

Phase 196 — the wavelength-scaled uvw family Phase 163 flagged as a

real, unimplemented gap: mscal.uvwwvl()/uvwwvls(),

uvwj2000wvl()/uvwj2000wvls(), uvwapp()/uvwappwvl()/

uvwappwvls() — and a real bug found while writing the live

cross-check for the last of these

Continuing the phase-by-phase sweep. Read derivedmscal/DerivedMC/ {Register,UDFMSCal,MSCalEngine}.cc directly for the wavelength-scaled uvw functions Phase 163 identified but left unimplemented. UVWWVL()/ UVWWVLS() (UDFMSCal::setupWvls+toWvls) scale the STORED UVW column by the row's spw reference/channel frequency divided by c (itsWavel[spw] = refFreq/c, confirmed via itsTmpVector *= itsWavel[...] to have units of 1/metre, so uvw_metres * itsWavel is uvw_metres / wavelength_metres — "uvw in units of wavelengths"). UVWJ2000()/UVWAPP() (ColType NEWUVW, the ctor's second int arg selects asApp in getNewUVW) and their *WVL(S) siblings are the same per-baseline computation mscal.uvw_j2000() already does (Phase 137's _mvuvw_construct), with UVWAPP adding one more step: getNewUVW's asApp branch converts the freshly-built J2000 Muvw via Muvw::Convert(..., Muvw::Ref(Muvw::APP,...)) — i.e. measconvert(::MuvW{J2000}, APP; frame) (Phase 75) applied to the J2000 baseline uvw. Also found: UVWJ2000()/UVWAPP() (and their *WVL(S) siblings) DO take casacore's usual optional direction argument (setupDir — they are not in setup()'s {STOKES,SELECTION,GETVALUE,UVWWVL,UVWWVLS} exclusion list, unlike the wvl-only pair, which take none), so mscal.uvw_j2000() gained that too (it previously always used FIELD.PHASE_DIR unconditionally, since it predates this investigation) — a pre-existing test asserting mscal.uvw_j2000('SUN') throws was corrected to assert it now works.

A shared _uvw_j2000_cols3/_uvw_j2000_row pair (factored out of the Phase 137 uvw_j2000 branch, now reused by the wvl siblings) memoizes the per-baseline ITRF→J2000 linear map per (direction, TIME) — valid because that whole hop (an MBaseline rotation composed with MVuvw's own construction) genuinely is one consistent linear rotation applied to the antenna-difference vector, so converting 3 orthonormal ITRF basis vectors once and recombining linearly with the real baseline gives the exact same answer as converting the baseline directly.

A real bug, found while live-verifying uvwapp() against real casacore for the first time: an initial _uvw_app_cols3 implementation tried to reuse that SAME "convert 3 basis columns, recombine linearly" trick for the J2000→APP step — but real casacore's MCuvw:: toPole/fromPole is documented as a pure rotation, while THIS package's own measconvert(::MuvW/::MBaseline, APP; frame) (Phase 75) composes through measconvert(::MDirection, APP; frame), whose J2000→APP step applies annual/diurnal ABERRATION — a direction- dependent additive shift, not a fixed rotation matrix. Applying that conversion independently to 3 basis vectors pointing in very different sky directions, then linearly recombining, does NOT reconstruct the same result as converting the actual combined baseline vector directly — caught immediately by an internal self-consistency check (hypot(uvwapp()) ≈ hypot(uvw_j2000()), expected exact to float precision for a pure rotation) failing at ~1e-5 relative. Fixed by computing uvwapp() via one direct measconvert call per row on the actual J2000 baseline vector (_uvw_app_row, no cols3 memoization for this step) — mathematically correct regardless of whether the underlying conversion is a pure rotation or not, since MuvW's own construction explicitly preserves the input magnitude through the MBaseline step (r*ux,r*uy,r*uz with r unchanged).

Live-verified against real casacore (a sweep over 11 rows spanning ~130 m to ~7200 m baselines): wvl/wvls (a pure scalar scaling of the stored UVW column, no SOFA involved) match to float precision; uj/ujwvl(s) match to ~3e-5–7.5e-5 relative (the same SOFA-vs- casacore ephemeris scale seen elsewhere in this codebase's frequency/ direction cross-checks); app/appwvl(s) — even after the cols3 fix — run measurably higher and more variable, ~4e-5–1.8e-4 relative, a genuine architectural difference (this package's APP conversion includes aberration; real casacore's uvw-specific one does not), not a further ephemeris-precision effect, documented explicitly rather than papered over with a looser blanket tolerance.

New testset "TaQL-lite — mscal.wvl wavelength-scaled uvw family (Phase 196)" (66 assertions: structural + internal-consistency checks for all 7 new functions, the direction-argument extension, and the real-TaQL cross-check above); corrected the stale mscal.uvw_j2000('SUN')-throws assertion in the existing parser-unit testset. Standalone taql_mscal_tests.jl (plus measures_tests.jl, which it depends on) green in full (1133/1133), no regressions.

Phase 197 — two real bugs in src/tables/units.jl's casacore↔Unitful

mapping, both dead code since Phase 65/70: "M0"/"S0" (solar mass)

were unreachable on read, and writing an Msun-unit column

unconditionally errored — plus a real, broader gap: casacore's own

implicit unit-exponent grammar ("m2" == m²) had no handling at all

At the user's direction, swept src/tables/ (not yet covered by the recent phase-by-phase bug hunt, which had been focused on src/taql/ and src/measures/). Read casa/Quanta/UnitVal.cc/UnitMap4.cc directly, checking _normalize_unit's token-substitution regex against casacore's own actual unit-string grammar.

Read-side bug 1: _UNIT_ALIASES has an "M0" => "Msun" entry (M0 being casacore's spelling of the solar mass unit — confirmed real, casa/Quanta/UnitMap4.cc:107-111), but _normalize_unit's token-substitution regex was r"[A-Za-z°µ%]+" — letters only, no digits — so it could only ever match the bare letter "M" out of the string "M0", never the whole two-character unit name. The alias was provably dead code since it was added: _normalize_unit("M0") returned "M0" unchanged, and Unitful.uparse("M0") (Unitful has no such symbol) would then fail with an unhelpful error — any real MS column tagged QuantumUnits=["M0"] (plausible for a simulation or derived catalog with a mass column) would have hit this. No existing test exercised "M0"/Msun at all, on either the read or write side.

A real, much broader read-side gap, found while investigating why the regex excludes digits at all: casacore's UnitVal::field/::power (casa/Quanta/UnitVal.cc:181-247, read in full) show a genuine, general grammar feature this package had zero support for — a unit name immediately followed by a bare digit run, with no **/^ separator, is an implicit exponent: "m2" parses as m², "cm3" as cm³, "hm2" as hm² — this is how casacore represents any squared/ cubed unit compactly, not a fixed list of named units. (The one deliberate carve-out: the literal character '0' is explicitly whitelisted as a name character in UnitVal::field's own un2 regex, which is exactly why "M0"/"S0" are atomic unit names and not M^0/S^0 — confirming _UNIT_ALIASES's "M0" entry's own intent was correct, just unreachable.) Fixed _normalize_unit to: (1) substitute "M0"/"S0" (added — S0 is casacore's more fundamental "solar mass" definition, M0 := 1*S0, previously entirely unmapped) as whole tokens before the generic rule, since digits are now in play; (2) insert Unitful's ^ before any remaining bare digit run directly following a letter/°/µ/% character. Confirmed the existing "deg_2"/"arcmin_2"/"arcsec_2" handling (a different casacore convention — real, separately-registered unit names, UnitMap5.cc, already correctly flagged :unsupported in UNITS_NO_JULIA_COUNTERPART) is untouched by the new rule, since the digit there follows an underscore, not a letter.

Write-side bug 2, found live-verifying bug 1's fix end to end (the practically relevant direction — a Vector{Quantity} column with Msun units): _ms_ustring(UnitfulAstro.Msun) unconditionally raised "no known casacore spelling". Root cause: string(UnitfulAstro.Msun) prints the symbol "M⊙", not the identifier "Msun" — the general "atomic unit" fallback path looks up _UNIT_ALIASES_INV by the printed string, which only had the key "Msun", never "M⊙". The codebase already has an established pattern for exactly this printed-symbol-vs- identifier-name mismatch ("″" => "arcsec", "′" => "arcmin" — how UnitfulAngles prints arcsecond/arcminute) — followed it, adding "M⊙" => "M0" to _UNIT_ALIASES_INV rather than widening _MS_USTRING_KNOWN's documented (compound-unit-only) scope. No prior test exercised writing a solar-mass-unit column either.

Live-verified end to end: _normalize_unit/_ms_uparse for "M0", "S0", "m2", "cm3", "hm2" (dimension checks + up("M0") == up("S0")); _ms_ustring(Msun) now returns "M0" and round-trips; a real write_table/readtable round-trip of an Msun-typed column (QuantumUnits == ["M0"], columnunit reads back Msun, values unchanged). m²/cm³ write-side round-tripping (_ms_ustring on a computed squared unit) is left as a documented, clearly-erroring gap — speculative (no real MS column plausibly needs it) and would need a separate, more general superscript-printing fix, out of scope here. Extended test/units_tests.jl's existing testsets (12 new assertions across _normalize_unit, the casacore-vocabulary parse check, and _ms_ustring, plus a full write_table/readtable round-trip). Standalone units_tests.jl green in full (90/90), no regressions.

Phase 198 — a real bug in src/tables/column.jl: column()/

getcolumn()/getcell()'s precision= override was never

validated, so a typo silently narrowed nothing instead of erroring

the way readtable's identical check already did

Continuing the sweep of src/tables/. readtable(path; precision=…) validates its argument against (:half, :full, Float16, BFloat16, Float32) and raises a clear ArgumentError for anything else — but the identical precision= kwarg on column/getcolumn/getcell (the common per-column override, column(t, name; precision=…)) had no such check at all. _narrowtarget (the function that turns an "effective precision" into a narrow scalar target or nothing) is three if-style branches that each test for one specific value and fall through to return nothing (= "no narrowing") for anything they don't recognize — so a mistyped precision=:hal (for :half) or precision=:HALF (wrong case) silently read back full ComplexF32 instead of the requested ComplexF16, with no error at all. Live- verified: column(t, "DATA"; precision=:hal) gave Matrix{ComplexF32} with no warning, while readtable(t; precision=:hal) on the exact same typo correctly threw.

Fixed by extracting readtable's validation into a shared _normalize_precision(p) (in table.jl) and having _narrowtarget call it first — every precision= entry point (column, getcolumn, getcell, and — since they all funnel through column(::Table, …) — a RefTable's MappedColumn and a ConcatTable's ConcatColumn too) now validates identically, with no separate fix needed per table kind. Live-verified end to end: all 6 valid values (nothing, :half, :full, Float16, BFloat16, Float32) still behave exactly as before; :hal, :HALF, and Int32 now all raise the same clear ArgumentError on column/getcolumn/getcell directly and through a RefTable. New testset "precision — column()/getcolumn()/ getcell() validate precision= too (Phase 198)" (11 assertions). precision_tests.jl standalone green (74/74), plus a broader core-data-dependent-file sweep (metadata/ssm/tsm/ism/api/ tables/schema/precision_tests.jl together, 184/184) confirming no regressions.

Phase 199 — a real bug found applying the Phase 198 methodology note

broadly: storage= (the :multifile/:multihdf5 container option)

was validated in only ONE place, deep inside the write pipeline —

write_ms/copyms turned a caller's own typo into 18 misleading

"unsupported source" warnings before the true cause ever surfaced

Continuing the sweep, applying Phase 198's own freshly-added standing note (a validation check written in one function is not automatically inherited by a sibling entry point accepting the "same" parameter) — grepped for every other validated-parameter shape in the codebase and found storage= (write_table/copytable/write_ms/copyms/ create_ms/reference_copy all accept it) had the exact same disease, in a more consequential form. The only place storage was actually checked was deep inside with_container_sink, called partway through the per-DM-writer section of _write_table_core — well AFTER _write_table_core/write_ms/create_ms had already mkpath'd the destination directory.

Live-verified the immediate symptom first: create_ms(dir; storage=:bogus) correctly threw an ArgumentError, but left an empty directory behind at dir that didn't exist before the call — the same "claims to have failed, but silently created state anyway" shape this codebase's whole atomic-write design (Phase 13/21's table.dat-is-the- commit-point philosophy) exists specifically to avoid.

Then found something much worse live-verifying write_ms: its subtable loop wraps each subtable's _copy_table call in a broad catch e; @warn "skipping subtable $kw (unsupported source)" typeof(sub) err=e — which catches ANY error, including a caller's own invalid storage= value. On the sample MS this produced 18 separate misleading warnings ("skipping subtable ANTENNA (unsupported source)", "skipping subtable FIELD (unsupported source)", …), each blaming the wrong thing (subtable data compatibility) for what was actually one single, unrelated, global configuration typo — before the real ArgumentError finally surfaced only when MAIN (which has no such catch) was reached. A user staring at 18 "unsupported source" warnings would have no reason to suspect their own storage= argument.

Fixed with a shared _check_storage(storage) (mirroring Phase 198's _normalize_precision pattern exactly), called FIRST — before any directory is created or any subtable is touched — in every one of the five top-level entry points that accept storage= (_write_table_core, write_ms, create_ms; write_table/ copytable/reference_copy needed no separate fix since they delegate straight into _write_table_core with no mkpath of their own). with_container_sink now calls the same shared check instead of its own inline duplicate. Live-verified across all 5 entry points: an invalid storage= now fails immediately with no directory created and (for write_ms) zero misleading warnings; all 3 valid values (:sepfile/:multifile/:multihdf5) still behave exactly as before, confirmed via a full create_ms(...; storage=:multifile) → readtable → subtable round-trip. New testset "storage= bad value: no stray directory, no misleading warnings (Phase 199)" (11 assertions, using Test.collect_test_logs to positively confirm zero warnings fire, not just that the eventual error is correct). Standalone container_tests.jl green (75/75), plus a broader writer_tests.jl + edit_tests.jl + container_tests.jl sweep together, no regressions.

Phase 200 — a fourth mscal.time() NaN/Inf crash corner: a literal nan/inf in the WHERE-string spec itself, not just a non-finite column value

Continuing the Phase 192–195/128 standing methodology note (once one function in a corner has a "raw Int(...)/round(Int,...) on a value that can be NaN/Inf" bug shape, grep the rest of that corner for the same shape) — swept src/taql/mscal.jl's date/time helpers once more for a spot Phase 194 hadn't covered. Phase 194 fixed _mssel_time's default-row TIME (a column value) going non-finite; this phase found a completely independent reachability path through the spec string itself: tryparse(Float64, "nan") succeeds in Julia (and "inf" too), so a literal mscal.time('nan~01/02') — an ordinary WHERE-string typo or a copy-pasted placeholder, nothing to do with the underlying data — parses to a genuine NaN/Inf token value and used to crash the whole predicate two different ways:

• _mstime_incl_hi (a ~-range's upper bound, when it needs to inherit missing calendar fields from the lower bound) fed a non-finite lo_secs straight into round(Int, lo_secs * 1000) — the by-now-familiar InexactError. Live-verified reachable via mscal.time('nan~01/02') on an ordinary table with a perfectly finite TIME column.
• A second, more fundamental site, found by tracing the fix one level deeper: _mstime_secs (the shared "fill wildcard fields from a default calendar, then convert to seconds" helper every partial Y/M/D token goes through) tests fields[k] < 0 to decide whether a field is a wildcard needing the default substituted in — but NaN < 0 is false in Julia, so a NaN field silently survived the substitution meant to catch exactly this case and reached Int(f[1]) unguarded. Live-verified reachable via mscal.time('nan/02') alone (no range, no default-row involvement at all — a bare partial token with one NaN field is enough).

Fixed both, same "degrade to a well-defined sentinel rather than crash" convention as Phases 193–195: _mstime_incl_hi now guards isfinite(lo_secs) before the round; _mstime_secs's wildcard test became fields[k] < 0 || !isfinite(fields[k]), so a NaN/Inf field is treated exactly like an omitted/* field and falls back to the (always-finite) default calendar — giving a sensible resolved date rather than propagating NaN further, and strictly more useful than a NaN-sentinel degrade would have been here. New testset "TaQL-lite — mscal.time() doesn't throw on a NaN/Inf SPEC token (Phase 200)" (13 assertions) covering both crash sites plus the wildcard-fallback behaviour (>nan/02 legitimately matches every row, since the NaN day field falls back to the resolved default day). Standalone taql_mscal_tests.jl green in full (every existing testset, including the Phase 121/128/194 time-family ones), no regressions.

Phase 201 — addcolumn!'s kind= was validated nowhere: a typo silently produced a column whose declared manager disagreed with its actual on-disk encoding

Start of a src/tables/ sweep (the user asked to focus there until it's complete — this and the phases that follow stay in that directory). Investigated src/tables/record.jl/writer.jl first (the Record/TableRecord/keyword-set AipsIO framing) against casa/Containers/RecordRep.cc + RecordDesc.cc + tables/Tables/TableRecordRep.cc line by line — every read/write pairing (the TpRecord-nested-empty-subdesc convention, the version>1-gated comment field, the old-style scalar/array keyset type-order tables, the Map<String,void> framing) matches exactly, no bug. One genuine-looking lead (_write_aipsarray always emitting the object type name "Array<void>", where casacore's own putDataField uses a distinct name per element type — "Array<Int>", "Array<uChar>", …) turned out to be harmless on investigation: casacore's own generic operator>>(AipsIO&, Array<T>&) reads whatever type string is actually on disk and uses it purely for getstart/getend framing depth-bookkeeping — it never cross-checks that string against the static C++ type T the caller declared, so the element type dispatch comes entirely from the RecordDesc's own type enum (known ahead of time), not from the Array object's name. This package's own read_array is equally permissive (accepts any "Array"/"Array<…>" prefix). Confirmed both directions live (our writer → real Casacore.jl, and — implicitly, since every existing keyword-array round-trip test already exercises it — real casacore → our reader) with no divergence. Also re-verified _resolve_tabpath/_strip_directory (RefTable/ConcatTable relative-path resolution) against casa/OS/Path.cc's addDirectory/stripDirectory in full — the 0/2/≥4-leading-"./"-characters case split matches exactly; the one form we don't implement (a trailing "/." reverse-relative reference, stripDirectory's "target is a prefix of the referencing table's own path" branch) was already a documented limitation from Phases 14/15, not a new find, and our own writer never produces it.

The actual find was in src/tables/edit.jl: addcolumn!'s kind= kwarg (:ssm/:ism/:tsm/:tcm/:tcell) was accepted with no validation anywhere, and — worse than a simple silent-fallback — an invalid value fell through two independent unguarded ternaries that default to DIFFERENT branches for the same unrecognised symbol: _normalize_desc's kind === :ism ? "IncrementalStMan" : "StandardStMan" (defaults to :ssm's manager string) and _flush_regen's writer dispatch k === :ssm ? write_standardstman(...) : write_incrementalstman(...) (defaults to :ism's writer). Live-verified: addcolumn!(t, "B", data; kind=:ssn) (a typo for :ssm) produced a column whose ColumnDesc.manager field claims "StandardStMan" while the bytes on disk are genuinely IncrementalStMan-encoded — a real metadata/data mismatch. Both this package's own reader and a real Casacore.jl cross-check still opened the resulting table and read the right values, because the data-manager type actually used for dispatch on read comes from the per-instance string in table.dat's ColumnSet, not the per-column manager field — the same "informational, not load-bearing" role ColumnDesc.manager already has documented at _source_dm (create.jl:799-805, added specifically because a real reference MS mislabels its own ISM columns as StandardStMan) — but the wrong metadata is still real and user-visible (columndesc(t,"B").manager lies about the column's actual encoding).

Fixed with a shared _check_kind(kind) (same "validate at the API boundary, not deep in the pipeline" pattern as Phase 198's _normalize_precision / Phase 199's _check_storage), called at every addcolumn! entry point: both EditTable methods, and — found by tracing the delegation chain — RefEditTable's with-data method, which pushes straight onto t.parent.addcols rather than calling addcolumn!(::EditTable, ...) and so needed its own separate call (RefEditTable's no-data method and both ConcatEditTable methods already delegate through the now-guarded EditTable path). New testset "edit — addcolumn! kind= is validated (Phase 201)" (10 assertions) covering all four entry points rejecting an invalid kind, plus a positive check that :ssm/:ism still work and produce metadata that genuinely matches the encoding (cross-checked against real Casacore.jl). Standalone edit_tests.jl green in full (every existing testset, including the Phase 125-133 RefEditTable/ ConcatEditTable ones), no regressions.

Phase 202 — write_table's ism= kwarg was the one sibling of tsm=/tcm=/tcell=/dysco= with no name validation at all

Continuing the src/tables/ sweep. _write_table_core's tsm=/ tcm=/tcell=/dysco= kwargs (each a set of column-name groups) all validate every referenced name and raise a clear "unknown column" error — confirmed by reading the tiled-group and Dysco-group write loops directly. ism= (a plain set of column names, no grouping) is the one sibling that skipped this: it's consumed only via ism_i = findall(c -> c.name in ism, descs), and findall simply omits a name that matches nothing — no error, no warning, the column is silently never bound to IncrementalStMan. Live-verified: write_table(dir, "T", ["A" => ...]; nrow, ism = Set(["A", "NOTACOLUMN"])) used to succeed outright, writing "A" to ISM and quietly discarding the typo'd "NOTACOLUMN" with no indication anything was wrong.

Fixed with a small validation loop (mirroring tsm='s own pattern), placed right before ism_i is computed: every name in ism must match a real column, else a clear "ism: unknown column ..." error. Confirmed this matches — not diverges from — the existing tsm=/ tcm=/tcell=/dysco= behaviour in one respect worth noting: all of these checks run after _write_table_core's own mkpath(dir), so an invalid name (in ism= now, same as its siblings already) still leaves a stray empty directory behind — a real, Phase-199-shaped gap, but one that already applied uniformly to every group kwarg before this phase, not something this fix introduces or was scoped to close (closing it for all five kwargs at once would mean restructuring with_container_sink's relationship to mkpath, a larger, separate change). New testset "ism= an unknown column name errors (Phase 202)" (2 assertions: the typo'd case errors, a valid ism= set still binds correctly) in test/ism_writer_tests.jl, right next to the existing ISM writer round-trip tests. Confirmed no regression to the internal callers that already pass real, always-valid ism= sets — create_ms, copyms of the sample MS's MAIN scalar columns, and the Dysco/reftable/container fixtures that use ism= — all still green. Standalone ism_writer_tests.jl, writer_tests.jl, edit_tests.jl, schema_tests.jl, and container_tests.jl together, no regressions.

Phase 203 — readtable(refpath; precision=...) was silently ignored on a persisted RefTable/ConcatTable

Continuing the src/tables/ sweep, in column.jl this time — read it in full against the _eltype/_narrowtarget/Column/MappedColumn/ ConcatColumn machinery, no bug found (the ConcatColumn duplicate- offset/empty-part searchsortedlast behaviour, already documented as correct since Phase 14, was re-derived by hand and reconfirmed). While tracing how a RefTable's effective read precision is actually determined (column(t::RefTable, name) delegates to t.parent's own .precision), found the real bug one file over, in table.jl: readtable computes its own resolved prec (:half/:full/…) right at the top, from the table's own table.info Type — but the moment the table being opened turns out to be a RefTable/ConcatTable, it dispatches to _read_reftable/_read_concattable, and neither ever received prec (or the raw precision argument) at all — the parent(s)/part(s) were always reopened via a plain readtable(p) buried inside _open_referenced, so readtable(refpath; precision=:full) on a persisted RefTable/ConcatTable was a silent no-op: the parent still opened at ITS OWN auto-derived default.

This one stayed hidden longer than most of this sweep's findings because the common case looked completely correct: a RefTable's own table.info Type is always copied verbatim from its parent (Phase 15's write_reftable/real casacore's RefTable::setup both do this), so the auto-derived default the RefTable's own readtable call would have computed (had it been threaded through) is identical to what the parent independently re-derives on its own — the bug is entirely invisible unless you pass an explicit precision= override, which no existing test did for this specific read path (query()'s in-memory RefTable construction, and column(t::RefTable,...)'s own delegation to t.parent.precision, are different code paths that already worked correctly and masked the gap in the persisted readtable(refpath; precision=...) path specifically). Live-verified: readtable(rdir; precision=:full) on a write_reftable-persisted selection over a real MS's MAIN gave back ComplexF16 DATA regardless of the override.

Fixed by threading the raw (possibly nothing) precision argument — not the pre-normalized prec — through _read_reftable/ _read_concattable/_open_referenced into the parent/part readtable calls: a deliberately conservative choice, so the default (nothing) case is byte-for-byte unchanged (each parent/part still independently re-derives its own default from its own table.info Type, exactly as before — relevant for the edge case of a ConcatTable whose parts happen to have heterogeneous Type strings, which a "always inherit one resolved value" version of this fix would have quietly changed), while an explicit override now genuinely propagates. resync needed the mirror-image fix: a RefTable/ConcatTable carries no precision field of its own (it lives entirely on the underlying Table(s)), so resync(t::Union{RefTable,ConcatTable})'s own readtable(t.path) call (no precision at all) would have silently reverted an explicitly- opened :full/BFloat16 RefTable back to the auto-derived default on every resync — the exact same "explicit override lost across a re-read" shape, one level further out. Fixed with a new _effective_precision helper (Table → .precision; RefTable → recurse into .parent; ConcatTable → recurse into .parts[1]; GroupedTable → nothing, handling a nested RefTable-of-RefTable chain too) whose result resync now passes through explicitly, mirroring how resync(::Table) already preserves t.precision. New testset "precision — readtable(refpath; precision=...) on a persisted RefTable/ConcatTable (Phase 203)" (8 assertions: default unchanged for both RefTable and ConcatTable, an explicit :full and BFloat16 both now take effect, and resync preserves an explicitly-opened :full RefTable's precision). Standalone reftable_tests.jl and precision_tests.jl together, no regressions.

newer entries →