Phase 300 — solar-system-body direction random fuzz vs casatools: the Sun was off by the constant of aberration
Continuing the Phase 269–287/299 pattern — a single deterministic oracle point repeatedly missing bugs a broader random fuzz catches — this phase spread the solar-system-body direction cross-check (Phase 76, previously checked at exactly one fixed epoch/observer position across 6 bodies) across 10 random epochs (1970–2050) and random observer positions, in one casatools process.
Found a real bug: the Sun's converted direction came out a nearly constant ~20.2″–20.8″ away from casatools' own value at every single random case, while Mercury/Venus/Moon/Mars/Jupiter did not show anything like this pattern (their residuals were noisy and consistent with the already-documented plan94/moon98 ephemeris-accuracy floors). That magnitude and constancy is the unmistakable signature of the classical constant of aberration (Earth's own orbital speed × the ~499 s Sun–Earth light time, ÷ 1 AU ≈ 20.5″, varying ~20.1″–20.9″ across the year with Earth's orbital eccentricity) — not ephemeris noise.
Root cause: _body_geovec(::Type{SUN}, tdb, ::Any) retarded Earth's own position by the light time (_earth_helio(tdb - lighttime)) inside a 2-iteration fixed-point loop, mirroring the shape of the general-planet method's light-time iteration — but the general method (correctly) holds Earth's position fixed at the observation time (eb = _earth_helio(tdb), computed once, never retarded) and only retards the target's own position. The Sun has no such "own position" to retard at all: by the very definition of a heliocentric coordinate frame, the Sun sits at the origin at every instant, regardless of when you evaluate it. Retarding Earth's position instead spuriously injects Earth's own orbital displacement over that ~499 s into the computed Sun direction — exactly the aberration-constant-sized error observed. Fixed to a one-line, non-iterating _body_geovec(::Type{SUN}, tdb, ::Any) = .-_earth_helio(tdb).
This bug has been present since Phase 76 (the first solar-system-body-direction phase) and affected every conversion of MDirection{SUN} to any other frame. The pre-existing single-fixed-epoch cross-check in this same file never caught it purely by luck: the aberration constant's yearly range (~20.1″–20.9″) straddles the test's own 20″ SUN tolerance almost exactly, and that one fixture date happened to land just under it.
New testset in test/measures_tests.jl (60 cross-check assertions across the 10 random epochs × 6 bodies, fixed-seed MersenneTwister(300)); confirmed the fix closes every one of the 4 failures the fuzz found, with no regression to the existing single-point Sun/planet tests or any other body.
Phase 301 — mscal.el1()/az1()/azel1()/pa1()/last1()/uvw_j2000() vs real derivedmscal on randomised rows
mscal.itrf()/delay()/hadec1() got the real 80-randomised-row derivedmscal cross-check treatment in Phase 280 (which found a real bug); the rest of the direction/uvw family (el1/az1/azel1/pa1/ last1/uvw_j2000) had only ever been checked at a handful of fixed rows via a self-consistency reference (reusing this package's own measure/measconvert code, not an independent oracle). This phase extends the same randomised-row treatment to the rest of the family, on the identical randomised copy shape (80 rows, random TIME/ANTENNA1/ANTENNA2/FIELD_ID).
Found 1 (not a bug — a genuine, previously-undocumented divergence): real casacore's derivedmscal (derivedmscal/DerivedMC/Register.cc, read directly) registers AZEL/AZEL1/AZEL2 (the combined 2-vector) and PA1/PA2, but has no AZ1/AZ2/EL1/EL2 under any name at all — unlike the bare pa / underscored uvw_j2000 spellings (Phase 163/164), which are aliases of a real registered function, mscal.el1()/az1()/el2()/az2() are pure MeasurementSets-only convenience accessors with no real casacore counterpart whatsoever. Compared here against the real azel1() vector's own components (the only way to check them against an independent oracle at all) — both agree to the same ~3e-5 rad residual as azel1() itself, consistent with ordinary azimuth-near-zenith numerical degeneracy (a tiny sky-position difference maps to a much larger azimuth-angle difference near the zenith, where azimuth itself is ill-defined), not a functional issue. Documented in src/taql/mscal.jl.
Found 2 (a genuine upstream casacore bug, confirmed directly via source — not something to fix in this package's own code, the same category as the Phase 279 DELETE-drops-Bool-bits and Phase 284 gmax-of-an-all-negative-group findings): real casacore's derivedmscal.LAST/LAST1/LAST2 do not vary with the row's own TIME at all, in the casacore build available for cross-checking here. Confirmed with a controlled experiment: holding the antenna fixed and sweeping TIME across a full day, the real LAST value moved by under one second total, where a genuine sidereal-time computation must sweep through the full ~86164-second sidereal day; holding TIME fixed and varying the antenna instead, real LAST barely moved either (consistent with only the tiny antenna-to-antenna longitude difference showing through). This package's own mscal.last1()/last()/last2() (an independent SOFA.gst06a-based computation, already CASA-cross-checked at a single epoch since Phase 66/77) correctly sweeps through the full [0, 2π) range across the same random rows. Traced into derivedmscal/DerivedMC/MSCalEngine.cc's setData/getLAST: itsUTCToLAST.setModel(epoch) is called with a genuinely per-row-varying epoch whenever time != itsLastTime — the C++ source itself looks correct, so this is either an MEpoch::Convert caching defect specific to an epoch-to-epoch conversion route (as opposed to the direction-to-direction AZEL/HADEC/ITRF conversions, which do vary correctly with time in this same build) in this compiled library, or a version-specific regression — not chased further, since it is unambiguously not this package's own bug. last1() is therefore not cross-checked against the real oracle in the new test (it would fail — the oracle itself is broken in this build); a sanity check confirms our own value stays in range and genuinely varies with time.
Also confirmed (a tolerance-only fix, not a bug): azel1(), the el1()/az1() component comparisons, and pa1() all needed a slightly looser tolerance (~1e-4 rad) than itrf()/hadec1()'s already- established ~2e-5/1e-5 rad — consistent with the azimuth/position-angle-near-zenith degeneracy noted above, not a new divergence; and uvw_j2000()/uvwj2000() needed a relative (not fixed absolute) tolerance, since random antenna pairs produce baseline lengths spanning a wide range and the established SOFA-vs-casacore ephemeris residual (Phase 137/196/280) is itself relative (~1e-4).
New testset in test/taql_mscal_tests.jl; src/taql/mscal.jl's own header comment updated with both findings.
Phase 302 — epoch conversion random fuzz vs casatools, including TDB's position term (investigation only, no bug found)
The CASA-oracle epoch conversion cross-check (measures_fixture.py) only ever checked 3 fixed epochs — and, notably, called me.doframe(e) but never me.doframe(pos) before converting, so TDB's position-dependent term (_dtdb_loc's longitude/height-derived arguments to SOFA.dtdb) had never actually been exercised against a real oracle at all; every existing TDB check compared against casacore's own position-less default. This phase spreads the oracle across 15 random epochs (1975–2030, safely inside the UTC leap-second table) and random global observer positions, with the position genuinely set via me.doframe(pos) before every conversion — the first time this package's TDB position term has been checked against anything other than casacore's default.
No new bug found: UTC→{TAI, TT, TDB, UT1} all agree with casatools to the same tight tolerances already established for the position-less case (TAI/TT to <1e-9 days ≈ 0.1 ms; TDB/UT1 to <1e-7 days, the dtdb/no-EarthOrientation accuracy floor) across every one of the 15 random epoch/position combinations, confirming the position-dependent TDB term is correctly wired even though nothing had ever actually checked it against an independent oracle before.
New testset in test/measures_tests.jl (61 cross-check assertions, fixed-seed MersenneTwister(302)).
Phase 303 — MBaseline/MuvW frame conversion random fuzz vs casatools (investigation only, no bug found)
MBaseline/MuvW frame conversion (Phase 75) had no confirmed real-CASA oracle at all beyond uvw_j2000's narrow TaQL-level scope — Phase 75's own risk note explicitly flagged "no CASA oracle confirmed available for uvw/baseline". Found live that casatools.measures() genuinely has me.baseline(rf, x, y, z) / me.uvw(rf, x, y, z), each returned as a spherical (lon, lat, length) triple (the same convention as MPosition's own spherical ITRF representation, Phase 155) rather than Cartesian — converted to Cartesian for a direct comparison with this package's own MBaseline/MuvW (always Cartesian). Fuzzed across 12 random epochs/positions/directions and random synthetic baselines against 6 target frames for MBaseline (J2000/GALACTIC/B1950/ECLIPTIC/AZEL/HADEC) and 3 for MuvW (J2000/GALACTIC/AZEL).
No new bug found: every conversion matches to a relative error of at most ~1.1e-4 (MBaseline) / ~2.6e-4 (MuvW) of the baseline length — squarely the same SOFA-vs-casacore ephemeris/EOP residual class already established and accepted for uvw_j2000/itrf/delay elsewhere (Phase 137/196/280/300 all cite a similar ~1e-4-ish relative floor), not a new divergence. This closes the "no CASA oracle confirmed" gap Phase 75 flagged, for the general (non-uvw_j2000-specific) conversion machinery MBaseline/MuvW share with the direction-conversion code that Phases 280/300 both found real bugs in — genuinely reassuring given that track record, not merely a formality.
New testset in test/measures_tests.jl (109 cross-check assertions, fixed-seed MersenneTwister(303)).
Phase 304 — Dysco write-path random-parameter fuzz vs casatools decode (investigation only, no bug found)
Phase 19's write-direction interop oracle (our write_dyscostman → real CASA's own getcol() decode) was, like several other cross-checks this project has since found real bugs by widening (Phase 280's ITRF-aberration bug, Phase 284's gmax-of-all-negative-group bug, Phase 300's Sun-aberration bug), only ever exercised at one fixed shape/antenna-count/bit-width combination (nant=4, npol=2, nchan=4, dataBitCount=10, weightBitCount=12, rowsPerBlock=nbl — one block per timestamp) crossed with the 3×4 normalization/distribution grid. This phase spreads the same oracle across 8 random configurations: antenna count (2–7, driving _dysco_metacount's AF-normalization antenna-indexed metadata size), cell shape (npol∈1:3, nchan∈1:6), dataBitCount/weightBitCount (4/6/8/10/12/16 and 4/8/12/16 — bit widths the fixed-point test never exercised the generic bit-packer at), and block layout (one-block-per-timestamp / one-big-block / a deliberately uneven multi-block split forcing a short final block), on top of the existing random normalization×distribution×dither sampling.
No new bug found: every one of the 8 random configurations decodes via real casatools to within float32-rounding agreement (< 1e-3 for DATA, exactly 0.0 for WEIGHT_SPECTRUM's linear quantizer) — confirming _dysco_metacount's per-normalization header-size arithmetic and the generic LSB-first bit-packer are correct across the antenna-count/bit-width/block-layout space the original fixed-point test never touched, not just at the one combination it happened to use.
New testset in test/dysco_tests.jl (8 random cases × 3 assertions, fixed-seed MersenneTwister(304)).
Phase 305 — CompressFloat/CompressComplex/CompressComplexSD random-parameter fuzz vs Casacore.jl: found and fixed a real declared-type validation gap
Following Phase 304's pattern, spread Phase 12's CompressFloat/CompressComplex/CompressComplexSD engine cross-check (previously exercised only at one fixed cell shape, one fixed scale/offset per kind, against Casacore.jl, a genuine cross-implementation oracle) across random cell shapes, random positive scales spanning several magnitude decades, random nonzero offsets, and — for CompressComplexSD specifically — a deliberate mix of purely-real ("even") and genuinely-complex ("odd") cells, since scaleOnPut's real/imaginary dispatch is the one piece of encode logic the original fixed fixture never varied.
Found a real bug while constructing the fuzz data itself: an early draft generated ComplexF64 cell data for the CompressComplex case (a Julia promotion artefact — a Float64-typed amp scalar promoted the whole expression). write_table(...; engines = Dict(nm => (; kind = CompressComplex(), ...))) on that ComplexF64 data succeeded silently, producing a column declared TpDComplex (inferred purely from the caller's Julia array eltype, _casatype_of(eltype(vals))) — but CompressFloat/CompressComplex/CompressComplexSD are real casacore's fixed, non-templated VirtualColumnEngine<Float>/<Complex> classes (unlike the genuinely templated ScaledArrayEngine<S,T> /ScaledComplexData<S,T>), structurally incapable of ever storing anything but Float32/ComplexF32. Our own reader tolerated the mismatch (it narrows/widens transparently via the ordinary Phase-34/35 precision machinery, oblivious to the declared type being wrong), but real casacore genuinely refuses to open the table: "Invalid data type when accessing column Column C1 has data type Complex ; expected DComplex" — live-verified via Casacore.jl, confirming this is a genuine interop-breaking gap, not a theoretical one.
Fixed with a new _compress_expected_vtype(::CompressKind) (src/datamanagers/virtual.jl) validated in _write_table_core's engine loop (src/tables/create.jl), before any encoding work, raising a clear ArgumentError naming the mismatch and the required conversion — matching this project's established "validate the parameter, don't silently produce something a downstream reader chokes on" discipline (Phase 199/201/202/204/205/210 all fixed the same shape of gap elsewhere). ScaledArrayEngine/ ScaledComplexData (genuinely templated — any stored_type the caller picks) and MappedArrayEngine/ BitFlagsEngine (no precision axis to mismatch) are unaffected; the check is scoped to kind isa CompressKind.
Once the test data itself used correctly-precisioned Float32/ComplexF32 values, every one of the 10 random shape/scale/offset combinations × 3 engine kinds agreed with Casacore.jl's own decode bit-for-bit (not just "close enough") — confirming the encode/decode math itself was already correct across the wider parameter space; the real find was the missing type-consistency check, not a codec bug. A companion 5-case random-parameter fuzz of the autoScale (per-row scale/offset) path similarly found no codec bug once its own test data avoided a second, unrelated degenerate construction (a row whose imaginary part sat at exactly 0 while the real part carried a large offset — not representative complex data, and not what findMinMax's joint real/imaginary scan is meant to size a scale/offset for).
New testsets in test/engine_tests.jl: the random-parameter fuzz (10 cases × 3 kinds, fixed-seed MersenneTwister(305)), the autoScale fuzz (5 cases, MersenneTwister(3050)), and a permanent regression test confirming Float64/ComplexF64 input through any of the three CompressKind engines now raises a clear error while Float32/ComplexF32 input still works.
Phase 306 — BitFlagsEngine random-parameter fuzz: stored_type + FLAGSETS vs Casacore.jl (investigation only, no bug found)
Every BitFlagsEngine test since it was implemented (Phase 40) and its readmask/FLAGSETS bug fixed (Phase 156) had only ever used stored_type = TpInt — real casacore's BitFlagsEngine<StoredType> is genuinely instantiated for three distinct stored types (uChar/Short/Int, each separately auto-registered in DataManager::initRegisterMap), and _engine_typestr(::BitFlags, ...) builds the on-disk DM class-name string from _TYPEID[stored_type] — but the uChar/Short spellings had zero test coverage, let alone a real-casacore open. This phase spreads the Casacore.jl cross-check across a random stored_type (uChar/Short/Int), random cell shapes/row counts, and a random FLAGSETS key set (2–5 named bits, one deliberately holding bit 0 — the only bit that can ever read back true, since the raw stored value is always exactly 0/1 — the rest noise bits), with readmaskkeys a random subset in random order, sometimes including a nonexistent key name (exercising Phase 156's silent-skip path at scale, not just the original fixed 2-key fixture).
No new bug found: all 8 random configurations — across all three stored types — produce a DM class-name string real casacore accepts (BitFlagsEngine<uChar/Short/Int, confirming the 8-char padding is correct for every width, not just the one previously tested) and a mask-recomputed read result that matches both this package's own reader and Casacore.jl's decode exactly. Closes a real "never actually checked" gap — the uChar/Short stored-type paths were reachable, plausible-looking code with literally no live verification behind them until now.
New testset in test/engine_tests.jl (8 random cases × 3 assertions, fixed-seed MersenneTwister(306)).
Phase 307 — ForwardColumnEngine random column-type fuzz vs Casacore.jl (investigation only, no bug found)
The ForwardColumnEngine/reference_copy test (Phase 40) only ever forwarded a scalar Float64 column and a fixed-shape ComplexF32 array column, writing one Int32 scalar. getcell/getcolumn (src/datamanagers/forwardcol.jl) are a pure, type-agnostic pass-through (column(_fce_ref(fce), fce.vdesc.name) — no type-specific branch at all), so nothing in the engine's own code obviously restricted it to those two shapes — but a String column, a Bool column, and a genuinely variable-shape (ragged) array column had never actually been tried forwarded, despite being exactly the kind of column a real reference-MS copy (MSTableImpl::referenceCopy, what this engine models) would routinely need to carry across untouched.
Spread the cross-check (ForwardColumnEngine is auto-registered in real casacore, so Casacore.jl is a genuine interop oracle, not self-consistency) across 6 random source tables mixing String (including empty strings), Bool, Int32, and ragged Float64 array columns, a random subset marked writable (independent copy) vs. left forwarded, cross-checked both through this package's own reader and through Casacore.jl's decode, and through copytable's materialise-through-the-forward path.
No new bug found: every column type — forwarded or independently copied — round-trips correctly through both readers in all 6 random configurations, confirming the pass-through implementation is genuinely type-agnostic in practice, not just by the absence of an obvious type-specific branch in the source.
New testset in test/engine_tests.jl (6 random cases × 4 column types × 3 checks, fixed-seed MersenneTwister(307)).
Phase 308 — write_concattable random N-part fuzz: found and confirmed a genuine upstream casacore ConcatTable/String-column limitation (not a bug in this package)
The write_concattable test (Phase 15) only ever concatenates exactly 2 nonempty parts with one Int32 column. Phase 15's own risk note explicitly anticipated a genuinely empty interior part ("searchsortedlast with duplicate offsets ... correctly skips the empty part") but no test had ever exercised one. Spread write_concattable + ConcatColumn's row→part lookup across a random number of parts (2–5), random per-part row counts, and a mix of Int32/Float64/String columns, cross-checked against Casacore.jl.
Found a real, reproducible divergence while constructing the fuzz itself: a ConcatTable with an empty part positioned before a nonempty one (whether first or merely interior) and a String column threw "SSMIndex::getIndex - access to non-existing row 0" through Casacore.jl. Live investigation — reading ~/Development/CASACORE/tables/DataMan/SSMIndex.cc's getIndex (throws whenever itsNUsed==0, i.e. no row has ever been added to that table's SSMIndex) plus a byte-swap experiment (replacing only our written table.f<seq> with the bytes of a table CREATE TABLE ... LIMIT 0 itself wrote, keeping everything else ours, made the failure disappear) — narrowed it down. The decisive test: does the identical failure reproduce when the empty part is authored by real casacore's own CREATE TABLE ... LIMIT 0, not just by this package's writer? It does, byte-for-byte the same error message, whether the empty part came first or was merely interior — confirming this is a genuine, pre-existing upstream casacore ConcatTable/SSMIndStringColumn limitation (real casacore's own ConcatColumn implementation evidently bootstraps something, per nonempty part, from every earlier part including empty ones, that unconditionally probes row 0), not a divergence in this package's writer. An initial hypothesis ("only the first part matters") was itself refined mid-investigation once a broader fuzz case — an empty interior part followed by a nonempty one — reproduced the identical error; the correct characterization is "an empty part anywhere before a nonempty one", not "the first part specifically". Confirmed not to occur when an empty part is a pure trailing suffix (nothing nonempty follows it), in either this package's writer or real casacore's own — nor when the column isn't String/SSM-indirect. This package's own reader is completely unaffected either way — only real casacore's ConcatTable+ SSMIndStringColumn combination has the gap.
The fuzz's own row-count generation is scoped to trailing-only empty parts (the confirmed-safe shape) so it targets this package's own correctness against a real, non-broken oracle; the empty-before-nonempty case is documented and regression-tested separately, including the real-casacore-authored confirmation.
New testsets in test/reftable_tests.jl: the random N-part fuzz (6 cases, fixed-seed MersenneTwister(308)) and a dedicated testset pinning both the confirmed upstream limitation (our writer and a real-casacore-authored empty part fail identically) and the confirmed-safe trailing-empty case.
Phase 309 — TiledColumnStMan writer random shape/type-mix fuzz vs Casacore.jl (investigation only, no bug found)
The TiledColumnStMan writer test (Phase 11) only ever uses one fixed cell shape (a 3-element 1-D vector) across both its groups, with 3 types total. write_tiledcolumnstman requires every column in one group to share the identical cell shape (checked directly), but the types bound to one shared hypercube are free to differ and go through the same canonical-size tie-break sort _tile_layout uses everywhere else — already dedicated-tested for TiledShapeStMan's "equal-size types" case, but never for TiledColumnStMan specifically, nor with Bool's zero-canonical-size special case in the mix.
Spread random cell shapes (1-D and 2-D), random per-group column counts (1–3), and a random mix of Float32/Float64/ComplexF32/Int32/Bool types per group across multiple groups in one table, cross-checked against Casacore.jl.
No new bug found: all 57 assertions across 6 random configurations agree with Casacore.jl's decode — confirming _tile_layout's tie-break sort and the general TiledColumnStMan write path are correct across a much wider shape/type space than the original fixed-point test ever exercised.
New testset in test/tsm_multicol_tests.jl (6 random cases, fixed-seed MersenneTwister(309)).
Phase 310 — MultiFile container random blocksize/DM-mix fuzz vs Casacore.jl (investigation only, no bug found)
The write_table(...; storage=:multifile) round-trip test (Phase 21) only ever uses one fixed blocksize (128) with one fixed SSM+TSM column mix; the dedicated header-overflow test uses a second, also-fixed blocksize (64). _finalize_multifile's block-layout logic — assigning each data-manager file a contiguous 0-based block range sized to its own byte length, _mf_pack_index's run-length encoding of that range, and the fixed-point continuation-block convergence loop — all depend on how each DM file's byte length divides against blocksize, territory outside those two fixed values essentially untested.
Spread random blocksize values (64 — MultiFile's documented floor, Phase 211 — up to a few KiB), random row counts, and a random mix of SSM (Int32/Float64/String/Bool) and ISM columns plus one TSM column into one container, cross-checked against Casacore.jl for the non-TSM columns (a single-column TSM group is a known, unrelated pre-existing Casacore.jl interop gap documented since Phase 15/21, so TSM columns are checked only through this package's own reader, matching the existing test's own convention).
No new bug found: all 60 assertions across 6 random blocksize/row-count/column-type-mix configurations agree with Casacore.jl's decode — confirming the block-layout and pack-index logic hold across a much wider blocksize space than the two fixed values previously exercised.
New testset in test/container_tests.jl (6 random cases, fixed-seed MersenneTwister(310)).
Phase 311 — TiledCellStMan random per-row shape + type-mix fuzz vs Casacore.jl (investigation only, no bug found)
The multi-column TiledCellStMan success test (Phase 214) uses exactly 2 columns, one fixed shape family ((2, r+1)), one type (Float32), and 3 rows — and, unlike every other multi-column tiled writer test in this file, had no Casacore.jl cross-check at all for the shared-group success path (only the shape-mismatch error path was ever exercised against anything). write_tiledcellstman requires every column in a group to share the same cell dimensionality per row, but the extent per dimension is free to vary row by row — untested with genuinely random per-row extents, a random column count, or a random type mix.
Spread random per-row cell shapes (1-D and 2-D, random extents each row), a random column count (2–3), and a random mix of Float32/Float64/ComplexF32/Int32/Bool types across the shared group, cross-checked against Casacore.jl.
No new bug found: all 32 assertions across 6 random configurations agree with Casacore.jl's decode — the first genuine cross-implementation proof this package's multi-column TiledCellStMan writer has ever had for its success path, not just its validation-error path.
New testset in test/tsm_multicol_tests.jl (6 random cases, fixed-seed MersenneTwister(311)).
Phase 312 — IncrementalStMan random independent multi-column run-length fuzz vs Casacore.jl (investigation only, no bug found)
The "ISM writer round-trip" test (Phase 8) uses fixed run-length change points (splits at exactly 20/40 of 60 rows, runs of exactly 12) for every column; "ISM writer multi-bucket" forces more than one bucket but with only one column, one pattern (changes every row). Neither exercises multiple columns with genuinely random, independent, simultaneous run-length patterns spanning several bucket boundaries at once — _ism_colindex's per-column row-number/offset array construction is built per column independently, so misaligned bucket boundaries across columns with different change cadences is exactly the kind of interaction a fixed single-pattern fixture can't catch.
Spread a random number of columns (2–4), random independent run-length patterns per column (a random per-column "hold probability" so each column changes value at unpredictable, uncorrelated points), random types (Int32/Float64/Bool/String), and row counts large enough to force multiple ISM buckets, cross-checked at both the whole-column level and 10 random individual-row lookups per case (not just start/middle/end) against Casacore.jl.
No new bug found: all assertions across 5 random configurations — each genuinely spanning multiple ISM buckets with misaligned per-column change points — agree with Casacore.jl's decode, confirming the per-column bucket-index construction and lookup hold under column-to-column bucket-boundary misalignment, not just the single-column or fixed-pattern cases previously exercised.
New testset in test/ism_writer_tests.jl (5 random cases, fixed-seed MersenneTwister(312)).
Phase 313 — CI fix: CompressFloat/CompressComplex/CompressComplexSD engine cross-checks were architecture-dependent (loosened, not the implementation)
Real GitHub Actions CI (Linux x86-64) reported the Phase 305 fuzz testset's exact bit-for-bit decode comparison (test/engine_tests.jl, @test all(vc[i] == ct[Symbol(nm)][i] for i in eachindex(vals))) failing, even though the same test passed locally on the developer's Mac (ARM64) both standalone and in the full-suite run before the Phase 304–309/310–312 pushes.
Reproduced the failure on demand via the project's own established Docker-based x86-64 methodology (the Phase 192/193 precedent): started a genuine linux/amd64 julia:1.10 container over this repo, confirmed real x86_64 via uname -m, and ran a script mirroring the exact fuzz loop but printing every mismatch instead of asserting. The failure reproduced immediately — every mismatch was a tiny (last-Float32-mantissa-bit-scale) rounding difference between this package's own decode (vc[i]) and real casacore's C++ decode (ct[Symbol(nm)][i]) of the identical stored integers (stored[i] printed and confirmed equal for every mismatched pair, ruling out an encode-side/stored-integer divergence) — never a logic/formula bug.
Root cause: _decode's CompressFloat/CompressComplex/CompressComplexSD methods (virtual.jl) use muladd(stored, scale, offset) for precision. muladd is a request to fuse the multiply and add into a single-rounding FMA instruction when the target supports it — whether it actually fuses is an architecture/toolchain choice, made independently by Julia's LLVM backend and by whatever compiler/flags built real casacore's own scaleOnGet (stored*scale+offset) for its target. On this ARM64 Mac the two choices happen to coincide; on x86-64 Linux CI they don't, producing a consistent, tiny, legitimate last-ULP divergence — not a bug in either implementation, the classic signature of comparing two independently-compiled numerical implementations for bit-for-bit equality when only "close" is a meaningful guarantee.
Fixed the test's own strictness (mirrors the project's Phase 192 precedent: loosen the assertion, not the arithmetic) — replaced every remaining vc[i] == ct[...][i] cross-implementation check in test/engine_tests.jl (the original Phase 12 fixed-point test, the Phase 305 fixed-scale fuzz, and the Phase 305 autoScale fuzz — the last of which the Docker repro additionally found tripping the identical divergence, not reported in the original CI paste but caught rerunning the whole file on x86-64) with a small ULP-scale tolerance comparison (_ulp_close, 8 ULPs of Float32 at the compared magnitude).
Verified the fix on both platforms — the full existing cross-check suite (test/engine_tests.jl) and the targeted testsets, standalone, pass cleanly inside a fresh x86-64 Docker container after the fix, and the full local test suite (8110/8110) passes unchanged on ARM64. README/memory updated, merge on the user's word.
Phase 314 — Dysco READ random-parameter fuzz vs casatools-authored tables (investigation only, no bug found)
Phase 304 fuzzed the Dysco write direction (our writer → casatools decode). The read direction (casatools writes a Dysco table, we decode it) had only ever been checked at one fixed shape/bit-width point (4 antennas, 4×2 cells, 10/12 bits) plus the 12 normalization × distribution combos at that same point. Spread it across random antenna counts (2–7), integrations, cell shapes (1–4 pol × 1–9 chan), data/weight bit widths (4–16) and every normalization × distribution: 28 of 30 exploratory configs matched casatools' own getcol() to float rounding (both sides decode the same stored symbols); the other 2 never reached our reader — real casacore's own DyscoStMan aborts the process for ntime=1 ("flushed before at least two timeblocks were stored"), so the permanent test uses ntime ≥ 2.
Phase numbering note: "start phase 313" arrived after 313 was already used for the CI fix (PR #102), so this is 314.
New testset in test/dysco_tests.jl (8 random cases, fixed-seed MersenneTwister(314), gated on the CASA python like its siblings).
Phase 315 — StandardStMan writer random column-mix fuzz vs Casacore.jl (investigation only, no bug found)
The SSM writer's bucket sizing (rowsPerBucket from the summed cell widths), string buckets and bit-packed Bool / fixed-shape array cells depend on the column mix and row count, but had only been checked with a handful of fixed column sets. Spread 25 exploratory random configurations — 2–6 columns of Int32/Float64/Float32/Bool/String/ComplexF32/UInt8/Int16, scalar or fixed-shape array cells, strings from empty to 400 characters, 1 to 3000 rows (several buckets), both byte orders — over whole-column and per-row comparison against both the written data and Casacore.jl's decode: all agree. New testset in test/writer_tests.jl (12 cases, fixed-seed MersenneTwister(315)).
Phase 316 — write_reftable random row-list / select / chain fuzz vs Casacore.jl (investigation only, no bug found)
Phases 15 and 132 checked write_reftable with fixed row lists. Spread 30 exploratory random cases — random row lists (repeats, unsorted, and sorted so the rowOrder flag is exercised both ways), random column subsets with renames over Int32/Float64/String/array columns, and a second RefTable chained on the first (flattened to the root on write) — through both our reader and Casacore.jl: all agree. New testset in test/reftable_tests.jl (12 cases, fixed-seed MersenneTwister(316)).
Phase 317 — write → copytable → random edit-session pipeline fuzz; a documented addrows! divergence from real casacore on ISM columns (no bug)
Spread 25 exploratory random pipelines — write_table with mixed StandardStMan / IncrementalStMan / shared-TiledShapeStMan-group columns, copytable to a fresh table, then a random edit session (setcell! / addrows! / removerows!) — against a plain Julia model and Casacore.jl: after fixing my own model all agree, and the manager mix survives the copy.
The one thing the fuzz surfaced: for an IncrementalStMan column an appended row reads back as zero / "" here (the documented, manager-uniform addrows! contract — "until written"), whereas real casacore's ISM returns the previous row's value (confirmed with casatools addrows: [5,5,7,7,9] → […,9,9]), because its "store on change" file simply has no entry for the new rows. Kept ours (uniform, and users set the values), and documented the difference in the addrows! docstring. The Phase 9 plan text that said ISM appends inherit the last value was never what the regen path implements.
New testset in test/edit_tests.jl (10 cases, fixed-seed MersenneTwister(317)).
Phase 318 — taql SELECT … JOIN random fuzz vs real TaQL (investigation only, no bug found)
Phases 259/260 checked JOIN with a few hand-picked forms. Spread 40 exploratory random cases × 8 query shapes — random left/right/third tables with duplicate and unmatched keys, chained joins, rowid() index lookup, and WHERE / ORDER BY / LIMIT / GROUP BY / HAVING over joined columns — against real tableCommand: every query with a non-empty result matches exactly (unmatched rows get the type sentinels). The only differences are queries that select zero rows, where real TaQL throws an unexplained "Slicer error" (seen elsewhere in Phase 253) when the result columns are read, and ours returns 0 rows. (My first harness tripped the Phase 210 bare-[...]-literal promotion hazard — Int32 keys silently became Float64, which real TaQL rejects as a join key — worth remembering when writing fuzz fixtures.)
New testset in test/taql_command_tests.jl (12 cases × 8 queries, fixed-seed MersenneTwister(318)).
Phase 319 — taql SELECT ORDER BY / DISTINCT / LIMIT / OFFSET random fuzz vs real TaQL (no bug; one documented DISTINCT tie-order divergence)
Spread 40 exploratory random cases × 8 queries on low-cardinality columns (lots of ties): random 1–3-key ORDER BY with ASC/DESC, leading DESC, WHERE, LIMIT / negative LIMIT / OFFSET. All plain SELECTs match real tableCommand exactly (stable multi-key sort, tie order included). Queries with an empty or past-the-end window make real TaQL throw a lazy "Slicer error" (as in Phase 318); ours returns 0 rows or an ArgumentError.
SELECT DISTINCT … ORDER BY k returns the same set of rows, but the order among rows tied on k (and an ORDER BY on an unselected column, or a LIMIT cutting through ties) can differ: ours keeps the first row of each distinct tuple of the sorted result, whereas real TaQL dedups with a no-duplicates heap sort that keeps an arbitrary representative row per tuple (inferred from the tie patterns; small, tie-free cases agree). Not worth porting its heap sort — the tie order is unspecified in SQL terms — so it is documented in src/taql/commands.jl and the test compares DISTINCT results as multisets.
New testset in test/taql_command_tests.jl (12 cases × 8 queries, fixed-seed MersenneTwister(319)).
Phase 320 — Dysco copytable under random write parameters (investigation only, no bug found)
The Dysco copy-preservation test used one fixed configuration. Spread 25 exploratory random configurations — 2–6 antennas, 1–4 pol × 1–6 chan cells, data bits 6–16, weight bits 8–16, every normalization × distribution, and rowsPerBlock of one baseline-set / two / the whole table — through write → copytable: every compression parameter (normalization, distribution, both bit widths, rowsPerBlock, antennaCount, truncation) is preserved, and weights re-encode exactly. RowNorm/RFNorm copies decode essentially identically; AFNorm drifts by up to ~4% of the data magnitude at coarse bit widths because its iterative antenna/channel RMS solve is not idempotent on already-quantised data (expected; checked loosely at 10%).
New testset in test/dysco_tests.jl (10 cases, fixed-seed MersenneTwister(320); not CASA-gated).
Phase 321 — MDoppler conventions + rest-frequency bridge random fuzz vs casatools (investigation only, no bug found)
Phase 72's Doppler conversions and frequency ↔ velocity bridge were cross-checked against casatools at exactly one value (RADIO 0.01, one observed/rest pair). Spread the same oracle over 25 random physically-valid shifts (|β| < 0.9, rest frequency 1e8–5e11 Hz): all five conventions from a BETA value, doppler(f, rest), radialvelocity, frequency(d, rest) and restfrequency(f, d) agree with casatools to better than 1e-9 relative (pure algebra). New testset in test/measures_tests.jl (25 cases × 9 quantities, fixed-seed MersenneTwister(321), gated on the CASA python).
Phase 322 — random-operation fuzz for in-place edits through RefTable / ConcatTable views (investigation only, no bug found)
Phases 125–130 tested edit(rt::RefTable) / edit(ct::ConcatTable) with fixed rows and fixed parts. Spread 30 exploratory random sessions — random (sorted or shuffled) RefTable selections and random 2–3 part ConcatTables over scalar / String / tiled-array columns, with random cell writes and whole-column writes through the view — and compared the underlying parent / parts against a plain Julia model (our reader, and Casacore.jl for the RefTable case): all agree. The permanent testset also does a random addcolumn! through each view kind (RefTable: the view's rows get the data and every other parent row 0.0; ConcatTable: the data is split across the parts). New testset in test/edit_tests.jl (12 cases, fixed-seed MersenneTwister(322)).
Phase 323 — tiled-column random cell-shape fuzz incl. cells larger than a tile (investigation only, no bug found)
The TiledShapeStMan / TiledColumnStMan writers pick a tile shape for a ~1 MiB target, but were only tested with small, fixed cell shapes. Spread 60 exploratory random cases — 1–4 dimensions with extent-1 axes, cell types Float32/ComplexF32/Float64/Int32/Bool, shared one- or two-column groups, and cells up to several MB (so a tile holds less than one cell and the tile shape has to be clamped) — through write → read (column(...)[:], rawblock), an in-place cell edit, and Casacore.jl wherever it can fetch a cell (104 comparisons; it cannot index every fixed-shape tiled column, a known limitation): all agree. New testset in test/tsm_multicol_tests.jl (14 cases, fixed-seed MersenneTwister(323)).
Phase 324 — casacore ↔ Unitful unit mapping checked against casatools for every unit name: four silent mis-parses and four unreadable write spellings fixed
A round-trip fuzz of Unitful-typed columns (write → columnunit/qcolumn) over ~40 units turned up unit spellings that did not survive. Checking every unit name in casacore's UnitMap (152) against casatools' own canonical SI value (qa.canonical(qa.quantity(1, name))) showed the cause: casacore names that Unitful reads as a different unit entirely, silently giving wrong numbers:
h(hour) parsed as Planck's constant;a(annum) as the are (100 m²);G(gauss) as the gravitational constant;R(roentgen) as the gas constant;mindid not parse at all.
Fixed in _UNIT_ALIASES (h→hr, a→yr, min→minute, G→Gauss, plus Ohm, in, mile); R and Gb (gilbert), which have no Unitful counterpart, now raise the usual clear error instead of returning a wrong quantity. The write direction emitted strings casacore itself cannot read — hr, minute, Gauss, Å, Ω — now mapped to h, min, G, Angstrom, Ohm (confirmed accepted by casatools). Remaining differences are only older numeric constants in casacore (AU 2.7e-10, M0/S0 2.6e-4, u 3e-4, cal 1e-3 relative), documented in src/tables/units.jl. A TaQL spaced literal such as 3 h now means three hours rather than 3·Planck's-constant.
New testset in test/units_tests.jl: a static snapshot of casatools' canonical value for every unit name that parses (68 names), the previously mis-parsed names, the write spellings, and table round trips.
Phase 328 — CI fix: the Phase 320 Dysco copy test was platform- and run-dependent (test fix only)
GitHub Actions (x86-64 Linux) failed the Phase 320 test: an AFNorm copy drifted by 11.4% against its 10% bound. Two causes, both in the test, not the package. (1) The bound was empirical to one machine: re-running on a real x86-64 container showed AFNorm copy drift up to ~13% even at 16 bits (AF re-solves its antenna/channel RMS factors on the decoded data, which is non-idempotent and platform-dependent), so only a sanity bound (0.5) is meaningful. (2) copytable re-encodes Dysco with dither=true drawing from the global RNG, so the copy was nondeterministic run to run (one stray local failure, unreproducible in 5 reruns, exposed this); the test now seeds the global RNG as well as its own. Verified identical results over 6 repeated local runs and clean on x86-64 Docker.
Phase 325 — casacore's micro prefix u did not parse (all 24 SI prefixes diffed against casatools)
Following Phase 324's exhaustive name-table diff, compared casacore's 24 SI prefixes × m/Hz/g/s against casatools' canonical values. Everything agrees except: casacore's micro prefix is the letter u (uJy, us, um, uas), which none of Unitful's parsers accept — every microunit raised the "no Unitful equivalent" error — and the write direction emitted Unitful's μ, which casacore rejects. Fixed both (u<unit> → μ<unit> on read, with uas/uarcsec → μas; μ/µ → u on write); a bare u stays the atomic mass unit. Not fixed (rare, noted in the test): the 2022 prefixes Q/R/q/r (Unitful predates them) and das (casacore deci-arcsecond vs Unitful deka-second). New testset in test/units_tests.jl.
Phase 326 — mscal.stokes polarization-name table diffed against casacore's Stokes enum (no bug found)
Applying Phase 324/325's exhaustive name-table diff to polarization names: _STOKES_NAMES codes 1–20 (I Q U V, RR RL LR LL, XX XY YX YY, RX RY LX LY, XR XL YR YL) match casacore's Stokes::StokesTypes enum order exactly. casacore additionally defines PP PQ QP QQ (21–24) and RCircular / LCircular / Linear (25–27), which neither casacore's StokesConverter nor mscal.stokes converts; they raise a clear ArgumentError. Pinned by a static test in test/taql_mscal_tests.jl.
Phase 327 — MEarthMagnetic bare-code reference enum was off by one (all measure enums diffed against casacore)
The numeric enums used to decode a bare-code VarRefCol (no TabRefCodes) were diffed against casacore's Measures/M*.h Types enums and casatools' me.listcodes. Direction/uvw/baseline, epoch, frequency, radial velocity, Doppler and position all match. MEarthMagnetic did not: its Types enum is the direction list without B1950_VLA, so codes 5 and up decoded one frame too late (code 5 → B1950_VLA instead of BMEAN, …), and its model type IGRF is code 32, which was out of range. Fixed with a dedicated _EM_ENUM and the code-32 case. New testset in test/measures_tests.jl pinning every enum (123 assertions).
Phase 329 — bundled Observatories table diffed exhaustively against casatools (no bug found)
Phase 218 had spot-checked 14 of the bundled Observatories entries. All 59 names casatools lists were now compared: the 53 present agree to under 1 cm. The 6 deliberately omitted are unusable even in casacore: VLBA/EVN are ITRF (0,0,0) placeholders, SUNRISE is a balloon at (0,0,743 m), and OVRO_MMA/LOFAR/NGVLA store geodetic-looking (lon, lat, height) values under the ITRF type, so me.observatory(name) returns a ~1 km radius from the Earth's centre. Documented in src/measures/observatories.jl (also correcting its stale "falls back to antenna 0" note — it is the middle antenna since Phase 144). New test pins the omissions and that every bundled position is on the Earth's surface.