Phase 137 — found mscal.uvw_j2000() computed the WRONG uvw entirely (antipodal, since Phase 79); fixed to match getNewUVW exactly
While investigating mscal.delay() for Phase 136, read MSCalEngine:: getNewUVW in full and found it does something completely different from what mscal.uvw_j2000() (Phase 79) had implemented since day one.
Real casacore recomputes uvw fresh from the antenna positions. getNewUVW rotates each antenna's ITRF baseline to J2000 via a pure MBaseline rotation, then constructs the uvw via MVuvw's OWN constructor (casa/Quanta/MVuvw.cc:83-91, xyz = R·pos with R = Rx(dir.lat-π/2)·Rz(-dir.lon-π/2)) — it never transforms the stored UVW column at all. This package's implementation did the opposite: rotate the stored UVW via MCuvw's toPole/fromPole (Phase 75, R = Ry(-π/2+lat)·Rz(-lon)) — a genuinely different rotation basis from MVuvw's own constructor. Confirmed by direct numeric comparison that these two matrices are related by R_mvuvw = Rz(-π/2)·R_mcuvw — not equal, not a simple transpose, a real structural difference.
Live comparison against the sample MS confirmed the consequence: the old implementation's output was the exact antipode (all three components negated) of what getNewUVW's real algorithm gives for the identical baseline and epoch. Tracing the root cause further: the sample MS's stored UVW column follows the ANTENNA1-ANTENNA2 sign convention (confirmed directly — its w component exactly equals dot(direction, ap1-ap2), matching mscal.delay()'s own already-verified formula), while NewMSSimulator's own source (ms/MSOper/NewMSSimulator.cc:1600,1625-1627, an explicit code comment plus the actual uvwvec(i) = x2[i]-x1[i] assignment) computes and stores the opposite ANTENNA2-ANTENNA1 convention. This is a real, longstanding split within casacore itself between real observed data and its own simulator's synthetic output — not a bug on either side of that split, and not something this port introduced.
The fix: since mscal.uvw_j2000() must match what real casacore's getNewUVW actually computes to be correct — and getNewUVW always uses ANTENNA2-ANTENNA1 via a fresh from-antenna-positions reconstruction, regardless of what convention the stored column happens to follow — the implementation now discards the stored UVW column entirely and recomputes it exactly the way casacore does. A new _mvuvw_construct helper (pure trigonometry, no SOFA call beyond the existing MBaseline rotation) ports MVuvw's constructor directly. The antenna-0 baseline origin getNewUVW uses per-antenna cancels exactly in the final ant2-ant1 difference by linearity (both the MBaseline rotation and MVuvw's construction are linear maps), so the whole per-antenna two-step collapses to one combined linear map, memoized per (field, TIME) — the same memoization trick the old (buggy) implementation already used, just applied to the correct formula. measconvert(::MuvW, ...) itself (Phase 75's general uvw frame-conversion machinery) is untouched by this fix — the bug was specific to mscal.uvw_j2000() using the wrong algorithm for the job, not a defect in the general conversion function itself.
8 new/updated tests in test/taql_mscal_tests.jl, including a dedicated regression test pinning the sign relationship between mscal.uvw_j2000()'s w component and mscal.delay()'s already- verified value, and a rewritten hand-computation in the main mscal testset matching getNewUVW's exact algorithm instead of the old (wrong) stored-UVW-rotation approach.
Phase 138 — found mscal.pa*() was missing a real mount-type check
While re-reading MSCalEngine.cc in full for Phase 137, found that MSCalEngine::getPA returns a hard 0.0 unless the relevant antenna's MOUNT starts with "alt-az" (case-insensitive) — an equatorially- or otherwise-mounted antenna, or the suffix-less array-centre form (which has no real antenna's MOUNT to consult at all — setData's mount stays its 0 default), has no well-defined parallactic angle in casacore's own model and the function simply returns 0 rather than computing a meaningless value.
This package's mscal.pa()/pa1()/pa2() had no mount check at all — it always computed the geometric parallactic angle regardless of antenna mount, and the bare mscal.pa() form would return a nonzero value it should never return. Not observable on the committed sample.ms fixture (every antenna's MOUNT is "ALT-AZ"), so verified with a synthetic patch setting MOUNT to "EQUATORIAL".
Fixed in src/taql/mscal.jl: reads ANTENNA.MOUNT once (defaulting to "every antenna is alt-az" if the column is absent, a graceful fallback), and the pa* branch returns 0.0 whenever the relevant antenna id is -1 (the suffix-less form) or that antenna's own mount isn't alt-az. 8 new tests in test/taql_mscal_tests.jl ("mscal.pa*() mount-type check").
Phase 139 — documented a real, deliberate divergence: mscal.* interpolates a moving-target FIELD direction, real casacore never does
Continuing the MSCalEngine.cc read-through from Phases 137-138, found MSCalEngine::fillFieldDir (the function that populates the per-field direction cache every mscal.* direction function ultimately reads) caches dirCol(i).data()[0] — the direction array cell's FIRST element — once per field, and reuses that exact same value for every row regardless of TIME. A grep across the entire file confirms NUM_POLY and EPHEMERIS_ID are never read anywhere in MSCalEngine.cc — real casacore's derivedmscal UDFs are completely unaware that a FIELD row can be a moving target at all.
This package's mscal.* functions do the opposite by design (Phases 82/93): they interpolate the polynomial or ephemeris-driven direction at each row's own TIME, giving a physically meaningful time-varying direction for a genuinely moving target. This is a deliberate, intentional improvement — not a bug to fix — but it does mean this package's mscal.* output for a moving-target field will not numerically match real casacore's UDFs for such a field (a real MS essentially never has one in practice; PHASE_DIR is overwhelmingly a fixed-position Dims column). Documented explicitly in src/taql/ mscal.jl, the query.jl docstring, and docs/src/concepts.md.
Extended the existing "mscal.* with an ephemeris FIELD" test (test/taql_mscal_tests.jl) with a direct check that the underlying ephemeris_direction interpolation genuinely varies across the ephemeris table's own time grid (the fixture's real MAIN rows span too few seconds for the ramp to show up row-to-row on that MS, so the grid's own wider span is used to exercise the machinery directly). No production behaviour changed — comment/doc-only, plus the one new test assertion.
Phase 140 — found mscal.*'s direction argument only accepted a hard-coded whitelist of 3 FIELD column names
Continuing the MSCalEngine.cc/UDFMSCal.cc read-through, read UDFMSCal's actual string-direction-argument dispatch (derivedmscal/DerivedMC/UDFMSCal.cc:288-308): real casacore tries the string as a solar-system body/frame name FIRST (MDirection::makeMDirection), and only if that fails does it fall back to itsEngine.setDirColName(str) — accepting any FIELD column name, not a fixed set.
This package's direction-argument resolver (_djfor, Phase 85) did the opposite: it checked a hard-coded whitelist of exactly three column names (PHASE_DIR, DELAY_DIR, REFERENCE_DIR) before trying a body/frame lookup, and any string outside that whitelist went straight to the body/frame branch, erroring "unknown direction" if it wasn't a recognized name. A real (if unusual) MS with some other custom FIELD direction column — anything other than those three exact names — would be unreadable via mscal.*'s direction argument, even though the underlying column-read machinery was already fully generic (it calls measure(fld, dir, ...) with whatever name was given).
Fixed in src/taql/mscal.jl: _djfor now tries a body/frame name first (matching casacore's actual precedence), then falls back to checking whether the string names any real column of the FIELD subtable (fieldcols = Set(columnnames(fld)), computed once) — not a fixed list. Removed the now-dead _MSCAL_DIR_COLS constant. New tests in test/taql_mscal_tests.jl covering a genuinely custom direction column, confirming body-name precedence is unaffected, and confirming a name that is neither a body nor a real column still errors clearly.
Phase 141 — investigated a real source divergence in the GEO/TOPO frequency/RV hop; the "obvious" fix empirically made agreement with real CASA worse
Read measures/Measures/MCFrequency.cc and MCRadialVelocity.cc in full to re-verify the GEO/BARY/TOPO/LSRK velocity-composition machinery built in Phases 66/71. Found a real, textual divergence: casacore's own GEO_TOPO/TOPO_GEO hop (the diurnal-aberration term) projects onto the frame's apparent direction (frameDirection(...).getApp(...)), while every other hop (LSRK_BARY/BARY_GEO/GEO_BARY) uses the plain J2000 direction (.getJ2000(...)). This package's _n_hat supplies one uniform J2000 direction to every hop, including the diurnal-aberration term — textually not what casacore's own source does.
Implemented the "obvious" fix — a separate apparent-direction vector threaded into just the TOPO↔GEO dot product — and tested it live against the real CASA oracle (measures_tests.jl's casatools cross-check). The result was the opposite of the expected improvement: the frequency residual grew from comfortably within the test's rtol=2e-9 tolerance to about 6e-9 (failing), and the GEO/TOPO radial-velocity residual grew from the already-documented ~0.2 m/s (Phase 71) to ~1.8 m/s — an order of magnitude larger than the small arcsec-level correction should plausibly produce, and enough to exceed the test's own atol=0.5 m/s tolerance.
Reverted the code change — the existing uniform-J2000 implementation demonstrably agrees with real CASA better than the textually more faithful apparent-direction port, likely because some other SOFA- vs-casacore residual in the apparent-place computation swamps the intended correction rather than the two cancelling as hoped. Left a detailed comment in ext/SOFAExt.jl recording the investigation and its negative result, so the same "fix" isn't re-attempted without re-testing against the live oracle. No production behaviour changed; no new tests (the existing CASA cross-check already caught the regression during development, which is exactly what caught this).
Phase 142 — found mscal.stokes()'s WEIGHT conversion was missing a real casacore quirk: any zero input poisons the whole output
Re-verified mscal.stokes's rescale-factor logic (Phase 78 — 0.5 for codes 5-12, √2/4 for codes 13-20) and its FLAG conversion against ms/MeasurementSets/StokesConverter.cc directly — both confirmed exact matches, no bug. The WEIGHT/SIGMA conversion (StokesConverter:: convert(Array<Float>&, ...), .cc:395-414) turned up a real, previously unported behaviour: casacore loops over every input correlation regardless of whether its conversion coefficient is zero (a harmless 0/x no-op when it is), but if any input's weight is exactly 0 — even one with no coefficient at all for the output in question — the entire output for that (output, channel) is forced to 0 and the loop stops (else { outMat(i,j)=0; break; }).
This package's implementation instead skipped a non-contributing or zero-weight correlation individually and computed a value from whatever nonzero terms remained — a real, meaningfully different result whenever any input correlation's weight is exactly 0, which is the ordinary convention for an invalid/flagged visibility in a real MS, not a rare edge case.
Fixed in src/taql/mscal.jl's _stokes_convert (the AbstractMatrix{ <:Real} / WEIGHT-shaped method) to iterate over every input correlation and zero the whole output on any zero input, exactly mirroring casacore's own loop. New unit tests plus a live cross-check against real casacore (test/taql_mscal_tests.jl, "mscal.stokes(WEIGHT) zero-poisoning vs real TaQL") confirmed the fix — the WEIGHT column naturally has no zero cells on the committed sample.ms fixture, so the cross-check patches a copy to introduce one.
Phase 143 — verified mscal.uvdist() (no bug) and confirmed a real casacore MS-mutation bug in mscal.corr()'s underlying grammar (not replicated here)
Re-verified two more mscal.* selection functions against their real casacore source, following Phases 80/81/83/84's original convention- and-live-TaQL-based build.
mscal.uvdist(): ms/MSSel/MSUvDistParse.cc has two code paths — a "slow" one (explicitly commented "here for testing — should ultimately be removed") using the full 3-D √(u²+v²+w²) uv-distance, and the actual default "fast" path (doSlow=false, ~60× faster per its own comment) using SQUARE(UVW[1]) + SQUARE(UVW[2]) — the 2-D projection only. This package's Phase 81 implementation already uses the 2-D form, matching the real, actually-used default path exactly. The wavelength-unit scaling (uvDist_lambda = uvDist_m · refFreq / c) also matches the slow path's own formula (the fast path scales the bound the other algebraic way, but the two are mathematically identical). No bug found.
mscal.corr(): reading MSCorrParse::selectCorrType (what msCorrGramParseCommand, hence mscal.corr(), actually calls) found the core selection logic matches this package's implementation exactly (DATA_DESC_ID IN the set of data-desc ids whose POLARIZATION. CORR_TYPE contains the requested code). But the real function also has a genuinely alarming, undocumented side effect along the way: it reopens the very MS being queried in writable mode and unconditionally adds (replacing any existing one) a SELECTED_DATA column, copying a slice of DATA into it — as a side effect of evaluating what should be a read-only WHERE-clause predicate. Using mscal.corr() (or a native WHERE CORR = 'RR' selection) against a real, writable MS in real casacore genuinely mutates that MS on disk. MSFeedParse.cc (the mscal.feed() counterpart) has no such pattern — this is specific to MSCorrParse.
This package's mscal.corr() is a pure, read-only, in-memory Bool computation with no such side effect — confirmed as the correct, deliberate choice, not a divergence to fix; replicating casacore's destructive behaviour would be a regression. Documented in a comment above mscal.corr's implementation (src/taql/mscal.jl) for anyone reading the source later. No production behaviour changed.
Phase 144 — found a real bug: mscal.*'s array centre was looked up per row via OBSERVATION_ID instead of once for the whole engine
Continuing the sweep of MSCalEngine.cc for real bugs (following Phases 136-143), re-read MSCalEngine::attachColumns's array-centre resolution in full. Real casacore computes the array centre used by every suffix-less mscal.ha() / azel() / pa() / itrf() / delay() once for the whole engine, from OBSERVATION row 0's TELESCOPE_NAME (falling back to a table-level TELESCOPE_NAME keyword, then to the middle antenna's position, itsAntPos[0][nant/2], 0-based) — never per MAIN row via that row's own OBSERVATION_ID.
This package (since Phase 90) looked the telescope up per row, indexed by OBSERVATION_ID into the OBSERVATION subtable — a real divergence for any MS with more than one OBSERVATION row (rare, but real — e.g. a concatenated/combined dataset). Live-verified against real tableCommand on a synthetic two-observation MS ("VLA" row 0, "ALMA" row 1, half the MAIN rows tagged OBSERVATION_ID=1): real casacore's mscal.ha() was bit-identical across the OBSERVATION_ID split (as expected — it never looks at the per-row id at all); this package's own output jumped by ~0.7 rad at the boundary before the fix. The same read also caught a second, harder-to-observe divergence in the no-telescope-found fallback: this package fell back to antenna 0, casacore falls back to the middle antenna — both fixed together.
Fixed in src/taql/mscal.jl: centrepos is now a single MPosition resolved once (OBSERVATION row 0 → the bundled Observatories table → a table keyword TELESCOPE_NAME → the middle antenna), not a Dict{Int,Any} keyed by OBSERVATION_ID. New testset test/taql_mscal_tests.jl "mscal.* array-centre: one lookup per engine, not per OBSERVATION_ID (Phase 144)" — the synthetic two-observation cross-check above, plus a same-value-across-the-split assertion on our own output. Full mscal suite green (510/510, standalone).
Phase 145 — redid a lost investigation: mscal.field()/mscal.state() vs FLAG_ROW is spec-form-dependent, not a blanket "never filters"
A Phase 144 investigation into whether mscal.field()/mscal.state() exclude FLAG_ROW-flagged rows was lost (never committed) before a context-summary cut. Redone from scratch, live-verified against real tableCommand, with a more complete result than the original: the original probes ('0', '0~0', '>=0' — a parse error, '<1' — also a parse error at the time) concluded "field/state selection never respects FLAG_ROW" and were about to revert a matching code change. Redoing it with a wider set of specs (and, critically, comparing a flagged fixture against an unflagged one for the same spec, rather than assuming a parse error meant "unsupported syntax") shows the real behaviour is spec-form-dependent:
- A bare id (
'0') or~-range ('0~0') spec does not exclude a flagged field/state — confirmed live, both return every row. Real casacore's grammar routes these throughMSFieldParse:: selectFieldIds(MSFieldParse.cc:68-79), which is a plaincolumnAsTEN_p.in(fieldIds)—FLAG_ROWis never consulted here. - A comparison spec (
'<N'/'>N') or a name/pattern spec ('3C286') does exclude it — confirmed live: the identical spec succeeds on an unflagged fixture and fails ("No field ID found" / "No match found for name") once the only matching field/state is flagged. These route throughMSFieldIndex::matchFieldIDLT/GT/GTAndLTandmatchFieldNameRegexOrPattern(MSFieldIndex.cc:103,224), which DO check!flagRow.MSStateIndex.cchas the identical structure (.cc:104,130) — inferred by symmetry, not independently re-tested.
So this package's mscal.field()/mscal.state() (which never filter by FLAG_ROW at all, for any spec form) match real casacore only for the bare id/range case — a genuine, confirmed divergence for comparison and name specs remains, now precisely characterised. Not fixed in this phase — documented as a comment above the field/state handling in src/taql/mscal.jl (Phase 145) for a scoped follow-up, since a correct fix needs spec-form-aware routing this function doesn't currently have. No production behaviour changed.
Phase 146 — fixed mscal.field()/mscal.state() to respect FLAG_ROW for comparison and name specs (per the Phase 145 finding)
Implements the Phase 145 finding: real casacore's FLAG_ROW exclusion for mscal.field()/mscal.state() is spec-form-dependent — a bare id ('0') or ~-range ('0~0') spec never checks FLAG_ROW (MSFieldParse::selectFieldIds), while a comparison ('<N'/'>N') or name/pattern spec does (MSFieldIndex's matchFieldIDLT/GT/GTAndLT/ matchFieldNameRegexOrPattern).
_mssel_resolve/_mssel_idset (src/taql/mscal.jl) gain an optional flagged vector (0-based-id-indexed, nothing = no filtering, default everywhere): a bare-id or ~-range term ignores it entirely; a comparison, regex, glob, or exact-name term intersects its match set with the unflagged ids. Only the field/state branches of _mssel_one now build and pass a real flagged vector (from that subtable's own FLAG_ROW column); baseline/spw/scan/array/ obs keep the default nothing (Phases 118-119 confirmed those never filter by FLAG_ROW in real casacore either).
Live-verified against real tableCommand on a FIELD-row-0-flagged fixture: mscal.field('0')/'0~0' are unaffected (still select the flagged field, matching real casacore exactly); mscal.field('<1')/ a name spec on the flagged field now correctly exclude it (real casacore instead raises a grammar error in this degenerate all-excluded case — an acceptable, pre-existing difference in error- handling style, not a semantic one — this package returns an empty result rather than erroring, consistent with how '>0' already behaves). New testset test/taql_mscal_tests.jl "mscal.field()/ mscal.state() FLAG_ROW: spec-form-dependent (Phase 146)". Full mscal suite green (521/521, standalone).
Phase 147 — fixed a real copyms gap (missing FLAG_CATEGORY CATEGORY keyword); found a severe, pre-existing crash bug in this environment's linked casacore for mscal.state()/scan()/array()/obs()
Started by investigating whether mscal.field()/mscal.spw() (which this package accepts >=/<= comparisons for, uniformly across every mscal.<sel>() function) actually support that syntax in real casacore. Reading MSFieldGram.yy/MSSpwGram.yy/MSStateGram.yy confirmed those three grammars declare GE/LE tokens but never reduce them in a production — live-verified against real tableCommand with a clean "Parse error at or near '=0'" for all three. This package's uniform acceptance of >=/<= for field/spw/state is therefore a benign, minor over-permissiveness (extra accepted syntax, still semantically correct) — not a bug, left as-is.
MSScanGram.yy/MSArrayGram.yy/MSObservationGram.yy, by contrast, DO have real GE INT/LE INT (and GE INT & LE INT) productions wired to selectRangeGEAndLE-style calls — so the natural next step was to confirm those three (and state, sharing the same UDFMSCal::setupSelection MeasurementSet ms(table) construction pattern) actually accept >=/<= live. That attempt uncovered two separate things:
A real, fixed bug: every one of
mscal.state()/scan()/array()/obs()threw "Missing CATEGORY keyword in FLAGCATEGORY column" against acopymsof the committed sample fixture — becauseUDFMSCal::setupSelectionconstructs a fullMeasurementSet ms (table)for these four (notfield/spw, which don't), andMeasurementSet's C++ constructor requires aCATEGORYkeyword on `FLAGCATEGORYthat it self-heals on a writable open but throws on read-only (the exact Phase 121 finding, previously fixed only forcreatems/addcolumn!, never forcopyms). The committed sample fixture's ownFLAGCATEGORYlacks it (real MSes commonly don't write it), andcopymsfaithfully carried the gap forward. Fixed:copytablecols(src/tables/create.jl) now stamps the same emptyflagcategorykw()whenever the source column lacksCATEGORY, leaving a source that already has a real value untouched. New test intest/writer_tests.jl`.A severe, pre-existing crash bug — NOT in this package, in the linked casacore build itself: once the
CATEGORYgap was fixed, invokingmscal.scan()(bare id'1', and separately'>=1') through realtableCommandsegfaults the whole Julia process (TableExprNodeBinary::getCommonTypes, reached viaMSScanParse::selectScanIds/selectScanIdsGTEQ→TableExprNode:: newEQ/newGE) — for essentially any spec, not just the comparison forms. A segfault cannot be caught bytry/catch, so this is a real operational hazard: no test in this suite has ever exercisedmscal.state()/scan()/array()/obs()against realtableCommand(confirmed by grep — every existingmscal.<sel>() vs real TaQLcross-check only triesbaseline/field/spw/corr/feed/uvdist), so nothing was silently crashing full-suite runs before now. Documented prominently intest/writer_tests.jlas a standing hazard — never call_taqlcmd/tableCommandwithmscal.state/scan/array/obsin this environment. Not a MeasurementSets.jl bug and not something this package can fix; no further live-testing of these four functions was attempted once the crash was confirmed, to avoid repeat crashes.
Full suite standalone: mscal (521/521), writer+edit+schema (219/219, +4 new).
Phase 148 — swept three more leads (ECLIPTIC frame, ICRS approximation, mscal.corr()'s selection condition); no confirmed bug, no code behaviour change
Continuing the sweep discipline, three leads investigated this phase:
ECLIPTICframe epoch-handling — re-checkedMDirection.h's frame documentation directly: casacore's plainECLIPTIC(as opposed toMECLIPTIC/TECLIPTIC) is explicitly listed among the handful of epoch-independent frames (fixed at the J2000 mean ecliptic/equinox, paired one-to-one withJ2000in the conversion table,MCDirection.cc:77-78) — confirmed this package'sECLIPTICimplementation (a fixed-epochSOFA.eceq06/eqec06call, no frame epoch needed) matches exactly. No bug.- ICRS≈J2000 approximation — re-confirmed as an already-documented, deliberate design choice (~0.02″), not a new finding.
mscal.corr()'s selection condition — re-readingms/MSSel/ MSCorrParse.ccmore closely than the Phase 143 pass found a real discrepancy with that phase's own "confirming the core selection logic here is correct" claim:selectCorrType'scorrtypeargument toMSPolarizationIndex::matchCorrTypeis the WHOLE, unfilteredPOLARIZATION.CORR_TYPEcolumn (flattened across every row), never actually narrowed to the requested correlation-type string's own code — which, permatchCorrType's own matching semantics (MSPolIndex.cc:105-138), looks like it would make the real selection non-selective for any MS with more than one distinct polarization setup (masked by the sample fixture's single setup). Not independently confirmed live:mscal.corr()/mscal.feed()are not registered as callable TaQL UDFs in this environment's casacore build at all (same limitation Phase 84 already documented). Per this project's standing discipline, an unverified source-reading hunch is recorded as an open question, not treated as an established bug — the correction is now insrc/taql/mscal.jl's comment above_parse_corr_types, alongside the original Phase 143 note. This package's ownmscal.corr()is unaffected either way (it correctly filters by the requested type regardless of what real casacore's condition actually computes).
No production behaviour changed; full mscal suite green (521/521, standalone, unchanged).
Phase 149 — found and fixed a real bug: update!'s SET list evaluated every item against one pre-update snapshot instead of applying items in order (real casacore does NOT swap SET A=B, B=A)
Investigated real casacore's TableParseUpdate/doUpdate after the Phase 30 plan's own claim that "all RHS are evaluated against the pre-update values (so SET A = B, B = A swaps)" had never actually been checked against real TaQL. It's wrong: reading TableParseQuery::doUpdate (tables/TaQL/TableParseQuery.cc:575-614) shows the real loop is row-outer, SET-item-inner — for (row) { for (item) { item->updateColumn(...) } } — and each updateColumn writes straight to the live, writable table column. Live- verified: UPDATE t SET A = B, B = A does not swap in real casacore — A takes B's old value first, then B's own item reads that already-updated A, so both columns end up equal to the old B.
This package's update! computed every SET item's RHS from one shared, frozen cols snapshot loaded before any writes — genuine "swap" semantics, a real, confirmed divergence for any SET list where one item's RHS references a column an earlier item in the same call also targets (the classic swap idiom, but also any multi-column SET in general once a later item happens to reference an earlier target).
Fixed in src/taql/commands.jl: update! now applies specs in their literal given order (dropped the previous "group by target column" step), and after every write — whole-column, per-row, sliced, or masked — immediately reflects the new value back into the live cols dict (and, for a sliced/masked write's full-precision continuation, a new curval cache) so a later item genuinely observes an earlier one's write, matching casacore exactly. Required materialising a SET target column into an owned, Any-typed Vector (previously an array-eltype column could be a lazy, non-setindex!-able Column view — safe since only actual write targets are touched, not every referenced column). The (D, M) => ... masked-array sugar (Phase 59) needed its own fix in tandem: its expansion now pushes the mask entry before the data entry, since the mask always re-evaluates the RHS expression from scratch and must see the pre-update data, not the data entry's own just-written result.
Docstring updated to state the real (non-swap) semantics; the existing hand-computed swap test corrected to the real expected values, and a same-string real-TaQL cross-check for SET A = B, B = A added (passes). Full suite green: commands 241/241, broad query/groupby/join 867/867, writer+edit+schema 219/219, mscal 521/521 (all standalone, no regressions).
Phase 150 — verified mscal.stokes()'s hardcoded conversion matrices are byte-exact against casacore's own construction (no bug)
Swept the six hardcoded 4×4 IQUV/circular/linear conversion matrices (_M_LIN_FROM_IQUV, _M_IQUV_FROM_LIN, _M_CIRC_FROM_IQUV, _M_IQUV_FROM_CIRC, and the composed lin↔circ pair) that mscal.stokes()'s _STOKES_BASE (Phase 78) has relied on since it was written — the Phase 78 plan asserted these are "standard" without ever independently re-deriving them from casacore's own construction.
Read ms/MeasurementSets/StokesConverter.cc::initConvMatrix directly: casacore builds its IQUV→linear matrix from a literal Slin[4][4], and IQUV→circular as Scirc = kron(h, conj(h)) · Slinear where h = (1/√2)·[[1,i],[1,-i]] (a Kronecker product via directProduct, whose exact operand order — kron(A,B) vs kron(B,A) — isn't stated in the header and had to be resolved empirically). Independently computed Slin and both candidate Kronecker orderings in a scratch Julia session (not the package under test) and compared:
_M_LIN_FROM_IQUVmatches casacore's literalSlinarray exactly, element for element.kron(h, conj(h)) · Slinear(the correct operand order, confirmed by matching) is byte-identical to_M_CIRC_FROM_IQUV._M_IQUV_FROM_CIRC/_M_IQUV_FROM_LINare exactlyinv(Mcirc)/inv(Mlin)computed independently.- The two composed matrices (
_m4mul(_M_CIRC_FROM_IQUV, _M_IQUV_FROM_LIN)forlin→circ, and the mirror forcirc→lin) match casacore's owntmp = Scirc; tmp *= Slinear.inverse()/tmp = Slinear; tmp *= Scirc.inverse()composition order exactly — confirmed the(from, to)key convention in_STOKES_BASE's own comment (to_vec = M * from_vec) lines up with which matrix is multiplied by which inverse.
Also confirmed, while re-reading RefTable::root()/BaseTable::root() for a different question (whether write_reftable's Phase 132 _flatten_to_root should also flatten through a ConcatTable parent): ConcatTable does not override root(), so real casacore's own RefTable constructor (btp->root()) stops at a ConcatTable parent exactly like this package's _flatten_to_root already does (only RefTable parents recurse) — confirmed correct, no change needed.
Every one of the six Stokes matrices this package hardcodes is a genuine, byte-exact match to real casacore's derivation — no bug found. This is a stronger result than Phase 78's original "standard, can be verified independently" claim, which was never actually checked against the source until now. No production code changed. Standalone mscal suite green (521/521, unchanged).
Phase 151 — found a real small gap in mscal.spw's channel-frequency units (missing THz); confirmed real casacore's own velocity-unit channel selection is disabled too
Investigated whether ms.msselect() (real casacore's MSSelection C++ library exposed via casatools, called directly rather than through the crash-prone derivedmscal/TaQL UDFMSCal wrapper — Phase 147's segfault hazard) could safely give a live oracle for mscal.scan/ array/obs's GE/LE grammar forms, since those functions can never be called via tableCommand at all. Confirmed msselectedindices()'s 'scan' key returns the interpreted range bounds of the spec, not the actual matched row/scan values against the table's data — an unreliable oracle for this purpose, not pursued further (no working independent test path found for mscal.scan/array/obs/state's GE/LE forms in this environment).
Redirected to mscal.spw's channel-frequency-range unit table (_CHAN_FREQ_UNIT, Phase 83), which had never been checked against casacore's own unit grammar. Read ms/MSSel/MSSpwGram.{ll,yy} and MSSpwIndex::convertToMKS directly: the MKS conversion factors there (k→1e3, m→1e6, g→1e9, t→1e12) match this package's hz/khz/mhz/ghz exactly — no bug — but real casacore's grammar also lexes a t<hz> (THz) prefix that this package was missing, a real small gap, now fixed (_CHAN_FREQ_UNIT["thz"] = 1e12). Separately confirmed something reassuring while reading the same grammar: real casacore's own velocity-unit (km/s, m/s) channel selection unconditionally throws ("Velocity units support temporarily disabled") the moment the parser reduces one — this package's own long-standing "velocity units on a chan range" non-goal was never actually a divergence from upstream, since upstream doesn't support it either.
New unit tests for _parse_chan_elem's THz form, a mscal.spw/ mscal.chan query test confirming a THz-unit range spanning the same window as an existing GHz test gives an identical result, and a new entry in the real-TaQL cross-check list (mscal.spw('0:0.0079~0.0081thz'), matching the 7.9–8.1 GHz window of an existing GHz spec) — passes. Standalone mscal suite green (666/666).
Phase 152 — investigated mscal.feed() against MSFeedIndex.cc; found an unconfirmed divergence (feed-id validation against the FEED subtable), documented rather than implemented
Read ms/MSSel/MSFeedGram.{ll,yy} + MSFeedParse.cc/MSFeedIndex.cc (what real mscal.feed() calls, UDFMSCal.cc:528-550) directly. The &/&&/&&&/;-negation grammar and its setTEN accumulator turned out to be byte-identical to MSAntennaParse::setTEN (the baseline grammar already ported in Phases 80/115/119/120) — confirmed no bug, since mscal.feed() already reuses the same _mssel_baseline_pred machinery unchanged. The ~ range separator (lexed as a token named DASH but mapped to the literal "~" character — a misleading legacy name inherited from the antenna grammar, not an actual -) also matches what this package already accepts — an initial reading of the token name looked like a real divergence from mscal.baseline's own ~ ranges until the lexer rule itself was checked.
One real but UNCONFIRMED divergence found: MSFeedIndex::matchFeedId intersects the requested feed-id set against the FEED subtable's own FEED_ID column values and THROWS ("No match found for requested feeds") if the intersection is empty — i.e. real casacore validates a requested feed id against the subtable's actual content, not just against what appears in MAIN's FEED1/FEED2. This package's own mscal.feed() derives its valid id range purely from max(FEED1, FEED2) in MAIN and never consults the FEED subtable at all — an in-range-but-never-assigned feed id (a gap in FEED_ID) would silently read as "matches nothing" here, where real casacore raises an error. Attempted to live-verify via _taqlcmd and confirmed mscal.feed() is not registered as a callable TaQL UDF in this environment's casacore build at all ("TaQL function mscal.feed (=derivedmscal.feed) is unknown") — the identical gap Phase 84's own writeup already found for mscal.corr()/mscal.feed() both. Per this project's own standing discipline (mscal.corr()'s own comment, Phase 148), an unverified source-reading finding is recorded as an open question in src/taql/mscal.jl, not implemented as a behaviour change with no way to test it. No production behaviour changed; standalone mscal suite green (666/666, unchanged).
Phase 153 — upgraded mscal.state() vs FLAG_ROW's "inferred by symmetry" note to a confirmed source-read match (no bug)
Phases 145/146 found mscal.field()'s FLAG_ROW filtering is spec-form-dependent (a bare id/~-range spec never filters, a comparison/name spec does) and applied the same fix to mscal.state() "by symmetry" — MSStateIndex.cc was never actually read directly, since mscal.state() itself can't be live-tested at all (Phase 147's crash bug). This phase closes that gap: read MSStateGram.yy + MSStateParse.cc + MSStateIndex.cc directly. Confirmed byte-for-byte the same structure as MSFieldParse/MSFieldIndex: the grammar's bare-id/~-range production (stateidrange, MSStateGram.yy:194-210) builds a raw id list with no index-table lookup at all, which MSStateParse::selectStateIds (MSStateParse.cc:65-73) turns into a plain TEN.in(stateIds) — no FLAG_ROW check; the </>/<>&<> bound forms (stateidbounds, .yy:214-243) and the OBS_MODE name/regex/pattern form both route through MSStateIndex:: matchStateIDLT/GT/GTAndLT (MSStateIndex.cc:216-251) / matchStateObsModeRegexOrPattern (.cc:68-104), each of which builds its selection mask as ... && !flagRow().getColumn(). This package's existing _mssel_idset's flagged kwarg (already applied to both field and state, Phase 146) implements exactly this per-term-form split — confirmed correct via direct source reading, not just symmetry with a sibling function. No production behaviour changed (comment-only — replaced the "inferred by symmetry, not independently tested" hedge with the confirmed citations); standalone mscal suite green (666/666, unchanged).
Phase 154 — fixed a mis-citation for baseline/spw/scan/array/obs's "never filter by FLAG_ROW" claim; independently re-confirmed it via direct source reading
Investigated mscal.array()/mscal.obs() (can't be live-tested, Phase 147's crash bug) by reading MSArrayParse.cc/ MSObservationParse.cc directly: both build their WHERE condition from a plain comparison against columnAsTEN_p (MAIN's own ARRAY_ID/OBSERVATION_ID column) with no subtable Index class and no FLAG_ROW column anywhere in the code path at all — genuinely cannot filter by FLAG_ROW, structurally, not merely "doesn't happen to". MSScanParse.cc is byte-for-byte the same shape.
While re-deriving this, noticed the existing comment above _mssel_notflagged (src/taql/mscal.jl, introduced in Phase 146 per git log -S) attributes this "never filters" claim to "Phases 118-119" — checked, and that citation is wrong: Phase 118 was mscal.baseline()'s antenna diameter/mount investigation and Phase 119 was its BLREGEX grammar support; neither touched FLAG_ROW at all. The comment's technical content was already correct (it also independently notes the check is "commented out in MSAntennaIndex.cc/MSSpwIndex.cc") — only the phase attribution was wrong. Independently re-verified that technical claim directly rather than trusting the old comment: MSAntennaIndex:: matchAntennaRegexOrPattern (baseline name/glob matching) and MSSpwIndex.cc's equivalent both have their !flagRow() term DELIBERATELY commented out of the active mask expression — e.g. MSAntennaIndex.cc: maskArray(i) = ((ret>0) != negate); //&& !msAntennaCols_p.flagRow().getColumn()(i)); — the check was written and then disabled, not simply never implemented.
Fixed both citations in src/taql/mscal.jl to point at this investigation instead, with the specific evidence (the commented-out mask term for baseline/spw, and the no-Index-class structure for scan/array/obs) rather than a phase-number pointer alone. No production behaviour changed; standalone mscal suite green (666/666, unchanged).
Phase 155 — added a real CASA cross-check for meas.wgs()/meas.itrfxyz()'s geodetic transform; found and fixed a misleading comment about MPosition{WGS84}'s own semantics
Investigated whether meas.wgs()/meas.itrfxyz() (Phase 106's geodetic ↔ Cartesian ellipsoidal transform, _geodetic_to_itrf/ _itrf_to_geodetic in ext/SOFAExt.jl) had ever been checked against a real oracle — it hadn't, only self-round-trip. Live-verified against casatools: me.position('WGS84', lon, lat, height) → me.measure(..., 'ITRF') for a VLA-like site (-107.6°, 34.0°, 2124 m) matches this package's _geodetic_to_itrf to sub-micrometre precision (both use the identical WGS84 ellipsoid constants — a=6378137 m, 1/f=298.257223563, confirmed identical to SOFA.eform(:WGS84)). Added this as a permanent cross-check (test/measures_fixture.py + test/measures_tests.jl), not just a scratch script.
While setting this up, this same live test also proved something else: the code comment above _mconv(::MPosition, ...) (ext/SOFAExt.jl) claimed "casacore stores the SAME geocentric Cartesian vector under both [ITRF and WGS84] refs" as the justification for implementing that conversion as an identity on (x,y,z) — that claim is false. Real casacore's MPosition::WGS84 genuinely represents a geodetic (longitude, latitude, height) position, and MCPosition.cc's ITRF_WGS84/WGS84_ITRF cases perform a real ellipsoidal transform (confirmed in source: they use MeasTable::WGS84(0)/(1), the ellipsoid semi-major axis and inverse flattening, in a Bowring-style iteration) — exactly the live-verified behaviour above, definitely not a passthrough. This package's own MPosition{R} is, by its own docstring, ALWAYS geocentric Cartesian metres regardless of R — so the identity behaviour for measconvert(::MPosition{WGS84}, ITRF) is still the correct thing for THIS package to do (no real MS POSITION column ever uses MEASINFO Ref="WGS84", so nothing in this project actually depends on the real geodetic semantics through that path) — but the comment's stated REASON was wrong, and could mislead a future investigator into thinking this mirrors real casacore. Fixed the comment in ext/SOFAExt.jl and the MPosition docstring (src/measures/types.jl) to state the real, verified fact (casacore's conversion is a genuine ellipsoidal transform) and the actual reason this package diverges (a deliberate, documented scoping choice, not an accurate port), pointing to meas.wgs()/meas.itrfxyz() as the real geodetic transform. Also corrected a test comment (test/taql_query_tests.jl) that called the identity "a Cartesian identity" without noting it's this package's own convention, not real casacore's. No production behaviour changed (comment/docstring fixes + one new permanent test); standalone measures suite green (486/486, including the new geodetic cross-check), query suite green (1008/1008).
Phase 156 — found and fixed a real bug: BitFlagsEngine's ReadMaskKeys/WriteMaskKeys crashed instead of silently skipping a key missing from FLAGSETS
Re-verified mscal.stokes()'s Ptotal/Plinear/Pangle/PFtotal/PFlinear formulas once more, this time against the actual VALUE-computation code in StokesConverter::convert(Array<Complex>&, ...) (not just the weight/flag setup code checked in Phase 122) — confirmed byte-for-byte match to the already-fixed implementation (sqrt(|Q|²+|U|²+|V|²) for Ptotal, /abs(I) for the PF* fraction forms, atan2(Re(U),Re(Q))/2 for Pangle using real parts only) — no third bug, Phase 122's fix was complete. Also independently re-derived AiryBeam's annular-aperture voltage response formula (_airy_voltage) against the standard Born & Wolf obstructed-aperture diffraction formula (A(x) = [2J₁(x)/x − ε²·2J₁(εx)/(εx)] / (1−ε²)) — an exact match, confirming what Phase 99 had only checked as "textbook optics, not cross-checked" — no bug.
The real find: reading BitFlagsEngine.cc's BFEngineMask::makeMask (the function that recomputes a ReadMaskKeys/WriteMaskKeys-based mask from the stored column's FLAGSETS record, Phase 40) shows it silently SKIPS any requested key not isDefined in FLAGSETS — for (key : itsMaskKeys) if (rec.isDefined(key)) mask |= rec.asuInt(key); — and ends up with mask == 0 (not an error) if NONE of the requested keys are found. This package's own _bfe_mask (src/datamanagers/virtual.jl) instead did reduce(|, UInt32(fs[k]) for k in ks; ...) — a plain Record index (fs[k]) that throws a KeyError for a missing key — so reading a BitFlagsEngine column whose ReadMaskKeys/WriteMaskKeys named even ONE flag category not present in that particular table's FLAGSETS would CRASH instead of just ignoring it, exactly the scenario real casacore handles gracefully (a table where only some flag categories are defined, or a ReadMaskKeys list that's broader than what one particular MS actually stamped). Fixed: for k in ks if haskey(fs, k) — skip an absent key instead of indexing it. Two new regression tests (a key partially missing → the found key(s) alone form the mask; every key missing → mask 0, matching casacore's uInt mask = 0; default, read back as all-unset rather than raising), both cross-checked against real Casacore.jl (auto-registers BitFlagsEngine<Int>) and agreeing exactly with this package's own reader. Full standalone engine suite green (255/255, was 253 before the two new tests).