GitHub

← older entries

Phase 90 — bundled Observatories table

observatory("VLA")     # -> MPosition{ITRF}(...)  (case-insensitive)
query(main, "mscal.ha() > 0")   # array centre from OBSERVATION.TELESCOPE_NAME
  • New src/measures/observatories.jl — observatory(name) returns the ITRF position of a known telescope (~50 entries: VLA / EVLA / ALMA / ACA / APEX / ATCA / GBT / WSRT / GMRT / LOFAR / MWA / ASKAP / MeerKAT / SKA-MID / SKA-LOW / Effelsberg / NOEMA / SMA / JCMT / Arecibo / FAST / …). A bundled snapshot of casacore's geodetic/Observatories data table, converted to ITRF Cartesian; station-array placeholders (VLBA / EVN, position 0,0,0) omitted.
  • The suffix-less mscal.ha() / mscal.hadec() / mscal.azel() / mscal.pa() / mscal.itrf() / mscal.delay() now use the real array centre — OBSERVATION.TELESCOPE_NAME (per row via OBSERVATION_ID) looked up in the table — instead of antenna 0. A lookup miss falls back to antenna 0 with a one-time warning.
  • observatory exported.
  • New test/aqua_tests.jl runs Aqua.test_all (ambiguities, undefined exports, stale deps, compat bounds, unbound type parameters, project extras, persistent tasks — all clean). Base.delete! / Base.insert! (the taql-style write commands, Phases 30-31, which accept a path string) are whitelisted for the type-piracy check.
  • Filled in the missing [compat] bounds Aqua flagged — Dates, Mmap, Random (stdlibs), and the test-only Aqua / Casacore / CxxWrap / Test extras. Every dependency now carries a bound.
  • README gains an Aqua badge. The CI workflow (Phase 67) already runs the full test suite, so Aqua runs on every PR.

measconvert for MFrequency / MRadialVelocity now handles the two remaining casacore velocity frames:

  • LGROUP — the Local Group barycentre (casacore MeasTable::calcVelocityLGROUP, 308 km/s).
  • CMB — the cosmic-microwave-background rest frame (the dipole, F. Ghigo's 369.5 km/s toward the galactic-coordinate direction).

VEL_LGROUP / VEL_CMB added to src/constants.jl; the LGROUP_BARY / CMB_BARY routes in ext/SOFAExt.jl are the same Doppler-shift form as LSRK↔BARY. Round-trip verified; the constants are copied verbatim from MeasTable.cc (no casatools oracle for these two).

Also: corrected the stale "RefOff is parsed-and-carried" comment — it is ignored (no standard MS column carries one).

query(main, "PHASE_DIR[1] > 10h30m AND PHASE_DIR[2] BETWEEN 40d AND 50d")
  • The tokenizer recognises a <number><unit> run whose unit is a sexagesimal token (h / h30m / h30m15s → hour angle, d / d51m / d51m16 → degrees) and emits a :num token with the value in radians (via the Phase-86 _parse_sexagesimal). _sexagesimal_unit does the classification.
  • A unit that isn't a sexagesimal token (30deg, 1.4GHz, 10m) stays a Phase-69 quantity literal — no regression.
  • 10h now means "10 hours of hour angle = 150°", not a 10-hour duration (matches casacore TaQL); use arithmetic on seconds for a duration.
query(main, "mscal.el1('10h42m31.3, 45d51m16')")     # a J2000 direction
query(t, "abs(RA - angle('10h30m')) < 0.01")         # angle() in any expression
  • _parse_sexagesimal(s, kind) (src/taql/functions.jl) — kind ∈ :ra (h/m/s time, ×15 → degrees) / :dec (d/m/s degrees). Accepts 10h42m31.3s, 10:42:31.3, 10 42 31.3, a leading sign, or a bare decimal (degrees). Returns radians.
  • A new angle('...') TaQL-lite function — sexagesimal string → radians (h in the string ⇒ hour angle).
  • mscal.* direction argument (Phase 85) now also accepts a comma-separated sexagesimal 'RA, DEC' string, in addition to the [ra, dec] radian pair.
  • Non-goal: a bare sexagesimal literal in the grammar (WHERE RA > 10h30m) — needs a tokenizer + node change; angle('10h30m') covers it.

The Phase-77 direction functions (ha / hadec / azel / az / el / pa / itrf / delay) take an optional direction argument instead of FIELD.PHASE_DIR, mirroring casacore's derivedmscal help text.

query(main, "mscal.el1('SUN') > 0.35")               # elevation of the Sun
query(main; select = ["az" => "mscal.az1('DELAY_DIR')"])
query(main, "mscal.hadec1([2.0, 0.5])")              # a fixed J2000 direction
  • The argument is a solar-system body name ('SUN' … 'NEPTUNE', 'MOON'), a FIELD direction column ('PHASE_DIR' / 'DELAY_DIR' / 'REFERENCE_DIR' — ephemeris-aware via Phase 82), or a [ra, dec] J2000 pair in radians. No argument → FIELD.PHASE_DIR (unchanged).
  • mscal.last (sidereal time) and mscal.uvw_j2000 are direction-intrinsic and stay 0-argument.
  • TQLMScal gains a dir field; the sentinel is "mscal.<fn>::<dir>". _mscal_columns resolves the per-row J2000 direction (_djfor) and keys its frame-conversion memo by (antenna, direction, TIME).

Phase 84 — mscal.corr() / mscal.feed() selection

The two remaining derivedmscal selection UDFs, completing the mscal.* selection set.

query(main, "mscal.corr('RR,LL')")           # rows whose pol setup has RR or LL
query(main, "mscal.feed('0 & 1') AND mscal.baseline('DA*')")
  • mscal.corr('spec') — a comma-list of correlation names (RR / XX / I / … via the Phase-78 _STOKES_NAMES) or integer Stokes codes. A per-row Bool: true if the row's polarization setup (POLARIZATION.CORR_TYPE via DATA_DESCRIPTION.POLARIZATION_ID) shares any code with the request.
  • mscal.feed('spec') — the mscal.baseline antenna-grammar form on FEED1 / FEED2 (L & R feed-pair, L && R, !, comma-lists of ids / N~M ranges), with numeric feed ids only.
  • Both in src/taql/mscal.jl — _parse_corr_types, _mssel_one corr / feed branches (feed reuses _mssel_baseline_pred).
  • mscal.corr / mscal.feed are not registerUDF-registered in the casacore build here (only advertised in the help text), so the real-derivedmscal cross-check skips them cleanly.

mscal.spw('spec') now takes the MSSelection spwid:chanlist form, and a companion mscal.chan('spec') returns the per-row channel mask.

query(main, "mscal.spw('0:5~20')")                    # spw 0, chans 5-20 non-empty
query(main, "any(mscal.chan('0:100~200MHz'))")        # rows with a channel in the band
query(main; select = ["m" => "mscal.chan('0:5~20;40~50')"])   # per-row BitVector
  • spec is a comma-list of <spwterm>[:<chanlist>]. <spwterm> is a single MSSelection-lite term (N / N~M / >N / <N / a name / glob / /regex/ against SPECTRAL_WINDOW.NAME / *). <chanlist> is a ;-list of a (single 0-based index), a~b (inclusive range), a~b^step, f1~f2GHz (a CHAN_FREQ range, Hz/kHz/MHz/GHz), or <f / >f.
  • mscal.spw stays a per-row Bool — a channelled spw matches only if at least one of the row's channels is selected (so 0:100~200 on a 64-channel spw excludes the row). mscal.chan returns a per-row BitVector of the spw's channel count (all-false for an unselected spw), for any(...) / count(...) / a Phase-60 masked array.
  • All in src/taql/mscal.jl — _parse_spw_spec / _parse_chan_elem / _chan_mask; _mssel_one's spw branch gains the channelled path, new chan branch. Threading unchanged.
  • mscal.spw('0:5~20') row count cross-checked against real derivedmscal.
  • Non-goals: velocity units on a channel range (casacore disables them too); ^step on a frequency range; per-DATA_DESC_ID distinct channel geometry within one spw.

Phase 82 — solar-system ephemeris (MeasComet) tables

A moving-target FIELD row — a non-negative EPHEMERIS_ID with a matching EPHEM<id>_*.tab polynomial position table — now gets its time-dependent direction from that table, closing the Phase-76 plan94-accuracy gap for comets and planets.

e = field_ephemeris(subtable(ms, "FIELD"), 0)          # nothing if not a moving target
d = ephemeris_direction(e, 60454.4)                    # MDirection at that MJD
measure(fld, "PHASE_DIR", 1; epoch = MEpoch{UTC}(60454.4))
query(main, "mscal.hadec1()")                          # uses the ephemeris automatically
  • New src/measures/ephemeris.jl: Ephemeris struct + open_ephemeris (reads the MJD0 / dMJD / NAME / posrefsys keywords + MJD / RA / DEC / Rho / RadVel columns), ephemeris_direction / _radvel / _distance (linear interpolation of the (ρ,RA,Dec) Cartesian vector between bracketing rows — casacore's MeasComet::get), and field_ephemeris(fld, field_id) (globs EPHEM<id>_*.tab in the FIELD directory).
  • measure(t, col, row; epoch) — new kwarg; for a direction cell of a FIELD table with an ephemeris it returns the ephemeris position (shifted by the stored PHASE_DIR offset).
  • _mscal_columns (mscal.*) _fielddir — an ephemeris field is evaluated per (field, TIME) (converted to TDB), then to J2000.
  • write_table gains a keywords::AbstractDict kwarg for scalar / string-array table-level keywords (used to build ephemeris fixtures).
  • Verified by an open_ephemeris / ephemeris_direction round-trip and an end-to-end mscal.hadec1() on a copied MS with a patched moving-target FIELD. No casatools oracle (needs a real ephemeris table + me.framecomet — documented gap).
  • Non-goals: polynomial (numpoly > 0) PHASE_DIR interpolation; the DiskLong / DiskLat sub-Earth point; the aipsrc measures.comet.directory lookup; MeasComet as a MeasFrame for a COMET-coded direction column outside FIELD.

Phase 103 — mscal.pbresponse per-baseline + elliptical/squint specs

query(main, "mscal.pbresponsebl('gaussian:0.008727') > 0.5")     # both antennas tracking
update!(ms; set = ["DATA" =>
    "mscal.pbcorrbl(DATA, 'ellipse:0.012:0.008:0.3:squint:0.0005:0.0')"])
  • mscal.pbresponse's beam-spec mini-language gains an "ellipse:HMAJ: HMIN:PA" form (EllipticalGaussianBeam) and an optional trailing ":squint:DLON:DLAT" on any spec (wraps the base beam in a SquintBeam) — _pb_response_fn (src/taql/mscal.jl) now returns an offset::(dlon,dlat) -> power closure uniformly; a circular beam (gaussian/airy) ignores the offset direction via power_response's existing generic 2-D-offset fallback (Phase 100), so the rewrite is a no-op for the Phase 101/102 forms.
  • The main-loop dispatch now computes the real 2-D tangent-plane offset (pointing_offset, Phase 100) of the nominal direction from the antenna's actual POINTING.DIRECTION, instead of the Phase 101 great-circle-magnitude-only _tql_angdist — required for a direction- aware ellipse/squint beam, and value-identical to the old scalar path for a circular beam (same magnitude, direction discarded downstream).
  • New mscal.pbresponsebl('spec' [, dir]) — the same beam evaluated at both ANTENNA1's and ANTENNA2's own actual pointing (needs ANTENNA2) and multiplied — the joint baseline response, for e.g. query(main, "mscal.pbresponsebl(...) > threshold") to cut baselines where either antenna has drifted off source. mscal.pbcorrbl / mscal.pbattenbl are the write-path sugar (valexpr / mscal. pbresponsebl(...) / valexpr * mscal.pbresponsebl(...)), mirroring Phase 102's pbcorr/pbatten.
  • Verified via controlled synthetic POINTING fixtures (the Phase 101/102 pattern — no CASA/casacore oracle exists for a MeasurementSets-only extension): both antennas of one baseline given independently known pointing offsets, the per-baseline response checked against the hand-computed product of the two Gaussian responses; the ellipse/squint results checked against power_response/SquintBeam called directly on an independently recomputed pointing_offset; pbcorrbl∘pbattenbl round-trips a known DATA value; a squint exactly onto the source gives response ≈ 1 regardless of the base beam's own off-axis response there.
  • Non-goals: a PolynomialBeam spec form; a per-baseline ellipse/squint variant beyond what mscal.pbresponsebl('ellipse:...') already gives (each antenna's own response, still direction-aware); an Observatories-array-centre (suffix-less) per-baseline response (baseline responses are inherently per-antenna-pair).

Phase 104 — complete the meas.* TaQL UDF subset

query(main, "meas.freq('TOPO', 'LSRK', 1.4e9, TIME/86400, X, Y, Z, RA, DEC) > 1.399e9")
query(main, "meas.doppler('RADIO', 'BETA', 0.01) > 0.009")          # no SOFA needed
query(fld, "meas.riseset(RA, DEC, TIME/86400, X, Y, Z)[1] < TIME/86400")
  • meas.freq('SSCALE', 'TSCALE', freq, mjd, x, y, z, ra, dec) / meas.rv('SSCALE', 'TSCALE', v, mjd, x, y, z, ra, dec) — frequency / radial-velocity frame conversion (topo/geo/bary/lsrk/lsrd/ galacto/lgroup/cmb), built on the existing MFrequency/ MRadialVelocity measconvert machinery (Phases 66/71/88) via a new _meas_full_frame(mjd,x,y,z,ra,dec) helper (epoch + ITRF position + J2000 source direction — the full frame a spectral conversion needs).
  • meas.doppler('SCONV', 'TCONV', value) — Doppler-convention algebra (radio/optical(z)/ratio/beta(true,relativistic)/gamma), a thin wrapper over the Phase 72 MDoppler measconvert — pure arithmetic, the only new meas.* function that needs no import SOFA.
  • meas.riseset(ra, dec, mjd, x, y, z[, elev0]) → [rise_mjd, set_mjd] — the rise/set UTC MJD of a J2000 direction for the day containing mjd, from the standard hour-angle-at-elevation formula (cos H₀ = (sin elev₀ − sin φ·sin δ) / (cos φ·cos δ), apparent place at local noon) plus a Newton inversion of the sidereal-time relation (_mjd_for_lst, 3 iterations against the real SOFA gst06a — the mean sidereal rate makes LST close enough to linear in UT1 over a day that this converges to sub-second precision). (NaN, NaN) if the source never reaches elev0 that day; (⌊mjd⌋, ⌊mjd⌋+1) if circumpolar. New core stub _riseset (src/measures/types.jl, mirrors _lst's stub/ext split) + the real implementation in ext/SOFAExt.jl. Not a byte-exact port of casacore's own iterative Rise/Set search — an independently-derived, documented approximation (no casacore/CASA oracle for a MeasurementSets-only convenience wrapper).
  • _meas_two_scale_args factors the "first two arguments are string literal frame/convention names" validation shared by meas.freq/ meas.rv/meas.doppler.
  • Verified: meas.freq/meas.rv cross-checked directly against measconvert on an equivalent MeasFrame; meas.doppler against measconvert(MDoppler{...}, ...); meas.riseset checked for rise-before-set + positive elevation at the rise/set midpoint (for a source below the horizon at the UTC-day boundary), a tighter elevation cutoff narrowing the window, and the NaN/circumpolar sentinel paths.
  • This closes the last item in Phase 97's meas.* non-goals list (meas.riseset, meas.freq/meas.doppler/meas.rv); the column-MEASINFO-driven direction-argument form and meas.pos/meas.itrfxyz/meas.wgs position UDFs remain non-goals.

Phase 105 — mscal.riseset(): automatic rise/set for the row's own antenna

query(main, "mscal.riseset1()[1] < TIME/86400")             # already risen
groupby(main, "FIELD_ID"; select = ["s" => "gmean(mscal.riseset1(0.2)[2])"])
  • mscal.riseset[1|2]([elev0][, dir]) → [rise_mjd, set_mjd] wires the Phase 104 meas.riseset / _riseset machinery into the automatic per-row mscal.* geometry — ANTENNA1's/ANTENNA2's own ITRF position (1/2 suffix; bare = array centre, same convention as every other mscal.* direction function) and dir (default FIELD.PHASE_DIR, same optional-direction-argument grammar as mscal.el1('SUN') etc.), for the UTC day containing the row's TIME. elev0 (rad, default 0) is a numeric literal baked into the function's parsed key (like mscal.pbresponse's beam spec), not a per-row expression.
  • Memoized per (antenna-or-centre, direction, UTC day) rather than per exact TIME — rise/set only changes once a day, so this is effectively free even over a MAIN table with many integrations per day.
  • Bug found and fixed during implementation (only visible once a query used two different elev0 values in one call): the memo key didn't include elev0, so mscal.riseset1(0.2) silently returned the elev0=0.0 result whenever both were evaluated in the same _mscal_columns call. Fixed by keying the memo on (antenna, direction, day, elev0).
  • No casacore/CASA oracle (mscal.* extension over Phase 104's own meas.riseset, itself independently derived). Verified: parser unit tests for the encoded key + dir/elev0 threading; on the sample MS, both rise and set are finite for a real antenna/field/day, the 2 suffix gives ANTENNA2's own (slightly different) window, and a tighter elevation cutoff never widens the window (rise2 >= rise, set2 <= set at every sampled row) — this is exactly the assertion that caught the memo-key bug above.

Phase 106 — meas.pos() / meas.itrfxyz() / meas.wgs(): position UDFs

query(ant, "meas.wgs(X, Y, Z)[3] > 2000")                     # height > 2 km
query(cat, "meas.itrfxyz(LON, LAT, 0.0) == meas.pos('WGS84', 'ITRF', X, Y, Z)")
  • meas.pos('SSCALE', 'TSCALE', x, y, z) — MPosition frame conversion (itrf/wgs84). This closed a real, previously-undiscovered gap: MPosition had no measconvert method at all before this phase — measconvert(::MPosition, ...) always hit the generic core stub's "needs SOFA.jl" error, even with SOFA loaded, since no _mconv method existed for it. The fix (ext/SOFAExt.jl) is an identity on (x,y,z): casacore stores the same geocentric Cartesian vector under both ITRF and WGS84 — the refs only differ in which ellipsoid a geodetic (lon/lat/height) view of that vector uses, so there is nothing to rotate.
  • meas.itrfxyz(lon, lat, height) / meas.wgs(x, y, z) — the real conversion: WGS84 geodetic ↔ geocentric Cartesian ITRF, via SOFA.gd2gc / SOFA.gc2gd (the same pair _frame_site already uses internally for AZELGEO). New core stubs _geodetic_to_itrf / _itrf_to_geodetic (src/measures/types.jl, mirror the _lst / _riseset stub/ext split) + real implementations in ext/SOFAExt.jl.
  • Closes the very last item in Phase 97's meas.* non-goals list (the column-MEASINFO-driven direction-argument form remains a non-goal — every meas.* argument is still an explicit expression, not inferred from a column's own MEASINFO).
  • Verified: meas.pos against measconvert(MPosition{ITRF}(...), WGS84) directly (exact — no arithmetic, an identity); meas.wgs ∘ meas.itrfxyz round-trips a real VLA-antenna ITRF position to atol = 1e-6 m.

Phase 107 — fix the UnitfulExt precompile method-overwrite bug

WARNING: Method definition _tql_known_unit(AbstractString) in module
MeasurementSets at src/tables/units.jl:98 overwritten in module
UnitfulExt at ext/UnitfulExt.jl:84.
ERROR: Method overwriting is not permitted during Module precompilation.
  • Root cause: _tql_known_unit(s::AbstractString) (core, src/tables/ units.jl) and MS._tql_known_unit(s::AbstractString) (ext/ UnitfulExt.jl) used the identical signature — a genuine redefinition, not an added dispatch. Every other core/extension stub pair in this package (_tql_quantity, _tql_unit_attach, _tql_write_strip, _ms_ustring, _quantity_column_spec, _lst, _riseset, _geodetic_to_itrf, _itrf_to_geodetic, every _mconv method, …) gives the core fallback a strictly looser signature (args..., an untyped positional, or an abstract/Union type) so the extension's concrete method is a genuine specialization, not an overwrite — Julia forbids the latter during extension precompilation. _tql_known_unit was the one place that pattern was broken.
  • Fix: drop the ::AbstractString annotation on the core definition (_tql_known_unit(s) = ...) — one line, matches the convention every other stub in the file already follows (_tql_write_strip(x, u) is the closest sibling: untyped core, ::Unitful.AbstractQuantity-typed extension).
  • Swept every other MS._* extension method across all four extensions (SOFAExt, EarthOrientationExt, HDF5Ext, UnitfulExt) against its core counterpart — confirmed no other instance of this bug exists.
  • Verified: import Unitful, UnitfulAngles, UnitfulAstro then using MeasurementSets precompiles cleanly (no warning, no error); _tql_known_unit correctly defers to the real _ms_uparse-backed check when the extension is loaded (e.g. "erg" — not in the core _COMMON_UNITS fallback set — now resolves true); the full units_tests.jl + taql_query_tests.jl standalone run is unchanged (760 tests, no count change — a pure precompile-hygiene fix).

Phase 108 — running* / boxed* sliding-window array reductions

query(main, "runningmedian(DATA, 2)[1,1] > 0")            # 5-channel median smooth
query(cat, "boxedaverage(SPECTRUM, 4) > threshold")        # 4-channel block average
  • running<X>(arr, hwidth) / boxed<X>(arr, bwidth), X ∈ average (mean)/median/min/max/variance/stddev/sum — array-cell sliding-window smoothing (one MAIN row's own array, reduced along its own axis/axes), the last item on the Phase 25 "not yet in TaQL-lite" list. running is a centred window ([i−h, i+h] per axis, shrinking at the edges — output the same shape as the input); boxed is non-overlapping bins of size bwidth (output shape cld(n, b) per axis, a partial trailing bin if bwidth doesn't divide evenly). The width argument is a scalar (same on every axis) or an array literal (ndims(arr) elements, one per axis) — runningaverage(V, [1,3]) smooths axis 1 with half-width 1 and axis 2 with half-width 3.
  • Shared generic engine (src/taql/functions.jl): _tql_window_widths (scalar-or-per-axis-array → an Int tuple, arity-checked), _running_reduce/_boxed_reduce (a plain CartesianIndices loop — no attempt at a separable/incremental-sum fast path; these operate on one row's small array cell, not a bulk column). min/max/sum keep the input's element type; average/median/variance/stddev promote to Float64 (matching the plain, non-running mean/median/ etc. reductions already in the function library).
  • Masked-array (TQLMArray) input is a documented non-goal — pass arraydata(...) first; a non-array (scalar) first argument raises a clear ArgumentError rather than silently treating it as a 1-element window.
  • No casacore/CASA oracle for this phase (self-contained numerical routines) — verified by hand-computed 1-D and 2-D references (edge- window shrinking, partial trailing bins, per-axis widths), agreement with a direct Statistics.var/std/median call on the same explicit window, and a query-string round-trip against the same functions called directly on each row's array.

Phase 109 — mscal.stokes() pseudo output types

query(main, "mscal.stokes(DATA, 'Ptotal')[1,1] > threshold")     # total polarized intensity
query(main, "mscal.stokes(DATA, 'I,Ptotal')")                    # physical + pseudo, one call
  • mscal.stokes(col, 'types')'s types now also accepts casacore's derived pseudo output types: Ptotal = √(Q²+U²+V²), Plinear = √(Q²+U²), Pangle = ½·atan2(U,Q) (rad), PFtotal/ PFlinear (the same totals divided by I) — non-linear combinations of Stokes I/Q/U/V, unlike every other mscal.stokes output (a plain matrix multiply against the correlation cell).
  • Implementation (src/taql/mscal.jl): pseudo types are encoded as negative internal codes (_STOKES_PSEUDO_CODES, never collide with a real 1-20 correlation code, thread through the existing _stokes_key/_stokes_setups sentinel machinery unchanged). StokesSetup gains an outtypes field (to tell pseudo rows apart) and an iquvmat (the input frame's own I,Q,U,V conversion matrix, built once, lazily — only when a pseudo type is actually requested). _stokes_convert's Complex method computes the ordinary linear rows as before, then — only if any output is a pseudo type — computes I,Q,U,V per channel once and derives each pseudo row's value from it; a query mixing physical and pseudo types in one mscal.stokes(DATA, 'I,Ptotal') call shares that single I,Q,U,V pass.
  • Bool (FLAG) / real (WEIGHT) input with a pseudo type requested raises a clear ArgumentError — the pseudo formulas are only meaningful for a complex (DATA-like) cell.
  • Closes the pseudo-output-type non-goal from Phase 78.
  • No casacore/CASA oracle for the pseudo-type formulas themselves (matches the documented Stokes::StokesTypes definitions) — verified by injecting a known DATA cell (RR=3+1i, RL=0.5-0.2i, LR=0.5+0.2i, LL=2-1i → I=5, Q=1, U=-0.4, V=1) via edit() and checking every pseudo type's value against the formula computed directly from the same I/Q/U/V; the I==0 edge case (fractional forms → 0, not NaN/Inf) checked directly on StokesSetup.

Phase 110 — column-MEASINFO-driven meas.<frame>() direction argument

query(fld, "meas.azel('PHASE_DIR', TIME/86400, X, Y, Z)[2] > 0")   # was: ..., 'J2000', PHASE_DIR[1], PHASE_DIR[2], ...
  • meas.<frame>(['SRC',] lon, lat[, mjd[, x, y, z]]) gains a second form: meas.<frame>('COLNAME', mjd[, x, y, z]) — instead of a literal 'SRC' frame name plus two explicit lon/lat expressions, a single string names a direction column, and the source frame is read from that column's own MEASINFO (measinfo(t, colname), fixed Ref only — a per-row VarRefCol column raises a clear error pointing at measure(t, col, row) instead). Closes the one remaining meas.* non-goal from Phase 97.
  • Disambiguated from the existing numeric form at parse time, not by argument count: the first string literal is a column name iff it is not a recognized frame name (_MEAS_DIR_FRAMES — j2000/b1950/ app/galactic/gal/ecliptic/ecl/azel/hadec/itrf/icrs) — so meas.b1950('J2000', RA, DEC) (existing) and meas.b1950('PHASE_DIR', TIME/86400) (new) both parse to the right form with no new syntax.
  • New AST node TQLMeasColDir (src/taql/functions.jl) threads through the query engine the same way mscal.*/mscal.stokes do: a "::measframe::COLNAME" sentinel resolved once per table (not per row) by a new _measframe_split/_measframe_cols pair, wired into both _tql_cols (query/groupby/join) and _vtq_prepare! (VirtualTaQLColumn — also fixed a stale docstring there that incorrectly claimed measures functions weren't supported in CALC expressions at all).
  • No casacore/CASA oracle (a MeasurementSets-only convenience over the existing measconvert machinery) — verified against measconvert called directly on the same column value, and against the equivalent explicit-'SRC'-plus-lon,lat numeric form on the same data; error paths (no MEASINFO, a non-direction MEASINFO, a VarRefCol column) each checked.

Phase 111 — UPDATE/DELETE ORDER BY + LIMIT

update!(t; set=["FLAG" => "true"], where="SNR < 3", orderby=["TIME"], limit=100)  # the 100 oldest
delete!(t; where="SCAN_NUMBER == 5", orderby=["TIME" => :desc], limit=20)         # the 20 newest
taql(t, "DELETE FROM t WHERE A > 3 ORDER BY TIME DESC LIMIT 2")
  • update!/delete! gain orderby/limit kwargs — TaQL's "update/ delete the N oldest/newest rows matching a condition" form. orderby is the same shape as query's do-block form (a bare column name/Symbol, ascending, or name => :asc/name => :desc); the matched rows are sorted by it (reusing the existing TQLOrderKey/ _apply_orderby machinery from ORDER BY), then limit (an Integer) keeps only the first limit of them — or, for limit < 0, the last |limit| — before the mutation runs.
  • taql()'s UPDATE/DELETE string forms parse a trailing ORDER BY k [ASC|DESC], … [LIMIT n] clause (new _taql_orderby_list helper) and pass it through to update!/delete!.
  • update!'s whole-column fast path (t[c][:] = [...], used when where === nothing) is now also gated on orderby/limit being unset — a orderby/limit-restricted update always goes through the per-matched-row path, even with no where.
  • Closes the Phase 30 non-goal ("DELETE/UPDATE ORDER BY+LIMIT … chain query then delete! by the selected condition instead").
  • Correction, found only once the full suite ran against real Casacore.jl: real TaQL's own UPDATE/DELETE ORDER BY ... LIMIT n does not sort the matched rows by the given key before LIMIT truncates them — a live spike showed UPDATE $1 SET A=A+100 WHERE A>3 ORDER BY T LIMIT 3 gives byte-identical results to the same command with ORDER BY T deleted entirely; T's actual values play no role. (LIMIT's own row-selection turned out direction/sign-dependent in a way not worth fully reverse-engineering for this phase.) This package's orderby/limit are a deliberate MeasurementSets extension implementing the genuinely useful "N oldest/newest rows" semantics the original non-goal text described — a real sort-then- limit — not a port of real TaQL's own behaviour; documented plainly in the update!/delete! docstrings. The planned real-TaQL cross-check test was replaced with the hand-computed-selection tests (already present) plus a comment recording the live-spike finding, in keeping with this project's established pattern for a documented MeasurementSets-only semantic choice.
  • Verified via hand-computed row selections (ascending/descending, positive/negative limit, the Julia and taql string forms agreeing).

Phase 112 — Hypercolumn_* keyword preservation on copy

copytable(dst, readtable(src))     # src's Hypercolumn_* private keywords now survive
  • copytable/copyms/write_ms (the Table source path) now preserve a source table's Hypercolumn_<name> private-keyword declarations (casacore TableDesc::defineHypercolumn) — closing the fidelity gap documented since Phase 11.
  • Found and fixed a real bug, not just a missing feature: _copy_table_cols was unconditionally passing private=Record() to _write_table_core, discarding the entire table-level private keyword set on every copy — even though the outer _copy_table methods already correctly defaulted private to the source's own (t.desc.private) and threaded it all the way down as a parameter that was then silently dropped at the very last call site.
  • New _filter_hypercolumns(private, keptnames) (src/tables/create.jl): passes every non-Hypercolumn_* private keyword through unconditionally, and preserves a Hypercolumn_<name> entry only when every column it names (HCdatanames/HCcoordnames/HCidnames) is still present, under the same name, in the copy's actual output column set — a renamed or dropped column silently drops just that one stale declaration rather than writing a reference to a column that no longer exists.
  • Our own reader never needed this keyword at all (Phase 11 — a hypercube's layout comes entirely from the storage manager's own on-disk header); this only matters for an external tool that inspects the TableDesc directly (e.g. tb.getdminfo()).
  • A RefTable/ConcatTable source still drops the private keyword set (_copy_table's RefTable/ConcatTable methods keep their own deliberate private=Record() default — a selection/projection may rename or drop the very columns a declaration names) — only the plain-Table source path (copyms/write_ms's common case) changed.
  • Verified: a hand-built Hypercolumn_TestCube keyword (via _write_table_core's private= kwarg directly) survives a full copytable round-trip intact; _filter_hypercolumns unit-tested for both the "all referenced columns kept" and "one is missing" cases, plus an unrelated private key passing through either way. No casacore/CASA oracle needed (pure keyword passthrough, not a new binary format).

Phase 113 — taql(): INSERT INTO t SELECT ... FROM 'path'

taql(t, "INSERT INTO t SELECT * FROM 'src.ms/POINTING'")
taql(t, "INSERT INTO t SELECT A AS X, B FROM 'src' WHERE A > 7")
  • taql()'s INSERT string form gains the row-copying variant: INSERT INTO t SELECT col [AS a], … FROM 'path' [WHERE cond] [LIMIT n] — * selects every source column as-is, an explicit list may rename a source column to match t's own name (A AS X), WHERE is an ordinary TaQL-lite condition over the source table, and LIMIT reuses insert!'s existing cycling/truncating semantics. Implemented as a thin wrapper: parses the clause, builds a query of the source table, and calls insert!(target; values=result, limit) — no new mutation machinery. Closes the Phase 31 non-goal.
  • Unlike Phase 111's UPDATE/DELETE ORDER BY/LIMIT surprise, this form was live-verified against real Casacore.jl first (INSERT INTO $1 SELECT A, B FROM 'src' WHERE A > 7, SELECT *, and an AS rename all spiked before writing the test) and matches our own implementation's semantics exactly — a genuine real-TaQL cross-check testset was added, not just hand-computed references.

Phase 114 — array-cell values in taql INSERT VALUES string form

taql(t, "INSERT INTO t (A, V) VALUES (9, [[1.0,2.0],[3.0,4.0]])")
  • taql()'s INSERT INTO t VALUES (...) string form now accepts an array-cell value as a bracketed literal — flat ([1.0, 2.0, 3.0], already worked once TQLArrayLit stopped being IN-only) or nested ([[1.0,2.0],[3.0,4.0]], new) — instead of requiring the Julia insert!(t; values=["V" => matrix]) form for an array-shaped column. New _taql_const-internal _nest_to_array: a rectangular nesting of vectors becomes a real multi-dimensional Array (stack); a ragged one is left as nested Vectors (which then fails, clearly, when written to an array-shaped column).
  • Element order matches real casacore TaQL's own nested-array-literal convention, confirmed by a live cross-check: the nesting is reshaped column-major — each inner vector becomes one column of the result, not one row ([[1,2],[3,4]] → [1 3; 2 4], not [1 2; 3 4]) — stack's own default axis order happens to match casacore's exactly, so no transpose/reshape juggling was needed once this was spiked against real TaQL (INSERT INTO $1 (A, V) VALUES (9.0, [[1.0,2.0],[3.0,4.0]]) on a real casacore table, then compared cell-for-cell against our own reader's output for the same insert). Closes the remaining half of the Phase 31 non-goal (the row-copying INSERT ... SELECT ... FROM half closed in Phase 113).

Phase 115 — mscal.baseline() ;-separated multi-term specs

query(main, "mscal.baseline('DA01&DV01;DA02&DV02')")
  • Investigated the standing Phase 80 non-goal ("baseline regex lists, blregexlist") against real casacore and found the plan's own characterization did not hold: mscal.baseline('[0,1]&2') and '{0,1}&2' are both flatly rejected by real tableCommand ("mismatched [ and ]" / parse error near {) — no bracketed antenna-name-list syntax exists in this casacore build. A first implementation attempt (splitting the whole spec on every top-level comma into separate OR'd pair-terms, with [...] grouping antenna names within one term) was built, then discarded before being committed once live cross-checking showed real casacore does the opposite: a plain comma extends one antenna-set list — even across & — rather than separating whole pair-terms ('DA01,DA02&DV01' is one pair-term with a 2-antenna LHS, not two terms; this already worked with zero code changes, since the pre-Phase-115 code only ever split on the first &, leaving every comma inside each side to _mssel_idset's own union logic).
  • The genuine gap, found instead: ; is casacore's real multiple-baseline-pair-term separator ('DA01&DV01;DA02&DV02', OR'd) — confirmed live across 7+ combinations (single/multi-antenna sides, duplicate terms, &&/&&& terms, 2–3 terms), every one matching a plain OR of each term's own match set exactly.
  • ! negation combined with ; is deliberately NOT implemented: 4 further live probes ('!A&B;C&D', 'A&B;!C&D', and their variants) showed the second half of a ;-list is silently dropped whichever term carries the ! — not a reproducible boolean combination, almost certainly a real casacore parser limitation rather than a defined feature. Replicating a bug isn't useful, so combining ! with ; raises a clear ArgumentError instead of a silent wrong answer; a ;-free leading ! (already supported before this phase) is unaffected.
  • _mssel_baseline_pred split into _mssel_baseline_term_pred (one &-pair term, unchanged logic) + a thin wrapper doing the ;-split
    • !-combination guard + OR.
  • Real-TaQL cross-check testset (5 multi-term specs, all matching); docs updated (the Phase 80 comment block, _mssel_baseline_pred's own docstring-comment recording the specific probes).

Phase 116 — RetypedArrayEngine feasibility investigation

  • Investigated the one remaining unsupported data manager's Phase 40 hand-wave ("needs the C++ source-type class") by reading casacore/ tables/DataMan/RetypedArrayEngine.{h,tcc} end to end. Confirmed genuinely infeasible, not just under-scoped: S in RetypedArrayEngine<S,T> is a C++ class template parameter — its dataTypeId()/set()/get() conversion functions and binary layout are arbitrary code compiled into whatever third-party program created the table, with no fixed wire format to target; the on-disk DM type string itself (className()) is built from S::dataTypeId(), and registerClass() must be called explicitly per instantiation by that program (casacore's own libcasa_tables never auto-registers any, unlike Dysco / BitFlags / ForwardColumn). grep -rl RetypedArrayEngine across the entire casacore source tree turns up zero real callers anywhere in casacore itself — including every Measurement Set-related file — only its own demo/test code (DataMan/test/dRetypedArrayEngine.{cc,h}).
  • Outcome: still unsupported, as before — the original assessment was correct. The only change is a clearer, investigation-backed error message + header comment in src/datamanagers/forwardcol.jl's _UnsupportedDM (cites the specific finding instead of the terser original text), so a future reader hitting it understands it's a structural dead end rather than a missing feature to file. No behavior/test change (the existing engine_tests.jl dispatch test — _dmtype("RetypedArrayEngine<Float>") === _UnsupportedDM — already covers the unchanged code path).

Phase 117 — primary-beam width-parameter validation

GaussianBeam(-1.0, 1.4e9)                   # ArgumentError: hpbw must be finite positive
AiryBeam(25.0; blockage = 25.0)             # ArgumentError: 0 <= blockage < diameter
EllipticalGaussianBeam(0.005, 0.02, 0, 1.4e9)  # ArgumentError: hpbw_major >= hpbw_minor
  • Every PrimaryBeam constructor (GaussianBeam, AiryBeam, PolynomialBeam, EllipticalGaussianBeam, SquintBeam) and every power_response method's freq/offset argument now validates its inputs — a non-positive/non-finite width, diameter, or frequency, a blockage outside [0, diameter) (blockage == diameter is a 0/0 singularity in the annular-Airy formula, > diameter is unphysical), hpbw_major < hpbw_minor (silently unenforced before this phase despite the docstring's ≥ claim), or a NaN/Inf offset/coefficient now raises a clear ArgumentError immediately instead of silently propagating to a NaN/Inf power response several calls downstream.
  • New shared helpers _pb_finite/_pb_positive/_pb_check_freq/ _pb_check_offset (src/beam/beam.jl); every constructor gained (or kept, for PolynomialBeam's pre-existing default one) an inner constructor doing the check-then-convert; power_response methods check freq and the θ/(dlon,dlat) offset at entry. SquintBeam's own power_response delegates its freq check to the wrapped base beam (no duplicate check) but validates its own offset before subtracting the squint.
  • A useful side effect: mscal.pbresponse('ellipse:HMIN:HMAJ:PA') (a transposed hmaj/hmin typo in a spec string) now raises a clear error at parse time instead of silently computing a rotated-wrong beam.
  • 41 new tests (test/beam_tests.jl, "Phase 117 parameter validation"); full existing beam + mscal.pbresponse/pbcorr/pbatten test suites (104 + 500 tests) pass unchanged — no valid existing usage anywhere in the codebase violated any of the new invariants.

Phase 118 — mscal.baseline() antenna diameter/mount selection investigation

  • Investigated whether real casacore's baseline-selection surface has any selection-by-physical-property (DISH_DIAMETER, MOUNT) syntax beyond name/id/glob/regex. Traced mscal.baseline(spec) (derivedmscal/DerivedMC/UDFMSCal.cc:445-465, the BASELINE case of UDFMSCal::getDataNode) to confirm it is a direct pass-through to real casacore's own MSAntennaGram/MSAntennaParse (msAntennaGramParseCommand) — not a separate mini-grammar, so its full syntax surface is exactly whatever the real MSSelection antenna grammar supports. grep -rin "diameter|mount" across every file in ms/MSSel/ (all grammar .yy/.ll files, every *Parse.cc) and a direct read of MSAntennaIndex.h's public interface (matchAntennaName / matchAntennaRegexOrPattern / matchStationName / matchAntennaNameAndStation / matchId — id, name, and station only) both confirm zero diameter/mount selection anywhere in casacore's own antenna-selection machinery. Confirmed absent, not missing — matches the earlier (correct) assumption; no code change needed. Achievable today anyway via a plain WHERE/join on ANTENNA.DISH_DIAMETER / ANTENNA.MOUNT (already fully general), just not through mscal.baseline's own spec-string syntax (which real casacore doesn't have either).
  • Incidental discovery, flagged for a future phase, not implemented here (out of this phase's chosen scope): reading MSAntennaGram.yy/ .ll end to end while investigating turned up the real form of the Phase 80 non-goal "blregexlist" — Phase 115 investigated and discarded a [name1,name2]-bracket-list guess (real casacore rejects it). The actual grammar production is blregexlist: BLREGEX (COMMA BLREGEX)*, where a BLREGEX token is a /…/-delimited regex whose body contains a literal & (the lexer's own discriminator, MSAntennaGram.ll:76-86: a /…/ regex containing & becomes BLREGEX instead of a plain per-name REGEX) — matched via MSAntennaParse::selectBLRegex against the whole "name1&name2" baseline string, not against each antenna name separately. Separately, MSAntennaGram.yy:150-163 shows gbaseline: NOT baseline | baseline and indexcombexpr: gbaseline | indexcombexpr SEMICOLON gbaseline — each ;-joined term can syntactically carry its own independent NOT, which suggests Phase 115's live-probed "! combined with ; silently drops the other term" finding may have a real, traceable explanation in how MSAntennaParse accumulates results across ;-joined terms (rather than being a bug) — worth revisiting with this grammar-level context before either implementing real BLREGEX support or reconsidering the Phase 115 !+; refusal.

Phase 119 — mscal.baseline() real BLREGEX support

query(main, "mscal.baseline('/DA01&DV.*/')")            # DA01 as ANTENNA1, any DV* as ANTENNA2
query(main, "mscal.baseline('/^DA01&DV01/')")            # NOT that exact ordered pair
  • Implements the real "blregexlist" mechanism found while investigating Phase 118, closing the Phase 80 non-goal correctly this time (Phase 115's [name1,name2]-bracket guess was a different, real-casacore- rejected idea). Confirmed by reading MSAntennaGram.yy/.ll + MSAntennaParse::selectBLRegex in full: a spec element is a /…/ regex whose body contains a literal & — the real lexer's own discriminator between a per-name REGEX and a BLREGEX — FULL- matched against the whole "name_i&name_j" string for every ordered pair of antenna indices (self-pairs included); a literal leading ^ inside the slashes negates just that one pattern (NOT the regex anchor — casacore repurposes the character); several comma-separated patterns OR their match sets; the outer ! this package already supports negates the whole list's result, not the first element.
  • Every one of the above was live-verified against real Casacore.jl before writing any code or tests (this session's established discipline, since a bracket-list guess was wrong once already): 10 representative specs run through both mscal.baseline and real tableCommand, all row counts matching exactly — including the subtle "negated pattern OR'd with a plain one still unions, doesn't intersect" case and the "outer ! distributes over the whole list, not just the first pattern" case (!/A&B/,/C&D/ = NOT(A∪C&D), not NOT(A)∪C&D).
  • New _mssel_is_regex_elem / _mssel_is_blregexlist (the detection — checked before the existing &&&/&&/& counting logic in _mssel_baseline_term_pred, since a BLREGEX pattern's literal & would otherwise misfire that counting) and _mssel_blregex_pred (the match-matrix builder, a direct port of selectBLRegex's nested loop — trivial cost, O(n_antennas²)). _mssel_baseline_pred / _mssel_baseline_term_pred gain an optional names::Vector{String} kwarg, threaded from mscal.baseline's own call site (which already reads ANTENNA.NAME); mscal.feed (no name table — feed ids are bare integers) passes nothing, so a BLREGEX-shaped feed spec now raises a clear error instead of silently misparsing the pattern's & as an ordinary L&R split.
  • Composes cleanly with the Phase 115 ;-multi-term machinery (no interaction with that phase's !+; refusal, which is about a different ambiguity — ;-joined whole baseline terms, not one blregexlist's own internal comma list).
  • 26 new tests (test/taql_mscal_tests.jl, "mscal.baseline() regex pair lists (BLREGEX)"), including the real-TaQL cross-check above; full existing mscal suite (500 tests) unchanged.

newer entries →