Phase 177 — found and fixed a real bug: week() used ISO-8601 week numbering, but casacore's own MVTime::yearweek() is a different, non-ISO convention
Another real bug found, continuing the same "check the sibling functions in this file" sweep from Phases 175-176: checked the numeric date-component functions (year/month/day/weekday/dow/week) against TableExprFuncNode's yearFUNC/monthFUNC/dayFUNC/ weekdayFUNC/weekFUNC (tables/TaQL/ExprFuncNode.cc:558-566) and the underlying MVTime methods (casa/Quanta/MVTime.cc:156-206).
year(), month(), day(), weekday(), dow() were all already correct — Julia's Dates.year/month/day/dayofweek happen to agree exactly with casacore's MVTime::year/month/monthday/ weekday (both use the same Mon=1..Sun=7 weekday numbering). But week() used Dates.week — ISO-8601 week numbering — while casacore's MVTime::yearweek() (built on yearday(), a classic day-of-year formula) is a different, non-ISO convention: at a year boundary where the ISO week wraps to week 52/53 of the previous year, casacore's own algorithm instead returns 0 for those early-January days that don't yet belong to a "full" week of the new year. Found live: 2022-01-01 (a Saturday) is ISO week 52 of 2021, but real casacore's week() gives 0, not 52.
Fixed with a direct port of MVTime::yearday/yearweek (_tql_yearday/_tql_yearweek, src/taql/functions.jl) — Julia's div/rem truncate toward zero and preserve the dividend's sign exactly like C++'s //% on Int, so this is a literal translation, not a re-derivation. Live-verified against real casacore across a 75-date sweep spanning five consecutive year boundaries (2020-2024) — every value matches, including every ISO-vs-non-ISO edge case.
New testset "Phase 177 — week() (casacore's non-ISO MVTime::yearweek)" (10 assertions, incl. the confirmed 2022-01-01 divergence and a regression check on the already-correct year/month/day/weekday/ dow) plus "Phase 177 — week(), real-TaQL cross-check" (375 assertions across 75 dates × 5 functions, _HAVE_TAQL-gated). Standalone taql_query_tests.jl green in full.
Phase 178 — swept normangle() and angdist(); both confirmed correct via a direct real-TaQL cross-check that had never existed before
Continuing the same "check every function in this corner of the file" sweep as Phases 175-177, since three of the last four checks turned up real bugs. normangle and angdist had only ever been checked against hand-computed unit-test expectations (normangle also had one indirect row-selection check inside a real-TaQL testset, but never a direct value comparison; angdist had no real-TaQL exposure at all) — exactly the "invented, never oracle-checked" pattern that produced Phases 174-177's four bugs, so both were worth a direct check.
Read TableExprFuncNode's normangleFUNC (tables/TaQL/ExprFuncNode.cc:849-853, fmod-based range reduction to (-π, π]) and angdistFUNC/angdist() (.cc:835-847, calling the shared angdist(lon1,lat1,lon2,lat2) free function) and live-verified both directly against real casacore via tableCommand — normangle for a spread of angles including exact π/multiples-of-2π/values a floating-point epsilon either side of the (-π,π] boundary, angdist for ordinary point pairs, an antipodal pair, and a near-pole pair. Both match to floating-point precision — no bug found. rem2pi(x, RoundNearest) (Julia stdlib) turns out to be exactly equivalent to casacore's own fmod+branch construction, and _tql_angdist's SOFA-seps-style atan2 form agrees with casacore's own angdist() everywhere tested.
New testset "Phase 178 — normangle()/angdist(), real-TaQL cross-check" (16 assertions, _HAVE_TAQL-gated) — the first direct value-level oracle check either function has ever had. Standalone taql_query_tests.jl green in full; this also closes out the src/taql/functions.jl date/time-and-angle-formatting sweep opened by Phase 174 — every function in that corner of the file has now been either fixed (174-177) or confirmed correct (178) against a real oracle.
Phase 179 — found and fixed two more real bugs: square()/sqr() computed the wrong thing for complex values, and min()/max() couldn't compare complex values at all
Two more real bugs found, in a different part of src/taql/ functions.jl (the general math-function table, not the date/time corner) — but the exact same "invented, never checked" root cause that produced Phases 174-178's findings, and both bugs directly affect DATA/MODEL_DATA/CORRECTED_DATA columns (complex-valued) in ordinary MS queries, not just an obscure edge case.
square()/sqr(): readTableExprFuncNode'ssquareFUNC(tables/TaQL/ExprFuncNode.cc:505-508(Int),658-661(Double),889-892(DComplex)) directly: for a complex argument, real casacore computes ordinary complex multiplicationx*x(a COMPLEX result —square(3+4i) == -7+24i). This package'ssquare/sqrwere instead aliased toabs2(a REAL magnitude-squared result —abs2(3+4i) == 25), silently discarding the phase of every visibility a query squared.norm()(.cc:678-683) IS the real casacoreabs2-equivalent function — a genuinely different function this package'ssquare/sqrwere conflated with.cube()was already correct (x^3, ordinary complex exponentiation). Fixed by changingsquare/sqrto_ew(x -> x^2), matchingcube's existing pattern.min()/max(): read the same file'sminFUNC/maxFUNC(.cc:899-921) andComplex/DComplex's own norm-based comparison operators (casa/BasicSL/Complex.h:174-206, ties return the first argument for both precisions): real casacore's 2-argumentmin/maxcompares a complex pair BY MAGNITUDE and returns the actual complex value with the smaller/larger magnitude. This package'smin/maxused Julia's plainmin/max, which has no ordering defined forComplexat all —min(DATA, x)on a visibility column raised a rawMethodError(islessundefined forComplex) instead of comparing by magnitude. Fixed with new_tql_min2/_tql_max2helpers (src/taql/functions.jl) — complex-aware magnitude comparison with a first-argument tie-break exactly matching casacore's operators; real-valued operands fall straight through to ordinarymin/max, unaffected.
Live-verified both fixes against real casacore via tableCommand for representative complex and real values — every case matches exactly. New testset "Phase 179 — square()/sqr()/min()/max() vs complex values" (11 assertions) plus "Phase 179 — square()/min()/max(), real-TaQL cross-check" (8 assertions, _HAVE_TAQL-gated, comparing this package's query() computed-select output directly against real casacore for 8 expressions). Neither bug had ANY prior test coverage (the entire package had zero existing references to square/sqr anywhere, and every existing min/max test used real-valued columns only) — a real, previously-invisible gap now closed. Standalone taql_query_tests.jl green in full.
Phase 180 — swept variance()/stddev()/mean() against a complex array cell; confirmed correct, plus a benign rms() permissiveness note
Continuing the complex-value sweep opened by Phase 179 — a reduction that's wrong for complex data is exactly the same bug shape, so variance/stddev/mean (all used on DATA-like columns in practice) were checked against real casacore's actual computation, not just assumed fine.
Read casa/Arrays/ElementFunctions.h:218-245 (SumSqrDiff's complex-type specialization) and ExprFuncNode.cc:788-808 (arrvariance0FUNC/arrstddev0FUNC): real casacore's complex variance sums (Δre)² + (Δim)² per element — the squared magnitude of each deviation from the mean, the standard definition of a complex random variable's variance — then takes the (already-real) result. This turns out to be exactly what Julia's Statistics.var/std already compute for a Complex vector, so _red(Statistics.var)/_red( Statistics.std) needed no change — confirmed correct via both source reading and a live cross-check against real casacore (variance/stddev/mean of a 4-element complex cell all match exactly).
Also confirmed, in passing: real casacore's rms() does not support a complex argument at all (tableCommand throws "function argument is not real"), while this package's rms computes the RMS magnitude for one — a benign extension, not a divergence to "fix" (there is no real casacore behaviour to match or diverge from).
New testset "Phase 180 — variance()/stddev()/mean() vs a complex array cell" (4 assertions) plus "Phase 180 — variance()/stddev()/mean(), real-TaQL cross-check" (3 assertions, _HAVE_TAQL-gated). Neither function had any prior complex-argument test coverage. Standalone taql_query_tests.jl green in full.
Phase 181 — checked whether Phase 179's min()/max()-vs-complex fix needs to extend to the group-aggregate gmin()/gmax(); confirmed it doesn't
Natural follow-up question after Phase 179: _TQL_AGGRS's gmin/gmax/gmins/gmaxs use plain Julia minimum/maximum — the exact same shape of implementation that turned out wrong for complex scalars in the plain min/max functions. Worth checking whether the group-aggregate versions have the same gap.
Read TableExprGroupFuncBase::makeGroupAggrFunc's dtype declarations (tables/TaQL/ExprAggrNode.cc:110-152) directly: real casacore restricts gminFUNC/gmaxFUNC/gminsFUNC/gmaxsFUNC/grmsFUNC/ grmssFUNC/gmedianFUNC to NTReal — there is no complex overload for any of them at all (checkDT(dtypeOper, NTReal, ...), so a GROUP BY query using one of these on a complex column is rejected at TaQL parse time, not silently computed with some behaviour to match). So, unlike the plain min/max/rms case (where casacore DOES have a defined complex behaviour this package was missing), there is no real casacore behaviour here for a complex gmin/ gmax/grms/gmedian to diverge from — Phase 179's fix correctly does not need to extend to these siblings. This package's own raw MethodError on a complex gmin/gmax is less polished than real casacore's TableInvExpr, but is not a "wrong value" bug.
Separately confirmed (same source read): gsum/gproduct/gmean/ gvariance/gstddev DO support complex in real casacore (.cc:286-300, NTComplex cases exist for all five) — but each reuses the exact same Julia primitive (sum/prod/Statistics.mean/ _pop_var/_pop_std) already live-verified correct for the plain, non-aggregate forms in Phases 179-180, so there was no separate divergence risk to check there either; confirmed via a self-consistency test against a hand-computed per-group reduction (a direct real-TaQL GROUP BY oracle comparison hit an unrelated tableCommand parsing quirk in this environment — "A GROUPBY key cannot have data type dcomplex" even with the complex column only ever referenced inside an aggregate — not pursued further since the NTComplex-support fact is already unambiguous from source, and the underlying Julia primitives are independently already proven correct).
New testset "Phase 181 — group-aggregate min/max/rms/median vs complex (scope check)" (6 assertions): gsum/gproduct/gmean on a complex column match a hand-computed per-group reduction; gmin/gmax on a complex column raise an error rather than silently misbehaving. No production code changed — this phase closes an open question, not a bug. Standalone taql_query_tests.jl green in full.
Phase 182 — found and fixed a significant real bug: running*() computed a completely different thing at every array-edge position than real casacore
A significant real bug found, in a fresh area (Phase 108's sliding-window array functions, self-described at the time as having "no oracle" and only ever hand-computed) rather than a further sweep of the already-closed-out functions.jl complex-value corner.
running<X>(arr, hwidth) was documented and implemented as a shrinking-window filter: at every position, including near an array boundary, it reduced whatever portion of the [i-h, i+h] window actually fit within the array. Read casacore's real implementation directly — slidingArrayMath (casa/Arrays/ArrayPartMath.tcc: 1060-1104) — and found this is not what real casacore does at all. TaQL's 2-argument running<X>(arr, shape) always calls the C++ function with its default fillEdge=true (confirmed: checkNumOfArg (2, 2, nodes) in ExprFuncNode.cc — TaQL exposes no 3rd argument to select the alternative mode), under which:
- the OUTPUT has the SAME shape as the input (matching what this package already did), but
- an edge position — anywhere within
hwidthof a boundary, where the FULL2·hwidth+1-wide window does not entirely fit — is set tozero(T), not a reduction over a truncated window, and - only genuinely interior positions (where the full window fits) get a real computed value.
Live-verified against real casacore via tableCommand: runningsum([1..8], [1]) gives [0, 6, 9, 12, 15, 18, 21, 0] — the first and last elements are exactly 0, not 1+2=3 / 7+8=15 as this package's shrinking-window implementation produced. This affects every running* function (runningsum/runningmean/runningmin/ runningmax/runningmedian/runningvariance/runningstddev) at every array boundary — for a typical spectral smoothing use (runningmean(DATA, [k]) over a channel axis), that's 2k channels at each edge of every spectrum silently computed as a value they should never have received, rather than the zero real casacore reports there. boxed* (the non-overlapping-bin sibling) was independently checked against boxedArrayMath (.tcc:1021-1053/fillBoxedShape, casa/Arrays/ArrayPartMath.cc: 29-48) and confirmed already correct — its trailing partial bin genuinely is a partial-window reduction in real casacore too (no edge/fill concept there at all), matching this package unchanged.
Fixed _running_reduce (src/taql/functions.jl) to compute only the genuinely-interior positions (zeros(T, size(arr)) pre-filled, then only the range where the full window fits gets overwritten). Live- verified against real casacore across 5 half-widths × 7 functions on a 12-element 1-D array plus a 2-D case — every value matches exactly, including the hwidth=0 (every position is its own full window, so the whole array is "interior") and hwidth ≥ n/2 (no position has a full window, entire output is zero) boundary cases.
Corrected the stale hand-computed assertions in the existing "Phase 108" testset (several previously asserted the old, wrong shrinking-window values) and added a new testset "Phase 182 — running*() edge semantics, real-TaQL cross-check" (36 assertions, _HAVE_TAQL-gated, covering all 7 functions across 5 widths plus a 2-D case). Standalone taql_query_tests.jl green in full.
Phase 183 — found and fixed two more real bugs: median()'s size-dependent averaging quirk, and gmedian()'s "never average" convention
Two more real bugs found, reading casa/Arrays/ArrayMath.tcc directly right after Phase 182's own array-math source dive — the median() family turns out to have not one but two distinct, non-obvious conventions this package's uniform Statistics.median usage completely missed.
- Plain
median()(arrmedianFUNC's default overload,median(a) = median(a, false, a.nelements()<=100, false),.tcc:1066-1107): for an EVEN-length array, casacore averages the two middle order statistics only when the array has ≤100 elements. Above that threshold it silently returns just the lower of the two — no averaging at all. Live-verified:median(1.0:128.0) == 64.0in real casacore, not64.5. This directly affects any wideband spectral-window array — 128/256/ 3840-channel bands are common, and both even and well over 100. gmedian()(the GROUP BY aggregate) does not go throughmedian()at all — it's built onTableExprGroupFractileDouble (this, 0.5), i.e. casacore's GENERICfractile()(.tcc:1138-1161), which never averages, regardless of size — a third, distinct convention from bothStatistics.medianand plainmedian()'s size-gated rule. Live-verified:gmedianof the 4-row group[1,2,3,4]is2.0in real casacore, not2.5. Unlike the >100-element trigger for plainmedian(), this one is routinely hit — GROUP BY groups are very often small and even-sized. The same investigation also confirmedrunningmedian/boxedmedian(Phase 182's own sliding-window fix) sharegmedian's "never average" convention (slidingMedians/boxedMedianshardcodetakeEvenMean=false, with no TaQL argument to change it) — so Phase 182's fix, while correct on the edge-fill semantics, still had the wrong even-window tie-break forrunningmedian/boxedmedianspecifically.
Fixed with two new helpers (src/taql/functions.jl): _tql_median (plain median()'s size-dependent rule) and _tql_fractile/ _tql_median_lo (the always-no-average convention shared by gmedian/runningmedian/boxedmedian). Live-verified against real casacore across the <=100 boundary (50/100/102/128 elements) for plain median(), and against a real GROUP BY for gmedian().
Corrected two stale test assertions that had baked in Statistics.median's always-average behavior (_boxed_med's existing Phase-108 unit test, and the groupby — correctness testset's gmedian assertion — every group there happens to be even-sized, so it was silently exercising the exact bug). New testset "Phase 183 — median()/gmedian() vs casacore's actual (non-Statistics.median) conventions" (7 assertions) plus "Phase 183 — median()/gmedian(), real-TaQL cross-check" (8 assertions, _HAVE_TAQL-gated). Standalone taql_query_tests.jl green in full.
Phase 184 — found and fixed a real bug: round() used Julia's
ties-to-even instead of casacore's round-half-away-from-zero
Continuing the sweep into the general math corner of src/taql/functions.jl (the same area Phase 179 fixed square()/ min()/max() in). Read TableExprFuncNode::getDouble's roundFUNC case (tables/TaQL/ExprFuncNode.cc:737-742) directly:
case roundFUNC:
{
Double val = operands_p[0]->getDouble(id);
if (val < 0) {
return ceil (val - 0.5);
}
return floor (val + 0.5);
}This is round-half-away-from-zero — every exact .5 value rounds outward, regardless of parity. round() in src/taql/functions.jl was wired straight to Julia's own round, whose default is round- half-to-even (banker's rounding) — a silent divergence at every .5 boundary landing on an even integer, invisible unless you specifically probe a tie value (non-tie inputs like 2.4/2.6 already agreed, which is presumably why this went unnoticed since Phase 25 first registered the function). Live-verified against real casacore: round(2.5) == 3.0 (Julia: 2.0), round(0.5) == 1.0 (Julia: 0.0, with the added footgun of a signed -0.0), round(-2.5) == -3.0 (Julia: -2.0) — every one of these is a case a real query (WHERE round(CHAN_FREQ/1e6) == …, a decimation/rounding filter) could plausibly hit.
Fixed with a new _tql_round(x) = x < 0 ? ceil(x - 0.5) : floor(x + 0.5) (src/taql/functions.jl) — a direct port of the quoted C++ — wired into _TQL_FUNCS["round"] in place of Julia's round. (The several other internal round(Int, …) call sites in src/taql/functions.jl/src/taql/mscal.jl are unrelated millisecond- formatting helpers for the date/time string functions, not the exposed round() TaQL function — left unchanged.)
New testset "Phase 184 — round() vs casacore's round-half-away-from- zero" (12 assertions: unit tests on _tql_round across every tie/non- tie/sign combination, plus a query()-level check) and "Phase 184 — round(), real-TaQL cross-check" (14 assertions, _HAVE_TAQL-gated, both literal-argument and column-argument forms). Standalone taql_query_tests.jl green in full.
Continuing the same source dive (per the standing methodology: once one function turns out wrong, check its siblings the same way immediately), swept the rest of the general-math corner of ExprFuncNode.cc and found three more real bugs, all the same "raw C++ std:: call, NaN not throw" shape:
pow(x,y)/ `** (powFUNC,.cc:655-657— and, which live-verified is the identical runtimepowFUNC/std::powpath, not a separate implementation): a negative base with a non-integer exponent isNaNin casacore, but Julia's^for twoReal`s **throws aDomainErrorin exactly that case. A real crash risk for any column that can go negative (pow(UVW[1], 0.5)). Fixed with_tql_pow(x,y) = x < 0 && !isinteger(y) ? NaN : x^y(an integer-valued exponent, even as aFloat64, does not throw in Julia either —(-1.0)^2.0 == 1.0— matching casacore, so only the fractional case needs a guard), wired into bothpow()and the**operator's_parse_power!.A subtlety found while testing this: casacore's operator precedence (like this package's own) puts unary minus OUTSIDE
**, so-2.0 ** 0.5parses as-(2.0 ** 0.5), notpow(-2.0, 0.5)— a literal negative base needs thepow(-2.0, 0.5)function-call form (each argument parses independently) or a genuinely negative-valued column to exercise; live-verified both engines agree on this precedence.sqrt/log/log10(.cc:668-670,653-654) andasin/acos(.cc:713-716): all four are a baresqrt/log/log10/asin/acoson aDouble— out-of-domain (sqrt/log/log10of a negative number;asin/acosof|x|>1) isNaNin C++, but Julia's ownsqrt/log/log10/asin/acosfor aRealall throwDomainErrorin that case. Live-verified against real casacore:sqrt(-4.0),log(-1.0),log10(-1.0),asin(2.0),acos(2.0)are allNaN. The same crash risk aspow— any real-valued column that can go out of a function's domain (sqrt(WEIGHT - threshold),asin(UVW[1] / baseline_length)). Fixed with_tql_sqrt/_tql_log/_tql_log10/_tql_asin/_tql_acos, each a::Real-specific domain guard returningNaNwith a fallback method passing aComplexargument straight through (Julia's ownComplexoverloads already never throw — they return the analytic-continuation branch, matching C++'sstd::complexoverloads — so only theRealmethods needed a guard).sign()(signFUNC,.cc:727-735): a manualif(val>0) 1; if(val<0) -1; else 0— aNaNinput falls through both comparisons to theelse 0branch, sosign(NaN) == 0.0in casacore, notNaN(Julia's ownsign(NaN) == NaN). Found while sweeping the surrounding functions for the same shape, and directly relevant now that the fixes above let moreNaNs flow downstream into asign()call than before (live-verified:sign(sqrt(-1.0)) == 0.0in real casacore). Fixed with_tql_sign(x) = isnan(x) ? zero(float(x)) : sign(x).
New testsets: "Phase 184 — pow()/** vs a negative base + non-integer exponent" (11 assertions) + its real-TaQL cross-check (13 assertions, covering both the function-call form and a negative-valued column through **); "Phase 184 — sqrt()/log()/log10()/asin()/acos() vs out-of-domain real args" (34 assertions) + its real-TaQL cross-check (20 assertions — tolerating a 1-ULP cross-library libm difference on in-domain values with ≈ rather than exact ==, live-verified as an ordinary floating-point variance and not a logic bug); "Phase 184 — sign() vs NaN" (4 assertions) + its real-TaQL cross-check (1 assertion). Corrected two stale .op === (^) unit-test assertions (**'s AST node now carries _tql_pow, not raw ^).
A fifth bug in the same family turned up directly as a consequence of the fixes above: int()/integer() (intFUNC, ExprFuncNode.cc:552-553) is a raw C++ Int64(double) cast — a NaN/out-of-range argument saturates rather than raising in C++ (live-verified: int(0.0/0.0) == 0, int(1.0/0.0) == typemax(Int64), int(-1.0/0.0) == typemin(Int64)), but Julia's own trunc(Int, …) throws an InexactError in all three cases. Before this phase, int(sqrt(-1.0)) was an unreachable combination (the old sqrt would already have thrown on the negative argument); now that sqrt()/log()/etc. correctly return NaN instead, that NaN flows straight into int(), which needed the identical fix. Fixed with _tql_int, a saturating cast matching the observed real casacore behavior. New testsets "Phase 184 — int()/integer() saturate instead of throwing" (8 assertions) + its real-TaQL cross-check (7 assertions, including through int(sqrt(A)) with a negative A).
A sixth bug, of a different (non-throwing) shape, turned up while checking isnan/isinf for the same Complex-argument corner once isfinite was under scrutiny: isfinite() on a Complex value. casacore's isFinite(Complex)/isFinite(DComplex) (casa/BasicSL/Complex.cc:123-129) is isFinite(re) || isFinite(im) — an OR, not the logically-expected AND (arguably a bug in casacore itself — "finite" should mean both parts finite — but real and reachable via TaQL's isfinite()). Live-verified: isfinite(complex(0.0/0.0, 5.0)) == true in real casacore. Julia's own isfinite(::Complex) uses AND — exactly backwards from casacore for a mixed finite/non-finite value. isnan/isinf on Complex (.cc:76-105) are already || in both casacore and Julia, so only isfinite needed a fix. Fixed with _tql_isfinite(x::Complex) = isfinite(real(x)) || isfinite(imag(x)).
Deliberately left nonfinite/isnonfinite (a MeasurementSets-only extension, not a real casacore function, used by the Phase 59/60 masked-array default-mask sugar) on Julia's own !isfinite rather than the new casacore-matching _tql_isfinite — for a masking predicate, "not finite" should mean either part is bad, which is exactly what !isfinite (AND, then negated → OR) already gives.
New testsets "Phase 184 — isfinite() vs a mixed finite/non-finite complex value" (10 assertions) + its real-TaQL cross-check (3 assertions, through a genuine complex column since TaQL-lite has no complex(re,im) constructor function to build one inline — a separate, larger gap noted but out of this phase's bug-fixing scope).
Standalone taql_query_tests.jl green in full.
Phase 185 — found a real missing-function gap: runningsamplevariance()/
runningsamplestddev()/boxedsamplevariance()/boxedsamplestddev()
Continuing the same corner of src/taql/functions.jl — the running*/boxed* sliding-window family Phase 182/183 already fixed two real bugs in. Reading TableParseFunc.cc's function-name table turned up something this package had entirely missed: casacore has two ddof variants of running/boxed variance and stddev —
funcName == "runningvariance" -> runvariance0FUNC (ddof=0, population)
funcName == "runningsamplevariance" -> runvariance1FUNC (ddof=1, n-1-corrected)
funcName == "runningstddev" -> runstddev0FUNC
funcName == "runningsamplestddev" -> runstddev1FUNC
funcName == "boxedvariance" -> boxvariance0FUNC
funcName == "boxedsamplevariance" -> boxvariance1FUNC
funcName == "boxedstddev" -> boxstddev0FUNC
funcName == "boxedsamplestddev" -> boxstddev1FUNC— mirroring the already-implemented gvariance/gsamplevariance group-aggregate split (_TQL_AGGRS already has both). This package's _running_var/_boxed_var (already correct — corrected=false, ddof=0, matching the plain name) had simply never been given *sample* siblings at all; calling runningsamplevariance(...) raised "unknown function" rather than computing the n-1-corrected value. Live-verified both variants are real and genuinely different: runningsamplevariance(1:8,[2]) gives 2.5 where runningvariance gives 2.0 over the same 5-element window.
Fixed by adding _running_svar/_running_sstd/_boxed_svar/ _boxed_sstd (Statistics.var/Statistics.std's own default corrected=true, i.e. ddof=1) and wiring all four new names into _TQL_FUNCS.
A second, smaller divergence turned up immediately while testing the edge cases: a window/bin with fewer than 2 elements is mathematically undefined for the n-1-corrected sample variance (unlike the population variant, which is well-defined — 0 — at n=1), and live-verified, real casacore genuinely throws ("Need at least 2 elements") rather than silently returning NaN the way a bare Statistics.var([x]) would. Reachable via a half-width/box-width of 0/1, or (more realistically) a trailing partial box with exactly one element. This is the same "match casacore's real behavior exactly" discipline the rest of this sweep has applied throughout Phase 184 — just pointed the other direction (there, several functions needed to stop throwing and start returning NaN; here, one needs to start throwing instead of silently returning NaN). Fixed with a shared _tql_need2 guard raising a clear ArgumentError.
New testsets "Phase 185 — missing runningsamplevariance()/ boxedsamplevariance() family" (21 assertions, including the <2-element throw cases and confirming the ddof=0 variant stays well-defined at n=1) + its real-TaQL cross-check (11 assertions, including that both engines throw for a same trailing-1-element-bin case). Standalone taql_query_tests.jl green in full.
Phase 186 — found five more missing functions: avdev(), runningavdev(),
boxedavdev(), runningrms(), boxedrms()
Continuing the same TableParseFunc.cc function-name-table check that found Phase 185's *samplevariance* gap — this time turning up a whole avdev (mean absolute deviation from the mean) family this package never had at all, plus the running/boxed siblings of the already-implemented scalar rms():
funcName == "avdev" -> arravdevFUNC (missing — new)
funcName == "runningavdev" -> runavdevFUNC (missing — new)
funcName == "boxedavdev" -> boxavdevFUNC (missing — new)
funcName == "runningrms" -> runrmsFUNC (missing — new)
funcName == "boxedrms" -> boxrmsFUNC (missing — new)avdev() (casa/Arrays/ArrayMath.tcc:1022-1043) is mean(|xᵢ − mean(x)|) — casacore's own per-element sum uses std::abs, so it already generalises to a Complex array with no separate branch (the magnitude-based abs makes the result real by construction; ExprFuncNode.cc:808-814's wrapping real(...) is a no-op). rms(), already implemented (_tql_rms), needed no formula change at all — runrmsFUNC/boxrmsFUNC are dtin=NTReal (no complex overload, unlike avdev's NTNumeric), which is irrelevant here since _tql_rms already only ever uses abs2 (correct for both real and complex) and was never given a complex-unsafe shortcut to begin with.
Both new reducers (_tql_avdev, reused directly as a sliding-window reducer via the existing _running_reduce/_boxed_reduce machinery — no new plumbing needed) live-verified against real casacore: avdev(1:8) == 2.0, runningavdev(1:8,[2])[3] == 1.2, runningrms(1:8,[2])[3] ≈ 3.3166247903554, all matching a hand computation over the same windows exactly.
While checking the function-name table for these, found (but did NOT implement — a real, larger feature gap flagged for a dedicated future phase, not this same-day bug-fix sweep) that casacore also has an entire "s"-suffixed axis-collapse family — sums, means, mins, maxs, products, medians, variances, stddevs, avdevs, rmss, fractiles, anys, alls, ntrues, nfalses (arrsumsFUNC etc.) — that reduce a multi-dimensional array cell along specific axes given as an argument, leaving the other axes intact. This is entirely distinct from this package's own gs* masked group aggregates (Phase 62) despite the superficially similar naming, and none of it exists in TaQL-lite today.
New testsets "Phase 186 — missing avdev()/runningavdev()/boxedavdev()/ runningrms()/boxedrms()" (14 assertions, including a Complex-array avdev case) + its real-TaQL cross-check (6 assertions). Standalone taql_query_tests.jl green in full.
Phase 187 — found a real bug (ltrim()/rtrim() strip too much) and
two more missing functions (capitalize(), sreverse()/reversestring())
Moved to the string-function corner of src/taql/functions.jl, not yet swept this batch. Read ExprFuncNode.cc's ltrimFUNC/rtrimFUNC/ trimFUNC cases directly and found a real, confirmed divergence: casacore's ltrim()/rtrim() use the regexes leadingWS/trailingWS ("^[ \t]*"/"[ \t]*$", .cc:973-974) — stripping only space and tab, never newline or carriage-return — whereas trim() (String::trim(), casa/BasicSL/String.cc:105-112) strips all four (space/tab/\n/\r) from both ends. This package's ltrim/ rtrim were wired to plain Julia lstrip/rstrip (no predicate), which strip every Unicode whitespace character — a real divergence whenever a string's leading/trailing whitespace includes a newline. Live-verified: ltrim("\n\t X \t\n") in real casacore is the string completely unchanged (it starts with \n, which [ \t]* never matches), while the old Julia-lstrip-based implementation stripped it down to "X \t\n". trim() itself happened to already agree with casacore for every plain-ASCII case (Julia's broader whitespace set is a superset of casacore's narrower 4-char one) but was narrowed to the exact 4-char set anyway, for genuine fidelity rather than an accidental agreement. Fixed with _tql_trim/_tql_ltrim/_tql_rtrim.
While checking the surrounding string functions for more of the same, found two entirely missing ones: capitalize() (title-cases each "word" — a maximal run of letters/digits; any other character, including _/., is a word boundary — first character of each word uppercased, the rest lowercased; String::capitalize(), casa/BasicSL/String.cc:323-334) and sreverse()/reversestring() (a plain character reversal, String::reverse()). Live-verified: capitalize("hello world") == "Hello World", capitalize("3d star_field.name") == "3d Star_Field.Name" (the leading digit 3 starts a "word" too, per casacore's own isdigit(*p) check, but has no letter case to change). sreverse/reversestring matches Julia's own reverse(::AbstractString) exactly. Also added the missing to_upper/to_lower aliases for upcase/downcase (casacore accepts all four spellings; this package only had three of the four).
New testsets "Phase 187 — ltrim()/rtrim() strip too much; capitalize()/sreverse() missing" (15 assertions) + its real-TaQL cross-check (9 assertions, covering both the whitespace corner case and the new functions). Standalone taql_query_tests.jl green in full.
Phase 188 — found a real missing function: shape()
Continuing the same TableParseFunc.cc function-name-table check that found Phases 185-186's gaps, now applied to the general "misc" function corner instead of the running*/boxed* family: shape() (shapeFUNC, ExprFuncNodeArray.cc:1041-1053) — the per-axis extents of an array cell as an Int array — was entirely absent from TaQL-lite. Confirmed casacore's own non-C-order default style returns the axes in exactly the order this package already stores/indexes arrays in (the C-order-reversed form only applies under an explicit USING STYLE PYTHON-family selector, which TaQL-lite has no concept of at all — a non-issue). Live-verified: shape(B) for a (3,4)-shaped cell gives [3, 4], matching Julia's own size(B) == (3, 4) directly, no reversal needed; a scalar's shape is the empty Int array. Fixed with _tql_shape.
While at the same corner of the name table, found three more introspection-style functions this package doesn't (and likely won't soon) implement, each for a different, real reason:
regex()/pattern()/sqlpattern()— build a regex/glob/SQL pattern object DYNAMICALLY from an arbitrary string expression (not just the fixed literal~ p/.../grammar Phase 24 already supports), usable in a subsequent~/==comparison. A real, meaningfully different capability (NAME ~ regex(PATTERN_COL), matching against a computed/column pattern) — but properly supporting it needs a new "compiled pattern" value type threaded through the comparison operators, not a one-line function addition. Flagged for a dedicated future phase, same treatment as the axis-collapse family (Phase 186).isdefined()/isnull()— this package has no "null"/"undefined cell" concept anywhere (confirmed by the existinggcountdesign note from Phase 26), so these would be close to meaningless no-ops; not worth the surface area without a real use case.iscolumn()/iskeyword()— check table-level metadata by a name string, not a per-row column value — architecturally different from every other TaQL-lite function (which only ever seescols[name][i], never the table object itself). Would need table-level context threaded through the whole function-eval machinery; a structural change, not a quick addition.
New testsets "Phase 188 — missing shape() function" (4 assertions) + its real-TaQL cross-check (1 assertion). Standalone taql_query_tests.jl green in full.
Phase 189 — found a real missing function family: sumsqr()/sumsquare()
and its running/boxed/g* siblings, via a full function-name-table diff
Phases 185/186/188 each found a missing function by re-reading one corner of TableParseFunc.cc's function-name table by hand. This phase instead dumped casacore's complete function-name table and diffed it against everything registered in _TQL_FUNCS/_TQL_AGGRS — a more systematic version of the same check. Found sumsqr()/ sumsquare() (the sum of elementwise squares, Σxᵢ² — ordinary multiplication, not abs2; for a Complex array this matches the Phase 179 square() finding exactly: z*z, not the magnitude) and its running/boxed/g*/gs* siblings — runningsumsqr/ boxedsumsqr (sliding-window) and gsumsqr/gsumsqrs (group aggregate + per-element) — entirely missing.
arrsumsqrFUNC's own C++ (ExprFuncNode.cc:776-782 for Double, :948-953 for DComplex) confirmed the exact formula: val*val for a scalar, sumsqr(array) (elementwise square then sum) for an array, identical for both real and complex. Live-verified: sumsqr(1:8) == 204.0 (== sum((1:8).^2)), runningsumsqr(1:8,[2])[3] == 55.0, boxedsumsqr(1:8,[2])[1] == 5.0, gsumsqr of the group [1,2] is 5.0, and sumsqr([1+1im, 2+0im]) == 4.0+2.0im == sum([1+1im,2+0im].^2) (ordinary complex square).
Fixed by adding _tql_sumsqr (the scalar array reduction, reused directly as a _running_reduce/_boxed_reduce reducer — no new plumbing needed, same pattern as Phase 186's avdev/rms) and _tql_gsumsqr (the group-aggregate reducer, shared between the :scalar and :perelem modes exactly like every other _TQL_AGGRS entry). sumsqrs/sumsquares (the "s"-suffixed AXIS-COLLAPSE variant) is deliberately excluded — it belongs to the same larger, already-flagged (Phase 186) axis-collapse feature, not this sweep-for-a-missing-sibling pass.
New testsets "Phase 189 — missing sumsqr()/gsumsqr() family (found via a full function-name-table diff)" (12 assertions, including a Complex-array case and a real groupby cross-check) + its real-TaQL cross-check (6 assertions, covering the scalar, running, boxed, and group-aggregate forms). Standalone taql_query_tests.jl green in full.
Phase 190 — sweeping src/taql/functions.jl to completion (user request):
a real hms()/dms() bug, missing hdms(), the rest of running*/boxed*,
and six more standalone functions
At the user's explicit direction, continued sweeping src/taql/functions.jl toward genuine completion rather than stopping at the next isolated finding. Two threads:
A real, confirmed bug, found while checking hdms()'s implementation against hms()/dms()'s: casacore's getArrayString (ExprFuncNodeArray.cc:2424-2454) shows hms()/dms() apply ELEMENTWISE to an array argument, not just a scalar — but this package's registration called _tql_hms(float(x)) directly with no _ew wrapping, so hms(a_PHASE_DIR_column) threw a MethodError instead of formatting each element. Live-verified: hms([1.0, 0.5]) == ["03h49m10.987", "01h54m35.494"] in real casacore. Fixed by wrapping both in _ew (the standard elementwise-or-scalar dispatch already used throughout this file). While there, found hdms() — an array-only sky-position formatter that alternates hms/dms by (0-based) index (hdms([ra1,dec1,ra2,dec2]) == [hms(ra1), dms(dec1), hms(ra2), dms(dec2)]) — entirely missing; added _tql_hdms.
Completed the running*/boxed* family: a full diff of TableParseFunc.cc's funcName == "running..."/"boxed..." chain (.cc:353-488) — the same technique Phase 189 introduced — against _TQL_FUNCS turned up EIGHT more entirely missing pairs (16 functions) plus two missing aliases:
runningproduct/boxedproduct— product per window/bin.runningfractile/boxedfractile— a THIRD argument (the fraction, 0–1) inserted before the half-width/box-width, using the SAME never-averagefractile()convention already implemented forgmedian/runningmedian/boxedmedian(_tql_fractile, Phase 183) —runningmedian(arr,[h])is in fact justrunningfractile(arr,0.5,[h])under the hood in casacore itself.runningany/runningall/boxedany/boxedallandrunningntrue/runningnfalse/boxedntrue/boxednfalse— plainany/all/count-true/count-false per window over aBoolarray; the Phase-182 edge zero-fill naturally becomesfalse/0.runningavg/boxedavg— real casacore ALIASES forrunningmean/boxedmeanthis package never registered.
Live-verified every one against real casacore: runningproduct(1:8,[2])[3] == 120.0, runningfractile(1:8,0.5,[2])[3] == 3.0 (matches _tql_fractile([1,2,3,4,5],0.5) exactly), runningany/runningall/ runningntrue/runningnfalse on a [T,T,F,T,T,F,T,T] array all match a hand count.
Six more standalone functions, found continuing the same TableParseFunc.cc name-table sweep past the running*/boxed* chain:
c()— the speed of light (cFUNC), the same value already insrc/constants.jlasC_LIGHT.near(a,b[,tol])/nearabs(a,b[,tol])— standalone FUNCTION-CALL forms of approximate equality.nearis the SAME relative-magnitude algorithm as the~=operator (_tql_near, Phase 47), but with a much tighter DEFAULT tolerance (1.0e-13, not~='s1e-5) when no 3rd argument is given.nearabsis a different, ABSOLUTE-difference check (|a-b| <= tol, casacore's ownnearAbs) with no default tolerance concept shared withnearat all — a barenearabs(a,b)also uses1.0e-13. Live-verified:near(5.0, 5.1) == false(tol1e-13,|5.0-5.1|=0.1far too big),near(5.0, 5.1, 0.5) == true,nearabs(5.0, 5.05, 0.1) == true,nearabs(5.0, 5.2, 0.1) == false. Fixed with a new_tql_nearabs(_tql_nearalready existed and slotted in directly).gfractile(col, frac)— the group-aggregate sibling ofgmedian, with an explicit constant fraction instead of the hardcoded0.5(gmedianis literallyTableExprGroupFractileDouble(this, 0.5)internally in casacore,gfractileis the same class withfracsupplied). The fraction is evaluated once, not per row, so it must be a numeric-literal argument — mirrors how this file already handles a handful of other constant-argument functions (mscal.pbresponse's beam spec,mscal.riseset's elevation cutoff). Live-verified:gfractile([1,2,3,4], 0.25) == 1.0, matching_tql_fractile's existing formula exactly.countall()— the SQL-standardCOUNT(*)spelling, byte-for-byte the same row countgcount()computes (TableExprGroupCountAllvsTableExprGroupCount— casacore's own two classes do the identical thing), just never taking a column argument.mask— a missing alias for the already-implementedarraymask().cweekday— a missing alias for the already-implementedcdow().
New testset "Phase 190 — sweeping functions.jl to completion: hms/dms array bug + hdms + the rest of running/boxed" (49 assertions) + its real-TaQL cross-check (25 assertions). Standalone taql_query_tests.jl green in full — the whole file's ~116 testsets ran end to end with no failures.
Remaining functions.jl names not yet resolved this phase (continuing next): bool/boolean, str/string (needs a getPrintFormat-style width/precision spec), rand, rowid, cones/anycone/findcone (needs a spatial cone-search index), the rest of the masked-array native functions (negatemask/replacemasked/replaceunmasked/ nullarray), the array-reshaping family (array/arrayflatten/ flatten/diagonal/diagonals/resize/transpose/reversearray), regex/pattern/sqlpattern (Phase 188, needs a new value type), isdefined/isnull/iscolumn/iskeyword (Phase 188, needs a null-cell concept or table-level context), substr/substring/ replace (the Phase 25 "string index-base rabbit hole" non-goal), gaggr/ghist/ghistogram/growid/gstack (the Phase 26 non-goal), and the "s"-suffixed axis-collapse family (Phase 186's flagged larger feature).