Phase 330 — standard MS schema diffed against a casatools-built MS (no bug found)
Every column of a casatools-simulator-built MS (134 columns over MAIN and 12 subtables) was diffed against stdtable: all value types agree, no required schema column is absent from the casacore-built MS, and every fixed-rank schema shape has casacore's rank. Differences are representational only (casacore repeats a unit per component, m,m,m, where the schema has m; MODEL_DATA/CORRECTED_DATA are casacore-added optional columns; three SOURCE columns report no rank). Pinned by a static snapshot test in test/schema_tests.jl.
Phase 331 — hard-coded format constants diffed against casacore's headers (no bug found)
ColumnDesc::Option (Direct=1, Undefined=2, FixedShape=4), StorageOption::Option (MultiFile=0, MultiHDF5=1, SepFile=2) and the ColumnSet version codes all match what the writer/reader hard-code; the 8-character type ids in the ScalarColumnDesc<…> class names are already proven end to end (real casacore refuses an unknown class name, and the Phase 265 type matrix round-trips every type). Pinned by a static test in test/writer_tests.jl.
Phase 332 — datetime('<string>') accepts the rest of casacore's date grammar (random format fuzz vs real TaQL)
Spread 400 random date strings over 9 formats against real datetime(): ours rejected or misread about a third of what casacore's MVTime::read accepts. Now matching (live-probed): Y/M/D with any of / T space - before a time, with month/day unbounded so 2020/02/30 rolls to 1 March; times as H:M[:S[.f]] or 10h30m15s (including after the dash-numeric 2020-02-12/10h30m); D[-]Mon[-]Y with a 2- or 4-digit year and a /time (12Feb20, 3Aug2033/01:49); and — oddly but really — a bare number is an MJD day count (58000, 58000/12:00 = 58000.5, and even 20200212 is 20200212 days, not a date). D Mon Y with spaces and D/M/Y are not dates in real TaQL either and stay errors. After the change all 400 strings agree with real TaQL (361 equal values, 39 rejected by both). New testset in test/taql_query_tests.jl (23 probe strings + 60 random, fixed-seed MersenneTwister(332)); the existing date/time testsets and mscal.time tests are unchanged.
Phase 333 — single-field h/d literals are plain time quantities; time ↔ angle coercion (unit-literal fuzz vs real TaQL)
Fuzzing col <op> <number><unit> comparisons against real TaQL on columns with units found a real bug: a single-field 6.06h / 3.73d was lexed as a sexagesimal angle (RA hours / degrees → radians), so TM > 1h on a column in seconds compared against ~0.26 rad. Real TaQL reads single-field Nh / Nd as plain hour / day quantities (only the multi-field 10h30m / 30d15m are sexagesimal angles), and converts between time and angle with 24 h = 2π — both TM > 1h30m on a seconds column (= 5400 s) and A > 12h on a radian column (= π). Fixed: _sexagesimal_unit needs a following m…/s field; with Unitful loaded the multi-field forms become radian quantities (rad is dimensionless, so a unitless column still compares as a plain number); _bcast retries a DimensionError through an extension hook (_tql_dim_coerce) that converts a time operand to an angle when the other is an angle. Metre-vs-second style mismatches (TM > 30m) still error, as in real TaQL. The Phase 87 single-field-h assertions were updated. New testset in test/taql_query_tests.jl cross-checking 17 unit-literal forms against real TaQL.
Phase 334 — string functions are byte-oriented like casacore (random-string fuzz vs real TaQL)
Random strings (including a non-ASCII é) through 26 string forms against real TaQL: everything agreed for ASCII, but every non-ASCII string differed, because casacore's string functions work on C char bytes. Now matching: strlength counts UTF-8 bytes (é is 2), upcase/downcase/capitalize change ASCII letters only and treat any non-ASCII byte as a word break, and substr/sreverse cut and reverse bytes (so they can yield invalid UTF-8, as in casacore). Comparisons, +, IN, LIKE, ~/!~ and replace/trim already agreed. New testset in test/taql_query_tests.jl (hand-checked units + a seeded real-TaQL fuzz, MersenneTwister(334)).
Phase 335 — random LIKE / glob / regex pattern fuzz vs real TaQL (investigation only, no bug found)
600 random patterns over LIKE, ILIKE, glob ~ p/../ (with *, ?, [ab], [!a], {a,bx} and the i flag) and partial / full regex ~ m/../, ~ f/../, against 60 random strings, agreed with real TaQL on every row. Confirms the Phase 157 line-by-line read of casacore's fromPattern / fromSQLPattern at scale. A seeded 150-pattern slice is kept in test/taql_query_tests.jl (MersenneTwister(335)).
Phase 336 — IN [...] elements may be arbitrary expressions (28-form probe vs real TaQL)
Real TaQL evaluates each IN [...] element as an expression per row: A IN [A, 3], A IN [2*2, sqrt(16)], S IN ['s1', 's'+'2'], A IN [K+1, K-1] and ranges with expression bounds ([1+1:4], [1:K], [K:K+2]). TaQL-lite only accepted literals and literal ranges, so all of those raised a parse error. Elements now parse at arithmetic level; an all-literal element stays static and is validated at parse time as before ([5:1], step 0 still error), anything else becomes a TQLExpr / TQLDynRange resolved per row (_in_resolve) in both the row and group evaluators, with the visitors (_tqlrefs!, _has_aggr, _has_qty, _sg) following the sub-expressions. 15 forms cross-checked against real TaQL. Still lenient where real errors: IN [] (matches nothing), A IN [true], NOT A IN [3:20].
Phase 337 — integer arithmetic is 64-bit like real TaQL (mixed-type WHERE probe vs real TaQL)
A 100-form probe mixing Int32/Int64/UInt8/Float/Bool/String columns against real TaQL found two value differences: U * U > 100 and ~U == 250 on a UInt8 column wrapped at 8 bits here, whereas real TaQL promotes integers to 64 bits (U * U for 16 is 256, ~5 is -6). Integers narrower than Int64 are now widened before every arithmetic / bitwise / comparison broadcast (_widen in _bcast / _bcast_raw), which also removes Int32 / Int16 overflow in expressions like I * 100000000. The remaining differences in the probe are all forms real TaQL rejects (bool arithmetic, S == 1, P & Q, ...) that TaQL-lite accepts leniently. New testset with a real-TaQL cross-check.
Phase 338 — computed SELECT result types match real TaQL (68-form eltype probe)
Comparing the column type of 68 SELECT <expr> AS Z forms against real TaQL: real computes in double precision, so Float32 operands give Float64 and ComplexF32 give ComplexF64, and results are plain Int64 / Float64 columns. TaQL-lite left Float32/ComplexF32 results, produced abstract eltypes (Signed, AbstractFloat) when rows mixed Int32/Int64, and returned Irrational for pi(). Now Float32/Float16/ComplexF32 operands are widened to double alongside the Phase 337 integer widening, computed columns that came out abstract are concretised to Int64 / Float64 (_tql_concrete), and pi() / e() are Float64. All 68 forms agree on type; new testset with a real-TaQL cross-check.
Phase 339 — GROUP BY aggregate result types match real TaQL (47-aggregate eltype probe)
Same finding as Phase 338 for the g* aggregates: real TaQL aggregates in 64-bit integers / double precision, so gsum(UInt8), gmin(Int32), gmax(Int16) and gfirst(UInt8) are Int64 and gmean/gmin/gmax/gvariance/ gmedian/gproduct/glast/gsum of Float32 or ComplexF32 are Float64 / ComplexF64; TaQL-lite kept the column's own narrow type (and gsum(UInt8) was UInt64). Aggregate arguments are now widened per row (_widen, Phase 337/338) before the reduction, which also removes UInt8 / Float32 accumulation error. Result types of 47 aggregate forms now agree with real TaQL (only the array-of-array representation of growid/gstack/gaggr differs, with matching element types); gsum/gmean of a Bool column still work although real TaQL rejects them.
Phase 340 — array expressions: 64-bit/double reads, kept subscript axes, elementwise near() (87-form probe vs real TaQL)
Result element type and shape of 87 array-valued SELECT expressions against real TaQL found three differences. (1) Real TaQL reads columns as Int64 / Double inside expressions, so sum(U), min(U), transpose(B), resize, flatten, B[1,1], boxedsum, sums ... of UInt8 / Int32 / Float32 cells are 64-bit / double; a column read inside an expression is now widened (_widen, extending Phases 337–339; plain projections and group-key columns keep their stored type). (2) A scalar subscript among slices keeps a length-1 axis (B[1:2,1] is 2×1, shape(B[1,1:2]) is [1,2]); only an all-scalar subscript gives a scalar. This corrects Phase 42's claim that scalar axes are dropped (that is Julia's rule). (3) near(a,b[,tol]) / nearabs are elementwise on array cells. New testset with a real-TaQL cross-check.
Phase 341 — computed SELECT columns keep QuantumUnits when persisted (probe vs real TaQL GIVING)
Persisting SELECT <expr> AS Z ... (INTO / GIVING) and comparing the new column's type, shape and units against real TaQL: types and shapes agree (real writes computed array columns variable-shape, ours fixed-shape — left), but a computed column lost its QuantumUnits, and so did every column of a SELECT that mixed projections with computed expressions (only an all-projection SELECT, a lazy RefTable, kept them). Real TaQL propagates units through expressions; probing 44 forms gave the rules now implemented: a plain column keeps its unit; + - % // keep it (unitless operand neutral); * joins (m.Hz), / divides (m/(Hz), (m)-1, equal units → none); unary minus, abs/min/max/mean/floor/round/ real/iif keep it, square squares it; **, comparisons, trig, sign, int and the rest are unitless (mismatched + and sqrt(X) are unit errors in real TaQL and stay lenient here). GroupedTable gained a units field (persisted by copytable / INTO); TQLFunc records its name. Also fixed: an expression that reduces to a bare column (+X, (X)) is a projection of that column, not a lookup of its source text. New testset with a real-TaQL cross-check.
Phase 342 — unit conversion inside expressions and UPDATE (probe vs real TaQL)
Probing UPDATE ... SET (19 forms) and SELECT / WHERE (33 forms) on columns in m, km, Hz, rad and deg: real TaQL converts units inside expressions while TaQL-lite used the raw numbers (SET X = KM stored 1 instead of 1000 for a metre column, X + KM added km to m). Now, as in real TaQL: the right operand of + - % //, of a comparison, a BETWEEN bound, an IN [...] element, a min/max/iif branch and the divisor of / of the same dimension is converted to the left operand's unit, and UPDATE SET X = <expr> converts the value to X's unit; assigning an incompatible dimension (Hz into a m column) is an error. * joins units without converting. Implemented as an AST rewrite (_unit_rw, _unit_conv, _unit_assign) that multiplies the conversion factor in as a literal; the factor comes from the Unitful extension (_unit_factor, nothing/no conversion without it). Reads stay lenient for mismatched dimensions (real TaQL errors) and real TaQL's odd BETWEEN behaviour with a unitless bound is not reproduced. New testset with a real-TaQL cross-check.
Phase 342 follow-up — CI fix: Phase 335 pattern guard generated invalid regexes
GitHub Actions (Linux x86-64, Julia 1.10) failed the Phase 335 random pattern guard on 4 of 150 patterns, all regex (~ m/../, ~ f/../) forms the generator built with stacked quantifiers such as ?* and **. Real casacore on Linux accepts those, while PCRE (and casacore on macOS) rejects them, so the two engines disagreed only on invalid patterns. Reproduced in an x86-64 Linux container; the test now collapses stacked quantifiers in the regex forms. No package change.
Phase 343 — meas.* epochs are seconds; more direction frames (random conversion fuzz vs real TaQL)
A random fuzz of meas.<frame>([lon,lat], 'SRC', epoch, 'OBS') over 13 frames, epochs and observatories against real TaQL's meas UDFs found that real casacore reads a plain-number epoch as seconds since MJD 0 (the unit of a TIME column) — meas.app(d,'J2000',60454.0) is the year-1858 answer, 60454d or 60454.0*86400 the 2024 one. The Phase 249 value-first forms assumed MJD days (that held only for the d forms), so any observer-dependent conversion given an epoch was wrong by an epoch error. Plain epochs are now seconds, unit quantities (60454d, 30deg, 5 m) are converted by the Unitful extension, and the older source-first numeric forms (meas.azel('J2000', lon, lat, mjd, x, y, z)) keep MJD days; datetime() / mjd() values are days here, so multiply by 86400 to use one as an epoch. Also added azelgeo, azelsw, azelswgeo and supergal to the meas.* direction frames (and AZELGEO/AZELSW/AZELSWGEO as epoch/position-needing sources). After the fix 369 random conversions agree with real TaQL within the usual SOFA-vs-casacore residual (observer frames ~arcsec; B1950↔AZELSW up to ~4″). Still unsupported: the mean/true/natural frames JMEAN, JTRUE, JNAT, BMEAN, BTRUE, MECLIPTIC, TECLIPTIC. New testset with a seeded real-TaQL cross-check.
Phase 344 — meas.epoch as real casacore has it: seconds, source scale, sidereal scales (random fuzz vs real TaQL)
Following Phase 343, probing the other meas.* families against real TaQL showed meas.epoch('TARGET', value [, 'SOURCE' [, pos]]) takes and returns seconds since MJD 0 (so meas.epoch('TAI', TIME) works on a TIME column directly), accepts a source scale, and also converts to the sidereal scales GMST1/GAST/LMST/LAST (seconds of the sidereal day; the local ones need a position). TaQL-lite only had the day-valued meas.epoch('TAI', mjd). It is now the real form (the old day-valued call is gone: use meas.epoch('TAI', TIME) instead of meas.epoch('TAI', TIME/86400.0)); the sidereal scales use a new _sidereal helper (SOFA gmst06 / gst06a). 200 random conversions over UTC/TAI/TT/TDB/UT1 sources and ten targets agree with real TaQL (scales to ~ms, sidereal to ~50 ms). Left alone: meas.freq / meas.rv / meas.pos / meas.itrfxyz / meas.riseset use different calling conventions in real casacore (value-first, positions as [x,y,z], itrfxyz an identity on xyz, riseset(dir, epoch, pos) returning MJD days) and are a later phase. New testset with a seeded real-TaQL cross-check.
Phase 345 — real-casacore value-first meas.freq / meas.rv / meas.riseset / meas.pos (random fuzz vs real TaQL)
Real TaQL's spectral conversions are value-first, like the direction and epoch forms of Phases 343–344: meas.freq('TARGET', value [, 'SRC'] [, [ra, dec]] [, epoch] [, pos]) (also meas.frequency; meas.rv / meas.radialvelocity likewise), with the optional pieces told apart by type and no SRC meaning the value is already in TARGET. TaQL-lite only had the all-scalar meas.freq('S', 'T', f, mjd, x, y, z, ra, dec). Live probing also found that real's plain radial-velocity unit is km/s (LSRK→BARY shifts 1000 by 9.277, not 9277) and that its plain epoch is seconds. Added the real forms (the scalar forms are unchanged and still m/s with MJD days), plus meas.riseset([ra, dec] [, 'J2000'], epoch, pos [, elev0]) → [rise, set] MJD days (agrees with real to ~2 min) and meas.pos('ITRF', 'OBS') / meas.itrfxyz('OBS') → ITRF xyz of an observatory. 300 random frequency / radial-velocity conversions over TOPO, GEO, BARY, LSRK, LSRD, GALACTO, LGROUP, CMB agree with real TaQL (frequency to 3e-9 relative, velocity to 1 m/s). Not copied: real's meas.wgs, meas.restfreq, and the odd meas.rv('LSRK', v) with no source. New testset with a seeded real-TaQL cross-check.
Phase 346 — ALTER TABLE (probed and cross-checked against real TaQL)
TaQL-lite had no way to change a table's structure from a command. taql(table, "ALTER TABLE \$1 clause ...") now does what real ALTER TABLE does (32 commands compared on twin tables): ADD COLUMN name type [[NDIM=n, SHAPE=[..]]] [, ...] [DMINFO ...] (types B U1 I2 U2 I4 U4 I8 R4 R8 C4 C8 S and the long names; zeros / empty strings / false, undefined cells for an NDIM-only array column), DROP COLUMN a[, b], RENAME COLUMN a TO b[, ...], SET KEYWORD k=expr[, ...] (a constant expression; COL::k for a column keyword), DROP KEYWORD, RENAME KEYWORD a TO b, several clauses per command. Real quirks reproduced: setting an existing keyword moves it to the end of the set, integers are Int64 and a mixed [1.5, 2] is Float64, failures leave the table unchanged. The metadata operations are also Julia functions — renamecolumn!(path, old, new), setkeyword!(path, name, value; column), removekeyword!(path, name; column) — implemented as a table.dat rewrite that reuses the storage-manager blocks verbatim (a data manager addresses its columns by position, so no data file is touched; a virtual engine's stored-column link and Hypercolumn_* lists follow a rename). Not copied: real leaves an uninitialised fixed-shape column's cells as garbage (zeros here). New testset with a real-TaQL cross-check.
Phase 347 — CREATE TABLE / DROP TABLE as TaQL commands (cross-checked against real TaQL)
Continuing the TaQL command set after Phase 346's ALTER TABLE: taql("CREATE TABLE 'path' [A I4, B R8 [NDIM=1], C R4 [SHAPE=[2,3], UNIT=\"m\", COMMENT=\"..\"], ...] [LIMIT n] [DMINFO ...]") creates a table of default-valued rows (zeros, empty strings, false; undefined cells for an NDIM-only array column), with AS [storage="multifile", blocksize=n] and an IncrementalStMan DMINFO, and DROP TABLE 'path' / taql(path, "DROP TABLE \$1") deletes a table (never a directory that is not a table). Both work without a target table (taql("CREATE ...")). 24 CREATE forms compared with real TaQL: column types, shapes, units, comments, data managers and default rows agree; types are case-insensitive, names keep their case, LIMIT takes a constant expression, duplicate columns / unknown types / NDIM vs SHAPE disagreement / unsupported attributes are errors. Not copied: real leaves a fixed-shape array column's cells uninitialised, lists the DMINFO manager first, and segfaults on LIMIT -1 (an error here). Also: write_table gained shapes= and comments= keywords. COUNT results cannot be read back through Casacore.jl, so COUNT stays unimplemented.
Phase 348 — CREATE TABLE random-spec fuzz vs real TaQL
60 random CREATE TABLE specs (12 types, NDIM/SHAPE, UNIT, COMMENT, LIMIT incl. 0/absent, DMINFO) compared with real TaQL. Fixed one bug: a variable- or fixed-shape array column with no rows ([A R4 [NDIM=2]] without LIMIT, or LIMIT 0) had element type Any and failed to write. Also probed meas.jmean/jtrue/jnat/bmean/btrue/mecliptic/tecliptic: real TaQL's meas.* does not expose those frames at all, so they are not a TaQL gap (only reachable through casatools measures).
Phase 349 — ALTER TABLE random-clause fuzz vs real TaQL
Random 1–2 clause ALTER TABLE commands (columns and table/column keywords) compared with real TaQL, outcome and resulting table. Fixed: SET KEYWORD replaces an existing keyword in place (the Phase 346 "moves to the end" claim was a misread of Dict order) and refuses a value of another data type (Int→Double, String→Int, scalar→array; a one-element array replacing a scalar is that scalar); RENAME KEYWORD keeps the keyword's position (new exported renamekeyword!); RENAME COLUMN X TO X is an error; each ADD COLUMN clause needs its own DMINFO [..]; and a command whose later clause fails leaves the table untouched (all clauses are checked on the column/keyword names first, _alter_dryrun). Also listed the Phase 346 verbs in the API docs. Real TaQL's RENAME KEYWORD a TO b onto an existing b writes a duplicate key; not copied (error).
Phase 350 — sub-query random fuzz vs real TaQL (+ docs CI fix)
Random SELECT ... FROM [(SELECT ..)] WHERE x [NOT] IN (SELECT ..) / [NOT] EXISTS (SELECT ..) queries compared with real TaQL (640 queries over 32 tables). Fixed: sub-queries in the WHERE of a FROM (SELECT ..) query name the original table, not the inner selection; EXISTS (... LIMIT n) is false when fewer than n rows match (real errors for the positive form). Also fixed the taql docstring, which a Phase 347 comment had detached from its method — that broke the Documenter build (no docs found for 'taql') in CI on main.
Phase 351 — string / string-array expression fuzz vs real TaQL
630 random expressions over a string column and a string-array column (upper/lower/trim/capitalize/sreverse/substr/replace/+/iif/string/strlength, ==/</LIKE/~/IN/bool) compared with real TaQL. Fixed: LIKE / ILIKE / ~ / !~ map elementwise over a string-array cell (they raised a TypeError), and on an array cell substr does not clamp — a start beyond an element's length is an error, as in real TaQL (a scalar string never errors).
Phase 352 — complex expression fuzz vs real TaQL (no bug found)
1100 random complex scalar / array expressions compared with real TaQL. No TaQL-lite bug; the handful of differences are floating-point noise in casacore's own complex functions (sqrt through the polar form, exact equality of C**2 and conj(C)**2, Inf/NaN intermediates of division by zero), so the guard avoids those forms.
Phase 353 — date / time / angle function fuzz vs real TaQL
1400 evaluations of the date, time and angle functions over random MJDs and angles compared with real TaQL. Fixed: dms of an angle beyond 999 degrees prints *** in its 3-wide degree field. Not copied: real's one-day error and wrong time() sign for negative MJDs (before 1858) and 24h00m00 for exact full-turn multiples in hms; datetime(<number>) stays lenient (real errors).
Phase 353 follow-up — Linux x86-64 guard fixes (test only)
The Docker run found two guards that depended on platform / Julia version: the Phase 349 ALTER TABLE fuzz could rename a keyword onto an existing one (real TaQL writes a duplicate key; not copied), so renames now target fresh names; and the Phase 352 fuzz compared arg() of complex values, whose ±π depends on the sign of a zero imaginary part, so it is left out.
Phase 354 — quantity-literal fuzz vs real TaQL (no bug found)
800 random col op N<unit> / arithmetic / BETWEEN / IN conditions over length and angle columns with literals in other units agree with real TaQL. Not copied: real TaQL rejects a time- or frequency-unit literal (s, h, Hz, …) against a numeric column; TaQL-lite compares them after unit conversion.
Phase 355 — GROUP BY expression keys / ORDER BY / LIMIT fuzz vs real TaQL
550 random grouped queries (expression keys, aggregate expressions, WHERE, HAVING by alias, ORDER BY alias/key/DESC, LIMIT) compared with real TaQL in order. Fixed: an all-descending ORDER BY of a grouped result is the reversed ascending sort (fully tied groups come out reversed), as for a row ORDER BY (Phase 246); mixed directions keep ties in first-seen group order.
Phase 356 — result-column type fuzz vs real TaQL (SELECT .. GIVING)
800 random SELECT expressions: the type and scalar/array nature of the persisted result column compared with real TaQL's GIVING table. Fixed: iif returns the promoted type of its branches and a scalar beside an array fills the array's shape; complex values order by magnitude in < <= > >=; a fixed-shape array column with one element per cell (the result of collapsing the only axis, sums(AF, 1)) crashed the StandardStMan writer. Not copied: real's VariableDims declaration for computed array columns, its errors for some Float32 / Int16 mixes, and real(<string>).
Phase 357 — GROUP BY aggregate result-column fuzz vs real TaQL (no bug found)
1300 random g*(col) / gs*(arraycol) aggregates: result column type, scalar/array nature and values compared with real TaQL's GIVING table — no divergence. Not copied: real rejects Bool / String aggregates and gfirst / glast of an array column; TaQL-lite accepts them.
Phase 357 follow-up — Linux guard fix (test only)
The Docker run found the Phase 357 guard drew a case where gmaxs hit real casacore's all-negative-group DBL_MIN bug (Phase 284); gmaxs is now left out of the guard.
Phase 358 — JOIN column type / value fuzz vs real TaQL
540 random SELECT … FROM $1 a JOIN $2 b ON a.K == b.K queries over Int32 / Int64 keys and right columns of every type (compared in order with real TaQL). Fixed: an unmatched row of a right array column is an empty array (any array column in the right table made the whole JOIN fail with "column of type SubArray… is not supported"), and right-table Float32 / ComplexF32 columns are widened to Double / ComplexF64 like real TaQL's result columns. Not copied: real rejects Double join keys; TaQL-lite matches them.
Phase 359 — INSERT / UPDATE value-coercion fuzz vs real TaQL
550 random INSERT … VALUES / SET / LIMIT commands (ints, floats, strings, arrays into every column type) compared with real TaQL on twin tables, all cells and column types. Fixed: an integer into a narrower integer column wraps (SET H=40000 → -25536 in an Int16 column, U=-1 → 255); a fixed-shape array column takes a scalar (broadcast to every element) or an array of exactly its shape — a wrong-sized array was silently truncated by UPDATE and accepted by INSERT; an UPDATE scalar fills the cell's current shape (also in variable-shape columns). Not copied: real rejects Bool into numeric columns and a string / number into the wrong kind; TaQL-lite is more lenient.
Phase 360 — epoch-measure columns are dates (date-function fuzz on MS-style TIME columns)
Real TaQL types a column with an epoch MEASINFO (every MS TIME) as a date: year(TIME), mjd(TIME), date(TIME), TIME > datetime('…') and TIME > mjd('…') read its seconds as a date. TaQL-lite took the seconds as MJD days, so every datetime comparison on a TIME column was silently wrong. Fixed: in a query that uses a date function the epoch columns held in seconds are converted to MJD days (a bare number beside such a column is then days too — a documented difference), and mjd('<date string>') parses the string. 300 random date conditions over a seconds and a days epoch column agree with real TaQL.
Phase 361 — date functions on epoch columns inside GROUP BY
Date-function queries on MS-style TIME columns in SELECT … GROUP BY (compared with real TaQL, 120 queries): the Phase 360 seconds → MJD days conversion was lost on the expression-key path (the intermediate table has no MEASINFO), so GROUP BY year(TIME) gave year 13012392. Fixed: the intermediate table carries the epoch columns already converted when the command uses a date function. Not copied: real TaQL puts every row in one group for GROUP BY date(<epoch column>).
Phase 362 — DELETE with date functions / quantity literals
DELETE FROM t WHERE … loaded its columns without the parsed condition, so the Phase 360 seconds → days conversion of epoch columns and unit attachment of quantity literals did not apply (DELETE … WHERE year(TIME) > 2011 deleted every row). Fixed; checked against real TaQL for DELETE and UPDATE with year / month / weekday / datetime / mjd('…') conditions and for unit-literal conditions.
Phase 363 — date functions / quantity literals through a JOIN
SELECT … FROM $1 a JOIN $2 b ON … WHERE year(a.TIME) > 2010 kept every row and WHERE a.LL > 1km raised a DimensionError: the joined result is a plain GroupedTable that lost its source columns' units and epoch flags, and query(::GroupedTable) evaluated its WHERE on raw columns. A GroupedTable now carries per-column units and an epoch-column set (kept through JOIN, ORDER BY and WHERE filtering, and reported by columndesc as QuantumUnits), and its WHERE uses the same column loading (_tql_cols) as a table's — unit attachment, epoch-date conversion. Verified against real TaQL for year / datetime / mjd / unit-literal conditions and select expressions over the joined columns.
Phase 364 — tiled fixed-shape columns keep their FixedShape option on rewrite
Random-parameter sweep of real-casacore-created tables (random column mixes, StandardStMan BUCKETSIZE/BUCKETROWS/PERSCACHESIZE, IncrementalStMan bucket/cache sizes, TiledShapeStMan tile shapes and cache sizes, 1–3000 rows, mutated by TaQL UPDATE/INSERT/DELETE, then edited by us), ours vs Casacore.jl: values agree everywhere (75 cases). One schema bug found: _normalize_desc wrote option 0 for every column bound to a tiled shape / cell manager, so a casacore-made fixed-shape tiled column (option 4) turned into a declared-variable one whenever edit regenerated it (removerows!, …), and our own uniform tiled columns were written that way too — Casacore.jl then typed the column as 1-D and could not size it. The FixedShape bit is now kept for a fixed Dims shape. (Real casacore refuses DELETE on any table with a tiled column — not copied.)
Phase 365 — our tiled fixed-shape columns vs Casacore.jl across every type and shape
Follow-up to Phase 364: the type matrix had never compared tsm= columns with Casacore.jl (its declared-variable shape made them unreadable there). With the FixedShape option kept, 132 combinations (11 element types × 4 fixed shapes × tsm/tcm/tcell) written by us read identically in Casacore.jl — no bug; the type matrix now includes the fixed-shape tsm cross-check.
Phase 366 — variable-shape arrays (incl. tiled) written by us, read by real TaQL
Casacore.jl cannot read variable-shape tiled columns, so those were checked ours-only. Real TaQL can: 216 combinations (9 element types × 1–3 dims × tsm/tcell/ssm/ism) written by us give identical nelements / ndim / sum per row in real TaQL — no bug; a 7-type × 3-dim × 4-manager version is kept as a test.
Phase 367 — extreme values round-trip bit-exactly
NaN, ±Inf, −0.0, subnormal / maximal floats and integer limits written through StandardStMan and IncrementalStMan (scalar, fixed and variable arrays; both byte orders; ISM with long runs) read back bit-identically in our reader and Casacore.jl (132 combinations) — no bug; kept as a test.
Phase 368 — column keywords survive every table rewrite
Random column keywords (Int / Double / String / Bool scalars, string, numeric and Bool arrays) on SSM / ISM / variable-array columns stay intact through edit regeneration (removerows! + addrows!), addcolumn!, removecolumn!, renamecolumn! and copytable with and without a row selection (40 random tables) — no bug; kept as a test.
Phase 369 — casacore-made fixed-shape arrays stay indirect through a rewrite
Column descriptions (type, shape, option, comment, keywords) of random real-casacore-created tables (SSM / ISM, scalar / fixed / variable columns, units, comments) were compared before and after our edit regeneration and copytable (40 tables). The only drift: a casacore-made fixed-shape numeric array column is stored indirect (option 4, FixedShape without Direct) and we rewrote it as a direct column (option 5). _normalize_desc now keeps the indirect storage; descriptions are identical afterwards and the rewritten data reads identically in Casacore.jl (checked for 8 element types × SSM / ISM × 3 shapes).
Phase 370 — table-level metadata survives every rewrite
Table keywords (scalars, arrays, nested records, set by us or by real casacore ALTER TABLE … SET KEYWORD), type / subtype / readme, table name and comment and column order are identical after edit regeneration, addcolumn! + removecolumn! and copytable with and without a row selection (70 tables, SSM / ISM) — no bug; kept as a test. (Harness note: a real-casacore handle still alive in the creating frame stops a following ALTER from flushing; create tables in a helper function.)
Phase 371 — reference tables in non-ASCII directory names
Path handling swept across readers, edit, copytable, persisted RefTable / ConcatTable, reference_copy, copyms and moved directory trees, with plain, spaced, quoted, #, accented and CJK directory names, relative / .. / trailing-slash / symlinked paths. One bug: _strip_directory sliced the stored relative path with a character count used as a byte index, so under a non-ASCII directory write_reftable, write_concattable and reference_copy stored e.g. ./é/t instead of ./t and the result could not be reopened. Fixed (ncodeunits).
Phase 372 — LIKE / glob / regex match bytes, like casacore
Non-ASCII string literals and values in TaQL (WHERE, UPDATE, INSERT, DELETE; comparisons, IN, BETWEEN, string functions, LIKE / ILIKE, glob and regex forms) compared with real TaQL on random accented / CJK / emoji strings (3 seeds × 60 tables × 17 conditions + write commands). Everything agreed except that casacore's _ (LIKE), ? (glob) and . / [..] (regex) match one BYTE while ours matched one character ('ß' LIKE '__', 'ßñ' LIKE '_ñ%'). Patterns and subjects are now viewed bytewise (_bview), consistent with the byte-oriented string functions of Phase 334.
Phase 373 — backslash-escaped column names in TaQL
Real TaQL lets any character in a name be escaped with a backslash (a\ b, a\-b, x\:y), the only way to reference a column whose name is not a plain identifier; TaQL-lite rejected the backslash everywhere. The tokenizer now accepts escaped names, and the taql() command parsers unescape them in SELECT / INSERT column lists and SET pairs and in ALTER TABLE ADD / DROP / RENAME COLUMN and SET / DROP / RENAME KEYWORD (including COL::kw). 27 SELECT / UPDATE / INSERT / DELETE forms and 10 ALTER forms with a\ b, a\-b, x\:y agree with real TaQL. (Real TaQL rejects non-ASCII names and a bare keyword such as select; ours stays lenient.)
Phase 374 — a MeasurementSet acts as its MAIN table
nrow(ms) did not exist, and neither did most other table verbs on a MeasurementSet (only ms[:COL], ms.SUBTABLE and Tables.jl worked). An audit of 22 verbs found 17 raising MethodError. nrow, columnnames, columndesc and keywords now describe the MAIN table, and column / getcolumn / getcell / query (string and closure) / groupby / join / copytable / write_reftable / measure / measinfo / columnunit / qcolumn / rawblock / subtables / update! / delete! / insert! / taql / edit accept a MeasurementSet and act on its MAIN table (src/msforward.jl). The three-argument subtable forms (column(ms, "ANTENNA", "NAME"), …) are unchanged.
Phase 375 — UPDATE / DELETE on a RefTable or ConcatTable
An API-symmetry audit of 28 verbs across Table / RefTable / ConcatTable / GroupedTable found the read verbs uniform; the write commands were not. Real TaQL (checked on persisted RefTables) writes UPDATE through to the parent rows, applies DELETE to the reference only and rejects INSERT. update! / taql("UPDATE …") now accept a RefTable or ConcatTable (object or the path of a persisted one) and write through the edit views; delete! / taql("DELETE …") on a persisted RefTable rewrites its row list (a ConcatTable cannot lose rows, an in-memory query result has no directory). Nine UPDATE / DELETE forms (ORDER BY … LIMIT, slice assignment, array columns, IN) agree with real TaQL.
Phase 376 — corrupted table files raise errors, never crash or exhaust memory
A random-corruption fuzz (truncate / flip / zero a run / huge word / bump a word in every file of tables using every storage manager, big-endian, MultiFile, engines, Dysco, reference tables; 12,000+ cases) found three classes of bug in the readers: (1) a segfault — IncrementalStMan's whole-column read wrote out[r] under @inbounds with row numbers taken from the file; (2) a corrupt row count in table.dat made every whole-column read try to allocate tens of gigabytes (killed at 137) — casacore refuses such a table ("mismatch in #row"), and every IncrementalStMan / StandardStMan / TiledStMan now checks its own row count when it is opened; (3) corrupt element, axis, column, hypercube, IPosition and container-header counts preallocated gigabytes or built arrays / tuples of billions of axes (one read took 76 s) — they are now checked against the bytes that exist (_check_count, _chk_count). test/corrupt_tests.jl keeps a deterministic row-count test per manager and runs the fuzz in child processes (a deterministic ISM word sweep crashes the old code and passes now).
Phase 377 — TaQL-lite refuses garbage input with ordinary errors (parser mutation fuzz)
A mutation fuzz of the TaQL-lite parser and taql() dispatcher (about 8,000 mutated queries / commands plus deep-nesting, long-chain and huge-list garbage) found that extreme input crashed with exceptions that are not ordinary query errors, and that several of them differ from real TaQL (all checked live against tableCommand): (1) an integer literal beyond Int64 (decimal or hex) threw OverflowError; real TaQL saturates it to the maximum (later Int64 arithmetic wraps), and a float literal beyond the double range (1e999) is ±Inf and one below it (1e-999) is 0.0 (Julia ≥ 1.13 refuses to parse those); (2) a few thousand nested parentheses / function calls / `/NOT** overflowed the stack ("program state may be corrupted"); the parser now stops at 400 nesting levels with anArgumentError; (3) **$N` table references** that name no table were ignored (`SELECT FROM $2`) or crashed (`$0indexedothers[-1]); real TaQL says "Invalid temporary table number" and so do we; (4) **array subscripts**: an out-of-range index raised a rawBoundsError,V[1-1e999]anInexactError,V[3:1]silently selected nothing andV[0]was an error; real TaQL resolves **0 to the last element** (it subtracts the origin 1, so 0 becomes -1), counts negatives from the end, and errors for out-of-range indices, a range ending before its start, a stride ≤ 0 and non-integer subscripts — 20 forms now agree; (5)any/all/ntrue/nfalse(and thesforms) of a non-Bool operand raisedTypeError/MethodError; real TaQL rejects them ("function argument is not Bool"), now with anArgumentError; (6) an empty command and a JOIN condition naming an unknown column (KeyError) are clear errors. The slow cases the fuzz reported (6–15 s) were first-call compilation, not algorithmic. Also loosened a stale 1e-6 tolerance in the (SOFA-gated, normally skipped) Phase 249meas.galacticcheck: real casacore's J2000→GALACTIC differs from SOFA'sicrs2gby 0.3″. A seeded mutation-fuzz guard (test/taqlfuzztests.jl) asserts that garbage input never produces a stack overflow,BoundsError,InexactError,OverflowError,KeyError,StringIndexErrororDomainError`.
Phase 378 — garbage MSSelection specs (mscal.<sel>('…') mutation fuzz)
About 5,000 mutated and random specs for every selection function (baseline / field / spw / chan / scan / state / array / obs / time / uvdist / corr / feed). Two failure classes: ids beyond the machine range ('0~99999999999999999999', '>29999999999999999999') raised OverflowError, now saturating like TaQL's own integer literals (Phase 377); and numeric ranges were expanded integer by integer, so '0~199999999' took 9 s and a 20-digit upper bound exhausted memory — ranges now expand against the ids that exist (plus the in-between ones only for small ranges). An out-of-range mscal.time date field (InexactError) is an ordinary ArgumentError. Seeded guard in test/taql_mscal_tests.jl.
Phase 377 (x86 CI follow-up) — Dysco cell shape checked against the stored block size and the DATA column
The x86-64 Docker run of the Phase 376 corruption fuzz found a flipped bit in table.dat that gave a Dysco WEIGHT_SPECTRUM column a 9764866 × 4 cell shape: a weight block holds one value per channel, so nothing contradicted it and the read built thirty 156 MB zero-filled cells (0.6 s on the ARM dev machine, 35–80 s and swapping on x86). open_dyscostman now requires every column's stored block size to match the one its shape, bit widths and normalization imply, and a weight column's shape to equal the DATA column's of the same instance. The corruption-fuzz child also re-times a slow read before flagging it.
Phase 379 — a MEASINFO that does not fit its column is an ordinary error
A fuzz of garbage measure descriptions (about 2,500 random kind / frame / VarRefCol / unit specs written onto columns of strings, integers, scalars, empty, short, NaN and huge data, then read with measinfo, measure and measconvert) found that a declared-measure column whose cells cannot be that measure raised MethodError (_scalar(::String), _vec3(::Float64), _lonlat), BoundsError (a 2-element cell for a 3-vector measure), KeyError (a VarRefCol naming a column that does not exist) or MethodError again (float ref codes); TabRefCodes longer than TabRefTypes indexed past the end; and a date before SOFA's tables (MJD 1) raised AssertionError from inside SOFA.jl. All are now ArgumentErrors that say what was wrong. Test in test/measures_tests.jl.
Phase 380 — the Julia API refuses invalid arguments with ordinary errors
A fuzz of about 8,000 calls to the Julia API (reads, copytable, write_reftable, query / groupby / join keywords, every edit verb, update! / insert! / delete!, write_table, create_ms, taql) with out-of-range, non-integer, huge, wrong-typed and unknown arguments found: addrows!(t, 2^40) and create_ms(nrow = 2^40) got the process killed (out of memory) — both now check the row count against a memory budget and say how much would be needed, and create_ms validates its sizes; copytable(rows = [out of range]) wrote a table with no columns and only a warning ("skipped unreadable columns") — copytable / write_ms now validate rows= (and subtable_rows=); removerows!(Inf), assigning Inf / 1.5 / 3e9 to an Int32 cell, and insert! of a range or an unconvertible number into a scalar column surfaced as InexactError from deep in the writer — all are ArgumentErrors at the call. Seeds 1–20 of the fuzz are clean afterwards. Test in test/robustness_tests.jl.
Phase 381 — extreme numeric arguments of TaQL-lite functions
A fuzz of about 6,000 expressions whose function arguments are sizes, widths, indices and angles of 1e308, 2^63-1, NaN, ±Inf, fractions and negatives found: array(1.0, 99999999999) (and resize) got the process killed by allocating the array — a single TaQL array may now use at most an eighth of the machine's memory; integer conversions of huge or fractional arguments (substr('abc', 1e308), runningsum(V, 0.5), V[1e308]) raised bare InexactError — whole numbers are required (clear ArgumentError) and huge ones saturate; year(1e18), cdate, hms, dms(99999999999) overflowed Int64 and now degrade like NaN (Phase 193); fractile(x, 1e999) / fractile(x, 1.5) raised InexactError / returned garbage — a fractile must lie in [0, 1]; angdist(Inf, …) raised DomainError where C++ gives NaN. Six seeds of the fuzz are clean afterwards. Test in test/taql_query_tests.jl.
Phase 382 — docstring placement lint (the Documentation job failed on main after Phase 380)
Phase 380 inserted _check_copy_rows and _check_row_budget between the copytable / addrows! docstrings and their methods, so Julia silently dropped both docstrings and the Documenter job failed on main (no docs found for 'copytable' / 'addrows!') — the fifth time this mistake (a comment or helper between a docstring and its definition) has broken the docs build (Phases 160, 212, 228, 379). The helpers moved above the docstrings, and test/docstring_tests.jl now scans every file in src/ and ext/: the line after a docstring must contain the name its signature line starts with and must not be a comment. The check fails against the Phase 380 layout. The "merged" step's CI check caught this; a local docs/make.jl run before pushing would have too.
Phase 383 — junk column data given to write_table
A fuzz of about 3,500 write_table calls with junk column data (scalars, Dates, Symbols, missing, nothing, matrices, arrays of arrays, mixed and wrong-length vectors, Inf / NaN) under every storage option (tsm / tcm / tcell / ism / dysco / engines / virtualtaql / units / measures / shapes / multifile / both byte orders) found bare MethodErrors from collect(::Date), _measure_column_spec(::Matrix), Int64(::Vector) (a cell that is an array of arrays), vec(::Float64) (a scalar column named in a tiled group or in shapes=), size(::String) and zero(String) (a string column in a tiled group, which then failed on read). Column data must now be a vector (a generator or tuple is still collected), a cell must be a scalar or an array of scalars, tiled storage managers refuse scalar and string columns (casacore's tiled managers hold numbers and Bool arrays), and shapes= only names array columns. Seeds 2–7 of the fuzz are clean afterwards. Test in test/writer_tests.jl.
Phase 384 — wrongly-typed arguments to the query verbs
A fuzz of about 5,000 calls to query / groupby / join / update! / delete! / insert! / taql with arguments of the wrong Julia type (numbers, nothing, missing, tuples, pairs, closures and Dicts where a column name, expression or collection of names is expected) found bare MethodErrors from a String(x) deep inside: groupby keys, select names and expressions, where / having, orderby entries, grouping_sets, cols; join column selectors and oncols; update! SET targets, expressions and where; insert! column names. All now raise an ArgumentError naming the argument (new _tql_name / _tql_check_cond); a bare name is accepted for groupby(...; cols=). A string assigned to a numeric column (or a number to a String column) by update! / insert! / EditColumn is refused where it is assigned (was a MethodError from the writer at flush). Argument-type rejections by Julia's own dispatch or ::Type keyword annotations are left as they are. The long runs seen in the fuzz were first-call compilation, not a slow path. Test in test/robustness_tests.jl.