Phase 385 — TaQL-lite functions given operands of the wrong type
A fuzz of about 25,000 function calls (every function in the library, 0–4 operands drawn from string, numeric, Bool, complex, array and string-array columns and literals) found 142 in 3,000 raising a bare MethodError (sqrt('a'), year(S), upcase(I), replace(1, 2, 3), atan2(S, X), boxedmean(VS, 1) ...), plus TypeErrors (iif(I, a, b) with a non-Bool condition) and InexactErrors (marray(X, 2.5)) from inside the implementations. A function evaluated on operands it does not handle now raises an ArgumentError naming the function and the operand types (_tql_call, used by the row and group evaluators). Integer fmod(x, 0) is x like real TaQL (was a DivideError). Six further 24,000-call runs are clean. Test in test/taql_query_tests.jl.
Phase 386 — operators, group aggregates and meas./mscal. calls given the wrong operand types
Continuing Phase 385 with the rest of the expression layer: random operator expressions ('a' * 2, -S, NOT I, S < I, I BETWEEN 'a' AND 'b', V + VS, I LIKE 'a'), every group aggregate over operands it cannot reduce (gmean(S), gall(I), gany(S), gmeans(VS), gmedian(S)) and about 7,000 meas.* / mscal.* calls with junk arguments. Operators, aggregates, LIKE/~ and mscal.stokes on a non-array now raise an ArgumentError naming the operator (or aggregate) and the operand types (was MethodError / TypeError), meas.<frame>() with no argument, or a bare number where a [lon, lat] direction belongs, raises one too (was a BoundsError), and a function whose implementation hits an AssertionError (meas.lst(0, ...): UTC before 1960) reports it as an ArgumentError. Seven further 7,000-call runs of each fuzz are clean. Test in test/taql_query_tests.jl.
Phase 387 — every exported function called with junk arguments
A signature-driven fuzz of the whole exported API (about 100,000 calls: each exported function's own method signatures filled with type-correct but meaningless values — out-of-range rows, NaN/Inf, strings, other measure types, empty arrays, a Table, a MeasurementSet) found the API already well-behaved: the failures left were MethodErrors from the untyped arguments of a few helpers. A rest frequency given to doppler / radialvelocity / frequency / shiftfreq / restfrequency that is not a number or an MFrequency (_hz), and a freq or rm given to faraday_rotation / derotate_angle that is not a Real / MFrequency, now raise an ArgumentError saying what is required. Out-of-range rows (getcell(t, name, -1)) stay a BoundsError and a missing required keyword stays an UndefKeywordError — ordinary Julia. Test in test/measures_tests.jl.
Phase 388 — unusual column names through the whole pipeline (sweep, no bug found)
About 800 random column names (non-ASCII, emoji, spaces, dots, dashes, quotes, brackets, parentheses, colons, commas, %, =, <>, backslashes, tabs, the empty string) taken through write_table, readtable, real casacore (Casacore.jl), copytable, edit, addcolumn!, renamecolumn!, query, groupby and update! (the last three via the backslash escapes of Phase 373) all read back identically; no bug. A name that begins with a digit is the one thing TaQL cannot spell: real TaQL rejects \9a and reads 9a as the quantity 9 a (annum), while TaQL-lite resolves 9a to the column when one has that name — more lenient, left as is. A fixed list of 18 names is kept as a regression test in test/robustness_tests.jl.
Phase 389 — columns obey the AbstractVector indexing contract
A property fuzz of every column kind (Column, the lazy MappedColumn of a RefTable, an order-reversing RefTable, ConcatColumn, BlockColumn, a GroupedTable column) indexed with integers, ranges, step and reversed ranges, integer vectors, Bool masks, : and end, compared against the materialised Vector (about 9,000 cases), and of EditColumn assignment with the same index kinds (about 900 sessions), found one bug: a Bool mask (col[col .> 3], col[mask]) failed with ArgumentError: invalid index: true of type Bool on a plain Column and a ConcatColumn — Bool <: Integer, so the integer-vector method indexed by true / false. A mask now selects the true rows like any Vector (a wrong-length mask is a BoundsError, as for a Vector). Everything else — iteration, views, first / end, Tables.getcolumn, assignment through ranges, vectors and masks — already matched. Test in test/api_tests.jl.
Phase 390 — Tables.jl contract across every table kind
A check of the Tables.jl surface of every table kind (Table, RefTable ascending and reversed, ConcatTable, groupby and computed-select results, an empty result, a MeasurementSet) — schema vs column names, every value an instance of its schema type, getcolumn by name and index, columntable, rowtable, row iteration and property access, DataFrame conversion, matrix, partitions, and a write_table round trip of the result — found one bug: a MeasurementSet returned the struct's own field names from Tables.columnnames (:path, :data, :tables, ...) and had no Tables.getcolumn, so Tables.columntable(ms) / DataFrame(ms) carried the wrong column names and Tables.getcolumn(ms, :TIME) failed. It now delegates columnnames and getcolumn to its MAIN table like the rest of the interface. Everything else was already consistent. Test in test/tables_tests.jl.
Phase 391 — verb × table-kind symmetry
Every public verb (nrow, columnnames, columndesc, keywords, subtables, column, getcolumn, getcell, t[name], show, length, iteration, validate, query in all its forms, groupby (string and closure), join, copytable, write_table from a table source, write_reftable, resync, is_stale, measinfo, rawblock, edit, the Tables.jl conversions) was called on a Table, a RefTable, a ConcatTable, a GroupedTable and a MeasurementSet. Two gaps: a MeasurementSet had no length and could not be iterated (for row in ms), although every other table kind and the MS's own MAIN table could; and edit accepted a path, a MeasurementSet, a RefTable and a ConcatTable but not an open Table. Both added. The remaining refusals are intended (write_reftable / edit of an in-memory GroupedTable, rawblock of a RefTable / ConcatTable, which have no single backing buffer). Test in test/api_tests.jl.
Phase 392 — reading one table from many tasks at once
A stress test under julia -t 8 — 16 tasks reading random cells and whole columns of the same freshly-opened table (StandardStMan, IncrementalStMan, tiled, indirect arrays, strings, a compression engine, a VirtualTaQLColumn, and the sample MS) against the serially-read values — crashed about one run in two: a segmentation fault, or UndefRefError. The first access to each column races to open its data manager, and the opened managers lived in a plain global IdDict (_DM_CACHE) that every task mutated; the tiled managers' lazily-filled layout / mmap Dicts and VirtualTaQLColumn's multi-field lazy setup had the same shape. _dm_instance and the cache deletions in resync now hold a reentrant lock (_DM_LOCK; reentrant because opening a virtual engine opens its stored column's manager from inside open), the tiled caches are copy-on-write (a published Dict is never mutated, so readers need no lock), and VirtualTaQLColumn preparation is serialised. Six 60-round runs at 8 threads are clean afterwards. Test (a 4-thread child process, test/threads_child.jl) in test/robustness_tests.jl — a probabilistic guard: before the fix it failed in about one run in six at 15 rounds, and it runs 30. Reads were never promised thread-safe in the docs; they are now.
Phase 393 — parallel TaQL-lite queries and concurrent writers
Continuing Phase 392's thread-safety sweep to the rest of the shared state (every module-level mutable Dict / Ref). Twelve tasks evaluating TaQL-lite queries at once under julia -t 8 — including regex() / pattern() values, mscal.*, meas.*, mscal.stokes and aggregate expressions on the sample MS — failed in five runs out of five: the compile cache for regex() / pattern() / sqlpattern() values (_TQL_PATTERN_CACHE) was a plain global Dict mutated by every query that built a new pattern (AssertionError: Multiple concurrent writes to Dict detected!, UndefRefError). It is now guarded by a lock. The one-time IERS table refresh in the EarthOrientation extension (_ensure!, which downloads) is serialised too. Twelve tasks each writing, reading, editing, update!ing and copytable-ing their own tables at once, and reading the shared sample MS, were already clean (the lock registry and atomic file replacement hold). Test (a 4-thread child, test/threads_query_child.jl) in test/robustness_tests.jl; it fails three times out of three without the fix.
Phase 394 — tables are garbage collected (the data-manager cache leaked every table)
Opening a table, reading a column and dropping it 2,000 times grew the process by 1.6 GB (maxrss 544 → 2,187 MB) and left 2,000 entries behind: the opened data managers were cached in a global IdDict{Table,Dict{Int,Any}}, whose strong keys kept every Table ever read — together with its mmapped tile files and the whole-file buffers of its StandardStMan / IncrementalStMan instances — alive for the life of the process. A long-running session, a script that loops over many measurement sets, or a server would eventually exhaust memory. The cache is now a field of the Table (Table.dmcache; the 13-argument constructor still works), so it is collected with the table: the same loop grows by 65 MB, and resync / the new _evict_dms! empty the field instead of deleting from a global. Phase 392's lock (_DM_LOCK) still guards every access. Test in test/robustness_tests.jl.
Phase 395 — leak sweep beyond Phase 394 (no bug found)
After Phase 394's cache leak, a sweep for other resources that accumulate: open file descriptors and the lock registry over hundreds of written / read / edited / copied tables (separate files and MultiFile; MultiHDF5 too), the number of open HDF5 objects, and resident memory by table kind (separate files, tiled, incremental, MultiFile, MultiHDF5, compression engine, VirtualTaQLColumn, the sample MS) and operation (open and touch every column, whole-column reads, query, groupby, MeasurementSet with subtables, measures). Descriptors, lock-registry entries and HDF5 files stay flat; resident memory plateaus after the first few hundred operations in every case (an allocator / mmap high-water mark, not growth). A finalizer on the MultiHDF5 container was tried and dropped — the HDF5 files were already being closed. A constant-size guard (descriptors and _HELD_LOCKS after 120 tables) is kept in test/robustness_tests.jl.
Phase 396 — the documentation examples are run
Every code block in README.md and the index / concepts / guide pages (92 statements) was executed against the sample MS, and 83 now run unmodified; the other nine are illustrative fragments (an install snippet, placeholders such as pointing and apparent_flux, a hypothetical table with columns A and B). Three real problems turned up. ms.MAIN, which the README and guide use in eleven places for the MAIN table, raised KeyError — getproperty only knew the subtable keywords, so it had never worked; ms.MAIN now returns the MAIN table (and is listed in propertynames). The README and guide called Tables.schema / Tables.rowtable after using DataFrames, which does not export Tables; they now say using DataFrames, Tables. The do-block groupby example used mean without using Statistics; the example imports it, and the guide notes that groupby and nrow are also exported by DataFrames.jl (qualify them with MeasurementSets. when both are loaded). test/docs_examples_tests.jl keeps the examples honest: it runs every statement of those four pages (skipping the illustrative fragments by substring) and fails on any error.
Phase 397 — docstring signatures checked against the methods (no drift found)
The signature lines at the top of every exported function's docstring (106 of them: query(t, wherestr; select=...), power_response(beam, θ, freq=reffreq(beam)), ...) were parsed and checked against the real methods — some method takes that many positional arguments (counting x=default as optional positionals) and every keyword named is a keyword of some method, with the weak-dependency extensions loaded so the stubs' real methods exist. No docstring has drifted from its API; test/docstring_signature_tests.jl keeps it that way (a renamed keyword or a dropped argument now fails the suite). The same check found nothing wrong with the two exported types that carry hand-written signatures either.
Phases 392–393 follow-up — the thread-safety tests skip under Rosetta
The x86-64 Docker run on an Apple-silicon host (Rosetta, /proc/cpuinfo says VirtualApple) segfaulted in the two 4-thread child tests (Phases 392 and 393). It is the emulation, not the package: plain Julia code with no MeasurementSets in it — 16 tasks that mmap a file, compile a Regex and allocate — segfaults there with -t 2 and more (9 runs out of 9), and passes on one thread. The two thread-safety tests are skipped when that environment is detected; they run on real hardware, including CI.
Phase 398 — very large tables, and a commit that survives being killed
Large tables. casacore switches to 64-bit forms above 2³¹−1 rows (Table version 3, the ColumnSet row count, ISMIndex version 2, a 64-bit row-number flag in each ISM bucket, 64-bit row blocks in the SSM index, TiledStMan version 3) and above 2 GiB (TSMFile / TSMCube version 2 records). Tables built by casacore with 3×10⁹ to 5×10⁹ rows (an IncrementalStMan table stores only the changes, so it is a 33 KB file; a Bool column is 0.5 GB) were read back and compared cell by cell around 2³¹ and 2³²: StandardStMan, IncrementalStMan and TiledColumnStMan; and a 2.4 GB TiledCellStMan table whose hypercubes start beyond 2 GiB, plus a 2.8 GB single-hypercube TiledColumnStMan table. Our writer's output was read by casacore in both directions for the over-2 GiB tiled files and for a 2.2×10⁹-row IncrementalStMan table.
- Fixed: a tiled table with more than 2³¹−1 rows could not be opened ("corrupt hypercube dimensionality 0"):
TiledStMan::headerFilePutwrites the persistent cache size as a 32-bit uInt in every version, we read it as 64-bit in version 3. - Fixed:
editof a very large table ran for hours and grew until the machine swapped: the row map alone is 8 bytes per row, and every touched column was copied cell by cell (3×10⁸ rows: 150 s to materialise one scalar column, 245 s to flush). An untouched scalar column is now read in one pass (150 s → 2 s, flush 245 s → 31 s), and an edit session, a whole-column read andcopytablefirst check the memory they need against half the machine's memory and raise anArgumentError. - Fixed: writing 2³² rows or more failed deep in a writer with
InexactError: trunc(UInt32, …);write_tableandcopytablenow say so up front, before creating anything. (Tables of any size are read; casacore opens what we write up to 2³²−1 rows.) - Fixed: assigning a cell of the wrong size to a fixed-shape array column (
t[:V][6] = Float32[1, 2]for a(6,)column) was accepted and failed at flush with aBoundsErrorfrom the writer, losing the session's changes; it is anArgumentErrorat the assignment. - A casacore-written table with 5×10⁹ rows and one with 4.5×10⁹ tiled Bool cells are fixtures (
test/data/ism64.tbl,tcm64.tbl; the 563 MB tile file is created sparse by the test); the over-2 GiB tile file and cube offset test builds a sparse 3 GB file.
Crash consistency. Killing the process while an edit flush replaced several files used to leave a table that nothing could open ("mismatch in #row", or a BoundsError from reading one file with another's layout): the storage-manager files, table.dat and table.info were renamed one at a time, then the sync blob written. A kill after any rename but before the last (every kill point of an addrows!, removerows!, removecolumn!, …) failed this way. The files of one flush are now committed as a group: they are written under hidden names, a journal table.commit is renamed into place (the commit point), the files are renamed over their targets, the sync blob is written and the journal removed. A process killed before the journal leaves the old table; one killed after it is rolled forward by the next readtable / edit. A flush that fails while writing (a full disk) removes its hidden files and leaves the table untouched; edit also removes the hidden files of a writer that died before its commit point. Every kill point of eleven editing operations was tested by killing a child process: all end in either the old or the new table (before this, 51 of the 90 simulated kills did not).
- A flush that patches tiles in place AND replaces other files now takes the regenerating path, so it is all-or-nothing; a flush of tile edits alone is still patched in place, cell by cell, which no journal can undo.
- Not covered: power loss (files are not fsynced before the rename, as in casacore), and a casacore process that opens the table while a commit is half done.
Phase 399 — static analysis, a mutation fuzz of taql(), process races and path edge cases
JET (report_package, all extensions loaded) gave 251 reports over 2158 definitions: 236 are Union{Nothing,…} from regex captures and the lock's io field that the code guards, and the other 15 were read — all inference artefacts except two real gaps: insert!(target, source; limit = …) took no keyword arguments (Base.insert!(target, source) now passes them on, so insert!(ms, other; limit = 2) works), and a sub-query containing INTO (… IN (SELECT K INTO 'p')) reached nrow(::String); it is an ArgumentError. JET is not a test dependency (it tracks the Julia version closely).
Mutation fuzz of the taql() dispatcher (466 command strings taken from the tests; tokens deleted, duplicated, swapped, replaced or inserted; 43,000 mutants) found one bug: an array assigned to a scalar column — UPDATE t SET S=['u','v'], SET X=['a'], SET K=[1,2] — passed the cell check and failed at flush with a MethodError from the writer (or was silently reduced); it is now an ArgumentError at the assignment, for edit and update! alike. The rest were deliberate errors (a missing column or keyword is a KeyError). 600 mutants of 20 commands are a seeded guard in test/taql_fuzz_tests.jl.
Edit sessions belong to a task. Flushing an edit(path) session from a different task than the one that opened it used to commit and then fail to release the lock ("unlock from wrong thread"), leaving the table locked for the rest of the process; it is now refused before anything is written, with a message saying so. Setting cells from other tasks is unaffected.
A table opened before another writer committed and read afterwards (a long-lived handle) failed with "mismatch in #row" and no hint; the message now says the table was modified after it was opened and to reopen or resync it. In a stress run of one writer process (3,600 edits that each changed the row count and every column) against two reader processes, 22,000 fresh readtables gave no inconsistent state and no error; of 190 handles opened before a commit and read after it, 12 failed loudly and none returned wrong data.
Paths — directories named with spaces, quotes, #%, parentheses, non-ASCII, trailing dots; trailing and doubled slashes; ./.. and relative paths; a symlinked table or MeasurementSet; a MeasurementSet copied with cp -r; a RefTable moved together with its parent — all read, edit and copy correctly.
Phase 400 — Bool columns unpack eight to a store; allocation guards for the hot read paths
The Julia-vs-casacore-C++ comparison on the real 9.8M-row MS was re-run (whole columns, per-cell and random access, subtables; geometric mean 0.29 of C++'s time): everything is faster than casacore except one column — whole-column FLAG_ROW, a bit-packed Bool in StandardStMan, was 9.5× slower (3.8 ms against 0.4 ms). Its unpacker read every bit individually into a temporary vector that was then copied. It now unpacks straight into the result, eight Bools per 64-bit store through a 256-entry table (little-endian machines; others keep the bit loop): 3.8 ms → 0.27 ms, faster than C++. The same routine (_rd_bits!) serves the tiled FLAG reads, whose per-cell times are unchanged. A fuzz of 19,000 random byte/bit offsets and lengths against a bit-by-bit reference found no difference; reads past the bytes are still checked.
test/perf_guard_tests.jl asserts allocation (not time) on the committed sample MS — whole columns cost about their payload, a cell of an opened column a few KiB, subtable arrays and strings, readtable, query and groupby — at about three times today's figures, plus the bit-unpacker fuzz and Bool columns of awkward sizes. Every large slowdown found so far (Phases 234–239) was an allocation bug first.
Phase 401 — the element types real data has: Union{T,Missing} columns, and what to do about unsupported ones
A column read by DataFrames or CSV has the element type Union{Float64,Missing} even when nothing is missing, and write_table refused it with a message about Unitful quantities. A Union{T,Missing} column (or array cells with such an element type) with no missing value is now stored as T, in write_table and addcolumn!; one that does hold missing values is an ArgumentError naming the first row and the count — casacore columns cannot hold them (assigning missing into a cell, or insert!ing it, says the same). copytable of a joined result with unmatched rows therefore fails with that message instead of the Unitful one.
For element types casacore has no column for, the error now names the type, lists what casacore columns hold, and says what to do: Int8 → Int16.(x), UInt64 → Int64.(x) if the values fit else Float64.(x), Symbol / Char → String.(x), Rational / BigFloat / BigInt → Float64.(x), Date / DateTime → a Float64 or an MEpoch measure column, a tuple → an array per row. (Half-precision and complex-half floats, SubStrings, views and ranges were already accepted.) Nothing is converted silently.
Phase 402 — tables from every kind of Tables.jl source
Twenty-eight kinds of input were fed to write_table (a row table, a column table, a matrix table, a DataFrame and a view of one, dictionaries, vectors of pairs, query / join / groupby results, partitioned tables, tables with no rows or no columns, duplicate and empty names).
- Fixed: a row-oriented table — a
Vector{NamedTuple}, whatTables.rowtableand most JSON / CSV readers return — was written as a table with no columns and no error: the writer asked the source for its columns directly, and a row table has none (itscolumnnamesis empty). Sources are now read throughTables.columns(a source without column access is materialised withTables.columntable), so every Tables.jl table works. nrow=is optional for a table with columns: it is the first column's length (a table with no columns, or a bad source, still needs it or says so).- Fixed: duplicate column names (
[:A => x, :A => y]) wrote two columns with the same name; it is anArgumentError. An element that is not aname => vectorpair (a partitioned table passed ascolumns) is anArgumentErrorsaying whatcolumnsmay be. - Fixed: a column named
""was written without complaint, and both our reader and casacore then returned another column's data for it; an empty name is refused bywrite_table,addcolumn!andrenamecolumn!.
Phase 403 — whole-MS workflows
The read side was checked first — sum / mean / sort / broadcast / stack / reduce(hcat, …) over lazy columns, the REPL display of every table kind (including a 9.8M-row column: it shows a few rows and reads no more) — and found nothing. Then the things done with a whole MS: append one MS's rows to another, cut a selection into a new MS, flag or delete rows, add the calibration columns.
insert!(target, source)takes the path of a table or MeasurementSet assource(a string used to be taken as the values, and refused);INSERT INTO t SELECT FROM 'path' [WHERE …]accepts an empty column list (every column), as TaQL does.copymsaccepts an openedMeasurementSetas well as a path, socopyms(ms, dst; rows = query(ms, …).rows)cuts a selection into a new MS.MODEL_DATAandCORRECTED_DATA(what CASA calibration and imaging add; present in every casatools-simulated MS, which the Phase 330 comparison already listed) are optional columns of the schema, soaddcolumn!(t, "CORRECTED_DATA")works — it failed with "not a standard column". The default cells of those two and ofWEIGHT_SPECTRUM/SIGMA_SPECTRUMhave the shape of the row'sDATAcell (they used to be empty arrays).
Phase 404 — documentation completion
An audit of which exported names the prose pages (README, index, concepts, guide) never mention found 31 — mostly the persisting and in-place-editing verbs added after the guide was written. The Guide now covers them, all inside the executed docs-example test (every ```julia block runs against a copy of the sample MS):
- Persisting a selection:
write_reftable(dst, parent, rows)(a lazy row-reference table other tools can open),write_concattable(dst, parts)(a virtual row-wise concatenation), andwrite_ms(a whole MS, with row slices per subtable). - Tables.jl sources into
write_table:Tables.columnssources and row tables, optionalnrow=, and howUnion{T,Missing}columns are stored (asTwhen nothing is missing, a clear error naming the row when something is). - Editing in place:
setcell!/setcolumn!(with cell validation), editing through aRefTableview (edit(sel) do t …), and the alter verbsrenamecolumn!,setkeyword!,renamekeyword!,removekeyword!.
The docs build itself is back under Documenter's size limits with zero warnings: the generated changelog pages are cut smaller (CHANGELOG_CHUNK_BYTES 55 000 → 48 000, the largest page 103 → 90 KiB), the API reference's primary-beam section moved to its own page (api-beams.md; api-2.md 101 → 87 KiB), and the search-index warning threshold is raised to 2 MiB (the full changelog dominates the index; it only ever warned).
Phase 405 — opening something that is not a table, overwriting something that is
A probe of common user mistakes found three unclear outcomes. readtable / MeasurementSet / edit of a directory without a table.dat raised a bare SystemError (and edit of a missing path an IOError from readdir); they now raise an ArgumentError saying it is not a table directory (a directory holding an interrupted commit's journal still opens). write_table onto an existing table silently overwrote it, leaving the old storage-manager files behind; every other writer already refused an existing destination, so it now refuses too (a directory you created first, or one without a table.dat, is still fine).
Phase 406 — a closed edit session refuses further writes
Probing edit-session misuse found a silent data loss: after flush(t) (or an aborted do-block) the session had released its lock and committed its state, but t[:A][1] = 5, addrows!, removerows!, addcolumn! and removecolumn! on it still returned normally — the change was simply never written, and a second flush was a no-op. All of them now raise an ArgumentError saying the session is closed and to open a new one with edit(path). flush stays idempotent, reads from a closed session still work, and views onto a RefTable/ConcatTable inherit the check (they delegate to the same sessions).
Phase 407 — copy and persist verbs: overlapping destinations, out-of-range rows, zero-row create_ms
A second pass over user mistakes, this time with the copy/persist verbs, found four problems. copyms(src, joinpath(src, "inner")) succeeded and wrote the copy into the very MS being read; write_ms / copyms / copytable now refuse a destination inside the source or a source inside the destination (compared through realpath, so symlinks do not hide it), before anything is created. write_reftable with a row beyond the parent's length wrote a table that failed with a BoundsError on first read; the rows are now checked up front. create_ms(nrow = 0) failed with an obscure tiled-writer message because a zero-row array column has no cell to take its rank from; it now creates an MS that validates and can be extended with addrows!. A check that the parts of a write_concattable share one schema was tried and dropped: MMS parts legitimately differ in optional columns.
Phase 408 — comparing a string with a number is an error
Probing the query verbs with typical slips turned up one silent failure: query(t, "NAME == 5"), or a number quoted by mistake (ANTENNA1 == 'a'), returned an empty selection with no complaint, because Julia compares a string and a number as simply unequal. Real TaQL rejects all of these (checked live for ==, != and < between a string and a number, column and literal), so the comparison operators now raise an ArgumentError saying a string cannot be compared with a number — in WHERE, HAVING, select expressions and groupby, including string-array cells. Comparing strings with strings, numbers with numbers, and an unknown (missing) value are unchanged; the Phase 386 test that had pinned S == I to "all false" was updated. The other probes (keyword typos, wrong-typed select/rightcols, unknown columns, aggregate over strings, name clashes) already gave clear errors.
Phase 409 — measures refuse nonsensical inputs
Probing the measures API with plausible slips found values that were accepted and produced NaN or garbage: a direction with a latitude beyond the poles (MDirection{J2000}(0.0, 5.0), e.g. degrees given as radians) converted to a Galactic position that meant nothing; doppler(f, 0.0) gave NaN and doppler(f, -1e9) a silent 0.0 (the shift is squared, so the sign vanished); earthfield at the Earth's centre returned a NaN field. All now raise an ArgumentError naming the offending value: measconvert checks the latitude of the measure and of frame.direction (a round-off excess of 1e-9 rad is tolerated), doppler(f, rest) requires finite positive frequencies, earthfield a finite position away from the centre. The other probes (missing frame parts, unsupported frame for a measure kind, unknown columns, wrong precision, bad beam widths, unknown observatory → nothing) already behaved clearly. An epoch outside the IGRF's 1900–2030 span still extrapolates.
Phase 410 — taql write commands: a repeated INSERT column, SELECT … INTO inside the source
Probing the taql write and DDL commands with plausible mistakes (28 cases) found two silent outcomes; the rest (missing SET, wrong value counts and types, unknown or duplicate columns, existing destinations, unknown DROP TABLE targets, a plain directory handed to DROP TABLE) already gave clear errors. INSERT INTO t (A, A) VALUES (1, 2) inserted a row, silently keeping one of the two values; a column named twice is now an ArgumentError. SELECT … INTO '<path inside the table>' wrote the new table inside the table being read; Phase 407 had closed this for copytable of a plain table, but a query result (a RefTable, which has no path of its own) slipped through. copytable now checks every on-disk table the source reads from, through nested RefTables and ConcatTable parts.
Phase 411 — write_table option misuse is refused up front
Probing the storage and metadata options of write_table (25 cases) found silent acceptance of nonsense. A column named in two storage-manager groups (tsm=[["V"],["V"]], tsm= with ism=, tsm= with tcm=) was written under one of them without a word; it is now an ArgumentError naming both groups. endian = :middle was accepted (as little-endian); nrow = -1 reported "3 values, expected -1"; both now have their own messages. measures = Dict("X" => (; kind = :epohc, ref = "UTC")) or a reference frame that does not belong to the kind (ref = "NOPE") wrote a MEASINFO nothing can read back; the kind and fixed reference are now checked against the known frames (a MeasInfo struct is still written as given, for crafting odd fixtures). A single name (tsm = "V", tsm = [:V]) is accepted as one group instead of failing with a MethodError. Everything is checked before the destination is created.
Phase 412 — assigning into a read-only column says how to edit
A pass over read-only misuse (assigning into or resizing a column, assigning a column into a table or MS, editing through an MeasurementSet, mutating columnnames/keywords, show/copy/collect/broadcast of lazy columns; 30 cases) found everything already failing or behaving sensibly, with one weak message: column(t, "TIME")[1] = 0.0 raised Julia's CanonicalIndexError: setindex! not defined for Column{Float64}, which says nothing about what to do instead. Assigning into any lazy column (Column, MappedColumn, ConcatColumn, BlockColumn, so also fill! and friends) now raises an ArgumentError pointing to edit(path) do t; t[:COLUMN][i] = value; end. Edit-session columns are unaffected.
Phase 413 — opening a table is 4.5x faster
The one remaining benchmark line where the package trailed real casacore was table open (metadata parsing), left unchased since the Phase 234–240 performance rounds. Profiling readtable showed the time sat in the AipsIO primitive readers: the reader's io field was typed as the abstract IO, so every read_scalar / read_string of a table description was a dynamic dispatch. AipsIO is now parametric (AipsIO{I<:IO}), so the field is concrete and every read is statically dispatched. Opening the sample MAIN table went from 441 µs to 98 µs (98 → 65 KiB allocated), ANTENNA 212 → 74 µs, SPECTRAL_WINDOW 416 → 97 µs, POINTING 390 → 84 µs; the storage-manager headers read through the same type get the same speed-up. No behaviour changed. A guard test checks the field stays concrete and bounds the allocation of an open.
Phase 414 — WHERE conditions run 20–150× faster: compiled row predicates
A throughput benchmark of the query engine on a 1M-row table (the first one since the engine was written) showed 130–380 ns per row for a plain WHERE, 65–250 B allocated per row. The interpreter walks the AST once per row: every node is a dynamic dispatch on an abstract TQLExpr field, every column reference a Dict{String,…} lookup and a boxed value. _compile now turns the common nodes (column, literal, comparison, arithmetic, AND/OR/NOT, negation, BETWEEN, IN over plain literals, LIKE, function calls, rownumber()) into nested closures over the concretely typed column vectors, once per query; any other node falls back to _tqleval, so the semantics are the interpreter's by construction (checked on 22 expressions against it, plus the 6600 existing query/command/fuzz assertions). Over 1M rows: A == 3 137 → 1 ms, B > 0.5 AND C < 0.5 337 → 7 ms, A IN [1,2,3] 128 → 29 ms, ORDER BY after a filter 172 → 39 ms, sqrt(B)+C > 1 383 → 178 ms (function dispatch still dominates). Used by query, the groupby where, join conditions, update! right-hand sides and computed select columns. Compilation only happens for tables of ≥ 2000 rows and expressions of ≤ 40 nodes (COMPILE_MIN_ROWS, COMPILE_MAX_NODES): below that the interpreter is as fast, and a very large expression would only make the compiler's closure types huge. Aggregate arguments inside groupby still use the interpreter.
Phase 415 — copying a scalar column is twice as fast and allocates a fifth
Profiling copytable of a 1M-row table (the second throughput target after Phase 414's WHERE) found three avoidable costs in the path every copyms, copytable and SELECT … INTO takes. _read_cells boxed every cell of a scalar column into a Vector{Any}; a whole scalar column now keeps its concrete element type. The StandardStMan writer then converted and wrote each cell through a dynamic dispatch on that Any; the loop now sits behind a function barrier on the element type (_ssm_put_scalars!). And _wrbytes! built a temporary one-element array for every value it wrote; for the plain bits types it is now a bounds-checked unaligned store. Copying a 1M-row, four-column table: 654 → 314 ms and 900 → 168 MiB allocated; a two-column table went from 490 to 106 B/row (the string column is unchanged). The bytes written are identical (a test compares the fast path to the generic one for every bits type in both byte orders). update! of a whole column is still ~1.2 s per million rows, dominated by the edit session's rewrite, not the evaluation.
Phase 416 — ORDER BY is 20× faster
The next throughput target after Phases 413–415, from a 1M-row benchmark of the rest of the engine, was sorting: ORDER BY B on one numeric key took 1.1 s and allocated 1.2 GiB. The sort was a merge sort over a comparator closure that, for every comparison, looked each key up in a Dict{String,…} and read two cells out of an untyped vector. It now sorts by least-significant-key-first passes: each key is gathered once into a concretely typed vector and sortperm'd (Julia documents sortperm as stable, also with rev = true), which gives exactly the old ordering — isless per key, ties in the original row order, all-descending keys the reversed ascending sort (Phase 246), missing last ascending, NaN after every number. Over 1M rows: ORDER BY B 1095 → 44–58 ms (1.2 GiB → 63 MiB), ORDER BY K, B DESC 37 ms, a string key 217 ms, and groupby(...; orderby = …) shares the same sort. An ORDER BY <expression> key is now evaluated by the compiled evaluator of Phase 414 (330 → 152 ms for sqrt(B)+C). A randomised equivalence test compares the new sort with the old comparator on 600 random key sets (ties, NaN, ±0.0, missing, strings, mixed directions, shuffled input rows), and an allocation guard bounds the bytes per row. Still open from the benchmark: update! of a whole column (~1.2 s per million rows, the edit session's rewrite) and copying a string column (670 ms vs 314 ms for numeric columns).
Phase 417 — aggregate arguments and function calls in compiled expressions
Two leftovers of Phase 414's compiled predicates. A groupby aggregate argument (gsum(B*C)) was still interpreted per row and boxed into a Vector{Any} before the reduction; on large tables it is now compiled once per query (cached per column dictionary, so the entry goes away with it) and collected into a concretely typed vector. And a compiled function call (sqrt(B), iif(...), pow(...)) went through a dynamic call on an abstractly typed callable plus a map over a tuple; the callable is now a type parameter of the closure (_mk_funcall, with one- and two-argument specialisations) and the error wrapping (does not accept operands of type …) moved to a shared _tql_call_fail, unchanged. Over 1M rows: groupby … gsum(B*C) 312 → 115 ms, gmean(B) 165 → 105 ms, gmean(sqrt(B)+C) 415 → 159 ms, WHERE sqrt(B)+C > 1 234 → 18 ms (13×), ORDER BY sqrt(B)+C 152 → 60 ms. Results are checked against direct reductions and the earlier interpreter's error messages; an allocation guard bounds both paths.
Phase 418 — whole-column update! and string-column copies
update!(path; set = ["B" => "B*2.0"]) on a 1M-row table took 1.5 s and 1.2 GiB. Profiling split the cost three ways. The update itself kept the current value of every written cell in a Dict{Tuple{String,Int},Any} (a million entries) and called a boxed fit_cell closure per cell even for scalar columns; the whole-column value is now one vector, later SET items still see it (a test chains a whole-column write, a slice write and a WHERE-restricted write on the same column), and scalar columns skip fit_cell. Assigning a whole column in an edit session (t[:B][:] = vals) validated and stored cell by cell after reading the old column only to overwrite it; it now validates once and replaces the override. And rewriting the column's StandardStMan file copied every inline string into a per-row Vector{UInt8} held in a Dict; the string is now written straight from the source, and a little-endian Vector of plain numbers is written with one memory copy per bucket (the generic bounds-checked loop remains for big-endian or boxed input, with a test that both agree for every integer/float type). Over 1M rows: whole-column update! 1476 → ~1020 ms and 1.2 GiB → 0.7 GiB, copytable with a string column 670 → 559 ms (386 MiB). What remains is inherent to the edit session: it regenerates the whole storage-manager file, so updating one column rewrites its siblings too, and a WHERE-restricted update still reads the column back as boxed cells (≈1.8 s per million rows).
Phase 419 — an edit-session flush no longer boxes untouched scalar columns
Phase 418 left a restricted update! (SET B = B*2.0 WHERE K == 3, ~1% of the rows) at 0.73 s per million rows, and profiling showed the time in the storage-manager writer's generic loop: the flush resolves every column of the regenerated file, and an untouched scalar column was read in bulk and then boxed into a Vector{Any} (_read_cells_any), which the writer then had to unbox value by value and could not copy in one go. _resolve — read-only, used only by the writers and the engine re-encoder — now hands the typed vector straight through (concrete = true); _materialize!, which the edit setters mutate, still gets the boxed copy. Over 1M rows the restricted update went 729 → 580 ms (577 → 507 MiB), a whole-column update! 1018 → 843 ms (729 → 661 MiB), and K = K + 1 1214 → 1013 ms, so a whole-column update is now about 1.7× faster than at the start of Phase 418. A test checks the untouched columns resolve typed, an edited column still reflects the edit, and an update with an appended and a removed row writes the right values.
Phase 420 — joins look their keys up in bulk
Having found ISM writes already cheap (1M rows × 3 columns in 200 ms), a benchmark of the rest of the engine's verbs found the equi-join as the slow one: a 1M-row table joined to a 100-row table took 426 ms. The match step read the left key column cell by cell through the lazy Column (a storage-manager lookup per row) and looked each key up as a boxed tuple in a Dict{Any,Int}. The key columns are now bulk-read (_load_col) and looked up by a Dict keyed on the plain element type (a typed tuple for composite keys); the M:N (multi = true) index is typed the same way. 1M × 100 N:1 join 426 → 9 ms (54×), multi = true 443 → 42 ms; the index-lookup on = "ANTENNA1" form reads its key in bulk too. A randomised test checks every match against a naive search — Int64 keys against Int32 and Float64 keys, strings, composite keys, a missing left key, duplicates in the M:N case, and the refusal of a non-unique right key — and an allocation guard bounds the bytes per row.
Phase 421 — reading one numeric cell is 3–6× faster
Random access to single cells (column(t, "TIME")[i], getcell) allocated 80–96 B and took 160–200 ns for a plain numeric StandardStMan cell: the generic reader built a one-element vector (and looked up the column's dimension tuple) to take [1] of it, then the Column dispatched dynamically on its untyped data-manager instance. getindex(::Column{T}, i) now takes a typed path for a numeric scalar column on a StandardStMan (no narrowing): the element type comes from T, so the cell is a single bounds-checked typed load — 159 → 51 ns for Int32, 200 → 33 ns for Float64, 0 allocated bytes. The data manager's own getcell has a scalar method too (the route getcell(t, name, i) and the TaQL engine take), 80 → 16 B. A test reads rows around bucket boundaries of every numeric type (including complex and big-endian files) and compares them with the generic reader; an allocation guard bounds the cost. Strings and Booleans keep the generic path.
Phase 422 — removerows! and addrows! are 4.5× faster
Structural edits regenerate the storage-manager files of the table, and the profile of removing 1000 rows from a 1M-row table (1.56 s) showed the time in reading the surviving cells back: once the row map is no longer the identity, _read_cells_any read each scalar cell individually through the lazy column (a storage-manager lookup per row) into a boxed vector, and the flush then re-boxed it again into a Dict{String,Vector{Any}} before the writer unboxed every value. A scalar column is now read in bulk and gathered through the row map into a typed vector (appended rows take the column's default, typed too), and the flush keeps the resolved vectors typed. Removing 1000 of 1M rows 1557 → 343 ms (786 → 367 MiB), appending 1000 rows 1588 → 349 ms (817 → 409 MiB); together with Phases 418–419 an edit session's flush is now dominated by the unavoidable file rewrite. A randomised test removes and appends rows on SSM and ISM tables (both byte orders) with Int32, Float64, String, Bool and array columns and checks every value, including the defaults of the appended rows; an allocation guard bounds both operations.