API reference (part 2)

write_table(dir, name, columns; nrow, endian=:little,
           tsm, tcm, tcell, ism, engines, virtualtaql, dysco, dysco_spec,
           measures, units, storage=:sepfile, blocksize=DEFAULT_MF_BLOCKSIZE,
           type="", subtype="", readme="")

Write a CTDS table at dir. columns is an iterable of name => vector pairs (or a Tables columns source). Column metadata (units, comments, exact class names) is taken from SCHEMAVER2[name] when available.

dysco/dysco_spec compress one or more columns with DyscoStMan – see _write_table_core for the exact shape. virtualtaql maps a column name to a TaQL-lite CALC expression (a VirtualTaQLColumn – the passed values for that column are ignored, only the expression is stored). storage/blocksize pack every StandardStMan/IncrementalStMan/TiledStMan private file into one table.mf/table.mfh5 – see _write_table_core.

A column whose Julia element type is a Unitful quantity or a Measure (MEpoch{UTC}, MDirection{J2000}, MPosition{ITRF}, MFrequency, MRadialVelocity) is stored as plain numbers with its QuantumUnits (and, for a Measure, MEASINFO) column keyword stamped automatically – so write_table -> readtable -> qcolumn / measure round-trips. measures = Dict(col => (; kind, ref[, varrefcol, units])) and units = Dict(col => "Hz" | ["rad","rad"]) stamp those keywords explicitly and override the auto-detection for that column.

When name is a standard Measurement Set table ("MAIN", "ANTENNA", "FIELD", ...) its columns also get the standard QuantumUnits / MEASINFO / CATEGORY keywords casacore requires of an MS (standard=false turns that off), so a table written piece by piece is a valid MS.

A table keyword (keywords = Dict(name => value)) may be a number, string, array, Dict / Record, or MeasurementSets.SubTable("././ANTENNA"), a link to a subtable directory (relative to this table's directory), so a MAIN written into a directory that already holds the subtables becomes a Measurement Set.

write_ms(dir, ms::MeasurementSet; rows=Colon(), subtables=Colon(),
        storage=:sepfile, blocksize=DEFAULT_MF_BLOCKSIZE)

Write ms to a new MeasurementSet directory dir: MAIN restricted to rows, and every subtable in full (or only those named in subtables, a collection of keyword names). A column the reader cannot decode is skipped with a warning. storage/blocksize, if not :sepfile, pack every table written (MAIN and each subtable) into its own table.mf/ table.mfh5 – matching what a real casacore MS created under a global StorageOption would look like (one container per table, not one shared container for the whole MS tree).

copyms(src, dst; rows=Colon(), subtables=Colon(),
      storage=:sepfile, blocksize=DEFAULT_MF_BLOCKSIZE)

Copy the MeasurementSet at src to a new directory dst (MAIN rows optionally sliced; subtables optionally restricted to a set of names). storage/blocksize – see write_ms.

copytable(dst, t; rows=Colon(), name="TABLE",
         storage=:sepfile, blocksize=DEFAULT_MF_BLOCKSIZE) -> dst

Deep-copy t into a fresh plain table at dst. For a Table / RefTable / ConcatTable each column's storage-manager / virtual-engine kind is preserved (casacore's GIVING ... AS PLAIN); for a GroupedTable the columns are materialised (name sets the table's schema name). rows selects/reorders rows (1-based into t); storage/blocksize pack the destination into one table.mf / table.mfh5 – see _write_table_core. This is "SELECT ... INTO" for any query result.

create_ms(dir; nrow=10, nchan=4, ncorr=2, nant=3, nrec=2,
         storage=:sepfile, blocksize=DEFAULT_MF_BLOCKSIZE)

Synthesise a minimal, validate-clean MeasurementSet v2 at dir (zero / default-valued data). MAIN's DATA/FLAG go through TiledShapeStMan; scalars and fixed-shape arrays through StandardStMan direct cells; variable-shape array columns (CHAN_FREQ, CORR_TYPE, POLARIZATION_TYPE, …) through StandardStMan indirect arrays. storage/blocksize, if not :sepfile, pack every table (MAIN and each subtable) into its own table.mf/table.mfh5 – see write_ms.

reference_copy(dst, src; writable=String[]) -> dst

Create a new table at dst whose columns are ForwardColumnEngine references to src (a path or a plain Table) – each read forwards to the same-named column in src, and no data is copied. Columns named in writable are real independent copies instead. Mirrors casacore's MSTableImpl::referenceCopy.

write_reftable(dir, parent, rows; select) -> dir

Persist a row-number reference to parent (a Table, RefTable, or ConcatTable) at dir, in casacore's RefTable format — openable by casa / python-casacore. rows are 1-based row indices into parent (any order, repeats allowed). select is output_name => parent_name pairs in output order (default: every column of parent, unrenamed). A RefTable parent is flattened to its true root before writing (matching casacore's own RefTable writer — see _flatten_to_root).

write_concattable(dir, parts; subtabnames=String[]) -> dir

Persist a virtual row-wise concatenation of parts (same-schema tables) at dir. subtabnames lists keyword subtables to also concatenate (rarely used; default none). Each part must already be persisted to disk (an in-memory query()/groupby()/join() result is not valid — persist it first).

edit(path) -> EditTable
edit(f, path)            # runs `f(t)`, then `flush(t)`

Open the CTDS table at path for update.

The write lock is acquired here, for the whole session — not just inside flush. Two concurrent edit() sessions on the same table (from different Tasks / Threads.@spawned callers) each materialise their own in-memory snapshot of the untouched columns and, if either outlives the other, flush regenerates a whole storage-manager file from that snapshot — so a flush-only lock (the earlier design) let a second session start reading before the first had written, and whichever session flushed last would silently overwrite the other's already-committed changes to any row/column it hadn't itself touched (found live, via Threads.@spawn-ing two edit() sessions on disjoint row ranges of the same table: one session's writes were silently lost, and — depending on flush order — the other could also crash on a colliding atomic-rename). Holding the lock from edit(path) through flush makes a second session's own edit(path) call — specifically its own lock acquisition, before it ever reads a byte of table data — block until the first session's flush has completed, so the second session's snapshot is taken after the first session's writes, not concurrently with them. edit(f, path) releases the lock in a finally even if f throws, so an aborted edit never leaves the table locked for the rest of the process.

edit(rt::RefTable) -> RefEditTable
edit(f, rt::RefTable)                # runs `f(t)`, then flushes the parent

Open the table rt is a view of for update, through rt's own row selection / column projection — t[name][i] = v writes to the parent table's mapped row. A thin wrapper over edit(path): every write goes through the exact same fast/regen machinery as editing the parent directly. rt.parent must be a plain Table (not another RefTable/ConcatTable — matches copytable's own restriction).

Row-count changes have no RefTable analogue in casacore itself (a selection's rows are fixed at query time, and removing a RefTable row only shrinks the in-memory selection, never touching the parent) — addrows!/removerows! are not supported on a RefEditTable; build a new RefTable via query instead. addcolumn! adds to the parent's schema, matching RefTable::addColumn; removecolumn! only hides a column from this view (the parent keeps it), matching RefTable:: removeColumn — see its own docstring.

edit(ct::ConcatTable) -> ConcatEditTable
edit(f, ct::ConcatTable)             # runs `f(t)`, then flushes every part

Open every part of ct for update, through ct's own row-offset mapping — t[name][i] = v writes to whichever part row i actually belongs to. Every part must be a plain Table (a part that is itself a RefTable/ConcatTable/etc. is not supported).

Row-count changes have no ConcatTable analogue in casacore (canRemoveRow/canRemoveColumn/canRenameColumn are all false, and removeRow/removeColumn/renameColumn all throw outright) — addrows!/removerows!/removecolumn! are not supported. addcolumn! IS supported — it adds the column to every part, matching ConcatTable::addColumn.

addrows!(t, n)

Append n rows. Appended cells read back as zeros / "" (or a VariableShape cell as the previous row's shape) until written.

This is uniform across storage managers. Real casacore differs for an IncrementalStMan column, whose appended rows read back as the previous row's value (the "store on change" file simply has no entry for them) – write the values you want rather than relying on either default.

removerows!(t, rows)

Delete rows (1-based indices into the current row set). Forces the regen persist path on flush.

addcolumn!(t, name; kind=:ssm)
addcolumn!(t, name, data; kind=:ssm, type=nothing, shape=nothing)

Add a column. With no data the column is taken from the standard MS v2 schema and filled with default cells; otherwise data (length length(t))) supplies the values and its element type / shape are inferred unless type / shape are given. kind binds the column to a new/shared StandardStMan (:ssm), IncrementalStMan (:ism) or a per-column TiledShapeStMan (:tsm).

addcolumn!(t::RefEditTable, name; kind=:ssm)
addcolumn!(t::RefEditTable, name, data; kind=:ssm, type=nothing, shape=nothing)

Add a column, visible through this view — matches RefTable::addColumn (addToParent=true): the column is added to the parent's schema, sized to the parent's full row count. With no data, the column comes from the standard MS v2 schema and every parent row (not just this view's) gets the default cell — identical to addcolumn!(::EditTable, name). With data (length length(t), one value per row of this view), the parent's other rows get the column's default cell and only this view's mapped rows get data's values — mirroring what a real RefTable::addColumn + a follow-up put on just the selected rows does in casacore.

addcolumn!(t::ConcatEditTable, name; kind=:ssm)
addcolumn!(t::ConcatEditTable, name, data; kind=:ssm, type=nothing, shape=nothing)

Add a column, matching ConcatTable::addColumn: the column is added to EVERY part in turn. With no data, every part gets the standard-schema column with its own default cells (identical to addcolumn!(::EditTable, name) on each part). With data (length length(t), one value per row of the whole concatenated view), it's sliced by t.offsets and each part gets its own slice — each part independently infers its own column type/shape from that slice.

removecolumn!(t, name)

Drop a column. If it takes a whole storage-manager instance with it, that instance's files are deleted on flush. Removing "ANTENNA1" / "ANTENNA2" while a DyscoStMan-compressed column remains is rejected — see _dysco_dependents.

removecolumn!(t::RefEditTable, name) -> t

Hide name from this view — matches RefTable::removeColumn: only the VIEW's own column list is edited. The parent's actual column (real storage, and anything else about the parent's own edit session) is left completely untouched — a column "removed" from a RefTable view is still there in the table it's really stored in, exactly as in casacore.

setcell!(t, name, i, v) -> t

Set cell i of column name in the edit session t to v — the verb behind t[name][i] = v.

setcell!(t::RefEditTable, name, i, v) -> t

setcell!(t::ConcatEditTable, name, i, v) -> t

setcolumn!(t, name, vals) -> t

Replace the whole of column name in the edit session t with vals (length must equal the current row count) — the verb behind t[name][:] = vals.

setcolumn!(t::RefEditTable, name, vals) -> t

setcolumn!(t::ConcatEditTable, name, vals) -> t

renamecolumn!(path, old, new) -> path

Rename column old of the plain table at path (a metadata-only change: no storage-manager file is touched). The links other keywords hold to a renamed column – a virtual engine's stored-column name, Hypercolumn_* lists – follow it. A column forwarded to another table by a ForwardColumnEngine cannot be renamed.

setkeyword!(path, name, value; column=nothing) -> path

Set (add or replace) keyword name of the plain table at path – or of column column – to value: a number, Bool, string, array of those, or a Dict / Record. Integers are stored as Int64 and floats as Float64, and an existing keyword is replaced in place (a new one is appended) and must keep its data type, like real TaQL's ALTER TABLE ... SET KEYWORD. A metadata-only change.

removekeyword!(path, name; column=nothing) -> path

Remove keyword name of the plain table at path (or of column column); an absent keyword is a KeyError.

renamekeyword!(path, old, new; column=nothing) -> path

Rename keyword old (in place – it keeps its position) of the plain table at path or of column column. An absent old is a KeyError, an existing new an ArgumentError; old == new changes nothing.

resync(t) -> Table / MeasurementSet

Re-open the table if is_stale; otherwise return t unchanged. The returned object replaces t – keep it and drop the old one.

is_stale(t) -> Bool

Whether the table.lock sync blob has been updated (by another process, or by our own edit) since t was opened.

is_multiused(dir) -> Bool

Whether another process currently has the table open (holds the byte-1 "in use" read lock). Uses a fresh probe fd because F_GETLK never reports a conflict with the calling process.

query(t::AbstractTable, wherestr::AbstractString;
     select=[n=>n for n in columnnames(t)]) -> RefTable | GroupedTable

Row-filter t with a small TaQL-like WHERE expression:

• comparisons ==/=, !=/<>, <, <=, >, >=;
• AND/&&, OR/||, NOT/!, parentheses;
• arithmetic + - * / % // ** and unary - on numeric operands (A + 1 > B, ANTENNA1 % 4 == 0, 2 ** N) — Julia numeric semantics (/ yields a float, // truncates, % is rem);
• col IN [v1, v2, lo:hi, lo:hi:step, lo:, ...] / col NOT IN [...] (a range is a discrete lattice, default step 1; lo: is unbounded above);
• col LIKE 'pat' / ILIKE (case-insensitive) / NOT LIKE — SQL glob (% = any run, _ = one char); and TaQL's col ~ p/glob/, ~ m/regex/, ~ f/regex/ (and !~), delimiters / % @, optional trailing i;
• functions NAME(args...) (case-insensitive, TaQL's aliases) — scalar math (abs, sqrt, exp, log/ln, log10, trig, floor/ceil/ round, sign, int, pow, fmod), complex parts (real, imag, arg/phase, conj, norm), array-cell reductions (mean/avg, sum, product, median, variance, stddev, rms, min/max, any, all, ntrue/nfalse, nelements/count, ndim), running<X>(arr, hwidth) / boxed<X>(arr, bwidth) (X ∈ average/ mean, median, min/max, variance, stddev, sum — sliding- window array-cell smoothing: running is a centred window (SAME shape, shrinking half-windows at the edges), boxed is non- overlapping bins (SMALLER shape); the width is a scalar or a ndims(arr)-element array, one per axis), string ops (strlength/len, upper/lower, trim/ltrim/rtrim), array reshaping transpose/reversearray(arr[, axes...])/flatten/array(v, shape...)/ resize(arr, shape)/diagonals(arr)/nullarray/isdefined/isnull and iscolumn('NAME'), axis-collapse reductions sums/products/means(avgs)/mins/maxs/ medians/variances/stddevs/samplevariances/samplestddevs/avdevs/ rmss/sumsqrs/anys/alls/ntrues/nfalses(arr, axes...) and fractiles(arr, frac, axes...) (1-based axes, scalar/array/several; the collapsed axes are dropped from the shape), isnan/isinf/isfinite/nonfinite, masked arrays (marray(d, m), arraydata, arraymask; V[boolexpr] yields a masked array whose reductions skip the excluded elements), iif(cond, a, b), rownumber() (1-based) / rowid() (0-based), pattern values S == regex('..') / pattern('..') (shell glob, incl. {a,b}) / sqlpattern('..') (full-string match), table / column keyword access ::NAME / COL::NAME[.field] and iskeyword('NAME'), observatory('VLA') (a telescope's ITRF [x,y,z]), meas.<frame>(['SRC',] lon, lat[, mjd[, x, y, z]]) — or, in place of 'SRC', lon, lat, a single 'COLNAME' naming a direction column whose own MEASINFO supplies the source frame (Phase 110; a fixed Ref only — a per-row VarRefCol column errors, use measure(t, col, row) directly; disambiguated from 'SRC' at parse time: the string is a colname iff it is not a recognized frame name) — / meas.epoch('TAI', secs[, 'SRC'[, pos]]) (real form, seconds) / meas.last(mjd, x, y, z) / meas.freq('SSCALE', 'TSCALE', freq, mjd, x, y, z, ra, dec) / meas.rv('SSCALE', 'TSCALE', v, mjd, x, y, z, ra, dec) (frequency / radial-velocity frame conversion, SSCALE/TSCALE ∈ topo/geo/bary/ lsrk/lsrd/galacto/lgroup/cmb) / meas.doppler('SCONV', 'TCONV', value) (Doppler-convention algebra, SCONV/TCONV ∈ radio/optical/ratio/beta/gamma – no frame needed) / meas.riseset(ra, dec, mjd, x, y, z[, elev0]) → [rise_mjd, set_mjd] / meas.pos('SSCALE', 'TSCALE', x, y, z) (position frame conversion, SSCALE/TSCALE ∈ itrf/wgs84 – an identity, both share one Cartesian vector in casacore) / meas.itrfxyz(lon, lat, height) (WGS84 geodetic → ITRF Cartesian) / meas.wgs(x, y, z) (the inverse) (measure conversions – all but meas.doppler need import SOFA), pbgaussian(θ, hpbw) / pbairy(θ, diameter, freq[, blockage]) / pbellipse(dlon, dlat, hpbw_major, hpbw_minor, pa) (primary-beam power response, see src/beam/beam.jl), pi, e;
• an optional trailing ORDER BY key [ASC|DESC], ... — each key any expression (A+B, abs(A), A>3); a leading ASC/DESC (ORDER BY DESC A, B) is the default for keys without their own direction; when every key is descending, ties come out in reverse row order (as real TaQL).

Column names are case-sensitive and must name a column of t; keywords and function names are case-insensitive. Only the columns actually referenced (by the WHERE expression or an ORDER BY key) are read.

select is "out" => rhs pairs. When every rhs is a bare column name (or Symbol) the result is a lazy RefTable (no data copied; persist it with write_reftable(dst, result)). When any rhs is a computed expression ("X * 2", "sqrt(abs(V))", "iif(K==0, 1, 0)" — same grammar as WHERE, aggregates excepted) the result is an in-memory GroupedTable with those columns evaluated per matched row. A computed column that is a masked array ("V[FLAG]") persists as its plain data. A ("val", "mask") => "expr" pair entry (TaQL's expr AS (v, m)) instead emits two columns — the data and the mask — so [("D", "F") => "marray(DATA, FLAG)"] reads a column together with its mask column.

A bare "ORDER BY ..." (no WHERE) matches every row, sorted.

mscal.* derived-MS functions (Phase 77, needs import SOFA and an MS MAIN table with ANTENNA / FIELD subtables): mscal.ha1() / ha2() / ha() (hour angle, rad), mscal.hadec1() ([ha, dec]), mscal.azel1() ([az, el]), mscal.az1() / el1() (scalar), mscal.pa1() (parallactic angle — 0.0 unless that antenna's own ANTENNA.MOUNT starts with "alt-az", case-insensitive; the suffix-less mscal.pa() always returns 0.0, matching MSCalEngine:: getPA exactly, Phase 138), mscal.last1() (local apparent sidereal time, rad), mscal.itrf() ([lon, lat] of PHASE_DIR in ITRF), mscal.uvw_j2000() ([u, v, w] m — the baseline's ANTENNA2 - ANTENNA1 uvw in J2000, recomputed fresh from the ANTENNA positions exactly like casacore's getNewUVW, not a transform of the stored UVW column; Phase 137 — a real MS's stored UVW usually follows the opposite ANTENNA1 - ANTENNA2 convention, a genuine, longstanding casacore quirk, not a bug here), mscal.delay() (baseline geometric delay ap1-ap2, s), mscal.delay1() / delay2() (Phase 136 — that one antenna's delay relative to the array centre; genuinely different from the bare form, not implemented until this phase) — the delay family defaults its direction to FIELD.DELAY_DIR, not PHASE_DIR (matches derivedmscal's own default exactly; an explicit direction argument overrides it as usual), mscal.riseset[1|2]([elev0][, dir]) (Phase 105) → [rise_mjd, set_mjd] for that antenna's own ITRF position and dir (default FIELD.PHASE_DIR), for the UTC day of the row's TIME — the same rise/set machinery meas.riseset() exposes, wired into the automatic per-row geometry the way mscal.pbresponse wires in the beam models; memoized per day, not per row. The 1 / 2 suffix picks ANTENNA1 / ANTENNA2; no suffix uses the array centre — OBSERVATION.TELESCOPE_NAME looked up in the bundled Observatories table, falling back to antenna 0. The direction functions (ha / hadec / azel / az / el / pa / itrf / delay) take an optional direction argument instead of FIELD.PHASE_DIR — a body name (mscal.el1('SUN')), any real direction-valued column of the FIELD subtable (mscal.az1('DELAY_DIR'), or a custom one — Phase 140, matching casacore's own precedence: a body/frame name is tried first, then the string is looked up as a FIELD column), a [ra, dec] J2000 pair in radians (mscal.hadec1([2.0, 0.5])), or a sexagesimal 'RA, DEC' string (mscal.el1('10h42m31, 45d51m16')).

Note (Phase 139): for a moving-target FIELD (a polynomial PHASE_DIR or an EPHEMERIS_ID), these functions interpolate the direction at each row's own TIME — real casacore's derivedmscal UDFs do NOT (MSCalEngine::fillFieldDir always uses the FIRST element of the direction cell, ignoring TIME entirely); this package's behaviour is a deliberate improvement, not a divergence to fix, but the two won't numerically agree for such a field.

mscal.stokes(col [, 'types'] [, rescale]) (Phase 78) converts a DATA / FLAG / WEIGHT array cell between correlation bases. types (default 'IQUV') is an alias (IQUV / CIRC / LIN) or a comma-list ('I', 'I,V', 'XX,YY', mixable with the pseudo types below); the input basis comes from POLARIZATION.CORR_TYPE row 1. The result is a (nOut, nchan) matrix. types also accepts the derived pseudo types Ptotal (√(Q²+U²+V²)), Plinear (√(Q²+U²)), Pangle (½·atan2(U,Q), rad), PFtotal/PFlinear (the same divided by I) (Phase 109) — non-linear, so they need a complex (DATA-like) input; Bool/real (FLAG/WEIGHT) input raises a clear error. A WEIGHT-like conversion matches casacore's own quirk exactly (Phase 142): if ANY input correlation's weight is exactly 0, the whole output is 0 too — even a correlation with no coefficient at all for that particular output.

mscal.<sel>('spec') (Phase 80) — MSSelection-lite row selection, returning a per-row Bool. <sel> ∈ baseline / field / spw / scan / state / array / obs. spec is a comma-list of terms (N, N~M, >N / <N, an exact / glob / /regex/ name match against the type's NAME column); a !-term is subtracted. mscal.baseline also takes L & R / L && R (baseline between two antenna sets; & drops autocorrelations) and a whole-spec !. mscal.time('t0~t1') selects a TIME range (endpoints are ISO / YYYY/MM/DD[/HH:MM:SS] datetimes or a bare MJD-days number); mscal.uvdist('a~b[unit]') selects a 2-D uv-distance range, unit ∈ m (default) / km / lambda / klambda / mlambda (wavelength units scale per row by SPECTRAL_WINDOW.REF_FREQUENCY). mscal.corr('RR,LL') selects rows whose polarization setup contains any of the named correlations (Stokes names or integer codes); mscal.feed('0 & 1') is the mscal.baseline form on FEED1 / FEED2.

mscal.spw('0:5~20') — the spw selection takes an optional :chanlist (a ;-list of a~b, a~b^step channel-index ranges, single indices, or f1~f2GHz / <f / >f CHAN_FREQ ranges); a channelled spw selects the row only if at least one of the row's channels is selected. mscal.chan('0:5~20') returns a per-row BitVector (the spw's channel count) — the selected-channel mask, for use with any(...) / count(...) / a masked array.

mscal.pbresponse('gaussian:HPBW' | 'airy:D:FREQ[:BLK]' | 'ellipse:HMAJ:HMIN:PA' [':squint:DLON:DLAT'] [, dir]) (Phase 101/103, a MeasurementSets extension — not a real derivedmscal UDF) — the GaussianBeam / AiryBeam / EllipticalGaussianBeam power response toward dir (default FIELD.PHASE_DIR) as seen through ANTENNA1's actual pointing (POINTING.DIRECTION), i.e. the attenuation from a pointing/tracking error, needs a POINTING subtable. A trailing :squint:DLON:DLAT on any spec wraps the beam in a SquintBeam (radians). mscal.pbresponsebl(...) is the same beam evaluated at both ANTENNA1 and ANTENNA2's own pointing and multiplied — the joint baseline response. mscal.pbcorr(valexpr, spec [, dir]) / mscal.pbatten(valexpr, spec [, dir]) (Phase 102) desugar to valexpr / mscal.pbresponse(spec, dir) / valexpr * mscal.pbresponse(spec, dir) — usable as an update! SET RHS to primary-beam-correct an array column in place; mscal.pbcorrbl / mscal.pbattenbl (Phase 103) do the same with mscal.pbresponsebl.

Deliberately a subset of real TaQL's grammar, not a look-alike: no boolean-mask array subscripts. Supported: 1-based array element/slice indexing (DATA[1,1], V[1:4,1], UVW[-1], V[end-2:end,1]; as in real TaQL 0 and -1 both mean the last element, other negatives count from the end, and an out-of-range index, a range ending before its start or a non-integer subscript is an error), array literals [a, b, ...], BETWEEN / NOT BETWEEN (inclusive), bitwise & | ^ ~ (^ is xor – use ** for exponentiation), ~= / !~= approximate equality (casacore's near, relative tolerance 1e-5), scientific-notation number literals (1.4e9), quantity literals (1.4GHz, 10arcsec, 30deg, or spaced 1.4 GHz / 3 km – compared against a column carrying a QuantumUnits keyword; needs the Unitful extension), sexagesimal literals (10h30m, 45d51m16s, 12h, 45d – an h-prefix is an hour angle ×15, a d-prefix is degrees; the value is radians), and date/time + angle functions: datetime/mjd/mjdtodate/date/time (all MJD-day Float64), year/month/day/week/weekday, cdate/ctime/cmonth/cdow/ ctod, hms/dms (radians → sexagesimal string), angle('10h30m') (sexagesimal string → radians), normangle, angdist/angdistx (4 scalar radians or two [lon, lat] arrays).

query(f::Function, t::AbstractTable; cols=nothing, orderby=nothing,
     select=[n=>n for n in columnnames(t)]) -> RefTable | GroupedTable

Row-filter t with a Julia predicate f(row) -> Bool (do-block friendly: query(t; cols=[...]) do row ... end). row is a Tables.AbstractRow supporting row.COLNAME property access. cols restricts which columns are actually read (default: every column — f is an opaque closure, so unlike the string-based query its column use can't be inferred; pass cols explicitly on a wide table to avoid materialising columns f never touches). orderby sorts the matched rows by one or more columns: each entry is a bare column name/Symbol (ascending) or a name => :asc/name => :desc pair. select — see the string-based query above.

query(gt::GroupedTable, wherestr; select=identity) -> GroupedTable

Filter / project / sort an in-memory columnar result. wherestr is a TaQL-lite WHERE expression (+ optional trailing ORDER BY) over the result's column names; select is outname => source_name pairs.

query(f, gt::GroupedTable; cols=nothing, orderby=nothing, select=identity) -> GroupedTable

Closure form: keep the rows for which f(row) -> Bool (row.OUTNAME property access). See the string form above.

groupby(t, groupcols; select, cols=nothing, where=nothing, having=nothing, orderby=nothing) -> GroupedTable

Group the rows of t by groupcols (a column name / Symbol, or a vector of them; an empty vector = one group over the whole table) and compute one result row per group. t is any AbstractTable — a Table, a RefTable, or another groupby / join / query result, so the verbs chain.

select is outname => rhs pairs, where rhs is one of:

• a string — a TaQL-lite expression that may use g-prefixed aggregate functions over the group: gcount() / gcount(x) (row count), gsum(x), gproduct(x), gmean(x) / gavg(x), gmedian(x), gmin(x), gmax(x), gvariance(x) / gsamplevariance(x), gstddev(x) / gsamplestddev(x), grms(x), gany(x), gall(x), gntrue(x), gnfalse(x), gfirst(x), glast(x) — plus the group-key columns and scalar expressions of them. An aggregate's argument is normally a scalar per row (wrap an array cell in mean(...) / sum(...)), OR a masked array (gmean(V[!FLAG]) pools every row's unmasked elements). The s-suffixed per-element variants (gsums, gproducts, gmeans / gavgs, gvariances / gsamplevariances, gstddevs / gsamplestddevs, grmss, gmins, gmaxs, ganys, galls, gntrues, gnfalses) reduce the group's array cells per position — over the rows where that cell is unmasked (all cells if unmasked), giving one array. Also growid() (the group's 0-based row ids), gaggr(x) / gstack(x) (collect the group's values; arrays are stacked along a new last axis) and ghist(x, nbins, lo, hi) (nbins + 2 counts: underflow, nbins left-closed bins, overflow).
• a Symbol — shorthand for a bare column name (:K ≡ "K").
• a function g -> value — called with a GroupSlice (see the do-block form below); use for aggregates the g* set can't express.

where pre-filters rows (a WHERE string, or a row -> Bool closure). having filters groups (a HAVING string over aggregates/keys, or a g -> Bool closure). cols restricts which columns are loaded onto a GroupSlice — only relevant when select/where/having use a closure (default: every column of t). orderby sorts the result rows by output column name(s): "N" (ascending) or "N" => :desc.

rollup = true / cube = true / grouping_sets = [...] compute multiple SQL grouping sets and stack the result rows. rollup groups by each key prefix (keys, keys[1:end-1], …, ()); cube by every subset; grouping_sets by exactly the sets you list — each an iterable of key names, () for the grand total (grouping_sets = [("K1","K2"), ("K1",), ()]). At most one of the three. In a row where a key is aggregated away, that key column is missing — a bare :K / "K" select entry emits missing; a closure should build its key fields from g.keys ((; g.keys..., N = length(g))).

GROUPING(K) — usable in a select or having string, and as g.grouping.K in a closure — is true when key K is rolled up in that row (SQL's GROUPING()), e.g. select = ["label" => "iif(GROUPING(K2), 'ALL', K2)", …] or having = "GROUPING(K1) == 0".

(casacore parses ROLLUP/CUBE/GROUPING SETS but does not implement them, so this is plain SQL semantics with no real-TaQL cross-check.)

Returns a GroupedTable.

groupby(f, t, groupcols; cols=nothing, where=nothing, having=nothing, orderby=nothing) -> GroupedTable

Closure form (do-block friendly). f(g) receives a GroupSlice for each group and returns a NamedTuple — that group's output row. Every group must return the same field names; they become the result's columns, in that order.

groupby(t, [:ANTENNA1]; cols=["ANTENNA1", "DATA"]) do g
    (; ANT = first(g.ANTENNA1), N = length(g), AMP = mean(abs.(g.DATA)))
end

cols restricts which columns are loaded onto g (default: every column of t — f is opaque). where / having — a string or a predicate closure (row -> Bool / g -> Bool). orderby — see the string form. rollup / cube / grouping_sets — see the string form; a multi-set closure should use g.keys for its key fields ((; g.keys..., N = length(g))) so aggregated-away keys come back as missing, and may test g.grouping.K.

GroupSlice

The per-group value passed to a closure-form groupby (the do-block argument, and the argument of a select / where / having closure). g.COLNAME is a materialised Vector of that column's values for the group's rows; length(g) is the group size; propertynames(g) lists the loaded columns.

Only the columns named in cols= (or every column, when cols is omitted) are available — a closure's column use cannot be inferred. The names cols, rows, keys, level are struct fields / synthesised properties, so a column with one of those names is unreachable as g.<name> (a non-issue for MS column names).

g.keys is a NamedTuple of the grouping-key values for this group (one scalar per key). g.level is the number of active grouping keys — always length(groupcols) for a plain groupby, but for a rollup=true subtotal level the trailing keys are inactive and their g.keys entries are missing. A rollup closure should build its key output fields from g.keys ((; g.keys..., N = length(g))).

update!(target; set, where=nothing) -> Int

Change column values in the CTDS table at target (a path or an open Table / subtable(ms, …)). set is "COL" => "expr" pairs, applied in the given order, each RHS seeing any earlier entries' already- written values for that row — matching real casacore's own per-row, per-item UPDATE semantics (TableParseQuery::doUpdate, verified live: set = ["A" => "B", "B" => "A"] does not swap — A becomes B's old value, then B reads that already-updated A, so both end up equal to the old B; to actually swap, stage a temporary column, or give both new values from a pre-computed constant). The set key may also be an array-slice or boolean-mask target, "COL[subscripts]" => "expr" (TaQL's UPDATE … SET NAME[i,j] = … / NAME[maskexpr] = …) — 1-based, end-relative and range subscripts, or a single Bool-array subscript acting as a write mask — which writes only the addressed elements, leaving the rest untouched; a scalar RHS fills the region. col[slice][mask] and col[mask][slice] both work.

A set key may also be a (datacol, maskcol) tuple (TaQL's UPDATE … SET (NAME, MASKNAME) = …). ("D", "M") => "dexpr" writes dexpr to D and, to M, the mask of dexpr when it is a masked array (SET (D, M) = V[goodcond] → M gets !goodcond), else a Bool array flagging where dexpr's result is non-finite. ("D", "M") => ("dexpr", "mexpr") writes mexpr (any Bool expression) to M instead. Either name may be a slice / mask target.

where is a TaQL-lite WHERE string, a row -> Bool closure, or nothing (every row). orderby (like query's — a bare column name/Symbol, ascending, or a name => :asc/name => :desc pair) sorts the matched rows before limit (an Integer) keeps only the first limit of them (limit == 0 keeps all, limit < 0 all but the last |limit| – real TaQL's LIMIT convention) — "update the N oldest/newest rows matching a condition", e.g. update!(t; set=[...], where="...", orderby=["TIME"], limit=10). A deliberate MeasurementSets extension, not a port of real TaQL's own UPDATE ... ORDER BY ... LIMIT n: real casacore's UPDATE/DELETE ORDER BY does not sort the matched rows by the given key before LIMIT truncates them (verified live — ORDER BY T LIMIT 3 and plain LIMIT 3 give byte-identical results on the same data; T's values play no role). This package's orderby/limit do a genuine sort-then-limit instead. Returns the number of rows changed.

target may also be a RefTable or ConcatTable (an object, or the path of a persisted one): the new values are written through to the real table(s), like UPDATE on a selection in casacore.

taql(target, command::AbstractString, others...)

Run one TaQL-lite command against target (a path or an open table); extra tables are $2, $3, … in the command:

• UPDATE [t [AS a]] SET c1 = e1, c2 = e2 [WHERE cond] [ORDER BY k [ASC|DESC], …] [LIMIT n] → update!, returns Int
• DELETE [FROM t [AS a]] [WHERE cond] [ORDER BY k [ASC|DESC], …] [LIMIT n] → delete!, returns Int
• SELECT [DISTINCT|ALL] [*|expr [AS a], …] [FROM t [AS a] | (SELECT …) | $2] [JOIN $2 b ON a.K == b.K] [WHERE cond] [GROUP BY k, …] [HAVING cond] [ORDER BY k] [LIMIT n [OFFSET m]] [(INTO|GIVING) 'path'] → the result table, or with INTO / GIVING the saved path (copytable)
• INSERT INTO t [(c1, c2)] VALUES (v1, v2), (…) [LIMIT n], INSERT [LIMIT n] INTO t SET c1 = v1, c2 = v2, INSERT INTO t [(c1, c2)] SELECT … FROM 'path' [WHERE cond] [LIMIT n] → insert!, returns Int
• ALTER TABLE t ADD|DROP|RENAME COLUMN …, … SET|DROP|RENAME KEYWORD …; CREATE TABLE 'p' [name type, …] [LIMIT n] [DMINFO …]; DROP TABLE 'p'

The syntax follows real TaQL (checked command by command against casacore): keywords in any case, # … comments to the end of the line, a trailing ;, K IN [1, 2] / K IN (1, 2) / K IN (SELECT …), EXISTS (SELECT …), unquoted date literals (2020-02-12, 12Feb2020), unit literals (1.4GHz), string literals that touch concatenate ('a''b' is ab), an unnamed computed column is called Col_<position>, and LIMIT 0 means no limit. INSERT … VALUES/SET values must be constant. Clause keywords (SET / WHERE / INTO / GIVING / FROM / SELECT) are found by a case-insensitive split; a quoted literal containing one of them is not supported — use the Julia functions for that.

join(left, right; on, rightcols, leftcols=nothing, where=nothing,
     unmatched=:error, multi=false, oncols=nothing, orderby=nothing) -> GroupedTable

Join left and right (extends Base.join). Either side is any AbstractTable — a Table, a RefTable, or another groupby / join / query result, so the verbs chain. The default (multi = false) is an N:1 lookup join — each left row matches at most one right row, right columns pulled in per left row (TaQL's JOIN … ON semantics). multi = true is a general M:N equi-join.

on is either

• a column name (String / Symbol) — that left column holds a 0-based row index into right (the MS subtable convention: ANTENNA1 → the ANTENNA subtable row); N:1 only;
• a Pair "LKEY" => "RKEY" — equi-join, matching left.LKEY against right.RKEY (must be unique when multi = false);
• a vector of pairs — a composite key (all must match);
• a 2-arg predicate (lrow, rrow) -> Bool — a general non-equi join (a MeasurementSets extension; TaQL's own JOIN is == / IN only). lrow / rrow support row.COLNAME. Evaluated by a nested loop over every (left, right) pair — O(nrow(left) × nrow(right)); query / select each side down first for a large table. oncols = (leftnames, rightnames) restricts which columns are loaded onto the predicate rows. unmatched sets the join type as for multi = true; multi itself is ignored;
• a string condition — a TaQL-lite expression with L. / R. table-qualified column references ("L.TIME BETWEEN R.T0 AND R.T1"). The declarative form of the predicate above — same nested loop, same cost, same unmatched semantics. Must reference at least one L.<col> and one R.<col>; a bare column name is still the index-lookup join.

rightcols lists the right columns to attach — "NAME" or "NAME" => "ANT_NAME" to rename. leftcols (default: every left column) likewise selects/renames left columns. Output names must be unique across both. where filters the assembled result (a string over the output column names, or a row -> Bool closure).

unmatched — for multi = false: :error (default — throw on a dangling key), :drop (exclude that left row), :missing (keep it, right columns missing). For multi = true it sets the join type: :drop = inner, :missing / :left = left outer, :right = right outer, :full = full outer, :error = every left row must match ≥ 1. Unmatched rows on either side get missing in the other table's columns.

orderby sorts the result by output column name ("N" / "N" => :desc). Returns a GroupedTable whose columns are lazy views except where missing-fill forces materialisation.

delete!(target; where=nothing, orderby=nothing, limit=nothing) -> Int

Remove rows from the CTDS table at target (a path or an open Table). On a persisted RefTable (its path or the object) the rows are removed from the reference only, the real table is untouched; a ConcatTable cannot lose rows. where is a TaQL-lite WHERE string, a row -> Bool closure, or nothing (every row — leaves a 0-row table). orderby/limit — see update! — sort the matched rows then keep only limit of them (limit == 0 keeps all, limit < 0 all but the last |limit|) before deleting, e.g. "delete the 10 oldest rows matching a condition": delete!(t; where="...", orderby=["TIME"], limit=10). Returns the number of rows removed. Extends Base.delete!.

insert!(target; values, limit = nothing) -> Int

Append rows to the CTDS table at target (a path or an open Table). values is one row (["A" => 1, "B" => 2.5] or (; A = 1, B = 2.5)), a vector of those, or any Tables.jl source (another table, a query result, a Vector{NamedTuple}). Columns of target not supplied get their default (0 / "" / a same-shape zero array). Scalar values are coerced to the target column's element type. Returns the number of rows inserted. Extends Base.insert!.

limit controls how many rows are appended (TaQL's INSERT … LIMIT): nothing or 0 (default) inserts one row per values row; a positive limit inserts exactly that many, cycling through the values rows; a negative limit inserts nrow(target) + limit rows (also cycling), clamped at zero.