Guide

The examples below assume using MeasurementSets and a Measurement Set at /path/to/my.ms. They are illustrative, not doctested.

Opening a Measurement Set

ms = MeasurementSet("/path/to/my.ms")
subtablenames(ms)                        # ["ANTENNA", "SPECTRAL_WINDOW", …]

t = readtable("/path/to/my.ms")          # the MAIN table on its own
nrow(t)
columnnames(t)

readtable also opens a single subtable directory (readtable("/path/to/my.ms/ANTENNA")) or any casacore table.

Reading columns

column returns a lazy Column (<: AbstractVector):

ms[:DATA][42]                            # one cell: a 4×64 ComplexF32
ms[:UVW][1:100]                          # first 100 baselines' UVW
column(ms.data, "TIME")[:]               # the whole column (fast path)

Column and table metadata:

columndesc(t, "DATA")                    # a ColumnDesc (type, shape, keywords, DM binding)
keywords(t)["MS_VERSION"]                # 2.0f0
columndesc(t, "UVW").keywords            # per-column keywords (units, MEASINFO)

Precision

A MAIN table's TpComplex visibility columns — DATA, MODEL_DATA, CORRECTED_DATA — read back as ComplexF16 by default. The on-disk bytes are ComplexF32; the visibilities derive from 8-bit samples, so nothing real is lost and the working set halves.

eltype(column(t, "DATA"))                     # Matrix{ComplexF16}
eltype(column(t, "DATA"; precision=:full))    # Matrix{ComplexF32}
readtable("my.ms"; precision=:full)           # every column wide
readtable("my.ms"; precision=BFloat16)        # DATA + WEIGHT + SIGMA all 16-bit
MeasurementSet("my.ms"; precision=:full)

TpFloat columns (WEIGHT, SIGMA, WEIGHT_SPECTRUM) stay Float32 under the default and under precision=Float16 (real weights exceed Float16's 65504 range). Pass precision=BFloat16 to narrow them too — BFloat16 has Float32's exponent range (overflow-safe) and a 7-bit mantissa that matches 8-bit-derived data; DATA then comes back as Complex{BFloat16} and WEIGHT as BFloat16.

Float64 (TIME, UVW), ComplexF64 and Bool are never narrowed. Non-MAIN tables default to :full. copyms / copytable / edit always read at full precision, so copies and in-place edits stay byte-exact; a narrowed column written back upcasts to Float32 on disk.

Tables.jl interop

Every table is a Tables.jl source — column and row access — so subtables drop straight into the data ecosystem:

using DataFrames, Tables
DataFrame(subtable(ms, "ANTENNA"))
Tables.schema(subtable(ms, "SPECTRAL_WINDOW"))
Tables.rowtable(subtable(ms, "FIELD"))

Validating against the standard schema

The MS v2 standard schema (NRAO Memo 229) is available as data:

validate(ms)                             # String[] when conformant; never throws
stdtable("SPECTRAL_WINDOW").columns
stdcolumns("ANTENNA")

Reference frames

measure(t, col[, row]) reads a measure-valued column as a typed value with its reference frame; import SOFA (+ optionally EarthOrientation) converts between frames.

import SOFA, EarthOrientation

e = measure(main, "TIME", 1)                       # MEpoch{UTC}
measconvert(e, TT)                                 # → MEpoch{TT}

fr = MeasFrame(epoch = e,
               position = measure(subtable(ms, "ANTENNA"), "POSITION", 1),
               direction = measure(subtable(ms, "FIELD"), "PHASE_DIR", 1))
measconvert(MDirection{J2000}(2.0, 0.5), AZEL; frame = fr)
measconvert(measure(subtable(ms, "SPECTRAL_WINDOW"), "CHAN_FREQ", 1)[1],
            LSRK; frame = fr)

See Concepts for the full frame list, accuracy notes, and the measures = write keyword.

Writing and copying

copyms("/path/to/my.ms", "/tmp/copy.ms"; rows = 1:2000)
create_ms("/tmp/synth.ms"; nrow = 100, nchan = 64, ncorr = 4, nant = 6)

write_table("/tmp/spw", "SPECTRAL_WINDOW",
            ["NUM_CHAN"  => [64, 32],
             "CHAN_FREQ" => [collect(1.0:64.0), collect(1.0:32.0)]];  # ragged
            nrow = 2)

copyms / copytable preserve each column's storage-manager, virtual engine and Dysco compression. write_table / create_ms pick StandardStMan by default; tsm= / ism= / engines= / dysco= choose a different layout, and storage=:multifile / :multihdf5 packs the result into one container file.

copytable is also SELECT … INTO — it materialises any query result:

copytable("/tmp/cal.tab", query(ms.MAIN, "FIELD_ID == 3"))

A selection can also be kept as a reference instead of a copy — write_reftable stores only the row numbers (casacore's RefTable, what a TaQL SELECT … GIVING '<path>' writes), and write_concattable a virtual concatenation of same-schema tables:

sel = query(ms.MAIN, "ANTENNA1 == 3")
write_reftable("/tmp/field1.tab", ms.MAIN, getfield(sel, :rows))       # a few KB, whatever the selection holds
nrow(readtable("/tmp/field1.tab")) == nrow(sel)

a = write_table("/tmp/part_a", "T", ["X" => [1.0, 2.0]]; nrow = 2)
b = write_table("/tmp/part_b", "T", ["X" => [3.0]]; nrow = 1)
write_concattable("/tmp/ab", [readtable("/tmp/part_a"), readtable("/tmp/part_b")])
column(readtable("/tmp/ab"), "X")[:]                                   # [1.0, 2.0, 3.0]

write_ms(dst, ms; rows, subtables, subtable_rows) is the general form of copyms (it takes an opened MeasurementSet; copyms takes either).

Any Tables.jl source can be written: write_table(dir, "T", df) takes a DataFrame, a Vector{NamedTuple} or a dictionary of vectors, and nrow is the first column's length. A Union{T,Missing} column without a missing value is stored as T; a missing value is an error that names the row, because casacore columns cannot hold them.

Editing in place

edit("/tmp/copy.ms") do t
    t[:FLAG][5] = trues(4, 64)           # a tiled cell -- patched in the tile file
    t[:SCAN_NUMBER][10] = 7
    addrows!(t, 10)                      # every storage manager's row count grows
    for r in 91:100
        t[:TIME][r] = 4.6e9 + r
    end
end

Row and schema mutation take a whole-file regeneration path:

edit("/tmp/copy.ms") do t
    removerows!(t, [2, 5, 9])
    addcolumn!(t, "WEIGHT_SPECTRUM")
    removecolumn!(t, "FLAG_CATEGORY")
end

The flush runs under an exclusive table.lock and updates the sync blob, so a concurrent casacore reader re-syncs. is_stale / resync let a long-lived reader notice an external write.

t[:A][i] = v and t[:A][:] = vals have function forms, setcell!(t, "A", i, v) and setcolumn!(t, "A", vals). A cell is checked when it is assigned: a string into a numeric column, an array into a scalar column, or an array of the wrong shape into a fixed-shape column is an ArgumentError there, not a failure at flush.

A RefTable or ConcatTable can be edited too — edit of one writes through to the real table (or the right part) at the translated row, as casacore does:

sel = query(ms.MAIN, "ANTENNA1 == 3")
edit(sel) do t
    t[:SCAN_NUMBER][1] = 99                  # row 1 of the selection = its parent's row
end

The schema and keywords of a table on disk can be changed without reading it: renamecolumn!(path, "OLD", "NEW"), setkeyword!(path, "KW", value), renamekeyword!(path, "OLD", "NEW") and removekeyword!(path, "KW") — the same as ALTER TABLE through taql.

The files one flush replaces are committed together: they are written under hidden names and a journal (table.commit) is renamed into place before they are renamed over the table's files. A process killed before that leaves the old table; one killed after it is finished by the next readtable or edit. A flush that fails while writing (a full disk) leaves the table untouched. Tile edits made on their own are patched into the tile file in place and are not covered by this. Tables of any row count are read (casacore's 64-bit forms above 2³¹ rows and 2 GiB are supported); edit, whole-column reads and copytable of a table too large for memory raise an error saying so, and the writers store at most 2³²−1 rows.

Querying

query row-filters a table and returns a RefTable:

sel = query(ms.MAIN, "ANTENNA1 != ANTENNA2 AND mean(abs(DATA)) > 3 ORDER BY TIME")

The WHERE string supports comparisons, AND/OR/NOT (or &&/||/!), parentheses, IN [...], arithmetic (+ - * / % // **), LIKE / ~ p/glob/ / ~ m/regex/ pattern matching, a function library (abs, sqrt, mean, any, rownumber(), …), and a trailing ORDER BY. There is also a closure form:

query(ms.MAIN; cols = ["ANTENNA1", "ANTENNA2", "UVW"]) do row
    hypot(row.UVW...) > 500.0
end

groupby computes one row per group:

groupby(ms.MAIN, "FIELD_ID";
        select = ["FIELD_ID" => :FIELD_ID,
                  "N"        => "gcount()",
                  "AMP"      => "gmean(mean(abs(DATA)))"],
        having = "gcount() > 100",
        orderby = ["AMP" => :desc])

or, for aggregates the g* set can't express, a do-block returning a NamedTuple:

using Statistics: mean
groupby(ms.MAIN, [:ANTENNA1]; cols = ["ANTENNA1", "DATA", "WEIGHT"]) do g
    (; ANT  = first(g.ANTENNA1),
       N    = length(g),
       WAMP = sum(mean.(abs, g.DATA) .* g.WEIGHT) / sum(g.WEIGHT))
end

join (a method added to Base.join) attaches a subtable's columns to each MAIN row — an N:1 lookup join, TaQL's own JOIN … ON semantics:

join(ms.MAIN, subtable(ms, "ANTENNA");
     on        = "ANTENNA1",                    # 0-based row index into ANTENNA
     rightcols = ["NAME" => "ANT_NAME"],
     where     = "ANT_NAME ~ p/DA*/")

on is a column name (that left column is a 0-based row index into the right table), a "LKEY" => "RKEY" pair (equi-join), or a vector of pairs (composite key). unmatched is :error (default), :drop or :missing.

Every result is an AbstractTable, so the verbs chain, and any result persists with write_table(dst, "T", result; nrow = nrow(result)) or copytable.

Row-level write commands

update!, delete! (extends Base.delete!) and insert! (extends Base.insert!) change a table in place:

update!("/tmp/copy.ms/ANTENNA";
        set = ["MOUNT" => "'ALT-AZ'"], where = "STATION ~ p/PM*/")

delete!("/tmp/copy.ms/FLAG_CMD"; where = "APPLIED")

insert!("/tmp/copy.ms/STATE"; values = (; OBS_MODE = "CALIBRATE_PHASE", SIG = true))
insert!("/tmp/dst.ms/STATE", query(other, "OBS_MODE == 'CALIBRATE_PHASE'"))

update!'s set RHS are TaQL-lite expressions evaluated against the pre-update row (so ["A" => "B", "B" => "A"] swaps). where is a WHERE string, a row -> Bool closure, or nothing (every row).

taql wraps all four as one string dispatcher:

taql("/tmp/copy.ms", "UPDATE t SET UVW = UVW * 2 WHERE ANTENNA1 == 0")
taql("/tmp/copy.ms", "DELETE FROM t WHERE FLAG_ROW")
taql("/tmp/copy.ms", "INSERT INTO t (A, B) VALUES (1, 2.5), (3, 4.5)")
taql(ms.MAIN,        "SELECT A, B AS BB WHERE A > 5 INTO '/tmp/out'")