Performance

Everything here is measured on an Apple-silicon laptop with a 1M-row synthetic table (four columns) or a 20 000-row Measurement Set; the figures are there to show what matters, not to promise exact numbers. The examples run against the sample Measurement Set in the test suite.

using MeasurementSets
t = readtable("/path/to/my.ms")

Reading cells and columns

A Column is lazy. Read it in bulk when you want all of it, and keep the lazy column when you index into it repeatedly:

times = column(t, "TIME")[:]                  # one bulk read: ~1 ms for a million rows

function total(col)                            # pass the lazy column in; `col[i]` is 20-50 ns for a numeric cell
    s = 0.0
    for i in 1:length(col)
        s += col[i]
    end
    return s
end
total(column(t, "TIME"))

A query / groupby over a tiled array column (DATA, FLAG) streams it in 8 MiB blocks instead of decoding one cell per row (memory stays bounded; any(FLAG) over 100 000 rows: 31 → 9 ms, real(DATA[1,1]) > 0.9: 115 → 73 ms). A column of array cells (UVW, DATA, FLAG, ...) comes back from [:] as a lazy BlockColumn over one buffer, and rawblock gives you that buffer as a single dense array with the cell axes first — the cheapest way to hand a column to numerical code:

uvw = rawblock(t, "UVW")                       # a 3 × nrow matrix, no per-row objects

Selecting many rows at once reads the column once when the selection is dense (at least 4096 rows covering an eighth of the column) and cell by cell otherwise, so both column(t, "TIME")[rows] for a large rows and column(query_result, "TIME")[:] are cheap:

rows = findall(>(0), column(t, "ANTENNA1")[:])
sel  = column(t, "TIME")[rows]

A run of consecutive rows of a tiled array column — column(t, "DATA")[a:b] — is decoded a block at a time as well (20 000 visibilities: 18.6 → 6.3 ms; the cells are owned arrays, as before).

The StandardStMan, IncrementalStMan and tile files are memory-mapped, not read: opening a table and reading one cell or one column touches only the pages it needs (the first cell of a 20M-row table: 35 ms and 611 MB → 0.2 ms and 1 MB; a fresh query on it 108 → 26 ms).

Precision

A MAIN table reads its ComplexF32 columns (DATA, MODEL_DATA, CORRECTED_DATA) as ComplexF16 by default, which halves the memory of a visibility cube, and decodes straight into the narrow type. Ask for the on-disk precision when you need it:

full = column(t, "DATA"; precision = :full)    # ComplexF32; about twice as fast per cell as the narrowed read

Queries

query, groupby, join, update!, delete! and taql evaluate their conditions and expressions as compiled closures over the loaded columns once the table has at least 2000 rows, so a WHERE over a million rows is milliseconds (A == 3: 1 ms; sqrt(B) + C > 1: 18 ms; ORDER BY a numeric key: ~50 ms; an equi-join of 1M rows to 100: 9 ms; groupby ... gsum(B*C): ~65 ms). Prefer the string forms to the closure forms: a closure is opaque, so every column is loaded unless you pass cols=.

r = query(t, "ANTENNA1 == 3 AND sqrt(TIME) > 0 ORDER BY TIME")
g = groupby(t, "ANTENNA1"; select = ["ANTENNA1" => "ANTENNA1", "n" => "gcount()", "m" => "gmean(TIME)"])

On the real 9.8M-row MS a WHERE on scalar columns takes 5–40 ms (0.03–0.15 of real TaQL's time; FIELD_ID == 1 5 ms, ANTENNA1 != ANTENNA2 40 ms), and one that reads an array column such as UVW or FLAG is bounded by the block read: UVW[1] > 100 1.3 s (real TaQL 2.0 s), sumsqr(UVW) > 1e6 1.6 s (2.1 s), UVW[1]*UVW[1]+UVW[2]*UVW[2]+UVW[3]*UVW[3] > 1e6 5.4 s (11.7 s). A subscript that does not depend on the row (UVW[1], DATA[1,2], V[end]) is compiled like the rest of the condition.

Editing and writing

An edit session rewrites the storage-manager file of every column it touches (and the columns that share the file), so do all your changes in one session rather than one session per cell:

update!("/path/to/my.ms"; set = ["INTERVAL" => "INTERVAL * 1.0"])    # a whole column: ~1 s per million rows

Overwriting a few single cells of numeric or Bool (flag) StandardStMan columns (fewer than one in eight rows) patches those cells in the file in place (1000 cells of a 1M-row table: ~12 ms, a single cell ~10 ms; it was ~0.38 s); anything else touching a column regenerates its data-manager file. Removing or appending rows regenerates the table's files (removerows! of 1000 of 1M rows: ~0.35 s); a tiled visibility cube is rewritten completely. copyms / create_ms of a 20 000 × 64 × 4 MS take about 0.2 s.

Reference-frame conversions

measconvert to AZEL, AZELGEO, HADEC, APP or ITRF builds SOFA's astrometry context for the frame once and caches it, so converting many directions under one MeasFrame costs ~7 µs each; create the frame once and reuse it:

fr   = MeasFrame(epoch = MEpoch{UTC}(60454.42), position = observatory("VLA"))
dirs = [MDirection{J2000}(0.1i, 0.5) for i in 1:1000]
azel = [measconvert(d, AZEL; frame = fr) for d in dirs]

Threads

The caches (the per-table Column objects, the data managers, the compiled expressions, the astrometry context) are safe to use from several tasks at once; concurrent reads, queries and conversions are tested under julia -t 8.

Copying a table larger than memory

copytable and copyms of a plain table copy the tiled array columns (DATA, FLAG, WEIGHT, UVW, ...) in chunks of rows: each chunk is read with one block decode, packed and appended to the new tile file, so the memory needed no longer grows with the size of the visibilities. On the real 9.8-million-row MAIN table (20 GB of DATA) a copy of 3 million rows peaks at about 2.2 GB (physical footprint) and runs at roughly 75,000 rows/s; the scalar columns (TIME, ANTENNA1, ... about 0.4 KB per row) are still held whole. Columns are chunked when they have 100,000 rows or more and the table is written to plain files (not storage=:multifile). Julia's garbage collector lets the heap grow well past what is live; starting Julia with --heap-size-hint=2G keeps a long copy near the live size (1.5 GB in the same test) at some cost in time.

First call in a session

The package image carries a precompile workload (a small table is written, queried, grouped, joined, updated, edited and copied while the package is precompiled), so the first query, groupby or taql call of a session takes a few hundredths of a second. Before the workload the first taql("... GROUP BY ... INTO ...") compiled for 19 s and the first query for 2.6 s. Loading the package takes about half a second; the extensions (SOFA, Unitful, ...) compile the first time a conversion or a unit literal is used.