Support for OI-FITS data in Julia
[![License][license-img]][license-url] [![Build Status][github-ci-img]][github-ci-url] [![Build Status][appveyor-img]][appveyor-url] [![Coverage][codecov-img]][codecov-url]
The OIFITS.jl
package provides support for OI-FITS data in Julia language.
OI-FITS types
OI-FITS is a standard to store optical interferometry data as a collection of data-blocks. In the second revision of the standard (see Ref. 1 and Ref. 2), an OI-FITS file may contain the following data-blocks:
- an
OI_TARGET
data-block stores a list of observed targets; - each
OI_ARRAY
data-block describes a given array of telescope stations; - each
OI_WAVELENGTH
data-block describes a given instrument notably the effective wavelengths and bandwidths of its spectral channels; OI_CORR
data-blocks store correlation data;OI_VIS
data-blocks store complex visibility data;OI_VIS2
data-blocks store squared visibility (powerspectrum) data;OI_T3
data-blocks store triple product (bispectrum) data;OI_FLUX
data-blocks store spectral flux data;OI_INSPOL
data-blocks store instrumental polarization data.
These data-blocks are stored as binary tables in a FITS data file. The support for FITS files is provided by the FITSIO.jl
package.
The Julia type of an OI-FITS data-block is named after the corresponding OI-FITS extension (see table below). In addition to these types for individual OI-FITS data-blocks, objects of type OIDataSet
contain several OI-FITS data-blocks and are efficient representation of the contents of a compliant OI-FITS file.
OI-FITS extension | Julia type |
---|---|
OI_ARRAY | OIArray |
OI_CORR | OICorr |
OI_FLUX | OIFlux |
OI_INSPOL | OIInsPol |
OI_T3 | OIT3 |
OI_TARGET | OITarget |
OI_VIS2 | OIVis2 |
OI_VIS | OIVis |
OI_WAVELENGTH | OIWavelength |
Reading and writing OI-FITS files
Reading and writing OI-FITS data-sets
Reading an OI-FITS data file in Julia yields a data-set and is done by:
using OIFITS
ds = read(OIDataSet, input)
where input
it the name of the OI-FITS file or an instance of FITSIO.FITS
which represents an open FITS file. The above read
call is equivalent to the shortcut:
ds = OIDataSet(input)
It is possible to merge the contents of several OI-FITS file, say inp1
, inp2
, etc., by one of:
ds = read(OIDataSet, inp1, inp2, ...)
ds = OIDataSet(inp1, inp2, ...)
or to merge them into an existing data-set ds
:
read!(ds, inp1, inp2, ...)
Creating an OI-FITS file is as simple as writing the data-set ds
:
write(filename, ds)
Overwriting is forbidden by default, but the keyword overwrite=true
may be specified to allow for silently overwriting an existing file.
Reading individual OI-FITS data-blocks
It may be useful to read individual OI-FITS data-blocks, to debug or to fix the contents of a non-compliant OI-FITS file. To that end, you must open the FITS file and can then read a given HDU as an OI-FITS data-block:
using FITSIO, OIFITS
f = FITS(filename, "r") # open FITS file for reading
tgt = OITarget(f[i]) # read OI_TARGET extension in i-th HDU
tgt = read(OITarget, f[i]) # idem
db = OIVis2(f[j]) # read OI_VIS2 extension in j-th HDU
db = read(OIVis2, f[j]) # idem
...
any OI-FITS data-block type can be used in that way. If the type of the i
-th extension is not known, OIDataBlock
can be used instead but the result is not type-stable:
db = OIDataBlock(f[i]) # read OI-FITS extension extension in i-th HDU
db = read(OIDataBlock, f[i]) # idem
Writing individual OI-FITS data-blocks is also possible:
using FITSIO, OIFITS
f = FITS(filename, "w") # open FITS file for writing
write(f, db) # write db in the next HDU of f
To fix a non-compliant OI-FITS file (usually duplicate target or instrument names), you can read all the data-blocks, fix those which are wrong and push them in order in an OIDataSet
to have a consistent data-set which you can then directly use or write in an OI-FITS file for later. Thanks to the automatic rewriting of target identifiers and of the fact that targets (and other dependencies) are identified by their name and consistently merged, it is possible to push an OITarget
with multiply defined identical targets (apart maybe their identifiers).
Accessing the contents of data-blocks and data-sets
The contents of OI-FITS data-blocks and data-sets may be accessed by the dot notation but also by indexation.
Contents of data-sets
The dot notation can be used on a data-set object, say ds
, storing a consistent set of OI-FITS data-blocks. The following properties are available:
ds.target
is theOITarget
data-block of the OI-FITS structure.ds.instr
is a list ofOIWavelength
data-blocks indexed by a regular integer index or by the instrument name:ds.instr[i] # yields the i-th OIWavelength data-block ds.instr[insname] # yields the OIWavelength data-block whose name matches insname
Matching of names follows FITS conventions that case of letters and trailing spaces are ignored. An exception is thrown if the index (integer or name) is not valid. The
get
method can be used to provide a default value, for example:get(ds.instr, insname, nothing)
would yield
nothing
ifinsname
is not found inds.instr
instead of throwing an exception.ds.array
is a list ofOIArray
data-blocks indexed likeds.instr
except that interferometric array names are assumed.ds.correl
is a list ofOICorr
data-blocks indexed likeds.instr
except that correlation data array names are assumed.ds.vis
is a vector ofOIVis
data-blocks.ds.vis2
is a vector ofOIVis2
data-blocks.ds.t3
is a vector ofOIT3
data-blocks.ds.flux
is a vector ofOIFlux
data-blocks.ds.inspol
is a vector ofOIInsPol
data-blocks.
Other fields of data-sets shall be considered as private and not accessed directly.
Using the dot notation, it is easy to access the different data-blocks containing measurements. For instance:
for db in ds.vis2
...
end
is convenient to loop across all OIVis2
instances stored by ds
.
Contents of data-blocks
The contents of a data-block, say db
, may also be accessed by the dot notation. As a general rule, db.key
or db.col
yield the value of the keyword key
or the contents of the column col
of the OI-FITS table corresponding to the data-block db
. In order to follow Julia conventions and to accommodate for a number of restrictions, key
or col
are the FITS keyword or column name converted to lower case letters and with non-alphanumeric letters replaced by underscores. For instance db.date_obs
yields the value of the keyword DATE-OBS
, that is the UTC start date of observations. The revision number corresponding to the keyword OI_REVN
is however accessed as db.revn
, this is the only exception. Other properties are also accessible via this syntax:
db.extname
yields the OI-FITS name of the extension corresponding to the data-blockdb
(for all data-block types);db.array
yields theOIArray
data-block associated with data-blockdb
(only forOIVis
,OIVis2
,OIT3
,OIFlux
, andOIInsPol
data-block). Beware that the association with anOIArray
is optional, sodb.array
may be actually undefined; this can be checked byisdefined(db,:array)
.db.instr
yields theOIWavelength
data-block associated with data-blockdb
(only forOIVis
,OIVis2
,OIT3
, andOIFlux
data-block).db.correl
yields theOICorr
data-block associated with data-blockdb
(only forOIVis
,OIVis2
,OIT3
, andOIFlux
data-block).db.name
is an alias fordb.arrname
forOIArray
instances, fordb.insname
forOIWavelength
instances, and fordb.corrname
forOICorr
instances.
Of course, getting a given property must make sense. For example, db.sta_name
is only possible for an OIArray
data-block but not for an OIWavelength
data-block. The dot notation can be however be chained and:
db.instr.eff_wave
can be used to access the effective wavelengths of the measurements in db
via the instrument associated to db
. Shortcuts are provided:
λ = db.eff_wave # get effective wavelength
Δλ = db.eff_band # get effective bandwidth
for OIWavelength
data-blocks but also for OIVis
, OIVis2
, OIT3
, and OIFlux
data-blocks.
Some fields of a data-block db
may however be undefined because:
the field is not yet defined (the data-block is being constructed);
the field is optional in the revision
db.revn
of the data-block;the field (for example
db.instr
for anOIVis
data-block) involves links with other data-blocks (the dependencies) and these links are only defined when a data-block is part of a data-set (see Building of data-sets below).
OITarget
data-blocks
For efficiency, instances of OITarget
data-blocks do not follow the same rules as other types of OI-FITS data-blocks whose properties are the columns of the corresponding OI-FITS table: in an OITarget
instance, all parameters describing a target are represented by an OITargetEntry
structure and all targets are stored as a vector of OITargetEntry
. An OITarget
instance, say db
, has the 3 following properties:
db.extname # yields "OI_TARGET"
db.list # yields a vector of OITargetEntry instances
db.revn # yields the revision number
The list of targets db.list
can be indexed by an integer (as any Julia vector) or by the target name (case of letters and trailing spaces are irrelevant).
As an OITarget
data-blocks is essentially a vector of target entries, it can be used as an iterable and it can indexed by an integer index or by a target name:
length(db) # the number of targets, shortcut for `length(db.list)`
db[i] # the i-th target, shortcut for `db.list[i]`
db[key] # the target whose name matches string `key`, shortcut for `db.list[key]`
Standard methods get
and haskey
, applied to db.list
or directly to db
, work as expected and according to the type (integer or string) of the key. For the keys
method, the default is to return an iterator over the target names, but the type of the expected keys can be specified:
get(db,key,def) # yields `db[key]` or `def` if `key` not found
keys(db) # iterator over target names
keys(String, db) # idem
keys(Integer, db) # iterator over target indices
keys(Int, db) # idem
The method OIFITS.get_column
is a helper to recover a single target field as a vector:
OIFITS.get_column([T,] db, col)
yields the column col
of an OI-FITS data-block db
. Column is identified by col
which is either sym
or Val(sym)
where sym
is the symbolic name of the corresponding field in OITargetEntry
. Optional argument T
is to specify the element type of the returned array.
To build an OITarget
instance, you may provide the list of targets and the revision number:
OITarget(lst=OITargetEntry[]; revn=0)
yields an OITarget
data-block. Optional argument lst
is a vector of OITargetEntry
specifying the targets (none by default). Keyword revn
specifies the revision number.
A target entry may be constructed by specifying all its fields (there are many) by keywords, all of which but category
are mandatory:
x = OITargetEntry(;
target_id ::Integer,
target ::AbstractString,
raep0 ::AbstractFloat,
decep0 ::AbstractFloat,
equinox ::AbstractFloat,
ra_err ::AbstractFloat,
dec_err ::AbstractFloat,
sysvel ::AbstractFloat,
veltyp ::AbstractString,
veldef ::AbstractString,
pmra ::AbstractFloat,
pmdec ::AbstractFloat,
pmra_err ::AbstractFloat,
pmdec_err ::AbstractFloat,
parallax ::AbstractFloat,
para_err ::AbstractFloat,
spectyp ::AbstractString,
category ::AbstractString = "")
It is also possible to specify another target entry, say ref
, which is used as a template: any unspecified keyword is assume to have the same value as in ref
:
x = OITargetEntry(ref;
target_id = ref.target_id,
target = ref.target,
...)
Note that, when an OITarget
instance is pushed in a data-set, target identifiers (field target_id
) are automatically rewritten to be identical to the index in the list of targets of the data-set.
Building of data-sets
Pushing data-blocks to data-sets
Reading an OI-FITS file is the easiest way to define a data-set but a new OI-FITS data-set may be built by creating an empty data-set with OIDataSet()
, and then pushing OI-FITS data-blocks in order with push!(...)
. Indeed, in order to ensure the consistency of a data-set, it is required to push the dependencies (OITarget
, OIArray
, OIWavelength
, and OICorr
data-blocks) before the data-blocks containing measurements (OIVis
, OIVis2
, OIT3
, OIFlux
, and OIInsPol
) that may refer to them.
For example, building a new data-set, say ds
, looks like:
ds = OIDataSet() # create empty data-set
push!(ds, arr) # push OIArray data-block(s)
push!(ds, ins) # push OIWavelength data-block(s)
push!(ds, cor) # push OICorr data-block(s)
push!(ds, tgt) # push OITarget data-block
push!(ds, db1) # push data
push!(ds, db2) # push more data
push!(ds, db3) # push even more data
...
with the dependencies:
arr
anOIArray
instance defining the interferometric array (zero or any number of such instances may be pushed),ins
anOIWavelength
instance defining the instrument (several such instances can be pushed),cor
anOI_COORREL
instance defining the correlations (zero or any number of such instances can be pushed),tgt
anOITarget
instance defining the list of observed targets (at least one such instance is required, if more such instances are pushed in the same data-set, they are merged in a single one);
and where db1
, db2
, db3
, etc., are instances of OIVis
, OIVis2
, OIT3
, OIFlux
, or OIInsPol
that provide measurements.
You may push all data-blocks in a single push!
call:
ds = push!(OIDataSet(), arr, ins, cor, tgt, d1, db2, ...)
and the following shortcut is implemented:
ds = OIDataSet(arr, ins, cor, tgt, d1, db2, ...)
These two are equivalent to the multi-line example above, but remember that pushing data-blocks in order (i.e., dependencies before they may be referenced) is required to have a consistent data-set. Apart from this constraint, dependencies may be pushed in any order before the data-blocks with measurements and data-blocks with measurements can be pushed in any order after dependencies.
As a benefit of the constraint of pushing data-blocks in order, data-blocks with dependencies are automatically linked to these dependencies when pushed on the data-set (which implies that the dependencies already exist in the data-set). This allows for syntactic sugar like:
ds.vis2[i].eff_wave # the wavelengths of the i-th OIVis2 data-block in ds
ds.t3[i].array # the interferometric array for the i-th OIT3 data-block in ds
ds.vis[i].instr # the instrument used for the i-th OIVis data-block in ds
Without linked dependencies, the first above example would require to (1) find in the data-set ds
the OIWavelength
instance, say instr
, whose name is matching ds.vi2[i].insname
and (2) extract the field eff_wave
of instr
. The latter step is as simple as instr.eff_wave
but the former one has some overheads and scales as O(n)
with n
the number of OIWavelength
instances in the data-set.
Since an OI-FITS data-set has a single list of targets (an OITarget
instance accessible via ds.target
), a mean to merge list of targets had to be defined. The adopted rule is pretty simple:
The
target_id
field of any data-block that is part of a data-set corresponds to the index of the target entry in the list of targets stored by the data-set.
As a consequence, whenever a data-block is pushed into a data-set, the target identifiers of the data-block have to be rewritten according to this rule. Of course, this does not apply for data-blocks with no target_id
field such as OIArray
, OIWavelength
, and OICorr
.
To summarize, here is what happens under the hood when a data-block db
is pushed into a data-set ds
:
When an
OIArray
,OIWavelength
, orOICorr
instancedb
is pushed in a data-setds
, it is appended to the corresponding list (ds.array
,ds.instr
, ords.correl
) unless this list already has an entry with a name matchingdb.name
. In this latter case, nothing is done unless that an assertion exception is thrown if the two data-blocks whose names are matching do not have the same contents (to prevent building inconsistent data-sets).When an
OITarget
instance is pushed in a data-set, the new targets (according to their names) are appended to the list of targets in the data-set and their identifiers set to their index in this list. This also re-initializes an internal dictionary used to perform the conversion from all the target identifiers of theOITarget
instance that has been pushed to the target identifiers in the data-set. Until it is reinitialized (by pushing anotherOITarget
instance), this mapping is used to rewrite the target identifiers of subsequent data-blocks pushed in the data-set.When an
OIVis
,OIVis2
,OIT3
,OIFlux
, orOIInsPol
instancedb
is pushed in a data-setds
, it is appended to the corresponding list (ds.vis
,ds.vis2
,db.t3
,db.flux
, ords.inspol
), after it has been linked to its dependencies (OIArray
,OIWavelength
, etc., which must already exist in the data-set), and its target identifiers have been rewritten according to the mapping defined by the lastOITarget
instance previously pushed to the data-set. Rewriting of the target identifiers may be avoided by using the keywordrewrite_target_id=false
, this assumes that the target identifiers in the pushed data-block are already set according to the index in the list of targetsds.target
.
Pushing a data-block in a data-set does check the consistency of the data-block. This is to allow for building the data-blocks step by step so that they not need to be consistent at all times (just when pushed into a data-set).
Pushing a data-block in a data-set lefts the data-block unchanged. A swallow copy of it is added to the data-blocks stored by the data-set. Most members of the pushed data-blocks are shared by the one stored by the data-set with the notable exception of the target identifiers which are rewritten and the links to the dependencies which are updated.
While it sounds complicated, the default rule of rewriting the target identifiers just amounts to assuming that the target identifiers of OIVis
, OIVis2
, OIT3
, OIFlux
, or OIInsPol
instances pushed in a data-set refer to the last OITarget
instance previously pushed on the same data-set.
Pushing several groups of data-blocks, each group making a consistent data-set, in the same data-set is easy. Typically:
# First push dependencies for group 1.
push!(ds, group1_arr) # push OIArray
push!(ds, group1_ins) # push OIWavelength
push!(ds, group1_cor) # push OICorr
push!(ds, group1_tgt) # push OITarget (reinitializing target_id mapping)
# Then push data for group 1 (using current target_id mapping).
push!(ds, group1_db1)
push!(ds, group1_db2)
...
# First push dependencies for group 2.
push!(ds, group2_arr) # push OIArray
push!(ds, group2_ins) # push OIWavelength
push!(ds, group2_cor) # push OICorr
push!(ds, group2_tgt) # push OITarget (reinitializing target_id mapping)
# Then push data for group 2 (using current target_id mapping).
push!(ds, group2_db1)
push!(ds, group2_db2)
...
Since they are referenced by their names, it is not necessary to push OIArray
, OIWavelength
, and OICorr
dependencies if they already exist in the data-set (according to their name), but it doesn't hurt. It is however mandatory to push an OITarget
instance with all targets and their identifiers as assumed by the subsequent data-blocks.
Merging data-sets
Two OI-FITS data-sets (or more), say A
and B
, can be consistently merged together by:
C = merge(A, B)
As much as possible, the resulting data-set C
will share its contents with A
and/or B
but without affecting A
and B
which are guaranteed to remain unchanged. As for pushing data-blocks, the target identifiers (the target_id
field) may be rewritten in the result.
Merging of data-sets assumes that the two merged data-sets are consistent and compatible. Here compatible means that targets and dependencies with matching names must have the same contents. This is checked during the merge operation.
It is also allowed to merge several data-sets and/or merge data-sets in-place:
ds = merge(ds1, ds2, ds3, ...) # merge ds1, ds2, ... in new data-set ds
merge!(ds, ds1, ds2, ds3, ...) # merge ds1, ds2, ... in existing data-set ds
Note that merge!(ds,...)
yields the destination ds
.
Also note that, after merging, the internal dictionary used for rewriting target identifiers is left with the mapping built from the targets of the last merged data-set.
Credits
The development of this package has received funding from the European Community's Seventh Framework Programme (FP7/2013-2016) under Grant Agreement 312430 (OPTICON) and is currently supported by the Jean-Marie Mariotti Center (JMMC).
References
Pauls, T. A., Young, J. S., Cotton, W. D., & Monnier, J. D. "A data exchange standard for optical (visible/IR) interferometry." Publications of the Astronomical Society of the Pacific, vol. 117, no 837, p. 1255 (2005). [pdf]
Duvert, G., Young, J., & Hummel, C. "OIFITS 2: the 2nd version of the Data Exchange Standard for Optical (Visible/IR) Interferometry." arXiv preprint [arXiv:1510.04556v2.04556].
[doc-stable-img]: https://img.shields.io/badge/docs-stable-blue.svg [doc-stable-url]: https://emmt.github.io/OIFITS.jl/stable
[doc-dev-img]: https://img.shields.io/badge/docs-dev-blue.svg [doc-dev-url]: https://emmt.github.io/OIFITS.jl/dev
[license-url]: ./LICENSE.md [license-img]: http://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat
[github-ci-img]: https://github.com/emmt/OIFITS.jl/actions/workflows/CI.yml/badge.svg?branch=master [github-ci-url]: https://github.com/emmt/OIFITS.jl/actions/workflows/CI.yml?query=branch%3Amaster
[appveyor-img]: https://ci.appveyor.com/api/projects/status/github/emmt/OIFITS.jl?branch=master [appveyor-url]: https://ci.appveyor.com/project/emmt/OIFITS-jl/branch/master
[codecov-img]: http://codecov.io/github/emmt/OIFITS.jl/coverage.svg?branch=master [codecov-url]: http://codecov.io/github/emmt/OIFITS.jl?branch=master