Introduction

Now operational, e-MERLIN [1] is a major upgrade to the Multi-Element Radio-Linked Interferometer Network, MERLIN [2], the United Kingdom’s national facility for radio astronomy and Very Long Baseline Interferometry (VLBI). The network consists of five remote antennas operated together with telescopes at Jodrell Bank Observatory (JBO) as an aperture synthesis array. With a maximum baseline length of 217 km, the angular resolution of e-MERLIN at 5-GHz is 40 milli-arcseconds, comparable to that achieved by the Hubble Space Telescope in the visual band. Table 1 shows the expected final capabilities of e-MERLIN once the upgrades are complete.

The e-MERLIN upgrade has increased the sensitivity of MERLIN by more than an order of magnitude, increased the frequency coverage, and enabled new modes of observation. These improvements are enabling a wide variety of new science, and are a combination of new broadband receivers and associated telescope electronics, together with a dedicated optical fibre network connecting each telescope at a bandwidth of 30 Gb/s to a new flexible high-capacity correlator at JBO. The new correlator, built by DRAO in Penticton, is highly flexible and allows for much more complicated observing modes, and hence a much more diverse range of science, than was possible with the old system.

Compared to MERLIN, datasets from e-MERLIN can be very large, especially when observations are made in ‘mixed-mode’ with additional high-resolution spectral windows covering astrophysically interesting spectral line transitions. Datasets are pre-processed by e-MERLIN staff using the first few stages of the pipeline described here, and made available to users as UVFITS files with associated tables containing flags and calibration information. Depending on the complexity of the research project and the experience of the user, data may be provided to the user either raw, or partially or fully processed. Options for making data available in other formats (such as CASA measurement sets [3]; ALMA/EVLA SDM and binary data) are under consideration for the longer term.

The staff of the national facility are developing scripts and pipelines to improve the accessibility of data products and images, including an on-line data archive for users to access their data once it has been observed and verified. This paper presents the data reduction pipeline for e-MERLIN, intended to reliably automate the process of data loading, housekeeping, averaging, interference rejection, calibration and imaging. The software is available to all users either from the e-MERLIN website, or via Github.

Briefly, the processes involved in processing a typical e-MERLIN continuum dataset are as follows. Each of these steps can be carried out by the pipeline either individually, or all of them in sequence, by varying the input parameters. Firstly the raw data (exported from the local archive in FITS-IDI format after correlation) must be loaded into AIPS. At this point a typical observation consists of multiple files, at least one per observed astronomical object. The data are then sorted if required, and some other housekeeping procedures are carried out. Once the data are loaded, a flag mask may be applied; this step creates a flag table for each dataset using a list of frequencies where interference is known to affect e-MERLIN telescopes. Averaging in time and/or frequency is then carried out, with the amount of averaging determined by the user depending on the needs of the experiment. If required, the data may be sorted at this point. The pipeline then produces a series of pre-calibration diagnostic plots which can be used to assess the quality of the data before proceeding with calibration. Automated flagging with SERPent [4] is then carried out, removing further interference-affected data, and post-flagging diagnostic plots may also be produced. At this point the averaged, flagged data may be backed up to disk outside of AIPS. The next step concatenates together all of the individual datasets, producing one dataset which can be calibrated. The data are then calibrated using standard procedures for interferometric arrays, including: fringe fitting, flux calibration, calculation of bandpass solutions, phase calibration, and optional self-calibration. Imaging may then be carried out, either as a single field, or in wide-field mode where the dataset is split up into small chunks, phase-rotated, imaged, and then mosaicked together. Any or all of these steps may, of course, be carried out by hand by the user without making use of the pipeline. A detailed description of the process of e-MERLIN data reduction covering all possibilities (including recommendations for dealing with mixed-mode spectroscopic data, polarisation calibration, astrometric observations, etc.) is beyond the scope of this paper since they are not yet implemented in the pipeline, the recommendations for dealing with these modes are still evolving, and current guidelines are provided in the e-MERLIN Cookbook [5] which is regularly updated.

Papers resulting from e-MERLIN datasets which have so far passed through the pipeline as part of the testing process include [6], [7], [8] and [9]. A large part of the development was carried out using the data published in [6], and data published in [7], [8] and [9] was also used to directly test pre-release versions of the code. Other publications which have resulted from data passed through development versions of the pipeline include: [10] [11] [12] and [13].

Implementation and architecture

Written in Python, the pipeline uses ParselTongue [14] to interface with AIPS [15], one of the main standard software packages for radio astronomy data reduction. The pipeline distribution consists of several files which are required for operation. The pipeline itself is contained in ‘eMERLIN_pipeline.py’, and requires ‘eMERLIN_tasks.py’ which contains many task definitions. A plain text file of inputs is also required; this can be named anything but must be specified on the command line at runtime. Table 2 shows the main control parameters, used to select which tasks to perform. Many of these tasks have additional parameters, all of which are documented in the example file ‘doall.inputs’ which is contained in the pipeline distribution. Any of the steps can be performed by setting the appropriate parameter to equal 1. The order of the steps is such that a raw dataset can be fully processed, in the correct order (described above), by simply telling the script to run everything.

Depending on which tasks are to be run, additional files may also be required. When applying the flag mask to flag (blank out) certain frequencies based on known sources of interference seen at e-MERLIN antennas, the file ‘flagmask512.fg’ must be in the working directory. Likewise, if applying the SERPent autoflagger [4], ‘SERPent.py’ must be present. The user must have write permissions in the current directory, and (if applicable) also in the output locations specified for fits backups and diagnostic plots.

The pipeline is executed by running > parseltongue pipeline.py inputs.txt on the command line, where inputs.txt is a plain text file containing a list of inputs necessary to tell the pipeline where the data are located and what steps are required (see Table 2). When downloaded from the repository, the pipeline comes with an example inputs file (‘doall.inputs’) which includes comments describing each parameter. In most cases, if a parameter is missing from the inputs file, a sensible default is used. When executed, the inputs are printed to the terminal window, along with a listing of the specified AIPS catalogue, before any tasks are performed, and the user is asked to check the inputs and confirm whether or not to continue.

Parameters which can be calculated from the data (such as the pixel size for imaging) can also be specified to override the numbers calculated in the pipeline. Sanity checks are made on some parameters to ensure that the numbers are sensible (a useful pixel size for example, or that the specified reference antenna is actually present in the dataset), or that the file catalogue within AIPS is in the expected state compared to the tasks requested.

Quality control

Extensive testing of the pipeline has been carried out by competent staff using a variety of production e-MERLIN datasets. The loading, housekeeping, flag mask and concatenation sections of the pipeline work reliably and are now used on every observed project. For well-behaved datasets, those without serious problems during either observation or correlation, the calibration section of the pipeline will produce useful data products. However the complexities of an operational interferometer and the nature of calibration are such that a suitably-trained human will always do a better job than an unintelligent piece of software; users are advised to check carefully the data products from this stage of the pipeline. Testing of the pipeline has, in fact, brought to light some interesting problems occurring further upstream in the e-MERLIN system.

For verification purposes, the main test datasets from v0.7 onwards are two observations carried out for use at a data reduction school held at JBCA in Manchester in January 2014. These datasets both consist of standard continuum phase-referencing observations of the gravitational lens system B1938+666, observed at frequencies of 1.4 and 5 GHz. Both datasets have been carefully processed by a human so that the data are well-understood, and reference data products are available to provide a direct comparison to the automated pipeline output. Figure 1 shows a pipeline-processed (with v0.7) image of the C-band verification dataset, together with an archive image of the same source produced by a human as part of a radio survey. For training and testing purposes, both of these verification datasets are available on the e-MERLIN website for download [5]. These two datasets will be run through each new release of the pipeline for verification, before the code is publicly released on Github.

Operating system

The pipeline will run on any linux system capable of running the underlying packages: AIPS (31DEC13 or later) [15], Obit [16], and Parseltongue [14].

Programming language

Python 2.7

Additional system requirements

If running the complete pipeline from raw correlator output, sufficient disk space is required to cope with twice the size of the input dataset (since AIPS essentially copies the data on loading), plus additional space to hold the averaged datasets. If data has been pre-processed by e-MERLIN staff, much less disk space is required, but performance may still be limited by disk I/O capacity.

Dependencies

Python 2.7; Obit; AIPS 31DEC13 or later; ParselTongue 2.0

List of contributors

The following have all provided segments of code: Neal Jackson, Rob Beswick, Nick Wrigley, Pierre-Emmanuel Belles (JBCA/e-MERLIN); Danielle Fenech (UCL)

Support

User support for e-MERLIN, including use of the pipeline, is available from e-MERLIN staff located at Jodrell Bank Centre for Astrophysics (e-merlin@jb.man.ac.uk).

Software location

Code repository

(e.g. SourceForge, GitHub etc.) (optional)

Language

The pipeline is written in Python using ParselTongue, the python interface to AIPS.