Sifting the GOTO transient stream with transient-optimised Bayesian source classification

GOTO web page

GOTO prototype paper

The Gravitational-wave Optical Transient Observer (GOTO) is a fully autonomous wide-field telescope array, designed specifically to hunt the afterglows of gravitational wave events. It is a multi-node, multi-site project, with each node made up of 16x40cm f/2 astrographs, capable of reaching magnitude 19.5 in 60s exposures, with a combined field of view of 47 square degrees. This combination of automation, depth, and wide field makes it uniquely suited for following up a wide range of fast transients, as well as providing a ~nightly cadence all-sky survey for large swathes of the sky.

This datastream is significant, with vast scientific discovery potential. GOTO is already accessing regions of transient parameter space that are difficult to probe with other facilities - the GW follow-up strategy is sensitive to a range of fast transients typically inaccessible with the slower cadence of other surveys. However, this is only possible with high-performance pipelines and machine learning filtering - with the burden placed on these scaling with increasing size of facilities. It is clear that the techniques used for last generation's surveys cannot be used in the upcoming LSST era, let alone with current sky surveys like GOTO and ZTF. It is therefore crucial to keep improving our classifiers, with even marginal percentage gains in accuracy adding up to 10s of thousands less candidates for humans to vet over a typical week. These algorithms have slowly climbed in accuracy over the past years with the introduction of deep learning.

We generate our training set in a fully automated way, to avoid human biases and enable large (~400k examples) datasets to be generated rapidly.

• Positive class (i.e. real transients): cross-match difference image detections with known asteroids in the field using the SkyBoT cone search HTTP API. Minor planets are a common contaminant in time-domain astrophysics - yet are useful here as they are 'real' PSFs - advantage over modelling.
• Negative class: remove all known minor planets and variable stars, and randomly sample the remaining difference image detections to generate a diverse set of bogus examples. Eyeballing indicates that this approach has <0.5% contamination, and is many orders of magnitude faster than human labelling.

With this approach, we generate a 400,000 example training set in under 24h, significantly larger than the largest human-labelled dataset. This set also accurately represents the intrinsic data distribution (magnitude, FWHM, etc.) and is free of human bias.

GOTO's specific science goals are focused around finding extragalactic transients, associated with galaxies. To optimise our classifier for the recovery of these types of objects, we extend our method above to generate realistic synthetic transients typical of those we are trying to detect. We do this as follows:

1. Select a known minor planet from the image
2. Search in the GLADE galaxy catalog for nearby hosts
3. Inject the minor planet at some random offset from the host by summing the stamps.

This generates very convincing supernovae that we can use to augment our positive classes.

To avoid the classifier 'cheating', we also inject blank galaxies as negative examples. In testing, without doing this, the classifier would classify anything near a galaxy as positive, which is counterproductive and is a trivial way to maximise recovery (at the cost of an overwhelming false positive rate!)

Uncertainty-aware predictions are generated using Monte Carlo dropout to perform approximate Bayesian inference. Gal and Ghahramani (2016) revealed a link between dropout as a regularisation method and approximate Bayesian inference. By sampling from realisations of the model with weights 'dropped out', set to zero, we can approximately sample from the predictive posterior of the network, gaining valuable information about the confidence of a given prediction. One such posterior distribution is plotted below.

We briefly experimented with active learning, and saw no gains over a random, naive selection. We think this is possibly because the active learning overprioritised weird artifacts which don't generalise to better performance.

We can combine classifier confidence as obtained via BNNs with other techniques to look deeply at the way our classifier has learned from the training set. In the below plot, we can identify specific areas of latent space the classifier finds harder to learn (i.e. has lower confidence) -- these lie largely where we expect (on class boundaries), but also indicate that specific morphologies of bogus detection (i.e. cosmic rays) are often confused with minor planets.

• Incomplete: we know that our completeness fractions for galaxy catalogs very rapidly fall off beyond ~100 Mpc. Therefore we might see the transient host in the image, but not in our meta-catalog
• Incorrect: Misclassifications are a natural part of large-scale catalogs, for example white dwarfs being misclassified as AGN in some photometric catalog.
• Inconclusive: Multiple catalogs may report multiple different object types for the same astronomical source, and it is unclear how to choose the correct one.

It is clear that different approaches are needed to perform nuanced classifications, whilst also taking into account the above issues.

To provide real-time contextual information to the classifier, we build an optimised database containing subsets of 12 catalogs of galaxies, AGN, and variable stars.

We classify under an 'apparent' scheme, using the labels

VS: variable star // OR: orphaned source // NT: nuclear transient // SN: supernova // BS: bogus source

We then train a CNN on images only, and feed this score into a random forest using the catalog information. We censor data randomly when training the random forest to promote balance between the CNN score and context.

On images alone, we obtain 90.5% accuracy, climbing to 98.5% accuracy when all contextual information is included. Below is a sample confusion matrix,

With the now relatively-pure stream provided by our real-bogus classifier, effort has shifted to more general source classification. This can be used in multiple ways (which we are currently developing):

• Automated prioritisation of candidates: minimises human vetting effort
• Automated real-time reporting to e.g. TNS
• Fully autonomous follow-up campaigns with other connected facilities

Most modern transient surveys detect variability via difference imaging, by subtracting a template observation to reveal objects that have changed in brightness since then. This is a complex process, as PSFs must be matched between images, and differential background corrected for. For various approaches to this, see Alard and Lupton (1999), Becker (2015), and Zackay (2016).

The vast majority of detections in this 'difference' image are merely artifacts of the process, arising from imperfect PSF matching or bad subtractions. There are also clear artifacts from bright stars, and cosmic rays that have been broadened into PSF-like shapes by convolution with the PSF-matching kernel.

With the high rates of false positives via the difference imaging method, how can we detect transients in real-time, with minimal human effort, and maximal recovery of real objects? This is where machine learning can help! By training a classifier on metadata of difference image detections with labels of positive (real) or negative (bogus), we can create a fully automated classifier to remove the majority of spurious detections.

Deep learning-based techniques currently lead this, which can represent far more complex decision boundaries than simple machine learning, and features are learned automatically via the process. Our new classifier is based on a convolutional neural network architecture, which we optimise in contrast to other works via hyperparameter tuning.

Email: t.killestein@warwick.ac.uk

Web: [institute] [personal]

Paper link: https://arxiv.org/abs/2102.09892

To verify real-world performance, we built a test set of over 700 spectroscopically confirmed transients seen in GOTO commissioning data via the TNS. We obtain a balanced accuracy of 97.2%, broadly in line with the scores we obtain on the test set (see below). We use subsets of this to check the consistency of performance with changing magnitude and host offset, plotted below:

Of particular interest: >30% improvement in recovery of faint (>19.5) transients, ~14% improvement in recovery of nuclear transients over a classifier trained purely on minor planets.

As is normal with ML approaches, we reserve 10% of our training dataset as a held-out test set, to confirm accuracy on data that has not yet been 'seen' by the classifier.

This shows excellent performance,  attaining balanced accuracies of 99.49% and 99.19% (F1 score: 0.9925) on the minor planet and synthetic transient test datasets respectively after extensive hyperparameter tuning.