Assessing Your Observatory's Impact: Best Practices in Establishing and Maintaining Observatory Bibliographies

Uta Grothkopf 1 ,  with collaboration of Observatory Bibliographers Collaboration (OBC) 

  • 1 European Southern Observatory (ESO), Garching

Abstract

Observatories need to measure and evaluate the scientific output and overall impact of their facilities. An observatory bibliography consists of the papers published using that observatory’s data, typically gathered by searching the major journals for relevant keywords. Recently, the volume of literature and methods by which the publications pool is evaluated have increased. Efficient and standardized procedures are necessary to assign meaningful metadata, enable user-friendly retrieval, and provide the opportunity to derive reports, statistics, and visualizations to impart a deeper understanding of the research output.

In 2021, a group of observatory bibliographers from around the world convened online to continue the discussions presented in Lagerstrom (2015). We worked to extract general guidelines from our experiences, techniques, and lessons learnt. We are crafting a paper which explores the development, application, and current status of telescope bibliographies, and the direction they are moving. The paper briefly describes the methodologies employed in constructing the databases, along with the various bibliometric techniques used to analyze and interpret them. We explain reasons for non-standardization, why it is essential for each observatory to identify metrics that are meaningful for them; caution the (over-)use of comparisons among various facilities that are, ultimately, not comparable through bibliometrics; and highlight the benefits of telescope bibliographies, both for researchers within the astronomical community and for stakeholders beyond the specific observatories. There is tremendous diversity in the ways bibliographers track publications and maintain databases, linked to parameters such as resources (personnel, time, budget, IT capabilities), type of observatory, historical practices, and reporting requirements to funders and outside agencies. However, there are also common sets of Best Practices. This poster describes some of our results from this effort.

Observatory Bibliographies & Bibliographers

Observatory Bibliographers Collaboration. (2023) "Assessing Your Observatory’s Impact: Best Practices in Establishing and Maintaining Observatory Bibliographies." arXiv: 2401.00060

The European Southern Observatory's (ESO) Telescope Bibliography (telbib) is maintained by the ESO library. It contains refereed publications that directly use ESO data and is the basis for metrics and visualizations that help characterize telescope data use over time.


https://telbib.eso.org/

An observatory bibliography consists of the refereed science papers published using that observatory's data or data product(s).


Science papers are defined as publications using data or data product(s) from the observatory to form the basis for reaching a new scientific conclusion. Papers that contain previously published data are included in the bibliography as long as the way the results are presented constitute new analysis or use. 

Observatory Bibliographers

  • Search the scientific literature, often by utilizing ADS for manual searching or through the ADS API.

  • Assess publications for inclusion based on parameters outlined in institution's data citation policies and in bibliographer community's best practices. 

  • Assess each publication manually to determine program(s)/data used, facility, instrument, dates, and more.

  • Assign metadata, which must be continually reviewed to account for changes in concepts, programs, and institutional goals.

  • Make bibliographies and metrics publicly accessible through ADS bibgroups and public libraries, and through institutional websites.

  • Provide publication data and metrics for reports to leadership and funding agencies.

Value to Researchers and to Archives

MAST/STScI Bibliographic Search. Proposal teams can use this tool to understand who has used their data to make new discoveries.

https://archive.stsci.edu/publishing/bibliography#/

 

Value to Researchers

  • Show researcher or research team's impact and reach, e.g., how many new discoveries relied on research team's data?

  • Deepen understanding of impact in astrophysics community, e.g., did my brown dwarf research lead to later exoplanet discoveries?

  • Inform collaboration: identify other researchers and teams using the same archival data to answer similar questions.

 

Snapshot of a few of the publications using data from JWST program 2107, "A JWST-HST-VLT/MUSE-ALMA Treasury of Star Formation in Nearby Galaxies"


Value to Archives

  • Measure reach and extent of archival data use in the astronomical community.

  • Make it easier to locate data used in publications through data links and DOIs indexed alongside publications in ADS.

  • Demonstrate connections between data usage — both new and archival — over time.

 

ADS record showing data link from publication to Chandra (CXC) archive

https://ui.adsabs.harvard.edu/abs/2024MNRAS.527L..47C/abstract

 

HST Paper Type Plot: shows increased growth of publications that rely on (green) Partial Archival + (orange) Pure Archival vs. new (blue) Guest Observer data

https://archive.stsci.edu/hst/bibliography/pubstat.html

 

Value to Institution & Funding Agencies

NSF NOIRLab Publication Metrics Dynamic Dashboard showing publication counts for NOIRLab telescopes by date; linked to ADS public libraries for individual NOIRLab telescopes. https://noirlab.edu/science/library/publications/metrics

Value to Institution

  • Demonstrate institution's scientific output.

  • Supply data on observatory, mission, telescope, archive, and instrument use.

  • Contribute to leadership reporting and decision-making.

  • Act as templates for new or revised bibliographies.

  • Support discovery of included papers through bibliographic search interfaces.

Value to Funding Agencies

  • Demonstrate utility and level of use of facilities and missions by astronomical community.

  • Show return on investments.

  • Inform funding agencies of support priorities.

  • Illustrate how observatory is advancing science. 

  • Guide decisions for future facilities development.

Observatory Bibliographies: Connecting the Community

Diagram showing connections between community, data, ADS, bibliographies, and funding

Comparisons

Comparing Observatory Bibliographies can be problematic and is not recommended.

  • Criteria for Inclusion vary between observatories, dependent on goals, reporting requirements, and leaderships' requests.

  • Differences in proposal process and data availability can impact or limit reuse of data in later publications.

  • Space-based observatories may have a distinct advantage over ground-based when it comes to total observing time.

  • Scientific goals of observatory influence which community uses the data (heliophysics vs. astrophysics).

  • Survey missions generate more data than targeted/object-oriented observatories.

  • Projects that deliver ready-to-use data products will be used and cited more frequently than those with data that require manipulation before being science-ready.

  • Journal coverage may vary; fewer journals assessed means fewer publications to include.

  • Extent of staff/resources dedicated to bibliographic work impacts bibliography.

Contributors

Contributors:
This poster has been created by the Observatory Bibliographers Collaboration (OBC), comprised of bibliographers from ADS, Center for Astrophysics / Harvard & Smithsonian, CXC, ESO, IAC, IPAC, IRSA, NAOJ, NRAO, NSF NOIRLab, NSO, and STScI.