野狼社区

This Open Research Perspective, contributed by Dr. James Bird, Technical Specialist in Research IT, explores the relationship between research trustworthiness, evaluability and impact, using data accessibility as one practical lens. It argues that while Open Research has improved the reach of research outputs, greater attention is still needed to ensure those outputs are genuinely assessable, reusable and trusted. Against that backdrop, James considers how Electronic Research Notebooks (ERNs) could help strengthen research workflows and support more FAIR, trustworthy research.

Disclaimer

I must clarify right from the start that this Perspectives article intentionally avoids providing definitions of research quality, value or impact. The purpose is not to wade into the debate on what they may mean, or how any might be used for the purpose of research assessment. Each should be interpreted in its most general sense, as they are referenced only to explore where research trustworthiness may enter the (intentionally, oversimplified) equation. The hope is that I might convey the role that software can play in supporting trustworthy research output generation. Hopefully it is recognised that on this basis, the metrics discussed are used responsibly ( at a minimum) and are not genuinely being used for any formal research assessment.

Framing "impact": An oversimplified model

For this exercise, I will make several modelling assumptions; simplified conditions intended to make the real-world challenge more tractable for this framing.

1. Greater reach equates to connecting with a greater proportion of the world.
2. An individual research output has a maximum value of 鈥渋mpact鈥�, independent of time, which relates to some inherent value of that output to the world.
3. Tying the first two together, I suggest that that maximum value of "impact' should be reached at a rate that correlates to both the reach and trustworthiness of that output.

The basic framing is therefore simple: the more people who can both reach and trust an output, the greater the rate at which maximum "impact" is reached. There are no doubt numerous other variables, and there is no suggestion of a clear linear relationship. One final implied assumption is that for this basic model, we need not be concerned with who is reached; the greater the cross-section of society accessing the research, the greater the absolute number of individuals that can make use of it.

Defining trustworthiness

Egon Guba and Yvonna Lincoln are often cited (Amin et al., 2020; Stahl and King, 2020; Ahmed, 2024) as the first to define the core components of trustworthiness in qualitative research, as Credibility, Transferability, Dependability and Confirmability, in a book published in 1985 (Lincoln and Guba, 1985). There is also an earlier article from 1981, formally credited to Guba alone(Guba, 1981), which states the same four components.

Jumping forward 45 years, perhaps the most recent distillation of decades of Metascience on trustworthiness was published this February (Nosek et al., 2026). The authors attempt to define a framework for research at large which both maps to, and updates those previously recognised components to suggest trustworthy research must be: Accountable, Evaluable, Evaluated, Well-Formulated, able to Control Bias and Reduce Error. The authors claim that these components are relevant across three levels: 'the research itself, researchers conducting and evaluating the research, and organisations including institutions, funders, and journals鈥揻acilitating and supporting the research'.

Evaluable research

I'm going to home in on the evaluable facet of trustworthy research, which I will argue bears some relationship to its 'FAIRness' (Wilkinson et al., 2016). Whether the research can be evaluated depends on:

• Firstly, whether it is Findable and Accessible (both of which should correlate to reach).
• Secondly, what exactly is shared. Arguably, everything and anything should be shared that led to the findings. The trustworthiness framework (Nosek et al., 2026) suggests plans, data, materials, code and outcomes be shared.
• Finally, how all these outputs are shared. Interoperability and Reusability (i.e. FAIR) should ensure that they are evaluable.

always appears in my mind when thinking 'FAIR', and no doubt reminds many of us of a time we sought to evaluate or reuse outputs.

Is our research evaluable and FAIR?

To explore in our local context whether our research outputs are evaluable, below I have tried to use a so-called 'trust marker' to assess some aspects of the Findability and Accessibility of research data. The trust marker I will use, so-called within the ^*, is the presence of a Data Availability Statement (DAS) in publications.

Again, putting caveats at the forefront, I will not seek to instil much trust in these data, as the irony would be far too great. Without intending to discredit the tool or developers, I personally, at the time of writing, am unable to verify that the tool was trustworthy when the data was exported, in September 2024. I was unable to 'look under the hood' at the way in which the tool identified publications of which the University of 野狼社区 researchers are authors, how they were categorised into Fields of Research, how the presence of DAS were confirmed, or how the contents of them were classified. Nonetheless, the tool was said to carry out these functions, which I have reinterpreted to generate the plots below. You can, however, evaluate my treatment of the data using the attached Jupyter notebook and environment file, if you wish.

Given the notable caveats, I will note some basic observations, such as the increase in the use of DAS over time, across all fields of research, on average (c.f. top-left figure). Looking outside the data, we generally know this to be true considering wholesale changes to the attachment of data availability statements to research publications. As for the classification of those DAS (c.f. top-right figure), we see that those identified as stating the data is 'available on request', 'location not stated', and 'not publicly available' trend upwards, whilst those statements classed as holding data either 'in online repository' and 'in supplementary files' trend down. On grouping the DAS classifications where we might presume the data associated with the publications are either directly accessible, or not required for evaluation ('in publication', 'in online repository', 'in supplementary files' and 'not applicable'), we observe a decrease in the proportion of publications containing immediately accessible datasets (c.f. bottom-left figure). Finally, on multiplying the mean percentage across all publications, independent of Faculty-alignment, which carry a DAS, with the percentage of DAS which indicate data accessibility, we arrive at a measure for the percentage of all identified University publications which point to immediately accessible datasets (c.f. bottom-right figure).

The implications of these findings, if they can be trusted, are those Findable publications (by the tool, at least) are increasingly using DAS. DAS adoption has not led to an increase in data Accessibility, as a proportion of overall DAS adoption, and the majority of DAS indicate data are not immediately accessible. The immediate accessibility of datasets within the growing number of DAS-containing publications, increased on average at a rate of ~1.5% per year, up to 9.2 卤 4.5% of all papers found in 2023.

To top and tail these implications, I'll state a few additional caveats:

• The data provided clearly only cover the period 2018-2023.
• Large error bars represent the relatively wide distribution of DAS usage across Fields of Research.
• The sum of all DAS classifications for a given year is typically ~105% (most apparent in the bottom-left figure), so the tool is responsible for some double-counting.
• No comment can be made on the Accessibility of data for which there was no DAS.
• No comment can be made on the Interoperability or Reusability of presumed Accessible datasets.

Is this unFAIR?

On making the (presumed entirely incorrect) modelling assumption that non-DAS containing publications have inaccessible data, based on these data, it would take until 2085, on average, for all our research data associated with publications to be immediately Accessible. The growth in open access publishing has no doubt increased Findability and the reach of research outputs, but by the , have we struggled historically with meeting the broader principles, such as making "outputs [sic] freely accessible as soon as possible under conditions that maximise reuse to amplify social, economic and research benefits"?

The role of 'organisations supporting the research'

If there is some underlying truth to these assumptions and data, and the observed trends have continued, what might we be able to do about it? I want to explore how the third player in research trustworthiness, 'organisations [sic] supporting the research', might better support more evaluable, and perhaps therefore more trustworthy, research output generation. Optionally, this might be viewed as accelerating research 'impact' (by the oversimplified model provided) and increasing alignment to Open Research practice.

We know that the generation and collection of new data is a decreasingly analogue activity. We also know that research data has numerous start and end points: we're not generally short on data-generating equipment nor data repositories or journals. We're collecting more digital data to back up our research findings, and publishing in digital-only journals, but somehow those data pipelines are being disrupted. We seem to be short on a sort of cyber-glue which could bind it all. Surely one answer therefore lies in improving digital research infrastructure. Crucially, to support FAIR data publication, I feel we must enhance data coordination throughout the entire research lifecycle and across the numerous contributors and collaborators that typically need to engage along the way.

Enter Electronic Research Notebooks

From my biased perspective, as a prior Electronic Research Notebook (ERN) user for my PhD here at the University, one fix for this small aspect of the much bigger picture starts with providing trustworthy data-coordinating software for research. I am not the only one to think this, however. The idea has been around for decades and many publications exist on the topic (c.f. my final 'blog' post as part of a series ), industry are known to have adopted them more liberally than academia, and I know that many of you have too.

The software family of ERNs, or Electronic Laboratory Notebooks, ELNs, for lab-oriented products, can be positioned centrally to the research data lifecycle, to act as that cyber-glue across the digital research journey. The graphic below was produced to demonstrate just that. The ERN can sit at the centre of the research data lifecycle (), allowing metadata and data ingress and egress across the numerous programs researchers make regular use of. The labelled programs are known to be used at The University, but their placement is of course not restricted to any single phase of the lifecycle. 

In this way, ERNs can coordinate data, metadata, protocols, inventories and more, across teams and collaborators, to ensure that when it comes to the sharing of research outputs, there need not be an extensive data-wrangling exercise to facilitate it. Audit trails, timestamping, digital signatures and user authentication are some of the ways that individual contributions are recognised throughout. Advanced searchability features can bring everything together on request, to be exported into an interoperable format, supporting long-term reusability. In short, they could pump some AIR into FAIR.

The ERN Project

A free to use, centrally supported ERN product is on the way (see our previous announcement ). If you wish to see what it will look like, you can get hands-on or watch a video on the project page . If you're intrigued enough to want to join the conversation and receive regular updates, you can join our Teams channel .

Footnotes

This article was written in a development version of the ERN software to be implemented. You can see how the ERN helped coordinate data for this article and evaluate my findings if you wish. [Attach Figshare link once content confirmed].

*The Dimensions Research Integrity app is now seemingly part of a much broader product, .

Dr. James Bird, Technical Specialist in Research IT

References

Ahmed, S.K. (2024) 鈥楾he pillars of trustworthiness in qualitative research鈥�, Journal of Medicine, Surgery, and Public Health, 2, p. 100051. Available at: .

Amin, M.E.K. et al. (2020) 鈥楨stablishing trustworthiness and authenticity in qualitative pharmacy research鈥�, Research in Social and Administrative Pharmacy, 16(10), pp. 1472鈥�1482. Available at: .

Guba, E.G. (1981) 鈥楨RIC/ECTJ Annual Review Paper: Criteria for Assessing the Trustworthiness of Naturalistic Inquiries鈥�, Educational Communication and Technology, 29(2), pp. 75鈥�91. Available at: .

Lincoln, Y.S. and Guba, E.G. (1985) Naturalistic Inquiry. SAGE. Available at: .

Nosek, B.A. et al. (2026) 鈥楢 framework for assessing the trustworthiness of scientific research findings鈥�, Proceedings of the National Academy of Sciences, 123(6), p. e2536736123. Available at: .

Stahl, N.A. and King, J.R. (2020) 鈥楨xpanding Approaches for Research: Understanding and Using Trustworthiness in Qualitative Research鈥�, Journal of Developmental Education, 44(1), pp. 26鈥�28. Available at: .

Wilkinson, M.D. et al. (2016) 鈥楾he FAIR Guiding Principles for scientific data management and stewardship鈥�, Scientific Data, 3(1), p. 160018. Available at: .