Tracking the Use of Open Educational Resources Sounds Simple. It Isn’t.

Open educational resources (OERs)—that is, free-to-use monographs, textbooks, and other educational materials—are having something of a moment in Australia and New Zealand. Through a very high membership (42 participating institutions in 2024), the Council of Australasian University Librarians (CAUL)’s OER Collective has been a significant driver of interest and participation in institutional publishing of OERs through libraries.

But what do we know about their use? The four of us looked into this question and fell down a very deep hole. It turns out the data is scattered, challenging to collect, and incomplete. Here is what we learned as we dug our way out.

The Original Question

This enquiry began when one of us, Dr. Danny Kingsley, was working as a community manager for OAPEN (Open Access Publishing in European Networks), the global library of open access books. Danny proposed a study into the use in teaching and learning of open monographs in the OAPEN library or listed in the Directory of Open Access Books (DOAB). This study initially considered the question of whether OAPEN books—which, by definition, are not textbooks—were being used for teaching and learning in Australia and New Zealand.

The rest of us—Sarah Brundrett (University of South Australia), Dr. Kay Steel (Federation University Australia), and Richard White (University of Otago)—answered a call for people interested in contributing to the project. Along with Danny, we set out to develop and share a research method that could be replicated in other jurisdictions. The plan was to test it against our own region and then share with our community across the globe for comparison. These were lofty plans indeed.

Initial Thinking

We were sure someone had already tried to measure the usage of open access monographs for teaching. But our (admittedly light-touch) literature search did not surface any study that had looked at this particular issue.

As with any research question, the first decision we needed to make was how to approach it. For example, what did we mean by OERs (there are several PhDs in that question alone), and how do open monographs fit into that definition? Identifying data we could use was also a very early consideration and challenge. There is great variability in sources of data on open teaching material and the majority of institutions deliberately do not track users.

One option was to look at university course reading lists. However, given the complexities of how these lists are managed across institutions and the multiple platforms in place, it was not clear whether we would be able to structure questions for a survey of universities that would give us meaningful responses.

A more streamlined approach was to use data provided by the Open Syllabus project, which ingests and analyzes syllabi, primarily those that are openly available online. Open Syllabus pulls out the readings from these syllabi and runs them against the Open Textbook Library and DOAB. The Open Syllabus Analytics service showed that open resources are currently a small proportion of texts assigned in syllabi (less than five percent overall). Within that five percent, the number originating from DOAB is almost the same as the number originating from Open Textbook Library.

This clearly answered the original question: open monographs are being used for the purposes of teaching and learning, almost to the same extent as open textbooks. However, it reinforced the broader question: what are we talking about when we talk about an open educational resource in an educational setting? We were also aware of the limitation that Open Syllabus’s analysis overwhelmingly relies on open syllabi, which can present a skewed view of an institution. This means the data are an indicator rather than a definitive count.

Triangulation

We decided to take a different approach to understanding usage of OERs by looking at the downloads of a specific text. A member of our group, Richard White, was one of a small team from half a dozen universities across Australia and Aotearoa New Zealand who “hacked”—that is, collaboratively wrote and edited in real time—a book in a weekend in 2013. Titled Media Studies 101, the book sits on a WordPress site and is available under a CC BY license. There is one other copy, on a Canadian university website.

Richard took multiple approaches to understand usage of this book, which he described in detail in this blog. The WordPress site on which the book is hosted provided some statistics, which show a clear pattern of uptake: views peaked at over 100,000 in 2017 and dropped afterward. This likely reflects the text being an early OER and that other resources have become available since 2018.

The download patterns also allow us to make some assumptions. For example, we see a peak in use during early parts of semesters in both the northern and southern hemisphere and the heavy use of specific chapters. These patterns indicate use for teaching, but, again, it is hard to know definitively.

However, once we move beyond the downloads from the specific host site things start to become much more difficult. Richard tried multiple search methods, including web search for title/author and ISBN and search using Gen AI chat bots. He widened his search to include OER indexing sites and DOAB before moving to reviews on indexing sites, because reviews often indicate use at the reviewer’s institution. This last method is clearly not scalable—it is time intensive, non-automatable, and has low return on investment.

The results from this work came in a variety of formats, some more downloadable than others. So once again, there appears not to be any standard, simple way to gain any understanding of the extent of use of open works for teaching and learning (or for any purpose).

Change of Direction

According to OAPEN, Australia is the sixth highest user of OAPEN books worldwide. But our question was whether they were being used for teaching and learning. This is quite challenging to determine.

OAPEN and DOAB metadata can be ingested into university library catalogues, which in turn can report on usage. OAPEN also collects information about downloads at the book and chapter level for a given institution based on IP addresses and the physical location of the primary campus. This information is available to any supporter institution through the OAPEN dashboard (a video of the dashboard can be seen here.) The dashboard provides real-time detail about downloads of specific titles using IRUS (Institutional Repository Usage Statistics) data and follows the COUNTER Code of Practice, which strips out bot activity.

We looked for usage patterns in the dashboard information—anything that might indicate a correlation between usage and the teaching or exam periods of a given institution. While we could see some promising patterns in individual institutions, we were unable to identify a consistent pattern across institutions.

A closer analysis of the download data yielded some interesting observations. In one research-intensive institution, the pattern of usage across the year did seem to indicate the materials were being used for teaching and learning. A significant proportion of the top 25 downloads in 2023 were related in topic and likely all part of a single course. A search into the courses at the institution identified a likely candidate.

This laid open a secondary consideration which we then pursued—how much can the uptake of OERs in an institution be attributed to individual practice as opposed to institutional policy? Is the usage of these monographs from OAPEN due to a few champions, or is there a policy or support in place that is systematically encouraging it?

Following the Policy Trail

To answer this question, our first step was to establish where policies existed in Australasian universities. We were also interested in other support for OERs, or even the idea of “no cost/low cost” texts. The team started by conducting a basic Internet search and querying the subscription resources available via our institutions in June 2024, followed by an email query via the CAUL Deputies discussion list and a follow-up to both members of CAUL’s Australian Scholarly Communications Community of Practice and the CAUL Open Educational Resources Collective, in which we asked questions about existing OER policies or support.

This was not a formal survey, yet the responses were significant in terms of volume and content. By October 2024 we had responses that revealed a variation in the extent of explicit support for OER development and use in the Australasian university sector. In total, 24 universities (21 out of the 42 universities in Australia, and 3 of the 8 in New Zealand) responded.

The support offered for OER development and use was varied. An impressive 74 percent of respondents (17) had a LibGuide on OERs, 4 institutions were offering grants to create OERs, and 3 were running open presses. A large number of respondents indicated membership in the CAUL Open Educational Resources Collective and several mentioned other advocacy work they are doing within the university.

For a causal relationship to be established between the existence of a policy, procedure, or guideline and an increased uptake of OERs, the team would have needed a decent sample size. However, only eight institutions indicated any formal position, in each case apparently developed under the aegis of the library or university teaching and learning portfolios and often framed within institutional IP policies.

James Cook University has an Open Scholarship Policy, Queensland University of Technology an Open Education Resources Policy, and Royal Melbourne Institute of Technology an Open Scholarship Policy. Curtin University mentioned their Open Scholarship Policy had recently been approved. Griffith University refers to OER in their Scholarly Resource Strategy and Sustainability Strategy.

Charles Darwin University has an Open Educational Resources Procedure and Southern Cross University’s Assessment, Teaching and Learning Procedure promotes the adoption of OERs as the preferred option for textbooks.

At the University of Otago, the Open Access Guidelines encourage the use of open teaching resources.

These policies and procedures are too few in number, and in some cases too recent, for us to establish a causal link between them and an uptake of OERs.

A Ray of Light

There is one service that looks at the usage of Open Access books at the publisher level—the Book Analytics Service (BAS) hosted by OAPEN and developed with the Curtin Open Knowledge Initiative. The BAS attempts to address exactly the issues we have identified—inconsistencies between what is being counted and the form of the outputs. The BAS synthesizes outputs from publishers into a form that can be more easily understood. (See, for example, the dashboard for Australian National University Press.) This is a welcome addition to the landscape, and we encourage other open presses to engage and support this capability as it moves from project funding to becoming self-funded.

While a great leap forward, the BAS is, by its nature, focused on open books. It is a piece in a very patchy puzzle. We have no equivalent for other types of OERs.

Where We Ended Up

We started this work by wanting to understand whether open monographs were being used for teaching and learning. This was relatively straightforward to answer—they clearly are—but the investigation opened a new line of enquiry. The second query aimed to determine the availability, accessibility, and standardization of information about the use of OERs such as open monographs and open textbooks. We found:

• There is some information available, comprising a mix of public/open data and information accessible through memberships or subscriptions. The type of detailed and specific information provided by OAPEN for open monographs is an outlier.
• Currently, the information differentiates between “textbooks” and “monographs,” despite both being used as OERs.
• The information lacks standardization, making it fragmented and challenging to locate and, therefore, to analyze.

As a result, we can’t answer questions about usage of OERs or causality of policies on uptake of use of OERs. That said, there is sufficient publicly available data to give a general idea of use and to prompt some next steps.

We recommend a sector-wide initiative to standardize and consolidate information about the use and reuse of both open monographs and open textbooks. Such an initiative would:

• Allow stakeholders—including authors, libraries, and publishers—to understand the extent and nature of resource usage.
• Justify the time and effort invested in OER initiatives and build support for their expansion.
• Promote recognition at senior institutional levels of the value of OERs.
• Demonstrate the value of OERs to academics and institutions and encourage their reuse, thereby reducing the need for recreating new OERs on the same topics.

This research project sent us down many different lines of enquiry, often leading to dead ends that prevented us from coming to any concrete conclusions. What we have concluded is that poor OER metrics are a sector-wide problem, needing both forward and comprehensive thinking to manage. Let’s get on with it!