The need to invest in research

This article is based on a journal article about the success and failure of South Africa’s ten-year innovation plan as measured by research output. But gauging research output is about much more than counting publications.

Research policies are the instruments that prescribe the direction in which funding for research flows. One would assume that any investment in research would have a positive impact on society. Unfortunately, this is not always the case. There is often a gap between the research and its applicability outside of academia – hence the strong call for relevant and interdisciplinary research, and hence the various instruments we have to measure the societal impact of research.

At macro level, societal impact would be reflected in the Gini coefficient. The Gini coefficient, a measure of inequality, typically measures socioeconomic status. South Africa has one of the highest Gini coefficients in the world, which is a great developmental challenge. It would thus be prudent for policymakers to address this challenge. This is the overarching purpose of the Ten-Year Innovation Plan (TYIP), published in 2008 by the South African Department of Science and Technology (DST). The DST claimed the ‘highest socioeconomic returns’ would be achieved by funding the following Grand Challenges: 

• The ‘Farmer to Pharma’ value chain (hereafter Farmer to Pharma) 
• Space science and technology (hereafter Space Science) 
• Energy and security (hereafter Energy) 
• Global change science with a focus on climate change (hereafter Global Change) 
• Human and social dynamics (hereafter Social Dynamics). 

An assumption driving the TYIP is that funding incentives can change the behaviour of researchers. Therefore, by looking at the behaviour of researchers, the effect of funding can be seen. But the relationship between funding and increased research output is not that simple. 

One study, which involved eight countries, found no relationship between competition for funding and research efficiency, as measured by publication output. However, in South Africa, research output has grown in past years in response to the funding model that financially rewards researchers on publication.

It would follow then that one of the measures of success would be publication counts, as outlined in the TYIP. Publication count is frequently used to evaluate research policy, but raw publication counts might simply show the direction in which all fields are moving. It would therefore be better to use the share of research output, which would show whether one field is growing faster than another (or whether one country’s output is growing faster than another’s). But this metric also comes with challenges. 

The TYIP’s aim was to increase South Africa’s share of the global research output to 1%. In 2002, it was reported at 0.5% and in 2006 it was still at 0.5%, with the goal for 2018 set at 1.5%. Now that the ten-year period has drawn to a close, a question is raised: Have South African researchers responded to the public declaration of research focus areas? 

Are our researchers really supporting the Grand Challenges?

Publication counts, as a measure of research success, fall within the scope of bibliometrics – the measure of research through output and citation counts. There has also newer methods, referred to as scientometrics, that estimate interdisciplinarity, impact, and a variety of other metrics. Bibliometric or scientometric methods are typically used to explore specific disciplines, gain insight into publishing patterns, examine the behaviour of researchers, and evaluate policies.

The second question that this article sought to address was whether the publications were aligned with the Grand Challenges, and whether South Africa has a specialisation in any of these areas. The country’s activity index, which shows the relative specialisation of a country or a university, varies widely between the different Grand Challenges: Global Change saw a decrease, Space Science and Farmer to Pharma appear to have changed minimally, and there was a slight increase in the specialisation of Energy and a significant increase in Social Dynamics. 

The third question relates to institutional differences: Which universities demonstrated an increased specialisation in line with the Grand Challenges over time? Nearly all the institutions showed a relative strength aligned with one or more of the Grand Challenges, and not only the research-intensive institutions, as one might expect. However, without intimate knowledge of the research drivers within each institution, it is impossible to say whether the specialisations noted in the study were intentional. 

The fourth question looked at possible problems with the metrics relating to research output.

This study wanted to determine the following:

1. Have South African researchers responded to the TYIP by publishing more research output? 
2. Is there evidence that South Africa’s areas of research specialisation have aligned with the Grand Challenges? 
3. Which universities in South Africa have demonstrated increased specialisation in line with the Grand Challenges over time? 
4. What are the possible problems with the metrics relating to research output? 

The database used for this study is Web of Science by Clarivate Analytics. It indexes highly ranked journals, creating citation links between the articles. It is said that about 62% of South African research can be found on Web of Science. Despite this, various South African studies have been based on Web of Science data. 

The first question that this article sought to address was whether South African researchers have responded to the TYIP by publishing more research output. Looking at the country’s share in the world’s research output from 1999 to 2018, an increase over time is evident – it has grown from just under 0.004% to 0.008%. Unfortunately, the TYIP’s goal of achieving a global share of 0.5% (or 1.5% in 2018) has not been achieved.

The second question that this article sought to address was whether the publications were aligned with the Grand Challenges, and whether South Africa has a specialisation in any of these areas. The country’s activity index, which shows the relative specialisation of a country or a university, varies widely between the different Grand Challenges: Global Change saw a decrease, Space Science and Farmer to Pharma appear to have changed minimally, and there was a slight increase in the specialisation of Energy and a significant increase in Social Dynamics. 

The third question relates to institutional differences: Which universities demonstrated an increased specialisation in line with the Grand Challenges over time? Nearly all the institutions showed a relative strength aligned with one or more of the Grand Challenges, and not only the research-intensive institutions, as one might expect. However, without intimate knowledge of the research drivers within each institution, it is impossible to say whether the specialisations noted in the study were intentional. 

The fourth question looked at possible problems with the metrics relating to research output.

Five possible reasons why our research output does not seem to increase

When evaluating the effect of the TYIP on the number of publications, it would seem that the TYIP has failed. Neither the identification of the Grand Challenges nor the bigger TYIP showed the desired effect of increasing the share of publications from South African authors to 1.5% of the world’s publications. There are five possible reasons for this.

Reason 1: The funding provided to address the Grand Challenges was obscured by other funding opportunities. One UK study speculated that unsuccessful applicants would seek, and likely find, funding from other sources, which would obscure the effects of national funding. In the case of the TYIP, funding for research interests outside the Grand Challenges was available, and it could well be that these other funding agencies obscured the impact of the TYIP. 

Reason 2: It could be that the study on TYIP outputs was premature. It has been shown that funding could have an immediate direct effect on output, and a delayed, indirect effect through collaboration. In the case of the TYIP, output from the funding could still be forthcoming in the next few years. In the case of Space Science, this would seem particularly likely. The National Research Foundation funded the construction of a giant radio telescope that will be a source of astronomical data – and research outputs – for decades to come. 

Reason 3: South Africa’s economy is not mature enough. One study looked at the productivity and efficiency of research and development financing in African countries, pointing out that countries need a level of maturity before they can claim a balance between productivity and efficiency (Dragos & Dragos, 2014). In terms of productivity, South Africa far exceeds other African countries. This could imply that South Africa has reached the required level of maturity and that there is likely a relationship between funding and productivity. 

Reason 4: Funding directed at the Grand Challenges had the opposite of the desired effect, namely it reduced the number of publications in certain areas. This contention could be explained by a study conducted in Italy, where a new policy led to the consolidation of research units (Coccia & Rolfo, 2007). The policy intended to increase the productivity of the research units, but it had the opposite effect. The authors concluded that the creation of large research units reduced the flexibility that smaller units offered. Large research units require significant funds, while smaller units were able to produce research with little or no funds. Taking these learnings into the South African environment, perhaps the TYIP has had the unintended consequence of reducing research by attracting researchers to bigger, well-funded research units and away from smaller, flexible research teams. 

Reason 5: Insufficient funding or available funding that is spread too thinly is another reason why outputs addressing the Grand Challenges do not have the desired effect. Perhaps a critical mass needs to be attained before the desired result will become visible. 

While there certainly could be more reasons behind the lack of success of the TYIP, these five reasons are offered as a starting point for future research and as input into future policies.

What now?

Ascribing the lack of desired output purely to the TYIP and its funding is too simplistic. Various factors can influence the productivity of researchers. Strategic direction and associated funding are just two possibilities. One must ask: What would the publication output have been in South Africa without any funding allocated to the Grand Challenges? Would South Africa have fallen further behind in the identified missions? It is impossible to say, but this is worth considering before one dismisses the TYIP as a failure. Not all Grand Challenges showed slow growth in publication output. This growth could well be the result of the TYIP and its funding. The TYIP provides several other success measures. It would be a worthwhile evaluating the success of the TYIP based on those measures too.

There are two suggestions for examining the impact of research that could provide an entry point into this discussion for future researchers. Firstly, McNie, Parris, and Sarewitz (2016) provided a typology that researchers can use to unpack who might find value in a research project. Secondly, Bhogal (2018) explained the contribution of the Square Kilometre Array to the South African economy. The SKA received most of the funds earmarked for Grand Challenges from the National Research Foundation. Despite South Africa not showing a specialisation in the area of Space Science, the funding towards this Grand Challenge has had a positive effect on the economy. 

This article showed how South African researchers have responded to the Ten-Year Innovation Plan (TYIP) of the Department of Science and Technology, particularly in light of certain areas of specialisation. It discussed the specialisation among the universities in the country. It commented on the problems with metrics relating to research output in an attempt to explain why South Africa has not shown the desired output. The activity index, which shows the relative specialisation of a country or a university, was used as the primary metric to evaluate the effectiveness of the TYIP. Publication counts were drawn from the Web of Science. It was found that South Africa fell short of the goal expressed in the TYIP. In areas of specialisation, the five Grand Challenges showed varying changes: from a decrease in ‘global change science with a focus on climate change’ to a significant increase in ‘human and social dynamics’. Universities in South Africa are shown to be multi-focused in their research. Five possible explanations were put forward as to why the output was not at the expected level.

The bottom line is this: Insights into the publishing patterns of South African researchers could help policymakers to define measures of success more effectively and draft policies that will more likely lead to the desired outcomes.

• Find the original article here: Skelly, L. (2021). The success and failure of South Africa’s ten-year innovation plan (2008) as measured by research output. Mousaion: South African Journal of Information Studies, 39(1). https://doi.org/10.25159/2663-659X/7921
• Bhogal, N. (2018). The role of the Square Kilometre Array in South Africa’s economic development strategy. South African Journal of Science, 114(3–4), 1-7. https://doi.org/10.17159/sajs.2018/20170297
• Coccia, M., & Rolfo, S. (2007). How research policy changes can affect the organization and productivity of public research institutes: An analysis within the Italian National System of Innovation. Journal of Comparative Policy Analysis: Research and Practice, 9(3), 215-233. https://doi.org/10.1080/13876980701494624
• Dragos, C., & Dragos, S. (2014). Scientific productivity versus efficiency of R&D financing: Bibliometric analysis of African countries. Current Science, 106(7), 942-945. https://www.jstor.org/stable/24102377
• McNie, E. C., Parris, A., & Sarewitz, D. (2016). Improving the public value of science: A typology to inform discussion, design and implementation of research. Research Policy, 45 (4), 884-895. https://doi.org/10.1016/j.respol.2016.01.004