Research quality measures 1

Literature appraisal

Prof. Atif Iqbal (Fellow, IET (UK) and IE (India))

Adapted from original lecture of:

Simon Judd, Prof.

Chemical Engineering, QU Research Methodologies, July 2018

Subject areas, cohort

Cursory papers appraisal

You have three papers, all concern a phenomenon called Antiscale Magnetic Treatment (AMT):

• Cho, Y. and Choi, B-G. (1999), International Journal of Heat and Mass Transfer, 42, 1491-1499.

• Gabrielli, C., Jaouhari, R., Maurin, G. and Keddam, M.

(2001), Wat. Res. 35(13) 3249-3259.

• Vedayasan, C. (2001), Desalination, 134, 105-108.

Which one is the best and which is the worst, and why?

You have 10 minutes.

Research quality measures 1

Perception is that:

• the “best” papers are published in the “best” journals

• the “best” journals are the ones with the highest “impact”

• the impact is quantified as the Impact Factor (IF)

For a given year, the Impact Factor of a journal is given by the ratio:

number of times articles published in journal are cited number of citable items published by that journal

for articles published in the previous two years, where a

“citable item” is usually a paper or review and is decided by the editor of the journal.

Thomson Reuters database

• Contains all journal IF values in “journal citation reports” (JCR)

• Database has two sections: Physical Science and Engineering (“Science”) and Social Science

• It is searchable, but the search criteria are quite specific:

• search for specific journal

• list journals in specific disciplines or subject areas

• It provides a lot of journal data, the most useful being the IF (1y and 5y values) and the immediacy index

An immediacy index is a measure of how topical and urgent work published in a scientific journal is.

SCOPUS/Science Direct

• These databases contains all journal publications by the publisher Elsevier, plus many others

• It is searchable by a number of different criteria, including:

• word(s) in article title, abstract, and keywords

• author name and/or affiliation

• journal title, or word(s) in the title

• publication year

• The database provides information categorised by:

• author

• paper

Impact factor: limitations

• Some types of publications, e.g. letters and commentaries, are used to count citations (the numerator), but are not counted as

“papers” (the denominator), and hence inflate the journal’s IF;

• IF depends on the number of references, which differs among disciplines and journals;

• Inclusion of journals in the database depends solely on

Thomson Reuters, a private company, and not on the fields’

practitioners;

• Exact IF published by Thomson Reuters cannot be replicated using publicly available data;

• Distribution of citations/paper is not normal: at the very least the mode or median should be used instead of the mean;

• The two-year span for papers followed by one year for citations is completely arbitrary, and favours high-turnover over long- lasting contributions; and

• Journal editors can manipulate and artificially inflate their IFs by selecting review papers for publication.

Impact factor: the reality

• Journals of high IF tend to attract good quality papers

BUT

• Not all good quality papers end up in high IF journals

• The journal IF is not an absolute measure, because it will depend on the subject matter.

• What is the range of IF values for journals dedicated to the following disciplines:

• computer science, software engineering

• electrical engineering

• cultural studies

Citation ..?

1. What’s a citation?

2. What type of papers are likely to get cited a lot?

3. If you were a journal editor, how would you raise the IF of your journal?

4. What might be a better way of judging an individual’s research quality?

Research quality measures 2

The impact of an individual

researcher is generally assessed according to:

• number of papers published

• number of citations received

The most accepted representation of the above is the h index:

• A researcher has index h if their N papers have at least h citations each, and the remaining (N-h)

papers have no more than h citations each

h index

1. If you were working in research, what would be a good way of artificially increasing your h factor?

2. What about if you were (a) reviewing, or (b) writing a paper?

3. What sort of researcher would be unfairly penalised by having a low h index?

Who has more impact?

PROF SIMON JUDD

• 148 publications

• mostly since 2000

• 4716 cites in 33 y

• 8 papers > 100 cites

• 4 papers > 200 cites

• h index of 33

PROF STEPHEN HAWKING

• 128 publications

• mostly before 2010

• 17135 cites in 48 y

• 39 papers > 100 cites

• 25 papers > 200 cites

• h index of 56

Summary

1. The quality of a research article is quantified by:

a) the impact factor of the journal in which it is published, and

b) the number of citations it receives over a period of time

2. The quality of an individual researcher is quantified by the h index

3. All these measures are flawed to some extent

Criteria for appraising papers:

quick method (metrics)

POSITIVE

1. Journal IF

2. Journal ranking

3. Number of citations (in SCOPUS or Web of Science)

4. Number of recent publications cited

5. Percentage of papers in peer- reviewed journals

6. Journal IF of the cited publications

NEGATIVE

a. Number/% of elderly references b. Number/% of non peer-reviewed

publications

c. Number/% of self-cites

Criteria for appraising papers:

more rigorous method, 1

Read the Conclusions and assess whether they are:

• specific

• semi-quantitative

• contextualised

Look at the Results and assess whether:

• trends are clear

• they concisely expressed

• they support the Conclusions

• statistical analysis has been conducted

• measurements have been replicated

Criteria for appraising papers:

more rigorous method, 2

Read the Discussion and check that:

• the results are not misrepresented or misinterpreted (i.e.

claiming a strong correlation when there isn’t one)

• work by others is cited

Read the Materials and Methods and check that it is:

• sufficiently detailed

• where necessary refers to the appropriate methods published either in text books (Standard Methods) Read the Introduction and check that:

• benchmark reviews or recent key texts are cited

• the knowledge gap is clearly and correctly identified

Papers: positive and negative attributes

POSITIVE

A good paper will:

• provide clear correlations of processed data which

support new insight into the state of the art

• cite recent relevant papers by other authors to place the work in the appropriate context

• possibly supporting or contradicting a theory or current understanding

• obviously add to knowledge

NEGATIVE

A poor paper will:

• have few meaningful

correlations, and (possibly) a disproportionate amount of raw data

• cite few references from the peer reviewed literature and display no awareness of the state of the art

• may simply be stating what is already known or

repeating something which has already been done

Literature appraisal

Exercise

Presentation for Research Methodologies, July 2018

A Three-pass approach for reading a research

paper

Three-Pass Approach

• Gives you a general idea about the paper.

First pass

• Let you grasp the paper’s content, but not its details.

Second pass

• Helps you

understand the paper in depth.

Third pass

The First Pass

A quick scan to get a bird’s-eye view of the paper.

Decide whether you need to do any more passes.

1. Carefully read the title, abstract, and introduction

2. Read the section and sub-section headings, but ignore everything else

3. Read the conclusion

4. Glance over the references, mentally ticking off the ones you’ve already read

1

At the End of this Pass

…

You should be able to answer the five Cs:

1. Category: type of paper?

2. Context: problem?

3. Correctness: valid assumptions?

4. Contributions: main contributions?

5. Clarity: well written?

More passes?

1

The Second Pass

Read with greater care, but ignore details such as proofs.

Identify main idea and key points Make comments in the margins.

Look carefully at the figures, diagrams.

Remember to mark relevant unread references for further reading

• this is a good way to learn more about the background of the paper.

2

1

After this Pass …

Sometimes, you won’t understand it even at the end of the second pass 

• subject is new to you, with unfamiliar terminology.

• proof or so that you don’t understand.

• poorly written

• it’s just late at night and you’re tired!

2

1

After this Pass …

You can now choose to:

a) set the paper aside, hoping you don’t need it to succeed b) return to the paper later, perhaps after reading background

material or

c) persevere and go on to the third pass.

2

1

The Third Pass

To fully understand the paper (e.g., reviewing) Requires great attention to detail.

Attempt to virtually re-implement the paper Identify and challenge every assumption Write down ideas for future work.

3 2

1

4-5

hours

r

At the End of this Pass

…

Should be able to reconstruct the entire structure of the paper from memory.

Should be able to identify its strong and weak points.

Should be able to pinpoint implicit assumptions, missing citations to relevant work, and potential issues with

experimental or analytical techniques.

3 2

1

Bibliometrics

POSITIVE

1. Journal IF

2. Journal ranking in cohort (position/total) in JCR database 3. Number of citations (in SCOPUS), and citations/year

4. Number/% of recent publications cited (within past five or ten years)

5. Number/% of papers in peer-reviewed journals (journals listed in JCR database) 6. Journal IF and ranking of the cited publications (in JCR database)

7. Number of recent papers cited in the “Discussion” section 8. Number of correlations presented

NEGATIVE

a. Number/% of plots of raw data (typically transients)

b. Number/% of elderly references (more than 10 years old) c. Number/% of non peer-reviewed publications

d. Number/% of self-cites

Analysis

• Conclusions

• specific, semi-quantitative, significant, reflection of paper outputs

• Results

• critically appraised for precision, accuracy and scope

• Context/Discussion

• reference to the state of the art

Bibliometrics

Journal name and IF Water Research

Journal ranking in cohort IF 5.323

ENGINEERING, ENVIRONMENTAL 3/44 ENVIRONMENTAL SCIENCES 9/215

WATER RESOURCES 1/79

Number of citations (in SCOPUS) 70 in 13 years, 5.4/yr Number of recent publication cited (prev 10 y) 13 out of 24, 54%

% papers in peer-reviewed journals 15 out of 24, 63%

Journal IF of the cited publications Generally high Number recent papers cited, "Discussion" 6 out of 11 Number of correlations presented 43 (G), 10 (T)

Number/% plots of raw data 32 (74%)

Number/% of elderly references 8 (33%) Number/% non peer-reviewed publications 9 (38%)

Number/% of self-cites 3 (13%)

Analysis, summary

Conclusions

• Qualitative but corroborated by data Results

• Very broad scope, with theoretical predictions provided Context

• Recent papers cited to provide context/reassurance

Experimental details

• Permanent magnet, two configurations used

• Magnetic field strength of 0.16T

• Simulant hard waters, with & w/o salinity:

• 0.5 g/l CaCO₃ at pH7

• “Salted water” at pH8

• 0.01-1 m/s flow velocity

• Efficacy measured by Ca concn. reduction &

electrochemical/gravimetric method V WELL DETAILED

Data presented

Comparison of electrochemical & gravimetric methods (Fig. 3)

Comparison of treatment efficacy for the two waters (Fig. 4) Impact of flow velocity (0-3.6 m/s) on Ca level (Fig. 5)

Data for salted water, electrochemical & gravimetric methods:

• impact of flow velocity (Fig. 6)

• comparison of two configurations (Fig. 7)

• impact of residence time (Fig. 9), with accompanying Ca level data (Fig. 8)

• impact of material (Fig. 11), with accompanying Ca level data (Fig. 10)

tabulated summary data on nucleation times

Discussion

Extensive reference to literature

Empirical expression for calcium level change with k₀, B and v presented

Bibliometrics

Journal name, IF Int J Heat & Mass Transfer, IF 2.522

Journal ranking in cohort

ENGINEERING, MECHANICAL 13/126

MECHANICS 10/138

THERMODYNAMICS 8/55

Number of citations (in SCOPUS) 35 in 15 years, 2.3/y Number of recent publication cited 13 out of 20, 65%

% papers in peer-reviewed journals 10 out of 20, 50%

Journal IF of the cited publications Generally moderate Number recent papers cited, "Discussion" 3 out of 3

Number of correlations presented 10

Number/% plots of raw data 100% (all transients)

Number/% of elderly references 8 (40%)

Number/% non peer-reviewed publications 10 (50%)

Number/% of self-cites 6 (30%) + 1 by their student

Analysis, summary

Conclusions

• Quantitative, and corroborated by data Results

• Scope limited to deposit thickness (as measured by resistance)

Context

• Very little: discussion limited to appraisal of results

Experimental details

• Electronic conditioning device (EAF)

• CaCO₃ precipitation on heated surface, 302K

• Simulant hard water:

• Fixed concentrations of Ca²⁺ + HCO₃^-

• pH & conductivity

• 27-76 ml/min (0.28-0.78 m/s, Re=1870-5020)

• Uncertainty of 2.5-7.8% in measurements

• Definition of fouling resist. & heat transfer coeff., which define efficacy

V WELL DETAILED

Data presented

Trends in fouling resistance with time (5 off):

• with and without EAF

• different flow velocities/temperature differentials

• two different concentrations SEM:

• Change in crystal habit with EAF

Conclusions

Fouling rate a function of:

• flow rate

• concentration

• temperature

Key findings/claims:

• Results claimed to support solution-based crystallisation

• No mechanistic explanation of findings

Bibliometrics

Journal name, IF Desalination, IF 3.96

Journal ranking in cohort

WATER RESOURCES 9/80

ENGINEERING, CHEMICAL 9/133

Number of citations (in SCOPUS) 8 in 12 years, 0.7/y Number of recent publication cited 2 out of 3

% papers in peer-reviewed journals NR Journal IF of the cited publications NR

Number recent papers cited, "Discussion" No discussion Number of correlations presented 8 (T)

Number/% plots of raw data Impossible to say Number/% of elderly references 0

Number/% non peer-reviewed publications 2 (66%)

Number/% of self-cites 0

Analysis, summary

Conclusions

• Unsubstantiated/exaggerated Results

• No explanation as to the time at which the data in Tables 1-2 was taken

Context

• None

Experimental details

Membrane plant performance data:

• transmembrane pressure, rejection, power consumption Details missing:

• Definitions (SDI, SWC, GFD, psi..)

• EMF device design and operation (strength, AC or DC, positioning, flow..)

• Basis of cost benefit analysis

Paper has nothing to do with magnetic fields!

Summary

• In order of quality, the papers are:

• Gabrielli et al, 2001

• Cho and Choi, 1999

• Vedavyasan, 2001

• Corroborated by bibliometrics, but bibliometrics not 100% reliable:

• IF changes with time

• In the example given, Desalination has a much higher IF now than in 2001

• landmark papers cited regardless of quality

• It is still necessary to read the paper

And lastly ..

If you haven’t done so already ..

JOIN THE GROUP!!

For:

• hints

• response to queries, requests for clarification

• networking with your peers