Predicting the “strugglers”: a case-control study of students at Nottingham University Medical School

Identification of strugglers and controls

We studied five consecutive year groups of medical students who should have completed the whole course at the time of data collection. The dates are not specified precisely to protect the confidentiality of the individuals in the study. We identified struggling students in one of four ways: students who attended the academic progress committee (APC), which meets several times each year and interviews students experiencing major academic difficulties and agrees various strategies of support; students who had their course terminated, usually for academic reasons; students who left the course voluntarily, whether for personal or academic reasons; and students whose course was suspended temporarily, for personal or academic reasons. These categories were not necessarily mutually exclusive—for example, a student who attended the academic progress committee might subsequently leave the course.

For each struggler, we identified four control students from the same year group cohort, but who did not fulfil the above criteria, by a computer generated randomisation programme that used each student's unique course identification code. We chose the 1:4 case:control ratio after statistical advice to minimise bias through chance yet avoid excessive data entry.

Data collection

For all students (strugglers and controls) we extracted data manually from two sources: the UCAS forms submitted by the students at the time of selection and the students' progress records once they were on the medical course. We supplemented these data by additional information provided directly by UCAS. Table 1 gives the details.

GCSE (general certificate of secondary education) and A level passes were scored (at GCSE, A* = 6, A = 5 etc, and at A level, A = 10, B = 8, etc) to create mean examination grades (MEGs). We used mean grade cut-off thresholds of < 5 for GCSE (5 is the equivalent of an average of grade A for all GCSE subjects, the current GCSE screening threshold in our selection process) and < 9.33 for A level (9.33 is the equivalent of AAB grades; the minimum required A level grades during the period of the study were ABB but most students exceeded this level).

“Negative comments” were remarks made in the academic references on the UCAS form and generally referred to possibly adverse aspects of personality or behaviour. They were extracted verbatim and then summarised as a yes/no variable (details are available from the corresponding author on request). In general the negative comments fell into one of five categories: academic (such as lack of ability, motivation, or self discipline); personality (such as excessive anxiety, lack of self confidence, or reluctance to participate); attitude (such as intolerance of others, arrogance, overconfidence); ongoing medical or social problems (such as chronic debilitating illness, severe family problems, or poor adaptation to cultural change); and late decision to read medicine (lack of appropriate work experience). We validated our assessment of what constituted a negative comment by asking 23 (50%) of the members of the current interviewing panel, who were not aware of the details of the study, to assess 25 comments as potentially positive, negative, or neither, by means of a one page postal questionnaire. We included eight comments that we thought were “positive” among 17 negative ones. We assessed inter-rater reliability of the identification of negative comments by asking an independent researcher to review 62 statements (a 10% random sample) against our criteria of negative comments.

Statistical analysis

We used SPSS v11 for data analysis. Inter-rater reliability of the identification of negative comments was tested with the κ statistic. Our primary outcome variable was being a struggler. For univariate analysis we compared dichotomous variables with χ² tests and continuous variables by Mann-Whitney tests. For multivariate analysis we tested for independent predictors by binary logistic regression, using the backwards logistic regression option and including significant univariate factors in the modelling. We included year of entry to the course as a categorical variable to adjust for change over time.

General

Overall, 123 students from the five cohorts were identified as strugglers and there were 492 controls. Fifty three strugglers were initially identified as APC attendees, nine as course terminations, 30 as voluntary withdrawals, and 31 as course suspensions. Thus of the 961 students admitted over the five years, 123 (12.8%) experienced problems, including 41 (4%) who left the course before graduation, as described below.

Extent of strugglers' problems

The strugglers experienced considerable academic problems. Seventy of the 123 (57%) failed three or more preclinical exams and only 92 (75%) had graduated as BMedSci (bachelor of medical science) at the end of the third year; of the 31 remaining, 21 left the course voluntarily, eight had their course terminated, and two were still on the preclinical course. At a date when all students should have completed their training, a further 12 had left during the clinical course, eight were still on the clinical course, and only 72 (59%) of the original group of strugglers had graduated BMBS (bachelor of medicine and bachelor of surgery).

In addition, 56 of the 123 (46%) had disruptive personal or medical problems noted in their undergraduate records, including a high incidence of depressive illness (strugglers 29/123 (24%) v controls 6/492 (1%); χ² 91.6, P < 0.001). Overall, 34 (28%) of the strugglers had attitude problems noted in their undergraduate records.

Preadmission factors associated with strugglers (univariate analyses).

Table 2 and Table 3 summarise these results. The non-academic factors that were significantly more common in the strugglers were the presence of negative comments in the head teacher's reference, the late offer of a place, male sex, and slightly older age. Non-white ethnicity was significant in UK students. There was no significant difference between UK strugglers and controls in social deprivation (Townsend score6).

The significant academic associations with the strugglers were lower mean examination grade at GCSE, lower mean examination grade at A level, and not achieving a grade A at A level biology. Such students were also more likely to have significantly lower UCAS points (point score for top three A levels) and total tariff scores (point score for all A levels taken). There were no significant differences between the groups in terms of A level chemistry grade or having undertaken a previous degree.

Preadmission factors significantly and independently predicting strugglers (multivariate analysis)

We entered preadmission factors that had shown significant univariate associations with struggler status (above) into a binary logistic regression. Table 4 shows the data for all students excluding ethnicity as an explanatory variable because it was not known for overseas students, and data for UK students including ethnicity as that was recorded for most of these.

A lower mean examination grade at A level and the late offer of a place were highly significant predictors in all students (likelihood ratio 2.19, 95% confidence interval 1.37 to 3.51, and 1.98, 1.19 to 3.30, respectively) and in UK students alone (2.36, 1.40 to3.99, and 2.25, 1.27 to 3.99). Negative comments were strongly predictive in the whole group (2.25, 1.44 to 3.50) but not in UK students alone. Men and those with lower GCSE science scores were more likely to struggle in the entire group (1.70, 1.09 to2.65, and 2.13, 1.12 to 4.05) but again not in UK students alone. Non-white ethnicity was a highly significant predictor in UK students (2.77, 1.52 to 5.05).

Validation of negative comments

Eighteen members of the interviewers' panel returned completed questionnaires (78% response rate). For the 17 comments that we (JY and DJ) thought were potentially negative, overall the interviewers agreed they were more likely to be negative. For the eight comments that we (JY and DJ) considered positive, again, the interviewers agreed they were more likely to be positive. The results of the negative comment survey are available from the corresponding author.

The review of 62 statements by an independent researcher revealed satisfactory inter-rater agreement with the identification of negative comments (κ = 0.69).

Strengths and limitations

We collected a wide range of robust quantitative data, with a 1:4 case:control ratio and random selection of controls to avoid matching on discriminatory factors. We tested the implicit assumption that our controls were representative of the year groups by comparing them with non-selected students—that is, those who were neither strugglers nor controls—for five pre-admission factors (age, sex, where they normally live, A level points score for the top three grades, and having a previous degree). There were no significant differences. We were unable to test the impact of ethnicity for the whole group as data were unavailable for overseas students.

In comparing strugglers with controls we chose to analyse our strugglers as a single group because there was no definitive way to split them up. Their problems comprised academic, personal, medical, or social components in varying degrees. Hence, we could define a struggler only as a student who failed to make satisfactory progress and was identified in any one of the categories listed. We accept that there may be subtle associations between the categories and the predictors that we were unable to identify, but a subgroup analysis of those strugglers who attended the academic progress committee and those whose course was terminated showed that the same range of factors emerged as predictors of struggling (see bmj.com for results).

The inclusion of “negative comments” as a defining variable can be criticised on the grounds of subjectivity, especially as there was a wide range of comments and the researcher (JY) was not blind to control/struggler status. The validation that was carried out in the form of a questionnaire distributed to interviewers and the independent review of statements provided some justification for the scoring. However, in an unstructured reference in which teachers are usually trying to accentuate their pupils' positive characteristics, we accept that the scope for finding negative ones may be limited. Furthermore, the scoring is still subjective. Additional studies using more formal statement analysis techniques, which we have used in previous research, are under way.7

Although we could not fully assess the independent predictive value of ethnicity in the whole group, we would expect non-white status to be comparably predictive in non-UK students as they are predominantly of non-European origin and also have a higher proportion of strugglers (22/86 (26%) v 101/529 (19%) of home students).

Comparison with other studies

We chose a different design and outcome to many other studies, which have used full cohorts and measured rates of examination success. However, our work confirms that risk factors for failure—lower school examination grades, non-white ethnicity, and being male—were as expected, being the opposite of those factors that have been reported to predict success.1 8 9 10 11

We are not aware of any other studies that have used “negative comments” in the academic references. Powis et al found that interviewers' negative remarks had some predictive value for course withdrawal in a study using 1:1 partial case-control matching, but the association was relatively weak.12 Papadakis et al suggested some correlation between negative statements made during the undergraduate years regarding professional conduct and later disciplinary action in graduates.13 A previous study at Nottingham showed no relation between the number of positive themes in the academic reference and course success, but the reverse was not examined.7 We found many negative comments that conveyed the opposite of those factors that are linked to success,7 14 15 although they tended to be less clear cut. Statement review is an integral part of the selection process at Nottingham, and our results confirm that interviewers should be alert for negative comments, not in a discriminatory sense but to explore them more fully with the student.

Similarly, we are not aware that others have examined the timing of course offers in relation to undergraduate progress, but our study shows that those who fail to impress the interviewers initially or apply at a late stage, perhaps after rejection by other schools or through “clearing,” are at greater risk of struggling than those who received an early offer of a place.

Future policy and research

Many medical schools in the UK are exploring more varied admissions policies, perhaps incorporating elements of the successful Australian policies of lower examination grades accompanied by psychometric testing.3 Their outcome evaluations, especially in relation to non-traditional students, may be important in guiding future policy across the UK.2 16 17 Our data suggest that the current four stage approach to student selection is sound, but we now have concerns that the introduction by UCAS of open references will reduce the opportunities for head teachers to draw attention to personal qualities or difficulties that might make it difficult for a student to succeed in medicine; structured references might be more helpful.

Pastoral support at Nottingham includes informal meetings with personal tutors and more intensive formal mechanisms, yet some students still hide, or deny, their difficulties until they reach a crisis point. In the course of this research we noticed a high incidence of depressive illnesses in strugglers, which is of particular concern. We intend to review our strugglers and our pastoral practices more closely to see what further support could be offered, perhaps as targeted interventions to those at greatest risk. Research elsewhere has identified personal, social, cultural, and financial pressures that may particularly affect students from non-mainstream backgrounds and that may need to be addressed explicitly and proactively.18 19 Failure in clinical examinations may have a sex related or cultural basis because the current emphasis on patient centred, empathetic care may be more natural for women than for men20 and may present a considerable difficulty for students from more paternalistic cultures.21 22 23 24 Language barriers may be important because fluency in standard English may not be adequate for medical and colloquial needs.25 26 27

In conclusion, this study supports our existing selection practices at Nottingham relating to previous academic achievement and the critical review of students with negative references. Those who were initially rejected or who were “reserves” may require additional support during their course. We plan further investigations into the nature of the negative comments and the characteristics and difficulties of those who do less well or fail on the course.