Longitudinal Education Outcomes (LEO) Data Analysis

 

How To Interpret LEO


BY ANDREW McGETTIGAN

One of the final acts of the Coalition government was to pass the Small Business, Enterprise and Employment Act. This law made it legal for the relevant Secretary of State to approve the use of HM Revenues and Customs data when it could be used to inform education policy and assess the performance of post-16 educational institutions.

Eighteen months after the 2015 Act was passed the first Longitudinal Education Outcomes data was released. A large amount of raw data was published on the Department for Education’s website with graduate earnings presented by “graduate characteristic” (sex, ethnicity, age, home region and prior attainment at A level) and, more radically, by subject choice and institution. “LEO”‘s first cohorts left university in 2003/04 and it is now possible to look at their earnings 1,3,5 and 10 years after graduation. The figures for subsequent cohorts covered such years as they were able, with 2008/09 graduates the most recent with 5-year figures.

View the raw data from the government

In June, the latter group came into focus as LEO data was updated to show earnings breakdowns for course and institution combined. That is, it became possible to look at the earnings of Social Studies graduates from the University of Sunderland compared to those who had studied English at Durham, but only up to five years out of university.

In all cases, what is available is the bare bones of a “box plot”: we are given the lower and upper quartiles plus the median. That is, if we rank all graduates in the selection we are told what the  person at the mid-point earns (the median), and the boundaries above and below which a quarter of individuals fall (upper and lower quartiles respectively). The figure below, taken from the December LEO release shows the annual earnings of all 2003/04 graduates by subject, ten years after graduation. Note that this shows graduates who were in some sustained employment – those out of the labour market are missing from the earnings data.


Earnings of 2003/04 graduates by subject, ten years after leaving university
Longitudinal Education Outcomes, 2016


You might think there are few surprises in such a chart – and you’d be right. At this level, the raw data reveals little, although it is worth bearing in mind that median annual earnings for non-graduates is in the region of £23,000, leaving some subjects looking like a poor choice as careers.

Indeed, most of the newspaper reaction to LEO has tended to focus on the specialist art and design institutions, where median graduates place below the median non-graduate. In putting the blame on universities, this perhaps ignores the casualised gig economy that marks out the creative professions. But it does offer a sobering insight into what is a very popular choice: the Creative Arts category covers Music, Drama, Fine Art, Design and is second only to Business in terms of graduate numbers.

Before we move on to talk about the more informative interpretations of the data, it’s worth flagging up some key points and caveats.

The government is still ambivalent about this data. As we’ll see, it tells a story about graduate earnings that is at odds with official defences of the current fee-loan funding system. But at the same time, the government would like this data to inform applicant choices insofar as the data can support the idea that the earnings of graduates tells us something about the quality of teaching at their alma mater (and how likely a graduate is to repay their loans!).

LEO promises to do two things. First, it is likely to supplant more traditional league tables, such as those offered by the Guardian and the Times Good University Guide, which rely on ‘inputs’ — how selective a university is and how much resources go into teaching — and an unreliable measure of ‘value add’: how many good degrees are awarded. And indeed, the Economist magazine has had a first attempt at utilizing LEO to develop a model which measures value add on the basis of earnings: a good university boosts actual graduate earnings by more than would be expected from certain key indicators about its intake. [1]

> View the British university rankings methodology – The Economist

Second, it should be less and less possible for politicians to appeal to overall average figures for the “graduate premium”. Most recently, Jo Johnson, minister responsible for universities, said, “Over a graduate’s lifetime, a degree is on average worth an additional £170,000 in salary for a man, and £250,000 for a woman.”

> View Jo Johnson’s speech

LEO shows that those averages are fundamentally misleading. But at the same time LEO should support the government’s desire for applicants to have more information about universities before making their choices. The danger is that applicants, particularly young applicants, are likely to be overwhelmed by this data dump and be misled by simplistic readings of what’s available. Tellingly, the government’s LEO consultation response talks of the importance of Careers Advisors in mediating LEO.

Caveats

What follows summarises some excellent “explainers” available from the specialist HE policy website, Wonkhe. Wonkhe has done a very good job in filling in the gaps around the data and its ‘beginners guide’

> View Wonkhe’s beginner’s guide

and ‘caveats guide’ are good ports to call on after reading this piece.

> View Wonkhe’s ‘caveats guide’

  1. The data is not comprehensive. Sizeable numbers of graduates are missing. Graduates may have left the country (so have no pay being recorded by HMRC); they may be self-employed (a huge gap, particularly for creative arts subjects, which the government is currently trying to rectify); they may have left the labour force for various reasons (including pregnancy and caring responsibilities).
  2. There is no allowance for part-time working. Individual earnings is simply total earnings.
  3. There are severe time lags. What would appear to be the most informative data — ten years out from university — represents individuals who would have applied to university circa 2000! Applicants today are effectively looking at subjects and institutions two decades ago. On the other hand, the five-year data is arguably too soon to get a real sense of divergent salary outcomes that are more likely to unfold as graduates hit their mid-thirties and early forties. That 2008/09 cohort also started their working careers just after the financial crisis and so graduated into a severe recession.
  4. Prior attainment data is only based on A levels. Those without A levels are classed as ‘Not Matched’ – a reference to the National Pupil Database.
  5. We do not know which students if any went into further study, and therefore might have received an earnings boost from a postgraduate qualification.
  6. Similar, but independent work by the Institute for Fiscal Studies showed that graduates from wealthier backgrounds had large earnings advantages: “The average gap in earnings between students from higher- and lower-income backgrounds is £8,000 a year for men and £5,300 a year for women, ten years after graduation.” The June LEO release added POLAR data, but this dataset has been subject to criticism and doesn’t capture parental household wealth.[2]

In sum, this is historical data which has large gaps. Some of those gaps may be addressed in future iterations and, intriguingly, such lagged data may combat misplaced prestige, which is also based on such lags.

Key Findings and Considerations

Subject choice matters but within those overall figures, there is a large variation by institution. A piece I wrote for Wonkhe goes into detail about the problems facing the creative arts.

> View ‘Lessons From Leo – it’s time to think differently about subject funding.

It is clear that specialist drama and music conservatoires have their earnings data dragged down by the failure to include earnings from self-employment. But regional specialist institutions and some former polytechnics do very badly here.

Arts graduates (2008/09), Earnings five years after graduation


More surprisingly for a professionally oriented degree, Law graduates show similar problems with a long tail.

Law graduates (2008/09), Earnings five years after graduation


Whilst the median Oxbridge law graduate is earning over £55,000 pa, there are twelve universities whose graduates have median earnings of £20,000 or less and a similar number where a quarter are earning under £15,000 per year. The 2014/15 median salary for all those in their mid to late twenties (grads and non-grads) was £20,800.

The law data also indicates that applicants should be wary of the ‘Russell Group’ badge: plenty of university graduates are performing better than those who studied Law at Liverpool and Leeds.

LEO raises lots of questions, but doesn’t offer explanations. We have to conclude that for some subjects, the outcomes are uneven and inequitable. Despite the headline fee of £9250pa being charged at all universities, outcomes differ. Should one conclude that Oxford does better because it has a better intake and more close contact with students through tutorials or that is has better captured a particular segment of the jobs market? Certainly employers are inclined to view entry into a selective university as a sign of talent. The “tail” may also be a sign that there is an oversupply of law graduates, meaning that some do not get the opportunity to pursue the career they planned. Whatever theories turn out to be correct, what this data indicates is that it may be a mistake to choose what looks like a professional degree for the sake of it. Applicants will need to do more digging, ask more questions about courses at particular institutions and be clear-eyed about where opportunities really are.

There is a large pay gap between men and women which may be partly explained by the higher percentage of women who go to work in the voluntary and public sectors, where wages are lower than in the private sector. Half a million women work in the voluntary sector (68%) of the workforce, and they also make up 64% of the public sector.[3]

2016 Longitudinal Education Outcomes


Education Outcomes

Some subjects seem to be better for women than others, such as Economics. While a choice like Languages serves women better than men on average. It should be emphasised that women do benefit more from having a degree than men, when compared to non-graduates.

It is also possible to examine courses at particular institutions for differing outcomes between men and women, though again it is hard to interpret the results. Wonkhe has uncovered the strange case of male nurses from the University of Hertfordshire who have some very high earners amongst them.

2016 Longitudinal Education Outcomes


Outcomes

It is not however possible to look at outcomes by ethnicity for particular courses at particular institutions. In fact it is only possible to look at the overall earnings data by this characteristic. The results show that British graduates with Black Caribbean, Bangladeshi or Pakistani heritage suffer weaker outcomes, while those from Chinese, Indian or other Asian backgrounds outperform the average (and White British results).

2016 Longitudinal Education Outcomes


Longitudinal Education Outcomes

Wonkhe has further broken down these figures by sex to produce a table showing differences between median earnings against White men and between men and women of different ethnicities.

> View the table on Wonkhe’s website

This reveals that women from Pakistani and Bangladeshi backgrounds are even more disadvantaged, with median annual earnings of those working at £5000-£6500 below white male counterparts.

Wonkhe, ” Prior attainment, gender gaps and other lessons from LEO”, December 2016
http://wonkhe.com/blogs/prior-attainment-gender-gaps-and-other-lessons-from-leo/


Unfortunately, this data, such as it is, can only illustrate a systemic problem needing large-scale solutions; it can’t be turned into practical careers advice, except to remind individuals to be sceptical. The averages conceal the stories of individuals and groups! Note that the average earnings are for those in work, those backgrounds with lower average earnings are also less likely to be in ‘sustained employment’.

2016 Longitudinal Education Outcomes


Prior Attainment also matters. As already noted, LEO categorises graduates by the combined UCAS tariff of their best three A level results. [4] The five categories are: 360+, 300-359, 240-299, below 240 and “not matched”. 64.0 per cent of graduates in the ‘not matched’ group were ‘mature’ students.

For students with lower prior attainment (or without A levels), Computer Science and Architecture, Building and Planning turn out to perform strongly. It is worth noting that the majority of these graduates are in the “below 240” and “not matched” categories, whereas these are not at all common in Mathematical Sciences and Economics

2016 Longitudinal Education Outcomes


Note also, that the students with the best A level results see a host of high-paying opportunities open up if they study Law, Business, Computer Science, Mathematics or Economics.

Regional disparities. There are two geographical aspects to LEO’s data. Which region students come from and the regions in which their universities are located. The data suggests that it might be wise to avoid going away from home to study in a region, town or city with a weak local economy. This applies in particular to those considering going away from home to study Creative Arts and Design full-time.

But London also demonstrates a split personality — those coming from the South-East earn more if they end up in work, but they are also much more likely to be missing from the earnings data:

“There was limited variation in the proportion of graduates with a positive study or employment outcome by home region, with almost all regions differing by less than 3 percentage points from the national average (see figure 5). The only exception to this was London, in which only 72.7 per cent of students were in further study, sustained employment or both, 5.1 percentage points below the national level. This was driven in equal parts by a higher than average rate of students in the activity not captured and in the no sustained destination categories.”

2016 Longitudinal Education Outcomes, page 12.


2016 Longitudinal Education Outcomes

A final point. Once the earnings data has been adjusted for inflation, it reveals that real pay for young graduates has dropped in the decade since the financial crisis.

> View Wonkhe blog on how LEO shows graduate pay has been squeezed 

Subjects that lead to public sector careers such as Education have had their pay squeezed the most. Unfortunately, this also applies to non-graduates so the relative advantage of having a degree in terms of earnings has stayed the same.

In conclusion, LEO has been produced by government in order to provide more information for applicants, particularly about the previous graduates outcomes for the courses to which they intend to apply. In many ways, it does do that, but it also has the potential to be very misleading. For instance, Birkbeck and Open University are specialist part-time institutions, many of their graduates were already advanced in their careers before beginning study and worked throughout. Their earnings are therefore higher than similar institutions primarily catering for full-time school-leavers.

It is hard to distil the first releases into practical careers advice. Degrees vary and lead to very different opportunities, not all of those opportunities and experiences can be captured by earnings data. The data is no substitute for visiting and asking questions at Open Days – perhaps the best way of utilising LEO is help frame sensible questions: “what do you do to help students and graduates find jobs?”; “why are so many of your recent graduates earning such low sums?”; “what jobs and careers do your graduates enter?”.

As already said, it is probably ill-advised to choose Law or Business, simply because they are professional oriented – without looking at how the outcomes vary by institution. Similarly, doing a creative degree because it’s a degree, or choosing a degree subject because its familiar from school or college will also likely be misguided. LEO at least indirectly tells you what courses well-paying employers value and also indicates that courses associated with public sector jobs (like Education with school teaching) have seen earnings eroded in the last twenty years. But above all, it tells you that applicants need to think very hard about the choice to be made. Perhaps the most responsible advice would be to relieve the pressure on seventeen year olds.

University degrees are not going to run away; future changes in government may see the cost reduced; and for many people, Birkbeck and the Open University or deferring the choice entirely may actually be better. If nothing else, LEO ought to work against the cultural inertia that sees young adults choose a degree, any degree, because it pleases their parents or because it’s the social expectation.




Notes

[1] “To estimate which universities make the biggest positive impact on their students, we have built a statistical model that estimates how much graduates from each university can expect to earn. It is based on how selective a university is, what subjects its students study, the share of students from lower-income areas, how many students enter after age 20, what share of students attended private schools and where the university is located. We then compare these estimates of expected earnings with the actual figures. This difference, we argue, measures the value added by the institution.”

[2] “The Participation of Local Areas (POLAR) classification places local areas into five groups, based on the proportion of 18 year olds who enter higher education at age 18 or 19. Quintile 1 contains the most disadvantaged areas.” LEO June 2017 dataset footnotes.

[3] https://data.ncvo.org.uk/a/almanac12/who-works-in-the-voluntary-sector/

[4] The old, pre-2017 UCAS tariff is used where an A* is 140pts, an A 120, a B 100, a C 80, a D 60 and an E 40.