Can You Trust MBA Rankings?

With some business schools liable to misinform—if not manipulate—rankings methodologies, it raises the question: How far can you trust the MBA rankings?

John Byrne, the editor of Poets & Quants who broke the Fox rankings scandal, suspects other business schools may now have cause for concern. “I think a lot of schools fudge the data and don’t get caught.

“While the Temple dean and people who fake the numbers are responsible, the truth is US News bears some responsibility as well.”

Verifying the data

Like every MBA ranking, US News has its own unique methodology, giving more or less weight to different metrics.

35% of a US News rankings position is dependent on a school’s placement success—including data on salaries, bonuses, and employment rates of MBA graduates sourced from the schools. Academic reputation accounts for 25%; quality of students 25%; recruiter reputation 15%.

Bob Morse (pictured right), the chief data strategist at US News who develops the methodologies and surveys for its major rankings, says measures are being put in place to prevent misreporting of data happening again.

Already, he says, US News analyzes the data schools provide, comparing it to the previous year to track any major changes—although that’s proved less than effective when a school consistently misreports its data year after year.

At QS Quacquarelli Symonds—best-known for its World University Rankings—MBA rankings data is compiled using three independently-verifiable surveys: a 40,000-strong employer survey which is given the most weight, an academic survey, and a survey of business schools

“The global employer survey is completely independent; no school can influence that,” claims QS CEO and founder Nunzio Quacquarelli (pictured below).

QS data from schools is verified with the help of the MBA Career Services Council. QS also analyzes alumni outcomes via its own independent lists of successful alums—people who run their own businesses or lead government organizations and nonprofits, as well as the directors of major firms.

“No one can be perfect; there’s ambiguity in data,” Nunzio continues, “but we put a huge effort and resource into the verification of the data we collect.”

A level of trust

While both US News and QS collect some of their data from business schools, neither organization surveys students or school alums. Alumni, Nunzio argues, have a vested interest in creating a bias response. “They have an interest in boosting their institution because it helps boost their own brand,” he says.

However, the Financial Times Global MBA rankings—one of the most respected in the world—does. The FT dedicates 59% of its ranking weight to its alumni survey—more than any other metric. In 2017, the FT surveyed over 22,00 alumni, receiving over 8,000 responses.

The FT surveys alumni three years after graduation and requires that a minimum 20% of alumni respond to the survey—with at least 20 completed responses—for schools to be included. Alumni are asked about their salaries, bonuses, career progression, level of seniority, and how highly they rate their alma mater.

Laurent Ortmans, responsible for the nine business education rankings the FT publishes each year, doesn’t buy the criticism:

“If people say they don’t survey alumni because the data is not valid, I don’t accept that at all. I think some rankings don’t survey alumni because they don’t have the data—that’s all. Schools trust us enough to share alumni contact details with us.

“A few alumni may exaggerate a bit, but we survey them directly; we have full control of process; we’re very confident that the data we collect is strong.”

Alongside its alumni survey, school data accounts for 31% of the FT’s Global MBA Ranking; research rank 10%. While some commentators have expressed doubts over the reliability of alumni-sourced data, the FT takes a step no other rankings organization does when it comes to validating the data it receives from schools: a comprehensive audit conducted by KPMG of 25 schools each year.

Still, Laurent admits that when it comes to collecting data for rankings, there is always a level of trust involved: “We spend a lot of time explaining what we measure and how we want it to be measured. We’re not trying to catch people out. We explain things and we hope schools follow our guidance.”

Ranking the rankings

So, to trust the MBA rankings you have to trust the alumni, business schools, and even employers who have the power to influence them. Furthermore, you have to trust in the rankings organizations and their unique methodologies—all created to stand out from one another and throw out different results.

With these issues in mind, John at Poets & Quants (pictured above) created his own composite MBA rankings, with five major business school rankings each weighted individually based on his view of their credibility. US News is weighted at 35%; Forbes 25%; FT and Bloomberg Businessweek 15%, and The Economist—a ranking entirely based on surveys of schools and students—at 10%.

Despite the Fox scandal, John says he won’t be re-thinking the weight he gives to US News, largely due to the amount and the quality of the data it collects.

US News data, John says, is the most up-to-date of any ranking—collected from schools in the fall of each year before the rankings are released in March of the next. He thinks the way US News gathers data has made US business schools become more transparent than some of their European or Asian counterparts.

“What I love about US News is that the outcomes are raw. You’re getting the real data; not data that’s put into a black box, manipulated, and then spit out for public consumption which is what the FT does,” John claims.

Conquer the rankings

US News; The Economist; the FT; The Wall Street Journal and Times Higher Education are teaming up to launch their own MBA rankings later this year. MBA rankings is probably the most crowded rankings field there is.

How far can MBA rankings be trusted? How far can we believe that MBA rankings are reliable and honest? Recent scandal has thrown question marks over this. The marketplace of public opinion puts a heavy price on misreporting data, but that’s not to say it won’t happen again.

At the same time, rankings, in terms of what they try to do—i.e. ranking business schools—are often flawed. But that’s not too say they aren’t useful.

For business school applicants, rankings should be a starting point; a source of data with which to compare schools of interest.

They should be studied, investigated, critiqued, and compared. Applicants should assess which rankings’ methodologies best match what they want to get out of business school. Applicants should look at rankings data over time; get a helicopter view.

No one ranking must be taken as fact. What makes a business school ‘good’ is always down to interpretation.