On the ratings of universities: we separate the grains from the chaff

Already more than once or twice on Habré I had to come across in passing abandoned phrases in the style of "this university is also in the first hundred of the world ranking" or "this university is bad, it hangs in the ratings somewhere in the fourth hundred." It turns out that all the features of a particular institution, all its advantages and disadvantages are somehow magically displayed in a single numerical indicator, which serves as a clear confirmation of this or that thesis. In reality, the situation is much more complicated and interesting. I would like to share my subjective look at it today. I apologize for the large amount of text, but less does not work.

Preliminary notes

To begin with, I will express the obvious idea that it is quite difficult to give an exact name to any integral indicator. Obviously, the rating in some way reflects “quality”, but it is hard to say what exactly is reflected. It is unlikely that an advanced car owner would come up with the choice of a car according to “rating” (assuming a reasonable price): rather, some indicators like power, reliability or spaciousness, plus a subjective feeling of comfort from a particular car, will be used. Perhaps a closer example is wine or cheese. The only objective indicator of the “quality” of a product is the price per unit, however, in practice, the choice will be determined more by relevance and personal taste, rather than by price (if we are not talking about some super-luxury products, of course).

Thus, the rating can serve as a quick indicator of "goodness", but to understand the situation, you should dig deeper. After all, many of us read weekly feature articles and forums to choose a graphics card or TV. But the much more important choice of an educational institution sometimes happens on a hunch or based on bare numerical ratings. I will try to show that each edge of the rating table has its own troubles. Relatively speaking, one does not have enough money for microscopes, while others will buy microscopes only if it will benefit the rating. Both the happiness and misfortunes of universities are so diverse that to reduce them to one figure is a fairly naive occupation.

Just in case, I add that I fully welcome the compilation of any ratings. Rating is a measuring device with which you can improve the atmosphere and productivity of a university, but you must clearly understand the purpose of the device and its indirect nature. The device can be deceived, but there is no need to deceive yourself. For example, if I want to become a healthier person, I will try to lose weight and go in for sports. At the same time, simple scales can help me track success. However, if I just set myself the goal of moving the weight arrow a few divisions to the left, nothing good will come of it: the body can be depleted by some kind of fierce diet, and health will not increase.

I do not plan to engage in full-scale studies of various ways of ranking universities, but at least briefly to list the most popular indicators found in various methods, it is necessary. So, what factors most often affect the final position in the ratings?

- The number of publications in the most prestigious scientific journals (ARWU, SCImago ratings)
- Various citation indexes calculated (ARWU, THE, QS, SCImago
ratings
)
- Expert estimates, that is, data from various kinds of profiles (THE, QS ratings)
- Number of graduates , the proportion of graduate students among students, the proportion of "employee / student" and similar indicators (ratings of THE, QS)
- Proportion of foreigners among students and staff, joint publications with foreigners (THE, QS, SCImago ratings)
There are also more exotic indicators, such as profitability ratings (THE rating) or the number of Nobel laureates and field medal winners (ARWU rating).

Preliminary findings

Before delving into the details, I will share my final opinion. Highly rated universities, as a rule, are really good as research centers, attract mostly sensible students and have developed ties with commercial companies, the state and foreign laboratories. Moreover, subjective “goodness” from the point of view of a student and employee (including a serious scientist) can differ greatly from the indicators obtained. A very low rating, most likely, really indicates major problems with financing or management policies. At the same time, even in low-ranking universities, there may be first-class "islands" for work, but in general the situation is sad. The “middle peasants” can turn out to be both “puffy” low-ranking institutions, and wonderful places that “sagged” in terms of indicators, которые администрация не желает или не может подтянуть в силу разных причин.

And the conclusion, again, is the same: you can read the ratings, but choosing a place to work or study based on them is like watching movies in descending order of stars on IMDB. Absolutely hack-work is eliminated, but it’s far from the fact that you will watch without stopping. Alas, you can’t do with little blood. Read forums, chat with people, ask. In the end, the film lasts only a couple of hours, and it will take many years to study or work.

And now - scattered fragments of the mosaic.

Undergraduate

The correlation of the rating and quality of education of bachelors is most bizarre. In theory, it should look like this: a world-class scientist works in a good university, who will communicate daily with young fools, gradually turning them into world-class specialists. And all this against the backdrop of technology parks, biorobots, spaceships and other surroundings, which are famous for the best universities in the world.

In practice, everything is sadder. Oboltus people need to tell matan, linear algebra and the basics of algorithms, explain how the process differs from a thread (which thread) and generally deal with any kind of mumbo jumbo, not always related to your own research. I would say that it’s almost never related, because the content of basic subjects has already settled down, and few people are working on improving quicksort or Dijkstra’s algorithm. In addition, everywhere there is such a sad phenomenon as the "educational program", in accordance with which the university must provide some set of basic subjects. Moreover, not all really working specialists with these subjects want to deal with (and many have already forgotten them). So books like Teaching What You Don't Know - This is not at all from the department of humor and not from the practice of the “collective farm and design” universities. And, frankly, the professor has very little motivation to work on improving his own teaching skills. Who cares (by personality), works, who doesn’t - scores.

It’s probably cool to hear about Christofides’s algorithm from the mouth of Christofides himself. I also believe that the very presence of people of this level somehow affects the student’s subconscious, but we’ll be fair: for basic courses, the skill of the teacher as a teacher is much more important, but it is not reflected in the ratings in any way. Well, nothing at all, except for student polls. In addition, unfortunately, abstract Christofides is not so interested in communicating with first-year students as he is with him. Because bachelors are basically a gray mass. Yes, among them there are sensible and very sensible, but what to take from them? They study from morning till night, while they still don’t know anything, and after distribution to the laboratories there’s just nothing left (let’s say, two years) to bring them up to date, get some kind of diploma paper and say goodbye forever . Как бы печально это ни звучало, студент-бакалавр для профессора — это не очень перспективное вложение времени.

At least, this is how I see it both from my own experience and from communication with colleagues. The task of the bachelor is to learn basic subjects, compose a diploma and safely leave the university. Those who are more sophisticated are also not very eager to get involved in a large laboratory project. He is a young man, they are interested in trying himself in different roles. Well, right, let them try; Yes, but I have no special time to mess with them (not a day of rubber), and usually they will not entrust serious work to them.

However, of course, in a high-ranking university, in principle, the situation should be quite healthy. Students are stronger, teachers are stronger, random people do not linger. But overall, this is an indirect factor; and in any case, we can talk about the first hundreds of rating institutions. I would say that several flagship universities of any good developed country should provide a healthy atmosphere. Regardless of the rating points.

Bike from life . One of the largest Russian universities, a brand and all that. The dean of the faculty is a great scientist, without fools. Students know about it and vying to enroll in his diplomas. Dozens annually, because it’s fashionable, because the name of the big person will be in the diploma. In fact: the dean is constantly on the move or simply on business, talks with his wards once a month (if not less often) on a schedule, people protect diplomas on some side exhaust from his work, after a year 99% of graduates are placed in ordinary offices of various profiles , where even the name of the scientific field of the dean will not tell people anything.

Master

The master's program is a paradoxical reversal of undergraduate studies, in which, again, many important formal criteria at the university level are not very significant for a particular student. Russian "bologization" has not yet been brought to mind, and the "bachelor" by most people is perceived as a "nonspecialist" who did not have the mind to finish his master's program. Within the framework of those educational systems where the division at the bachelor / master level has existed for a long time, the perception is completely different. Bachelor is an absolutely full-fledged graduate, ready to work in the industry. A master is a person who has spent an additional two years on improving research skills . This is not just a declaration, it is a different learning style and a different style of perceiving the result. The master's program is not primarily a mechanical addition of new courses to those already attended, but work on a research project in a university laboratory. Take as an example International IT Master's program from the University of Eastern Finland: of the 120 “credit units” of the master's minimum, 50 units comprise research in its purest form (a diploma project as a scientific work plus an “IT Project”, that is, software corresponding to the project), and another 7-10 units can be equated to studies conditionally (they are given for compiling a personal curriculum, the course "University study skills", scientific English and "master's seminar"). In addition, such forms are practiced as obtaining loans for a certain clearly defined work (analogue of a course student) and “book exams”, that is, passing an exam based on the results of independent study of a particular book.

Thus, it is not the university as a whole that is important for the student, but the specific laboratory in which he will work, and the personality of a particular leader. Of course, in general, there are always more good laboratories in a good university, but particulars must be carefully studied. It's like talking about whether New York City is good or bad. General reasoning is good from the outside, and from the inside a resident of Harlem sees a completely different picture than a resident of the West Village. Even in very good universities, a variety of sad stories happen when supervisors squeeze out all the juices from their subordinates for the success of the project and prevent them from defending for several years (because "the work is still damp"). After all, undergraduates and graduate students are the main labor force of laboratories, and behind each successful project (which make the university a name! ) A train of hundreds of sleepless nights of students and staff is reaching. Each laboratory is a personal estate of merit, and it is necessary to treat distribution in this or that place as to the device for work. Projects are important, the working atmosphere is important, relationships between people are important, the personality of the boss is important. Yes, it’s very difficult to know all the details in advance, but you won’t do anything.

Bike from life . I knew one professor who had only newcomers in his wards, preferably from poor countries. Local students did not rush at him at all (word of mouth worked), and the new arrivals turned out to be easy prey. In addition, you could always put pressure on them: here you will work poorly, I will drive you out of the department, your visa will not be extended, and roll back to your beautiful country.

Teacher or scientist?

Honestly, I am not too competent in this matter, so I will try to refrain from evaluations. I just want to say that there are different points of view on the mission of the university, and the specific embodiment of a particular point of view can greatly affect both the position in the ranking and the internal culture of the organization.

In the Soviet Union, science and education were separated from each other. It was believed that research institutes should be engaged in science, and the university’s task was to educate students. At the same time, the same people could both teach and work at the research institute, and the research institutes themselves maintained contact with both universities and industry.

Now we are trying to introduce a “Western model”, which consists in transferring research activities to universities. The idea seems to be viable, since it works in this very West (although they also have National Labs, corporate research centers and other non-university offices), but in general it is not too obvious. Proponents of the old system believe that you can’t do any special science with university students, because this requires serious guys from research institutes who work there for a long time and are able to take on long-term projects. And the connection between the "leading edge" of science and the learning process can be made by those same part-time workers from research institutes that conduct one or two classes a week at the university.

Reformers can say that the industry has already collapsed anyway, and, therefore, the three-legged table "Research Institute - University - Plant" still no longer has one support. In fact, universities are brewed in their own juice, and research institutes in their own, and everywhere everything is gradually rotting. There is a sweet dream that universities will earn on targeted research ordered by industry, but so far we do not see any special results.

In this case, I do not want to take sides, both of them are right in their own way. There is an objective separation between science and teaching, and how to combine these things is not always clear. A scientist can not understand anything in pedagogy and teach how God will put a soul on. He has no corresponding education either. Moreover, he can sincerely hate teaching and conduct classes anyhow, while serving labor duty. The teacher can be a brilliant storyteller, however, far from scientific work.

In my view, undergraduate students need more specialized teachers, and in graduate and postgraduate studies, scientists, but I will not insist. There are attempts to cross each other in the position of “teaching professor”, but such a practice cannot be called generally accepted. Specifically in at my university, teaching is practically not appreciated. Well, that is, the professor has little motivation, in addition to purely internal, to improve and polish his own courses. The prestige of the university, the receipt of grants and the career success of a scientist depend on scientific results, and not on successes in the teaching field. I recently heard a similar assessment from a professor at a prestigious British university, so the situation is typical.

University ratings are also “tailored” to the Western model of universities as research centers, and our teachers are not ready for such a turn of events. It should be understood that in scientific activity there is really 5% of “science” and 95% of “activity”, that is, purely technical skills related to setting up the general scheme of work for writing articles, participating in conferences, establishing contacts and fighting for grants. People may not believe in this scheme or may not be able to work in it (or simply not be in time, because the load of teaching and any bureaucracy is exorbitant). However, I think that on the whole we will move in this direction, because even if we assume the great efficiency of the Soviet system, it is hardly possible to maintain it in the modern world: you either build your own local paradise, or merge into a common system, albeit not the most reasonable. Всё равно участие в глобальном процессе принесёт больше плодов, чем гордое эффективное одиночество.

English factor

We often like to complain that universities in the former countries of the British Empire have an insurmountable linguistic advantage: they all quote, they all read, but we can only count on quotations from friendly Ukrainians, Belarusians and someone else in a similar class. However, it is enough to look at the first hundred of the QS rating to recall the proverb about a bad dancer, who also has inborn features that prevent him from reaching the heights of mastery. Therefore, the Germans, Japanese, Koreans, French, Chinese, Danes, Finns, Swedes and others cope with the language barrier, but we can’t.

But one cannot deny the obvious either. The English language gives a lot of non-obvious bonuses to the rating, to which there is no concern for a simple normal student or employee, but which clearly contribute to his - rating - growth.
If you are writing an article about the intricacies of the syllable of the 15th century Japanese poet Sogi , it is quite reasonable to compose it in Japanese, imagining a typical reader as a specialist in medieval Japanese literature. And there are far fewer of them in the world than Chaucer's experts and, accordingly, potential readers of the article about the peculiarities of the Canterbury Tales.

That is, a researcher who is engaged in national culture is already a loser: his article will never be published by a prestigious English-language journal and thousands of other scholars will never cite it. Of course, in the natural sciences this problem is not so acute, but the matter is not limited to the natural sciences, and you can come up with other examples. Say there are huge collections of diverse statistics collected in the USA. They are analyzed and quoted everywhere. However, many of them are collected to meet local needs, and their use outside the country is simply a side effect: the language is English, the data is good. The Japanese also love statistics and numbers, but their research for local needs, of course, is published in Japanese, and it is unlikely that anyone will start studying them abroad. Demanding the publication of employment or demographic figures of some provincial peripheral prefecture in English is simply absurd; however, for Japan's rating as a whole, this is a minus.

Recall also that an important part of the rating is “international connectivity,” that is, the proportion of foreigners among students and staff. And here, any American-British backwater outperforms many excellent universities outside the English-speaking world. We assume that scientists and intelligent students are more or less fluent in English. How many of them are ready to go to a little-known country and learn a completely new language for communication outside the university walls? In general, of course, many, but in percentage terms it wouldn’t be so. The English-speaking world is more familiar (in films, books), more understandable. Even if it’s a kind of self-deception, a false feeling (“I’ll go to Santa Barbara, I already know everyone there”), practice still shows that people are more likely to relocate to the United States or Australia than to Korea or even France.

Bike from life . One of my comrades put it this way: "Canada is almost like Finland, only you don’t need to learn Finnish." And he left Finland for Canada.

A little bit about fashion

Look again at the ratings. A special place in them is occupied by quantitative indicators related to the number of publications and citations, and other similar “scientometrics”. Compilers of ratings, like the character of a joke, are looking for keys under the lantern, because it is light there. Bibliometric indicators are among the few means of an objective, numerical assessment of the university’s contribution to science, so they are widely expected to be used. At first glance, everything is true: if you publish in a prestigious journal and are widely quoted, then your contribution to world science is significant. However, in practice, all sorts of “corrections” and “coefficients” have to be introduced into the methodology in order to take into account various nuances, so that in the end the initial simple idea turns out to be very blurred.

For example, in computer science, an average article is written by two or three authors, and biologists are not uncommon among biologists, since all involved, including washing flasks and injecting mice, are included in the ranks of the authors. Therefore, by injecting mice for three projects, you can quickly become a co-author of three articles. In computer science, the average article has been “living” for several years, after which no one will refer to it. Physicists can still easily quote original articles by Einstein and Bohr, that is, their citation indices continue to grow. The same physicists, in principle, are more likely to quote each other. Counted that the physicist will have to collect 2073 citations for his work in order to be included in 1% of the most quoted physicists in the world, and our brother computer science will have enough 149 citations to achieve a similar goal. I don’t know whether this is due to the greater friendship of physicists or to the fact that computer science is a kind of “umbrella” science that unites too diverse parts.

Of course, you can try to introduce “correction factors” and conditionally equate one and a half hundred quotations of computer science to two thousand citations of a physicist, but to achieve complete justice, hundreds, if not thousands of coefficients will be required. From a scientific point of view, the wars in Rwanda can be no less interesting than the wars of Britain and France, but it is easy to guess that the "fashionable" countries are much more interesting. There are fashionable countries, there are fashionable topics, subtopics and directions within subtopics; writing a paper on Method X to solve Problem Y, I can predict in advance how many citations this work can bring to me. Further, articles on fashionable topics are published in fashionable magazines, and for a different narrow-minded topic, a fashionable magazine is generally not in nature! So then explain why for the sake of you are published in the specialized journal X, если он неизвестен широкой публике.

One might argue that fashion reflects some kind of objective utility, but this, of course, is not entirely true. A lot of factors of an extra-scientific plan influence fashion, and how can a paradigm shift be achieved if fashion is always followed? There must be separate talents developing un-groomed trails. I agree that the distribution of fame and money can be uneven: publishing in Nature, engaging in a fashionable topic - great, here you have the money, the job title, and the interview on the central channel. Pulling your bed - well, pull on, let's see what grows out of this. In practice, unfortunately, thanks in large part to the ratings, the strong push the weak. If the university is in the third or fourth hundred of the ranking, there will be no special repression, but representatives of the elite can be rather jealous of their position in the final list. Особенно стараться ради прыжка на пару пунктов в третьей сотне как бы и незачем, а вот скакнуть с седьмого места на пятое — уже большое достижение.

Bike from life . Actually, the “professor of a prestigious British university” mentioned above told me about this. He was quietly fused to retirement (now he is emeritus), with the motivation "give way to someone who will bring the university more ratings and grant money."

Again, I am not against highly cited works and large grants, but in my ideal world there should be enough space for everyone - both those who participate in the race and those who philosophize in nature. It remains to be seen whose contribution will be more significant. It is bad when the measuring tool itself turns into an object of experimentation. At first, the child understands that if the thermometer shows 37.5, then you can not go to school, and then guesses that you can knock on the thermometer and make it show 37.5, regardless of the actual temperature. Unfortunately, similar actions are common in the scientific community.

Bike from life . I’ve already talked about staff optimization, and now I’ll talk about conferences. A good conference is a big plus for the organizers (international relations, quoting, raising the status of the institution, extra money ...) The question is how a “bad” conference differs from a “good” one. There are few objective signs, and among them one can single out the “acceptance rate”, that is, the percentage of submitted works that, as a result, were accepted for publication. At good conferences, the rate can be 20-25%, that is, seven to eight works out of ten submitted are rejected, and, in theory, the best ones remain. In reality, some unscrupulous organizers phoned all their friends with a request to send "any rubbish" for show. The nonsense is then rejected, and the remaining articles of slightly less nonsense are accepted. It seems that a beautiful rate is respected, but the quality of the conference is still lame. Unfortunately, over the years, such tricks are used more and more actively.

In general, for the sake of metrics, any sacrifices are made. Semi-fake PhDs defend themselves (I heard that in some countries in good countries they offer graduate students to defend two different, but very overlapping dissertations, so that the counter of defenders grows), not very useful articles are generated, spam messages like “write to our journal” or “ participate in our conference ”and so on.

East and West

In different countries and just in different universities, a completely different corporate culture is developing, which is not reflected in formal indicators, but at the same time is quite important for people within the system. You need to understand who is who and who needs what from life.

For example, in Europe, a strategy was adopted to expand the “pan-European scientific space”, as a result of which, in particular, large amounts of money began to be distributed not from within countries, but at the global European level. Universities are fighting for grants. The winner of the grant for the project hires employees for the duration of the project, and upon completion, the employees dissolve (if the next grant is not won, of course). The system seems to provoke competition and the general idea of ​​“people follow money and projects”, but it has a flip side in the form of coiled employees living “from grant to grant” and devoting half of their work to writing the next project plan. In theory, the crown of a young scientist’s career should be a professorship, but in practice there are very, very few permanent positions. On the other hand, mobility and cooperation are high. People move everywhere and are very glad to any cooperation, because the probability of winning a grant together with a good team is much higher. It is easy to make contacts, and many will be happy to think about how the two of you combine scientific topics to get a joint project.

In Japan, there are problems with mobility and friendship, but stability (sounds familiar, isn’t it?) People here are easily taken to permanent positions, and the question of providing yourself with a salary is not worth it. Grants are won for additional funding. Therefore, if anyone has the desire and energy to play these games, he plays. But no - does not play. In Japan, worse with students. Europe, in principle, is more social - there are different scholarships that pay, encourage measured thorough work. A career of the type “studied at the university, worked a little at the company, learned again, went to travel, worked” does not surprise anyone. In Japan, students are somewhat similar to Russian students: everyone at the same age enrolled in universities, they sound for four or six years without interruption at the same university, then they leave to work without delay. The thought of that the thesis may not be good enough and, perhaps, it makes sense to polish it for another year, will be met without understanding, although this is a common thing in Finland. Therefore, the sense of Japanese students is not very much. Well, of course, “gathering employees for the project” is not often, so you have to do it yourself.

I have hardly encountered American universities, but the situation there is “middle to half” according to my feelings. Scholarships are by no means given to everyone, but more willingly than the Japanese. It is difficult to earn a permanent position, but still easier than in Europe. Attracting a grant is cool, in many ways even cooler than scientific achievements. Americans in a personal resume list the won grants next to the list of publications. The Americans know how to file and sell themselves well, this cannot be taken away from them. Well, in general, their culture in general encourages curiosity, knowledge. So among them there are many sincerely partial people. Although a huge social stratification should not be forgotten either. They told me about one good, seemingly, American university, which is located behind a high, closed fence in the center of some eerie ghetto. The government decided that local adolescents should be brought to the people, and that the university is more profitable than prison. Social project, so to speak. In Japan or Finland, such things are unthinkable.

Epilogue

An inexperienced user comes to a cell phone store and asks "the best phone is not more expensive than N rubles." The seller will easily offer some popular model, and the buyer will leave satisfied. The user-geek will not be satisfied with bare "stars" and will figure out the features of a particular model. Perhaps Model A is generally cooler, but Model B has the ability to insert two SIM cards or the battery lasts longer.

Universities - matter is more subtle than telephones. Although you can be an "inexperienced user" and choose a university "by stars", I believe that it is much better to become a "geek" and try to understand the problem thoroughly. In the end, you can sell the phone or throw it away if you don’t like it, and it’s much more difficult to part with your place of study or work, so the choice should be made on the basis of a comprehensive analysis.