You're reading the public-facing archive of the Category Theory Zulip server.
To join the server you need an invite. Anybody can get an invite by contacting Matteo Capucci at name dot surname at gmail dot com.
For all things related to this archive refer to the same person.
Morgan Rogers (he/him) (Aug 31 2026 at 15:42):Following Beppe Metere's post linked in , I wonder if anyone has ideas for reliable measures of scientific ability / integrity / accomplishment that could plausibly come to replace the detestable bibliographic metrics that are the current default.
An extant competing one, with its own problems, is that of direct peer evaluation: in order to determine the quality of a candidate, one asks other experts in their field to evaluate them. That system:
It isn't terrible, but I can think of plenty of research that hasn't been picked up because this system incentivises experts to support work close to their own rather than a diversity of topics, since this strengthens their position in the community.
Peva Blanchard (Aug 31 2026 at 16:53):We could possibly start by listing a few reasons why bibliometrics are detestable?
Amar Hadzihasanovic (Aug 31 2026 at 18:38):Something that seems to be a pattern about metrics is that they are quite reliable (and interchangeable) in recognising "insufficient" candidates who do not meet the minimal requirements, but arbitrary and unstable in the ranking and evaluation of "sufficient" candidates. I have long thought that there is a good argument for two-stage selection processes where any reasonable method is used to identify all the candidates that meet the requirements, and then a lottery is used to pick among those.
David Michael Roberts (Sep 01 2026 at 03:31):@Peva Blanchard for a start: https://www.ams.org/notices/200808/tx080800968p.pdf which is the summary of https://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf
eg
The sole reliance on citation data provides at best an incomplete and often shallow
understanding of research—an understanding that is valid only when reinforced by other
judgments. Numbers are not inherently superior to sound judgments.
David Michael Roberts (Sep 01 2026 at 03:33):And also this other one:
Nothing (and in particular no semi-automatized pseudo-scientific evaluation that involves
numbers or data) can replace evaluation by an individual who actually understands what
he/she is evaluating. Furthermore, tools such as impact factors are clearly not helpful or
relevant in the context of mathematical research
https://www.mathunion.org/fileadmin/IMU/Report/140810_Evaluation_of_Individuals_WEB.pdf
Beppe Metere (Sep 01 2026 at 13:28):Hi @Morgan Rogers (he/him) . Indeed, direct peer evaluation certainly has problems, including biases, personal relations, and power dynamics. Nevertheless, in the present situation I think it remains more reliable than bibliometric evaluation, especially when it is conducted with rigor and evaluators are required to explain their choices.
Furthermore, LLMs are making bibliometric indicators even less reliable. They can increase the production of papers, citations, and variations of existing results without a corresponding increase in scientific understanding. Bibliometric indicators were already indirect measures of quality; now the distance between scientific output and scientific value can become even larger.
Of course, peer evaluation is not objective. Experts can favour areas, methods, and communities close to their own. But I think that trying to automate and objectivise evaluation is illusory. Every evaluation involves choices about what is important, original, or promising. An algorithm does not eliminate these choices: it only hides them. We need evaluators who explain their judgements and assume responsibility for them. This is something LLM cannot do: taking responsibility of their own choices.
The fact that some areas of mathematics try to prevail over others, sometimes suffocating them, is a political problem. It cannot be solved simply by changing the evaluation method. If we want to preserve a diversity of topics and approaches, we must make explicit choices of scientific policy.
The problem is then moved: if we reject superficial bibliometric evaluation, what should we evaluate?
Perhaps we should discuss directly with candidates their research, objectives, methods, and future plans, including how they use LLMs and how they verify their results. We could also evaluate mathematical fluency: the capacity to move between definitions, examples, proofs, and intuitions; to recognise incomplete arguments; to formulate good questions; and to explain why a problem or result is significant.
This leads to a more general question: should we search for mathematicians with exactly the same competences as those who entered academia before LLMs? Some capacities remain essential, such as conceptual understanding, critical judgement, and verification. But the intelligent and responsible use of LLMs may also become part of mathematical competence.
The question may therefore be less whether a candidate produced a paper without assistance, and more whether they understand and control the mathematical process, can verify what has been produced, and can defend their scientific choices.
Snow Walker (Sep 01 2026 at 15:13):Without significant effort, an AI is still not capable of writing an innovative Annals or Acta paper : they can give interesting examples, but most of the AI results I've seen have not been especially innovative in terms of method, they just connect things that already exist in the literature in a superhuman way.
It further goes without saying that even if AI competely takes over, a talented mathematician armed with AI is always going to ask better questions than an worse mathematician with the same AI. The problem solving gets pushed one level up. There's a reason why many of the striking results so far have been found by Levent Alpöge, a man who was an one of the most talented young mathematicians in the world before going to Anthropic.
The solution is to double down on what has always been valued in mathematics, quality over quantity. If every article below, say, the usual standards of, say, JPAA can be automated, perhaps we need to appropiately decrease the value of lower publications. It is already the case that in many areas of mathematics, publications in weak journals "don't count."
Direct peer evaluation is just a terrible way to do things. "Peer" here usually means unaccountable older person at some prestigious university who already has too much direct power over publication and hiring decisions, and is often much weaker than the young people in the field because they got their job in the 1970/80s in a less competitive time. I can think of at least one case where the old guard has treated a talented young mathematician, who was, if publication record reflects anything, much more distinguished than them, quite poorly because, as it appears to me, they didn't like their attitude and lack of deference.
Nicolai Kraus (Sep 02 2026 at 19:29):Here are my two cents on this. To me it seems that, when hiring PhD students and postdocs, decisions are generally very good. That's because experts spend a significant amount of time and effort to choose people in their own field. It's only at faculty level when decisions become based on questionable criteria, because people from other areas have no idea how to evaluate candidates, and maybe also don't have the time that would be needed. (Instead of "don't have the time" one could say "don't have the motivation", which would suggest that panel members are just too lazy to evaluate applications properly, but I think that would be a bit unfair -- developing enough insight about all the fields of all the applicants to come to fair conclusions just isn't possible.) Anyway, that does suggest that direct peer evaluation could work better than what is currently used. (I'm personally reasonably sure that it would work better than what is currently used, but I can't prove it.)
Unfortunately, even if better than what is currently used, direct peer evaluation is still quite difficult and potentially unfair when one has to compare people from very different areas, and I think all the issues that previous posters have mentioned are real issues. It also would make conferences quite a bit different, in a bad way. I'm in favour of what @Amar Hadzihasanovic () suggested:
Would that mean that the best people get hired? Of course not, because of the second step. But actually think that the luck one would need would be no more than the luck required in the current system, and I don't think the system would be worse than what we currently have. A main positive effect would be that people could stop gaming metrices and do what they actually want to do in their job. I also think the same principle would be suitable for grant application decisions and similar things that are too difficult to evaluate properly.
I agree with most things that @Snow Walker wrote above, but this bit seems unfair to the "unaccountable older person[s]":
Snow Walker said:
unaccountable older person at some prestigious university who already has too much direct power over publication and hiring decisions, and is often much weaker than the young people in the field because they got their job in the 1970/80s in a less competitive time.
I think this is unfair because:
a) Formulated as "having too much direct power" makes it sound like a privilege, but it's real work that someone has to do [at least in the current system], and it's not something that mathematicians actually want to do.
b) My impression is that many people who're nowadays professors are worse (or at least aren't better) mathematicians than they were when they were postdocs. Apart from the standard explanation about brain aging, I think there's also the effect that, as a postdoc, one has much more time to focus on maths, while afterwards, one is forced to do all sorts of staff that one didn't sign up for, which can make it challenging to keep up. As a result, it would be unreasonable to expect that current professors are better mathematicians than current postdocs who aspire to become professors. And while I have no doubt that the 1970/80s were less competitive from some point of view, they surely came with their own challenges. (I wasn't in academia at that time so I'm not qualified to judge.)
Peva Blanchard (Sep 02 2026 at 20:44):We are not very far from Goodhart's law. So any explicitly stated rule will almost surely find an adversarial counter-example.
From what I understand so far:
Peva Blanchard (Sep 02 2026 at 20:47):Maybe, another perspective on this topic is to emphasize the properties of the recruitment process that we'd find desirable. Here are a few on top of my head:
The last one is probably my favorite, because this is what enables improving based on learned lessons. But this raises questions of governance.
Amar Hadzihasanovic (Sep 03 2026 at 05:27):Re: the lottery system, I think in general it would be healthy if calls for applications were treated like "procurements": the university / the funding body is in need of these services, and is looking for candidates who are able to provide them. Once it is established that a group of candidates is capable of providing those services, each one of them is equally suited to getting the job / the funding, and it is irrelevant (with respect to the call) how they perform outside of that domain.
Amar Hadzihasanovic (Sep 03 2026 at 05:29):If one is looking for particularly "elite" candidates (wrt some metric), they can raise the threshold by introducing extra requirements, e.g. require "a proven ability to publish in Annals of Mathematics and/or Inventiones", at the risk of having the "procurement" fall through if no candidates meet them.
David Michael Roberts (Sep 03 2026 at 05:32):I don't like the idea of "proven ability to publish in X", because that inherits the biases from that journal or tier of journals. In job ads I used to look at they would say something like "proven ability (or strong promise) to obtain competitive grants". In my case there was exactly one grant scheme I could reasonably apply for, and only twice before never again, and that is well-known to be a very unstable process, where there is essentially a huge plateau of equally deserving applicants who sit around the boundary of success. This was noted above for jobs. Chaining together such processes leads to further fragility/instability in how success happens.
Amar Hadzihasanovic (Sep 03 2026 at 05:52):Yes, I think that's a terrible "requirement", but I don't think one can regulate "bad calls" out of existence; the point was to force requirements to be explicit and public, because with a lottery system in the second phase anyone who meets them can get the job. I still think it would be theoretically fairer to have "has published in journals X, Y, or Z" as a requirement, rather than an informal criterion by which candidates are privately ranked.
Matteo Capucci (he/him) (Sep 03 2026 at 15:06):Amar Hadzihasanovic said:
If one is looking for particularly "elite" candidates (wrt some metric), they can raise the threshold by introducing extra requirements, e.g. require "a proven ability to publish in Annals of Mathematics and/or Inventiones", at the risk of having the "procurement" fall through if no candidates meet them.
uhm that seems like bib metrics with extra steps then, though in spirit I vibe
Matteo Capucci (he/him) (Sep 03 2026 at 15:13):I recently learned of a Finnish tech startup's hiring process where (after a modicum of filtering) they just pay the candidate to show up and work with them for a week.
Eventually, all metrics are proxies for 'how well could we work with this person to deliver outstanding research?'. If someone shows up for a few days, gives a seminar, has conversations with peopl, then the group can _really_ feel in their gut if they would be a good fit or not.
The only downsides of this approach I can think of are (1) economical (you must pay travel for 1 week, or worse, ask the candidate to foot the bill) and (2) some people (I'm thinking some neurodivergent folks especially) might feel uncomfortable into such a situation of scrutiny.
Morgan Rogers (he/him) (Sep 03 2026 at 15:22):I like the sound of that @Matteo Capucci (he/him) , the paid version anyhow. For jobs in the arts (acting, design...) one often has to do a non-trivial amount of labour - producing a response to some brief - to compete for the position, and that labour is almost never compensated or even rewarded beyond the successful candidate getting hired. A more equitable version of the "show us what you can do" hiring process might be sustainable... except that with a shrinking job market every advertised position is getting hundreds of applicants, so some of the previously mentioned pre-selection processes will surely be necessary.
Snow Walker (Sep 03 2026 at 16:12):Thank you for your reply. I certainly should have been clearer that I was not making a generalisation to every older person in mathematics, many of whom are excellent.
For what it is worth, and since she has chosen to make this completely public information, I can mention that one of the cases i was thinking of was how Caramello was treated by the old guard of category theorists a few years back. She was denied a major UK grant on the basis of an anonymous peer review that almost certainly came from a senior figure in the field who believed her to be uppity. Regardless of your views on her, she had at that point managed to publish in more papers in Advances-tier journals almost any of the older people in the field. By any objective standard, her treatment was unfair.
I also don't think that it is unfair to say it is objectively harder now to get a position in a mathematics department than in the 1970/80s. If you want a position at a good US department today, it is now almost mandatory to have a publication at the level of Duke. In contrast, this is quite rare among the older generation, who often got positions with their best publications in mid-subject level journals.
Finally, I disagree that that looking at the prestige of the journal is a bad way to measure quality within a field. When I read AGT or JPAA, the papers are usually better than in HHA or TAC. When I read Transactions, Advances or IMRN, the papers are usually much clearly better than in JPAA. When I read Crelle or Duke, the publications are usually better still. When I read Annals or Inventiones, I am usually blown away. My experience of submitting papers, and how much work I must do on it, is is similar; with exponential jumps in either work or cleverness at each tier. Of course, these journals have selection biases between fields and there is much noise, but if we are in a situation where we must compare people within the same field they are usually quite accurate. I certainly don't think that random selection reflects this.
David Michael Roberts (Sep 03 2026 at 23:14):FWIW my highest cited paper by a long shot is in TAC, and I will not publish in Elsevier journals for ethical reasons which rules out JPAA and Advances, and have held that position for over a decade. Where then do I send pure category theory papers to be taken seriously, by this attitude?
David Michael Roberts (Sep 03 2026 at 23:14):Getting a pure CT paper in Crelle or Duke seems to me a pipe dream.
Nicolai Kraus (Sep 04 2026 at 00:44):Snow Walker said:
I certainly should have been clearer that I was not making a generalisation to every older person in mathematics, many of whom are excellent.
Sure, I certainly didn't think you meant to make such a generalisation, which would be absurd. My main point was that "having power" sounded like a too positive thing. I'm sure many older people would be very happy if they could get rid of this "power" and do maths instead. It's just that the current system doesn't work if everyone thinks like that. In general, it's better to give power to those people who don't want to have power, paradoxically.
For what it is worth, and since she has chosen to make this completely public information, [...]
You're absolutely right that there are cases like this that demonstrate clear problems of peer review. I don't think anyone believes that peer review can be considered "perfect" in any reasonable sense. The question is just whether we have anything that's better than peer review, and I don't think the answer to this is simple. (I personally don't think that peer review is the best we can do; see my original post.)
I also don't think that it is unfair to say it is objectively harder now to get a position in a mathematics department than in the 1970/80s. If you want a position at a good US department today, it is now almost mandatory to have a publication at the level of Duke. In contrast, this is quite rare among the older generation, who often got positions with their best publications in mid-subject level journals.
I don't necessarily disagree with your conclusion that it's harder now (and I don't think I'm able to judge). But your argument for this seems too simple. Your argument assumes that a publication in journal X in the 1970/80s was as difficult to achieve as a publication in the same journal today. You're now talking about a time when the modern internet didn't exist and people were lacking all sorts of tools (e.g., plain TeX was just created in the middle of that time period). Doing anything without these tools just seems to much harder and time-consuming to me than what we're doing today.
Finally, I disagree that that looking at the prestige of the journal is a bad way to measure quality within a field.
I'm not really a CT person, my field is type theory, so maybe that explains why I haven't made the experience you've made. In my experience, the same principle as with metrices and job applications apply: High-prestigue venues are reasonably good at filtering out (i.e. not publishing) bad papers, but they're not good at making choices between very good and excellent papers.
Graham Manuell (Sep 21 2026 at 08:52):Snow Walker said:
Finally, I disagree that that looking at the prestige of the journal is a bad way to measure quality within a field. When I read AGT or JPAA, the papers are usually better than in HHA or TAC. [...]
This whole discussion is moot post Navier-Stokes, but let me just say that my experience has been closer to the opposite of this.
Nathanael Arkor (Sep 21 2026 at 09:04):My experience too is that the correlation between the prestige of a journal and the quality or significance of the paper is relatively small. Other, more political factors, seem largely more relevant.
Nicolai Kraus (Sep 21 2026 at 16:06):I would argue that "quality or significance" is used in a too one-dimensional way, and to me it seems that journal prestige is sometimes confused with how specialised or general the journal is. Some journals are considered lower prestige simply because they're narrow, but if one is interested in exactly that narrow topic (which is often the case for experts!), the journal's papers are probably just what one wants, independently of the "prestige". If one thinks that "higher significance" means that a paper is relevant for a wider part of maths, then it's plausible that more "prestigious" journals have more such papers.
Oisín FC (Sep 21 2026 at 17:34):I guess relevant: https://academia.stackexchange.com/questions/96914/is-it-bad-to-publish-only-in-specialized-math-journals. Edit: Well, was relevant, haha
Morgan Rogers (he/him) (Sep 21 2026 at 18:50):I think it's premature to conclude that a big AI result is somehow going to overturn the status quo in academic publishing, but the fact that it's pushing us to examine what we could do instead is just what I hoped from this discussion
Nicolai Kraus (Sep 22 2026 at 13:30):Graham Manuell said:
This whole discussion is moot post Navier-Stokes
Genuine question: Why?
Peva Blanchard (Sep 22 2026 at 15:23):Here is how I understand Graham’s point. It is understood that the quality of a researcher is its inner ability to produce novel, meaningful insights or results. It is the thing that we honour and give prizes for. If machines were to enter the game, which the Navier-Stokes story seems to tell, then prizes would honour mostly those who have access to those machines. This breaks the honour ladder.
Morgan Rogers (he/him) (Sep 23 2026 at 08:34):I don't know if this was mentioned already somewhere, but it seems somewhat relevant for this discussion.
In France, some national and local organising is happening at the moment to work on the structure of higher education and research. I haven't had the opportunity to attend one of these meetings (they happen to fall in September when I'm teaching almost every day), but it's interesting to see what the centres of interest have been so far, including some discussion of changing how we measure scientific accomplishment.
Nicolai Kraus (Sep 23 2026 at 16:33):Peva Blanchard said:
Here is how I understand Graham’s point. It is understood that the quality of a researcher is its inner ability to produce novel, meaningful insights or results. It is the thing that we honour and give prizes for. If machines were to enter the game, which the Navier-Stokes story seems to tell, then prizes would honour mostly those who have access to those machines. This breaks the honour ladder.
Thanks for explaining. I can see the point, but I don't (yet) find it convincing, even if we assume that AI has found the NS proof independently, without human help, and without essentially stealing other human's work. The NS proof demonstrates the AI can solve hard problems, but there are other things that we value.
I think it is possible that the community has to re-evaluate what is most valuable, and if certain problem-solving can be done cheaply by AI, then maybe it will naturally become less valuable. However, until the AI can do everything that a mathematician has to do, I don't see the discussion as moot.
Peva Blanchard (Sep 28 2026 at 11:10):I am reading Michael Sandel's book The tyranny of merit, and, while discussing the issues of meritocracy embodied in the admission procedures in Ivy League's universities, he suggested the following procedure:
It's quite interesting that it is exactly the solution @Amar Hadzihasanovic suggested. The difference is that Sandel proposed this solution for reasons related to meritocracy.
ps: If I try to sum up his thesis, I'd say he attacks the idea that one "deserves his or her success", because that entails that "one deserves his or her failure", and this leads to all sorts of bad things. By relying on explicit randomness, winners will be less likely to "inhale too deeply their success".
Simon Burton (Sep 28 2026 at 12:59):There's an inherent tension here, between wanting an objective measurement of value and the AI's ability to optimize for that. My own perspective is that "value" is just not objective at all, and the ongoing AI developments are just going to force us all to come to terms with this.
Peva Blanchard (Sep 28 2026 at 19:14):I think it all boils down to the fact that there is a limited number of positions available, and we are thinking about practical means to allocate these scarce resources.
The meritocratic mind, I guess, will state that the resource should go the most accomplished scientist; hence all the questions about pragmatic criteria to recognize one, and opportunities for optimization (see Goodhart's law).
But @Amar Hadzihasanovic 's method is interesting in that it tries not to "deterministically" find the most accomplished scientist. Obviously bad candidates are sorted out, and then a lottery is used to select the one among the rest. This is fairer, in a sense, and probably more robust. Once you're in the sampling pool, you can't optimize your way to have a better chance at winning the lottery (assuming the lottery can run fairly of course).
Graham Manuell (Oct 01 2026 at 18:11):Nicolai Kraus said:
Graham Manuell said:
This whole discussion is moot post Navier-Stokes
Genuine question: Why?
Sorry for not checking this thread in a while. You're right, sorry. I misspoke when I said "this whole discussion". One could alternatively say the original question in this thread is more important than ever.
What I meant was the discussion of "raising the bar" as suggested by @Snow Walker, not the general problem of finding better metrics, though I do think fixed legible metrics will always to subject to Goodharting. Saying "this whole discussion" was a bad way to express what I meant.
But yes, as mentioned by @Peva Blanchard, the Navier-Stokes result does complicate things. It means that something which previously would be have been incredibly prestigious cannot be seen as quite so prestigious at this point, since AI might be very good as some things previously in this category. The non-equal access to the most powerful models makes this even more acute.