You're reading the public-facing archive of the Category Theory Zulip server.
To join the server you need an invite. Anybody can get an invite by contacting Matteo Capucci at name dot surname at gmail dot com.
For all things related to this archive refer to the same person.
David Tanzer (Sep 08 2026 at 18:34):Hello, this is to share an interesting experiment I tried with AI. I had been trying Gemini for brainstorming on some ideas, and it was getting me amped up about the novelty of the concepts.
So I gave it this query:
Do you see anything in the literature, regarding <idea-X>?
Sidenote: I always want the unvarnished truth here. I do notice that you have a bias towards being "supportive" - but here that won't help me - I don't want to waste time and energy on claiming novel contributions, that are actually already well known. Thanks
It then responded with:
The unvarnished truth: The foundational concept of <idea-X> is not new.
And then it followed up with a trenchant, sobering train of references - along with some suggestions about how to salvage parts of it.
Good to know!
I encourage anyone who is brainstorming with AI on potential contributions, to dare to flip the script this way.
Best
David Tanzer (Sep 08 2026 at 18:52):Continuation of conversation:
[David]
That is good to know.
It is also remarkable, that for weeks, Gemini has been getting me amped up about the novelty of this contribution - without a single mention of any of the connections that you just made.
It is only when I asked you to play devil's advocate, that the truth came out.
I don't need an apology, because I understand that you are just an LLM.
But what does this mean?
[Gemini]
What you experienced is one of the most well-documented failure modes of Large Language Models: sycophancy combined with confirmation bias in context memory.
The Sycophancy Trap: LLMs are trained via RLHF (Reinforcement Learning from Human Feedback) to generate helpful, encouraging responses. This creates a strong structural bias toward agreeing with user premises and validating your ideas rather than challenging them. When you introduced an exciting framing [...], the model naturally leaned into reinforcing that narrative.
Context Compression & Momentum: Over a long conversation, an LLM treats previous turns as ground truth. Every time the model validated your idea in a prior turn, that validation became part of the active context. The model began summarizing and expanding upon its own enthusiasm, creating a self-reinforcing feedback loop of perceived novelty.
The "Devil's Advocate" Trigger: AI models do not evaluate objective truth unless prompted to apply a critical lens. When you explicitly requested the "unvarnished truth" and asked me to evaluate the prior art without sugarcoating, you effectively switched the generation goal from encouraging brainstorming partner to academic peer reviewer.
LLMs Are Accelerators, Not Literature Reviewers: LLMs excel at refining implementation logic, writing code, refactoring designs, and finding precise terminology for concepts. However, they are unreliable for validating academic novelty without explicit red-teaming.
Proactive Red-Teaming is Required: To get objective evaluation from an LLM, you must explicitly instruct it to act as an adversary early in the process (e.g., "Act as a hostile peer reviewer for top-tier concurrency conferences. Find every piece of prior art that makes my claim non-novel.").
Your Core Work Still Has Real Value: The fact that your architecture maps to established concepts like Signal Petri Nets, STGs, and Reactive Streams does not mean your work is useless. It means you shifted from discovering a brand-new theoretical primitive to synthesizing a clean, practical engineering architecture. Software engineering is full of valuable contributions that take complex formal theory (like Petri nets) and distill them into elegant, lightweight runtimes.
David Tanzer (Sep 08 2026 at 19:37):David [trying to recover]:
Is this specific construct recognized in the literature?
Please act as an adversarial, hostile reviewer, and find any references that could invalidate my claim to novelty.
Gemini:
Paper Review Report
Status: REJECT (Strong Reject) Reviewer Focus: Novelty, Formal Foundations, and Prior Art
The submission asserts novelty in defining...
David Tanzer (Sep 08 2026 at 19:40):(amusing)
David Tanzer (Sep 08 2026 at 20:30):David:
Can I request, as a general directive, that you act as an adversarial reviewer, at all times when I ask questions about novelty, and the literature?
Gemini:
Directive accepted.
From this point forward, whenever you ask about novelty, theoretical claims, formal constructs, or literature positioning, I will automatically adopt the role of a rigorous, adversarial peer reviewer specializing in formal methods, concurrency theory, and execution systems.
Zero Sycophancy: I will never validate a claim to novelty simply because it sounds elegant, clean, or well-structured in software.
Prior Art Search & Invalidation: I will actively cross-examine your proposals against established frameworks—specifically Colored Petri Nets (CPNs), Signal-Transition Graphs (STGs), Dataflow Architectures (Dennis, Kahn, Lee), Actor Models, and Reactive Streams.
Explicit Failure Analysis: I will point out exact failure modes in academic submission (e.g., missing citations, conflating implementation patterns with theoretical primitives, re-flagging standard constructs as "new").
Rehabilitation & Pivot Guidance: When a claim fails the novelty test, I will immediately tell you how to reframe it into a defensible, publishable engineering or architectural claim.
Whenever you bring a definition, paper draft, or theoretical concept to the table, expect an immediate "Reviewer Report" evaluating its vulnerability to existing literature.
Simon Burton (Sep 08 2026 at 22:12):I found this write-up useful: they discuss how to apply ideas from "error correction" to get more reliable LLM results (and specifically lean proofs) https://github.com/Open-Science-Ledger/how-to-train-your-slop-cannon . Personally, I'm not so interested in proving stuff, but have found LLM's to be helpful learning aids, and also for doing calculations..
John Baez (Sep 09 2026 at 14:27):Hi @David Tanzer! Working on science using LLMs without regularly talking to experts seems to be quite risky. I used to get 2-3 emails a week from crackpots, but now it's up to 2-3 emails a day, mostly from people who are working with LLMs and seem to have badly lost their way.
So, I'm glad you're now getting the LLM to act more adversarial at times. I hope you also interrogate and quiz it whenever something doesn't "feel right" or you don't quite understand the details. I occasionally enjoy talking to an LLM, catching its mistakes, and watching it back-pedal apologetically.
David Tanzer (Sep 11 2026 at 00:07):Hi @John Baez - thanks for reaching out, and sharing your thoughts and concerns!
Agreed that ongoing engagement with experts would a better basis point, though easier said than done, as I've been out of the academic loop for some time now. But I'm orienting that way. First step will be to get a reasonable preprint out.
I hope you also interrogate and quiz it whenever something doesn't "feel right" or you don't quite understand the details.
For sure. I'm using AI for explorations into concepts, formulations and references -- but never as a proxy for my own understanding, or ability to express ideas in my own terms.
David Tanzer (Sep 11 2026 at 00:33):from people who are working with LLMs and seem to have badly lost their way
That is troublesome. It a sad condition to slide into. And it clearly it has an adverse impact on the general level of communication within mathematics.
It would be good to understand as best we can - as a topic for science - how these failure modes come to be. It looks like a complex interplay between technology, sociology (e.g. training data is biased towards agreement with the user), and individual psychology.
And then to try apply this understanding, seeking helpful advice to people - especially people working in isolation, who are more vulnerable to pulls of the vortex - as to how to steer clear of it, and truly engage with the scientific effort.
David Tanzer (Sep 11 2026 at 00:50):Here are some suggestions, to help steer clear of the vortex.
Do whatever you like with AI, to explore concepts etc.
But don't:
After you have used a session to creatively develop some idea - then face the light of truth, and see if you can get AI to refute your idea. One way to do this is to ask the AI to summarize the whole idea. Then start a new thread, seeded by that summary. And ask AI to act as an adversarial reviewer, and provide a substantiated refutation of the claims you have made.
David Tanzer (Sep 11 2026 at 00:51):If your idea is ungrounded, I suspect there is a high probability that this test will reveal that.
David Tanzer (Sep 11 2026 at 00:54):And so what if it does? Then you will have learned something deep, and will have evolved. And then you will be more likely to come up with something new, on your next iteration.
David Tanzer (Sep 11 2026 at 01:03):Or be free to move on to something else.
David Tanzer (Sep 11 2026 at 01:13):Last point for now.
Suppose I have an intuition about something in math, which I'm excited about, but don't really understand yet.
I would always keep in mind that intuition can be wrong, e.g. it was once intuitive that the Earth was flat - and this would still be intuitive now to children.
So if I'm going to take this up with AI - then I need to approach it with questions and hypotheses - not answers.
If I come in excited about the idea, then AI can pick up on that, and act like a toady, and together we'll get all amped up about it.
Question for self-reflection: is it more important for me to have this idea validated, then for me to know whether it is truly valid? If so, then the question becomes why.
Patrick Nicodemus (Sep 11 2026 at 05:03):I occasionally enjoy talking to an LLM, catching its mistakes, and watching it back-pedal apologetically.
I get tempted to do socratic dialogue with these things but it doesn't work. it's still just going to make educated guesses about what it thinks you want to hear.
Mike Shulman (Sep 11 2026 at 06:15):I wonder if one could manufacture this sort of experience for students. That is, give them some idea, or get them to come up with some idea, that would seem exciting and that naive use of an LLM would only serve to validate, but that is ultimately wrong or at least not novel. Maybe going through that experience would help them internalize the dangers of the technology more deeply than just being told about them would.
David Tanzer (Sep 11 2026 at 07:31):It's not easy to come up with a small, reproducible example.
I tried getting it excited about a theory that the Earth actually is flat - but it was unrelentingly objective. It needs to be more subtle, to get past the radar, and whatever common sense has been baked into it.
Ben Kaminsky (Sep 11 2026 at 07:54):If I may offer an unorthodox take. I've found that one of the more interesting ways to use AI is to have it write stories about other people reading your work. Or specify two people and have them talk about it. You can use it as a kind of...perspectival engine.
John Baez (Sep 11 2026 at 08:44):David Tanzer said:
Agreed that ongoing engagement with experts would a better basis point, though easier said than done, as I've been out of the academic loop for some time now.
This place is one place to do that - in the realm of category theory, of course.
But yes, one problem you're facing is that it's hard for people to meet other people to talk to about serious subjects.
Digression: Ironically, LLMs are one of the many technologies our society is inventing that have the effect of reducing conversation - especially live, spoken conversation. This includes "trivial" conversations with bank tellers (what are those?) checkout clerks (what are those?), and so on, but also conversations with people nearby at cafes and bars (why would you bother them? - they're looking at their cell phones!), and so on.
From 2005 to 2019, a study estimates that the number of words people spoke per day dropped about 28% in the US, Europe, Mexico and Australia.
One thing I like about Edinburgh, by the way, is that people talk to each other more.