You're reading the public-facing archive of the Category Theory Zulip server.
To join the server you need an invite. Anybody can get an invite by contacting Matteo Capucci at name dot surname at gmail dot com.
For all things related to this archive refer to the same person.
Sichu Lu (Jan 05 2023 at 20:49):https://arxiv.org/abs/2301.01445 This is absolutely amazing and I think the authors are here on this Zulip @Evan Patterson @James Fairbanks I just want to share as I'm reading this.
Evan Patterson (Jan 05 2023 at 20:54):Thank you for the kind words!
Sichu Lu (Jan 05 2023 at 21:00):I'm still digesting this paper so apologies if I am asking stupid things but do you see a possible synthesis of your work describing the mathematical observables with the work done by deepmind and others using large language models to empirically train useful models? see here https://www.nature.com/articles/s41592-021-01252-x
Evan Patterson (Jan 06 2023 at 02:38):I'm not familiar with that line of work, so I can't say much about it specifically. Here's a vague thought instead.
Our paper is about the conventional way to build scientific models: by using a lot of domain knowledge to handcraft a model with not-too-many parameters and then fitting it to data. When this works, it works really well because the model is latching onto reliable mechanisms and regularities in nature. But doing this requires a lot of labor and expertise, and it doesn't scale very well with traditional methods. (Part of what we're trying to do is make it more scalable by formalizing the compositional aspects of model building, but that's a different story.) Nowadays, people, like the authors of the paper you cite, are trying to use deep neural nets and other black-box prediction algorithms to replace all that bothersome labor with big piles of data. They've had some impressive successes, but these models also suffer from various problems, of which the most significant is, I think, the frequent failure of external validity. Ben Recht has a nice blog post about that.
It's natural to wonder whether and how these two approaches to modeling could be reconciled or integrated. That seems like the sort of thing that category theory could help with. But don't ask me how, because I don't know yet :)
Sichu Lu (Jan 06 2023 at 04:48):If you can't speculate on the internet then where can you? :D I guess where I am coming from is the classic nlab POV that these categories are the "right" descriptions/representations. and if you have the right abstractions, then you can use them to simplify the calculations and provide a constraint on the spaces. I notice you cited Uri Alon in your paper and I had very similar ideas while I was working through his youtube lecture/book which is why I was so excited to read your paper and immediately groked what you were trying to say here. This is just the motivating philosophy and of course I have to implement this onto an algorithm level. As to your point about the large language models not being very good at figuring what's ground truth if asked to do out of distribution tasks, I have been thinking going beyond what is being done with say basic reinforcement learning, like what deepmind has done with alphacode for example by just having a trained datasets that serves as the "ground truth". The idea is that you should be able to get large language models to generate their own hypothesis,(which is basically what they are doing anyway in these models) A friend and I have coined a term for this, we call this unreal computing because in some ways the model already has an internal world model, you just have to get it output it, using the right prompt engineering in some cases! Basically you can do hypothesis testing by getting it to output it's guesses about the world and then testing this against empirical reality. There is already a community of people doing this based on that deepmind paper I send. Here: https://openbioml.org/ I have been in their discord/meetings listening to them. I think hypothesis testing can be implemented(the details need to be fleshed out but there is this recent paper by Schmidhuber using actor-critic reinforcement learning: https://arxiv.org/abs/2212.14374) using reinforcement learning! Your work/paper here offers a way to maybe guide the reinforcement learning systems(or maybe this is another unsupervised system with expert rules??, the distinction blurs) towards the right rules and also serves a check on the mathematical modeling of real world objects using the most abstract nonsense there is.
Evan Patterson (Jan 08 2023 at 20:33):Thanks for the pointer to OpenBioML, seems interesting.
Sichu Lu (Jan 10 2023 at 01:02):oh yeah join the discord and come talk to the people. I have some vague intuitions on applying categories to the empirical stuff but that's a much more longer reply for another day. please do poke me here or on twitter if I forget
Van Bettauer (Jan 11 2023 at 01:41):Exciting work! I wish I had this paper when I was trying to learn gene regulatory networks from scRNA-seq data in my MSc. OpenBioML is great community. I'd be all in on a open source bio-cat-ml project :smile:
Evan Patterson (Jan 11 2023 at 03:56):Thanks! I will join the Discord and see what is going on there.
Sichu Lu (Jan 19 2023 at 02:22):https://www.biorxiv.org/content/10.1101/2023.01.13.523995v1 image.png just putting here because it is relevant to the discussion we were having
dan pittman (Feb 25 2023 at 03:35):Subscribing to this thread due to a vested personal interest. I have a kid with a rare genetic disorder for which I’ve been trying to find a way that my specific brain can be useful. I’ve watched the first lecture of Alon’s systems biology course, which was wonderful and I can already tell that this is damn near exactly the lens I’ve been looking for to grapple with all of this. Thanks for sharing!
Evan Patterson (Feb 26 2023 at 04:42):Thanks for the kind message and best wishes to you and your family.