You're reading the public-facing archive of the Category Theory Zulip server.
To join the server you need an invite. Anybody can get an invite by contacting Matteo Capucci at name dot surname at gmail dot com.
For all things related to this archive refer to the same person.
(=_=) (Apr 15 2020 at 08:31):I just stumbled upon this 2-year-old article about possible re-designs of the scientific paper:
https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/
(=_=) (Apr 15 2020 at 08:35):I've actually seen this "graphic novel" version of Collective dynamics of ‘small-world’ networks by Watts and Strogatz. Here's Strogatz's reaction to the re-design:
Strogatz admired Victor’s design. He later told me that it was a shame that in mathematics it’s been a tradition for hundreds of years to make papers as formal and austere as possible, often suppressing the very visual aids that mathematicians use to make their discoveries.
Morgan Rogers (he/him) (Apr 15 2020 at 09:24):I think it's a shame this paper rolls away on a total biographical tangent. The conclusion is just "we should use computational notebooks", which I guess is an upgrade on pdf in some respects but it's hardly a massive leap of creativity away from the format the writer is claiming is obsolete.
Morgan Rogers (he/him) (Apr 15 2020 at 09:28):There are plenty of ways to include diagrams in pdfs these days, so the 'suppression of visual aids' that Strogatz describes is mostly laziness on the part of the authors in this day and age. I was hoping that the article would propose something a lot more radical, like encouraging accreditation of original contributions to community projects, rather than just "people should make their work more interactive :smiley: "
James Wood (Apr 15 2020 at 09:43):Tomas Petricek has also had a go at doing something “better than PDF” before. http://tomasp.net/coeffects/
Fabrizio Genovese (Apr 15 2020 at 09:57):Morgan Rogers said:
There are plenty of ways to include diagrams in pdfs these days, so the 'suppression of visual aids' that Strogatz describes is mostly laziness on the part of the authors in this day and age. I was hoping that the article would propose something a lot more radical, like encouraging accreditation of original contributions to community projects, rather than just "people should make their work more interactive :smiley: "
Suppression of images is not just laziness, it's also page limit (esp. in conference submission). I guess a limit of 12 pages is fine for people working with "standard" mathematics, but try to let a paper about string diagrams fit within that limit...
Morgan Rogers (he/him) (Apr 15 2020 at 10:12):Okay, but a page limit is not really a problem with the medium, which is what is being denounced as obsolete, it's a problem with the system for publication. Which, based on the other topic in this stream, is a much bigger problem!
Morgan Rogers (he/him) (Apr 15 2020 at 11:15):This seems like a good opportunity to put forward my own idea of how this could work, at least for mathematics as a highly structured subject of research. The "traditional internet format" suffers from many the same restrictions that pdfs do. There's greater scope for interaction, but most pages about scientific subject matter largely consist of text blocks with static pictures or diagrams. If one is lucky the page author will have the facilities to format equations nicely. In a well-maintained online project (such as a community wiki) content may be well-connected in the sense that technical terms and names link to relevant other pages, and if one is really lucky the contents will be searchable...
The most glaring thing that's absent from this format (imo) is the visibility of structure. Text and pictures will, for the foreseeable future, be the most convenient final delivery method of detailed information, but there are things about a subject I should be able to understand without having to comb through these details. I imagine a network structure, a virtual map of a subject, with nodes linking to, say, entries in a wiki. At a finer level, one could construct networks based on the results presented in a given textbooks, expanding them with results that have been discovered since the book's publication.
Morgan Rogers (he/him) (Apr 15 2020 at 11:15):If implemented in a serious way, this could give:
Morgan Rogers (he/him) (Apr 15 2020 at 11:22):A lot of research goes into visualisation of knowledge every year (I should put a source here but I need to get back to "real work"). The problem is that it's specialised: one has to learn to use the tools that are being constructed, and that's a big enough obstacle that only a tiny fraction of it reaches mainstream. As soon as the sharing of ideas in a visually structured way becomes as easy as contributing to a community wiki or a stackexchange page, what I've described will become the next generation of those, I'm confident of it.
Morgan Rogers (he/him) (Apr 15 2020 at 11:24):(PS: You can probably tell from reading the above that making this idea a reality is a dream of mine, despite my limited programming experience. If I leave academia at any point, this is my backup plan.)
Fabrizio Genovese (Apr 15 2020 at 11:39):Fabrizio Genovese said:
Morgan Rogers said:
There are plenty of ways to include diagrams in pdfs these days, so the 'suppression of visual aids' that Strogatz describes is mostly laziness on the part of the authors in this day and age. I was hoping that the article would propose something a lot more radical, like encouraging accreditation of original contributions to community projects, rather than just "people should make their work more interactive :smiley: "
Suppression of images is not just laziness, it's also page limit (esp. in conference submission). I guess a limit of 12 pages is fine for people working with "standard" mathematics, but try to let a paper about string diagrams fit within that limit...
Precisely! I think this is the biggest problem right now, the format is way less problematic.
Fabrizio Genovese (Apr 15 2020 at 11:40):In any case, I think the really needed thing to change academic publishing is people willing to do it, which also means spending an _insane_ amount of time in building these platforms.
Fabrizio Genovese (Apr 15 2020 at 11:48):At this point, I find this kind of discussion a bit useless. I do not say acrimoniously, but from a personal experience point of view. All of us have ideas about what could be improved or not, and we could talk endlessly about this. For a time, Statebox wanted to really collaborate on creating a journal which would improve the current workflow in many ways. We discussed about this endlessly, how it should or should not look. We also got the endorsement of Mathematica so that we could naively support jupiter and mathematica computational notebooks to make the publication format more expressive. We even travelled all the way to London to meet other researchers (@Philipp Zahn and Andrew Lewis-Pye) in person to discuss this further, and get backed up by the LSE in the process. Well, guess what? It went nowhere. And the reason for this is that, all things considered, no one among anyone of us had the time you'd need to put into such a project. This kind of stuff, done the right way, is a thousands man hours project. So if we really want to change how the academic workflows are implemented, the first thing would be making sure we have people on board that can literally work on this fulltime. :slight_smile:
Morgan Rogers (he/him) (Apr 15 2020 at 11:48):Fabrizio Genovese said:
In any case, I think the really needed thing to change academic publishing is people willing to do it, which also means spending an _insane_ amount of time in building these platforms.
The plan is: solve a Millennium Problem, claim the prize, hire a small team of devs for two years to work full time on building an open-source platform and a way to automatically generate the structural component from existing wikis, run out of money, and then work out what to do next... :stuck_out_tongue:
Fabrizio Genovese (Apr 15 2020 at 11:51):I mean, I see a similar pattern with Compositionality. I submitted there ages ago, got reviews, made the requested edits, and never heard from them again. But I don't blame them. Managing a journal is an extremely difficult and time consuming project, especially if it happens on a volountary basis. It''s literally people giving away their time to read your stuff and evaluate it. We were just getting started with bootstrapping a journal and gave up. You can imagine how difficult it must be to run it once papers start coming in :slight_smile:
Fabrizio Genovese (Apr 15 2020 at 11:52):Now, if you also want to rely on "new infrastructure", you have to factor in stuff like bugs, that will have to be insta-fixed as papers start to be submitted, etc etc etc
Fabrizio Genovese (Apr 15 2020 at 11:52):The only way I see this happening is if a bunch of universities get a big ass grant to put people to work exclusively on this project for a couple of years.
Fabrizio Genovese (Apr 15 2020 at 11:53):Morgan Rogers said:
Fabrizio Genovese said:
In any case, I think the really needed thing to change academic publishing is people willing to do it, which also means spending an _insane_ amount of time in building these platforms.
The plan is: solve a Millennium Problem, claim the prize, hire a small team of devs for two years to work full time on building an open-source platform and a way to automatically generate the structural component from existing wikis, run out of money, and then work out what to do next... :stuck_out_tongue:
This is more or less what I was hinting at with the grant indeed. You need that kind of money and that kind of time window to make an endeavour like this happen, I believe.
Jules Hedges (Apr 15 2020 at 13:28):Huh. I assumed the reason Compositionality was slow was that not enough people submitted there. The first issue with 4 papers did come out a while ago. I have a bunch of stuff in mind to submit there still...
Jules Hedges (Apr 15 2020 at 13:32):I also kinda wish they (by which I really mean the executive board, which is Brendan, Nina and Josh) would communicate more regularly about it.... it's supposed to be a community journal, we do get updates once a year at ACT but that seems like a very 20th century method of community engagement
(=_=) (Apr 15 2020 at 13:44):Morgan Rogers said:
The conclusion is just "we should use computational notebooks", which I guess is an upgrade on pdf in some respects but it's hardly a massive leap of creativity away from the format the writer is claiming is obsolete.
This is actually quite meaningful in CS, because you can run code from within your document. When we eventually gain the ability to run proofs on our computers, the notebook would probably become more compelling as a format. If you look at the "comics" version of Watts and Strogatz that I referred to above, you can interact with the diagrams, which allows you to better appreciate the dynamics of the 'small-world' networks that they're studying in the paper.
Jules Hedges (Apr 15 2020 at 13:48):I did read a couple of HoTT papers at some point that were just literate Agda files. But the typography of code is eye meltingly ugly compared to even bad LaTeX
(=_=) (Apr 15 2020 at 13:49):Morgan Rogers said:
I imagine a network structure, a virtual map of a subject, with nodes linking to, say, entries in a wiki. At a finer level, one could construct networks based on the results presented in a given textbooks, expanding them with results that have been discovered since the book's publication.
It's actually quite gratifying to see that there is convergence in the kind of ideas that people come up with when they think about how to redesign the format of scientific publishing.
(=_=) (Apr 15 2020 at 13:51):Jules Hedges said:
I did read a couple of HoTT papers at some point that were just literate Agda files. But the typography of code is eye meltingly ugly compared to even bad LaTeX
Well, people do love the typewriter/Courier font. It makes the code "look like code"... :sweat_smile:
(=_=) (Apr 15 2020 at 13:53):Morgan Rogers said:
(PS: You can probably tell from reading the above that making this idea a reality is a dream of mine, despite my limited programming experience. If I leave academia at any point, this is my backup plan.)
Me too. :raised_hand:
(=_=) (Apr 15 2020 at 13:59):Morgan Rogers said:
A lot of research goes into visualisation of knowledge every year (I should put a source here but I need to get back to "real work"). The problem is that it's specialised: one has to learn to use the tools that are being constructed, and that's a big enough obstacle that only a tiny fraction of it reaches mainstream.
Not to mention that it's probably also going to be proprietary. For example, Web of Science has a feature called a "citation map", which allows you to see both the papers citing a particular paper (something available from Google Scholar) and the papers cited by that particular paper (which is not a feature that's publicly available anywhere else AFAIK). This is your:
- A way to keep track of updates to older publications, and see how they connect to more recent research.
Unfortunately, this feature is proprietary, very clunky, and allows you to go backwards or forwards for a meagre two steps.
(=_=) (Apr 15 2020 at 14:10):Now, let's go through your desiderata in more detail.
Morgan Rogers said:
- Potential to gain credit for regular small contributions to a project. Being part of a community wiki is great, but actually being able to see how much someone has contributed to a network visually could be a lot more persuasive than a number like reputation.
This is actually quite doable. I've mentioned something like a Github equivalent for maths. What the arXiv is, currently, is a repository system with version control, which is also what Github has. What arXiv is missing, however, is precisely what you're asking for here.
Github has all these:
So all you need to do is to add these Github features onto the arXiv. Simples. :sweat_smile:
Reid Barton (Apr 15 2020 at 14:14)::this: This is the only thing on the list I actually want regularly. I mean I could compile a list of errata and send them to the author, but now they face the problem of what to do with it.
(=_=) (Apr 15 2020 at 14:14):
- Eventually, a breakdown of long, intimidating publications into pieces made manageable by their connections to other resources.
You need better NLP and more code-like or modular mathematical writing. However, modularity isn't well supported under the current system, because you want to bundle results together to form a "publishable" and "substantial" paper. It may work once you have arXiv+Github in place.
Morgan Rogers (he/him) (Apr 15 2020 at 14:14):Rongmin Lu said:
Web of Science has a feature called a "citation map", which allows you to see both the papers citing a particular paper (something available from Google Scholar) and the papers cited by that particular paper (which is not a feature that's publicly available anywhere else AFAIK). This is your:
- A way to keep track of updates to older publications, and see how they connect to more recent research.
Unfortunately, this feature is proprietary, very clunky, and allows you to go backwards or forwards for a meagre two steps.
Right, this is the "content ignorant" version of that feature. I'm definitely looking forward to something more sophisticated, since in a bunch of fields (my most recent experience of this was in a signal processing course) there are one or two seminal papers which almost every paper in the field ends up citing in their background sections, so the resulting citation graph wouldn't be quite as illuminating.
(=_=) (Apr 15 2020 at 14:15):Reid Barton said:
:this: This is the only thing on the list I actually want regularly. I mean I could compile a list of errata and send them to the author, but now they face the problem of what to do with it.
Exactly. Also, your communication isn't public knowledge, and the author(s) can do whatever the heck they want with your contribution.
(=_=) (Apr 15 2020 at 14:17):Morgan Rogers said:
I'm definitely looking forward to something more sophisticated, since in a bunch of fields (my most recent experience of this was in a signal processing course) there are one or two seminal papers which almost every paper in the field ends up citing in their background sections, so the resulting citation graph wouldn't be quite as illuminating.
On the other hand, it does show you, roughly, which papers belong in that field.
(=_=) (Apr 15 2020 at 14:22):
A visual way to understand the structure of a subject, which immediately makes visible the cutting edge and open questions which normally are only known to specialised researchers. (it wouldn't give immediate comprehension of the statements of these problems; just where they lie in the grand scheme of things)
A direct visual link between abstraction and concreteness.
These two require a lot more NLP than what I think is currently available:
It can be done, but it'd be a multi-year research project. You'd probably need more DisCoCats, QNLP and whatever Bob Coecke and others can throw at it.
Morgan Rogers (he/him) (Apr 15 2020 at 14:36):I'm not asking for the open questions to be labelled in an automated way, they'll be identifiable either from the open ends of the network or the areas where active research is currently happening (another aspect that I haven't yet mentioned). Similarly, abstraction and generalisation will be apparent from connectivity between areas that were previously separate.
Ultimately my intention is that only the initial structural scaffolding needs to be automatically generated so that we don't have to build it from scratch; after that it will be added to manually, hopefully as easily as people are able to upload preprints to arXiv or push to Git today. The point of this system is not to automate research; I don't need a machine to do any of the things that you're suggesting, although I expect that others might like that possibility. Its purpose is to illuminate the structure of research for researchers, and for the general public.
(=_=) (Apr 15 2020 at 14:47):Morgan Rogers said:
I'm not asking for the open questions to be labelled in an automated way
I didn't suggest that they would be.
they'll be identifiable either from the open ends of the network or the areas where active research is currently happening (another aspect that I haven't yet mentioned).
Define "open". I imagine you might be thinking along the lines of "most recent", but many open questions are actually buried in highly cited papers.
Define "areas where active research is currently happening" as well.
If your network is actually a network of concepts, which is what it sounds like, you need NLP.
Similarly, abstraction and generalisation will be apparent from connectivity between areas that were previously separate.
That's a connection between two fields. Elliptic curves and modular forms were previously separate fields, but then people found some connectivity between the two. It's not really a generalisation, more the realisation that there's a bridge. Passing from Taylor-Wiles to the full modularity theorem is a generalisation, but it wouldn't appear as an emergent connectivity.
Ultimately my intention is that only the initial structural scaffolding needs to be automatically generated so that we don't have to build it from scratch; after that it will be added to manually, hopefully as easily as people are able to upload preprints to arXiv or push to Git today.
You need NLP for that "structural scaffolding".
It's not as bad as it sounds, really. That's something that the NLP side of ACT can work on, if they're keen. There is prior art: I'll elaborate in another comment.
(=_=) (Apr 15 2020 at 14:53):Morgan Rogers said:
- Connections between researchers who didn't realise that they were working in parallel on the same thing due to different naming conventions etc.
You'd be surprised at how often people independently come up with the same thing. And sometimes they use the same naming convention! It's crazy, I know.
Story time: I encountered this very problem with my thesis, and it was a really fun time.
So I had come up with this really weird gadget that my advisor had an idea for, based on something that someone had told him about in a conference. Now I was pretty sure I wasn't that smart to have been the first one to come up with it, so I took to the Googles and search the heck out of it. No dice.
There was a tiny problem: the name that I had come up with had a Greek letter in it, and for the life of me, I couldn't figure out how to get Google to accept the Greek letter as a keyword, which I was absolutely positive that anyone who came up with the same thing would have used.
Long story short, I didn't manage to figure out that someone else had already come up with the same thing before I submitted. Thankfully, nobody else knew about it because it was in a completely different field, but it took me a few years before I discovered the probable source of the notion that my advisor had heard.
As it turned out, two other people had come up with the same thing, using the same naming convention, but in wildly different fields! So yeah, point 3 here is a major problem.
(=_=) (Apr 15 2020 at 15:11):Rongmin Lu said:
That's something that the NLP side of ACT can work on, if they're keen. There is prior art: I'll elaborate in another comment.
So I promised I'd talk about the prior art I know of in this problem.
The problem is being able to process the natural-language material found in maths papers into some sort of network of concepts. One prerequisite for that is to be able to parse mathematics written in a natural language. Some of it has been done before.
Back in 2013 --- 2012/13 was pretty hectic in maths --- Tim Gowers ran a curious experiment on his blog and posted the results here. It turned out to be a demo of some software written by Mohan Ganesalingam to test that his grammar of mathematics, which he developed in his thesis, actually works. They eventually published the results here.
The point isn't that we can now look forward to automated theorem provers writing really convincing natural-language proofs. Rather, it's that there has been a prototype that can generate natural-language proofs, which presumably means it may also "understand" natural-language proofs, in such a way that it may be possible to extract concepts and do something useful with them.
Morgan Rogers (he/him) (Apr 15 2020 at 15:12):I'm not asking for the open questions to be labelled in an automated way
I didn't suggest that they would be.
Umm... then what did you mean by:
How will the machine figure out what the "open questions" are?
..?
Define "open". I imagine you might be thinking along the lines of "most recent", but many open questions are actually buried in highly cited papers. Define "areas where active research is currently happening" as well.
In "open problems" I mean those problems which have not yet been resolved. The system I described would enable people to identify when solutions to problems posed in a given text have later been resolved, as well as what people are currently working on (which is what I meant by "areas where active research is currently happening", I don't understand how that description was unclear...), either by users of this system actively signalling that or by viewing recent activity in the area of this network one is interested in.
Morgan Rogers (he/him) (Apr 15 2020 at 15:16):If your network is actually a network of concepts, which is what it sounds like, you need NLP.
When you say NLP, I presume (based on the anecdote you mentioned) you mean implemented in a computer, in which case, beyond constructing the basic scaffolding I mentioned, I don't see why you think I need it. I can draw one of these networks manually, and as I said I expect it to be added to manually. NLP would make this easier, for sure, since the system could suggest links, but I wouldn't say it's essential.
(=_=) (Apr 15 2020 at 15:17):Morgan Rogers said:
In "open problems" I mean those problems which have not yet been resolved. The system I described would enable people to identify when solutions to problems posed in a given text have later been resolved, as well as what people are currently working on (which is what I meant by "areas where active research is currently happening", I don't understand how that description was unclear...), either by users of this system actively signalling that or by viewing recent activity in the area of this network one is interested in.
All of this requires natural language processing (NLP). Unless your system is going to be manual, which is then the status quo, it will require more NLP than what we can currently do at the moment, as far as I'm aware.
The description was very clear... to a human. It's also very clear that you can't have a non-manual system to do that without better NLP.
I mean, I'm not trying to be pessimistic here. I'm actually excited. This is a lot of work that ACT can do. And if you can push NLP to this point, it's likely you can have commercial spin-offs.
(=_=) (Apr 15 2020 at 15:19):Morgan Rogers said:
When you say NLP, I presume (based on the anecdote you mentioned) you mean implemented in a computer, in which case, beyond constructing the basic scaffolding I mentioned, I don't see why you think I need it. I can draw one of these networks manually, and as I said I expect it to be added to manually. NLP would make this easier, for sure, since the system could suggest links, but I wouldn't say it's essential.
Then that is no different from the status quo, isn't it? The way you've previously described it requires automation. If you plan on having a manual system, you just need a board and some writing implements.
(=_=) (Apr 15 2020 at 15:21):Morgan Rogers said:
The plan is: solve a Millennium Problem, claim the prize
That's a very inefficient way to raise $1m. You might as well work on developing NLP further and then build startups to commercialise the technology.
Joachim Kock (Apr 15 2020 at 15:24):Fabrizio Genovese said:
I see a similar pattern with Compositionality.
Overall it is my impression that Compositionality runs well, at the speed it can. The high expectations about much faster refereeing than any other journal are difficult to meet, as the refereeing process depends on the same referees as the other journals :-(
(It is not really up to me to make judgments about the virtues of Compositionality, but I thought Fabrizio's 'pattern matching' should not stand uncontested.)
(=_=) (Apr 15 2020 at 15:26):Morgan Rogers said:
I can draw one of these networks manually, and as I said I expect it to be added to manually. NLP would make this easier, for sure, since the system could suggest links, but I wouldn't say it's essential.
I can also draw a citation map manually. Why would I need the Web of Science's citation map then? Why did people waste their time coding up that feature, when "clearly" people don't need it?
Morgan Rogers (he/him) (Apr 15 2020 at 15:28):Rongmin Lu said:
Morgan Rogers said:
The plan is: solve a Millennium Problem, claim the prize,
That's a very inefficient way to raise $1m. You might as well work on developing NLP further and then build startups to commercialise the technology.
:face_palm: jokes are a thing
Rongmin Lu said:
Then this is no different from the status quo, isn't it? The way you've previously described it requires automation. If you plan on having a manual system, you just need a board and some writing implements.
No, very much not! I can't currently see the structure I've described, and it certainly isn't central to the system. The automated aspect is just to get the system up to speed with existing research, so that people don't have to manually reproduce work they've done before. After that people would add their work directly into that structure as it relates to it, rather than being constrained to a long-form, linearly-structured format.
(=_=) (Apr 15 2020 at 15:32):Morgan Rogers said:
The automated aspect is just to get the system up to speed with existing research, so that people don't have to manually reproduce work they've done before.
This requires NLP. It seems I cannot reiterate that enough.
If you need the system to get "up to speed" with existing research, which is written in natural language, you need a way for the system to process it, ergo Natural Language Processing (NLP).
Guess who else can do NLP? Humans. But then that would be a manual system with no automated aspect.
Morgan Rogers (he/him) (Apr 15 2020 at 15:39):Rongmin Lu said:
The description was very clear... to a human. It's also very clear that you can't have a non-manual system to do that without better NLP.
There's some tension between us here, and I think this might be the source of it. The purpose of research papers is to communicate research to humans. The system I described above aims to do exactly this: make the structure of research more visible to people. I frankly don't care how able a computer is to read my research papers, or whether it can analyse the resulting network in order to identify the open problems; I care about helping people to do these things and avoiding exactly the kind of anecdote you shared above, for example.
As you've pointed out, the technology already exists to produce citation trees; it's not perfect, but that's a sufficient level of NLP to produce the initial scaffolding that we've been talking about. Anything further is a bonus, for the same reasons that improved NLP gives better search engine results, but it's not essential to get the system up and running.
Morgan Rogers (he/him) (Apr 15 2020 at 15:41):After all, this is a discussion of the format in which research is shared, not about attempts to automate aspects of it.
Todd Schmid (he/they) (Apr 15 2020 at 15:42):@Jules Hedges (In response to your link) Wow, I love this. It would be super nice to see how something like this could be impemented for more pure-mathematical work. I can definitely imagine myself playing with virtual knots and braids in the middle of the research-level paper where they're used as examples, or hitting the buttons on a labelled Markov process... The design is just a little too web-modern for my aesthetic taste, but that's of course not a real criticism!
John Baez (Apr 15 2020 at 22:40):Fabrizio Genovese said:
I mean, I see a similar pattern with Compositionality. I submitted there ages ago, got reviews, made the requested edits, and never heard from them again. But I don't blame them.
You should raise hell! That's ridiculous! I'm on the advisory board. Is your article still in limbo?
John Baez (Apr 15 2020 at 22:41):Jules Hedges said:
I also kinda wish they (by which I really mean the executive board, which is Brendan, Nina and Josh) would communicate more regularly about it....
Communicate to whom? The whole world? You? I can make them communicate a bit more if you tell me what needs to be communicated, to whom.
Fabrizio Genovese (Apr 15 2020 at 22:46):John Baez said:
Fabrizio Genovese said:
I mean, I see a similar pattern with Compositionality. I submitted there ages ago, got reviews, made the requested edits, and never heard from them again. But I don't blame them.
You should raise hell! That's ridiculous! I'm on the advisory board. Is your article still in limbo?
I wouldn't define it "limbo". They asked me to resubmit with corrections, so I guess they more or less liked the paper. They gave me a very long and detailed list of things to revise/change, which I appreciated, clearly reviews were well-done. This happened last august I believe. I managed to implement the corrections in December, and resubmitted including a file where I said, for each correction, if I had implemented it and, if I didn't, why I chose to do that. I didn't hear back from them ever since. Then today, exactly because of this conversation, I sent a mail to them asking for news. Dan Ghica almost instantly replied to me that reviewers are reviewing the corrections, and that the process "should be swift" ...I hope he's right!
Fabrizio Genovese (Apr 15 2020 at 22:48):In any case yes, maybe I should have pressed them a bit more. But then again, it's people that work on a volounteering basis, so even if the process is slow I still perceive it as a gift more than as a right... Maybe I'm wrong, but I feel that way!
(=_=) (Apr 16 2020 at 14:59):Morgan Rogers said:
I frankly don't care how able a computer is to read my research papers
Yeah, and therein lies the "tension".
I've been trying to tell you that your system requires an incredible amount of effort to do what you've specified.
The effort required isn't on the scale of thousands of man-hours, as Fab had very optimistically estimated. It's on the level of millions of man-hours: that's about a decade of work by 100s of people working full-time, basically a full-blown research program.
The purpose of research papers is to communicate research to humans. The system I described above aims to do exactly this: make the structure of research more visible to people.
Google Scholar helps to make research more available to humans as well. It took an incredible amount of effort to get to doing what it can do. And that's nowhere near the kind of system that you were specifying. Nowhere.
I care about helping people to do these things and avoiding exactly the kind of anecdote you shared above, for example.
Yes, and I care too.
But what I also care about is that there's still a ton of work to be done before you can achieve most of the features you've specified.
As you've pointed out, the technology already exists to produce citation trees; it's not perfect, but that's a sufficient level of NLP to produce the initial scaffolding that we've been talking about. Anything further is a bonus, for the same reasons that improved NLP gives better search engine results, but it's not essential to get the system up and running.
It will not achieve ANY of the features you've specified, and hence, it won't be able to help people the way you had envisioned your system would do. That was my point.
Antonin Delpeuch (Apr 16 2020 at 15:27):I think it's okay to have pub discussions (or the digital equivalent) about things like this, I would not lecture people about handwaving and work.
Morgan Rogers (he/him) (Apr 16 2020 at 15:30):I was sketching my vision of what I would like an alternative to scientific papers to look like, and we just happen to disagree about what that would involve to implement :shrug:. In order to upload a preprint, you go to arXiv and upload it. In the most basic version of the system I was trying to describe you add a node to a network instead; this idea, as stated, does not require computer-implemented NLP.
I agreed that some automation would be needed to set up the initial network, but the reason I would be satisfied with the existing NLP models for this task is that it wouldn't be the final network; this would just provide an initial geometry for the network, to be subsequently modified by the users of said network. Perhaps it was optimistic to hope that my suggestion 5 would ever be feasible manually, but otherwise the mere existence of a visible network of this kind with active users seems to me to achieve all of the points I gave.
Fabrizio Genovese (Apr 16 2020 at 16:09):I know it seems silly, but even "adding a node on the network" requires hundreds of man-hours. Let's take the simplest example: arXiv. According to your desiderata, arXiv does nothing, it's just the super-bare-minimum infrastructure.
Fabrizio Genovese (Apr 16 2020 at 16:09):It still required thousands of man-hours to be implemented, and there is a full time team working on it even now
(=_=) (Apr 16 2020 at 16:10):Antonin Delpeuch said:
I would not lecture people about handwaving and work.
It is extremely frustrating to outline what I had believed to be an interesting program, only to have it being swatted away as if it were a triviality.
Fabrizio Genovese (Apr 16 2020 at 16:10):I didn't even dare to venture into NLP stuff because that's absolutely outside of our current possibilities. And I don't mean just technically, even economically!
Fabrizio Genovese (Apr 16 2020 at 16:11):Neural networks require an insane amount of computational power to run and to be trained. So even if we were to implement such a thing, we basically couldn't afford to run it without some big ass grant. :D
(=_=) (Apr 16 2020 at 16:11):Fabrizio Genovese said:
I know it seems silly, but even "adding a node on the network" requires hundreds of man-hours. Let's take the simplest example: arXiv. According to your desiderata, arXiv does nothing, it's just the super-bare-minimum infrastructure.
Thanks, Fabrizio!
Fabrizio Genovese (Apr 16 2020 at 16:12):I understand that many people want this kind of stuff, or want to help to set it up. I want it too. But let's adopt a super-duper rational approach to the problem:
(=_=) (Apr 16 2020 at 16:13):Fabrizio Genovese said:
Neural networks require an insane amount of computational power to run and to be trained. So even if we were to implement such a thing, we basically couldn't afford to run it without some big ass grant. :D
There is some research being done on binarised neural networks that might reduce that computational requirement. People really want some deep learning power on edge devices.
Fabrizio Genovese (Apr 16 2020 at 16:14):What we have plenty:
| - Ideas
What we have little:
|- man hours
|- money
Really, the best way to make something like this happen is:
Morgan Rogers (he/him) (Apr 16 2020 at 16:15):Fabrizio Genovese said:
I know it seems silly, but even "adding a node on the network" requires hundreds of man-hours. Let's take the simplest example: arXiv. According to your desiderata, arXiv does nothing, it's just the super-bare-minimum infrastructure.
Of course it's going to take a lot of man-hours, I was only objecting to the strict need for those hours to be invested into NLP for the concept I had outlined.
(=_=) (Apr 16 2020 at 16:16):I've outlined another dot point: see if the CT we know can help with developing the tech. I'm pretty sure Bob Coecke's school of QNLP can help.
Fabrizio Genovese (Apr 16 2020 at 16:16):Again, I really do not want to sound hostile, I am not and I appreciate this idea. But it's just enriching a bucket full of resources of which we already have plenty. This kind of discussion is a bit of a trap: It's very interesting for everyone on a superficial level, but it's really very difficult to let it take off from there. This is why I said above that I find this discussions not useful anymore. In the end, it all runs down to wanting to do something and not having resources to do it, and hence, sadly, it ends up being more whishful thinking than anything else :frown:
Fabrizio Genovese (Apr 16 2020 at 16:17):So if I can try to let this stir to a more productive level, I'd suggest this:
(=_=) (Apr 16 2020 at 16:17):Morgan Rogers said:
Of course it's going to take a lot of man-hours, I was only objecting to the strict need for those hours to be invested into NLP for the concept I had outlined.
Google Scholar can be implemented with a team of research librarians and a card catalogue. Was there a strict need to develop all that tech just to search for keywords?
Morgan Rogers (he/him) (Apr 16 2020 at 16:19):Rongmin Lu said:
Antonin Delpeuch said:
I would not lecture people about handwaving and work.
It is extremely frustrating to outline what I had believed to be an interesting program, only to have it being swatted away as if it were a triviality.
It's great that you have your own idea about the way to improve scientific communication and your own idea of what it will take to get there, @Rongmin Lu. However, ranting at people and dismissing their ideas because they think differently from you, which is how you introduced your interesting NLP programme, is not the way to convince anyone of your point of view. You've been responsible for too many unnecessarily heated arguments already on this platform, please be more considerate.
(=_=) (Apr 16 2020 at 16:20):Fabrizio Genovese said:
In the end, it all runs down to wanting to do something and not having resources to do it, and hence, sadly, it ends up being more whishful thinking than anything else :(
You have the resources to do some of what I've proposed. For example, I've posted a link to the abstracts of the fifth conference on AI and Theorem Proving. People can take a look at them and see if they can apply CT there. Or at least that's what I thought ACT was about.
(=_=) (Apr 16 2020 at 16:22):Morgan Rogers said:
However, ranting at people and dismissing their ideas because they think differently from you
I respected your ideas, and elaborated on them by analysing what's required to realise them. You've dismissed my ideas by calling it "ranting".
Gershom (Apr 16 2020 at 16:26):I would like to interrupt whatever this discussion is to note that there is _already_ a well-funded and exciting and potentially transformative project underway to improve a key aspect of mathematical communication: Hales' Formal Abstracts Project: https://formalabstracts.github.io/
It has a vision of the tech necessary, and also the social changes necessary to get widespread adoption of formalized representation not of proofs, but of the dependencies, definitions, and _claimed theorems_ within a wide range of mathematical papers, in such a way that a universe of tools can be written surrounding such things.
Morgan Rogers (he/him) (Apr 16 2020 at 16:34):OH MY GOODNESS THAT'S WONDERFUL
Morgan Rogers (he/him) (Apr 16 2020 at 16:35):"We foresee tools for exploration such as a Google Earth for mathematics, providing an intuitive visual map of the entire world of mathematics, combining formal abstracts, computation, and other networked content – all displayed at user-selected resolution."
(=_=) (Apr 16 2020 at 16:50):For anyone who's interested in some history, Google Earth was launched in 2001, while "[t]he core technology behind Google Earth was originally developed at Intrinsic Graphics in the late 1990s." That technology is built from a wide range of technologies, including 3D graphics and GIS.
(=_=) (Apr 16 2020 at 16:59):Gershom said:
there is _already_ a well-funded and exciting and potentially transformative project underway to improve a key aspect of mathematical communication: Hales' Formal Abstracts Project: https://formalabstracts.github.io/
Thanks! I've forgotten about that, but it sounded like an exciting project when I first heard of it. However, it's stated there that
"We are not actively seeking contributions at this time."
Would you happen to know more about this?
Pastel Raschke (Apr 19 2020 at 13:00):github as a model has a big problem: issues, pull requests, and forks are not part of the git repository, but proprietary to the service. gitlab and other git hosting services have their own implementations of these features, and while i believe there is decent transport between these services (developers would riot otherwise), ideally you would want issues and prs to be versioned and stored alongside a repository (i bet there are extensions for this, but they might not interface natively with a hosting service).
i wouldn't be surprised if these were implemented as either separate repositories or extensions to the hosted repository on the backend, that are nevertheless not sent when you clone a github repo. i imagine communication and coordination workflow was not included in the repository because it didn't fit into the use of git as versioning a specific directory, and git does not support partial checkouts/fetches very well.
re wikis, zettelkasten and roam are very interesting as potential organizational infrastructure. roam is proprietary and cloud-hosted, but their ideas are certainly not chained to the trademark.
Henry Story (Apr 19 2020 at 13:04):To build a decentralized version of Github with decentralized bug features you'd need all the data behind the issue databases to be published as LinkedData and to write Apps that can read those too. Most of the standards to do that exist. Essentially Tim Berners-Lee's Solid project is about enabling such use cases.
Henry Story (Apr 19 2020 at 13:08):On the topic of redesigning the scientific paper, you may be interested in this recently published thesis Linked Research on the Decentralised Web
(=_=) (Apr 19 2020 at 14:24):Pastel Raschke said:
github as a model has a big problem: issues, pull requests, and forks are not part of the git repository, but proprietary to the service.
I did not know that! Thank you for bringing this up.
i wouldn't be surprised if these were implemented as either separate repositories or extensions to the hosted repository on the backend, that are nevertheless not sent when you clone a github repo. i imagine communication and coordination workflow was not included in the repository because it didn't fit into the use of git as versioning a specific directory, and git does not support partial checkouts/fetches very well.
The idea I'd proposed was to layer the non-git features of git hosting services onto arXiv, which has some version control built in. Unfortunately, what you've brought up seems to suggest that this idea would suffer from the same problems as the existing git hosting services, unless this extension is developed as an integral part of arXiv.
Daniel Geisler (Apr 19 2020 at 15:03):How about MediaWiki like Wikipedia uses? I use SiteGround which has great support. I can pull a wiki together with math type support in an hour. MediaWiki has built in version control with a diff utility. I'm updating my personal math website to use MediaWiki.
Daniel Geisler (Apr 19 2020 at 18:46):FYI - the calendar now lives in a MediaWiki website at http://categorytheory.world .
Alexis Hazell (Apr 20 2020 at 01:52):One alternative to GitHub, which I'm enthusiastic about, though it's currently in alpha: Sourcehut, https://sourcehut.org/. It provides an example of a project oriented towards data freedom and trying to avoid proprietary lock-in.
(=_=) (Apr 21 2020 at 06:43):I've just come across Damiano Mazza's 2012 proposal, in which he called for "open access, interactive paper repositories". His characterisation of the system is arXiv+EasyChair.
Jules Hedges (Apr 21 2020 at 10:21):I was thinking with this there's broadly 2 different approaches that are at risk of talking across each other [why is it always me that recognises that kind of situation?]. Broadly they are about human-readable vs machine-readable metadata. Both reasonable, but different. (The second one is hard, on the basis that semantic web is hard)
Morgan Rogers (he/him) (Apr 21 2020 at 11:06):That's more or less the argument that Rongmin and I had above :rolling_on_the_floor_laughing:
But Rongmin made me conscious of the fact that the sheer volume of existing research is such that automated machine-readability is necessary in order for the revolution to happen. Without it, the foundation off which people are supposed to build would need to be produced manually, which is not only a mammoth task but one which any person or community would struggle to handle in a systematic, reliable way.
Morgan Rogers (he/him) (Apr 21 2020 at 11:07):If people have to frequently reference resources that aren't integrated into a new system, then as the old system of preprints gets retired we'll end up with the same problems of missing refs as we do today due to people referencing unpublished work or folklore.
(=_=) (Apr 21 2020 at 13:10):Jules Hedges said:
Broadly they are about human-readable vs machine-readable metadata. Both reasonable, but different. (The second one is hard, on the basis that semantic web is hard)
As you've said, creating machine-readable metadata is hard like the semantic web is hard. It requires quite a bit of hand-crafting and a lot of manual labelling.
So my point was basically that we should learn to process the human-readable data we already have directly and automatically, i.e. develop better natural language processing (NLP) technology. The state of the art in NLP now is better than when Ganesalingam wrote his thesis, and there have been steps taken towards processing mathematical text as well.
Where I think ACT can come in is to provide a better framework for the engineers to design better. Right now, to use an analogy with engineering and physics, people know how to build bridges that wouldn't collapse when someone walks on it, but there's no theory of mechanics yet to guide the engineers on how to build better bridges.
There are embryonic theories, like Bob Coecke's DisCoCats(Circs) and the work sparked by Backprop as Functor. My hope is that these could blossom into some coherent theory that would help guide future work in ML/NLP.
Reid Barton (Apr 21 2020 at 15:09):Some of the issues being discussed here are addressed by Gerby, the software behind the Stacks project and Kerodon. In particular, the contents of the document can be extended over time and there is a comment system. Gerby also solves the problem of how someone else can refer to a specific item (like a proposition) in a document that is evolving.
(=_=) (Apr 21 2020 at 15:38):Reid Barton said:
Some of the issues being discussed here are addressed by Gerby, the software behind the Stacks project and Kerodon. In particular, the contents of the document can be extended over time and there is a comment system. Gerby also solves the problem of how someone else can refer to a specific item (like a proposition) in a document that is evolving.
Cool. It does involve generating metadata manually, which is a partial solution.
(=_=) (Apr 21 2020 at 15:44):@David Michael Roberts, have you used Gerby? Says here:
In case you were wondering, a gerbe is a kind of stack (in the mathematical sense), and the software was originally meant for the Stacks project.
:sweat_smile:
Valeria de Paiva (Apr 21 2020 at 17:15):Rongmin Lu said:
I've just come across Damiano Mazza's 2012 proposal, in which he called for "open access, interactive paper repositories". His characterisation of the system is arXiv+EasyChair.
well the guys in NLP/ML have this system running already at https://openreview.net/about
David Michael Roberts (Apr 21 2020 at 22:12):@Rongmin Lu no, just gerbes. :-)
(=_=) (Apr 22 2020 at 02:48):Valeria de Paiva said:
well the guys in NLP/ML have this system running already at https://openreview.net/about
Yup, I mentioned OpenReivew in an earlier topic.
Nikolaj Kuntner (May 29 2020 at 13:22):Rongmin Lu said:
I just stumbled upon this 2-year-old article about possible re-designs of the scientific paper:
https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/
What I don't liked is that the author seemingly oversees that if you were to turn papers into computing notebooks, that would mean scientists would have to decide on a language. The "what programming language is the best" does not have an answer, as we know from 80 years of computing.
And I don't see "too formal" as an issue. It depends on the subject.
And I agree with Morgan Rogers point that I was hoping for something more radical going in.
In fact, the article leaves me with a discouraged feel in this regard, since the presentation and biographical details makes one conclude that to change something, it needs a highjacking of ones own career and a tangent of 20 years. Probably what Fabrizio is also getting at later.
Morgan Rogers said:
(PS: You can probably tell from reading the above that making this idea a reality is a dream of mine, despite my limited programming experience. If I leave academia at any point, this is my backup plan.)
What exactly is the plan here? :D
(=_=) (May 29 2020 at 13:26):Oh gosh, not this thread, please... :sweat_smile:
Nikolaj Kuntner said:
What I don't liked is that the author seemingly oversees that if you were to turn papers into computing notebooks, that would mean scientists would have to decide on a language. The "what programming language is the best" does not have an answer, as we know from 80 years of computing.
Eh. Popular science writers.
In fact, the article leaves me with a discouraged feel in this regard, since the presentation and biographical details makes one conclude that to change something, it needs a highjacking of ones own career and a tangent of 20 years.
It does. Unless you want to devote your career to effecting change, it's not going to happen.
(=_=) (May 29 2020 at 13:28):Nikolaj Kuntner said:
Morgan Rogers said:
(PS: You can probably tell from reading the above that making this idea a reality is a dream of mine, despite my limited programming experience. If I leave academia at any point, this is my backup plan.)
What exactly is the plan here? :D
Trying to take over the world, clearly. :laughing:
If you read the thread from there, you'll understand. That's my dream as well, but clearly we had different interpretations of what that dream entails.
Nikolaj Kuntner (May 29 2020 at 13:28):@Fabrizio Genovese Thanks for sharing the "London story" about the failed attempt.
Nikolaj Kuntner (May 29 2020 at 13:29):@Rongmin Lu What hinders from directly using github as the scientific document repository?
Nikolaj Kuntner (May 29 2020 at 13:29):Why not this thread, what do you mean? :sweat_smile:
Nikolaj Kuntner (May 29 2020 at 13:30):So who of you is gonna be the Elon Musk of scientific publishing?!
(=_=) (May 29 2020 at 13:31):Nikolaj Kuntner said:
Rongmin Lu What hinders from directly using github as the scientific document repository?
Currently, Github only accepts text files. This means having to upload .tex files, but we usually read in .pdf.
However, we already have the arXiv as a document repository. The thing is, we'd like it to do more. Much more.
(=_=) (May 29 2020 at 13:31):Nikolaj Kuntner said:
So who of you is gonna be the Elon Musk of scientific publishing?!
Probably Morgan. :wink:
Alexis Hazell (May 29 2020 at 13:40):Rongmin Lu said:
Currently, Github only accepts text files. This means having to upload .tex files, but we usually read in .pdf.
Well, GitHub (and Git more generally) can store binary blobs such as PDFs, but I presume you're talking about being able to track changes in PDF contents?
Jules Hedges (May 29 2020 at 13:46):I'm totally on board with "kill the paper". Pdf doesn't support even basic things that readers might want to do like changing the font size and margins, let alone complicated things like rotating a 3d visualisation or interactively changing parameters of a model
Simon Burton (May 29 2020 at 13:47):Did anyone mention distil.pub yet? For example, https://distill.pub/2020/bayesian-optimization/
Oliver Shetler (May 29 2020 at 14:35):@Morgan Rogers I'm not sure if this is of interest to you, but have you checked out Douglas Walton's books Methods of Argumentation? And Argumentation Schemes?
One place where scientific papers could stand to improve is on how they present their arguments. Including author-coded metadata on what kinds of arguments scientists intend to make could drastically improve the ease and quality of lit reviews and meta analyses. Plus, it has the advantage of being an add-on rather than an overhaul of existing formats.
(=_=) (May 29 2020 at 14:47):Alexis Hazell said:
Rongmin Lu said:
Currently, Github only accepts text files. This means having to upload .tex files, but we usually read in .pdf.
Well, GitHub (and Git more generally) can store binary blobs such as PDFs, but I presume you're talking about being able to track changes in PDF contents?
Yes... ack. That sounds so wrong now that I'm reading it again. :sweat_smile:
But it's also that I don't think people are comfortable uploading .tex files for everyone to see in all their glory. The arXiv certainly does a good job of hiding .tex files.
(=_=) (May 29 2020 at 14:51):Jules Hedges said:
I'm totally on board with "kill the paper". Pdf doesn't support even basic things that readers might want to do like changing the font size and margins, let alone complicated things like rotating a 3d visualisation or interactively changing parameters of a model
The PDF format is a dumpster fire. Unfortunately, it's still the most popular platform-neutral document format. What you're suggesting is best implemented as a web document, but even then, I'd just print to PDF so I can read it offline.
Nikolaj Kuntner (May 29 2020 at 14:53):I've also heard critical voices of TeX as such.
(=_=) (May 29 2020 at 14:55):Nikolaj Kuntner said:
I've also heard critical voices of TeX as such.
Yup, yup. needs to die too, pace Donald Knuth. Same problem as PDF though: too big to fail.
(=_=) (May 29 2020 at 15:02):Oliver Shetler said:
I'm not sure if this is of interest to you, but have you checked out Douglas Walton's books Methods of Argumentation? And Argumentation Schemes?
Thanks! This is nice. I found some pictures too.
Including author-coded metadata on what kinds of arguments scientists intend to make could drastically improve the ease and quality of lit reviews and meta analyses. Plus, it has the advantage of being an add-on rather than an overhaul of existing formats.
I'm not in favour of hand-coded anything. And it wouldn't be just an add-on if authors are required to code metadata, it'd just become another format. Relevant xkcd.
(=_=) (May 29 2020 at 15:07):Simon Burton said:
Did anyone mention distil.pub yet? For example, https://distill.pub/2020/bayesian-optimization/
Apparently not, but I'm aware of it.
T Murrills (May 29 2020 at 15:08):This doesn’t count as a ”redesign of the scientific paper” but nonetheless seems relevant here: I’m thinking about document creation/structure, and the interface we use to do it. One thing I’ve been surprised at is how far behind being a “good, intuitive text editor for math” most LaTeX editors are. For a while I’ve used TeXmacs ( https://www.texmacs.org ), which, while it allows LaTeX-like input of math symbols, is a different document structure built (at least in large part) on a Scheme base. It’s a what-you-see-is-what-you-get, it’s got (mostly) easily learnable keyboard shortcuts (hit tab a couple times from similar symbols, for instance), it has plugins for CAS and interactive in-document coding sessions, has a graphical interface for diagram drawing, and best of all, it’s GNU-licensed/open-source. It’s not perfect, but I like to characterize it as the closest thing I’ve seen to a modern mathematical text editor.
(=_=) (May 29 2020 at 15:12):T Murrills said:
This doesn’t count as a ”redesign of the scientific paper” but nonetheless seems relevant here
I'm not sure what's relevant any more. There are several intertwined topics in this thread from the last time this was active. The thread feels like a wishlist for how to change scientific publishing as we know it.
I’m thinking about document creation/structure, and the interface we use to do it. One thing I’ve been surprised at is how far behind being a “good, intuitive text editor for math” most LaTeX editors are.
See, why can't we just scribble maths on our devices and have it automagically and beautifully rendered? How hard can that be? (Answer: Very)
T Murrills (May 29 2020 at 15:20):? TeXmacs lets you input mathematical text much more easily and reliably than writing raw LaTeX; there’s a lot of user-interface ground to cover between writing LaTeX and just scribbling on our devices. (Unless I’ve misunderstood the point you were trying to make?)
(=_=) (May 29 2020 at 15:38):T Murrills said:
there’s a lot of user-interface ground to cover between writing LaTeX and just scribbling on our devices. (Unless I’ve misunderstood the point you were trying to make?)
Yeah, I'm actually saying I want that. TeXmacs, as good as it is, is still old-school stuff, like Word. And you're right that there's a lot of UI stuff to cover as well. NLP stuff too, plus the use of symbols in mathematical writing is highly... "creative".
(=_=) (May 29 2020 at 15:40):Oliver Shetler said:
Including author-coded metadata on what kinds of arguments scientists intend to make could drastically improve the ease and quality of lit reviews and meta analyses.
Just popped back in to say: we already have that. It's called an abstract, and I can't tell you how many times I've seen an abstract that bears only a passing resemblance to the paper it's supposed to summarise. This is one reason why I'm not in favour of authors hand-coding metadata: I don't trust them to give me clean data.
T Murrills (May 29 2020 at 15:58):ah, I see! (I thought you were just being sarcastic :P )
a random thought about document creation, btw: I think the formats in which we write papers and read documents ought to be interchangeable. Maybe not exactly the same; maybe you want some more stuff exposed while you’re editing, and you want to present a cleaner version to an audience. But a reader ought to be able to choose to edit your document and expose all that stuff if they want—maybe by changing the extension or changing something in the document’s metadata, or simply by opening it differently.
I feel like this gives the structure of the document greater relevance, and so greater possible use (or something)—and makes our tools coincide, so we don’t have to build one thing and convert to another. It might make adding custom functionalities easier and more widespread...in an imaginary document format where adding functionalities to the document (like interactive sessions/visualizations) is already doable and easy, ofc. :) (You can write stuff like this for TeXmacs, but afaik it’s not exactly easy!)
In general, I think some of the different things being talked about in this topic might “coevolve” and help each other along, and that it would be harder to instead add them piecemeal or in sequence to the current state of affairs. Idk—just a speculation.
Peter Arndt (May 29 2020 at 16:01):Rongmin Lu said:
T Murrills said:
See, why can't we just scribble maths on our devices and have it automagically and beautifully rendered? How hard can that be? (Answer: Very)
Actually I am quite impressed by MathPix Snip doing just that. I am currently teaching linear algebra, and it saves me the nightmare of teXing my matrix calculations. It is even not bad at recognizing clean handwriting.
Oliver Shetler (May 29 2020 at 16:54):Rongmin Lu said:
Oliver Shetler said:
Including author-coded metadata on what kinds of arguments scientists intend to make could drastically improve the ease and quality of lit reviews and meta analyses.
Just popped back in to say: we already have that. It's called an abstract, and I can't tell you how many times I've seen an abstract that bears only a passing resemblance to the paper it's supposed to summarise. This is one reason why I'm not in favour of authors hand-coding metadata: I don't trust them to give me clean data.
I feel what you're saying... but taking things to that level of cynicism undermines the entire edifice of scientific discourse.
My opinion is that scientific papers are––on the whole––credible. It's just that they conceal too much of their logic and rhetoric. All of modern science is based on the same adversarial learning system that analytic philosophy and western law is also based on––a defeasible mode of inquiry unlike mathematics. However, many bodies of literature make it hard to process and recombine arguments from different sources because scientists aspire to be like mathematicians who conceal their problem-solving-based arguments in favor of neatly packaged results (theorems, corollaries, etc.). Unfortunately, this just leads to a mess in science because scientific results are defeasible rather than (contingently) immutable.
Whenever possible, I already code papers of interest using the schemes I know (though I'm working on expanding my repertoire into statistical arguments––which would be a lot more useful). It works pretty well as a knowledge management strategy. It's not perfect, but nothing is.
Rongmin Lu said:
Nikolaj Kuntner said:
I've also heard critical voices of TeX as such.
Yup, yup. needs to die too, pace Donald Knuth. Same problem as PDF though: too big to fail.
In defense I have to say that I know very few other alternatives with such a fine typography control
Fabrizio Genovese (May 29 2020 at 17:19):With a good knowledge of it is generally possible to publish near-to-perfect documents, from a typographic point of view. Indeed, I am quite sure most of the publishers, also outside of the scientific community, use it.
Fabrizio Genovese (May 29 2020 at 17:22):This strange thing happened to me once: I wrote a book together with a friend about the relationships between cryptography and secret services. I asked the publisher if they used latex, and what latex classes should I use. They said that they were using latex, but couldn't share their templates/classes. It never happened to them that an author had made such a request because they mainly publish stuff in political sciences. So I had to convert my file to word, only to allow them to convert it back to . Beaurocracy at its finest I guess...
Morgan Rogers (he/him) (May 29 2020 at 17:44):Oliver Shetler said:
Morgan Rogers I'm not sure if this is of interest to you, but have you checked out Douglas Walton's books Methods of Argumentation? And Argumentation Schemes?
One place where scientific papers could stand to improve is on how they present their arguments. Including author-coded metadata on what kinds of arguments scientists intend to make could drastically improve the ease and quality of lit reviews and meta analyses. Plus, it has the advantage of being an add-on rather than an overhaul of existing formats.
Thanks for pointing me to this! I'm basing my comments on the site with examples that @Rongmin Lu shared.
On the face of it, a lot of the schemes themselves won't be directly relevant to all areas of scientific research (we (modern) scientists are loath to appeal to authority in our arguments, for example). However, the very concept of these schemes is the crux of what one would want to use NLP to extract from existing literature: a schematic, and therefore compositional, presentation of the arguments, results and proofs presented in a paper. A jigsaw piece that can take its place in the magnificent puzzle of science.
Morgan Rogers (he/him) (May 29 2020 at 17:51):Simon Burton said:
Did anyone mention distil.pub yet? For example, https://distill.pub/2020/bayesian-optimization/
A pretty, open source, community project. As far as the online notebook idea goes, this seems like a very nice execution of it, and from their staff list it seems they have enough industry backing to have competitive legitimacy.
When both meta-research and research have as much clarity as some of the best articles on here, my dream will be fulfilled :heart_eyes:
Oliver Shetler (May 29 2020 at 18:50):Morgan Rogers said:
Thanks for pointing me to this! I'm basing my comments on the site with examples that Rongmin Lu shared.
On the face of it, a lot of the schemes themselves won't be directly relevant to all areas of scientific research (we (modern) scientists are loath to appeal to authority in our arguments, for example). However, the very concept of these schemes is the crux of what one would want to use NLP to extract from existing literature: a schematic, and therefore compositional, presentation of the arguments, results and proofs presented in a paper. A jigsaw piece that can take its place in the magnificent puzzle of science.
Yes. Walton's schemes are geared towards old school rhetoric and law right now, but I think the overall idea has a lot of potential. In particular, I think schematizing (1) statistical tests (2) common causal arguments (3) common arguments by constraint / other scientific arguments a la Walton could be useful.
Also, these people are working on applications of the argument schemes to AI. Idk about NLP per say, but I'm sure that's close at hand.
(=_=) (May 30 2020 at 05:58):Oliver Shetler said:
Also, these people are working on applications of the argument schemes to AI. Idk about NLP per se, but I'm sure that's close at hand.
For what it's worth, I'm heartened to see conferences like the AITP. Incidentally, there's an abstract there by Dennis Müller and Cezary Kaliszyk on Learning Semantic Annotations for LaTeX Documents, which seems to be something you might find interesting. It combines your metadata proposal with my insistence on letting the machine do the job.
(=_=) (May 30 2020 at 06:06):Fabrizio Genovese said:
In defense I have to say that I know very few other alternatives with such a fine typography control
There are few alternatives to for the fine control of typography.
There are few alternatives to C/C++ for low-level programming using a high-level language.
That doesn't mean these languages don't have design issues, nor that people aren't justified in seeking to find replacements for them.
It just means that they will still be around for a very long time.
(=_=) (May 30 2020 at 06:19):Morgan, I generally agree with most parts of your comment, except for this:
Morgan Rogers said:
we (modern) scientists are loath to appeal to authority in our arguments, for example
Have you not heard of citations?
Granted, most citations are justifiable, but appeals to authority need not always be fallacies. They are just defeasible arguments, and the point of Walton's work was to point out how these arguments can be defeated.
The same goes for citations. We've all seen spurious citations, or citations to three volumes of a work without specifying the page numbers or the volume being referred to, etc, etc. Sometimes Referee 2 will demand that you cite the (sometimes spurious) references they've supplied in their report before they would concede to the publication of your manuscript. And I've actually seen (recent!) maths stuff in which references to "a famous theorem" are made without actual citations!
(=_=) (May 30 2020 at 06:30):T Murrills said:
ah, I see! (I thought you were just being sarcastic :P )
You're right, it does read like I was being sarcastic, but I was merely being playful. This thread is for blue-sky ideas, not incremental ones, so I'm trying to push the envelope by proposing lofty goals like this.
But a reader ought to be able to choose to edit your document and expose all that stuff if they want—maybe by changing the extension or changing something in the document’s metadata, or simply by opening it differently.
You can do wikis, and I think the polymath project is an example of this. However, the academic system rewards individualistic efforts, at least in mathematics, so most people doing maths academically are rather attached to the idea that their work is their own for this idea to take off.
(NB: I'm not saying collaborations don't exist. It's just that even in those massive polymath projects, the people writing up the results and, hence, being credited for it are few in number.)
(=_=) (May 30 2020 at 06:32):Peter Arndt said:
Actually I am quite impressed by MathPix Snip doing just that. I am currently teaching linear algebra, and it saves me the nightmare of teXing my matrix calculations. It is even not bad at recognizing clean handwriting.
This looks awesome! Thanks for sharing your find. I think the future I was envisioning may not be so far off after all. :smiley:
(=_=) (May 30 2020 at 06:48):Oliver Shetler said:
Rongmin Lu said:
Just popped back in to say: we already have that. It's called an abstract, and I can't tell you how many times I've seen an abstract that bears only a passing resemblance to the paper it's supposed to summarise. This is one reason why I'm not in favour of authors hand-coding metadata: I don't trust them to give me clean data.
Whenever possible, I already code papers of interest using the schemes I know (though I'm working on expanding my repertoire into statistical arguments––which would be a lot more useful). It works pretty well as a knowledge management strategy. It's not perfect, but nothing is.
This thread was opened to discuss how to improve the scientific publishing process, so ideas discussed here are geared towards making things more perfect.
It's nice that you're practicing what you preach, but time-poor researchers aren't going to be nearly as conscientious as you are. Your proposal adds friction to an already time-consuming process, and the manual component in that proposal is the defeasible "weakest link" in that chain. This is one plank of my argument that I "don't trust" humans to give me clean data.
(NB: I use "don't trust" as a shorthand for "trust, but verify", which is essentially the same thing, but somehow the former always raises heckles. :shrug: )
(=_=) (May 30 2020 at 07:08):Oliver Shetler said:
My opinion is that scientific papers [...] conceal too much of their logic and rhetoric. [...] However, many bodies of literature make it hard to process and recombine arguments from different sources because scientists aspire to be like mathematicians who conceal their problem-solving-based arguments in favor of neatly packaged results (theorems, corollaries, etc.).
I'll treat this part of your argument first. I agree that authors can do better when it comes to academic writing. In the case of mathematics, however, the style of writing that you're not in favour of is due to Bourbaki, who has many fervent defenders: see #practice: applied ct > mathematical writing, particularly here.
Bourbaki's austere style has been much imitated, but prior to that, people wrote papers in the style you favoured, and concluded problem-solving arguments by packaging the results, instead of announcing them at the beginning. I called this the "Pokemon theorem" style here. This style is still practiced in certain areas of computer science, and I'll take reading a CS paper over a maths one any day.
There is an argument for packaging results though: it's also a kind of metadata that helps to organise the ideas in a paper.
(=_=) (May 30 2020 at 07:26):Oliver Shetler said:
Rongmin Lu said:
It's called an abstract, and I can't tell you how many times I've seen an abstract that bears only a passing resemblance to the paper it's supposed to summarise. This is one reason why I'm not in favour of authors hand-coding metadata: I don't trust them to give me clean data.
I feel what you're saying... but taking things to that level of cynicism undermines the entire edifice of scientific discourse. My opinion is that scientific papers are––on the whole––credible.
On the contrary, scientific discourse is being undermined because people are not being critical enough. You've surely heard of the replication crisis. Peer review is also broken because many professors farm out the work to their students, who are usually not in the best position to evaluate the manuscript confronting them. Yes, it's a convenient way to train students, who'd surely benefit from the work. However, it also shows how, when you don't have adequate incentives (peer review is generally unpaid work), people give peer review work a much lower priority than appropriate.
As for abstracts, people have all sorts of reasons for producing abstracts that are inaccurate, ranging from innocent misunderstandings of their own work through to bad faith. The point is, it's hard to tell what led to that inaccuracy, and I want to take the guesswork out of it. I would rather we have an open-source algorithm producing the abstract and metadata, because I can at least interrogate the algorithm without a lot of hassle.
(=_=) (May 30 2020 at 07:37):Oliver Shetler said:
All of modern science is based on the same adversarial learning system that analytic philosophy and western law is also based on––a defeasible mode of inquiry unlike mathematics. [...] scientific results are defeasible rather than (contingently) immutable.
Oh boy! You'd love this comment. Settle down with a nice beverage and brace yourself. I can wait.
Scientific results are defeasible because they rely on inductive reasoning, which reasons backwards from effects to causes: if you are ignorant of a cause, you may be misattributing an effect to an erroneous cause. Mathematical results appear to be less defeasible because they reason forward from hypotheses to conclusions using deductive reasoning.
However, the assurance of that indefeasibility of mathematical results rest upon the very adversarial learning system that you claimed mathematics is, somehow, not based on. As you may have inferred from my complaint about peer review, the adversarial system that peer review is ostensibly supposed to be is not working properly these days, and that is a problem.
(=_=) (May 30 2020 at 07:52):Oliver Shetler said:
Rongmin Lu said:
It's called an abstract, and I can't tell you how many times I've seen an abstract that bears only a passing resemblance to the paper it's supposed to summarise. This is one reason why I'm not in favour of authors hand-coding metadata: I don't trust them to give me clean data.
I feel what you're saying... but taking things to that level of cynicism undermines the entire edifice of scientific discourse.
Here's another reason why this is actually realism, not cynicism: we've done this experiment before with web documents. There is metadata in HTML documents that search engines read, and people have used the meta elements to do search engine optimisation (SEO) to boost their rankings in search results.
You can do SEO in a variety of ways though: you can write really informative content, or you can just stuff the keyword attribute of meta elements with your desired keywords. In the early days of the Web, many people do the latter, and users were bombarded with a lot of useless results. More recently, Google has cracked down a lot on the latter to improve the signal-to-noise ratio, so that people are now forced to supply useful relevant content.
Right now, there is also a lot of noise in scientific research, c.f. the replication crisis and -hacking. Hence, it would be counter-productive to introduce more noise by requiring metadata, because people can then hack that metadata to boost the visibility of their results. They have every incentive to do so, because getting ahead in their academic careers depends on gaining visibility.
@Henry Story: perhaps you'd like to comment more on the web aspects of this comment. In particular, I think the idea of the Semantic Web may be relevant here.
Morgan Rogers (he/him) (May 30 2020 at 14:41):Rongmin Lu said:
Morgan, I generally agree with most parts of your comment, except for this:
Morgan Rogers said:
we (modern) scientists are loath to appeal to authority in our arguments, for example
Have you not heard of citations?
Granted, most citations are justifiable, but appeals to authority need not always be fallacies. They are just defeasible arguments, and the point of Walton's work was to point out how these arguments can be defeated.
Okay, any good paper should contain some motivation, which may include arguments for why a particular problem is worth looking at, of which some may be of the form "here are some influential people who care about this". They should also contain relevant background material, which may include non-specific references. But these arguments should be independent of the scientific content of the paper.
There is a more philosophical aspect to "citations as appeals to authority", though, in that no one has time to check them all. Thus any quoted result from another author (or the same author) that goes unchecked must be accepted on the basis of that author's authority alone. It's like second hand stories, or second hand evidence... If too many reported results get changed together, the literature can get very difficult to stay on top of.
Another nice thing about building a meta-literary structure would be that cited results become "gluing points", and any 'real' appeals to authority are revealed as arguments which fail to be well-founded in this structure. And one could even find a shortest path from familiar (accepted) material and results to desired results, following only paths containing acceptable methods of proof.... :robot:
Oliver Shetler (May 30 2020 at 17:35):Rongmin Lu said:
Oliver Shetler said:
Whenever possible, I already code papers of interest using the schemes I know (though I'm working on expanding my repertoire into statistical arguments––which would be a lot more useful). It works pretty well as a knowledge management strategy. It's not perfect, but nothing is.
This thread was opened to discuss how to improve the scientific publishing process, so ideas discussed here are geared towards making things more perfect.
It's nice that you're practicing what you preach, but time-poor researchers aren't going to be nearly as conscientious as you are. Your proposal adds friction to an already time-consuming process, and the manual component in that proposal is the defeasible "weakest link" in that chain. This is one plank of my argument that I "don't trust" humans to give me clean data.
(NB: I use "don't trust" as a shorthand for "trust, but verify", which is essentially the same thing, but somehow the former always raises heckles. :shrug: )
I mean... that's kind of like saying "time poor researchers will not be bothered to learn rigorous mathematical thinking" or "time poor researchers will not be bothered to learn to actually understand statistical tests so we might as well give up on advocating that people think that way." Argument schemes aren't the kind of thing machines can reliably identify in natural language. Also, even if there was a way to unreliably code papers with argument schemes, that wouldn't necessarily reflect the author's intent. It would reflect the distorted opinion––in the hyper-abstract––of the theorist who developed the method that the machine learning engineer interpreted and implemented. The two main reasons for asking people to label their arguments is (1) to get people to think about what they are arguing / address more critical questions / openly highlight the loose ends; and (2) to gather information about authors' intent. Neither of these goals is accomplished by a purely machine learning solution. Maybe in the future, AI can assist us in using argument schemes––just like CAS systems help us use numerical methods.
The NLP for argument schemes idea could become useful for lit reviews, but only if people actually adopt the convention of using them first. So even in that case, that project would need a base of people who know how to do this stuff by hand. By analogy, if we didn't teach people arithmetic, and make them practice a lot, all the data scientists would disappear.
I think the way to go is to use the method, gain a demonstrable advantage, then tell people how you did it and encourage them to copy you. That would serve both goals.
Oliver Shetler (May 30 2020 at 17:44):Rongmin Lu said:
Oliver Shetler said:
My opinion is that scientific papers [...] conceal too much of their logic and rhetoric. [...] However, many bodies of literature make it hard to process and recombine arguments from different sources because scientists aspire to be like mathematicians who conceal their problem-solving-based arguments in favor of neatly packaged results (theorems, corollaries, etc.).
I'll treat this part of your argument first. I agree that authors can do better when it comes to academic writing. In the case of mathematics, however, the style of writing that you're not in favour of is due to Bourbaki, who has many fervent defenders: see #practice: applied ct > mathematical writing, particularly here.
Bourbaki's austere style has been much imitated, but prior to that, people wrote papers in the style you favoured, and concluded problem-solving arguments by packaging the results, instead of announcing them at the beginning. I called this the "Pokemon theorem" style here. This style is still practiced in certain areas of computer science, and I'll take reading a CS paper over a maths one any day.
There is an argument for packaging results though: it's also a kind of metadata that helps to organise the ideas in a paper.
Ok this is a fair criticism of my loose rhetoric. What I meant to say is that mathematics can get away with being abstruse about its arguments because––once a result is proven––all the 'argument schemes' in mathematics can collapse to 'by proof...such and such is true'. And––if the result has truly been proven––under the assumptions for the proof, there are no further critical questions. This means mathematicians can string together results without thinking too hard about how they got them. It could have been a beautiful explanatory proof, a convoluted pedantic proof or computer generated gobbeldy-gook. A proof is a proof is a proof.
I was raising the point that scientists often act like this with their citations, stringing results together, even though you can't do that for appeals to authority that are then contingent on underlying irreducibly defeasible scientific arguments.
Oliver Shetler (May 30 2020 at 17:51):Rongmin Lu said:
Oliver Shetler said:
All of modern science is based on the same adversarial learning system that analytic philosophy and western law is also based on––a defeasible mode of inquiry unlike mathematics. [...] scientific results are defeasible rather than (contingently) immutable.
Oh boy! You'd love this comment. Settle down with a nice beverage and brace yourself. I can wait.
Scientific results are defeasible because they rely on inductive reasoning, which reasons backwards from effects to causes: if you are ignorant of a cause, you may be misattributing an effect to an erroneous cause. Mathematical results appear to be less defeasible because they reason forward from hypotheses to conclusions using deductive reasoning.
However, the assurance of that indefeasibility of mathematical results rest upon the very adversarial learning system that you claimed mathematics is, somehow, not based on. As you may have inferred from my complaint about peer review, the adversarial system that peer review is ostensibly supposed to be is not working properly these days, and that is a problem.
Again, totally fair criticism of my loose language. I have a taxonomy in my head that I didn't lay out. In it, mathematics also shares a common ancestor with science, analytic philosophy and western law. It's just that the split goes back further. Mathematics has the nice closure that I mentioned above, whereas these other disciplines do not.
My main point was––as I mentioned––that scientists need to account for the fact that their reasoning is defeasible (though not necessarily inductive).
Scientific results are defeasible because they rely on inductive reasoning, which reasons backwards from effects to causes: if you are ignorant of a cause, you may be misattributing an effect to an erroneous cause.
This presupposes that your starting ontology is correct. It also presupposes that you have a coherent notion of causes in the domain in question. I don't think we can take either of these things for granted in science. It also presupposes that your choice of model is correct. Scientific induction (usually in the form of statistical tests) stand on top of a whole bunch of other reasoning that isn't inductive at all. It really depends on the field, the level of analysis, etc. Still, we agree on the main point–– that science is defeasible.
Oliver Shetler (May 30 2020 at 18:00):Rongmin Lu said:
Oliver Shetler said:
Rongmin Lu said:
It's called an abstract, and I can't tell you how many times I've seen an abstract that bears only a passing resemblance to the paper it's supposed to summarise. This is one reason why I'm not in favour of authors hand-coding metadata: I don't trust them to give me clean data.
I feel what you're saying... but taking things to that level of cynicism undermines the entire edifice of scientific discourse.
Here's another reason why this is actually realism, not cynicism: we've done this experiment before with web documents. There is metadata in HTML documents that search engines read, and people have used the meta elements to do search engine optimisation (SEO) to boost their rankings in search results.
You can do SEO in a variety of ways though: you can write really informative content, or you can just stuff the
keywordattribute of meta elements with your desired keywords. In the early days of the Web, many people do the latter, and users were bombarded with a lot of useless results. More recently, Google has cracked down a lot on the latter to improve the signal-to-noise ratio, so that people are now forced to supply useful relevant content.
This is a fair point. However, I'm having a hard time thinking of how the label "argument by [insert scheme here]" could be abused for search optimization in a way that would add much to the abuses related to tagging and keywords. The number of useful known schemes is likely in the low hundreds, so it's a finite and repetitive set. They have no bearing on topical relevance, so search relevance hacking would be hard.
Right now, there is also a lot of noise in scientific research, c.f. the replication crisis and -hacking. Hence, it would be counter-productive to introduce more noise by requiring metadata, because people can then hack that metadata to boost the visibility of their results. They have every incentive to do so, because getting ahead in their academic careers depends on gaining visibility.
As for the reproducibility crisis and p-hacking (which you also mentioned elsewhere). This is one of the main reasons why I like argumentation schemes. I wish that statistical tests were presented in the form of formal arguments and labeled schemes so that people could detect fishy rhetoric. While this would not directly address p-hacking, the it's probably fair to say that––often––the kinds of people who p-hack are the kinds of people who use sketchy or lazy reasoning. If the rhetoric was brought to the center, peoples critical thinking skills would kick in more. Non-statisticians could catch many errors that only experts currently see.
Also, let's be honest, a lot of the research claiming to 'debunk' whole disciplines relies fishy interpretations of the original studies. Very little attention gets paid to the competence of debunkers' interpretation of the experimental procedures they try to reproduce. Both parties would benefit from making their arguments were more explicit. Researchers could convey the most important parts of their research in a way that would allow replicators to correctly identify the crux of their experimental designs; and replicators would be constrained to justify their choices when attempting to reproduce the work.
(=_=) (May 30 2020 at 23:00):Morgan Rogers said:
Okay, any good paper should contain some motivation, which may include arguments for why a particular problem is worth looking at, of which some may be of the form "here are some influential people who care about this". They should also contain relevant background material, which may include non-specific references. But these arguments should be independent of the scientific content of the paper.
This is where the interesting problem is. Yes, there may be a lot of background material, but it's often not clear how it relates to the content of the paper. Did it motivate the present work? Is it just general background that is relevant to the scientific content of the paper? Is it general background, but of an inconsequential nature? Is it an opposing approach? Is it even spurious? Etc, etc.
(=_=) (May 30 2020 at 23:00):Morgan Rogers said:
Another nice thing about building a meta-literary structure would be that cited results become "gluing points", and any 'real' appeals to authority are revealed as arguments which fail to be well-founded in this structure. And one could even find a shortest path from familiar (accepted) material and results to desired results, following only paths containing acceptable methods of proof.... :robot:
So there may be some ambiguity in this criterion, because I can see that "arguments which fail to be well-founded in this structure" can include some of the instances of citations I've listed above. While we'd like to sieve out the spurious ones, should we also filter those that may be considered as valid background material, yet have contributed nothing to the arguments in the contents of the current paper, except for being a mere "gluing point" in the meta-literary structure?
The shortest path problem is probably a nice problem to solve, as I find myself asking the same question when I do a literature review.
(=_=) (May 30 2020 at 23:24):Oliver Shetler said:
that's kind of like saying "time poor researchers will not be bothered to learn rigorous mathematical thinking"
Exhibit A: physicists. If not for their lack of mathematical rigour, we wouldn't have path integrals or renormalisation, which later inspired new mathematics (note that this means the rigorous maths wasn't there in the first place).
I should qualify this by saying that I'm not against teaching them this, nor that the physicists themselves aren't aware of the importance of rigour, but that the physicist's mode of thinking often uses thinking that is considered to be non-rigorous to a mathematician.
In that sense, they're time-poor, because they don't have the time to engage in a decades-long program to formulate their thinking in a rigorous manner. Otherwise, they wouldn't be physicists: they'd be mathematicians!
(=_=) (May 30 2020 at 23:34):Oliver Shetler said:
"time poor researchers will not be bothered to learn to actually understand statistical tests
so we might as well give up on advocating that people think that way."
The first half is a statement of an existing problem, and I don't agree with the second half.
if we didn't teach people arithmetic, and make them practice a lot, all the data scientists would disappear.
Yet no working data scientist would routinely do complicated arithmetic by hand: they'd just reach for a calculator or type into Excel.
That's the point: learning how to understand statistical tests is friction. Learning what argument schemes are constitutes friction in the process. If you think learning these things are good, you need to advocate for training researchers to use them. However, you also need to make tools that'd help them use these things more efficiently in their work.
(=_=) (May 30 2020 at 23:48):Oliver Shetler said:
Argument schemes aren't the kind of thing machines can reliably identify in natural language.
Then, as an old argument goes, you haven't really understood argument schemes yet, since you haven't succeeded in teaching them to machines.
Also, even if there was a way to unreliably code papers with argument schemes, that wouldn't necessarily reflect the author's intent.
It's a reflection of the author's degree of fallibility. I think I've been clear in stating that I think the intent of the author is a red herring. The author being merely human --- and thus subject to various cognitive biases, which cannot be readily corrected by an external agent, that'd lead to unreliable coding --- is enough for my argument. This is why I'd prefer a machine, which plays the role of an independent agent in this situation, to do that coding.
The two main reasons for asking people to label their arguments [...]
I think it's a great idea to train people to use argument schemes for their own private refinements of their thought. We could also advocate clearer writing styles that makes the schemes manifest, so that NLP tools can more easily infer these schemes from processing the papers. This is how to reduce the friction in the process.
(=_=) (May 31 2020 at 00:01):Oliver Shetler said:
I was raising the point that scientists often act like this with their citations, stringing results together, even though you can't do that for appeals to authority that are then contingent on underlying irreducibly defeasible scientific arguments.
RIght, that's a fair point, especially when appeals to authority actually buttress an argument.
Here's a case that should be of interest to you: the argument that artificial neural networks (ANNs) are of little relevance in the study of neuroscience.
This argument was advanced by Francis Crick in a 1989 polemic in Nature and held sway for a very long time. The justification was that the power of ANNs came from backpropagation, which cannot exist in a biological neural network because neurons can only fire in one direction. That argument is defeasible, because it turns out that we weren't aware at the time of the function of glia and other components and processes in the brain's network to produce effects similar to backpropagation. There are now various "biologically plausible" theories of backpropagation, and Fong et al's Backprop as Functor shows that backpropagation is a mathematical necessity for the composition of neural networks to happen, which reflects the fact that ANNs only became powerful and exciting when backpropagation was implemented.
(=_=) (May 31 2020 at 00:14):Oliver Shetler said:
Rongmin Lu said:
You can do SEO in a variety of ways though: you can write really informative content, or you can just stuff the
keywordattribute of meta elements with your desired keywords. In the early days of the Web, many people do the latter, and users were bombarded with a lot of useless results. More recently, Google has cracked down a lot on the latter to improve the signal-to-noise ratio, so that people are now forced to supply useful relevant content.This is a fair point. However, I'm having a hard time thinking of how the label "argument by [insert scheme here]" could be abused for search optimization in a way that would add much to the abuses related to tagging and keywords. The number of useful known schemes is likely in the low hundreds, so it's a finite and repetitive set. They have no bearing on topical relevance, so search relevance hacking would be hard.
I think my argument wasn't clear there. The point was that labelling was a cheap way to organise information when the search capability wasn't sophisticated enough. It was only in recent years that Google started penalising web documents for not having meaningful content, and promoted those that do, which presumably requires a lot more sophisticated NLP tech than what was available in previous decades.
In the case of argument schemes, it's not very informative to just put labels saying such and such an argument was used here. The metadata should probably include the structure of the arguments in the paper and the corresponding classifying labels, which a crawler would then extract and feed into the search engine's database.
If you let authors hand-code their own metadata, they could introduce errors because of their own misapprehension of what they've done. This routinely happens: people can miss something in their work that may seem obvious to outsiders, or their perceived perception of the significance of their work may differ from outsiders. A historical example is that Newton reportedly thought his theological work concerning the apocalypse was more important than his calculus!
(=_=) (May 31 2020 at 00:27):Oliver Shetler said:
Also, let's be honest, a lot of the research claiming to 'debunk' whole disciplines relies fishy interpretations of the original studies.
That shouldn't detract from the fact that there's a systemic issue here: the dysfunction of the adversary in the adversarial system that we're supposed to have in academic research. I don't claim that the replication crisis is a justification for "debunking" entire disciplines, just that it's a systemic problem, one that will dog those disciplines and fuel political attacks for a long time.
However, there are certain schools of thought that need debunking, and people may have used the word "debunking" to highlight legitimate grave concerns. For example, the school of thought in economics known as "marginalism" historically, and "neoclassical economics" to its critics, is founded on many flawed mathematical premises which, many critics have argued, completely invalidates many of its conclusions and frameworks.
Yet, it is alleged, the prestigious journals in economics routinely reject papers that don't adhere to this school of thought, which is why people like Steve Keen have written books like Debunking Economics. Now, Keen isn't debunking his own discipline: he's just criticising the dominant school of thought, and arguing for a better mathematical foundation for economics.
Of course, I'm biased here as I'm sympathetic to his work, but I think some of the "debunking" out there isn't necessarily done in bad faith simply because it comes across as hostile.
Jules Hedges (May 31 2020 at 09:59):Rongmin Lu said:
Oliver Shetler said:
that's kind of like saying "time poor researchers will not be bothered to learn rigorous mathematical thinking"
Exhibit A: physicists. If not for their lack of mathematical rigour, we wouldn't have path integrals or renormalisation, which later inspired new mathematics (note that this means the rigorous maths wasn't there in the first place).
Let me stoke the fire by claiming that in many cases, rigour is in direct opposition to good communication. Part of the difficulty of good communication is finding the optimal balance between rigour and handwaving. To take it to the extreme, an example of mathematics written with a very high level of rigour is Russell and Whitehead's Principia, which is hardly a page turner
Jules Hedges (May 31 2020 at 10:01):A less silly example is Leibniz style calculus where you do things like "divide through by dx", which is much easier to communicate and remember than any of the various ways of making it rigorous it, which can be quite obscure - want to teach first year engineers about tangent bundles?
(=_=) (May 31 2020 at 14:24):Jules Hedges said:
Let me stoke the fire by claiming that in many cases, rigour is in direct opposition to good communication. Part of the difficulty of good communication is finding the optimal balance between rigour and handwaving.
Yeah, that was my argument against the Bourbaki style as well. Unfortunately, as you may recall, it brought out some ardent defenders as well, so I wanted to talk about another way in which a less "rigorous" approach has reaped benefits.
Oliver Shetler (May 31 2020 at 14:24):Jules Hedges said:
Rongmin Lu said:
Oliver Shetler said:
that's kind of like saying "time poor researchers will not be bothered to learn rigorous mathematical thinking"
Exhibit A: physicists. If not for their lack of mathematical rigour, we wouldn't have path integrals or renormalisation, which later inspired new mathematics (note that this means the rigorous maths wasn't there in the first place).
Let me stoke the fire by claiming that in many cases, rigour is in direct opposition to good communication. Part of the difficulty of good communication is finding the optimal balance between rigour and handwaving. To take it to the extreme, an example of mathematics written with a very high level of rigour is Russell and Whitehead's Principia, which is hardly a page turner
You both are completely right. Period. I think I wasn't clear about what my argument was intended to convey. I was suggesting only that being familiar with a method––knowing how to do the essentials by hand––is a prerequisite for any automated assistance to even be useful. In fact, my–– and your––lack of fluency in argumentation schemes is precisely why these misapprehensions keep happening (even with my sporadic efforts to correct this deficit). I'm not following my own advice in the fast pace of a chat because I still haven't really built the foundations of thinking that way. The best I'm capable of right now is coding papers according to standards nobody taught me––very slowly. (Automating parts of my manual process would be very useful. Removing me from the process would probably just make me a worse thinker.)
So I'm not saying everybody should follow the exact same method. "Lack of rigor" as you both have interpreted it is certainly not always a disadvantage. However, I'm suggesting that, especially at this early stage of adoption, knowing and using argumentation schemes manually is more important than automating any one part of the process––just like knowing and using calculus is more important than automating its use (though that certainly is important at this late stage of wide adoption). I also think automating the whole process would defeat the purpose of using them. It would be just another way to generate reports that nobody is proficient enough to use well.
(=_=) (May 31 2020 at 14:37):Oliver Shetler said:
However, I'm suggesting that, especially at this early stage of adoption, knowing and using argumentation schemes manually is more important than automating any one part of the process
I don't think argumentation schemes are anything new. What's missing here is the training needed to ensure widespread adoption amongst researchers, and the enforcement (that's supposed to come through peer review, but isn't) of best practices in publication.
What's not desirable, however, is to institute yet another checkbox that the researcher must tick, and manually at that, in order to get published. Researchers who already have training in argumentation will find this tedious and patently unnecessary. The point at which such training should take place is in the formative stage of a researcher's education, and NOT during the publication process, where the focus should instead be on enforcing those argumentation standards. This enforcement is perhaps best done with the aid of automation, because it's tedious and repetitive.
I understand that you have a pedagogical objective in mind, which is well and good, but it shouldn't involve making researchers code up metadata about argumentation schemes, because it's about as silly as asking people who routinely perform a lot of calculations in their daily work not to use a calculator or a spreadsheet.
Oliver Shetler (May 31 2020 at 14:45):Rongmin Lu said:
Oliver Shetler said:
Argument schemes aren't the kind of thing machines can reliably identify in natural language.
Then, as an old argument goes, you haven't really understood argument schemes yet, since you haven't succeeded in teaching them to machines.
Also, even if there was a way to unreliably code papers with argument schemes, that wouldn't necessarily reflect the author's intent.
It's a reflection of the author's degree of fallibility. I think I've been clear in stating that I think the intent of the author is a red herring. The author being merely human --- and thus subject to various cognitive biases, which cannot be readily corrected by an external agent, that'd lead to unreliable coding --- is enough for my argument. This is why I'd prefer a machine, which plays the role of an independent agent in this situation, to do that coding.
The two main reasons for asking people to label their arguments [...]
I think it's a great idea to train people to use argument schemes for their own private refinements of their thought. We could also advocate clearer writing styles that makes the schemes manifest, so that NLP tools can more easily infer these schemes from processing the papers. This is how to reduce the friction in the process.
Let's fill out the context here:
Between the two quotes you cited I said this:
Also, even if there was a way to unreliably code papers with argument schemes, that wouldn't necessarily reflect the author's intent. It would reflect the distorted opinion––in the hyper-abstract––of the theorist who developed the method that the machine learning engineer interpreted and implemented.
Clearly a machine isn't an "independent agent" just because it is a machine. Machines reflect the biases of those who code them and those who devise the methods used by those who code them. There is no such thing as an objective third party in the context of adversarial learning. The objective part comes in after scientists have built a quorum consensus about what an experiment or observation would mean––only then do results speak as the objective third party.
I think this point is really important in the question of how to use these tools because this misconception is a huge problem among technologists. We seriously need to stop letting people launder their opinions through algorithmic implementations (or mathematical models for that matter!). Clarifying what real scholars mean to say is the only way to converge on scientific conclusions. While it's true that the artful misinterpretation of others' ideas can be a great source of innovation, the only reason that works is because the new authors' intent improves on the old authors' intent. It still boils down to authors' intent.
In my view, any attempt to reform scientific rhetoric will be dead from the start if begin with this well-debunked premise from the old technocratic political movement.
On the other hand, I think it's great that you want to automate argumentation coding. I would certainly use and (if I can) contribute to such a project. I don't want to discourage it. I just want to convince you that if you don't convince people to use the convention by hand, there will be no cultural support, and your labor will be for naught. Just like a dead programming language.
Morgan Rogers (he/him) (May 31 2020 at 14:57):Rongmin Lu said:
Morgan Rogers said:
Another nice thing about building a meta-literary structure would be that cited results become "gluing points", and any 'real' appeals to authority are revealed as arguments which fail to be well-founded in this structure. And one could even find a shortest path from familiar (accepted) material and results to desired results, following only paths containing acceptable methods of proof.... :robot:
So there may be some ambiguity in this criterion, because I can see that "arguments which fail to be well-founded in this structure" can include some of the instances of citations I've listed above. While we'd like to sieve out the spurious ones, should we also filter those that may be considered as valid background material, yet have contributed nothing to the arguments in the contents of the current paper, except for being a mere "gluing point" in the meta-literary structure?
My short answer is yes: I don't want to eliminate that extra information, but I do want to separate it. I am of the opinion that the mathematical/formal scientific content should be separable from the pedagogical and philosophical content of a paper.
I appreciate that this is an ideal that may not be perfectly achievable, however: the distinction between an axiom and an assertion about reality is a rather subtle one.
Oliver Shetler (May 31 2020 at 14:57):Also, @Rongmin Lu I want to check in on tone. Text, with its flat affect, can sometimes fail to convey the intended congeniality and sportsmanship of a debate. @Jules Hedges comment about "stoking the fire" made me think now would be a good time to touch on that. Are you feeling like our mutual criticisms are constructive? Or would you prefer that we focus more on the parts we agree on going forward?
Oliver Shetler (May 31 2020 at 15:01):Some citation based arguments are of the form "you should believe my claim because this credible source said so" while others are of the form "you should check out this paper if you're unclear on this background idea". Both fit inside the framework. I don't think those background citations need to count as "extra information". Papers often convey many disjointed arguments. Not everything needs to be requred to connect to the central argument (if there is only one).
Morgan Rogers (he/him) (May 31 2020 at 15:04):Having had a conversation with Rongmin privately some weeks ago, I can speak to his great enthusiasm for the potential of this idea. He convinced me that we need to be optimistic about the potential of NLP in building this resource, but realistic about its necessity and about the time and resources required to make that happen.
Morgan Rogers (he/him) (May 31 2020 at 15:06):There are proof theorists out there who work with the formal structures we're talking about abstractly. I wonder if we can get any of them in on this discussion to comment on structural aspects we haven't thought about.
Oliver Shetler (May 31 2020 at 15:09):I bet it would be realistic to develop a few specialized tools to automate the process of using argument schemes. It might also be feasible to automate small parts of the coding process. For example, by developing special text search tools geared towards assisting manual coding. By the way----I think this is in Walton's Methods of Argumentation but I don't have the page number----they've done studies showing that human coding has a consistency score of 0.86, which is very high. That's promising both for the human adoption of these methods and for future NLP assistance.
Oliver Shetler (May 31 2020 at 15:13):One thing I've been meaning to do is develop a set of text-completion templates for existing argumentation schemes, so that once you identify an argument, you can automatically generate the scaffolding plus critical question suggestions. I've been torn between doing this in the form of an XML namespace or just as markup for personal use (that's my excuse for not following through anyway).
Oliver Shetler (May 31 2020 at 15:14):Do you think proof theoretic results would transfer to defeasible arguments?
Morgan Rogers (he/him) (May 31 2020 at 15:19):Yes, for the simple reason that (in a way that argumentation schemes render explicit), defeasible arguments have the same structural form as formal logical/proof theoretic ones; they can be treated as extra principles of deduction. I can fragment my logic by choosing to allow or deny these principles, just as constructivists decompose classical logic by banning uses of the law of excluded middle or AC.
Oliver Shetler (May 31 2020 at 15:23):I suppose if you allow for people to go in and manually code whether they think particular arguments go through, you could use that to automatically propogate results ---- which the argument schemes people are already doing with a declarative programming language.
Oliver Shetler (May 31 2020 at 15:25):I don't know enough about proof theory to know what major results could be transferred, but it sounds promising. Maybe it would be possible to have a sort of dash board of logic mods that you could use to tweak how you evaluate the same argument graph. Sounds interesting.
(=_=) (May 31 2020 at 18:52):Oliver Shetler said:
Jules Hedges comment about "stoking the fire" made me think now would be a good time to touch on that. Are you feeling like our mutual criticisms are constructive? Or would you prefer that we focus more on the parts we agree on going forward?
I feel it's constructive. What Jules meant by that phrase is that he's raising a potentially controversial opinion, but it's an opinion I had already advanced in an earlier iteration of this thread and one that's already implicit in the discussion we've had.
Oliver Shetler (May 31 2020 at 20:32):Good. We've never met so I just wanted to make sure. :hug:
(=_=) (Jun 01 2020 at 06:50):Oliver Shetler said:
Good. We've never met so I just wanted to make sure. :hug:
Morgan Rogers is an admin, so he'd be all over this if there was an actual "fire" going on. That he hasn't yet told me off is enough evidence for me that this discussion is still pretty constructive.
It seems like this is a case of being "divided by a common language". Some expressions that are similar to "stoking the fire" in intent are "stirring things up" and "doing a hot take". Jules was aspiring to be controversial with his comment; he wasn't making an observation that some heated conversation was happening.
(=_=) (Jun 01 2020 at 07:19):Oliver Shetler said:
Clearly a machine isn't an "independent agent" just because it is a machine. Machines reflect the biases of those who code them and those who devise the methods used by those who code them.
Clearly you haven't understood an argument that I made earlier, which you passed over without comment, that supported my position that I'd rather let a machine do it. Let me recall the relevant bit for you:
I would rather we have an open-source algorithm producing the abstract and metadata, because I can at least interrogate the algorithm without a lot of hassle.
This is the advantage I'm citing for a machine: I can interrogate the (open-source) algorithm it's running for biases with far greater ease than I can interrogate humans to understand what's going on between their ears.
Also, the biases for an algorithm include not only those things you've cited, but also the data you feed them, i.e. "garbage in, garbage out". This is what we've learned from (or at least I've learned) the Microsoft chatbot Tay and the recruitment tool that was being developed by Amazon. The formalism of Fong et al. also bears out this fact.
This means that trying to figure out what's going "wrong" in humans also involves reviewing a lifetime of "data" (i.e. experiences) the humans have ingested. Such a review on a therapist's couch is far more difficult and expensive to do than inspecting the data you've fed an algorithm.
(=_=) (Jun 01 2020 at 07:25):Oliver Shetler said:
We seriously need to stop letting people launder their opinions through algorithmic implementations (or mathematical models for that matter!).
And let them continue to launder their opinions through allegedly independent committees that are far more opaque than the open-source algorithms you disparaged?
(=_=) (Jun 01 2020 at 07:44):Oliver Shetler said:
In my view, any attempt to reform scientific rhetoric will be dead from the start if begin with this well-debunked premise from the old technocratic political movement.
And now we come to the crux of the matter: you're under the sway of a particular school of political thought, in this case the anti-technocracy crowd.
Let me just make a few quick points:
(=_=) (Jun 01 2020 at 07:54):Oliver Shetler said:
I just want to convince you that if you don't convince people to use the convention by hand, there will be no cultural support, and your labor will be for naught. Just like a dead programming language.
And I'm trying to inform you that there isn't, and won't be, any cultural support for your proposal either.
There is currently a program to formalise maths. Look up Kevin Buzzard and his projects. Homotopy type theory is also part of the story. The thing is, mathematicians like John Baez have zero (zilch, nada) interest in writing maths in the more formal way demanded by the existing technology that's being used by the formalisation crowd. John wishes them well, but he will not actively participate in their effort because he believes he has better things to do. As far as I can tell, his view is typical of most research mathematicians.
The reason for such inertia is now clear to me. It's because such efforts are labour-intensive and don't appear to be particularly enlightening (i.e. intellectually rewarding) activities to the detractors. The same goes for manually encoding argumentation schemes: most mathematicians have acquired an informal understanding that is sufficient for all intents and purposes, and see no point in belabouring (what they believe to be) the obvious.
That is why my focus has been to improve the user experience, instead of convincing people to do something they clearly and justifiably see no value in doing. I don't deny that there is pedagogical value in educating future researchers about argumentation schemes, but you should not expect working researchers to submit to demands that add more friction to what is already a time-consuming and taxing process.
Jules Hedges (Jun 01 2020 at 13:38):100% agree with this. (And I have the same opinion as John, despite knowing a decent amount about logic and type theory and being a functional programer on the side and even playing with a couple of proof assistants once)
Morgan Rogers (he/him) (Jun 01 2020 at 14:31):Rongmin Lu said:
The reason for such inertia is now clear to me. It's because such efforts are labour-intensive and don't appear to be particularly enlightening (i.e. intellectually rewarding) activities to the detractors.
This was also one of the reasons that I was initially resistant to your arguments that NLP was a crucial/inevitable component in this reformation project: the requisite NLP developments lie outside both my skillset and personal interests, so my instinct was to want to avoid/dismiss them. Fortunately I'm pretty confident that if and when this project gets going, the barriers to getting into an NLP mindset won't be prohibitively high.
(=_=) (Jun 01 2020 at 15:20):Morgan Rogers said:
This was also one of the reasons that I was initially resistant to your arguments that NLP was a crucial/inevitable component in this reformation project: the requisite NLP developments lie outside both my skillset and personal interests, so my instinct was to want to avoid/dismiss them. Fortunately I'm pretty confident that if and when this project gets going, the barriers to getting into an NLP mindset won't be prohibitively high.
Yeah, I care a lot about user experience, which is why I tried to word my proposal in a way that showed that I did. This actually led to frustration at your initial resistance, because I thought I had done the necessary due diligence. Oh well, at least now we're on the same page. :smiley:
I think when the barriers come down, it'd just be a matter of plugging in the correct APIs, no "NLP mindset" required. The point is, the project would only make sense once the NLP is seamless. In the meantime, either wait, or see how you can help to build the theories that'd enable those NLP APIs to become a reality.
(=_=) (Jun 01 2020 at 15:27):Here's another analogy for why a good user experience is important for adoption. Einstein's theory of relativity is crucial in computing the relativistic corrections that make GPS work. It is important that a person with a good scientific education be somewhat familiar with relativity, and for those who aspire to do physics, or require physics in their future work, to master relativity to some extent. That doesn't mean, however, that we have to require users to do those corrections by hand every time they need to use GPS. Otherwise, nobody would use a map app, satnav, or any other app that requires GPS, because there would be way less friction in using good old-fashioned paper maps.
Oliver Shetler (Jun 01 2020 at 17:30):Rongmin Lu said:
Clearly you haven't understood an argument that I made earlier, which you passed over without comment, that supported my position that I'd rather let a machine do it. Let me recall the relevant bit for you:
I would rather we have an open-source algorithm producing the abstract and metadata, because I can at least interrogate the algorithm without a lot of hassle.
This is the advantage I'm citing for a machine: I can interrogate the (open-source) algorithm it's running for biases with far greater ease than I can interrogate humans to understand what's going on between their ears.
Also, the biases for an algorithm include not only those things you've cited, but also the data you feed them, i.e. "garbage in, garbage out". This is what we've learned from (or at least I've learned) the Microsoft chatbot Tay and the recruitment tool that was being developed by Amazon. The formalism of Fong et al. also bears out this fact.
I think a tool for automatically summarizing literature would be amazing. If I could personally specify what my priorities are and get coherent abstracts for use in my personal investigations, that would be a huge boon for anybody who has lit to write review. I just also think that the task of writing an abstract is sufficiently ambiguous that it makes sense to let the author determine what to highlight in that part of the document. You might disagree with the authors' choices, and that's where your tool could become useful. As a computer assisted qualitative analysis tool.
As for your point about an algorithm being open source. If the algorithm is for communal use, it might work for a community of computer scientists or applied mathematicians... but for everybody else... it would definitely be quite a hassle for them to "interrogate the algorithm without a lot of hassle."
Oliver Shetler (Jun 01 2020 at 17:43):Rongmin Lu said:
Oliver Shetler said:
We seriously need to stop letting people launder their opinions through algorithmic implementations (or mathematical models for that matter!).
And let them continue to launder their opinions through allegedly independent committees that are far more opaque than the open-source algorithms you disparaged?
I'm not disparaging open-source algorithms. I'm saying that using them to summarize other peoples' work for them, without their input, unnecessarily smuggles the opinions of the algorithms' designers into a process that already emphasizes the opinions of the authors themselves. If a "independent committee" wrote everybody's abstract, I would raise the same objection. This isn't about algorithms vs other forms of top-down management. It's about allowing authors to say what they think is important, even if some authors are incompetent or dishonest. The paper is right next to the abstract. The risk of dishonest authors involves you being annoyed they wasted your time when you skim the paper. The risk of removing authors from summarizing their own work is that you introduce systematic bias into the whole process. The same risk would apply to letting committees write abstracts. Reviewers already introduce enough systematic bias into that process. It just seems more useful to let readers use such a powerful class of algorithms to automate parts of the literature search.
Oliver Shetler (Jun 01 2020 at 17:57):Rongmin Lu said:
Oliver Shetler said:
In my view, any attempt to reform scientific rhetoric will be dead from the start if begin with this well-debunked premise from the old technocratic political movement.
And now we come to the crux of the matter: you're under the sway of a particular school of political thought, in this case the anti-technocracy crowd.
Let me just make a few quick points:
- I am not saying the machine is infallible. I'm just arguing that any fallibility can be diagnosed and corrected, in principle, much more quickly, easily and ethically, than if a human were the source of the fallibility.
- I am therefore not advocating for the uncritical use of machines. In fact, we need to be there to check them, just as they need to be there to check us. We need that virtuous adversarial system you've been talking about.
- There are technocracy fans here: Jules is one of them with his on-going obsession about cybernetics.
Clearly I'm not an anti-technocrat. I knew enough about the obscure party to criticize the early movement. I'm all for many of their big ideas: giving priority to technical experts on technical issues, minimizing parochial politics on simple issues of infrastructure and sustenance, automating big swaths of unpleasant labor. My reason for citing the "old technocratic political movement" is because a lot of their prescriptions were un-workable top-down command methods that we now know simply do not work. A lot like an algorithm that puts words in authors' mouths. Great as a tool for individual use. Not so great as a way to reform scientific literature.
Oliver Shetler (Jun 01 2020 at 18:16):Rongmin Lu said:
Oliver Shetler said:
I just want to convince you that if you don't convince people to use the convention by hand, there will be no cultural support, and your labor will be for naught. Just like a dead programming language.
And I'm trying to inform you that there isn't, and won't be, any cultural support for your proposal either.
There is currently a program to formalise maths. Look up Kevin Buzzard and his projects. Homotopy type theory is also part of the story. The thing is, mathematicians like John Baez have zero (zilch, nada) interest in writing maths in the more formal way demanded by the existing technology that's being used by the formalisation crowd. John wishes them well, but he will not actively participate in their effort because he believes he has better things to do. As far as I can tell, his view is typical of most research mathematicians.
The reason for such inertia is now clear to me. It's because such efforts are labour-intensive and don't appear to be particularly enlightening (i.e. intellectually rewarding) activities to the detractors. The same goes for manually encoding argumentation schemes: most mathematicians have acquired an informal understanding that is sufficient for all intents and purposes, and see no point in belabouring (what they believe to be) the obvquote
As I understand it, people need to see that a method will give them a competitive advantage. Then some of them will adopt the method. If the method is sufficiently advantageous, it ends up being a game-theoretic invasive species. That's how something gets adopted. Maybe we're right and those of us who use these methods will gain a publishing advantage. Maybe we're wrong and we won't. If we're wrong, that's no skin off my nose. I'm wrong about lots of things that I try. If we're right, others will support the method eventually.
That is why my focus has been to improve the user experience, instead of convincing people to do something they clearly and justifiably see no value in doing.
I agree with the sentiment, but I disagree with the implementation. Maybe I'm misunderstanding you, but if I am perceiving your point correctly, you're saying we should make the tools more convenient so people adopt the method. If that is what you mean, here is my response:
I could be completely wrong about this, but it seems to me that trying create demand for a solution by increasing its supply won't work. My evidence is the graveyard of similar projects. It seems like the first step is to create demand by figuring out how to use a thing for personal gain. Then you share your notes and people crib your methods. (Or, if the advantage is really big you defend your "trade secret" for a while and people try even harder to replicate your methods).
If that isn't what you meant, ignore the response and explain what you actually meant.
I don't deny that there is pedagogical value in educating future researchers about argumentation schemes, but you should not expect working researchers to submit to demands that add more friction to what is already a time-consuming and taxing process.
I never said I would ask researches to "submit to demands." I doubt anybody in this community really takes that stance on anything. Which raises a question:
What can I do to interpret your arguments more charitably? The first thing I can think of is that I should ask more clarifying questions before refining or challenging a point I think you've made.
Oliver Shetler (Jun 01 2020 at 22:19):Morgan Rogers said:
Rongmin Lu said:
The reason for such inertia is now clear to me. It's because such efforts are labour-intensive and don't appear to be particularly enlightening (i.e. intellectually rewarding) activities to the detractors.
This was also one of the reasons that I was initially resistant to your arguments that NLP was a crucial/inevitable component in this reformation project: the requisite NLP developments lie outside both my skillset and personal interests, so my instinct was to want to avoid/dismiss them. Fortunately I'm pretty confident that if and when this project gets going, the barriers to getting into an NLP mindset won't be prohibitively high.
NLP is one of the most exciting applications of Category Theory. I'm totally into it. Though I have a feeling that coding and extracting argumentation schemes / argument graphs / main points of an argument is a ways away. It might require an almost human level of semantic comprehension. Though maybe I'm wrong about that. @Rongmin Lu Do you have any ideas about what sorts of known methods could be leveraged to that end? The work I know of only works on tools to make argument diagramming easier for people who already use semi formal arguments.
(=_=) (Jun 02 2020 at 07:28):Oliver Shetler said:
I just also think that the task of writing an abstract is sufficiently ambiguous that it makes sense to let the author determine what to highlight in that part of the document.
I disagree. By an "abstract", I mean an objectively accurate summary of claims and arguments made in the document. By "objectively accurate", I mean that the summary agrees with the consensus arrived at by experts in the field regarding what the claims and arguments of the document are, based solely on the text and ignoring any issues of experimental replicability at this point. By this standard, most abstracts currently fall short of the mark, because they depend too much on the authors' opinions on what to highlight in their document.
You might disagree with the authors' choices, and that's where your tool could become useful. As a computer assisted qualitative analysis tool.
Agreed. However, I think it can also be a tool to assist authors (who wish to act in good faith) as well, by helping them to check that they are indeed claiming and arguing what they intend to claim and argue.
(=_=) (Jun 02 2020 at 07:28):Oliver Shetler said:
As for your point about an algorithm being open source. If the algorithm is for communal use, it might work for a community of computer scientists or applied mathematicians... but for everybody else... it would definitely be quite a hassle for them to "interrogate the algorithm without a lot of hassle."
The contention was that it's less of a hassle to interrogate an open-source algorithm than a human author, who may well have any number of reasons and excuses to be opaque. Presumably, the communities you've cited would be early adopters, and for everybody else, the adoption would come when they can interrogate the algorithm easily with some user-friendly interface. In any case, research labs routinely employ people to work on software that assists in research, and this could be something to keep them busy.
(=_=) (Jun 02 2020 at 07:42):Oliver Shetler said:
I'm saying that using them to summarize other peoples' work for them, without their input, unnecessarily smuggles the opinions of the algorithms' designers into a process that already emphasizes the opinions of the authors themselves. If a "independent committee" wrote everybody's abstract, I would raise the same objection. This isn't about algorithms vs other forms of top-down management.
I've italicised all the assertions I object to in the above. Let me deal with them in order:
(=_=) (Jun 02 2020 at 07:49):Oliver Shetler said:
It's about allowing authors to say what they think is important, even if some authors are incompetent or dishonest.
This is a major problem. As the current process allows authors to state whatever they think is important in their abstracts, it creates dirty data when authors are mistaken in the assessment of their own work, for whatever reasons. I'd like to take the authors' opinions out of the equation.
The paper is right next to the abstract.
It's harder to extract information from the paper than from the abstract. There are only so many hours in a day.
The risk of removing authors from summarizing their own work is that you introduce systematic bias into the whole process.
Again, never said that, just that they shouldn't let their opinions interfere with the summarisation.
It just seems more useful to let readers use such a powerful class of algorithms to automate parts of the literature search.
It can be a tool for researchers in general, since they're both readers and authors.
(=_=) (Jun 02 2020 at 07:54):Oliver Shetler said:
My reason for citing the "old technocratic political movement" is because a lot of their prescriptions were un-workable top-down command methods that we now know simply do not work.
I agree with your stance, but I'd like to see your evidence for asserting that "their prescriptions" were "un-workable".
@Jules Hedges, would you like to address this as well?
Great as a tool for individual use. Not so great as a way to reform scientific literature.
This is more of a way to make access to information in the scientific literature easier.
(=_=) (Jun 02 2020 at 08:06):Oliver Shetler said:
As I understand it, people need to see that a method will give them a competitive advantage. Then some of them will adopt the method. If the method is sufficiently advantageous, it ends up being a game-theoretic invasive species. That's how something gets adopted.
if I am perceiving your point correctly, you're saying we should make the tools more convenient so people adopt the method. If that is what you mean, here is my response:
I could be completely wrong about this, but it seems to me that trying create demand for a solution by increasing its supply won't work. My evidence is the graveyard of similar projects. It seems like the first step is to create demand by figuring out how to use a thing for personal gain.
Maybe stick to the informal game-theoretic analysis next time, because neoclassical (aka "textbook") economics is a dumpster fire.
Yes, if you demonstrate that a method will give them a competitive advantage, then people would be inclined to adopt that method, and this increases the demand for that solution (and not the supply).
One way to give people a competitive advantage is to save them time, i.e. make a convenient solution, because all of us are time-poor: "valar morghulis" and all that jazz. My insistence on providing a good user experience subsumes the time-saving argument, which is also implicit in the arguments of Jules and Morgan. The resistance from mathematicians that I referred to earlier is there precisely because they don't perceive any competitive advantage from participating in the formalisation of maths.
(=_=) (Jun 02 2020 at 08:11):I won't rehearse the arguments research mathematicians have against formalisation here, but point you to the following threads:
(=_=) (Jun 02 2020 at 08:31):Oliver Shetler said:
Do you have any ideas about what sorts of known methods could be leveraged to that end? The work I know of only works on tools to make argument diagramming easier for people who already use semi formal arguments.
In terms of writing abstracts, there's an active field of research called "abstractive summarisation". I found a summary of the state of the art ca. 2019 here. My impression is that there's a lot to be done, because we haven't yet figured out how to extract the semantics from a document effectively.
To that end, within category theory, there's work by Bob Coecke and his school using CQM (categorical quantum mechanics) to encode semantics. There's a recent conference to launch what they call QNLP. They've done a demonstration using a quantum computer recently that Bob has talked about (Zulip thread here), but I think machine learning should do just fine as well: there is some evidence of deep neural networks exhibiting what appears to be quantum entanglement by Yoav Levine and others (see also this paper at OpenReview.net for an example of how open peer review can take place).
As for informal maths, Mohan Ganesalingam has formalised a grammar of that in his thesis, and he and Timothy Gowers did an experiment using software written based on that grammar (paper here).
Oliver Shetler (Jun 02 2020 at 14:36):Rongmin Lu said:
Oliver Shetler said:
I just also think that the task of writing an abstract is sufficiently ambiguous that it makes sense to let the author determine what to highlight in that part of the document.
I disagree. By an "abstract", I mean an objectively accurate summary of claims and arguments made in the document. By "objectively accurate", I mean that the summary agrees with the consensus arrived at by experts in the field regarding what the claims and arguments of the document are, based solely on the text and ignoring any issues of experimental replicability at this point. By this standard, most abstracts currently fall short of the mark, because they depend too much on the authors' opinions on what to highlight in their document.
You might disagree with the authors' choices, and that's where your tool could become useful. As a computer assisted qualitative analysis tool.
Agreed. However, I think it can also be a tool to assist authors (who wish to act in good faith) as well, by helping them to check that they are indeed claiming and arguing what they intend to claim and argue.
I think we've found the crux of our disagreement. If I'm understanding you correctly, you think there is such a thing as an "objectively accurate summary" of a document. Now this is a subtle issue and I don't entirely disagree. I think that a document can be compressed to a point without making any controversial decisions about what to leave out... But I think an abstract is, by design, too short to be considered an "objective" compression of the paper. My understanding is that an abstract ––usually a single paragraph––is far too short for its author (whether it's a person or an algorithm) avoid controversial decisions about what to include. It simply is not possible to make an "objective" abstract. I don't think that experts could ever come to a consensus about the particular choices made for writing any given abstract. They would likely come to agree that while some of the decisions are normative, many of the decisions are individual / discretionary.
My conclusion, then, is that––since academic discourse is centered around the intellectual property of its contributors (not in the monetary sense, but in the credit sense)––the rightful owner of the abstract is the author of the paper. Hence, the author should have the final say as to what goes into his or her abstract.
Having said this, I think that one could check how accurate an abstract is––in a very literalistic sense. That is to say, one could check for lies of commission, but not lies of omission. One could also measure how comprehensive an abstract is. One could ask "compared to the smallest possible objective summary, how many claims made it into this abstract?" However, the latter metric wouldn't necessarily tell us much about how "good" the abstract is. Depending on the size of the paper and the priorities of the author / publishers, standards of comprehensiveness would vary. If we had tools to measure those things, it would be a great tool for reviewers. I can imagine a new wave of desk rejections on the grounds of false claims in abstracts. Access to such a tool could also make writing accurate abstracts much easier. I'll bet a lot of scientists would make tweaks based on a grammarly-like abstract checking tool. I'll bet it would also lead to a wave of algorithm hacking, just like people resume hack due to the use of algorithms by hiring managers. But maybe it could still be an improvement.
Oliver Shetler (Jun 02 2020 at 14:51):@Rongmin Lu You said (1):
I didn't insist that authors would necessarily be excluded from the process. This is implicit in my characterisation of the process as an interactive one...
and you also said (2):
I'd like to take the authors' opinions out of the equation.
I'm having trouble reconciling these statements.
How can we remove the authors' opinions from the equation without removing the authors' agency?
Oliver Shetler (Jun 02 2020 at 15:00):Rongmin Lu said:
In terms of writing abstracts, there's an active field of research called "abstractive summarisation". I found a summary of the state of the art ca. 2019 here. My impression is that there's a lot to be done, because we haven't yet figured out how to extract the semantics from a document effectively.
To that end, within category theory, there's work by Bob Coecke and his school using CQM (categorical quantum mechanics) to encode semantics. There's a recent conference to launch what they call QNLP. They've done a demonstration using a quantum computer recently that Bob has talked about (Zulip thread here), but I think machine learning should do just fine as well: there is some evidence of deep neural networks exhibiting what appears to be quantum entanglement by Yoav Levine and others (see also this paper at OpenReview.net for an example of how open peer review can take place).
As for informal maths, Mohan Ganesalingam has formalised a grammar of that in his thesis, and he and Timothy Gowers did an experiment using software written based on that grammar (paper here).
Thank you for the resources. I've been following Coecke's work. I was only vaguely aware of the rest. Are you actively working on this project? If so, what are some of the open problems you are working on?
(=_=) (Jun 03 2020 at 04:15):Oliver Shetler said:
How can we remove the authors' opinions from the equation without removing the authors' agency?
I think you have a rather broad definition of what the authors' "opinions" mean in this context, but perhaps I wasn't being clear about what "opinions" mean for me. By "opinions", I mean the decisions made by the authors as to what to include or exclude (the "lies" of commission and omission in your language) in the abstract. I do NOT mean the claims, arguments and factual observations made by the authors in the course of their scientific work that are recorded in the paper. As I have outlined in my proposal, such a removal of those "opinions" can be done with the full participation of the authors themselves, using an abstractive summarisation tool of the kind I have proposed, so the authors have not lost any agency at all.
(=_=) (Jun 03 2020 at 04:27):Oliver Shetler said:
Are you actively working on this project?
No.
(=_=) (Jun 03 2020 at 04:45):Oliver Shetler said:
you think there is such a thing as an "objectively accurate summary" of a document.
Your subsequent arguments show that you've understood my proposal and agreed that such a thing can exist.
But I think an abstract is, by design, too short to be considered an "objective" compression of the paper.
I've proposed that we modify that design, because the current design is not good enough.
My understanding is that an abstract ––usually a single paragraph––is far too short [...]
How long is a single paragraph? It's like asking how long a piece of string is. I've seen paragraphs that are several pages long.
Hence, the author should have the final say as to what goes into his or her abstract.
I agree, but I think the authors should have some assistance in making those decisions, and that those decisions should not lead to the deterioration of the accuracy of the abstract.
Having said this, I think that one could check how accurate an abstract is––in a very literalistic sense. That is to say, one could check for lies of commission, but not lies of omission.
Your subsequent proposal is capable of checking for both sorts of "lies".
One could also measure how comprehensive an abstract is. [...] If we had tools to measure those things, it would be a great tool for reviewers. I can imagine a new wave of desk rejections on the grounds of false claims in abstracts.
It may be possible with an extension of the current NLP technology. What current NLP tech does is to transform text into vectors in some high-dimensional metric space, and then use the metric to estimate distances. One could conceive of extracting the arguments in a text, transforming those arguments into vectors, and then using a metric to estimate the discrepancies you've mentioned in your proposal.
(=_=) (Jun 03 2020 at 04:50):Oliver Shetler said:
Access to such a tool could also make writing accurate abstracts much easier. I'll bet a lot of scientists would make tweaks based on a grammarly-like abstract checking tool.
Yeah, that's the hope.
I'll bet it would also lead to a wave of algorithm hacking, just like people resume hack due to the use of algorithms by hiring managers. But maybe it could still be an improvement.
Yeah, people are optimisation machines, so some exploitation is inevitable. It's still an improvement because the existence of accurate abstractive summarisation tech can help researchers become more productive. What academia is suffering from now is an information deluge, which is causing a major bout of indigestion, partly in the form of a replicability crisis. Allowing researchers to process research output more efficiently, i.e. by providing them with more informative and detailed summaries of papers, should go some way to alleviate this problem.
Oliver Shetler (Jun 03 2020 at 13:40):(deleted)
(=_=) (Jun 06 2020 at 03:18):Rongmin Lu said:
Morgan, I generally agree with most parts of your comment, except for this:
[Mod] Morgan Rogers said:
we (modern) scientists are loath to appeal to authority in our arguments, for example
[...] And I've actually seen (recent!) maths stuff in which references to "a famous theorem" are made without actual citations!
A related example was noticed by David Michael Roberts, who was discussing a CT paper on #general > Every cat is one of ordinals. He wrote:
The more I look at this paper I wonder how it got though refereeing. There's three 'It is evident', one 'details left to the reader' and one functor that is just declared to have some non-obvious property and no example of such a thing given.