Category Theory
Zulip Server
Archive

You're reading the public-facing archive of the Category Theory Zulip server.
To join the server you need an invite. Anybody can get an invite by contacting Matteo Capucci at name dot surname at gmail dot com.
For all things related to this archive refer to the same person.

Stream: theory: applied category theory

Topic: imaging systems engineering

David Egolf (Dec 04 2020 at 20:06):

I am interested in better understanding imaging systems structurally.
For example, I would like to understand:
-Can we build up more complicated systems for forming images by sticking together smaller systems?
-Is there an underlying mathematical similarity between different imaging systems? (And if so, can we translate understanding about imaging from one system to another?)
-What enables imaging systems to work? If we stick together a bunch of pieces to form an imaging system, when will the resulting system also work properly? How does information flow around between the different pieces of imaging systems?

I have been doing research in an engineering medical imaging lab. My experience is that thinking of new ideas for imaging systems is often hit-or-miss, and implementation for testing is often slow and expensive. I'm wondering if answering some of the above questions will help accelerate concept iteration and better enable cross-context communication (e.g. between MRI researchers and ultrasound researchers).

My training is in engineering. However, I've seen expressed here the idea that category theory has something to contribute to engineers. I'm hopeful that its ability to provide a language for structure and for translation between structures might be of some use for this problem.

I'm just getting started, and am wanting to understand what work has already been done in this direction, and what categorical ideas I need to become more familiar with in order to make progress on this. However, this feels a bit daunting as I've not done a project that makes use of category theory before. Honestly, I'm not even sure what kind of coarseness I should be trying to model imaging systems at!

Is there somewhere a resource along the lines of "Using Applied Category Theory: An Engineer's Guide"? Failing that, perhaps there are some really good examples of how people have modelled systems using categories?

How can I get started as an (imaging system) engineer with applied category theory?
Any thoughts are much appreciated.

Jules Hedges (Dec 04 2020 at 20:16):

So right now there's a few engineering-focussed people in the applied category theory, but not many. They tend to be focussed on systems engineering, which might be good because it sounds to me like you're really asking systems engineering questions

Jules Hedges (Dec 04 2020 at 20:17):

You definitely won't be able to find an off-the-shelf answer to these questions, I think this stuff is still in the basic research stage

Jules Hedges (Dec 04 2020 at 20:18):

(One consequence of this fact is that this would be totally welcome and on-topic in #practice: applied ct)

Eric Forgy (Dec 04 2020 at 20:26):

Fellow "engineer" here :wave:

You might want to have a look at https://www.algebraicjulia.org/

Notification Bot (Dec 04 2020 at 21:31):

This topic was moved here from #learning: questions > Imaging Systems Engineering by Matteo Capucci

Fabrizio Genovese (Dec 05 2020 at 10:57):

I agree wth Jules, there's certainly an ongoing effort in bringing more engineering into ACT at the moment, but there's still so much work to do. Fields that are already active in this sense are mainly related to dynamical and cyberphysical systems.

Fabrizio Genovese (Dec 05 2020 at 11:01):

That said, one important issue in ACT right now is that many of us are pure mathematicians that would like to help by applying categorical techniques to various fields, but they do not know where to start: Interaction with domain specialists is paramount right now to bootstrap things. So, if I may suggest a possible course of action, this would entail exposing your problem (even handwavily), hope to get some interaction with some member of the community, and then maybe start working together on something more formal :smile:

Georgios Bakirtzis (Dec 05 2020 at 16:19):

@David Egolf I am writing something like this as part of my dissertation which I will share as soon as I have a draft; meanwhile check this out https://bakirtzis.net/talon/. @Gioele Zardini is also doing some work in this area

David Egolf (Dec 05 2020 at 17:59):

Thank you all for the helpful responses. :smile: I look forward to learning from this community and seeing what we can come up with together. As I work on this in the coming days, I'll take a stab at describing the goal in some more specific detail to provide a concrete starting point for (hopefully) some fruitful interaction.

Jules Hedges (Dec 05 2020 at 18:04):

I'm no engineer but the first question I would ask is, what do you think could be special about imaging systems in particular to distinguish them from general systems, from the perspective of what category theory could be useful. This could either be from something inherent to the systems, or from the type of questions people like to ask about them

John Baez (Dec 05 2020 at 18:25):

That's a good question; also perhaps a hard question, depending on how much @David Egolf has thought about imaging systems versus other systems, particularly from a mathematical (i.e. abstract) viewpoint.

John Baez (Dec 05 2020 at 19:26):

There's a lot of nice category theory lurking in elementary optics, by which I mean the theory of lenses - actual lenses, not the things category theorists call lenses. Lenses can be seen as morphisms in a category, and to a first approximation they give symplectic linear maps. This is discussed at the start of Guillemin and Sternberg's Symplectic Techniques in Physics. But any serious sort of modern imaging system would go much further than this, I imagine.

John Baez (Dec 05 2020 at 19:30):

A bet someone interested in the physics of optics could have a lot of fun with this... but that might not be what's needed for practical purposes!

Eric Forgy (Dec 05 2020 at 19:47):

Fabrizio Genovese said:

That said, one important issue in ACT right now is that many of us are pure mathematicians that would like to help by applying categorical techniques to various fields, but they do not know where to start: Interaction with domain specialists is paramount right now to bootstrap

It is fun to see worlds collide with CT solving engineering problems. There is still a significant gap between ACT and engineering with pure mathematicians on side and engineers on the other, but this community seems to be a great place to make these necessary connections.

Fabrizio Genovese said:

So, if I may suggest a possible course of action, this would entail exposing your problem (even handwavily), hope to get some interaction with some member of the community, and then maybe start working together on something more formal :)

What would be the most appropriate way for me to do this for applied problems I'm interested in? Create a topic here in #practice: applied ct ?

Fabrizio Genovese (Dec 05 2020 at 20:25):

Eric Forgy said:

Fabrizio Genovese said:

So, if I may suggest a possible course of action, this would entail exposing your problem (even handwavily), hope to get some interaction with some member of the community, and then maybe start working together on something more formal :smile:

What would be the most appropriate way for me to do this for applied problems I'm interested in? Create a topic here in #practice: applied ct ?

Most likely yes, or you could consider posting somewhere in #general , to which everyone is automatically subscribed

Jules Hedges (Dec 05 2020 at 20:55):

I'd propose staying here. This stream only has half as many subscribers as general, but the average ACTist thinks quite differently to the average normal CTist

Matteo Capucci (he/him) (Dec 05 2020 at 23:06):

(Also people do not subscribe to streams for a reason :laughing: so force-feeding them whatever they didn't subscribe to in #general defeats the purpose of having multiple streams)

David Egolf (Dec 08 2020 at 02:08):

A thought provoking question was asked above, along the lines of: What characterizes an imaging system as compared to other systems? Answering this will help to create a model with sufficient detail (specialized to imaging systems) that adds complexity specific to answering questions about imaging system.

I had some thoughts on this topic, although they are not precise.

I am most familiar with ultrasound imaging, and will use examples from that context to help provide concreteness. In brief, ultrasound imaging works like this:

1. There is some target we want to image (e.g. a portion of the heart in a patient)
2. The transmit array creates a pulse of ultrasound (high frequency sound) which then propagates into the body
3. The ultrasound scatters off the heart, this scattering being induced by variations in density and the speed of sound
4. The resulting echo propagates outwards and part of this echo is sensed by a receive array of sensors, each which generates a voltage over time signal. These receive signals are saved in a computer.
5. The recorded receive data is then processed to form an image (e.g. of the heart)

So, what distinguishes an imaging system?

First, imaging systems are communication systems. Here is a diagram form a 1948 paper by Shannon of a communication system:
Communication System

Consequently, imaging systems have a starting point and an ending point. The start is where the message is generated (e.g. biological processes generate the heart in some state). The end is where an image is displayed to the user (e.g. the display screen of an ultrasound machine). Diagrammatically, the bare-bones structure of an imaging system looks like this: o---o---o---o---o.

Second, imaging systems are often composed of smaller imaging systems. For example, consider the human vision system. This system is composed of two eyes, together with the brain. If we close an eye, the remaining subsystem is still able to form images. On a smaller scale, the rods and cones of the eye can be viewed as inducing even smaller imaging systems. The whole imaging system is then formed by combining together these smaller imaging pieces of hardware, together with appropriately more sophisticated processing. Similarly, in an ultrasound setting, the transmit array which generates the ultrasound wave is composed of many small transmitting elements, and the receive array is composed of many small sensing elements. On a higher level, multi-modal imaging systems seek to combine information from multiple sources to provide a more comprehensive picture (e.g. a camera and an ultrasound system in an endoscopy probe). Diagrammatically, the bare-bones "gluing" process might look like this: two simpler imaging systems o---o---o---o---o and o---o---o---o---o together yield a more complicated imaging system:
Composite Imaging System

Note that multiple sub-imaging systems can be combined across time, not just across space. For example, in ultrasound imaging, it is common to send pulses of ultrasound towards the target from several different angles. These different pulses are usually sent at different times. The final image formed then incorporates all the data collected from each angle.

Third, we care about the imaging speed and imaging quality of an imaging system. We care about the speed at which an image of a given quality can be produced, and the quality of the images produced at a given speed. Quality is hard to assess in general, but includes things like "contrast" and "resolution", and is probably related to the mutual information between the messages generated (e.g. the distribution generating heart tissues) and the images produced (e.g. ultrasound images of the heart). Imaging speed corresponds to how long it takes to generate an image. In an ultrasound setting, this often corresponds to the number of times we send ultrasound transmission into the body.

So, in summary: imaging systems are communication systems, often composed of smaller imaging systems glued together, and we care about their speed and image quality. The hope is that noting that they are distinguished in this way will help us model them effectively.

I'm guessing this is not at the level of precision we need to start making mathematical models right away, but hopefully it can provide a springboard for further discussion.

David Egolf (Dec 09 2020 at 19:55):

I've started to try and categorify things! I don't know much category theory, so I would be very happy to hear about better ways to do this.

As a first attempt, we can model imaging systems like this:
Imaging System Model

where M is the message, t is the transmission, S is the signal, o is the observation, R is the received data, p is the processing, and I is the image. For now, these are just objects and arrows in some category.

I want to define a category that allows us to compare imaging systems. As a first guess, vertical maps look like this:
Map $\rho$ between Imaging Systems

where each square is required to commute. I believe that this creates a category of imaging systems. As far as I can tell, there is an arrow from imaging system A to imaging system B when you can calculate everything in B from things in A. So, if $A \mapsto B$ then A "observes at least as much" as B.

I want to be able to describe the process of sticking together two imaging systems, as many real imaging systems are composed of many smaller imaging systems combined. One way to maybe do this is with a span that is a sort of terminal object, taking inspiration from the little bit I've heard about modeling with structured cospans:
Combining imaging systems

Here A and B are smaller imaging systems. A+B is then the imaging system defined by being the "least observant" system that observes at least as much as A and B, as it is required to observe no more than any imaging system C that observes at least as much as A and B.

So, I think this defines a way to combine two imaging systems. There are limitations with this, but I'll stop here for now in case someone has some thoughts about this approach.

John Baez (Dec 09 2020 at 20:07):

A+B would be the imaging system that you get by putting A and B on the same table, so to speak, not connecting them up at all.

John Baez (Dec 09 2020 at 20:08):

There's not usually a morphism from A+B to either A or B in category theory, so your diagram look a bit off. There are morphisms from A and B to A+B: these say how the parts are included in the whole. The resulting diagram

$A \to A+B \leftarrow B$

is an example of a cospan.

John Baez (Dec 09 2020 at 20:09):

You drew the (in general nonexistent) span

$A \leftarrow A+B \to B$

John Baez (Dec 09 2020 at 20:10):

Cospans and structured cospans get more interesting when you use them to connect systems to each other. Any sort of machine or system made of parts has a good chance of being describable using these. That's why I've been obsessed with structured cospans.

John Baez (Dec 09 2020 at 20:11):

Now I've got to get back to writing another paper on structured cospans. :upside_down:

John Baez (Dec 09 2020 at 20:12):

On a trivial note: you don't need to draw those big black dots in your diagrams; people usually just draw the letters.

David Egolf (Dec 09 2020 at 21:07):

Thanks @John Baez ! I really appreciate you taking the time to help out.

A+B would be the imaging system that you get by putting A and B on the same table, so to speak, not connecting them up at all.

I'm realizing that my notation is backwards. I realize now that $A+B$ usually means a coproduct, not my weird definition of $A+B$ as a terminal span over objects $A$ and $B$. I believe my definition corresponds to a product, so I should write my diagram like this:

Combined imaging system V2

Please correct me if I'm wrong, but I think that $A + B$ as a coproduct corresponds more to an intersection of imaging systems $A$ and $B$. This is because there is only a morphism from $A$ to $A + B$ if the information in $A$ is sufficient to calculate everything in $A + B$. So then the coproduct would be the "largest" possible imaging system that can have every part of it calculated from $A$ and also from $B$.

For clarity, this is now what I'm understanding as $A+B$ under my corrected notation:
An intersection of imaging systems?

John Baez (Dec 09 2020 at 21:20):

Your diagrams are now the correct definitions of product and coproduct. But A+B is not like an intersection, usually: in the category of sets it's the "disjoint union", where we put two sets side by side.

John Baez (Dec 09 2020 at 21:21):

The point is that both A and B map into A+B, so it has to be "big enough to contain both".

Morgan Rogers (he/him) (Dec 09 2020 at 21:48):

@John Baez from the definition that @David Egolf gave of morphisms, the existence of a morphism $A \to B$ means "$A$ is at least as observant as $B$", so perhaps the reason it seems strange to you is because it is dual to what you might expect: a morphism means that "every observation made by $B$ could have been made by $A$", or that there is a mapping in the other direction $\mathrm{obs}(B) \to \mathrm{obs}(A)$.

Morgan Rogers (he/him) (Dec 09 2020 at 21:49):

That is, the dual of David's category can be understood as a concrete category where we would indeed expect the coproduct to behave as a disjoint union. But what David has said seems to make sense to me based on his setup!

David Egolf (Dec 09 2020 at 21:50):

I think in my particular category defined above, $A+B$ is acting like an intersection. This is because in the imaging system category defined above, $A \mapsto B$ means that $A$ is a "bigger" (more observant) imaging system than $A$. So if $A$ and $B$ both map into $A+B$ it means that they are both big enough to contain $A+B$.

I suspect this is another notational problem, where I'm working in a category with the arrows running the opposite way to normal convention. We can fix it like this:
-Let $A_1$ and $A_2$ be imaging systems with maps $(t_1, o_1, p_1)$ and $(t_2, o_2, p_2)$. Let M, S, R, and I be objects corresponding to messages, signals, received data, and images.
-Then $\rho$ is a map from $A_1$ to $A_2$ exactly when the following diagram commutes:

morphism definition V2

Now if we say $A_1 \mapsto A_2$ this says that $A_2$ can calculate everything about $A_1$. So in this sense $A_1$ is smaller than $A_2$. To glue together two imaging systems, we want the smallest system that contains both. Under the new definition of morphisms (which has reversed all arrows), this is now the coproduct $A+B$. A co-span over $A$ and $B$ is now any imaging system that can calculate everything about $A$ and $B$.

David Egolf (Dec 09 2020 at 21:53):

@[Mod] Morgan Rogers Yes, I think that is the difference: we were assuming arrows running in opposite directions!

Morgan Rogers (he/him) (Dec 09 2020 at 21:55):

@David Egolf I personally think you should stick to your original definition, since the data matches the direction of the arrows, which is always preferable. But it might be useful to keep the hypothetical contravariant functor $\mathrm{obs}$ I described on your category (to $\mathbf{Set}$, say) to hand, since it can be helpful for reasoning about what limits etc. should be in your category! :grinning_face_with_smiling_eyes:

John Baez (Dec 09 2020 at 22:25):

Oh, okay! So David working in the opposite category from me.

David Egolf (Dec 09 2020 at 22:44):

@[Mod] Morgan Rogers I like the concept of $\mathrm{obs}$. Could you maybe elaborate a bit on how it can help determine what kinds of limits are interesting or wanted? I'm wondering if the idea is to look at limits that are in $\mathbf{Set}$ and then look at the corresponding limits in an imaging system category, but I'm unfamiliar with the design flow one takes when trying to build an interesting category.

David Egolf (Dec 11 2020 at 03:04):

There are some problems with the above approach. I'm describing them here on the off-chance that someone else knows of ways to get around them.

I'm realizing that this model for imaging systems is a bit coarse. We don't see explicitly in the model all of the pieces that an engineer is choosing to put into the system. For example, the pieces of hardware used as sensors are only incorporated implicitly in the map $o$, which describes how observations are performed.

As a (probably related) problem, sticking together two imaging systems using the product (or even just a span) is too idealized to model how we often stick together imaging systems in practice. This is because the definition of a morphism from imaging system $A$ to $B$ requires $A$ to be able to observe everything that $B$ can observe, and this is too stringent a condition. In practice, when we stick on another imaging component to our current system (e.g. add another sensor), the resulting system often can no longer observe everything that it could originally observe. So, if $S$ is the imaging system after we add another sensor, and $A$ was the original imaging system, often there will be no morphism from $S$ to $A$. So, we can't relate the new system to the old system under this definition of morphism.

Both of these reasons make me think we need a new definition of morphism between imaging systems, and probably a new definition of imaging systems too. Ideally these would be granular enough to relate all the components we want to design over, while also being relaxed enough to allow for morphisms between imaging systems in the case when neither of the two systems is strictly more observant than the other.

As a reminder, here is the (unsatisfactory) way I've been defining morphisms between imaging systems:
Current morphism definition.png

(t is transmission, o is observation, and p is processing)

Can we do this in a more "relaxed" way so that morphisms more often exist between imaging systems?

David Egolf (Dec 11 2020 at 22:45):

I'm realizing a few things:
-We need ways to "mix together" parts of imaging systems, not just snap them together like Lego blocks
-For an imaging system model to be useful for design, it should expose hardware and processing
-Before considering "translation" maps between imaging systems, we need a better model of an imaging system
So, I am hopeful that reading some more of "Seven Sketches in Compositionality" will be the way to make some progress. I would like to be able to make diagrams for imaging systems like this one from the preface:
System with Mixing

If nothing else, I am hopeful that the posts I'm making here will provide an example of some of the challenges engineering people can face as they try to figure out how to apply category theory to engineering design.

Morgan Rogers (he/him) (Dec 12 2020 at 13:56):

David Egolf said:

[Mod] Morgan Rogers I like the concept of $\mathrm{obs}$. Could you maybe elaborate a bit on how it can help determine what kinds of limits are interesting or wanted? I'm wondering if the idea is to look at limits that are in $\mathbf{Set}$ and then look at the corresponding limits in an imaging system category, but I'm unfamiliar with the design flow one takes when trying to build an interesting category.

The idea behind $\mathrm{obs}$ was that, looking at the higher level picture rather than the details of the object structure, I interpret the 'imaging system category' you describe as a "hardware category", where the objects in some sense represent measurement devices. So $\mathrm{obs}$ would send a device to the set of measurements it can make, or possibly to something more structured than a set, such as a collection consisting of the sets of 'interpretable messages', 'signals', 'data', and 'image', plus the mediating maps between them that you originally described as being the internal structure of the objects. After all, if you're considering systems, there should be a range of messages that the system can interpret, and to me it makes more sense to associate to a system its maximal set of data rather than a single instance.

Then $\mathrm{obs}$ would be contravariant with respect to the comparison morphisms you originally described because a strictly more powerful system can interpret all of the messages etc. that its less powerful counterpart can, so we get a mapping, indeed an inclusion from the set of interpretable messages (etc.) of the latter into those of the former. In the end I agree that this is not the most useful definition of morphism, however.

Morgan Rogers (he/him) (Dec 12 2020 at 14:23):

Here is an abstract suggestion for a framework that you might be able to adapt or work with abstractly, based on what you've described so far.

First, set up a category that represents the "image processing pipeline". A simple option is $[\mathcal{C},\mathbf{Set}]$, where $\mathcal{C}$ is the simple category $M \xrightarrow{t} S \xrightarrow{o} R \xrightarrow{p} I$ you described earlier; an object consists of a series of 4 sets with functions between them. A more nuanced set-up might have vector spaces in place of sets, or even a different type for each object and a morphism with specified properties between these, to properly capture the respective structures of the different pieces. To bound the space of things which count as messages, you might want to fix the set/space $M$. Whatever you go with, let's call this category $\mathcal{I}$.

Next, we set up a category of imaging systems over $\mathcal{I}$. The objects of this category will represent physical systems; the precise nature of the morphisms I shall leave up to your discretion. Since imaging systems can be "composed in parallel", it makes some sense to make this a monoidal category. Since the parallel composition can be imperfect (and I want to avoid extra hassle at the next stage) I would advise making your morphisms flexible enough to make sense of a morphism $A_1 \otimes A_2 \to A$ which compares an idealised parallel composition of two systems to the realistic composite system you actually care about. Let's call this category of imaging systems $\mathcal{S}$.

The final ingredient is a monoidal functor $\mathrm{obs}:\mathcal{S} \to \mathcal{I}$, where in the simplest situation the monoidal product on $\mathcal{I}$ is a coproduct (on the respective sets), but that can be substituted for whatever way of composing pipelines makes sense. Then the application of $\mathrm{obs}$ to the morphism $A_1 \otimes A_2 \to A$ mentioned earlier will be a (collection of) function(s) expressing the loss of information for the system $A$ compared with the idealised composite $A_1 \otimes A_2$.

Morgan Rogers (he/him) (Dec 12 2020 at 14:35):

The main goal of structuring a model in this way is that you can encode the subtleties of how the hardware fits together in $\mathcal{S}$ and the subtleties of the processing in $\mathcal{I}$, so these aspects are accessible but distinct. On the other hand, the set-up allows you to hide as much of that structure as is useful when reasoning about these systems at a high level, for example when you're in the early stages of design of a new system and you don't know exactly what ingredients will be present yet.

That last point is what I view as the real motivation for making a categorical model at all: you can refine the categorical model in tandem with your physical one, distilling the crucial features into an abstract setting where you can reason about how to proceed, or you can substitute or extend $\mathcal{I}$ or $\mathcal{S}$, or indeed tweak the functor $\mathrm{obs}$ to better reflect the specifics of a class of systems you care about, while carrying across any results that you were able to derive more abstractly using the features common to all of the systems you care about.

Morgan Rogers (he/him) (Dec 12 2020 at 14:36):

I hope that was helpful..! Good luck with your reading of 7 Sketches :tada:

John Baez (Dec 12 2020 at 17:45):

@David Egolf - you might find it helpful to take a certain limited very precisely defined class of imaging systems and mathematically formalize the things you actually do with them. "Imaging systems" seems like a broad and rather vague class to me, in part because I don't know much about imaging systems. What are we actually talking about here? Can you pick a very few kinds of imaging systems and work out the category theory of what we actually do with these, in great detail?

I found this approach helpful when working on the category theory of electrical circuits. Circuits made of just resistors are sort of boring and well-understood, but I decided it was worthwhile understanding them using category theory, as a warmup before studying more complicated circuits. And indeed it turned out to be very useful! My student Brendan Fong spent a couple of years on this and invented the framework of decorated cospans, which turned out to be useful very generally for describing systems made by sticking components together. Using this we wrote a paper on passive linear electrical circuits - e.g. resistors, inductors and capacitors.

The whole project took about 5 years, which shows that it can take time to figure out even simple things. But we learned a lot of lessons that apply much more generally, and Brendan explained them quite well in his thesis:

• Brendan Fong, The Algebra of Open and Interconnected Systems, Ph.D. thesis, University of Oxford, 2016

John Baez (Dec 12 2020 at 17:48):

I think the ideas he discovered could be useful to you, but even more I think it could be useful to follow the same strategy, where you start by focusing on a precisely limited small sub-class of the problems you're interested in, and then generalize later.

David Egolf (Dec 15 2020 at 01:09):

@[Mod] Morgan Rogers Thanks for the detailed response! It was helpful. I don't fully follow all the details of what you said, but I'm hopeful that Seven Sketches will help with that (e.g. the definition of a functor from a monoidal category).

If I understand correctly, some of the major points from your post are:
-Modeling hardware and processing in separate categories might be useful
-Similarly, modelling general aspects (e.g. structure of information flow) separately from specific aspects (e.g. specific hardware used) and then relating the two (via a functor?) may allow for generalization of abstract insights to multiple concrete imaging systems
-We can use a monoidal category to talk about composing (abstract?) imaging systems in parallel, and use this setting to relate idealized and actual combination of imaging systems. Taking a functor from this category to the "specific realization" category then should translate statements about abstract information flow into language about maps in a concrete imaging system.

I will keep this scheme in mind as I follow John Baez's advice to tackle a very precise class of imaging systems!

David Egolf (Dec 15 2020 at 01:58):

@John Baez
Thank-you for your suggestions and the resources. They are helpful!

I agree that it will be valuable to focus on a smaller, precisely defined subclass of imaging systems. Before getting to that, I found your reflection on the vagueness of an "imaging system" thought provoking, and had some thoughts that might help define this better.
Here are some reflections, leading to a proposed definition for "image" and "imaging system" below:

1. Imaging systems don't always just have an "image" at the end. For example, an imaging system might generate a rough photograph, and then have further steps involving processing of this photograph. It seems arbitrary to say that the processed photograph is an image and the earlier one is not.
2. Imaging systems don't always fit well in the conceptual "message->signal->received data->image" pipeline. For example, I think horizontal composition of two imaging systems following this pattern is often meaningful. In this case, we more naturally have: "message->signal->received data->image->message->signal->receive data->image".
3. So, the four step pipeline with an image at the end has problems. To fix these problems we could maybe: (a) allow for any number of steps in the pipeline, (b) allow for the data at each step to be considered on equal footing, as an "image".

These reflections indicate a better need for definition of "image". In a medical imaging setting, we generally care about images inasmuch as they let us answer a question (e.g. "is there a cyst? is there a fracture? what is the shape of some structure?"). Often this involves human interpretation, but this is not necessarily always the case - for example a simple algorithm can calculate contrast between two areas of an image. So, let us define an image to be some data $\mathcal{I}$ together with a question $Q$, an estimator (interpretation) algorithm $E$, and a distribution of possible messages $M$. The "goodness" of the image is then how well the interpretation of the data $E(\mathcal{I})$ answers $Q$ for a given $m$. For example, this could be $\|E(\mathcal{I}) - Q(m)\|$ for an input message $m$.

We can now better define an abstract notion of an imaging system. We have some starting object $M$, which corresponds to all possible messages (with some probability distribution). Then we have some arbitrary finite number of processing step arrows producing data objects:
$M \to D_1 \to D_2 \dots \to D_n$
Some of these steps may involve processing in a computer, while others may involve transmission through a physical medium, for example.

So far this just describes some processing of messages $m$ in $M$. To make this into an imaging system, we need an interpretation algorithm $E$ and a question $Q$. If $D_i$ is not in the domain of $E$, then the first $i$ processing steps of the pipeline is said to be an imaging system of goodness zero. Otherwise, define some way of measuring the closeness between the distributions $(E \circ p_i \circ \dots \circ p_1)(M)$ and $Q(M)$, where $p_i$ is the $i_{th}$ processing arrow. This closeness is then the goodness of the imaging system up to the first $i$ steps.

This model for an imaging system allows for any number of steps, and removes the (I believe) artificial distinction between data that is an image and data that is not an image. It also provides a framework for talking about the "goal" of an imaging system: makes its goodness as high as possible given design constraints.

David Egolf (Dec 15 2020 at 02:11):

Pictorially, the basic structure of an imaging system now looks like this:
Imaging System V2

This example has three processing steps, although any (finite) number is allowed. $A$ is the "answer space", $Q$ is a question, and $E$ is an "estimator" trying to find the answer to the question from the processed data.

Morgan Rogers (he/him) (Dec 15 2020 at 08:47):

This new description sounds to me like a manually applied/engineered version of a machine learning algorithm for identifying features of an image. :hushed:

David Egolf (Dec 15 2020 at 15:41):

This new description sounds to me like a manually applied/engineered version of a machine learning algorithm for identifying features of an image.

That is an interesting connection. I suppose that in some cases the $p_i$ could correspond to message propagation into deeper layers of a neural network. I do suspect that machine learning algorithms are going to be central to the future of imaging system design (due to their strong ability to incorporate prior information), so it's good that this approach looks like it could model them.

John Baez (Dec 15 2020 at 21:21):

I like your new thoughts, David.

One thing I'd like to argue is that when you start talking about "pipelines" that you can connect end to end, like "message->signal->received data->image->message->signal->receive data->image", you are really talking about morphisms. It's no coincidence that you're drawing arrows here.

John Baez (Dec 15 2020 at 21:23):

The fact that you want to think of these pipelines as objects is not in contradiction.

John Baez (Dec 15 2020 at 21:23):

In a double category, one has morphisms that also act like objects, because there are "2-morphisms" going between morphisms.

John Baez (Dec 15 2020 at 21:24):

My approach to applied category heavily uses double categories.

John Baez (Dec 15 2020 at 21:27):

You may wish to avoid them, but if you need them if you:

1) want to talk about morphisms between imaging systems,

and

2) also want to talk about "connecting imaging systems end to end", like taking a camera filter and attaching it to the lens of your camera, or taking the jpeg produced by a camera and putting it into some image processing software.

John Baez (Dec 17 2020 at 20:27):

I see you're thinking, @David Egolf. Someday I'm sure you'll be convinced of my point. The basic point is this: whenever you draw anything using an arrow, you are secretly treating it as a morphism, even if you don't know it.

David Egolf (Dec 18 2020 at 01:26):

Thanks @John Baez for your thoughts on double categories.
I suspect you are right that I do want to use them. My thought process follows, which I provide here largely to convince myself, and also in case it is of interest to some people.

To begin with, I thought of an imaging "pipeline" (e.g. message->signal->receive data->image) as a category. The morphisms are the data transformation maps, and the objects are the data we're mapping between. Then, the idea was to relate two data processing pipelines using a functor. My intuition for a functor is a "translation" of structure at one level of abstraction to (some subset of) that structure at another level of abstraction. So, the idea was to be able to describe similarities in structure in different data processing pipelines. A functor $F$ would act like this:
Functor translating between data processing pipelines

However, I recently realized that we want to be able to horizontally compose the data processing pipelines underlying imaging systems. This is because there isn't really a canonical "finish" to one of these pipelines, and we can often feed data from one of these pipelines to another one. In an attempt to preserve the idea of each pipeline as a category, I was thinking of this scenario like this:
Horizontally stitching together data processing pipelines

The idea was to have a higher level category where each data processing pipeline is an object, and stitching them together horizontally corresponds to morphisms. However, this doesn't actually work. This is because in a category we need to be able to compose morphisms. Assume we have pipelines connected with these connecting data transformations like this: A -p-> B -q-> C. Then, if we populate pipeline $A$ with some data it's handing off to $p$, then this data arrives to $C$ after we apply $p$, all the processing in $B$, and $q$. There is no way to make a single map $q \circ p$ that passes data from $A$ to $C$ with all this processing being done (as would be needed for composition), as it doesn't know about the internals of processing pipeline $B$.

So, the horizontally composed pipelines don't really act like objects with respect to horizontal composition. As you pointed out, they do act like morphisms: we can compose them nicely to get a bigger processing pipeline. In addition, this avoids an artificial distinction between pipelines and "transition maps" feeding the output of one to the input of the other: every processing map is on an equal footing as a morphism.

However, I do still want to relate imaging systems to each other. So, why not use functors for this, just viewing the entire processing pipeline as a category? The main downside I can think of is that this seems conceptually awkward, as it would require us to change the category we are working in every time we add another step to our processing pipeline.

It sounds like double categories could do what I want in a streamlined way: allow for horizontal composition of pipelines (viewed as morphisms) and then still allow for comparison of pipelines, without having to constantly change the category we are working in as we tweak a pipeline. I'm not sure exactly how the "answers a question" part of imaging systems fits into all this, but this seems reasonable for the data pipeline part of the model.

John Baez (Dec 18 2020 at 15:39):

I like this story, @David Egolf, because many people have gone through this kind of mental process, but people rarely talk about the evolution of their thoughts, and how various desires push them in different directions.

For composing processing pipelines, and comparing pipelines, double categories may be good. People use them for electrical circuits, signal flow diagrams, etc and they work well.

John Baez (Dec 18 2020 at 15:40):

But what's the "answers a question" part of an imaging system? A simple concrete example would help me understand what you mean by that.

Morgan Rogers (he/him) (Dec 18 2020 at 15:41):

From how you drew the pipeline earlier, it sounds like the "question-answering" aspect could be formulated in terms of a universal property. The 'question' map $Q$ from your diagram is a morphism which we may not have perfect information about in general, $E$ is some 'extension' of that along the morphism $p_3 \circ p_2 \circ p_1$, which is to say the best approximation of $Q$ that we can obtain after performing some processing.

Morgan Rogers (he/him) (Dec 18 2020 at 15:49):

John Baez said:

But what's the "answers a question" part of an imaging system? A simple concrete example would help me understand what you mean by that.

Assuming that my machine learning analogy from earlier is close enough, here is an example:

• We start with some raw data, such as the raw output of an xray scan of someone's hip.
• We want to go from this data to the answer of a question, such as "is the hip broken?"
• There is a 'perfect information' map that sends all possible iterations of xray data to their respective diagnoses, which is to say the space {"yes", "no", "that's not a hip, it's a fossilized t-rex fibia", ...}
• However, this map is hard to present in terms of the raw data! So we have an image processing pipeline.
• Processing inevitably loses information, but the aim is that we should be discarding irrelevant information, so that we can still get a faithful approximation to the 'perfect information' map.

In this example, the raw xray data is transformed into one or more images, and that data is passed to a doctor, and the doctor makes a diagnosis. Or the image is processed further to highlight any relevant features, resulting in a more accurate diagnosis.

Morgan Rogers (he/him) (Dec 18 2020 at 15:51):

In any case, the image is being processed with a final "question-answering" purpose in mind, and any analysis of the quality of the processing pipeline will depend on how well it fulfils this purpose!

John Baez (Dec 18 2020 at 16:01):

Thanks. This seems a lot more complicated than "image processing pipelines" - now we've got doctors and their diagnoses in the system. I think we could simplify your example while preserving some essential features by making the question be "is the image a dark square on a light background?"

John Baez (Dec 18 2020 at 16:02):

Even so, we have now left the realm of "image processing pipelines" unless we imagine an image processing device (perhaps a person) that takes an image as input and returns a bit as output: a yes or no answer to this question. (Or maybe a third alternative, "unsure".)

Morgan Rogers (he/him) (Dec 18 2020 at 16:06):

That last suggestion is roughly what I was hinting at. Once we impose some conditions on what the objects are (a simple choice is big-but-finite-dimensional vector spaces), we can interpret the output, or answer to the question, as data in a comparable form to the input.

Morgan Rogers (he/him) (Dec 18 2020 at 16:07):

There's no reason not to include the doctor in the pipeline, as far as I can see! The human eye was the most sophisticated image processing system available for millennia, after all (pun incidental, but not deliberately avoided).

David Egolf (Dec 20 2020 at 02:01):

Thank-you for your thoughts! @John Baez @[Mod] Morgan Rogers

I think we do want to allow "interpretation" steps into the data processing pipeline, as it feels arbitrary to exclude them. At the same time, an imaging system is more than just a data processing pipeline - it has a goal. As described below, I think we can define an imaging system as a related pair of data processing pipelines.

Define a data processing pipeline to be a finite sequence of morphisms starting from an object $M$ and continuing to objects $D_i$: $M \to D_1 \to D_2 \to ... \to D_n$.

These processing steps can involve the formation of "pictures" and interpretation of these. Formation might involve generating a collection of "brightness" values associated with points in space. Interpretation could involve computing the difference in brightness between two points.

So far this is just processing of data, but an imaging system has a purpose. In particular, it produces data that can help us measure a property of interest. So, to say that we have an imaging system we need two things: 1. some data and 2. a property of interest. We also need a relationship between these two that captures how well the data could help us measure the property of interest.

We can produce a property of interest by using a data pipeline $P$ (the "property oracle"). This takes in some initial data and spits out the value of the property of interest. We can use a second data pipeline $E$ (the "estimator") to produce some other data, which we hope is helpful for finding the value of the property.

Finally, we can then say that the estimator $E$ is an "imaging system" with respect to the property oracle $P$, together with a measurement $\mu$ of how much the output of $E$ could in theory help us determine the output of $P$. The imaging system has three parts: $(P, E, \mu)$. From a design perspective, it seems helpful to decouple the "potential usefulness" of $E$ from its usefulness under a given implementation for using $E$.
An example imaging system

As an example, a camera that takes a photograph of a house $H$ is an imaging system with respect to the property "the color of house $H$", together with some measurement $\mu$ of how well the photograph could in theory help us determine that color.

A challenge with this approach is how to define a good way to compare the "property" and "estimator" pipelines with $\mu$. I don't know how to do this in general. Working with a more specific class of imaging systems I think will help with this, but I'll stop here for the moment.

David Egolf (Dec 22 2020 at 00:44):

I am wondering if we can make an analogy between circuits and data processing networks. In the case of circuits, we can use a voltmeter to measure a voltage between two points. Perhaps in the case of data processing networks we can use an "information meter" (corresponding in spirit to $\mu$ above) to compare how much we know about one point in terms of observation at another point. If this is the right strategy, then we can just use the structure discussed earlier for data pipelines (a monoidal double category) and then consider how to make these measurements on it.

The challenge is still to think of a good way to measure how much the data at one point in the network relates to data at another point. Hopefully whatever mechanism we arrive it is compositional; so the goal should be to relate adjacent objects in a pipeline to begin with. I'm currently thinking about how to do this in a vector space setting.

John Baez (Dec 22 2020 at 05:57):

David Egolf said:

I am wondering if we can make an analogy between circuits and data processing networks. In the case of circuits, we can use a voltmeter to measure a voltage between two points. Perhaps in the case of data processing networks we can use an "information meter" (corresponding in spirit to $\mu$ above) to compare how much we know about one point in terms of observation at another point. If this is the right strategy, then we can just use the structure discussed earlier for data pipelines (a monoidal double category) and then consider how to make these measurements on it.

The challenge is still to think of a good way to measure how much the data at one point in the network relates to data at another point.

Mutual information can take any two random variables and tell you how much information they have in common. It's a powerful tool. But if I were you, I'd start by getting the double category you're alluding to worked out in an simple example or two.

David Egolf (Dec 28 2020 at 01:29):

I'm trying to write down the simplest double category that describes (some) data processing pipelines. I'm pretty new to all things category theory, so this is a slow process.

To form a double category, we need two categories as building blocks: the object category and the morphism category. As a starting point, I said the morphism category has objects corresponding to finite sequences of linear maps between finite dimensional vector spaces:
An object of the morphism category

For two objects corresponding to pipelines of the same length, we can define a morphism $\alpha$ as corresponding to a commutative diagram like this (which looks like a natural transformation):
A morphism in the morphism category

However, it seems more challenging to define a morphism between two pipelines of different lengths.
We can imitate the definition of a Petri net (given on slide 10 here) to define a data pipeline in a way that can describe pipelines of different lengths:

Data pipeline definition

Here $\mathsf{T}$ is a discrete category corresponding to a set with a finite number of elements. We think of each element as corresponding to a "transition" between vector spaces. $\mathsf{V}$ is the discrete category of finite dimensional vector spaces, and $\mathsf{L}$ is the discrete category with objects corresponding to linear transformations between finite dimensional spaces. $s$ and $t$ are functors that correspond to functions supplying a "source" and "target" for each transition. $p$ is a functor corresponding to a function that specifies a linear transformation for each transition.

David Egolf (Dec 28 2020 at 01:30):

We can view a data pipeline as the image of a functor to $\mathsf{Cat}$ from this simple category:
Shape category

So, maybe we can define a morphism of data processing pipelines as a natural transformation $\alpha$ between two functors into $\mathsf{Cat}$ from the shape category, having image of the form described above. Then, a morphism of data processing pipelines $\alpha$ corresponds to this commutative diagram:
Morphism

David Egolf (Dec 28 2020 at 01:34):

I'm not sure if this is what we want as a morphism of data processing pipelines. I'll have to try this out in some small concrete cases next time...

Any thoughts are very much welcomed, as always!

John Baez (Dec 28 2020 at 01:41):

However, it seems more challenging to define a morphism between two pipelines of different lengths.

It might help to think about what you want in applications. I'm guessing a pipeline like this

$V_1 \xrightarrow{p_1} V_2 \xrightarrow{p_2} V_3 \xrightarrow{p_3} V_4$

is supposed to represent a gadget that implements three separate stages of data processing: $p_1, p_2$, and $p_3$. Is that the idea?

John Baez (Dec 28 2020 at 01:42):

If so, what's a morphism from this to some other sort of pipeline supposed to represent? Is the morphism supposed to care about the individual stages of the processing or just the end result?

John Baez (Dec 28 2020 at 01:43):

For example, do you want there to be a morphism from the above pipeline to this one:

$V_1 \xrightarrow{p_3 p_2 p_1} V_3$

where you do all 3 stages of processing all at once?

John Baez (Dec 28 2020 at 01:44):

Conversely, do you want there to be a morphism from

$V_1 \xrightarrow{p_3 p_2 p_1} V_3$

to

$V_1 \xrightarrow{p_1} V_2 \xrightarrow{p_2} V_3 \xrightarrow{p_3} V_4$ ?

John Baez (Dec 28 2020 at 01:46):

(I still think that describing a really simple concrete example of what you're talking about would be the best way to speed up settling all these questions.)

David Egolf (Dec 28 2020 at 02:46):

John Baez said:

I'm guessing a pipeline like this

$V_1 \xrightarrow{p_1} V_2 \xrightarrow{p_2} V_3 \xrightarrow{p_3} V_4$

is supposed to represent a gadget that implements three separate stages of data processing: $p_1, p_2$, and $p_3$. Is that the idea?

Yes, that is the idea.

John Baez said:

If so, what's a morphism from this to some other sort of pipeline supposed to represent? Is the morphism supposed to care about the individual stages of the processing or just the end result?

The reason I want to have maps between data processing pipelines is to help us understand similarities between imaging systems in different contexts. For example, say we have a sequence of data processing steps that makes a photoacoustic image. In an ultrasound imaging setting, there might be an approach that is similar in spirit. I would then want there to be a morphism from the photoacoustic imaging system to the similar ultrasound imaging system.

When designing an imaging system, I would care about similarities between different steps in processing pipelines. For example, there might be an interesting approach for final processing done in an photoacoustic context that I could grab and try out in an ultrasound context - even though all the earlier steps in the imaging process could be different. So, I do think we care about similarities between individual steps, and not just the end result.

I think I would want both of the morphisms you describe to exist, but probably working out a concrete example is going to be helpful to determine that. Next on the list will be trying a more concrete example... probably with echoes in a 1D setting!

John Baez (Dec 28 2020 at 04:51):

The reason I want to have maps between data processing pipelines is to help us understand similarities between imaging systems in different contexts. For example, say we have a sequence of data processing steps that makes a photoacoustic image. In an ultrasound imaging setting, there might be an approach that is similar in spirit. I would then want there to be a morphism from the photoacoustic imaging system to the similar ultrasound imaging system.

The problem is that a morphism from X to Y always says something much more specific and precise than "X and Y are similar in spirit" ${}^1$

Thus, to define the morphisms in a category, you have to think about exactlywhat you want them to be. When you're done you'll be able to say things like "there are exactly 6 morphisms from X to Y" and know what this means.

John Baez (Dec 28 2020 at 04:54):

${}^1$ An exception is when your category is of a special sort called an "equivalence relation", where there's at most one morphism from any object x to any object y, and every morphism has an inverse. This is the same thing as a symmetric reflexive transitive relation.

For example, if you can detect with sure-fire accuracy when two objects x and y are "similar in spirit", and this is a symmetric reflexive transitive relation, then you've got an equivalence relation, which you can think of as a category.

David Egolf (Dec 28 2020 at 06:03):

Thanks as always for your thoughts!

I understand that morphisms must be defined precisely. However, I wasn't proposing "similarity in spirit" as a definition of a morphism, but rather as a motivating philosophy for design. The challenge then is how to capture aspects of similarity in a precise way.

I can think of two different approaches to applying category theory to capture similarity between pipelines:
1) Start with a guess at what a category describing pipelines could look like. Then use category theory to automatically generate some notions of similarity in this setting. Hope that these notions are relevant to application.

2) Start with some concrete examples of pipelines and try to describe what aspects of similarity in these examples are relevant to application. Then use category theory to encode these aspects precisely.

I think I've been proceeding along the lines of (1) so far. If I understand you correctly, you are proposing an approach more along the lines of (2). I like approach (2). It is challenging in that it requires us to figure out relevant aspects of similarity between pipelines on our own (instead of using category theory to automatically generate interesting notions of similarity). However, this is probably a good challenge.

John Baez (Dec 28 2020 at 06:08):

I keep recommending approach (2) because it's so easy to spend a lot of time drifting around in lovely fields of mathematics if you don't have a really concrete example in mind that you're trying to formalize. Also, trying to formalize vague concepts like "similarity in spirit" is ridiculously hard without a very concrete example to pin things down.

David Egolf (Dec 28 2020 at 06:24):

Yes. Something with reasonable scope and clearly-defined success sounds like the way to go!

Here's one example of a concrete pipeline to get started:
$D_0$: There is a point with some reflection amplitude $R$ sitting at some depth $d$ in 1D space
$p_1$: We fire a square pulse of sound traveling at speed $c$ of duration $T$ and height $A$ from the origin at time 0 towards the point. The sound bounces off the point and travels back towards the origin with amplitude $AR$.
$D_1$: We observe the reflected wave at the origin, sampling every time interval $\tau$ from time 0 to some ending time $t_{end}$.
$p_2$: We take the amplitude of the sample observed closest to time $2d/c$ and output this as an "image" of the point.
$D_2$: The "image" of the point output by $p_2$.

We have the pipeline:
$D_0 \to_{p_1} D_1 \to_{p_2} D_2$

David Egolf (Dec 30 2020 at 21:35):

The above concrete example has been helpful for reflecting on when I consider imaging systems to be "similar". Here are some examples of imaging systems I would consider similar in some way to the above example, which I will call the "reference system":
-The reference system, except the point being imaged having reflection amplitude $R'$
-The reference system, except the point being imaged being at some depth $d'$
-The reference system, except we fire a square pulse of duration $T'$
-The reference system, except we modify one of its parameters (this includes the above examples)

Here are some other systems that seemed similar to the reference system to me, but are more different from the reference system:
-A photoacoustic imaging system, different from the reference system except in its sampling hardware and sampling rate
-An MRI imaging system, different from the reference system except in the target that it is imaging

In summary, I seem to consider two imaging systems to be similar in some way, if some part of these imaging systems are the same. Generalizing this a bit, I think I also consider two imaging system to be similar if they have some parts that are "compatible". By this I mean that I can grab a part from one imaging system and slot it into the other one, and this would create a well-defined imaging system. This finds practical application when writing code to simulate an imaging system. If I find an algorithm in the literature that seems better for some step being carried out for my code, then I can try copying the algorithm and putting it in the code. This can happen even if this algorithm was being used in a different imaging system.

Morphisms that detect "compatible" parts of imaging systems - so one can be swapped out for the other - I think detect an aspect of similarity between imaging systems that is relevant to application. (Another analogy: Say I'm building a tower out of Lego blocks, and you are building a tower out of Lego blocks. Then there might be a section in your building that has the same volume and connections to the remainder of your building as a section in my building. I could then at least try replacing my version of that part of the tower with your version of that part of the tower. The tower might collapse, but it's something I can try at least. So, in this sense, our two towers are similar.)

David Egolf (Dec 30 2020 at 22:16):

Based on the above discussion, we want morphisms to detect compatible parts of imaging pipelines, so that one part could be swapped out for the other. Let us try to make this idea precise.

To start, let's consider some very simple pipelines consisting of linear maps between vector spaces:
Pipeline 1: $\mathbb{R^2} \to_f \mathbb{R}$
Pipeline 2: $\mathbb{R^2} \to_g \mathbb{R}$
We want there to be a morphism that captures the fact that we can make a new pipeline from Pipeline 1 by swapping out $f$ for $g$. That is, the morphism should detect the compatibility of $f$ and $g$.

We often define morphisms between objects to take structure from the source object and find an "echo" of that structure in the target object. For example, an invertible linear map sends a subspace to a subspace of the same dimension. As another example, the image of a subgroup under a group homomorphism is again a subgroup.

Continuing this pattern, we will define a morphism from Pipeline 1 to Pipeline 2 to map structure into compatible structure. In this case $f$ is compatible with $g$. So, the map from Pipeline 1 to Pipeline 2 sends $f$ to $g$.

Now we consider a case where we have non-compatible maps.
Pipeline 1: $\mathbb{R^2} \to_f \mathbb{R}$
Pipeline 2: $\mathbb{R^3} \to_g \mathbb{R}$
$f$ and $g$ aren't compatible because they expect different kinds of inputs. So, there should be no morphism between these two pipelines.

Next consider a slightly more complicated case:
Pipeline 1: $\mathbb{R^2} \to_f \mathbb{R} \to_h \mathbb{R}$
Pipeline 2: $\mathbb{R^2} \to_g \mathbb{R}$
The map $g$ is compatible with the map $f$. So, there is some way to "plug in" map $g$ into Pipeline 1. So, there should be a morphism from Pipeline 2 to Pipeline 1 that sends $g$ to its compatible "echo" $f$.

There should be a morphism from Pipeline 1 to Pipeline 2 that sends $f$ to $g$. However, I'm not sure what (if anything) this morphism should do to $h$, as there is no linear map in Pipeline 2 that is compatible with $h$. Perhaps it should be mapped along to some kind of trivial structure space (e.g. as excess dimensions are collapsed when mapping from larger to smaller vector spaces)?

As always, any thoughts/suggestions/questions/criticisms/reflections are very much appreciated.

David Egolf (Dec 31 2020 at 04:19):

I noticed that when this diagram commutes and the $\alpha_i$ are isomorphisms, then $g = \alpha_2^{-1} \circ h \circ f \circ \alpha_1$:
one pipeline fitting into another

This means that the steps being carried out in the upper pipeline are the same as those being carried out in the lower pipeline, up to a change in details (e.g. like names of variables). For example, in a simulation context, I could write $g$ in code or I could write $h \circ f$ in code (together with a "change conventions" block before and after it) and the result would be the same. In this case we have detected two portions of different imaging systems that are secretly the same.

It is unfortunate that the above construction only detects exact equality, as "approximate equality" is more interesting from an applications perspective. However, if we only require the trapezoid to commute, then the range of possible $\alpha_i$'s should carry information about how the two pipelines relate, even in the cases when the two segments of pipeline compared aren't equivalent.

This way of detecting similarity between pipelines seems a bit more detailed than described in the previous post. It doesn't just say: "this map is compatible with this one", but also compares them through the $\alpha_i$. (I suspect merely "compatible" maps are very often too different from each other to make detecting pairs of them very useful). We can detect perfect similarity (when the $\alpha_i$ are isomorphisms) but I think we can also detect other degrees of similarity, for example based on the kernel of the $\alpha_i$ maps.

I think this brings us close to being able to define morphisms for pipelines. It remains to check this concept makes sense with some concrete examples and then to write down a definition. I think we will end up with something like this in the case where each pipeline is a sequence of vector spaces connected by linear maps: "A morphism is a collection of linear maps $\alpha_i$, one from each source in-pipeline-object to some target in-pipeline-object, such that resulting diagram commutes and the mapping preserves the order of in-pipeline objects."

David Egolf (Jan 01 2021 at 00:57):

Now we need to properly define morphisms and their composition, starting in the special case of pipelines consisting of maps between vector spaces.

(I'm happy with the relevance of these morphisms to applications. In some small examples I've tried it seemed like these morphisms were able to detect levels of similarity between parts of imaging pipelines. Detecting equivalence between parts of imaging systems I think will have application to detecting secretly related imaging system aspects described using different jargon (e.g. MRI vs. ultrasound). Measuring similarity between imaging systems I think will be relevant to assessing the "non-ideality" of high speed imaging methods, by comparing them to the ideal imaging pipeline that directly extracts the property of interest.)

On the chance that someone here is familiar with morphisms like these, and how they should be composed, here is a picture of roughly what I want. Here is what I want a morphism to look like, where each square (and triangle!) is commutative:
morphism

And here is what I want composition to look like:
composition part 1
composition part 2

Intuitively, we form composed arrows by "flowing along" arrows from top to bottom, taking all possible paths.

I'll be working on trying to describe this formally, but I'm pretty new to categorical language. Any ideas on how to formalize this are appreciated.

David Egolf (Jan 01 2021 at 22:51):

I'm still making things too complicated! We can reduce complexity like this:
-Consider only pipelines of this form, which model photoacoustic imaging systems:
Photoacoustic pipeline

-Consider only morphisms that relate analogous parts to analogous parts. This is reasonable in the case where every pipeline is of known form. So, morphisms look like this:
Photoacoustic pipeline morphism
where each square is required to commute.

John Baez said:

"Imaging systems" seems like a broad and rather vague class to me, in part because I don't know much about imaging systems. What are we actually talking about here? Can you pick a very few kinds of imaging systems and work out the category theory of what we actually do with these, in great detail?

@John Baez Is this the sort of thing you had in mind when you mentioned a "very few kinds of imaging systems"? I'd appreciate any insight you could offer that might steer me in tractable directions.

John Baez (Jan 01 2021 at 23:06):

Yes, this is roughly what I had in mind. Personally I'd go further and specify the workings of a limited class of imaging systems exactly, in full mathematical detail. To do this I'd have to limit the class a lot - otherwise the chore would be immense. But having something completely spelled out means there's no questions anymore about what you've got... and conversely, all the pain involved in completely spelling something out can be very useful: there are always tricky issues one hadn't expected, and facing up to those is important.

John Baez (Jan 01 2021 at 23:10):

In your current setup you have not told me what things like "message", "sound waves", etc. actually are - at least, not precisely.

John Baez (Jan 01 2021 at 23:12):

Even more importantly, what are the things called $\alpha_1, \alpha_2, \dots, \alpha_4$?

Of course you can't say what these are - not precisely - until you say precisely what things like "message", "sound waves", etc. are.

David Egolf (Jan 01 2021 at 23:14):

Okay, that is very helpful! I am currently in the process of working out the painful details, which is indeed limiting things a lot more.

John Baez (Jan 01 2021 at 23:14):

If all that really matters is that each of these things like "message", "sound waves", etc. has a set of possible states, then maybe you can keep things really simple and just say they are sets.

John Baez (Jan 01 2021 at 23:14):

Then it's very natural to say that $\alpha_1, \dots, \alpha_4$ are simply functions - or if you want to allow randomness, stochastic maps (which are a generalization of functions).

John Baez (Jan 01 2021 at 23:15):

So that's one way to get out of the painful process of limiting things: to work with very general, simple mathematical concepts that happen to include what you care about as a special case!

John Baez (Jan 01 2021 at 23:16):

There's a perfectly nice double category with sets as objects, with functions as vertical and horizontal arrows, and with commuting squares as 2-morphisms.

John Baez (Jan 01 2021 at 23:17):

But then it may turn out that something you want to do doesn't work at this level of generality. All forms of composition work fine in the double category I just described - but maybe you want to do more!

John Baez (Jan 01 2021 at 23:19):

This idea of "generalizing the hell out of what you want to do, to make it easier" is pretty common and powerful. For example, suppose you wanted to design a spreadsheet for prices of optical instruments. Well, you'd probably start with a general-purpose spreadsheet program, which doesn't give a damn about whether you're talking about optical instruments.

Eric Forgy (Jan 01 2021 at 23:20):

From a casual observer's perspective (and as a fellow engineer), I'd think a natural setting for this kind of thing would be $\mathsf{Span(Set)}$. I'm working on an "applied" article now with an intro to the benefits of $\mathsf{Span(Set)}$ for simulation.

David Egolf (Jan 01 2021 at 23:21):

If we could just use sets that would be nice. However, I'm worried that this won't capture the structure well. The "Laser" map acts in a very structured way on points in space, entirely determined by how it acts on a single point. It does correspond to a function, but it's got a lot more structure than just that. I'm not sure how damaging this would be to the resulting model.

John Baez (Jan 01 2021 at 23:22):

You can always add more structure as needed.

David Egolf (Jan 01 2021 at 23:23):

True! Maybe working with sets is a good place to start then.

John Baez (Jan 01 2021 at 23:23):

If you don't know what extra structure you need... well, that's typical when one is getting started! So a common thing to do is start with something simple, like the double category of sets I just outlined, and then see when you run into a problem that requires more structure.

John Baez (Jan 01 2021 at 23:24):

The great thing about starting with something "off the shelf" like the double category of sets is that people know lots about it, so for any mathematical question you have about it, some mathematician will know the answer.

John Baez (Jan 01 2021 at 23:24):

If you cook up a fancy double category of your own right from the start, you're sort of on your own.

David Egolf (Jan 01 2021 at 23:26):

I appreciate this very practical advice. It's a challenge learning how to collaborate well with people from other fields.

Eric Forgy (Jan 01 2021 at 23:26):

David Egolf said:

I appreciate this very practical advice. It's a challenge learning how to collaborate well with people from other fields.

Amen brother :sweat_smile:

Eric Forgy (Jan 01 2021 at 23:37):

Eric Forgy said:

From a casual observer's perspective (and as a fellow engineer), I'd think a natural setting for this kind of thing would be $\mathsf{Span(Set)}$. I'm working on an "applied" article now with an intro to the benefits of $\mathsf{Span(Set)}$ for simulation.

Btw, by $\mathsf{Span(Set)}$, I mean the category (as opposed to bicategory) whose objects are sets and morphisms are isomophism classes of spans so that composition is associative. Three spans $E:V\to V'$, $E':V'\to V''$ and $E'':V''\to V'''$ can be composed in two different ways

$(E\times_V E')\times_{V'} E'':V\to V'''$ and $E\times_V (E'\times_{V'} E''):V\to V'''$

that correspond to different (but isomorphic) spans. Considering isomorphism classes of spans effectively lets us drop the parentheses and consider those two to be the same and work with $E\times_V E'\times_{V'} E'',$ which is simpler.

The more I learn about them, the more I'm growing to love spans. I think they should be taught early in science / engineering education.

John Baez (Jan 01 2021 at 23:46):

The question of whether you treat spans as morphisms in a category, a bicategory, or a double category depends on exactly what you want to do with them. These days I always use a double category because it's easy to get a bicategory out of that and a category out of that.

But much more important is this: Egolf's diagrams

pipeline

don't look like maps between spans, they look like maps between composable chains of morphisms. So I'm not seeing spans here.

(Of course I would want to sell him on my own favorite formalism for open systems, which involves cospans, but he's not going for those either.)

David Egolf (Jan 01 2021 at 23:49):

I have seen for example this that looks very applicable to me, and I've been trying to imitate it. So, I am indeed interested in structured cospans! I had the impression though that in order to use that formalism, I had to first define an object category (which includes morphisms) - and that's what I was trying to do above.

David Egolf (Jan 01 2021 at 23:51):

@Eric Forgy What kinds of things do you think spans are good for in engineering? Modeling things exposing interfaces is the main application I'm aware of so far.

Eric Forgy (Jan 01 2021 at 23:52):

Any directed graph is just an endomorphism in $\mathsf{Span(Set)}$ so I can still see spans in that image, but you're probably right. I haven't thought much about double categories of sets :+1:

Where I feel spans work nicely (and maybe double categories do too) is if any of the output arrows can connect to multiple inputs. If the output of one system connects to the inputs of multiple other systems, spans make sense to me (but maybe double categories do too).

John Baez (Jan 01 2021 at 23:53):

@David Egolf wrote:

I have seen for example this that looks very applicable to me, and I've been trying to imitate it. So, I am indeed interested in structured cospans! I had the impression though that in order to use that formalism, I had to first define an object category (which includes morphisms) - and that's what I was trying to do above.

Hmm! To me it seems your category is fancy enough to begin with that applying the structured cospan idea to it may be overkill. But I guess you should just try to wrestle this category to the ground and do things with it, and see what happens.

Eric Forgy (Jan 02 2021 at 00:02):

Structured cospans look very cool to me and seeing some recent presentations on open dynamical systems is one reason that drew me back to thinking about ACT recently. I hope to form a more informed opinion than they "look cool", but that is where I currently am. A lot of my work involves directed graphs / networks and so far $\mathsf{Span(Set)}$ has been sufficient for what I need to do.

John Baez (Jan 02 2021 at 01:36):

You would get interested in structured cospans iff you were working with "open graphs" - graphs with some vertices designated as "inputs" and "outputs" - and wanted to build big graphs by gluing together smaller open graphs. Open graphs are the simplest interesting example of structured cospans, and Kenny and I use them as one of our main examples in our paper Structured cospans.

Since David Egolf might get interested in building bigger pipelines from gluing together smaller ones, he might want to use structured cospans.

David Egolf (Jan 03 2021 at 01:26):

To start with, let's try to work in the double category of sets, in order to work in a well-known setting. To do this, we first describe a photoacoustic imaging pipeline in terms of sets and functions.

We introduce a category that sets the shape of the pipelines. The category $\mathsf{Shape}_4$ has four objects and the three displayed non-identity morphisms:
Shape_4

A photoacoustic imaging pipeline is then the image of a functor $P: \mathsf{Shape}_4 \to \mathsf{Set}$. It looks like this:
A photoacoustic imaging pipeline

Each $P(v_i)$ is a set, and each $P(f_i)$ is a function.

This lets us describe a photoacoustic imaging pipeline as a diagram in $\mathsf{Set}$. However, we want to describe it as living in the double category of sets.

David Egolf (Jan 04 2021 at 00:14):

We could view the above diagram as living in the double category of sets, with the arrows corresponding to horizontal arrows. I still want to use double categories (maybe with cospans) to stick shorter pipelines together end-to-end. However, I need to do some learning about double categories first.

In the meantime, we can still study photoacoustic pipelines with fixed length, say having three arrows. Let's introduce a category of "photoacoustic pipelines in Set" as a functor category, where objects are functor from $\mathsf{Shape}_4$ to $\mathsf{Set}$ and morphisms are natural transformations between these functors. The intuition is that the morphisms compare imaging systems part-by-part.

There are many things in this category that don't correspond to the kind of photoacoustic imaging systems I want to model. However, it does contain the things that I want to model!

We can immediately add a bit more structure, though, while being able to still describe the systems I want to model. In particular, I am happy to assume that every map in every pipeline is a linear map. I'm also fine with giving each set the structure of a vector space. I have always assumed this in all the imaging system engineering I've done, so it's an interesting setting for me. (Although it's good to note that physical things in imaging systems aren't usually vector spaces - sensors have a maximum amplitude they can sense, for example. They look like parts of vector spaces though.)

To formalize this, we introduce the category "photoacoustic pipelines in Vect" as a functor category. The objects are functors from $\mathsf{Shape}_4$ to $\mathsf{Vect}$ and the morphisms are natural transformations between these functors. So, we model photoacoustic imaging pipelines by the functor category $\mathsf{Vect}^{\mathsf{Shape}_4}$. Until we start sticking pipelines together, I think this should be a good category to work in.

David Egolf (Jan 05 2021 at 22:55):

The functor category $\mathsf{Vect}^{{Shape}_4}$ corresponds to the category of quiver representations on the quiver corresponding to ${Shape}_4$ . So, we can grab notions from quiver representation theory and use them to gain understanding about imaging systems. For example, this theory describes the kernel and cokernel of a map between quiver representations. These might be useful in describing how an ideal imaging pipeline relates to an non-ideal one: we can capture this difference in the kernel or cokernel of a map between the ideal and non-ideal pipelines. For example, I'm wondering if we can detect the "resolution" of a pipeline in terms of the kernel of a map from an ideal imaging pipeline to the pipeline in question.

John Baez (Jan 06 2021 at 02:07):

I hadn't expected that your idea of imaging systems would be so linear - I can easily imagine a lot of 'image enhancement' algorithms that are nonlinear - but if linear systems are interesting to you, that's very good, because linearity makes things easier.

In particular, as long as you work with categories of the form $\mathsf{Vect}^\mathsf{C}$ for some category $\mathsf{C}$, you're on extremely firm ground mathematically: people know lots about these, and it would take quite a bit of ingenuity to think of a purely mathematical problem about them that nobody has thought about yet. That's good!

John Baez (Jan 06 2021 at 02:12):

There's a lot of 'linear math' that goes beyond merely studying categories of the form $\mathsf{Vect}^\mathsf{C}$ - there's a lot more you can do with linearity. For example you might be interested in "pipelines of variable length", trying to incorporate categories like

$\mathsf{Vect}^{\bullet \to \bullet}$

$\mathsf{Vect}^{\bullet \to \bullet \to \bullet }$

$\mathsf{Vect}^{\bullet \to \bullet \to \bullet \to \bullet}$

etc. into some larger structure (like a category or double category).

John Baez (Jan 06 2021 at 02:12):

The power of linearity would still be helpful.

David Egolf (Jan 06 2021 at 03:26):

In the context of ultrasound imaging, with which I am most familiar, I suspect that the future lies with nonlinear reconstruction and processing - maybe using thing like neural nets. However, in this setting, a lot of imaging schemes can be modeled as being linear. For example, state-of-the-art methods (e.g. "plane wave compounding") for ultrasound imaging with extremely fast frame rates can be modelled as being linear (with the exception of some final "cleaning up" non-linear processing).

There is a huge number of degrees of freedom in the design of an ultrasound imaging system even while staying within the realm of linearity. This is because you have over 100 little vibrating elements that can (nowadays) be controlled individually over time. Necessarily engineers have only implemented a small fraction of the possible schemes. Developing a theory that helps suggest interesting things to try or helps explain why different schemes work (even in just a linear setting) sounds really interesting to me.

David Egolf (Jan 06 2021 at 03:27):

John Baez said:

There's a lot of 'linear math' that goes beyond merely studying categories of the form $\mathsf{Vect}^\mathsf{C}$ - there's a lot more you can do with linearity. For example you might be interested in "pipelines of variable length", trying to incorporate categories like

$\mathsf{Vect}^{\bullet \to \bullet}$

$\mathsf{Vect}^{\bullet \to \bullet \to \bullet }$

$\mathsf{Vect}^{\bullet \to \bullet \to \bullet \to \bullet}$

etc. into some larger structure (like a category or double category).

Yes, this is very much something I want to do, as this will create a setting where we can compare imaging systems with different numbers of steps. However, I'm trying to get a feel for what I can learn in a setting with fixed pipeline-lengths first.

John Baez (Jan 06 2021 at 05:44):

By the way, here's a fun category.

An object of this category is a category $\mathsf{C}$ and a functor $F: \mathsf{C} \to \mathsf{Vect}$. So, in your application it's a kind of "pipeline" where the shape of the pipeline is very arbitrary - not necessarily a "line". A morphism from

$F: \mathsf{C} \to \mathsf{Vect}$

to

$F': \mathsf{C}' \to \mathsf{Vect}$

in this category is a functor

$G: \mathsf{C}' \to \mathsf{C}$

such that

$F' \circ G = F$

This may seem a bit weird, but it says this: if we call the morphisms in $C$ 'pipes', each pipe in $\mathsf{C}'$ is being written in terms of pipes in $\mathsf{C}$. I think this should handle a number of commonly arising situations.

David Egolf (Jan 08 2021 at 20:05):

I think $G$ should be a functor $G: \mathsf{C} \to \mathsf{C}'$, right? This lets us define $F' \circ G$.

David Egolf (Jan 08 2021 at 20:18):

If I understand correctly, it seems that setting $G$ restricts $F$ and $F'$ more than I would like. For example, say I have two pipelines of the same shape. In particular, say $C = C' = \bullet \to \bullet \to \bullet$. If we set $G$ to be the identity endofunctor (which maps each pipe to itself), then to have $F' \circ G = F$ implies $F' = F$. However, there are in general a number of maps between distinct quiver representations of the same shape.

It seems like in this construction $G$ corresponds to finding parts of the pipelines that are the same. This could be handy, but I'm also interested in detecting parts of pipelines that are related in weaker ways.

John Baez (Jan 08 2021 at 22:49):

I got a bit mixed up about which way some arrows should go. I think I really want

$G: \mathsf{C} \to \mathsf{C}'$

as stated, but

$F' = F \circ G$

This formula expresses the new pipes (the morphisms $F'(f)$ where $f$ is a morphism in $C'$) in terms of the old pipes.

John Baez (Jan 08 2021 at 22:51):

David Egolf said:

If I understand correctly, it seems that setting $G$ restricts $F$ and $F'$ more than I would like.

I'm not fixing $G$ ahead of time here; any functor $G: \mathsf{C} \to \mathsf{C}'$

with

$F' = F \circ G$

counts as an allowed morphism from $F$ to $F'$.

John Baez (Jan 08 2021 at 22:52):

For example if $G$ is the identity, then we get the identity morphism from $F$ to $F' = F$ - but this is the most boring example imaginable; it doesn't really illustrate what's going on very well.

David Egolf (Jan 09 2021 at 00:22):

This reminds me of the contravariant Hom functor, $\mathrm{Hom}(\_, \mathsf{Vect}): \mathsf{Cat} \to \mathsf{Set}$ that acts on a category $\mathsf{C}$ by: $\mathrm{Hom}(\_, \mathsf{Vect})(\mathsf{C}) = \mathrm{Hom}(\mathsf{C}, \mathsf{Vect})$, which is the set of functors from $\mathsf{C}$ to $\mathsf{Vect}$. If we have a functor $G: \mathsf{C} \to \mathsf{C'}$ then it sends this to the map $G^*:\mathrm{Hom}(\mathsf{C'}, \mathsf{Vect}) \to \mathrm{Hom}(\mathsf{C}, \mathsf{Vect})$ which acts like this: $G^*(F') = F' \circ G$.

The Hom functor above takes in a pipeline with "shape" (source category of pipeline functor) $\mathsf{C'}$ and spits out a pipeline with shape $\mathsf{C}$. Each functor $G$ from $\mathsf{C}$ to $\mathsf{C'}$ corresponds to a strategy for doing this. That is, each $G$ provides a strategy $G^*$ for converting pipelines from shape $\mathsf{C'}$ to shape $\mathsf{C}$, by $G^*(F') = F' \circ G$.

This process then spits out $F = F'$ when $G$ is the identity map. That is, we don't have to change the pipeline at all if we aren't changing its shape. Similarly, if $F$ and $F'$ are different but $\mathsf{C} = \mathsf{C'}$ then I think often there will be no morphism between them. This is too restrictive, I think. As noted above, I want the morphisms between pipelines of the same shape to correspond to natural transformations between their functors.

John Baez (Jan 09 2021 at 17:38):

The obvious generalization of my idea, which allows for more morphism, is to say that a morphism involves a natural transformation

$\alpha: F' \Rightarrow F \circ G$

rather than demanding the equation

$F' = F \circ G$

(In general, equations between functors are "evil", while natural transformations are not: they're more flexible.)

Another option would be to use a natural transformation

$\alpha: F \circ G \Rightarrow F'$

I haven't played around with examples so I don't know which of these is "better".

John Baez (Jan 09 2021 at 17:39):

I started out with equations just because they're very easy to understand.

David Egolf (Jan 11 2021 at 03:32):

Yes, this at least gives what I want in the case of morphisms between two pipelines of the same shape that compare each part of the two pipelines part-by-part. (In that case, we can set G = id, and then a morphism is simply a natural transformation from F to F', as desired). I'll have to try it out with some maps between differently shaped pipelines to see if this gives what I want in those cases.

John Baez (Jan 11 2021 at 04:26):

I hope it does good things! It's a natural construction in category theory (something like a lax slice category or oplax slice category), so there should be a whole pile of theorems that apply to it, but I won't bother thinking about those unless it seems worthwhile.

David Egolf (Jan 13 2021 at 17:47):

I've been trying to understand how composition works in this setting. Say we let a morphism from $F$ to $F'$ be a functor $G$ together with a natural natural transformation $\alpha$ from $F$ to $F' \circ G$, as shown:
a morphism

Now we want to compose two morphisms, the first going from $F$ to $F'$ and the second going from $F'$ to $F''$. We get a diagram like this, where $\beta$ is a natural transformation from $F'$ to $F'' \circ H$:
two morphisms

I know there are vertical and horizontal ways to compose natural transformations, but this seems a bit more complicated. I am hoping this introduction to 2-categories and bicategories will be helpful.

Nathanael Arkor (Jan 13 2021 at 18:06):

You shouldn't need to think about 2-categories to understand how to compose natural transformations works: it's easy to figure it out for yourself. In this situation, there are two concepts that are necessary: (left) whiskering, and vertical composition.
I think it helps to draw the diagram like this:
image.png

Nathanael Arkor (Jan 13 2021 at 18:07):

You can only compose natural transformations with matching target/source, so from this diagram it's clear that $\beta$ needs to be "pulled to the left" somehow.

Then the first step is figuring out how to get a natural transformation from $F' \circ G \Rightarrow F'' \circ H \circ G$ using $\beta$ and $H$ (this is whiskering).
image.png

Nathanael Arkor (Jan 13 2021 at 18:09):

Then you need to work out what it means to compose $\alpha$ and $\beta \circ H$ (this is vertical composition). If you write out the definitions of both natural transformations (as families of morphisms in $\mathsf{Vect}$), it should be obvious!
image.png

Nathanael Arkor (Jan 13 2021 at 18:10):

Notice that this composite natural transformation $\alpha \circ \beta \circ H$ goes to $F'' \circ (H \circ G)$, which is exactly the form you expect.

Nathanael Arkor (Jan 13 2021 at 18:11):

Once you've worked it out yourself, composition of natural transformations will seem much less complicated! (Horizontal composition can be defined using whiskering, so you'll almost have that covered too.)

John Baez (Jan 13 2021 at 18:17):

Thanks, @Nathanael Arkor. You explained it better than I would have.

John Baez (Jan 13 2021 at 18:25):

If these diagrams are confusing to you, @*David Egolf, you can also just see how to build a morphism from $F(x)$ to $F''(H(G(x))$ for each $x \in C_1$, given that you have a morphism

$\alpha_x : F(x) \to F'(G(x))$

and a morphism

$\beta_y : F'(y) \to F''(H(y))$

for each $y \in C_2$. Clearly you want to take $y = G(x)$ to get something useful here. Then you get

$\beta_{G(x)} : F'(G(x)) \to F''(H(G(x)))$

Now notice that you can compose

$\alpha_x : F(x) \to F'(G(x))$

and

$\beta_{G(x)} : F'(G(x)) \to F''(H(G(x)))$.

This gives you the natural transformation you seek! To check that it's really a natural transformation, the more abstract approach provided by Nathanael is best. You can also just grunge it out. Or you can just believe me: we get a natural transformation

$\beta_{G(x)} \circ \alpha_x : F(x) \to F''(H(G(x)))$

David Egolf (Jan 14 2021 at 02:37):

Thanks to both of you! I was able to figure out the answer from Nathanael Arkor's diagrams, and was pleased to see that the answer I got matched what John Baez described.

Now we can combine "analogies" between imaging systems, even between imaging systems having different shapes (e.g. having a different number of processing steps in a pipeline)! That is, if we can capture part of the structure of imaging system A in the context of system B, and we can capture part of the structure of imaging system B in C, then we automatically can capture part of the structure of system A in system C.

John Baez (Jan 14 2021 at 02:39):

Yes, it seems nice.