Category Theory
Zulip Server
Archive

You're reading the public-facing archive of the Category Theory Zulip server.
To join the server you need an invite. Anybody can get an invite by contacting Matteo Capucci at name dot surname at gmail dot com.
For all things related to this archive refer to the same person.

Stream: theory: applied category theory

Topic: spans and images

David Egolf (Jan 18 2022 at 00:09):

Let $f: A \to B$ be a morphism in some category. We think of $A$ as a source we wish to image, $f$ as an observation process, $B$ as a collection of possible observations and $f(A)$ as the observations actually obtained.

If $f$ factors through some $g: A \to B$, so $f = h \circ g$ for $h: B \to B$, we think of the observations obtained by $g$ as determining the observations made by $f$. For example, if we are working in $Set$, we have $f(x) = h(g(x))$ for any $x \in A$. So, the observation $f(x)$ is determined by the observation $g(x)$. This can be used to describe the situation where one sensor (or collection of sensors) collects observations that determine the observations another sensor must collect. However, this situation is unusual.

Probably more common is the situation where two sensors collect different data, and where the observation collected by one can not be used to compute the observation collected by the other. However, the observations are still consistent with one another. That is, it is possible for both observations to happen at the same time. I am wondering if we can view the statement of consistency as the existence of a span $B \leftarrow_f A \to_g B$. Intuitively, this is saying there exists a complicated enough object $A$, so that there exist observations $f$ and $g$ of that object, so that the two sets of observations obtained $f(A)$ and $g(A)$ can be obtained simultaneously.

Does this make sense? Can spans be used to reason about "consistency of observations"? How is "consistency" modelled in applied category theory in other contexts?

Zhen Lin Low (Jan 18 2022 at 03:37):

I feel there is a gap between the application you describe and the mathematics you are using. If A is a source and B is the set of possible observations, then I would say the map f : A → B _itself_ is the observation. Concretely, you might think of A as the set of coordinates of sample points, B is the set of intensities, and f is the function that returns the intensity at each sample point. Consequently, taking the image of f is a disastrously lossy process.

David Egolf (Jan 18 2022 at 04:06):

I appreciate you sharing your thoughts, @Zhen Lin Low !

I am thinking about your example. I not sure the best way to model imaging, and alternate strategies for doing this are of interest to me.
One thing that is nice about your approach - where a function $f: A \to B$ is viewed as the observation itself - is that it naturally incorporates an indexing of the observed data. By looking at $f$ you can say, "we observed $f(a)$ at location (or time) $a$". The ordered pairs of $f$ correspond to observations, where we keep track of when/where data (e.g. intensity data) was collected.
This indexing of observation data is absolutely vital, as you note. Just marking down intensity values in your example, while ignoring the coordinates of sample points, would indeed be "disastrously lossy"!
However, I believe it is also possible to include indexing in the image. To rephrase your example in this form, let $A$ be a set with a single thing we want to image - say it is a painting. This is a set with a single element. Then, let $f$ send this single element to indexed observation data. The image of $f$ is a set, with elements consisting of ordered pairs - the first coordinate correspond to location in the painting, and the second coordinate corresponding to the intensity observed there. In this case, the image of $f$ retains the indexing data we need.

David Egolf (Jan 18 2022 at 04:10):

Does you feel this addresses the gap you were describing? I would appreciate any further thoughts you may have.

Zhen Lin Low (Jan 18 2022 at 06:37):

It would be more reasonable to say f is an "observation process" in that model, yes. But it remains unclear to me what the point of taking the image is, and the role of the map h is also unclear. As previously discussed, when A is just a point, any two maps f, g : A → B will be related by some h : B → B.

David Egolf (Jan 18 2022 at 16:36):

Zhen Lin Low said:

But it remains unclear to me what the point of taking the image is, and the role of the map h is also unclear. As previously discussed, when A is just a point, any two maps f, g : A → B will be related by some h : B → B.

Thanks again for your thoughts, @Zhen Lin Low. That is a really good point regarding what happens if A is a set with a single object. So the example I gave above doesn't appear to be describing an imaging system in the way I want.

Perhaps the problem is that I am modelling the "thing-to-be-imaged" as a set with a single element. Let me instead try to model it as a function $p: \{1, 2, ..., n\} \to \mathbb{R}$. Each element of $\{1,2,\dots,n\}$ corresponds to a position, and $p(m)$ is the intensity of the painting at location $m$. Considering this as an object, we want to figure out what an observation morphism should be.

I noticed that the painting now corresponds to a functor from the category $0 \to 1$ to $Set$. Call the source category $[1]$, following the notation for objects of the simplex category. So, our model for the painting corresponds to a functor $P: [1] \to Set$. We can think of the first object of [1] as corresponding to "where", and the second object of [1] as corresponding to "what".

Let us also model observations as functors from $[1] \to Set$. Then, let maps between the painting and its observations correspond to natural transformations of these functors. This seems to be an improvement over the previous approach - I think we no longer have the problem that any two observations of the painting are related in the way you describe above!

David Egolf (Jan 18 2022 at 23:02):

I was typing up some engineering research work on imaging, and I noticed some similarities and differences between the structure of the objects involved and what I described above.
In my engineering research, we have a (noiseless) model for observations: $y = \Phi(x)$ . Here $y$ is the observation observed (a finite dimensional vector), $x$ is a function describing an unknown source, and $\Phi$ describes the observation process.
In more detail, $x: \mathbb{R^2} \to \mathbb{R}$ is a function that return the absorbing strength of a source at each point in the plane. We can also view $y$ as a function, $y: \{1, 2, \dots, n\} \to \mathbb{R}$, as it keeps track of when (and where) the data was observed. Then $\Phi$ is a function between function spaces $\Phi:\mathbb{R}^{\mathbb{R^2}} \to \mathbb{R}^{\{1, 2, \dots, n\}}$.

This is similar in some ways to what was described above, but also different. We can view both our observation $y$ and the unknown source $x$ as functors from $[1]$ to $Set$. This matches what I was saying above.
However, the observation process is a function between function spaces - not a natural transformation. This doesn't match what I was saying above.

David Egolf (Jan 18 2022 at 23:07):

So, maybe this is a reasonable category for describing some imaging processes:
-The objects are sets of functions (thought of as unknown sources, or observations)
-The morphisms are functions between these sets, thought of as imaging processes

David Egolf (Jan 18 2022 at 23:09):

Unfortunately, we are going to have the problem described earlier again. Any two functions from a set with a single element (a function) will be "equally informative". Hmmm.

Zhen Lin Low (Jan 18 2022 at 23:13):

You seem to be fixated on the "equally informative" concept. May I gently suggest that, in light of growing evidence, that the definition is ... wrong?

David Egolf (Jan 19 2022 at 01:24):

Well, it is certainly possible. There is some intuition I'm trying to model. It's possible that I'm modelling it wrong.

For the record, here is the intuition I'm trying to model.
Imagine we drop a pebble in a pond. At some point in the pond, we have a sensor $a$ that can detect the height of the water there. It records the height of the water over time, recording a ripple pattern.
Imagine we have a priori information that the only disturbance of the water is due to a single pebble.
Now, imagine we have a second sensor $b$ that also observes the height of water.

Given the readings on sensor $a$, the locations of sensor $a$ and $b$, and the a priori information that the disturbance to the water is due to a single pebble - given all these things, we can compute the readings that occur on sensor $b$.
I would like to be able to model this situation, where one observation can be computed from the other given appropriate context.

David Egolf (Jan 19 2022 at 01:26):

I don't care so much about modelling things as being "equally informative", as I care about one observation being "at least as informative" as another.

David Egolf (Jan 19 2022 at 01:28):

As a simpler example, the readings on sensor $a$ and sensor $b$ together are enough to calculate the readings on sensor $a$. So, the two readings together are at least as informative as the reading on sensor $a$ in isolation.

David Egolf (Jan 19 2022 at 01:29):

I would very much welcome any ideas you have on how this situation could be modelled in a better way (instead of terms of morphisms factoring through one another).
(I'm continuing to think about this too!)

David Egolf (Jan 19 2022 at 02:12):

I think part of what's going on here is that the image of every function from a singleton set $A$ specifies $A$ up to isomorphism. So in this sense all such functions are "equally informative" about $A$. But a set with a painting of sunflowers is different in the context of imaging paintings than a set with a painting of a mountain.
Maybe a way forward is to work in a category with fewer morphisms, so that fewer things are isomorphic. Then any approach that can distinguish objects (up to isomorphism) will be able to distinguish more things.
This might involve modelling a thing-to-be imaged as something more structured than a set.

David Egolf (Jan 19 2022 at 16:28):

I'm still thinking about this. I'll try to come to some clearer conclusions before sharing my thoughts.

John Baez (Jan 19 2022 at 18:33):

I would agree with Zhen Lin that, coming from a background in category theory rather than image analysis, I'd naturally want to think of a category $A$ as something being observed and a functor $f: A \to B$ as some process of observation. Then, I would say $f: A \to B$ is at least as informative as $g: A \to C$ iff $g$ factors though $f$ up to natural isomorphisms, i.e. there is some functor $h: B \to C$ such that $g \cong f \circ h$, where $\cong$ means there's a natural isomorphism from $g$ to $f \circ h$.

John Baez (Jan 19 2022 at 18:34):

Here you can think of $h$ as some "data processing". (You can also think of it as an observation if you like: this very general framework doesn't distinguish between "observing" and "data processing", which makes sense since observations often include some data processing.)

John Baez (Jan 19 2022 at 18:36):

So, the idea is that $f$ is at least as informative as $g$ if you can get $g$, at least up to isomorphism, by doing $f$ and then doing some data processing.

David Egolf (Jan 19 2022 at 20:12):

Thanks again for your thoughts! I think they will be helpful going forward. The modification of factoring through up to natural isomorphism is interesting. I will have to mull that over - maybe try to think of a concrete example to illustrate that.

In the meantime, I've shifted my modelling approach a little. I am thinking there is a problem with having a category where objects are things we observe, and morphisms are observations of those. In particular, every category requires there to be an identity morphisms associated with each object. I am worried that this would require there to be a "perfect observation" of each object - something we generally don't have access to in applications.

As a result, I want to make things simpler and first consider a category of the environmental interactions of some fixed object, which we can think of as grounding the concept of "observation". So, in this setting, the thing being imaged does not appear as an object - instead the interactions of that thing with its environment correspond to the objects. As motivation for focusing on environmental interactions, in x-ray projection imaging, we don't directly observe the unknown object - we observe a sort of shadow in x-ray beams fired at the object. Also, in some cases, observing a particular special environmental interaction of an object is just as good as observing the object itself.

I hope to associate each environmental interaction with a specific range of space (and time) locations. If space and time can be split up in to little pieces (say, open sets), then this will allow us to talk about observations at a particular place (e.g. we observe the temperature in this open set at this time). I am hoping that the observations will naturally inherit structure (some kind of lattice?) from the structure of space and time.

David Egolf (Jan 19 2022 at 20:18):

We can also hope that this perspective of environmental interactions will lead to a way for thinking about imaging.
To illustrate, imagine we have a sequence of interactions (each "x->y" can be read as "x changes y"):
sun -> temperature field around sun -> temperature reading on probe
If we can go from "temperature reading on probe" to "temperature field around sun", we can view this as providing an explanation for the temperature reading on the probe. "The probe reads XXX because it is in an area with a temperature of XXX". Following the slogan: "Imaging is explanation of environmental changes", we can view this explanation as providing a sort of image of the temperature field.
So, if we can start talking about relationships between environmental interactions - and in particular deducing (or ruling out) some from others - this is not so different, necessarily, from talking about imaging.

David Egolf (Jan 20 2022 at 00:36):

I'm trying to connect these ideas to the discussion in Seven Sketches. To this end, I made a little diagram that tries to imitate the sort of diagrams we see in that book:
environemental interactions

David Egolf (Jan 20 2022 at 00:37):

"Imaging" corresponds to trying to go backwards along this chain of events - to figure out something about the sun from the sensor reading.

David Egolf (Jan 20 2022 at 00:56):

I hope that, at each point along such an "interactions chain", we can get a category of environmental interactions (with morphisms corresponding to inclusion of subsets, relating interactions over smaller and larger areas). It would be interesting to consider if there are functors between different categories along such a chain.
Anyways, I guess I feel that the stuff in Seven Sketches is starting to seem more relevant (I'm starting to want to think of imaging in terms of systems that link together and influence one another). Probably next on my list is to work through more of that!

John Baez (Jan 20 2022 at 18:17):

David Egolf said:

Thanks again for your thoughts! I think they will be helpful going forward. The modification of factoring through up to natural isomorphism is interesting. I will have to mull that over - maybe try to think of a concrete example to illustrate that.

For example, one observation might give you an output written in decimal numbers while another might give it written in binary; then we might use a natural isomorphism to translate between them. This example is vague because I haven't specified the precise categories and functors involved! Indeed I'm pretty fuzzy about the details of the categories you're talking about. I don't remember you describing examples of these - in complete details, where you specify the objects, morphisms, and how to compose morphisms. That makes me very nervous, but I can respond in an equally vague way.

David Egolf (Jan 20 2022 at 19:57):

Thanks @John Baez for your thoughts, as always.

Let me try to make a category that describes at least a little of what I want, and describe it precisely.

To start, some motivation. We want a category $C$ that is a category of "environmental interactions of an object with respect to a single property $H$ at a single moment in time". At a given moment in time, the environmental interactions of an object correspond to modifications it makes to its surroundings, in space. As a concrete example, say the object is a person, and the interaction with their surroundings at this moment in time corresponds to them standing in a field of fresh snow - so the snow is depressed under their feet. The interactions with the snow occur over space - each little part of each show of the person reduces the height of the top of the snow. We want a category where the objects are the different parts of these interactions. It seems reasonable to split up the modelling of the interactions over space, since each modification of the environment occurs at some location in space.

Now, to describe a category. We start with a topological space $(X, \tau)$, defined by a set $X$ and a topology $\tau$. This will play the role of the field on which the snow rests in the example above. Next, we need to specify the places in space where an interaction/modification has occurred. To do this, we specify a subspace $(Y, \tau_Y)$ of $(X, \tau)$, having the subspace topology. In our example above, this corresponds to where the shoes are resting. We can talk about different parts of the shoes using the open sets $\tau_Y$. Next, we want to talk about the modification made to the snow height for each piece of the shoes. To do this, we use a function $f_i$ for each open set $S_i \in \tau_Y$. Let $H$ be the set of possible values for the height of the snow. Then the function $f_i: S_i \to H$ specifies the height of the snow in the region $S_i$. These functions are "environmental interactions" and so they should be objects in our category $C$.

To summarize so far, we have a topological space $(X, \tau)$ and a subspace $(Y, \tau_Y)$ of $X$, having the subspace topology. We have a set $H$, and for each open set $S_i$ of $Y$, we have a function $f_i: S_i \to H$. So, we have one function per open set of $X$. Let these functions be the objects of $C$.

To specify a category, we need to specify morphisms. Let there be exactly one morphism from $f_i$ to $f_j$ iff $S_i \subseteq S_j$. This allows for at least some rough comparison of environmental-interactions.

There are a lot of limitations with this, but it's a start!

John Baez (Jan 20 2022 at 23:24):

I think this needs a lot of work. Your concept of morphism doesn't involve the functions $f_i$ at all, just the sets $S_i$, so I believe it is equivalent (in the technical sense) to the category of open subsets of the topological space $X$.

John Baez (Jan 20 2022 at 23:25):

This category is a poset.

John Baez (Jan 20 2022 at 23:26):

The posets of open subsets of an arbitrary topological space $X$ is a wonderful and much-studied thing, but I'd be hard-pressed to convince anyone it has anything to with snow, people's feet, or image processing.

John Baez (Jan 20 2022 at 23:29):

You said a bunch of things about a field of snow, footprints, shoes, etc. - but none of this made its way into the mathematical description of the category at the end. As far as the end result goes, you could have just said "consider the poset of open subsets of a topological space $X$".

David Egolf (Jan 21 2022 at 01:23):

You are right - it needs lot of work! I wanted to make a start, though.

I actually thought the described category was isomorphic to the category of open subsets of $Y$ with morphisms corresponding to inclusions. (But is it just an equivalence? I suppose it doesn't matter - we don't want either).

Something more clever needs to be done with the morphisms - we need a rule for specifying morphisms that cares about the functions $f_i$ and not just the corresponding set $S_i$. If the morphisms don't notice the modifications to the environment (the $f_i$) the category really doesn't reflect anything about them!

David Egolf (Jan 21 2022 at 01:31):

Here's an idea. Let us have a morphism $f_i \to f_j$ exactly when these two conditions are satisfied:

1. $S_i \subseteq S_j$
2. $f_i = f_j$ on the intersection $S_i \cap S_j = S_i$
   The idea is to have a morphism from $f_i$ to $f_j$ when "$f_i$ consistently fits inside $f_j$".

David Egolf (Jan 21 2022 at 01:35):

In addition, it seems unsatisfying to only consider a single function from each subset $S_i$, as this only lets us talk about one environmental interaction at a time for a given location. It would be nice to able to talk about different interactions so that we can compare them in some way.
So, let the objects of the category now be the functions from the open sets of $Y$ to $H$. For any given open set $S_i$, there are a bunch of associated objects, each one corresponding to a single function from $S_i$ to $H$.

David Egolf (Jan 21 2022 at 01:36):

Now the category is looking different than the poset of the open sets of $Y$, I think (I hope!).

David Egolf (Jan 21 2022 at 01:44):

I think this is describing the snow-field example a little better. If we pick a given object, and then consider a sequence of morphisms from it, this corresponds to considering successively larger areas of deformed snow, each of which contain the previous area of deformed snow inside of it (having the same deformation).

David Egolf (Jan 21 2022 at 01:48):

In a similar spirit, a cospan should correspond to two smaller areas of deformed snow contained inside a larger area of deformed snow.

John Baez (Jan 21 2022 at 03:56):

I actually thought the described category was isomorphic to the category of open subsets of $Y$ with morphisms corresponding to inclusions. (But is it just an equivalence? I suppose it doesn't matter - we don't want either).

It's equivalent, not isomorphic, because for each open subset the described category has many different isomorphic objects, one for each function on that subset.

John Baez (Jan 21 2022 at 03:59):

I'm also confused about whether you're talking about open subsets of $X$ or open subsets of $Y$, since you said both things here:

We have a set $H$, and for each open set $S_i$ of $Y$, we have a function $f_i: S_i \to H$. So, we have one function per open set of $X$. Let these functions be the objects of $C$.

John Baez (Jan 21 2022 at 04:05):

(If your category only involves the subspace $Y \subset X$ you don't need to mention $X$ in the first place, since any subspace of a topological space is a topological space in its own right, so you can just start with $Y$.)

John Baez (Jan 21 2022 at 04:37):

But this doesn't affect my main point. My main point is that if you want to apply category theory to image processing the way you're doing, I believe you should cook up some examples of categories that are convincingly connected to image processing.

David Egolf (Jan 21 2022 at 04:41):

John Baez said:

I actually thought the described category was isomorphic to the category of open subsets of $Y$ with morphisms corresponding to inclusions. (But is it just an equivalence? I suppose it doesn't matter - we don't want either).

It's equivalent, not isomorphic, because for each open subset the described category has many different isomorphic objects, one for each function on that subset.

Hmmm. I don't actually think this is the case (?). In the original formulation, I associate exactly one function with each open set. So there is a bijection between open sets and the objects $f_i$.

David Egolf (Jan 21 2022 at 04:42):

John Baez said:

I'm also confused about whether you're talking about open subsets of $X$ or open subsets of $Y$, since you said both things here:

We have a set $H$, and for each open set $S_i$ of $Y$, we have a function $f_i: S_i \to H$. So, we have one function per open set of $X$. Let these functions be the objects of $C$.

Oops! I mean open sets of $Y$.
The reason I'm mentioning $X$ at all (as a space containing $Y$) is that I am hoping in the future it will provide a sort of ambient space where multiple $Y_i$ will live. But it's true we don't use it right now.

John Baez (Jan 21 2022 at 04:44):

David Egolf said:

John Baez said:

I actually thought the described category was isomorphic to the category of open subsets of $Y$ with morphisms corresponding to inclusions. (But is it just an equivalence? I suppose it doesn't matter - we don't want either).

It's equivalent, not isomorphic, because for each open subset the described category has many different isomorphic objects, one for each function on that subset.

Hmmm. I don't actually think this is the case (?). In the original formulation, I associate exactly one function with each open set. So there is a bijection between open sets and the objects $f_i$.

Okay, I thought the $f_i$ were all possible (continuous?) functions $f_i: U \to \mathbb{R}$. If there's just one then you're right: your category is isomorphic to the poset of open subsets of $Y$.

John Baez (Jan 21 2022 at 04:45):

To a category theorist this means the functions $f_i$ are utterly irrelevant to the definition of your category.

John Baez (Jan 21 2022 at 04:45):

They might become important in some later construction, but not in defining that category.

David Egolf (Jan 21 2022 at 04:45):

John Baez said:

But this doesn't affect my main point. My main point is that if you want to apply category theory to image processing the way you're doing, I believe you should cook up some examples of categories that are convincingly connected to image processing.

Hmmm. In my mind the "snow-field" example is convincingly connected to the process of imaging. Let me see if I can convince you there is a connection.
One could break the process of imaging into a few steps:
there is an object -> it interacts with something -> we process those interactions to make an image
For example, for ultrasound imaging:
there is a heart -> we bounce ultrasound waves off it -> we process the ultrasound wave echoes to make an image

So, in my mind, creating a language for talking about interactions of an object with its surroundings is very much on topic to imaging - it describes the first part of the imaging process.

David Egolf (Jan 21 2022 at 04:46):

John Baez said:

To a category theorist this means the functions $f_i$ are utterly irrelevant to the definition of your category.

Do note, however, that (I think) they are very important in the updated definition I provided afterwards, where we define morphisms in terms of the $f_i$.

David Egolf (Jan 21 2022 at 04:46):

David Egolf said:

Here's an idea. Let us have a morphism $f_i \to f_j$ exactly when these two conditions are satisfied:

1. $S_i \subseteq S_j$
2. $f_i = f_j$ on the intersection $S_i \cap S_j = S_i$
   The idea is to have a morphism from $f_i$ to $f_j$ when "$f_i$ consistently fits inside $f_j$".

Namely in this formulation of morphisms ^

John Baez (Jan 21 2022 at 04:49):

David Egolf said:

John Baez said:

But this doesn't affect my main point. My main point is that if you want to apply category theory to image processing the way you're doing, I believe you should cook up some examples of categories that are convincingly connected to image processing.

Hmmm. In my mind the "snow-field" example is convincingly connected to the process of imaging. Let me see if I can convince you there is a connection.
One could break the process of imaging into a few steps:
there is an object -> it interacts with something -> we process those interactions to make an image
For example, for ultrasound imaging:
there is a heart -> we bounce ultrasound waves off it -> we process the ultrasound wave echoes to make an image

So, in my mind, creating a language for talking about interactions of an object with its surroundings is very much on topic to imaging - it describes the first part of the imaging process.

That's a bunch of words, but when it comes around to defining your category, you could skip all those and just say "let $Y$ be a topological space; my category is the poset of open subsets of $Y$".

And this is different from successful applications of category theory that I know, where the categories involved are often deeply connected to the mathematical details of the subject they're being applied to. For example there's a category of chemical reaction networks, where the objects are chemical reaction networks (some sort of mathematical gizmo that chemists study).

David Egolf (Jan 21 2022 at 04:49):

I don't think the current category is a poset.

David Egolf (Jan 21 2022 at 04:50):

(with the updated definition of morphisms)

John Baez (Jan 21 2022 at 04:50):

David Egolf said:

David Egolf said:

Here's an idea. Let us have a morphism $f_i \to f_j$ exactly when these two conditions are satisfied:

1. $S_i \subseteq S_j$
2. $f_i = f_j$ on the intersection $S_i \cap S_j = S_i$
   The idea is to have a morphism from $f_i$ to $f_j$ when "$f_i$ consistently fits inside $f_j$".

Namely in this formulation of morphisms ^

This is getting more interesting; it still has nothing to do with image processing per se, but this is an example of what category theorists call "the category of elements of a presheaf category".

David Egolf (Jan 21 2022 at 04:51):

Let me try to connect this more closely to image reconstruction.

David Egolf (Jan 21 2022 at 04:53):

Image reconstruction in the context of ultrasound imaging is the process of taking sound waves, and figuring out how they could have been produced. I believe this can be viewed as asking the question: what states of environmental interaction (through sound waves) are consistent with the sound waves we see now?
So if we can talk about interactions being consistent with one another, in my view we are long ways towards making images.

John Baez (Jan 21 2022 at 04:54):

If I understand correctly your new category is still a poset since if there's a morphism from $f_i$ to $f_j$ and also one from $f_j$ to $f_i$ they are equal.

John Baez (Jan 21 2022 at 04:54):

I'm afraid all your talk with words goes right past me; I look at the math. If I were feeling more energetic I would actively work to translate your words into interesting math, but I'm not.

David Egolf (Jan 21 2022 at 04:54):

In practice, we don't require total consistency - but rather approximate consistency.
The traditional reconstruction algorithm for ultrasound imaging basically asks: "What scattering amplitude should this point have, so that the data that we would expect to come from it looks as much as possible as what we actually see?"

David Egolf (Jan 21 2022 at 04:55):

John Baez said:

If I understand correctly your new category is still a poset since if there's a morphism from $f_i$ to $f_j$ and also one from $f_j$ to $f_i$ they are equal.

I should have clarified - it is not (I think) the poset of the open sets of $Y$ anymore. It might still be a poset.

David Egolf (Jan 21 2022 at 04:55):

John Baez said:

I'm afraid all your talk with words goes right past me; I look at the math. If I were feeling more energetic I would actively work to translate your words into interesting math, but I'm not.

Hmmm. Let me explain a reconstruction algorithm similar to "backprojection", a standard image reconstruction approach - using math instead of words.

David Egolf (Jan 21 2022 at 04:56):

We have a system model :
$y = \Phi x$
here $y$ is what we observe, $x$ is an unknown object of interest, and $\Phi$ is a matrix
Then to estimate $x_i$, we compute $\langle \Phi_i, y \rangle$

David Egolf (Jan 21 2022 at 04:57):

For me, this intuitively corresponds to asking "How similar is what we observed, to what we would expect to observe if there was a unit strength source at location $i$?"

David Egolf (Jan 21 2022 at 04:58):

As another example of this idea in image reconstruction, expressed in math, I am currently doing research on reconstructing photoacoustic images. The model looks like this:

David Egolf (Jan 21 2022 at 04:59):

$y = \Phi(x)$
$\hat{x} = \textrm{arg} \min_x \|\Phi( x) - y\|_2^2 + \tau \|x\|_1$

David Egolf (Jan 21 2022 at 04:59):

We estimate the unknown object by minimizing a loss function. Notice that this loss function includes a discrepancy term (the 2-norm term), comparing the observed data and the expected data for some guess $x$.

David Egolf (Jan 21 2022 at 05:00):

The point I'm trying to make:
Both of these examples involve talking about consistency (generally only partial) between what we observed, and what we would expect to observe under some guess for an unknown thing

(As another example of why I care about relating different environmental perturbations: often we have multiple sensors observing different parts of a sound wave. They then have to work together to make an image. Figuring out how to do this is important).

David Egolf (Jan 21 2022 at 05:00):

So, a language for talking about (ideally partial) consistency between different perturbations of the environment is interesting to me for purposes of image reconstruction.

The category you called "the category of elements of a presheaf category" I think starts to go in this direction, as morphisms correspond to a statement of consistency relating environmental perturbations over smaller areas to those over larger areas. I would also want to be able to talk about consistency across time (e.g. to describe a propagating wave), as well as space. But I was trying to make it as simple as I could to get started.

David Egolf (Jan 21 2022 at 05:02):

I don't know if this helps. I would like to describe this stuff in better math for you, but that's what I'm aiming to do - I haven't done it yet

David Egolf (Jan 21 2022 at 05:11):

I hope something I posted above is helpful. I'm happy to clarify anything if you would like. And as always, thanks for your time and interesting comments.

John Baez (Jan 21 2022 at 05:30):

Sometime when I'm feeling more peppy I'll see if I can dream up some math connected to what you just said.

John Baez (Jan 21 2022 at 05:31):

Btw, often it's not category that's needed, but some other sort of math. The bigger ones toolbox, the more likely one will find the appropriate tools.

David Egolf (Jan 21 2022 at 18:38):

For easy reference going forward, here is the most recent version of a category trying to describe environmental perturbations at a fixed moment in time, with morphisms enabling us to talk about consistency between perturbations:

• Let $Y$ be a topological space. Let $I$ be an indexing set for the open sets of $Y$, so we have a bijection $S: i \mapsto S_i$ between $I$ and the open sets of $Y$.
• For each open set $S_i$, we have the set of functions from $S_i$ to a set $H$, $Set(S_i, H)$. The objects of $C$ are all such functions: $Obj(C) = \{f | f \in Set(S_i, H) \textrm{ for some } i \in I\}$.
• If $f \in Set(S_f, H)$ and $g \in Set(S_g, H)$ there is one morphism $f \to g$ exactly when:
  - $S_f \subseteq S_g$
  - $f = g$ on $S_f \cap S_g = S_f$

As indicated by John Baez above, I believe this category has the structure of a partial order.

David Egolf (Jan 21 2022 at 18:42):

Using this category, let us say that $f \in Set(S_f, H)$ and $g \in Set(S_g, H)$ are "consistent" exactly when:

• there exists a $h \in Set(S_f \cap S_g, H)$ so that a morphism exists $h \to f$ and a morphism exists $h \to g$

(this is a span!)

David Egolf (Jan 21 2022 at 18:49):

Let us also say that $h \in Set(S_h, H)$ is "a consequence of" $f \in Set(S_f, H)$ exactly when:

• there exists a morphism $h \to f$

David Egolf (Jan 21 2022 at 18:55):

We can then consider the set $Q(f)$ "of all consequences" for any object $f \in Set(S_f, H)$.

• $Q(f) = \{h | \textrm{ there exists a morphism } h \to f \}$
  (It would be possible to get a slice category from this.)

David Egolf (Jan 21 2022 at 18:57):

This corresponds to the set of all environmental perturbations known to have happened if the environmental perturbation $f$ has happened.

David Egolf (Jan 21 2022 at 19:03):

The idea I'm inching towards here: an image is a special kind of consequence of some special environmental perturbation (namely the perturbations of our sensors) - e.g. if we observed this pressure-wave on our sensors, then there must have been this initial pressure distribution in the unknown object corresponding to an image.
To go further towards this idea, I think we will need to talk about consistency across time, which will involve including some notion of a time-update operation in our category.

David Egolf (Jan 23 2022 at 16:57):

Starting here John Baez explains how the category above is a special case of a "category of elements".
He also notes that putting a topology on $H$, enabling us to consider continuous function to $H$, could be useful.

Let me update (and re-describe) the category above using these helpful things.
We start with a topological space $Y$.
-- I conceptualize this as the "environment" of some object.
Let $Open(Y)$ be the partial order category of its open sets, with a morphism $A \to B$ exactly when $A \subseteq B$.
-- I conceptualize this as a way of keeping track of how the "nice subsets" of the environment relate to each other.
For each open set $U$ of $Y$, let $Top(U, H)$ be the set of all continuous functions from $U$ to a topological space $H$.
-- I conceptualize this as the set of all possible "nice" perturbations of some environmental property $H$ at location $U$.
Let $F$ be the functor $F: Open(Y)^{op} \to Set$ that acts by $U \mapsto Top(U,H)$ on objects. On morphisms it acts as follows: if $V \subseteq U$, so there is a morphism $(U,V): U \to V$ in $Open(Y)^{op}$, then the image of this morphism under $F$ is the function $F(U,V): Top(U,H) \to Top(V,H)$ which acts by $F(U,V): g \mapsto g|_V$, where $g \in Top(U,H)$ and $g|_V \in Top(V,H)$ is $g$ with its inputs restricted to $V$.
--I conceptualize this as describing how the possible nice perturbations of the environment relate to one another - which ones are physically consistent in the absence of additional contextual knowledge

We can now form the category described above using a somewhat different language:
--Every object is an element of $F(U)$ for some open set $U$ of $Y$, so an object is a function $f \in Top(U,H)$
--Every morphism is an element of a morphism $F(U,V)$ for some open sets $V \subseteq U$ of $Y$, so a morphism $f \to f_V$ is a function that restricts $f \in Top(U,H)$ to $V$, yielding a result $f_V \in Top(V,H)$

This is slightly different than what we described above:

• We now require maps to $H$ to be continuous
• We now have a morphism $f_i \to f_j$ exactly when $f_i$ restricts to $f_j$ (we would have previously said this was a morphism from $f_j$ to $f_i$.

David Egolf (Jan 23 2022 at 17:10):

Intuitively, it seems $F$ is inducing a preorder on functions (a "consistency relationship") by its action on morphisms in $Open(Y)^{op}$. The resulting preorder is described by the restriction of functions, and is inherited from the restriction of one open set to another.

David Egolf (Jan 23 2022 at 17:12):

I'm currently thinking about a next step to take in this modelling:

• Environmental perturbations caused by an object can be viewed as objects in their own right
• These objects can then cause subsequent environmental perturbations

Can we figure out a way to "chain together" environmental perturbations?

An example of why we care about this for imaging. We have this chain of perturbations:
heart -> ultrasound wave echoes -> changes in voltage on receive sensing array

David Egolf (Jan 24 2022 at 00:49):

I want to next model the situation where we get to observe only part of a perturbation.

• Let $P(Y)$ be the category of perturbations we've been discussing above. In $P(Y)$, objects are continuous functions from open sets of a topological space $Y$ to a topological space $H$, and there is a morphism $f \to g$ when $f$ restricts to $g$ on the open set that $g$ maps from.
• Let $P(W)$ be a similar category, but defined with respect to a topological space $W \subseteq Y$. I think of $W$ as an observation "window".
• Let an observation process $\alpha_W$ be a functor $\alpha_W: P(Y) \to P(W)$.
  - This functor sends an object $f: U \to H$ to a restriction of this function $\alpha_W f: U \cap W \to H$.
  - This functor sends a morphism $f \to g$ to a morphism $\alpha_W f \to \alpha_W g$.

David Egolf (Jan 24 2022 at 00:55):

Now, I think we can start talking about imaging processes.
Maybe:

• An imaging process $R$ is a function from the objects of $P(W)$ to a subset of $Obj(P(Y))^2$, the powerset of objects of $P(Y)$. So, it associates each object in $P(W)$ with a collection of objects in $P(Y)$.
• We require if $f \to g$, then $R(f) \subseteq R(g)$.
  The reasoning behind the last requirement is: if we observe more things, that should narrow the focus of our estimate for what was present

This definition doesn't require the imaging process to make good estimates for $\alpha_W^{-1}(f)$, which can be viewed as an ideal imaging process. Maybe that's something to think about going forward.

David Egolf (Jan 24 2022 at 16:23):

It seems like a lot of what I'm saying would be retained if we just talked about functors between partial orders. Since each partial order is a preorder, all the stuff on preorders in Seven Sketches should apply.
For example, I think the notion of "upper set" is going to be helpful.

David Egolf (Jan 24 2022 at 16:38):

Related to that, I wonder if a "reasonable" observation and imaging process is going to correspond to a Galois connection.

John Baez (Jan 24 2022 at 17:35):

If observation is a functor $F: A \to B$ and imaging is a functor going back, $G: B \to A,$ then it makes sense to me that we'd like them to be a Galois connection. But this only applies when "imaging" goes back to the original category. Is that really called "imaging"? It's more like reconstructing your best guess of the original scene - the one you're observing.

David Egolf (Jan 25 2022 at 01:38):

I'm having trouble deciding if you're right about this.
Probably that means I need to pick some particular example of something that is certainly "imaging", and see if it fits in this framework.

Some initial musings about something I think of as "imaging":
-We bounce ultrasound waves (with the resulting echoes living in $B$?) off a heart (living in $A$?)
-We then take the resulting echoes, as sensed by a sensor (still living in $B$?) and estimate what they bounced off of (lives in $A$?)
The functor from A to B here (if it exists) would be different than the "windowing" functor I described above, though.
Maybe I should try and work out other examples of "observation" functors besides restricting what part of the perturbation we get to look at (which I call "windowing").

David Egolf (Jan 25 2022 at 01:39):

On a tangentially related note - it feels like somewhere in here one should have to talk about the wave equation (for this example) or more generally a rule that propagates environmental disturbances.

John Baez (Jan 25 2022 at 02:27):

I agree that you should take some of your examples and try to fit them fully - with all relevant details - into some mathematical framework. For this, though, you should start with pathetically simple examples, so you can work them out in detail. (So, not involving the wave equation if you can help it: geometric optics is simpler.)

John Baez (Jan 25 2022 at 02:28):

I have trouble with examples that turn out to be something like "the category of elements of the category of some presheaf on a topological space", since this is so unstructured that I have trouble seeing how you'd get much mileage out of it in imaging.

David Egolf (Jan 25 2022 at 03:35):

Making a really simple example with geometric optics sounds like a great idea!
I don't know a lot about optics, but hopefully I can muddle my way through.
I'm thinking I want to start with making an image of a point source of light using a lens and a screen.

David Egolf (Jan 25 2022 at 16:21):

I see there is such as thing as the category of lenses, but this appears to actually be a concept from computer science.

John Baez (Jan 25 2022 at 16:23):

Yeah, many people here, like @Jules Hedges, @Matteo Capucci (he/him) and @Bruno Gavranovic, love those lenses - they're an application of category theory to computer science.

John Baez (Jan 25 2022 at 16:24):

They have nothing to do with optics.... but wait! "Optics" has also been taken over by category-theoretic computer scientists - it's another technical term that has nothing to do with optics as you knew it.

Fawzi Hreiki (Jan 25 2022 at 16:31):

Do you know where the name originated?

John Baez (Jan 25 2022 at 16:38):

It makes some intuitive sense... in its original application, a lens is a way to view and interact with a database, so it's like there's a big book but you're looking at a portion of it through a magnifying glass.

John Baez (Jan 25 2022 at 16:39):

But I don't know who invented the term and what they said at the time. Maybe the people I tagged know the history better.

David Egolf (Jan 25 2022 at 16:49):

The start of chapter 5 in Optics by Hecht has an interesting description of an "image" of a point.
The picture under discussion is:
optical system

Hecht describes the correspondence between the cone of rays emanating from $S$ to the cone of rays converging at $P$ as an image of $S$.
He then generalizes this idea:
image as energy flow

Here an "image" of point $S$ is described as a point $P$ where enough rays (with enough energy) from $S$ converge.

John Baez (Jan 25 2022 at 16:56):

That's nice! Yes, we don't tend to think often enough about how every object we meet is beaming waves of light in every direction. (The ancient Greeks thought the eye itself sent out beams:

Another established Greek theory of vision described the visual fire. This theory suggested that there was a light emitted by the eye that came into contact with the object being viewed. When the fire hit the object, it reflected back to the eye and the image was seen

David Egolf (Jan 25 2022 at 17:00):

That's a neat historical note!

An interesting thing about this notion of "image" is that it suggests that things can be more or less an image of something else. One could talk about "the extent to which $P$ is an image of $S$" by talking about the fraction of radiant energy from $S$ that travels to $P$, for example.

David Egolf (Jan 25 2022 at 17:11):

There is a more general notion of image going on here, actually:
virtual image

We say that there is a (virtual) image in this case, even though there are no physical rays converging to a point.

John Baez (Jan 25 2022 at 17:11):

Geometric optics is a lot of fun. I haven't studied it enough, but I know Guillemin and Sternberg use it at the start of their book Symplectic Techniques in Physics to motivate the category of symplectic vector spaces and symplectomorphisms.

John Baez (Jan 25 2022 at 17:12):

They use the linear approximation for lenses to treat them as morphisms in this category.

David Egolf (Jan 25 2022 at 17:28):

Let's keep it simple and say an image of $S$ is a point $P$ where multiple diverging rays from $S$ converge.

Let me start thinking about how this relates to the framework from earlier.

Let us say there is a point object $S$, sitting in a 2D plane. It perturbs its environment by creating a collection of spherically emanating rays. We might try to model these rays as a function $r: \mathbb{R}^2 \to \mathbb{R}^2$, so we specify a direction of propagation at each point in space. $r$ will be of a particular form to get a spherical collection of emanating rays.
Then, we introduce an optical system. This will modify the paths that rays take through space, yielding some new function $r': \mathbb{R}^2 \to \mathbb{R}^2$, which will have a more complicated form. We say that $r'$ contains an image of $P$ if there is a point in the plane where rays converge as described by $r'$.
A couple initial thoughts:

• We have begun to talk about a chain of perturbations: a point creates a bunch of rays, and then adding an optical system changes these rays
• The image in this case is not immediately a map from perturbations back to a cause. (However, I am hopeful that it can be viewed in this way, with some work).
• I bet there is a simpler - and more structured - way to talk about ray collections in the context of simple optical systems

David Egolf (Jan 25 2022 at 17:38):

I suppose another perspective on this example is this - the optical system can be viewed as performing processing on the environmental perturbation (rays) produced by $S$ - presumably with the intent to figure out something about $S$. So, perhaps we can think of the modified rays produced by the optical system as in some way an approximation of $S$.

John Baez (Jan 25 2022 at 17:47):

Digressing massively, this short article which just came out has a nice introduction to "lenses" in applied category theory - which as noted above, have nothing (?) to do with what we're talking about:

• John Baez, Simon Cho, Daniel Cicala, Nina Otter, and Valeria de Paiva, Applied category theory in chemistry, computing, and social networks, Notices of the American Mathematical Society.

David Egolf (Jan 25 2022 at 17:50):

Thanks for linking that! I always enjoy learning just a little about something I know nothing about.

David Egolf (Jan 25 2022 at 20:01):

I think there might be some connection, actually, between lenses and iterative (image) reconstruction schemes(?).

From the article linked above, a rough first description of a lens is the following two things together:

• A function $f: X \to Y$
• A function $b: X \times Y \to X$

Let $X$ be a set of unknown objects. Let $Y$ be a set of possible observations of them. Let $f: X \to Y$ send each object to its corresponding observation.
An "iterative reconstruction strategy" is...

• An update function on guesses for an unknown object, given the observed data $y \in Y$
• So, for each observation $y$, we have a function $u_y: X \to X$. This sends any particular guess value to a refined guess value.
• This corresponds to a function $b: X \times Y \to X$

However, to have a lens, there are some additional compatibility laws that need to hold. I haven't checked those yet.

David Egolf (Jan 25 2022 at 20:07):

This compatibility law doesn't appear to hold:
compatibility law

If we have an $(x, y)$, then $b(x,y) \in X$ is the updated guess for an unknown object given observation $y$ and current guess $x$. However, there is no guarantee that the observation that would be observed from the object corresponding to this updated guess $f(b(x,y))$ will actually match the observation obtained $y$. (Although a good optimization scheme will be trying to reduce the gap between the two).

David Egolf (Jan 25 2022 at 20:10):

This law should hold, though (for a reasonable iterative reconstruction scheme):
law 2

If we have some object $x \in X$, and we compute its observation $f(x)$, obtaining the pair $(x, f(x))$, then updating the "guess" $x$ given the observation $f(x)$ should leave $x$ unchanged. There is no way to improve $x$ to better match the observed data $f(x)$ in this case.

David Egolf (Jan 25 2022 at 20:17):

I don't think the third compatibility law would hold.
So, we don't get a lens in this way. Oh well!

David Egolf (Jan 26 2022 at 16:58):

Still digressing on lenses, this is a nice explanatory picture:
lenses

(image from: https://golem.ph.utexas.edu/category/2020/01/profunctor_optics_the_categori.html)
The idea is (apparently) that lenses can describe the process of:

1. specifying some sub-part of some data
2. updating that part of the data (which may change the type of the whole)

David Egolf (Jan 26 2022 at 17:01):

I believe the little $a$ here is called the "focus", continuing the optics terminiology

David Egolf (Jan 26 2022 at 17:07):

According to the nlab, there is a also a notion of "lenses without laws" or "bimorphic lenses".
I wonder if the iterative image reconstruction approach outlined above would have this weaker structure.

David Egolf (Jan 27 2022 at 00:31):

Anyways, returning to a different kind of lens... In an effort to strive towards a "pathetically simple" example of describing an imaging system mathematically, let us consider this sort of situation:
lens

(image source here: http://hyperphysics.phy-astr.gsu.edu/hbase/geoopt/lenseq.html)
Together with the thin lens equation:
$1/o + 1/i = 1/f$
Where $o$ is the "object distance", $i$ is the "image distance" and $f$ is the focal length of the lens.

David Egolf (Jan 27 2022 at 00:33):

Apparently the upside-down arrow produced here is called an "image". However, I feel that this is only appropriate together with an understanding of the optical system that produced it. I could have a black-box optical system that generates an upside-down arrow, and it would not necessarily mean anything helpful about an unknown object.

David Egolf (Jan 27 2022 at 00:34):

So, in my opinion, the "image" is more properly:

• The upside down arrow, together with
• The thin lens equation

David Egolf (Jan 27 2022 at 00:35):

This suggests that, more generally, we may want to think of an image as:

• Some observation, together with
• Some mathematical structure that "explains" the observation

David Egolf (Jan 27 2022 at 00:47):

Probably it would be helpful to work out this arrow-lens example in more mathematical detail.

David Egolf (Jan 28 2022 at 01:00):

I think there are three key pieces here needed as input to an image-making algorithm:

1. An observation - the position of the inverted "image" arrow on the right side of the lens, $i$
2. A "forward model" relating unknown objects to observations: $1/o + 1/i = 1/f$ relates $o$ to $i$.
3. A "feasible set" of unknown objects: in this case, arrows
   Then, to make a simple image, we can take $i$ - the position of the observed image, and solve the above equation for $o$, obtaining $o = 1/(1/f - 1/i)$. Then, using our feasible set of unknown objects, we say "there is an arrow at distance $o$ from the lens".

David Egolf (Jan 28 2022 at 01:03):

These three pieces are also what I need as input to an imaging algorithm I have implemented in MATLAB for photoacoustic imaging. In this case, the pieces are:

1. An observation - the ultrasound waves observed over time on each transducer element
2. A forward model - a map $\Phi$, which maps from measures that describe objects to expected observed ultrasound waves in a noiseless setting
3. A "feasible set" of unknown objects - a collection of "atomic measures", corresponding to a collection of point sources within some area and bounded above in a certain sense by some total amplitude (this incoporates a priori knowledge that the unknown object is a collection of point sources)

David Egolf (Jan 28 2022 at 01:06):

Trying to describe this in a general way, using math, maybe we can say the three key inputs needed for a system that makes images are:

1. An observation: An element $y \in Y$ of some set of possible observations.
2. A forward model: A function $f: X \to Y$, where $X$ is defined below.
3. A feasible set of unknown objects: A set $X$.

I guess a next challenge here would be to ask: Is there something in common about functions $f: X \to Y$ that correspond to "forward models" in the physical world, as opposed to arbitrary functions?

David Egolf (Jan 28 2022 at 01:08):

If you happen to have time, and interest, I would very much appreciate your thoughts on this direction @John Baez. I always find your comments helpful for directing my efforts in more productive directions.

David Egolf (Jan 28 2022 at 16:34):

Oh - on a related note!
I think I've thought of a category related to of our "elements of a presheaf" category we considered above, that should be capable of describing more context-specific structure, and doing more than just talking about spatial restriction of environmental perturbations.
For some indexing set $I$, let each $S_i$ for $i \in I$ be a set, called a "property type".
Let each $P_i$ for each $i \in I$ be a set, called "property values".
A function $f: S_i \to P_i$ is used to specify some property values of type $S_i$.
Define a category where:

• The objects are some subsets $F_i \subseteq Set(S_i, P_i)$. This describes the possible values of property $i$
• The morphisms are some functions $g: F_i \to F_j$. Each function corresponds to a "mode" for the object, and determines the value of property $P_j$ given the value of property $P_i$

Let's see how this could be used in the lens example.
Say there are two properties of the unknown object: its position $o$, and the position of its image as formed through the lens $i$.
The possible values for both of these properties are the element of $\mathbb{R}$, in the absence of prior knowledge.
So, each value for the object position property is a function $f_o: 1 \to \mathbb{R}$ and each value for the image position property is a function $f_i: 1 \to \mathbb{R}$.

To form a category, we need to pick some subsets of $Set(1, \mathbb{R})$ for object locations and also for image locations. For now let us have $F_i = Set(1, \mathbb{R})$ and $F_o = Set(1, \mathbb{R})$, indicating these properties could have any values as far as we know. If we want to incorporate prior information, we could have specified smaller subsets of these collections of functions.

Now we need to define morphisms. We will do this using the lens equation $1/o + 1/i = 1/f$. Let $g: F_i \to F_o$ be the function that send $* \mapsto i$ to the function $* \mapsto 1/(1/f - 1/i)$, where $*$ is the single element of the set $1$. We can similarly defined a $g^{-1}: F_o \to F_i$. These are the non-identity morphisms.

This is a very simple case, where the property are simple, we have no prior knowledge, and our "forward model" perfectly specifies one property in terms of another. It would be interesting to see if this structure can handle more complex cases!

David Egolf (Jan 28 2022 at 16:44):

Consider the example from above, of deformed snow in a field. Let $S$ be a topology, and the $S_i$ be the open sets of $S$. There is only one property value set $P_1=H$, corresponding to the height of the snow.
A continuous function $f: S_i \to H$ describes the height of the snow in set $S_i$.
This suggests we need to allow the $S_i$ to be more than just sets. We want to be able to view them (and $H$) as topological spaces here. This suggests a way to generalize the category we are constructing.

Category version 2:
For some indexing set $I$, let each $S_i$ for each $i \in I$ be an object of a category $C$, called a "property type"
Let each $P_i$ for each $i \in I$ be an object of $C$, called "property values".
A morphism $f: S_i \to P_i$ in $C$ is used to specify some property values of type $S_i$.
Define a category where:

• The objects are some subsets $F_i \subseteq C(S_i, P_i)$. This describes the possible values of property $i$.
• The morphisms are some functions $g: F_i \to F_j$. Each function corresponds to a "mode" for the object, and determines the value of property $P_j$ given the value of property $P_i$

David Egolf (Jan 28 2022 at 16:52):

Now we should be able to describe the "perturbed snow" example in this framework.
Let the category $C = Top$.
Let $S$ be a topological space, and the $S_i$ be its open sets, viewed as subspaces.
Let $P_i = H$ for each $i$, where $H$ is a topological space.
A continuous function $f: S_i \to H$ specifies the height of the snow in area $S_i$.
Define a category where:

• The objects are some specified subsets of $Top(S_i, H)$ for each $i$. We could specify some proper subset here to indicate prior knowledge - for example that the snow can't be more than 10 m high.
• The morphisms are some functions $g: F_i \to F_j$ for $F_i \subseteq Top(S_i, H)$ and $F_j \subseteq Top(S_j, H)$. Each function specifies the height of the snow in region $j$ given the height of the snow in region $i$. We would want to put some structure on these morphisms making use of physical considerations. For example, if $S_i$ contains $S_j$, that should significantly restrict the morphsims that exist between them.

David Egolf (Jan 28 2022 at 16:55):

This framework seems capable of describing a lot of the structure relevant to imaging, including incorporation of prior knowledge and "forward models"!
In general, the more prior knowledge we have, the fewer the objects and morphisms that will exist in the resulting category. The idea is that objects describe "possible property values" and morphism describe "possible relationships between property values" - so if we know more, then there are fewer possibilities.