Submitted by FresckleFart19 t3_z2hr4c in MachineLearning
dpineo t1_ixhd0tq wrote
Reply to comment by iamthesexdragon in [R] Category Theory for AI,AI for Category theory by FresckleFart19
I'm bullish on CT in ML. I predict that in 10 years, that the word "functor" will be commonplace, similar to how the word "tensor" is now.
I also predict that we will misuse the word "functor" to mean "function", the way "tensor" is currently misused to mean "array", and mathematicians will continue to curse us for screwing up yet another one of their terms.
Matsarj t1_ixiaoj1 wrote
I'm someone with a PhD in a field that uses category theory extensively and I now work in DS. I'm finding a lot of ideas in this post unmotivated, I guess I'm pretty bearish on CT being applied to ML. Can you explain what problems you see that category theory will be used to solve?
yldedly t1_ixigekm wrote
I stumbled on this thesis some time ago, where the author formulates a category of causal model, where arrows are structure-preserving transformations between models. Seems like it would be useful for causal model discovery.
Matsarj t1_ixijut0 wrote
That looks interesting, and I'm definitely not saying there's no intersection between CT and DS. There's some cool things I've seen with CT and probability theory recently. But to me it often seems like theory in search of a problem, and a far cry from functor becoming a common word among ML practitioners.
LazyHater t1_ixipeej wrote
CT is kinda good for ML if you have a complex topology of solution spaces. When programmers try to implement categories from a naive view, instead of applying sophisticated categorical constraints on their models, I definitely feel a sort of way about it. With that said, LLMs with analytic modules should be able to do categorical constructions in the not too distant future, which will be nice as hell. Optimizing functors might be a thing someday too, but its definitely not there mathematically yet
Im bullish on deriving (co)homologies using ML but it will be some time before we get there i think.
Matsarj t1_ixirp1l wrote
I guess I'm separating purely categorical applications from TDA applications, which I agree things like persistent homology will probably be useful.
LazyHater t1_ixiuzpu wrote
Persistent homology for real data is developing plenty of classical techniques, but we probably need a very good LLM with some very good HoTT to derive non-simplicial (co)homology for some given category -> some given abelian category
Matsarj t1_ixiwioj wrote
This sounds really interesting. Can you expand here or link to any resources related to this? I'm most interested in where you would apply these cohomology theories.
LazyHater t1_ixj3e2t wrote
Resources are quite scarce I'm afraid. Emily Riehl and company are working on (inf,1) categories to establish homotopy between derived functors, for applications in univalent foundations. For a computer algebra system or proof assistant, type equivalence is required to abstract away implementation details. To actually compute homotopy equivalence, it's better to compute cohomology equivalence, but simplicial cohomology is often too expensive to compute. So it's an open problem whether we can optimize a derived homology functor between a derived (enriched) functor and an abelian (enriched) category (which still lacks proper definition afaik). But its a goal I heard at a HoTT talk once to get non-simplicial cohomology of types instead of computing homotopy (which is computationally impossible at scale). Feel free to steal and spread this idea but it's kinda original and speculative.
tl;dr application is computing homotopy equivalence of types at a reasonable expense
Matsarj t1_ixjliwb wrote
So I'm pretty familiar with homotopy theory but don't know any type theory, homotopy or otherwise. What does determining whether types are homotopy equivalent get you in terms of ML applications?
LazyHater t1_ixjq0v8 wrote
In "laymans" terms, it gives you a) an environment for ML models to verify their proofs and b) and rich space for ML to study relations between different fields of mathematics, logic, philosophy, ethics, and everything else by default at that point.
propositions are implementations of types so just being able to say when propositions are equivalent in like a rigorous way is good for science anyways.
Phoneaccount25732 t1_iybqxxl wrote
Does category theory continue to be as insanely mind-blowing once you actually understand some of it?
LazyHater t1_iyeaom6 wrote
Yes and no. The fundamental ideas, once they start to sink in, show clear parallels between vastly different fields of analytic thought. The more you understand the framework though, the more its limitations can be concerning. Dependence on the axiom of choice, for example, and the naturality of choice in the field itself, leads some to speculate that if contradiction can be chosen true, the theory's implementation (with the vast majority of categorical proofs appealing to choice) is completely broken.
It's overwhelming at times how applicable category theory is from the right perspective, but underwhelming how its implementation in set theory can be expected to pan out.
tl;dr: category theory is dope but aoc is sus
dpineo t1_ixj7ot5 wrote
Sure. I see the potential of CT as being a language for expressing, reasoning about, and ultimately designing, AI/ML architectures abstractly.
In software development, we have the concept of "design patterns" that provide a common vocabulary with which we can describe common recurring patterns in software design at an abstract level. It cuts past the implementation details and allows us to focus on larger concerns, such as the composition and coupling of components, and the flow of information. This maturity in software development has allowed us to grow past brute forcing spagetti-code programs to developing robust enterprise-sized systems.
I believe that AI/ML is still in it's spagetti-code infancy. We have no idea how to build and compose AI/ML components into a system a disciplined way. To scale up to larger and more complex AI/ML systems, we're going to need to step back and look at AI/ML architectures more abstractly the way that software did with design patterns. I think CT may be able to help with that.
DeStagiair t1_ixiq07v wrote
I, for one, can't wait until abstract nonsense takes over probability theory*.
ML/DS definitely has problems with papers sometimes playing fast and loose with the theory and instead focusing on getting +0.1% on your favorite leader board. "We make model bigger and big model make good predictions." In all seriousness, I think stuff like this is interesting, but I don't think it's very useful as of now. The fact that everything is just an array in Python has always irked me a bit, but maybe there can be some way to leverage type theory in some way to make the ML code more robust. Or perhaps CT can be used to make it easier to compose probabilistic models.
> * From Disintegration and Bayesian Inversion via String Diagrams
Viewing a single comment thread. View all comments