Viewing a single comment thread. View all comments

patrickkidger t1_iv5vb05 wrote

Neural differential equations! The continuous-time limit of a lot of deep learning models can be thought of as a differential equation with a neural network as its vector field.

A survey is On Neural Differential Equations.

Also +1 for /u/betelgeuse3e08's recommendations, which are primarily neural ODEs encoding particular kinds of physical structure; c.f. Section 2.2.2 of the above.

You can find a lot of code examples of neural ODEs/SDEs/etc. in JAX in the Diffrax documentation.

This topic is kind of my thing :) DM me if you end up going down this route, I can try to point you at the open problems.

14

Gaussianperson t1_iv9bt1z wrote

Hey Patrick! Huge fan of your research :). It’s a really cool topic imho. Could you share the open problems?

1

patrickkidger t1_ivb3t7e wrote

See the conclusion of my thesis (linked above ;) )

TL;DR: everything neural PDEs, stable training of neural SDEs, applications of neural ODEs to ~all of science~, adaptive/implicit/rough numerical SDEs (although that one's very specialised), there's current work connecting NDEs with state space models (S4D, MEGA, etc.), ... etc. etc!

1