I think that this could totally work. My masters thesis was semi related to this, I used pytorch and instead of vectors x and y I was solving for trajectories that minimized a loss function. Try mocking up a toy version where it's easy to check that it works.