Viewing a single comment thread. View all comments

lustiz t1_isza6yn wrote

What is your goal here? If your evaluation metric doesn’t care about the form of F, i.e. only cares about how well your predictions do, this is just standard supervised learning. However, given the way you phrased it, it sounds like there are some geometric or structural properties of your problem that allow you to synthesize training data. Just note that, if you can exploit structural properties of your problem, it may be much faster using standard nonlinear 2nd order optimizers. Will depend on what you are trying to do, though..

1

joeggeli OP t1_it26unz wrote

Alright, I'll try to explain ...

My goal is to convert 3d scenes to bas-reliefs. The steps for this are:

  1. Compute the height and normal maps of a 3d scene, as seen from a specific point of view.
  2. Compress the heights in such a way that the overall height is reduced, while the shapes and details of the original 3d scene remain intact.

The first step is simple. The second step is a very difficult problem. There exists no method to do this in an optimal way. There are many approaches to formulate it as linear or quadratic optimization problems.

I'm currently solving it by formulating an optimization problem (which is basically the loss function L = G(x, y')), reformulating it into a poisson-like equation and then solving it using a multigrid method. This is a very standard way to compute such bas-reliefs from height or normal maps. The problems with this approach are:

a) It is relatively slow to transform a single image (1024x1024 takes 1-2 minutes). Neural networks could have the potential to be much faster. My goal is to be as close to real-time as possible. b) It is too restrictive because I have to formulate my loss function in such a way that it results in a linear system. This negatively affects the quality of the final result.

It is possible to compute how well a converted bas-relief y' = F(x) represents the original height map x, which leads to the loss function. As an example, you could compute the similarity of the normal vectors of x and y' (which will already result in a non-linear optimization problem). Similar normal vectors = low loss; different normal vectors = high loss. This type of non-linear optimization problem is what I want to solve, which is what lead me to the idea to solve this problem using a CNN.

1