vocdex OP t1_isofmvp wrote on October 17, 2022 at 2:35 PM

Reply to comment by jake_1001001 in [D] Clustering after instance segmentation by vocdex

Thanks for suggestions.

I don't have a labeled dataset but I can create one, for sure. The object here is asparagus in a greenhouse farm.

Here's the situation: I am using segmentation because in the future, I want to use this segmentation with depth maps to create point clouds. I have tried to do so with only bounding box detections but due to the presence of background and foreground pixels (different depth image values), I am getting quite bad point clouds. Then, I applied simple depth value based filter to crop out only the object without any back/foreground. This works but doesn't generalize well to all situations.

I thought that instance segmentation would give me only the object pixels and I can fuse this with depth values in order to get point clouds.

Moreover, there could be different clusters other than bent vs straight. So, I want the clustering algorithm to find those clusters in an unsupervised fashion. If this doesn't work, then yes, I guess I'll have to create a dataset and train another bent vs straight classification model.

Thanks for reading till here!

jake_1001001 t1_isoot8f wrote on October 17, 2022 at 3:43 PM

Aha, ok, use of segmentations to extract the object point cloud seems good and I have used similar approach for face reconstruction l.

Have you tried 3D approachs (ridgid and non rigid alignment)? How similar are the objects? you could use the dense alignment error to determine if the object is the same as a streight one.

But if we go back to image based methods, if your segmentation model is good, it may provide good embeddings already in the encoder. You could take those embeddings and compute thier distance to the embeddings of templates (straight, bent, etc). Kmeans may not cluster as you expect if there is a high variance in samples (shape, size, color, etc), which is why supervised methods could be preferred. Templates provide a prototype for your class to compute distance/similarity to (Euclidean, cosine similarity) . It is crude, but could work in constrained settings.

vocdex OP t1_isrjl7v wrote on October 18, 2022 at 4:16 AM

Ah, haven't considered 3D approaches but definitely check them out. Objects are quite similar (green color, just the shape is different). Thank you for your help