I am using pointnet

I have a point cloud segmentation problem. In my training data, I have 1 class, but on average only ~4% of all points per point cloud are of that class, and are usually found grouped together (same object).

How do I balance this?

If I remove most points that aren't in the class, then the point cloud will become sparse and it would be too easy to spot where the class is, since only ~8% of points will remain.

Or is there a way to train this well without balancing the training data?

Thanks!