I think you need a comprehensive benchmark, you might find your deep learning model to miserably fail even in a simple scenario. Thus I would recommend to double check the requirements, if your business does not particularly care about some possible bad predictions, then it should be fine, otherwise I would look for some more deterministic models.

1. Can’t you also create custom loss functions for XGBoost? I’ve never used it myself but it seems as easy as doing it for an ANN

2. Is it always trivial to get meaningful embeddings? Does taking the last hidden layer of ANN guarantee that representation will be useful in many different contexts? I think it might need more work than you expect. I’m actually looking for a write up about what conditions needs to be met for a hidden layer to provide meaningful embeddings. I think using a triplet loss intuitively favors that but I’m not sure in general.

3. XGBoost allows for this too, doesn’t it? The scikit-learn API definitely at least let’s you create MultiOutput models very easily. Granted it can be silly to have multiple models under the hood but whatever works.

Sorry I’m playing devil’s advocate here, but the vibe I’m getting from your post is that you’re excited to finally getting to play with DNN. Which I can relate to. But don’t get lost in that intellectual excitement: at the end of the day, people want you to solve a business problem. The fastest you can get to a good solution the better.

In the end it’s all about trade offs. People who employ you just want the best value for their money.

You can write your own custom functions for tree based models.

Here is an pretty good industry show case with lightGBM. Found it on reddit:

https://doordash.engineering/2021/06/29/managing-supply-and-demand-balance-through-machine-learning/

It comments on their custom optimizer.

Btw depending on the task, you can also check out temporal fusion transformers (forecasting, nlp, Image recognition)

So one consistent feedback I have got from financial services customers is transparency and explain ability. My understanding is these 2 factors are reason why the adoption of dl in low. Especially when you have tons of audit and regulatory requirements.

I am with you. While I generally favor trees for tabular data, there are some advantages of deep learning as you mentioned. I haven't heard many success stories out of industry for moving away from trees to deep learning, outside of Sean Taylor talking about using deep learning at Lyft. My guess is the extra complexity of using deep learning is probably only useful in a small set of use cases.

Deep learning is probably also useful in multimodal use cases. If people are using deep learning for tabular because of these advantages, I would love to hear about it.

Take a look at TabPFN, which uses meta-learned networks for tabular data prediction: https://www.automl.org/tabpfn-a-transformer-that-solves-small-tabular-classification-problems-in-a-second/

What’s the TLDR for this? I’ve meant to try it but never got around to

The authors trained a transformer that takes a tabular data set and can learn SoTA embeddings on categorical data in under a second

Don't get attached to any particular method. Train and combine them all (ensemble learning) to get the best possible loss on your validation data you could hope for.

Sweet

for datasets with ~1000 rows and 10 columns iirc

Yep exactly. Been keeping an site out but afaik, still the case. And I mean, what’s the point of that

Yeah, good luck maintaining those ensembles in production!

Small correction - 100 columns, and only classification (not regression) so yeah, not very practically useful yet, though a really cool idea

I agree with you, another benefit I might add is the scalability to very large data.

To give an example, the team I work on processes point cloud data, which is easily in the millions or billions of points for a single dataset. Random forest is popular to do per-point classification of the point cloud. However, you need distributed computing for large real-world datasets (think large geographic extents), whereas with a simple MLP, you can train the model in batches with a GPU. Multi-GPU is the next step for scaling. Very natural progression here. Inference is still blazing fast here too.

I personally see DL models as a flexible and modular way to build models with the benefit of improving a model through deeper networks, different activation functions, and network modules. If you need to go simple, just use less layers :-)

As others have mentioned, use the tool which fits the problem. But, a neural network does have the advantages you mentioned, and should also be considered.

For me, the only time deep learning was worth all the work was when I had a large unlabelled dataset and I've done pre-training with it before the main task.

Sorry, I’m not following, what’s the problem here? I had done this but the model inference only had to be run once in the morning every morning so it was a little different. Is there something else I’m missing with the maintenance of the ensemble?