Thanks for this. I knew PyTorch was dominant but didn’t realise it was this dominant for deep learning!

Yeah, not just competitive ML but the research community as a whole seem to have almost entirely switched to PyTorch now (based on the Papers With Code data). I was expecting to see some people using JAX though!

Really detailed analysis! For the Real Robot Challenge you mentioned (very cool!) were people able to test on the robot before the competition/during training?

Thank you for the analysis! Any insights on forecasting and time series related problems? Do most solutions use GRUs? Thanks!

Yeah that one is really cool! They had an initial competition stage, open to everyone, where evaluation was done in a simulation environment (in software) as opposed to real robots. Competitors were given data from dozens of hours of actual robot interaction which they could use to train their policies.

The teams that qualified there made it through to the real robot stage. At that point they could submit their policies for weekly evaluation on actual robots - so they could have a few practice runs on the actual robots before the final leaderboard run.

I found that for a lot of time-series problems, people often treated them as if they were standard tabular/supervised learning problems. There's a separate page of the report which goes into these in detail: https://mlcontests.com/tabular-data?ref=mlc_reddit

For example, for the Kaggle Amex default prediction competition, the data is time-series in the sense that you're given a sequence of customer statements, and then have to predict the probability of them defaulting within a set time period after that. The winner's solution mostly seemed to flatten the features and use LightGBM, but they did use a GRU for part of their final ensemble: https://www.kaggle.com/competitions/amex-default-prediction/discussion/348111

The M6 forecasting competition finished recently, I'm looking forward to seeing what their winners did: https://m6competition.com/

Good job u/hcarlens and MLcontests team! Thank you for the insights!

That’s a lot of work 🫡. Thanks

JAX does offer some general matrix math that can be more useful/fast than torch alone. I often do deep learning with torch and then use JAX on the top to train statistical models (i.e. fuse features from multiple models, raw features, etc. into a single regression/inference)

Amazing. This seems like a great way to learn how things are currently being done in ML

🫡 F

Can somebody elaborate on the discrepancy between PyTorch and TF?

I keep hearing that Torch is preferred for research and academic purposes, but TF seems to be very close in terms of accuracy and performance.

What factor makes Torch users over an order of magnitude more popular here?

This is amazing, thank you! Where do L2R problems fall in your classification? Tabular?

Awesome roundup. I wonder, was https://grand-challenge.org/ included in your analysis?

see this post : https://www.reddit.com/r/MachineLearning/comments/m3boyo/d_why_is_tensorflow_so_hated_on_and_pytorch_is/

Is pytorch also better than TF for usecases where I have to do training/inference on mobile?

Great one, thanks ;)

Interesting meta-study with many remarkable trends.

This is seen from the competitor's side, but what correctly the best website to set up a simple prediction competition? I'm asking this since I'm planning on creating a small competition for students of one of the courses given (no large files needed).

Thanks! I didn't create a separate category for learning-to-rank problems because they often overlap with other domains.

For example, some of the conservation competitions (https://mlcontests.com/state-of-competitive-machine-learning-2022/#conservation-competitions) are L2R problems on image data.

Or the Amazon/AIcrowd competitions (https://mlcontests.com/state-of-competitive-machine-learning-2022/#nlp--search) which were L2R with NLP.

In reality the mapping of competition:(competition type) is almost always one:many, and I'm planning on updating the ML Contests website to reflect that!

If I'd had more time and better data I would have sliced the data in multiple different ways to also look into e.g. L2R problems specifically in more depth.

Thanks! Yes, but I didn't manage to get as much data as I wanted for the competitions on there. I emailed some of the competition organisers but didn't get a response.

Another, older post on this: https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry/

Thanks! It depends on exactly what you're planning, but CodaLab (https://codalab.lisn.upsaclay.fr/competitions/) or their new platform CodaBench (https://www.codabench.org/) will probably work well.

They both allow you to run competitions for free, and people do use them for teaching purposes (you'll see some examples if you browse through the list of competitions).

I'm planning on writing a shorter blog post on running competitions soon.

Also a cool read, thanks!

What you accomplished is wonderful enough. I agree wholeheartedly with your 1:n mapping

For noobs like me: GBDT=gradient boost decision tree

This was a fascinating read, thanks a lot!

Thank you very much u/hcarlens !
This distills so much of information .

Do you think that we might be at a plateau in terms of new methods being tried out (say, since last 24 months) ?

Hi! I'm not sure I fully understand your question, but if you're asking whether the rate of progress in competitive ML is slowing down, I think probably not. A lot of the key areas of debate (gbdt vs nn in tabular data, cnn vs transformers in vision) are seeing a lot of research still and I expect the competitive ML community to adopt new advances when they happen. Also in NLP there's a move towards more efficient models, which would also be very useful.

Good point! Edited to clarify.

Thank you (you cleared my doubt) !