[deleted]

The results are interesting, but you should include more recent deep learning approaches (not only from GluonTS).

Also, how would someone know this particular combination of stats methods in the ensemble will produce good results beforehand?

>This ensemble is formed by averaging four statistical models: AutoARIMA, ETS, CES and DynamicOptimizedTheta. This combination won sixth place and was the simplest ensemble among the top 10 performers in the M4 competition.

IIRC there was a recent paper that demonstrated how almost all deep learning approaches for time series forecasting use flawed evaluation procedures, resulting in misleading claims about performance and terrible out-of-distribution performance.

[deleted]

The M3 dataset consists only of 3,003 series, a minimal improvement of DL is not a surprise. Everybody knows that neural networks require large datasets to show substantial improvements over statistical baselines.

What is truly surprising is the time it takes to train the networks, 13 days for thousand series

=> there must be something broken with the experiments

13 days to tune multiple deep neural networks is not at all unrealistic depending on the number of gpus.

NBEATs hyper-parameters are minimally explored in the original paper the ensemble was not tuned. There is something broken with the reported times.

14 days is 20k minutes, so it’s about 6.7 minutes per time series. I don’t know how many models are in the ensemble, but let’s assume it’s 13 models for even math, making an average deep model take 30s to train on an average time series.

Is that so crazy?

Depends on the problem. For physical phenomena, statistical techniques are very effective. For more abstract applications, like language and vision, I just don’t know how the purely statistical methods could compete.

All the models are global models, trained using cross learning. Not single models per series. Unless the experiments were done like that.

It's so so so important to set up good experiments with solid baselines and comparisons to other methods.

I’m going to read this paper in detail but I’m wondering if there’s any insight into why DL methods underperform in TS prediction?

I will stick to my Exponential Smoothing good old Boomer technology.

The obvious implication of publish or perish mentality is that you cannot trust papers anymore, given all the hastily produced and broken Deep Learning conference methods.

But those conference papered are Peer Reviewed (TM), the gold standard of those who Believe Science, and hence beyond reproach. You are hereby cancelled.

If you use a flawed evaluation procedure, does a solid baseline do you any good?

Even then. I dabble in image processing at work and haven't found a need for deep learning yet. Every time, there's some trick I can pull with a rule based classifier to address the business need. It's like Duck Hunt: why recognize ducks when you can scan for white vs. black pixels?

I would say forecasting in general is bs.

I agree, for univariate timer series an statistical model should be enough in the most of the cases; however, I still thinks that DL models would outperform statistical models in multivariate time series with a big set of variables, like the MIMIC-III database. Am I wrong with this belief?

Why would you say that?

Hahaha hundreds of days burning GPUs. One can only start to suspect that the purpose of the conferences and deep learning literature is to promote GPU usage rather than improve the forecasting methods knowledge.

💵💵🤖💵💵

That answer is hard to predict.

Just guessing here, but: overfitting.

A lot of people are doing hyperopt with how they set up their experiment to get better results so that they get in prestigious conferences.

The analysis relates to time series prediction problems. Isn't it fair to say vision and language do not fall under that umbrella?

By any chance do you have the name or a reference?

I'm not saying there is some conflict of interest, but...

A lot of time series' are really easy to predict to an acceptable level though.

Then what is your take on physics?

Consider spoken language, and you're back in the realm of time-series. Obviously simple statistical methods can't deal with those though.

Even for univariate time series, when you have the data & complexity, DL will obviously outperform simple methods. Show me the simple statistical method that can generate speech, a univariate time-series.

Upvoted because I agree with you-- for many simple image problems you can even just grayscale and use the distance from the Frobenius Norm of each class as input to a logistic regression and nail many of the cases.

I think you have a good argument for images, but language is more challenging because we rely on positional encodings (a kind of "time") to provide us with contextual clues which beat out the following form of statistical language model: Pr{x_{t+1}|x_0, x_1, ..., x_{t}} (Edit-- that is, predicting the next word in sequence given all preceding words in the sequence)

Right, even though language is a form of time series, in practice it doesn't use TSP methods. Transformers are not surprisingly being applied to TSP problems though.

Yes, DL is a sophisticated tool for the most intractable of tasks, and for most problems is like using the Death Star to crack a nut. This is well known and should just be a normal known thing any analyst of any flavour should have in mind - if you're using DL, especially anything not really big or that isn't natural language or image related, it should be for a good reason, because either a random forest, a GAM, or an autofit ARIMA will get you 80+% of the way there 80+% of the time for tabular data. Not everything needs to start with the biggest guns.

The current, rather excessive, employment of deep learning methods is majorly motivated by the desire to understand them better through the experience gained in applying them.

A good paper that puts this into perspective is from Lea Breiman called "Statistical Modeling: The Two Cultures". He argues in the paper that data based statistical models are preventing statisticians from new and exciting discoveries with algorithmic models. Coincidentally, the author is the creator of the ensemble idea that you are using here as explanation. Now take into account that this was written in 2001 where ensembles were what deep learning is in 2022.

Basically, deep learning is preferred in order to improve it to a point where it will by far outperform all other methods, which it is believed to have the potential for. For it may one day lead us to new and exciting discoveries.

My guess is it’s the same reason we don’t have self-driving cars: bad out of distribution performance. Teslas get confused when they see new leaves where they’ve never been seen before. In the real world, distributions change a lot over time.

I think so too, I’m confused why they would need to train for 14 days, from skimming the paper it doesn’t seem like the dataset itself is that large. I bet a DL solution that was parameterized correctly to the problem would outperform the traditional statistical approaches.

There's https://arxiv.org/abs/2205.13504, but that's specifically targeted at transformers.

[deleted]

I’m just curious, why do you think that?

And it's sometimes kinda hard to realize you're doing a bad job, especially if your bunk experiments give good results

I didn't have a ton of guidance when I was writing my thesis (so, my first actual research work) and was so disheartened when I realized my excellent groundbreaking results were actually just from bad experimental setup.

Still published tho! ^^jk

Isn’t it common knowledge that deep learning is usually not the best solution for modeling time series?

Eh, predicting where pedestrians are going, or predicting next frames in general. Even images have temporal forecasting use cases

While I agree with your general statement, my gut says a well parameterized/regularized deep learning solution would perform as well as an ensemble of statistical approaches (without the expertise needed to select the statistical approaches) but would be harder to explain/interpret.

Essentially simple statistical models are much more eco-friendlier to the planet!

…. am i failing to detect humour/sarcasm? those words don’t appear to say anything

I’ve been doing it for a decade+ and I’m inclined to agree but it pays well and there’s no shortage of buyers. Even straight up named a model GIPSy once with a crystal ball logo, had a pretty good run.

Amen we’ve been doing satellite image time series analytics and deep learning keeps getting pushed off in favor of classification models based on complex features

Sometimes I consider retracting my very first paper because of this.

In practice you don’t, but combiniation forecast still works. This is like saying, “how did someone know that the Total Stock Market Index would outperform bitcoin beforehand”. Combining forecast has been studied in the literature and in practice. It is effective.

To forecasters… yes.

I was thinking the same thing when I looked into this. I'm not sure if the experiments are necessarily 'broken' (there may be at least reasonable justification for why it took 13 days to train), but the first point about dataset size is a smoking gun.

I have not read the paper yet, but the time DL ensemble takes may be due to some kind of hyperparameter search

what do you mean, conflict of interest? doesn't anyone else just buy auditoriums for their besties?

The "and" in the post you replied to was a logical "and". The best evaluation procedure does not help if you use poor, underperforming baselines.

!RemindMe 3 days

I will be messaging you in 3 days on 2022-12-05 09:50:47 UTC to remind you of this link

2 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^(Parent commenter can ) ^(delete this message to hide from others.)

Little need to speculate in this case: they're trying to fit giant models on a dataset that's a fraction of a megabyte, without any targeted pretraining or prior. That's like trying to prove trains are slower than running humans by having the two compete in a 100m race from standstill. The biggest set (monthly observations) is around 105kB of data. If anyone is surprised your average 10GB+ network doesn't perform very well there, well... I suppose now you know.

[deleted]

Lost 9 months of this shit

Great writeup! Reminds me of the excellently named "Cracking nuts with a sledgehammer: when modern graph neural networks do worse than classical greedy algorithms" (https://arxiv.org/abs/2206.13211).

In addition to what other people have said, I'll add this: classical methods work really well. In fields like text and image generation, we didn't have great approaches 20 years ago, and DL models represented a massive improvement.

>like using the Death Star to crack a nut

Or a sledgehammer.

I completely agree with you. I instruct the juniors where I work to start with the most basic possible statistical tests and add complexity only when necessary. A good-enough linear regression is easier to implement, replicate, and understand than a slightly-improved DL model.

Depends how much data you have and how much signal there is. Separating signal from noise in a high-dimensional time series is always a challenge.

Wouldn't a DL model trained on a waveform just assume you were going to keep repeating the same words over and over?

Yes, but I've seen many proposals to apply DL to everyday problems where it's not well suited. Heck, even I briefly went down that rabbit hole with a graph theory problem at work. Tried out a basic greedy algorithm first and it worked well enough that I didn't see the need to get any more complicated.

When I first read your comment, I thought you were still talking about Duck Hunt. I'd read the heck out of that whitepaper.

You could already tape together a deep learning solution consisting of neural speech recognition, an LLM and Wavenet. Counts as a deep learning solution in my book. I'm not sure if anyone has built an end-to-end solution and I expect it would be worse, but I'm sure if someone put their mind and money to it you'd get decent results

If a model has more parameters than datapoints in the training set it can quickly just learn the training set resulting in an over-fit model. You don’t always need 16+ attention heads to have the best model for a given dataset. A single self attention layer with one head still has the ability to model more complex relationships among the inputs than something like arima would.

Yeah

> Just guessing here, but

is a common US English idiom that typically means, "Obviously".

You're absolutely right, though. Just by comparing the training data to the training process and serialized weights, you can see how clearly this should overfit. Once your model is noticeably bigger than a dictionary of X, Y pairs of all of your training data, it's very hard to avoid overfitting.

I volunteer with a group that develops interest and skills in science and tech for kids from historically excluded groups. I was teaching a lab on CV last month and my best student was like, "What if I train for 20 epochs, tho? What about 30?" and the performance improved (but didn't generalize as well). He didn't understand generalization yet so instead, he looked at the improvement trend and had a lightbulb moment and was like, "What if I train for 10,000 epochs???" I should check to see if his name is on the list of collaborators for the paper 😂

Pretty biased selection method: the best ensemble in the M4 competition, evaluated on the M3 competition. Although I'm not familiar with these datasets, they're from the same author, so presumably they have significant overlap and similarity. The real question is how hard is it to find such an ensemble without overfitting to the dataset.

Tabular data is another problem setting where DL has a tough time stacking up to simpler statistical or even ML methods.

How is it biased to try good-performing ensembles in another data set?

And how is that overfitting?

Furthermore, just because the data sets begin with "M" it does not mean that they "have significant overlap and similarity. "

Thanks for sharing the link! This’ll actually work really nicely for a paper I’m writing!

Happy to help!

This PLAGUES my research.

The amount of detail that most papers provide about their statistical methods used as a baseline is not enough to replicate. "We outperformed ARIMA". Didn't provide values, etc.

This is something I have found in my research. I keep seeing people making models with millions of parameters when I am able to achieve 99% of the performance with roughly 1k.

From my personal experience:

- Univariate with a few timesteps: XGBoost or statistical methods.

- Multivariate with many timesteps: NN-based models.

I agree with you

Because if there's overlap in the datasets, or they contain similar data, the exact ensemble you use is essentially an optimized hyperparameter specific for the dataset. It is exactly the reason that for any hyperparameter optimization cross-validation is used on a set separate from the test set. So using the results on the M4 dataset is akin to optimizing hyperparameters on the test set, which is a form of overfitting.

The datasets are from the same author, same series of competitions: https://en.wikipedia.org/wiki/Makridakis_Competitions#Fourth_competition,_started_on_January_1,_2018,_ended_on_May_31,_2018

"The M4 extended and replicated the results of the previous three competitions"

Makridakis Competitions

Fourth competition, started on January 1, 2018, ended on May 31, 2018

>The fourth competition, M4, was announced in November 2017. The competition started on January 1, 2018 and ended on May 31, 2018. Initial results were published in the International Journal of Forecasting on June 21, 2018. The M4 extended and replicated the results of the previous three competitions, using an extended and diverse set of time series to identify the most accurate forecasting method(s) for different types of predictions.

^([ )^(F.A.Q)^( | )^(Opt Out)^( | )^(Opt Out Of Subreddit)^( | )^(GitHub)^( ] Downvote to remove | v1.5)

There is no overlap at all. It’s a completely new dataset. There might be similarities in the sense that there are time series or certain frequencies but in no way could it be the talk of “training in the test” set.

Lol, I'm minus 7 and you're positive 7 karma yet agreeing 😂. Reddit is so stupid sometimes

DL is not working well on low-dimentional samples data, data with low correlation between sample elements, and especially bad for time series prediction which is both. Many people put that kind of senseless projects (DL for time series) on their CV and that is instant black mark for candidate, at least for me. They say "but that approach did work!" I ask "did you try anything else?" "No".