Looking at your background you're a recent (or soon to be recent) BS graduate? I'm asking because when you get into graduate level coverage of these topics there's considerable overlap. In terms of papers, etc I'm not sure you could find a clear line where one field starts and the other ends. Maybe if you tried hard you could say that one has more focus on data-driven methods or something, but I think you'd be able to find so many counter examples that I'd question the point of it.

So to answer your question, I'd say it's all around you. If you're looking, you've probably already seen it.

Modern machine learning is really an amalgamation of older disciplines (signal processing, control theory, optimization, estimation theory, etc…). One example is the LASSO algorithm was originally developed for signal processing applications. So to get to your question, it’d be impossible to capture its scope! Topics like adaptive signal processing, optimal signal processing, image processing, pattern recognition, and estimation & detection theory are all traditional signal processing topics that have merged into the AI field.

To add to this Receiver Operating Characteristic (ROC) curve also comes from signal detection theory which was established for detection of objects using radio signals.

Yeah, I'm a recent graduate. I have seen signal processing and ML overlap somewhat, but when I saw it written on the site, it felt there was some fundamental use or something.

>If you're looking, you've probably already seen it.

Getting wise master-ish vibes :)

I see. I graduated as a EE and found signal processing (somewhat along with control theory) pretty interesting cuz of all the math involved. Enjoyed ML too, but a lot of the work around me was more into programming applications using ML rather than tinkering with the math behind it, so was looking for a way to merge signal processing and AI theory...

Computer vision is built on top of image processing, which is basically just 2d signal processing

One of the bull's eye contributions of signal processing to deep learning was this paper. From a signal processing perspective, naive pooling is obviously problematic because you're decimating without limiting the signal bandwidth. That paper showed that in 2019. Shows how much computer vision has changed from an EE-dominant field to a CS field, where signal processing is not common knowledge.

[deleted]

Interesting perspective. I did not know Computer Vision was a EE-dominant field at one point, I mean I knew Image Processing is a EE thing, but Vision just gave the ... CS vibe, I mean when I took it in my Bachelor's it was labelled as a CS course.

So basically one way of looking at things could be, and as a EE I'm obviously biased, it's not the Signal Process Engineers moving into ML, it's the CS guys starting to use Signal Processing, cuz where I've been the impression is always that AI is completely a CS thing and the EE's coming in this field are coming due to the lack of job opportunities

At this point, I feel dumb for writing this post. If I ak not mistaken, Dr. Lecunn has a ECE background before his PhD, also evident from the invention of CNNs, that have convolution, fundamentally a EE thing

That's a more recent trend. Until the late 2000s, computer vision was basically combining machine learning techniques with image processing: Design filters to extract features, and slap them into a classifier. Naturally, lots of Fourier, wavelets, and other weird bases. Very different times.

Sounds fun. During my Bachelors I took ML courses along with Signal Processing and other EE stuff wanting to somehow specialize in some intersection, now ending up kinda in this identity crisis of being a CS or a EE. Make EE great again! :D

That’s a great paper! I like this paper too that looks at ViT’s through a signal processing lens and points out some potential flaws in the architecture for vision applications

Well like reinforcement learning uses a lot of markov chains, forward/backwards filtering/smoothing etc. Kalman filters are also a sort of Gaussian Process Regression. There is a huge overlap in the classical ML part with signal processing. No specific paper but it's just that ML and especially deep learning often takes already existing ideas from physics or ee and try to apply it on some data, see what happens.

The S4 Transformer uses structured state spaces which is a concept from EE that models the hidden state with differential equations. Seems to have SOTA results on a lot of tasks

CNNs are incorrectly named because they use autocorrelation functions which is super common technique from extracting signals from noise. The CNN is essentially learning filters to extract those signals from images

Correlations are the same as convolutions with just the kernel flipped.

I would argue that EE is actually a better major than CS for ML. It beefs up your Math and Statistics chops with DiffEQ, Quantum, and the like, and also includes enough Linear Algebra and Statistics to get you sorted. As a Math major doing ML research, I'm kind of embarrassed by how weak my background in Signal Processing is, and am working through a textbook on DSP in my spare time to fix that.

signal decomposition (Fourier, SVD, Laplace), compression, convolutions, it’s also adjacent to lots of linear algebra/search(inner product spaces, kernels, mean squares error, optimization, etc), I’m only scratching the surface

https://towardsdatascience.com/how-fourier-transform-speeds-up-training-cnns-30bedfdf70f6

Seems to make sense considering how convolutions are simply multiplications in the Fourier domain

Guess we shouldn't be taking the EE background for granted

RL sure seems like a fun field to get into

Thanks

Thanks, will look into it

I studied EE Signal processing in Bachelors and Masters, and now work as a machine learning engineer.

In signal processing, you take the signal, define a desirable output and try to create a system to get to that output.

In ML, you typically have the input and the output, and your machine figures out the system depending on the network architecture you provide.

Both are solving the same problem most times, and extensive signal processing knowledge helps a ton in understanding what's happening/what should happen within a network.

And FWIW EE signal processing has most requirements for AI covered in their coursework, and makes good ML engineers minus production level coding.

I see. As far as research is concerned, I guess, EE's in ML shouldn't be seen as "changing their field"