Submitted by a_khalid1999 t3_10ocalm in MachineLearning
On This page of Meta AI research where they mention AI theory as a topic, they mention that they use techniques from Signal Processing. As someone with an Electrical Engineering background, and interests in Mathematics and AI, I found this very intriguing. Can someone tell me some of the ways signal processing has been used in AI theory? Some papers or some work done?
To add to this Receiver Operating Characteristic (ROC) curve also comes from signal detection theory which was established for detection of objects using radio signals.
I see. I graduated as a EE and found signal processing (somewhat along with control theory) pretty interesting cuz of all the math involved. Enjoyed ML too, but a lot of the work around me was more into programming applications using ML rather than tinkering with the math behind it, so was looking for a way to merge signal processing and AI theory...
[deleted]
Looking at your background you're a recent (or soon to be recent) BS graduate? I'm asking because when you get into graduate level coverage of these topics there's considerable overlap. In terms of papers, etc I'm not sure you could find a clear line where one field starts and the other ends. Maybe if you tried hard you could say that one has more focus on data-driven methods or something, but I think you'd be able to find so many counter examples that I'd question the point of it.
So to answer your question, I'd say it's all around you. If you're looking, you've probably already seen it.
Yeah, I'm a recent graduate. I have seen signal processing and ML overlap somewhat, but when I saw it written on the site, it felt there was some fundamental use or something.
>If you're looking, you've probably already seen it.
Getting wise master-ish vibes :)
One of the bull's eye contributions of signal processing to deep learning was this paper. From a signal processing perspective, naive pooling is obviously problematic because you're decimating without limiting the signal bandwidth. That paper showed that in 2019. Shows how much computer vision has changed from an EE-dominant field to a CS field, where signal processing is not common knowledge.
Interesting perspective. I did not know Computer Vision was a EE-dominant field at one point, I mean I knew Image Processing is a EE thing, but Vision just gave the ... CS vibe, I mean when I took it in my Bachelor's it was labelled as a CS course.
So basically one way of looking at things could be, and as a EE I'm obviously biased, it's not the Signal Process Engineers moving into ML, it's the CS guys starting to use Signal Processing, cuz where I've been the impression is always that AI is completely a CS thing and the EE's coming in this field are coming due to the lack of job opportunities
That's a more recent trend. Until the late 2000s, computer vision was basically combining machine learning techniques with image processing: Design filters to extract features, and slap them into a classifier. Naturally, lots of Fourier, wavelets, and other weird bases. Very different times.
Sounds fun. During my Bachelors I took ML courses along with Signal Processing and other EE stuff wanting to somehow specialize in some intersection, now ending up kinda in this identity crisis of being a CS or a EE. Make EE great again! :D
I would argue that EE is actually a better major than CS for ML. It beefs up your Math and Statistics chops with DiffEQ, Quantum, and the like, and also includes enough Linear Algebra and Statistics to get you sorted. As a Math major doing ML research, I'm kind of embarrassed by how weak my background in Signal Processing is, and am working through a textbook on DSP in my spare time to fix that.
Guess we shouldn't be taking the EE background for granted
That’s a great paper! I like this paper too that looks at ViT’s through a signal processing lens and points out some potential flaws in the architecture for vision applications
Computer vision is built on top of image processing, which is basically just 2d signal processing
At this point, I feel dumb for writing this post. If I ak not mistaken, Dr. Lecunn has a ECE background before his PhD, also evident from the invention of CNNs, that have convolution, fundamentally a EE thing
The S4 Transformer uses structured state spaces which is a concept from EE that models the hidden state with differential equations. Seems to have SOTA results on a lot of tasks
Thanks, will look into it
CNNs are incorrectly named because they use autocorrelation functions which is super common technique from extracting signals from noise. The CNN is essentially learning filters to extract those signals from images
Correlations are the same as convolutions with just the kernel flipped.
signal decomposition (Fourier, SVD, Laplace), compression, convolutions, it’s also adjacent to lots of linear algebra/search(inner product spaces, kernels, mean squares error, optimization, etc), I’m only scratching the surface
Thanks
I studied EE Signal processing in Bachelors and Masters, and now work as a machine learning engineer.
In signal processing, you take the signal, define a desirable output and try to create a system to get to that output.
In ML, you typically have the input and the output, and your machine figures out the system depending on the network architecture you provide.
Both are solving the same problem most times, and extensive signal processing knowledge helps a ton in understanding what's happening/what should happen within a network.
And FWIW EE signal processing has most requirements for AI covered in their coursework, and makes good ML engineers minus production level coding.
I see. As far as research is concerned, I guess, EE's in ML shouldn't be seen as "changing their field"
Well like reinforcement learning uses a lot of markov chains, forward/backwards filtering/smoothing etc. Kalman filters are also a sort of Gaussian Process Regression. There is a huge overlap in the classical ML part with signal processing. No specific paper but it's just that ML and especially deep learning often takes already existing ideas from physics or ee and try to apply it on some data, see what happens.
RL sure seems like a fun field to get into
Seems to make sense considering how convolutions are simply multiplications in the Fourier domain
A_HumblePotato t1_j6dye7c wrote
Modern machine learning is really an amalgamation of older disciplines (signal processing, control theory, optimization, estimation theory, etc…). One example is the LASSO algorithm was originally developed for signal processing applications. So to get to your question, it’d be impossible to capture its scope! Topics like adaptive signal processing, optimal signal processing, image processing, pattern recognition, and estimation & detection theory are all traditional signal processing topics that have merged into the AI field.