Hi folks,

stylepoint here.

I have released the YouTube series discussing and implementing activation functions.

Videos:

GitHub: https://github.com/oniani/ai

Some notes about the series:

In every video, I discuss the activation function before implementing it.
In every video, I compute/derive the derivative/gradient of the activation function.
In every video, I provide two implementations for the activation function - manual and using PyTorch's autograd engine.
In every video, I use gradcheck to test the implementation.
Every video has timestamps, so you can skip parts that are not of interest.
There is not a lot of interdependence across the videos, so you can watch some and skip others.

Hope y'all will enjoy these vids!

Comments

Gemabo t1_ircq1tp wrote on October 7, 2022 at 1:12 AM

#49,145

Bookmarked!

mister-guy-dude t1_ircwrn7 wrote on October 7, 2022 at 2:09 AM

#49,435

This is great! Thanks for creating it!

itsstylepoint OP t1_ircxamq wrote on October 7, 2022 at 2:14 AM

#49,456

Replying to mister-guy-dude (#49,435)

Hey thanks for the kind words!
Will be making more AI/ML YouTube series in the future - in fact, working on one as we speak!

_chyld t1_irdmjiv wrote on October 7, 2022 at 6:39 AM

#50,360

Excellent, I'll check them out.

ElderberryFine t1_irdvi1k wrote on October 7, 2022 at 9:03 AM

#50,632

Thanks

awebb78 t1_irdxw4o wrote on October 7, 2022 at 9:42 AM

#50,713

Sounds very interesting

Erosis t1_irewcom wrote on October 7, 2022 at 3:24 PM

#52,170

Your videos have been great so far! Can't wait for more modeling content.

itsstylepoint OP t1_irfs7wv wrote on October 7, 2022 at 7:24 PM

#53,540

Replying to ElderberryFine (#50,632)

Thank you!

itsstylepoint OP t1_irft9uh wrote on October 7, 2022 at 7:33 PM

#53,589

Replying to Erosis (#52,170)

Thank you!
Yup, that is the plan! Will likely make a few more series (about gradient descent, optimizers, etc.) first. We need these for DL and if someone asks how things work, I could then cite the appropriate video series. After that, will dive into deep learning.

pm_me_your_ensembles t1_irjbb1t wrote on October 8, 2022 at 5:36 PM

#58,950

Do you go over numerical stability issues?

itsstylepoint OP t1_irjh4n3 wrote on October 8, 2022 at 6:23 PM

#59,206

Replying to pm_me_your_ensembles (#58,950)

Yup, all implementations are numerically stable.

Note that I do not discuss numerical stability issues for all activation functions, but for those where the intuitive implementation is not numerically stable (i.e., Sigmoid, Tanh).

I also have a separate video discussing numerical stability: AI/ML Model API Design and Numerical Stability (follow-up). But this is in the context of Gaussian Naive Bayes.

pm_me_your_ensembles t1_irjlb0n wrote on October 8, 2022 at 6:57 PM

#59,369

Replying to itsstylepoint (#59,206)

Great! Thank you for sharing! :D

CeFurkan t1_irmeyx2 wrote on October 9, 2022 at 11:24 AM

#62,877

how are you able to program in a such fashion that without doing any debugging and you are sure they are working correctly as intended?