Why do you even need to understand everything? FOMO?

Either you work on fairly general topics, or more generally applicable things, and then you don’t need to read a bazillion papers that propose yet another flavour of attention.

Or you work on something more specific within one domain, e.g. NLP, and then you don’t need to know the details of e.g. image generation that you brought up.

Pro-tip: high level understanding of the field as a whole, and solid understanding of your specific niche is what you’re after.

This. We need to start treating ML as a collection of fields with synergies. Just like they teach in engineering school, you're going for a T shaped skill tree. Be generally knowledgable about what's out there, but do a deep dive of one particular niche that interests you.

Also, it should be said that if you're not a researcher, one of the hardest parts of AI is the data. Everyone gets this wrong, particularly when they start chasing the latest backbone, or loss function, or a hundred things that see dozens of publications per week on. The ML engineering around standing up a good experiment, regardless of what you're actually putting into the model slot of this experiment, is where 90% of the effort should be going.

I have burned out trying to chase all the new papers. Recently I have just given up. I guess age is catching up but I want to take a break and just focus rather than reading papers all day long.

[deleted]

> I am wondering if ML needs to do what physics did a while ago, and just give up on trying to understand all of it.

I think a lot of people in ML already have been doing that. This doesn’t need to be a widely acknowledged shift. You research what you’re interested in, that’s how you specialize.

>Why do you even need to understand everything? FOMO?

FOMO is part of it. I'm interested in ML in general, and think that AI is coming (really unsure when) and I want to be part of all of it. I have this fear that someone is going to do something very important and I just won't know about it.

There is also the expectation (at least where work) that when you are a 'Machine Learning Engineer' that you have a pretty good grasp on the field as a whole. You don't want someone to say 'What do you think of Random Forests, how do they work?' and you go 'What's a Random Forest?' (hyperbolic example). I kind of sucks when you are the 'ML Guy' in a pretty big company so people come to you with (random) questions.

Finally, I'm old enough that I think that when ML started, I did know most of it. Having been raised on Duda and Hart and then was around when backprop made it's second renaissance, I remember when Elements of Statistical Learning came out. So, perhaps it's a sadness that the field has blossomed beyond me.

If we assume AI is a tool to solve problems and it's more prevalent as the years go by, we are not going to keep up with the whole new research, because specialists in many different industries will create models for specific purposes. We can try to be generalists, but just one human doesn't have enough time to consume the content generated by thousands. In the end, we can try to learn general models, have a superficial understanding of promising new techniques, and learn about a handful of specific problems to be a specialist in those.

I feel the opposite. While there is a lot more cool stuff out there then there was 10 years ago when I started. The work has not changed much. Everything that's interesting gets provided for free from universities, or google. This leaves me with the chore of sorting through large piles of data. Honestly the ai portion of every project I have been on is tiny compared to the overall infrastructure required while management only cares about what's new and shiny.

Lol I remember early in my career trying to keep up with all of CS before I gave up sometime in the late 90s.

CS in general is moving so fast that it is hard to keep up with what all the sub fields even mean now.

ML has hit a critical juncture and is moving at light speed. I’d be surprised if any mere mortal can keep up with all of ML now.

To answer questions from normal people you don't need to know the latest thing. You need to understand the basics really well, because that's what they don't 🙂 Anyone can read the latest paper but not everyone can put it into context and compare it to existing models.

You don't need to understand everything. You don't be generalist nowadays it's extremely demanding and time consuming.

Eh I just read stuff that’s relevant to my current work, and things which are very prominent in the zeitgeist.

I understand that. I don't even bother understanding what the latest developments in javascript libraries are or what release we're in with Java. I can barely keep up with changes in Python, ML libraries, and things specific to my work. Since (as I mentioned) I do applied stuff, I have to go between handling large data (dask, BigQuery, etc.) and ML frameworks (sagemaker, vertex ai, etc.) as well as the underlying algorithms. It's too much.

I agree with what you said. Still there remains the question of whether to split conferences. What's your take on this?

Aren’t there already a lot of specialized conferences? Like cvpr and iccv are computer vision, I know theres an NLP one I don’t remember the name.

RL is the only one that I can’t think of a famous specialized conference for but there probably are ones that just aren’t as famous.

"Computer Vision" on its own is already a fucking massive field though, covering generative modeling, scene understanding, edge hardware, and everything in between.

It always has been this way. In general, research is not about breadth, its about depth. A generalist will get you nowhere in the field.

if you work in applied ml, let the project scope or expected near future project scopes dictate the locus of your learning. learn the unchanging principles (how to diagnose errors, and what to do about that) and then ride the wave instead of trying to drink it. if this is too vague please ask me more
Edit: i think the scientists (researchers) are more in trouble than the engineers (which i what i call an applied ML practitioner)

I made a list of my "obsessions" which is where I will focus. I try to stay inside that list though I reevaluate it quarterly. I also use a knowledge management system (Obsidian) where I store summaries of what I read so I know I can find something again if I need it. These two habits have helped me.

I literally don’t care anymore. In school I felt like I needed to keep up on everything but after working for 3 years now I just don’t try. I do research related to my work, and live my life outside of work. I have literally zero interest in ML beyond work and even during work I have more pressing concerns.

NLP has several large conferences that people publish impressive research at.

ACL, EMNLP, COLING, LREC (for datasets!), and the various geolocated variants of each are all prestigious venues.

The trick I think is to stay current. If you don't get around to reading an important paper: it happens. You can't read everything. It's more important that you at least have a pulse on what is possible and what other people are doing. Look forward, not back.

Who in the world is expecting you to stay on top of every part of the field? That's not how any technical discipline works. It's not even how CS works.

It’s honestly not that difficult to keep up with the significant improvements in deep learning from a fundamental research perspective. AlphaTensor, amongst many others, is a cool application of reinforcement learning. It’s implications are huge by any standards but it’s not the discovery of a new model. Although diffusion models are also not new outside of ml, they recently started to gain attention as a new model type to train. For me, this is what’s interesting: new types of models, new ways to use SGD in a DL setup. CNNs, RNNs, Att,… are fundamental computation models. Extensions, improvements and upscaling is what I see in 99.99% of papers. GAN is a new model, VAE is a new model, actor critic is a new mode, etc. when it comes to fundamental dl research there are not many of the big ones imho.

Too much in AI for one person to know it all.

That's sounds... Boring

That’s how I feel about 99% of the research field. It’s all the same bullshit over and over.

ML umbrella now is too big, many fields were not labelled as ML some decades ago but now are under ML umbrella. My research areas are in game theory (economics), mathematical analysis, information theory, non-parametric statistic, ergodic theory, stochastic and non stochastic, etc. Now all of them are under the ML umbrella, in particular the reinforcement ML. Recently I found that my fields have some new names/ nomenclatures but similar concept: explainable AI, continual ML, online ML, lifelong ML, etc. I actually don't care the ML/AI buzzwords and the silly names they are trying to invent (as some analog terms used by different fields). I consider it is an expansion rather than advancement of ML research and it is good for multidisciplinary research, but I don't like the way some ML groups are trying to take all the light of the stage.

Generative image models probably need to fork off sometime soon, especially text-guided versions. It's a pet peeve of mine that we're calling it "vision". Vision, at least to me, implies seeing/making sense of what is actually there.

I kinda get where OP is coming from, though. With all the pop-sci ML stuff and big press releases for popular consumption hitting really shortly after actual publication, there's always a risk that some manager will be like "hey I just read about stable diffusion on Twitter, can we use it to do this?" and then you're a deer in headlights bc you weren't at the press conference where they introduced it and you have no idea what the manager is even talking about.

Anyu DX d,zzeww s

"This is a fairly new model and I do not know the details. If you seriously consider this, I can read up on the most recent work and then we have a meeting next week and discuss whether and how it could help us".

The awesome thing about solid basics is that you can do exactly this.

While text-guided image generation is flavour of the month, i don't think that it has broad enough impact to generate a consistent large enough amount of papers to be able to sustain its own conference.

Yepp... Me. I just don't have the time and the necessary problems to solve in order to keep up to date. And for most of my daily challenges a good old linear regression or random forest is all that's needed. And if I want to get fancy, I use xgboost.

Stuff like diffusors, transformers... fascinates me. Do I understand the principle ideas behind them? Kind of, I think... Would I be able to replicate the results? Hell no! Let alone use the principles to solve my own problems (that I don't even have)...

I miss doing AI science. But at some point you've got to let go and let fresher minds do the pushing.

I am/was a computational neuroscientist. There's way more papers published in neuroscience than any one person has a hope of reading.

The best way for me to keep up was to use an rss reader. The big publishers and publication portals (such as arxiv) all have rss feeds, usually per keyword, and you follow the ones that might interest you.

Then you use the filtering available in your feed reader to get rid of any item you obviously don't care about. I was typically left with ~100 papers every morning, with about 4-5 that I'd actually download.

ML is too broad to know everything. Pick a specialization.

I focus on stuff that'll help me in my day job (finance), so there are many areas of ML that are irrelevant.

When I'm curious about a problem, I google the idea and add machine learning to the search and wammo!

I am on a niche with NLP, transformers, GPT, BERT, T5, but I totally ignored diffusion models(yes, they are cool, but very hard to apply only to text) I also ignore mostly all things related to CV. I ve also worked with GNN, they intersect with transformers, and I want to start study RL

Meanwhile in most typical business settings summary stats/t tests, linear, logistic, and random forest/xgboost is all thats used

Same with... every field of everything... you don't know to know everything. It's impossible. Just work on what you want to.

Yes I feel the same exact way

It's not necessarily FOMO. look at job postings these days. They want the kitchen sink, the counter it was built into, the floor plan, the carpenter who did the rough framing of the whole damn house, how did you acquire the property to build the house, where did you get the money, prove it, and on.

Someone newish to the field thinks all these things are important because job posters just throw every damn AI/ML skill they've ever heard of or googled into the JD.

That needs to stop and the person describing the T method is right. Business and hiring managers have it bass ackwards.

Nowadays there are many niche conferences in different subareas of ML. However, I think it will take some time for those niche conferences to be "mainstream" in the specific subarea, because prestige matters (especially in job application) and those niche conferences are just not as prestigious as the top ML conferences, at least for now. See here for a discussion: https://www.reddit.com/r/MachineLearning/comments/vjqdom/d_niche_ml_venues_vs_top_ml_conferences/?utm_source=share&utm_medium=web2x&context=3

I suspect automated image generation is going to become an industry, if not a separate field of study, within the next three years, so we'll start seeing hybrid programmer/artist conferences springing up, with people leaning more towards demonstrations rather than papers.

Exactly. Being an expert in every single field of your profession is never a requirement. The need to be one is more of a sign of fomo or maybe even some underlying psychological issue.

But this bit:

>"This is a fairly new model and I do not know the details"

is hard! I understand having anxiety about being The ML Guy and not being able to immediately answer questions.

yes. I think at this point it is important to realize that in the exact moment you got hired by a company, your role changed.

You were the guy with a PhD straight from university who did top-notch research. Now, you are the guy hired to make this project work.

If your job description does not include "active research" or "follow the most recent advances in ML research" then it is not your job to know what is up - especially if it is an advancement in a subfield of ML your project is not actively interested in.

brumm brumm, beep bazing