Yeah I read through the whole thing and it's not surprising. Train-test contamination has been a problem for a while now.

Thanks!

It'd be nice to see the qualifications of the authors.

That's true for all of the models. I don't really need anything cool though, all I need is a solid model that can perform simple tasks like text classification or NER well.

Thanks for the detailed answer! My use case is that the company I work at currently uses image-based models for e-commerce purposes, but we want to use text-based models as well. The image-based model(s) are already taking up around 30-50M parameters so I didn't want to just bring in a 100M+ parameter model. Even 15M seems quite big.

I guess at least 100M+ parameters? I like to think of the BERT-base model as being the "starting point" of LLMs.

I hear my colleagues complain about the same thing. And then go back to doing AutoModel.from_pretrained(sdfsdf).

> ...just the hyperparameter was the optimizer design itself.

Probably one of the best things I've read today lol. Reminds me of when old colleagues of mine would have lists of different PyTorch optimizers and just loop through them.

Feedly. A great tool that curates news and other blogs/articles/social media via RSS.

Are there any Slack channels or Discord Servers for ML practitioners to talk about stuff?

Also make sure to change your random seed for each run and calculate the mean and variance for each runs' performance on the test set. As a principal, you should always set aside a test set that you never touch other than for performance purposes.

Frankly I feel the same way about diffusion models. It's nothing more than "whoa cool!" for me. After doing research in NLP in graduate school and doing NLP in the real world, I'm increasingly feeling a huge disconnect between academic research and the real world.

I'm also in NLP and I usually just use a pretrained model from HuggingFace as a backbone and build on top of that. It's not usually anything complicated though, maybe an MLP.

I think the word "misinformation" is a little dangerous to be using here. People can criticize the authors of not providing any actual novelty (which is actually super common) but pointing fingers and saying they're "spreading misinformation" is a little much.

Nah screw that. Everything should be publicly discussed.

There's not really any merit. That's why the reviewing process is such a mess. There's virtually no way to incentivize or penalize people for being shitty reviewers.

Thanks for the comment. I should probably have been a little more descriptive, but we're not really trying to optimize the metrics themselves. This is just one part that's going to be used for a bigger pipeline.

Thanks! That sounds like something that would be easily computable and reasonable.

Ah thanks for the comment but I think we won't have to account for sentiment information. I should have probably said "publicity" or something rather than "popularity" (it makes sense in my native language). Negative sentiment would also mean that something is trending, and that's what we're trying to measure rather than how positively people would view something.

Try looking up DensePhrases, it was made by a colleague of mine and may be what you're looking for. They also have an online demo you can try.

I'm not sure what you mean by "retrieval-based language model" though. I don't think there's any language model that's made solely for the purpose of retrieval.

Would open-domain QA models be relevant for this topic?

Why do we pronounce "ICLR" as "eye-clear" but not "ICML" as "eye-camel?"

Lol are you by any chance in Asia? As an Asian who was raised in the US but is living and working in his "mother country," everything about this post screams "Asia" to me.

You're not going to find theory-dense papers at major ML conferences. Most of the reviewers don't bother going through them and people usually find theory boring compared to "super duper cool" architectures that lead to 0.1% increase in performance. Like the top comment said, COLT is a good place to start.

I normally wouldn't entertain that, but upon checking your profile it seems like you're interested in financial ML. As someone who's briefly done research in financial ML I can tell you that 99% of the papers you read and ML code you run are pretty much BS. You're better off using ML very sparingly and using if-else statements.