Is this a "give me Amazon, but without the billions that went into it" post?

Are you wanting it to recommend products that go with what the person is purchasing?

Are you wanting a home-page like place that recommends products without context of their current cart?

How would you go about solving the problem?

I want to do a category wise Recommendation. So suppose you are iPhone page of my ecommerce site . You should get a Recommendation like

People also buyed

iPhone + Anchor charger Or iPhone + ear buds

Something like this. Here the product which is recommended is in same category (ie electronics).

I should not get out of category Recommendation like

IPhone + men's shirt

Amazon pretty much offers their recommendation engine throughAmazon Personalize. Instead, If you want build one from scratch, I strongly recommend you read up the many white papers that have been written on Amazon’s item-to-item recommendation engine. It’s pretty straightforward to implement. Here’s a link to one of them.. What I have seen is that the best performing algorithms are the simple ones that are based on “viewed-also-viewed” or “bought-also-bought”. Unless you are in the research space, stay away from algorithms that claim enhancements through “Deep Learning” or “NLP”. These fancy algorithms seldom lead to any real improvement in recommendation performance.

Google “collaborative filtering”

Approximate page rank GNN

But, Amazon recommendations are awful

We solved with lightgbm, for news to suggest.

if you have the tabular data, (user properties, buyed product, previous X products viewed), sets the buyed product as target and you had your recommender done.

There's a lot of examples doing it with lightgbm or xgboost (gbm is faster)

Collaborative filtering and matrix factorization are your best bets for this amount of data. Don’t try anything neural networks related unless you have millions of rows of training data

... u kidding right?

have you bought anything there?

Amazon will recommend the exact same coffee-maker you bought 1 week before, together with a bunch of irrelevant products

Amazon is a broken bazaar, they have been lucky with the server stuff, and even there I would say it is tic toc time for them

Hmm neural networks can be arbitrarily small. You can model logistic regression with a neural net. You could definitely try a simple neural network with one embedding layer for users and one for products and use a sigmoid activation for the output layer. Then you need to combine these layers in some ways, for instance a simple dot product or concatenation. You can start with no hidden layer and add more if the results are not satisfactory.

This already looks over complicated compared to the simply deployable and explainable alternatives that have been proven over time to be better at this problem for small datasets.

Isn’t the simplest version of what I described essentially matrix factorization?

Theoretically yes, but it sounds like it requires more coding skills since you need to create the network structure by hand instead of relying on solutions that were specifically built for this. Don't get me wrong, there is nothing bad with what you described, but sounds to me OP needs a tool out of the box.

Fair

How do you typically use the "previous X products viewed" in the model? As categoricals/text embeddings, somewhat? Are you doing any within user "ranking"/"group" losses, or just using an overall multi-classification model?

the previous X is the result of a top-k N id's or categories (we used various models) and with the output apply some business rules, like seasson, recently, etc