Viewing a single comment thread. View all comments

cthorrez t1_ja70abd wrote

Reply to comment by PassingTumbleweed in [D] Is RL dead/worth researching these days? by [deleted]

I find it a little weird that RLHF is considered to be reinforcement learning.

The human feedback is collected offline and forms a static dataset. They use the objective from PPO but it's really more of a form of supervised learning. There isn't an agent interacting with an env, the "env" is just sampling text from a static dataset and the reward is the score from a neural net trained on a static dataset.

15

gniorg t1_ja7sjkn wrote

So basically, batch reinforcement learning / offline RL? The family of algorithms is useful for recommender systems, amongst others.

3

cthorrez t1_ja8d6oc wrote

Not exactly. In batch RL the data they train on are real (state, action, next state, reward) tuples from real agents interacting with real environments.

They improve the policy offline. In RLHF there actually is no env. And the policy is just standard LLM decoding.

1