John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

There is this idea that the goal of RLHF is to make ChatGPT woke or as you put it to lobotomize it. I suspect that this is a conspiracy theory. There's a very good talk by John Schulman, chief architect of ChatGPT [0], where he explains that if you don't include a RL component in your training, you're essentially doing imitation learning. It's well known that imitation learning fails miserably when presented with conditions that are not in your training set, i.e., answering questions that don't exist on the Internet already. So the goal of RLHF is actually to reduce hallucination. [0] ▶ this video

in “Don Knuth Plays with ChatGPT” · View on HN

Sure, here you go: ▶ this video Summarizing: 1. Obviously you can't completely "solve" truthfulness because people disagree on what is and is not true. But you can go a long way. 2. The models do know what they don't know. Their level of uncertainty is not only expressed in the final token logprobs but also seems to be reified somehow, such that they can express their own level of certainty in words. 3. Many of the problems are introduced by subtle issues introduced during training which effectively teach the AI to guess. One example is by training on QA data sets where the answer is never "I don't know". RLHF introduces its own problems because the human trainers don't know what the model knows, so may reward it for getting a correct answer by guessing. 4. Many failures are caused by lack of access to information. Giving the LLM tools to let it search the web and access other databases can help a lot, and it's relatively easy to do. 5. In some cases you want it to guess. Like, it's much more useful when coding for it to spit out a screenful of code that has one or two minor errors, than to just refuse to try at all because the result might not be perfect.

in “It's not only you, there's an explosion of ChatGPT on HN” · View on HN

How is that "as good as an intern?" Interns, for the most part, do not hallucinate and confidently lie on topics they know nothing about. Good interns tend to ask a lot of questions and quickly learn the cultural cues around the office, allowing them to fit in and become productive. LLMs are incapable of any of this. John Schulman from OpenAI recently gave a talk about the hallucination and uncertainty issues [1]. He claimed that models already have information about their uncertainty but that it remains an open problem as to how to express that uncertainty in natural language. One of the big issues with trying to prevent the model from hallucinating is that you can err too much on the side of caution, causing the model to lie about things it actually does know the correct answer to. [1] ▶ this video

in “AI Safety and the Age of Dislightenment” · View on HN

This is exactly the reason OpenAI isn't afraid of the open-source community, like many kneejerk opponents of regulatory capture assume (they are probably still afraid of Google). Also why they still do the expensive and cumbersome RLHF training, instead of those deceptively cheap and fast finetunes. They understand their own tech and why there isn't free lunch. Recently, John Schulman explained the issue with behavior cloning and it's a very typical ML problem.[1] Basically: what are we training the model to do? The model updates after finetuning in a holistic manner, based on the sum total of its content and capability. Suppose GPT-4 can correctly answer to many requests because it knows correct answers, in the sense that it has something isomorphic to an internal knowledge graph and tools for querying it, and that graph contains sufficient data for its tools to derive an answer at inference. RLHF reinforces this behavior by constraining the distribution of outputs (essentially, steering the model away from applying inappropriate tools for respective inputs, e.g. employing fantasy-narrative or bad-yahoo-answers cognitive routines when asked something that looks like a straightforward factual question). Now suppose you teach LLaMA-13B to imitate those responses by SFTing it on a dump of successful GPT-4 conversations. But LLaMA doesn't have internals that would have enabled it to find the same answers; so on the object level it shallowly memorizes specific items of the post-t

in “The False Promise of Imitating Proprietary LLMs” · View on HN

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer. The author says he’s summarizing a talk by John Schulman of OpenAI [2] but I haven’t personally watched the video. In any case, this is an interesting insight. Say we set up a supervised learning scenario where we ask the model to use its internal knowledge to answer a question and compare its answer to one written by a human. If the two answers essentially say the same thing, but in different words, in the supervised learning case the model is penalized. In the RL case, it’s rewarded. That’s the difference. 1. [link] 2. ▶ this video

in “Training open-source LLMs on ChatGPT output is a really bad idea.” · View on HN