When Bad Data Leads to Good Models
arxiv.org
In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of “quality” from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model’s output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

  • Mr_Dr_Oink@lemmy.world
    English
    14·
    2 days ago

    So is it saying essentially that in order to not output garbage, it needs to know first what garbage is?

    Is it just me that things this seems like a no-brainer?

    It almosr draws parallels to many societal issues. Knowledge is power.

    People tend towards intolerance and hatred when they dont understand the thing they are angry at. The more they know the better they behave.

    • halowpeano@lemmy.worldEnglish
      8·
      2 days ago

      No it’s more of a technical discussion. Many people might believe that in order to avoid toxicity, you just train a model on “good” non-toxic data and then apply toxicity removal techniques to address emergent toxicity that the model might spit out. This paper is saying they found it more effective to train the model on a small percentage of “bad” toxic data on purpose, then apply those same toxicity removal techniques. For some reason, that actually generated less total toxicity. It’s an interesting result. A wild guess on my part, but I’m thinking training the model with toxic content “sharpened” the toxicity when it was generated, making it easier for those removal tools to identify it.