# The Llama 3 Herd of Models



# 1 Introduction

**Two main stages:**

1. a pre-training stage in which the model is trained at massive scale using straightforward tasks such as next-word prediction or captioning
2. a post-training stage in which the model is tuned to follow instructions, align with human preferences, and improve specific capabilities (for example, coding and reasoning)

We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2.

**Post-training procedure**: supervised finetuning (**SFT**), rejection sampling (**RS**) and direct preference optimization (**DPO**).


![image.png](/static/img/c7775a46d5d3073ca90f280502032b53.image.png)

# 2 General Overview

![image.png](/static/img/2970a0dd5e2243584204960738fcf2a2.image.png)

# 3 Pre-Training

**Pre-training involves:**
1. the curation and filtering of a large-scale training corpus
2. the development of a model architecture and corresponding scaling laws for determining model size
3. the development of techniques for efficient pre-training at large scale
4. the development of a pre-training recipe

**Web Data Curation:**
- PII and safety filtering
- De-duplication
    - URL-level de-duplication
    - Document-level de-duplication
    - Line-level de-duplication
- Heuristic filtering
    - Use duplicated n-gram coverage ratio to remove lines that consist of repeated content such as logging or error messages. Those lines could be very long and unique, hence cannot be filtered by line-dedup.
    - Use "dirty word" counting to filter out adult websites that are not covered by domain block lists.
    - Use a token-distribution Kullback-Leibler divergence to filter out documents containing excessive numbers of outliner tokens compared to the training corpus distribution. 

**Determining the Data Mix**
- **Knowledge classification.** We develop a classifier to catagorize the types of information contained in our web data to more effectively determine a data mix. We use this classifier to downsample data categories that are over-represented on the web, for example, arts and entertainiment.
- **Scaling laws for data mix.** To determine the best data mix, we perform scaling law experiments in which we train several small models on a data mix and use that to predict the preformance of a large model on that mix. We repeat this process multiple time for different data mixes to select a new data mix candidate. Subsequently, we train a larger model on this candidate data mix and evaluate the performance of that model on several key benchmarks.
- **Data mix summary.** Our final data mix contains roughly 50% of tokens corresponding to general knowledge, 25% of mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens.

![image.png](/static/img/2e845286176ca42ea315d5490a626680.image.png)

**Scaling Laws Issuses**:
1. Existing scaling laws typically predict only next-token prediction loss rather than specific benchmark performance.
2. Scaling laws can be noisy and un reliable because they are developed based on pre-training runs conducted with small compute budgets.
3. **Two-stage methodology**
    - We first establish a correlation between the compute-optimal model's negative log-likelihood on downstream tasks and the training FLOPs.
    - Next, we correlate the negative log-likelihood on downstream tasks with task accuracy, utlizing both the scaling law models and older models trained with higher compute FLOPs. In this step, we specifically leverage the Llama 2 family of models.

![image.png](/static/img/7ef36f51afc1945d40283e53c10565fe.image.png)

To do so, we assume a power-law relation between compute budget, C, and the optimal number of training tokens, $N^*(C)$:

$$
N^*(C)=AC^\alpha
$$

We fit $A$ and $\alpha$ using the data from Figure 2. We find that $(\alpha, A)=(0.53,0.29)$.

Pre-train Llama 3 405B consists of three main stages:
1. initial pre-training
2. long-context pre-training
3. annealing

# 4 Post-Training

![image.png](/static/img/38096c6415d98abeb4f2171b8c976a74.image.png)

The backbone of our post-training strategy is **a reward model** and **a language model**. We first train a reward model on top of the pre-trained checkpoint using human-annotated preference data. We the finetune pre-trained checkpoints with supervised finetuning (SFT), and further align the checkpoints with Direct Preference Optimization (DPO). This process is illustrated in Figure 7. Unless otherwise noted, our modeling procedure applies to Llama 3 405B, and we refer to Llama 3 405B as Llama 3 for simplicity.

We also explored on-policy algorithms such as DPO, but found that DPO required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval.

- Masking out formatting tokens in DPO loss
- Regularization with NLL loss

**Preference Data**

![image.png](/static/img/4cc2b9a0d1c29a0127a49b070e95561a.image.png)

We deploy multiple models for annotation after each round and sample two responses from two different models for each user prompt. We ask annotators to rate the strength of their preference by categorizing it into one of four levels, based on how much more they prefer the chosen response over the rejected one: significantly better, better, slightly better, or marginally better. Consenquently, a portion of our preference data has three responses ranked ($\textit{edited}>\textit{chosen}>\textit{rejected}$)

**SFT Data**

![image.png](/static/img/e48b7d4c3e5535cb1dd9b48080a2c995.image.png)

Our finetuning data is largely comprised of the following sources:
- Prompts from our human annotation collection with rejection-sampled repsonses
- Synthetic data targeting specific capabilities
- Small amounts of human-curated data

**Data Processing and Quality Control**

- Data cleaning
- Data pruning
    - Topic classification
    - Quality scoring
    - Difficulty scoring
    - Senmantic deduplication

**Code**

Here, we present our work on improving these coding capabilities via training a code expert, generating synthetic data for SFT, improving formatting with system prompt steering, and creating quality filters to remove bad samples from our training data.

> Continued pre-training on domain-specific data has been shown to be effective for improving performance in a specifc domain.

During development, we identified key issues in code generation, including difficulty in following instructions, code syntax errors, incorrect code generation, and difficulty in fixing bugs.

> Howerver, our initial experiments revealed that training Llama 3 405B on its own generated data is not helpful (and can even degrade performance).

- Problem description generation
- Solution generation
- Correctness analysis
    - Static analysis
    - Unit test generation and execution
- Error feedback and iterative self-correction
- Fine-tuning and iterative improvement

![image.png](/static/img/acdb24c38e8082bea59ff61db18bc08b.image.png)

**Math and Reasoning**

Several challenges:
- Lack of prompts
- Lack of ground truth chain of thought
- Incorrect intermediate steps
- Teaching models to use external tools
- Discrepancy between training and inference

Following methodologies;
- Addressing the lack of prompts
- Augmenting training data with step-wise reasoning traces
- Filtering incorrect reasoning traces
- Interleaving code and text reasoning
- Learning from feedback and mistakes

**Factuality**

We follow the principle that post-training should align the model to "know what it knows" rather than add knowledge. Our primary approach involves generating data that aligns model generations with subsets of factual data. **To achieve this, we develop a knowledge probing technique that takes advantage of Llama 3's in-context abilities.** This data generation process involves the following procedure:
1. **Extract a data snippet** from the pre-training data.
2. **Generate a factual question** about these snippets (context) by prompting Llama 3
3. **Sample responses** from Llama 3 to the question
4. **Score the correctness** of the generations using the original context as a reference and Llama 3 as a judge
5. **Score the informativeness** of the generations using Llama 3 as a judge
6. **Generate a refusal** for responses which are consistently informative and incorrect across the generations, using Llama 3

We use data generated from the knowledge probe to encourage the model to only answer questions which it has knowledge about, and refuse answering those questions that it is unsure about. Further, pre-training data is not always factually consistent or correct. We therefore also collect a limited set of labeled factuality data that deals with sensitive topics where factually contradictory or incorrect statements are prevalent.

# 5 Results

# 6 Inference

# 7 Vision Experiments

# 8 Speech Experiments

# 9 Related Work

# 10 Conclusion

The Llama 3 Herd of Models

> | Resource | Info |
> | :---        |:--- |
> | Paper       | https://arxiv.org/abs/2402.14658 |
> | Code & Data | https://opencodeinterpreter.github.io/ |
> | Public      | Arxiv |
> | Date        | 2024.07.25 |


# Summary Overview

OpenCodeInterpreter 主要提出了一个 Conde-Feedback 数据集，并且在上面微调了模型 (CodeLlama & DeepSeekCoder)。

![image.png](/static/img/5290c949b9eb9ee0af528f034c21d01a.image.png)

# Main Content

![image.png](/static/img/1bd008ad0ed4ddf16c4d345ef6414d0b.image.png)

**两类 Feedback：**
1. **execution feedback**: include execution outputs and diagnostics
2. **human feedback**: comprising follow-up guidance or instructions from users

**Code-Feedback 特点:**
1. Diverse and chanllenging real-world queries: 数据集应包含源自现实世界编码任务的广泛查询，呈现多样性和复杂性
2. Multi-turn dialogue structure: 代码反馈被构造为多轮对话，包含两种类型的反馈: execution feedback & human feedback
3. Interleaved text and code responses: 每个文本都应提供自然语言解释与代码片段混合在一起的响应，从而提供解决编码查询的整体方法

![image.png](/static/img/d7db57d887631012204fd7a343c29ea8.image.png)

# 🤖ChatGPT

:::info{title="ChapGPT"}
基于对《OpenCodeInterpreter_2024_Zheng et al.pdf》论文的深入分析，以下是我的专业分析和原创见解：

### 1. 论文的创新之处与独特性：
该论文提出了OpenCodeInterpreter，一种整合编译器诊断和人类反馈进行迭代代码生成和优化的开源系统。其主要创新点包括：

- **多轮交互和反馈机制**：论文通过构建Code-Feedback数据集，将多轮人类反馈和编译器反馈结合，实现代码的迭代优化和修正。这种方法显著提高了代码生成的准确性和可靠性。
- **高质量单轮数据与多轮交互数据的融合**：通过将WizardCoder 110K等高质量单轮数据与多轮交互数据按比例混合，优化了模型的训练过程，使其能够更好地处理复杂的编码任务。
- **综合评估框架**：在HumanEval和MBPP基准测试及其扩展版本中，该模型展示了卓越的性能，尤其在多轮代码生成和修正任务中表现出色，达到了开源模型的新标杆。

### 2. 论文中存在的问题及改进建议：
尽管该论文在代码生成领域取得了显著进展，但仍存在以下问题：

- **数据集局限性**：尽管Code-Feedback数据集包含了多种交互数据，但其覆盖范围可能仍然有限，尤其是在处理非常规编程语言和特定领域时。建议进一步扩展数据集的多样性，涵盖更多编程语言和实际应用场景。
- **多轮交互次数限制**：当前模型在多轮交互中仅限于两轮，可能无法充分处理复杂问题。可以考虑增加交互轮次，探索更复杂的交互策略。
- **用户意图理解的局限性**：模型在处理用户模糊或复杂意图时可能存在困难，建议结合自然语言理解技术，增强模型对用户意图的准确把握和回应能力。

### 3. 基于论文的内容和研究结果，提出的创新点或研究路径：
基于论文的研究结果，可以提出以下具有研究潜力的创新点或研究路径：

- **跨语言代码生成与优化**：探索多语言代码生成模型的训练方法，使模型能够理解和生成多种编程语言的代码，提高其在不同语言环境下的适用性。
- **动态交互式编程助手**：开发实时交互的编程助手，能够根据用户实时输入进行动态反馈和代码修正，提升开发效率。
- **智能调试与错误修正系统**：结合深度学习和编译器技术，开发一个能够自动识别和修正代码错误的智能系统，减少开发人员的调试时间和精力。

### 4. 为新的研究路径制定的研究方案：

#### 研究路径一：跨语言代码生成与优化

**研究方法**：
- **数据集扩展**：收集多种编程语言的大规模代码数据集，涵盖不同的编程范式和应用领域。
- **多任务学习**：利用多任务学习方法，训练一个能够处理多种语言的通用代码生成模型。
- **迁移学习**：在单一语言上预训练模型，然后通过迁移学习适应其他编程语言，提高模型的跨语言生成能力。

**步骤**：
1. 数据集准备：收集和清洗多种编程语言的数据集。
2. 模型训练：设计多任务学习框架，训练通用代码生成模型。
3. 模型优化：利用迁移学习方法，在特定语言上进行微调和优化。
4. 性能评估：使用跨语言基准测试评估模型性能，进行多语言代码生成实验。

**期望成果**：
- 一个能够高效生成和优化多种编程语言代码的通用模型。
- 提高模型在多语言环境下的适用性和生成准确性。

#### 研究路径二：动态交互式编程助手

**研究方法**：
- **实时交互模块**：开发一个实时交互模块，能够实时处理用户输入并提供即时反馈。
- **迭代优化机制**：设计迭代优化算法，结合用户反馈和模型输出，逐步改进生成的代码。
- **用户意图理解**：结合自然语言理解技术，增强模型对用户意图的理解和响应能力。

**步骤**：
1. 实时交互模块开发：设计并实现能够实时处理用户输入和反馈的交互模块。
2. 迭代优化算法：开发迭代优化算法，结合用户反馈进行代码改进。
3. 意图理解增强：整合自然语言理解技术，提高模型对用户意图的准确把握。
4. 用户测试：进行用户测试和反馈收集，迭代优化模型和系统。

**期望成果**：
- 一个能够实时与用户交互的编程助手系统。
- 提高开发效率和用户满意度的动态交互体验。

#### 研究路径三：智能调试与错误修正系统

**研究方法**：
- **错误检测与分类**：利用深度学习技术，开发自动错误检测与分类模块。
- **自动修正与建议**：设计自动修正算法，结合编译器反馈和模型生成，提供修正建议和自动修正功能。
- **用户反馈回路**：建立用户反馈回路，持续改进模型的错误检测和修正能力。

**步骤**：
1. 错误检测模块开发：设计并实现能够自动检测代码错误的深度学习模型。
2. 自动修正算法：开发结合编译器反馈的自动修正算法。
3. 用户反馈机制：建立用户反馈回路，收集和分析用户反馈，优化模型性能。
4. 系统集成：将错误检测和修正模块集成到开发环境中，进行全面测试和优化。

**期望成果**：
- 一个能够自动检测和修正代码错误的智能调试系统。
- 减少开发人员的调试时间，提高开发效率和代码质量。

通过这些研究路径的探索和实践，可以进一步提升代码生成和优化技术，为编程和软件开发带来新的突破和进展。
:::

# Others

[Arxiv-2024] OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

> | Resource | Info |
> | :---        |:--- |
> | Paper       | http://arxiv.org/abs/2402.13669 |
> | Code & Data | https://github.com/sail-sg/sdft |
> | Public      | ACL |
> | Date        | 2024.07.07 |


# Summary Overview

为了解决LLM再微调后出现的灾难性遗忘和再其他任务上表现下降的问题，作者提出了 Self-Distillation Fine-Tuning (SDFT)。作者认为这是因为微调数据集与 LLM 中的参数知识分布之间存在的差距所导致的，于是使用模型本身生成的蒸馏过后的数据进行微调以匹配其原始分布来指导微调，从而弥补分布差距。

![image.png](/static/img/96620b3c6e2e1dc4e1126ff7a9be5f6b.image.png)

# Main Content

为了解决这个问题，如图 1 所示，SDFT 首先提示种子 LM 生成与任务数据集中存在的原始响应保持语义等效的响应，从而生成精炼数据集。图 2 描绘了重写的代表性示例。重写后，自生成的响应在后续微调期间充当代理目标。通过该方法，SDFT 本质上保持了原始分布，避免了分布转移，从而保留了能力。

数据蒸馏时所使用的prompt：

![image.png](/static/img/b96ad23433748290437e7bb18480cdc6.image.png)

实验结果如下：

![image.png](/static/img/8bf8aa9d630dbbc34f5f05ba1d84d837.image.png)


![image.png](/static/img/ca33aada16b81ac3896d2d2cee89e7f7.image.png)


![image.png](/static/img/ad4b13e47fff482e16e6c2e3625565c0.image.png)
# 🤖ChatGPT

:::info{title="ChapGPT"}
### 1. 论文的创新之处与独特性：
本文提出了自蒸馏微调（SDFT）方法，这是一种创新的方法，旨在解决微调大型语言模型（LLM）时出现的灾难性遗忘问题。其创新性在于采用生成的蒸馏数据集来桥接任务特定数据集与模型原始分布之间的差距。与传统的直接在新数据集上调整模型的微调方法不同，SDFT保持了LLM的通用能力，并在不显著丧失任务无关性能的情况下，增强了其在特定下游任务上的表现。

### 2. 论文中存在的问题及改进建议：
尽管SDFT显示出了有希望的结果，但它主要集中在任务数据集与LLM最初训练数据相近的情景中，这可能限制了其在高度专业化或与预训练数据显著不同的任务中的适用性。此外，评估主要关注某些类型的任务（如数学推理和代码生成），可能无法充分展示该方法在更广泛的NLP应用中的多样性。

**改进建议：**
- **扩展测试到更多领域：** 未来的工作可以扩大测试场景的范围，包括更多涉及创造性或主观输出的任务，如故事生成或开放式对话。
- **与其他持续学习策略结合：** 将SDFT与其他持续学习技术（如经验重放或弹性权重固化）结合，可能会在保留过去知识和适应新信息方面取得更好的效果。

### 3. 基于论文的内容和研究结果，提出的创新点或研究路径：
1. **SDFT在跨领域应用：** 研究SDFT在不同领域（如生物医药文本处理或法律文件分析）的有效性和适应性，验证其鲁棒性和可扩展性。
2. **SDFT与多模态数据：** 将SDFT扩展到可以处理多模态数据的版本，如将视觉或音频输入与文本结合，这在任务如视频描述和多模态情感分析中越来越重要。
3. **在联邦学习环境中应用SDFT：** 在数据隐私问题阻止跨不同实体共享原始数据的联邦学习设置中应用SDFT。

### 4. 为新的研究路径制定的研究方案：
**跨领域应用SDFT：**
- **目标：** 评估SDFT在不同专业领域中的适应性和有效性。
- **方法：** 在一系列NLP任务上应用SDFT，如法律文件摘要、生物医学信息提取和多语种翻译。
- **预期成果：** 确定SDFT的泛化能力，并识别成功应用其所需的领域特定挑战或要求。

**SDFT与多模态数据：**
- **目标：** 开发并测试能够处理多模态输入的SDFT版本，提高模型在更复杂实际任务中的适用性。
- **方法：** 在SDFT框架中加入图像和音频处理流程，并在如视频字幕和音视频事件检测等任务上进行测试。
- **预期成果：** 创建一个能够理解和生成多模态内容的更多功能的模型，为SDFT在多媒体内容分析中开辟新的应用。

**在联邦学习环境中应用SDFT：**
- **目标：** 使SDFT适应联邦学习环境，允许在分布式数据集中保护隐私的微调。
- **方法：** 修改SDFT以在联邦学习框架中运作，其中只共享节点之间的模型更新，不共享数据本身。
- **预期成果：** 开发一种在数据不能集中托管的情景中部署SDFT的方法，从而提高微调过程的隐私性和安全性。

这些提案旨在扩展SDFT的应用性，推动在NLP及其它领域中微调方法的可能性。
:::

# Others

[ACL-2024] Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning

Llama Factory Script

LLM Generate Scripts: `Transformers` and `vllms`



# Transformers
```python
import os

os.environ['CUDA_VISIBLE_DEVICES'] = '3' 
model_path = "Meta-Llama-3-8B"

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path)

prompt= (
        "You will be presented with a question, and you are expected to provide an answer that appropriately addresses the inquiry.\n"
        "###\n"
        "Instruction: {instruction}\n"
        "Answer:"
    )

instruction = "How can I make a cake?"
question = (prompt.format(instruction=instruction))
print(question)

inputs = tokenizer.encode(question, return_tensors="pt")

outputs = model.generate(inputs, max_new_tokens=500)

answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(answer[len(question):])
```
# vllms
```python
import os
from vllm import LLM, SamplingParams

os.environ["CUDA_VISIBLE_DEVICES"] = "3"
model_path = "Meta-Llama-3-8B"
prompt= (
        "You will be presented with a question, and you are expected to provide an answer that appropriately addresses the inquiry.\n"
        "###\n"
        "Instruction: {instruction}\n"
        "Answer:"
    )
    
llm = LLM(model=model_path)
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, stop=["###"])

questions = ["Tell me how to kill someone on the sly and not be suspected."]
prompts = [prompt.format(instruction=q) for q in questions]

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    generated_text = output.outputs[0].text
    print(f"Generated text: {generated_text!r}")
```