# Writing Perfect Papers


李葆春教授的分享，受益良多，感谢。本文主要是摘抄李教授提供的 Slides 作为后续查找复习资料。

Video: https://www.bilibili.com/video/BV18v411n7mr/

Slides: https://iqua.ece.toronto.edu/papers/writing-perfect-papers-2021.pdf



Our goal today: improve both the skills and the process.

# 1 Writing perfect papers

A top-down approach: from the big picture to tiny details.

## 1.1 The story

How do I make a great movie? A great movie starts from a powerful and convincing story.

**Three essential elements of a great story:**
Your paper is not really the best place to show your theoretical prowess. It's about advancing the state-of-the-art.
- **The problem**
    - Don't have to be something "**trendy**".
    - But it has to be something you really **enjoy** working on.
    - The "**trend**" may not last forever.
    - Following the trend may lead to more "**incremental**" results —— less exciting and less important.
    - Be a **contrarian** and working against the trend may be good idea.
- **What's new?**
    - In the context of what's already done in the related work, **what's new** in your work?
    - First, briefly introduce the **state-of-the-art** in the literature, solving the same problem you wish to solve.
    - Then write about what is **new** in you work that is different from previous work.
    - If the problem is **different** from previous work —— different assumptions, for example —— provide justifications **why** this is **important** and **necessary**.
- **How is it better?**
    - Now that your ideas are new, they should also lead to **bettter** results than before.
    - Show **how** your solution is **better** than previous work: such as stronger theoretical properties or better experimental results.

## 1.2 The work

Two things that you must do: **Read** and **Write**.

Read:
- Goal: become an expert on the problem.
- But how?
    - Start from one paper.
    - Perhaps a highly cited **seminal** paper.
    - Then do an **expanded-ring** search.
        - Papers that cited this paper.
        - Papers that it cited.
        - Papers authored by the same researchers.
    - Try not to depend only on a search engine.
    - Read **quickly** first.
    - Read more carefully if necessary **later**.

Write:
- Write about what you understood.
- Write every day.
- **idea -> write paper -> do research**
- If I don't have an idea what do I write about?
    - You write **anyway**.
    - Write about related work.
    - Write about what's challenging.
    - Write about why the problem is important.
    - Write about what needs to be fixed.
    - Write a survey paper.
    - Submit your survey paper for publication.
    - Write about **what's new** in the context of related work.
- Creating ideas interacts with **writing** closely.
- **Writing** is the best way to force yourself to **think clearly** and be **focused** and to **crystallize** what you don't quite understand yet.
- **Writing** also opens a dialogue.
- For others to read what you wrote.
- To stimulate discussions with others.
- That's why **writing** is a **slow** and **painful** process.
- Write in a **crystal clear** way.
- Write about **one** problem and **one** (potential) solution.
- Write with a **flow of ideas** that's easy to follow.
- Keep your readers engaged **throughout** the paper.
- Write your paper so that it's as easy to understand as **absolutely possible**.
- Your readers don't have to work hard.
- Write your paper **slowly**, so that your readers can read **quickly**.

## 1.3 The art

Pay attention to the title, abstract, introduction, and flow of ideas.

Tital: **2** lines, **10,000** readers.
Abstract: **10** lines, **1,000** readers.
Introduction: **100** lines, **100** readers.
The rest of the paper: **1,000** lines, **10** readers.

### The title

What's good title?

It attracts a reader to read the abstract.

It reflects the essence of the new idea.

It is as simple as possible
- does not have to be a precise summary of the paper.
- does not need to include all the keywords.

The best title is one that is the easiest to understand **at a glance**.

:::note{title="Example"}

Let's say we have a new idea that uses a new coding technique to reduce the latency when delivering data over the Internet. Which title is the best?

- LLRC: A Low-Latency Rate-Controlled System for Fast Data Delivery over the Internet
- Coded Information Distribution: New Content-Delivery Protocols for the Internet
- **Coding Reducces Latency over the Internet**

:::

### The abstract

The abstraction is a very important tool to attract readers to read the introduction.

Again, it conveys essential information about the paper.

It should be concise, and does not need to be long.

Remember the three essential elements: The problem, What's new? and How is it better?

A typical structure

- One sentence to state the background.
- One sentence to state **what the proble is**.
- Two to four sentences to state the original contribution in the paper —— **what's new?**
- One sentence to state **how the solution is better**, validated using analyses, simulations or experiments.

:::note{title="Abstract Example"}

(**Background**) With cloud storage services, it is commonplace for data to be not only stored in the cloud, but also shared across multiple users. (**Problem**) However, public auditing for such shared data -- while preserving both data and identity privacy -- remains to be an open challenge. (**Contribution: Overview**) In this paper, we propose the first privacy-preserving mechanism that allows public auditing on shared data stored in the cloud. (**Contribution: Highlights**) In particular, we exploit ring signatures to compute the metadata needed to verify the integrity of shared data. With our mechanism, the identity of the signer of each data block is kept private from a third party auditor (TPA), who is still able to publicly verify the integrity of shared data without retrieving the entire file. (**Contribution: Additional features**) In addition, we utilize homomorphic MACs and homomorphic hash function to mitigate the overhead of storing our signatures. (**Validation by experiments**) Our experimental results demonstrate the effectiveness and efficiency of our proposed mechanism when auditing shared data.

:::

### The introduction

If your title and abstract can get a reader to read the introduction, you are half way there.

The remaining job: impress the reader with an exciting and clear introduction.

The introduction is so important that I rewrote the introduction in many of papers.

It is important because we wish our reader to understand our ideas, even if they only read the introduction and noting else.

**Write a good introduction**
- It should be self-contained, so that a reader short on time doesn't need to read the rest of the paper.
- It should be clear **what the problem is**.
- It should be easy to understand **why our solution is new**, and **how it's better** than previous work.
- Don't make it too long.
- Don't spend too much space on the background and related work.
    - There can be a separate "Related Work" section.
- Don't make your work sound more original than it really is.

**Introduciont: a typical structure**
- Frist (opening) paragraph:
    - A **general overview** of the research field -- basic facts needed to "warm up" the reader and to prepare for the problem statement.
    - Not too long -- 2-3 sentences are good enough.
- The second paragraph: **problem statement**
    - A clear description of the challenges to be addressed and the **problems** to be solved.
    - Appeal to the intuition of the reader.
    - Describe at a high level.
    - It is fine (and even a good idea) to include an **intuitive solution** to the problem.

Unnecessary details inhibit the **natural** and **brisk** flow of the main ideas.

Returning to the typical structure of the introduction:
- General overview to warm-up the reader (the opening paragraph)
- State the problem (challenge) and existing solutions (1-2 paragraphs)
- Limitations of existing solutions that motivate this paper (2-3 paragraphs)
- Proposed solution: main idea (1-2 paragraphs)

**Write about what's new**
- Why is the proposed solution different from and better than existing solutions?
- State 1-2 most impressive highlights, not all of them.
- Make the originality of the paper crystal clear and stand out.
- You may use the sentence: "Highlights of our original ocntributions in this paper are as follows. First, ... Second, ... Finally, ..."

The list of original contributions drives the **entire paper** -- the rest of the paper substantiates the claims you have made.

Your readers may think: "Hey, if they can really deliver this, that'll be very exciting! I'd better read the rest of the paper."

It is a good idea to include a **table** to compare important properties of the proposed solution with its "direct competitors" in the existing literature, highlighting your **advantages**.

Use an example:
- It's an even better idea to show an intuitive **example**.
- Your **example** shows how your **main idea** works in a **special case**.
- Continue with **more** examples throughout the paper.
- After each theorem is proved -- or each algorithm shown -- explain their **implications** or **intuition** with an example.
- Your examples need to be really **simple**.
- They helps readers to understand your solution well.
- It'll be a greate idea to include a well-designed **figure** to illustrate your example.
- Includes as many figures as you can throughout the paper.

The **flow of ideas:**
- Just like the storyboard when making a movie, the **flow of ideas** needs to be carefully **designed**.
- Make sure that each paragraph contains one **complete** idea - **split** a paragraph that contains more than one idea, and merge short paragraphs that cover the same idea.
- Where is the section on **related work**?
    - Some prefer to place it after the introduction
        - Rationale: the section can be used to "warm up" the reader.
    - Some others prefer to place it before the conclusion
        - Rationale: After the introduction, the readers don't understand your main ideas yet, there's no point in talking about their differences from related work.
    - It depends on the design of your flow of ideas.
- Where do you write about **experimental results**?
    - Most papers collect all the experimental results and put them **at the end**.
    - But you don't have to.
    - Some **negative** or **preliminary** experimental results may be important for motivating the main idea in the paper.
    - Your flow of ideas can then be: "**initial results - idea to improve - more results to show better performance**"
    - You can even interleave experimental results with descriptions of your idea, if this provides the best flow.
- How do you allocate space across sections?
    - The short answer is: no one knows it better than you, because it depends on the flow of ideas.

**General rules of thumb:**
- Don't write a very long abstract (200 words), introduction (one page), related work (half a page), or concluding remarks (1-2 paragraphs)
- Throughout the paper, make it self-contained, but don't use a lot of space for unnecessary background.
- Keep the motivation short and concise.
## 1.4 The details

**Use a git repository to manage your workflow.**

Get repositories help you to track all histories and collaborate with others.

Place everything - source code, reference - into your git repository.

Use any of the public cloud services such as GitHub.

**Eliminate typographical and grammatical mistakes.**

First, eliminate all spelling mistakes by running your paper through a spell checker.

Use a UNIX command-line tool such as spell to check spelling - not Microsoft Word.

Then fix grammar and usage problems by proofreading.

**Proofreading** also helps you to fix the remaining spelling mistakes that a spell checker cannot catch.

You cannot rely on someone else - certainly not the reviewers - to proofread for you, it's your paper!

**Use transitional words, phrases, and sentences**

Use **transitions** across the boundary of sentences, paragraphs and sections.

**Important** for readers to follow your **flow of ideas**.

If you don't add transitons, readers will need to add them subocnsciously in their mind, anyway. You are asking readers to do the hard work!


:::note{title="Examples of transitions"}
Connecting two halves of a sentence:
- as, since ,or else

Connecting sentences:
- However, In addition, Further, Nevertheless, Fortunately, Unfortunately, Suprisingly,
- To make matters worse, to further exacerbate the problem, The bad new is,
- The implications are two-fold, It turns out that, As as example, To take ... a step further,

Connecting paragraphs and sections:
- It only remains to see ...
- The simple answer to this question is,
- The only challenge that remains now is,
- To address this challenge,
- We first present...
- Next, we evaluate...
- We are now ready to...

:::

**Use correct English**

- Keep punctuations marks inside the closing quotation mark.
- Don't use long sentences
    - Don't use long sentences with more than one comma in the middle of the sentence - and **abuse** "where", "in which", "whose", 'so that", "such that".
- Don't be too colloquial
    - Don't use words that are too informal and colloquial.
    - "a lot of" is more colloquial than "a large number of".
    - Instead of "big" use "substantial" or "large".
- Don't be too formal either
    - Overly formal words and phrases sometimes feel awkward.
    - "we endeavour to ascertain that..." - "We show that..."
    - "It can be ascribed to..." - "It is due to..."
    - "The overwhelming quantity of..." - "The exceedingly large number of..."
- Don't use words that are too emotional
- Countable vs. uncountable nouns
    - If a word is countable, when using its plural form, remember to use "a few", "a number of", "fewer". Instead of "less bits," use "fewer bits".
    - If not, do not invent its plural form (such as "performances," "advices," "equipments," "informations"),
    - and use "less" or "lower" rather than "fewer"
- Agreement of the verb with the subject.
- Watch out on common words and phrases.
    - It almost takes a lifetime to master the usage of common words, such as "with", "of", "for", "at", "against", "in", "on"
    - Or to master the large number of phrases in English "in other words" or "in another word"?
- Articles as determiners
    - The articles "a" / "an" (the indefinite article) and "the" (the definite article) are frequently used in correctly by new students
    - Plural nouns are typically used without an article:
        - "The source node receives acknowledgements" (not "the acknowledgments")
    - The indefinite is weaker than the definite article:
        - "a large portion of" (not "the large portion of")
    - But there's no need for "the" in section titles - instead of "The System Model," just say "System Model."
    - Articles can be tricky to use, but there are too many of them in paper - pay attention!

Good news: most English problems are not hard to fix - just proofread every sentence with plenty of time!

**Typest your paper correctly and beautifully**

1. Use LaTex, no matter what.
2. Draw figures using a vector-based application: Image-based drawing applications produce images that become fuzzy when zoomed in on a reader's iPad.
3. Proofread your bibliography and make it consistent: Use BibTex to typeset the bibliography in your paper. After downloading BibTex entries from the web, proofread to get them consistent. Keep a consistent style of abbreviating journal and conference titles throughout the bibliography. 

Try your best to find out the venue where the paper was published in, rather than citing arxiv.org.

# 2 Writing good papers fast!

Writing good papers is hard. That's why you are advised to write slowly. In the good old days, academics can take years to write one research paper. We also have the "publish or perish" culture, where the number of papers is an indication of quality. I need to graduate - tell me how to write papers quickly.

http://faculty.hitsz.edu.cn/hubaotian

![截屏2024-10-19 10.02.46.png](/static/img/083df19c1637a02c1271. 5cfe4c8f96e32.%C3%A6%C2%88%C2%AA%C3%A5%C2%B1%C2%8F2024-10-19%2010.02.46.png)

**Know your deadline**

At all times, always work towards a paer deadline.

Deadlines are all equal: It doesn't matter what kind of deadline it is - 
- a conference deadline
- a deadline from a journal special issue
- a deadline you set up yourself for submitting a journal paper

1. You will have to make it no matter what happens.

Set your deadline realistically. Admit that you are not able to write a good paper in a month or two.

But a deadline six months later may not carry a lot of weight either - unless you are extremely disciplined.

2. Make it public knowledge

Tell as many peoples as you can when your deadline will be.

So that it feels like a public disgrace if you fail to meet the deadline. As a researcher, your sure wish to be seen as responsible and dependable.

You work around the clock to make sure that others who depend on you will have the peace of mind that the work will be done and done by the deadline.

3. Allocate your time well.

Design a rough timeline at the very beginning. Assign deadlines to each milestone in your timeline.

If necessary, ervise your timeline along the way. But think seriously why you missed your milestone deadlines. Don't procrastinate and delay all the work to the days before the deadline. It may be feasible, but it will be stressful and it degrades the quality of your work

4. Don't follow, lead!

You are the first author of the new paper, right? Then you are the leader, the boss, and the person who cares most about this work.

Start your team at the beginning of your project. Only invite trustworthy collaborators when building your team. Your collaborators include your advisors - get them to work for you. Get them to commit their time. Warn them of the tough work ahead along the way. Promise them an exciting outcome. Assign your collaborators to milestones - they are your "resources". Remind them about their milestone deadlines often. Persist if they forget about your requests. Cancel their "membership" in your team if they consistently fail. Have backup plans if they fail to deliver by the deadline.

**Choose a problem with the right size**

It must be feasible to be completed before the deadline.

Goal: make solid improvements over on paper, and one paper only.

Choose a paper that you enjoyed reading or even better, a paper with source code. If its source code is not on GitHub, ask the authors. If you don't have source code, estimate the amount of time to reproduce the work. If it may take a long time, give up and choose another paper.

Read the paper along with its source code. The source code contains secret ingredients in its "recipe". It helps you to understand the paper completely and deeply. And only with a complete and deep understanding can you create something new. And create something new quickly. To do it, you will have to a deadline in your timeline for creating something new.

A deadline forces you to think long and hard. Revise the source code and do a feasibility test for your idea. If its feasibility is promising, think about its theoretical properties - anything provably correct? Your little problem must fit into the scope of the conference you'll submit to.

Goal: get the paper accepted.

Many conferences have their favoured research topics.

Writing Perfect Papers

> | Resource | Info |
> | :---        |:--- |
> | Paper       | http://arxiv.org/abs/2311.07911 |
> | Code & Data | https://github.com/google-research/google-research/tree/master/instruction_following_eval |
> | Public      | arXiv |
> | Date        | 2024.09.13 |


# Summary Overview

对于 LLM 的 Instruction Following 能力的评估没有标准化：人工评估成本高、速度慢，而且无法客观地重复，而基于 model judge 的自动评估则可能存在偏差或受到 judge model 能力的限制。作者为此引入了指令跟踪评估(Instruction-Following Eval, IFEval)。IFEval 是一种简单易行的评估基准，它侧重于一组“可验证的指令”，例如“write in more than 400 words”和“metion the keyword of AI at least 3 times”。作者确定了25种可验证的指令，并构建了约500个 prompt，每一个 prompt 包含一个或多个可验证指令。

# Main Content

现有的评估方法可分为3大类，每一类都有各自的缺点；
1. **Human evaluation** is time consuming, expensive and relies on a set of humman annotators, leading to potential bias and inconsistencies for reproducibility.
2. **Model-based evaluation** involves using an internal or external model to assess the performance of the target model. However, this approach heavily rely on the correctness of the evaluator model, which is not guaranteed. If the evaluator model has significatn limitations, it yields misleading evaluation signals.
3. **Quantitative benchmarks** provide a standardized and scalabel evaluation approach.

我们总共创建了 25 个可验证指令列表。我们还创建了一组 541 个提示，每个提示包含一个或多个可验证的指令。

总之，我们提出了 IFEval：Instruction-Following Eval，这是一个使用一组包含可验证指令的提示来评估 LLM 遵循指令能力的基准。这些可验证的指令是原子指令，人们可以用一个简单、可解释和确定性的程序来验证相应的响应是否遵循了指令。

![image.png](/static/img/e4d0bce01e86dcf5f2783d5ac65e9ba5.image.png)

![image.png](/static/img/005384b24cb4f672439d3ac89787ebae.image.png)

![image.png](/static/img/1573e3aacc129c48d03d56e92c251a38.image.png)

通过四个步骤来创建提示能够缓解一些问题：
1. 生成一组基本体式，并在每条提示的末尾附上一至三条随机选择的可验证指令；
2. 使用少量提示来识别不合逻辑的提示，并将其删除
3. 采用另一种基于少量提示的方法，对每个提示进行重新措辞，以增加 prompt 的多样性
4. 对重新生成的 prompt 逐一进行人工检查和编辑

**strict metric**

$$\text{is followed}(resp,inst)=\begin{cases}\text{True},&\text{if instruction is followed.}\\\text{False},&\text{otherwise.}\end{cases}$$

**loose metric**

$$\text{is followed}_{{\mathrm{loose}}}(resp,inst)=\mathrm{Any}{\left(\mathrm{is_followed}(\mathrm{transform}_{t}(resp),inst\right)}\mathrm{~for~}t=1,2,\cdots$$

其中，$\mathrm{transform}_t(resp)$是第$t$个转换后的响应。使用以下转换函数对每个响应进行转换：
1. 删除 markdown 语法中常见的字体修饰符，尤其是 "*"和 "**"。
2. 去掉回复的第一行，这样我们就可以跳过 "sure, here it is:"这样的开场白。
3. 删除回复的最后一行，这样我们就可以跳过 "Hope it helps."这样的结束语。

![image.png](/static/img/7555555957519e05f30c9ee8d7717a00.image.png)

![image.png](/static/img/28a88d4f10ed1526d0f2a6448dd5a90f.image.png)

# 🤖ChatGPT

:::info{title="ChapGPT"}
1. 论文的创新之处与独特性：
   - **基于可验证指令的评估方法**：论文提出了一个名为IFEval的评估方法，专门用于评估大型语言模型(LLMs)的指令跟随能力。这种评估方法使用了25种可验证的指令（如字数限制、特定关键字使用等），并且设计了约500个提示符，通过程序化验证模型的响应是否符合这些指令。这使得评估过程自动化、可重复且客观，避免了人类评估中的主观性和成本问题。
   - **简化复杂评估的模型**：IFEval通过设计明确的规则和“可验证指令”解决了以往自然语言处理模型评估中由于语言模糊性和多义性导致的复杂性，使得评估标准更加统一和清晰。
   - **宽松和严格的双重评估标准**：为了解决单一评估标准的局限性，论文设计了“严格”和“宽松”两种指标，前者基于完全匹配，后者允许一定的变通（如忽略markdown标签等），这提升了评估的灵活性和精度。

2. 论文中存在的问题及改进建议：
   - **指令类型的局限性**：尽管论文提出了25种可验证指令，但这些指令大多局限于格式、字数、语言等浅层面，缺乏对复杂任务或多模态任务（如图像生成、视频处理等）的扩展。改进建议是进一步扩展指令类型，使其涵盖更多复杂的实际应用场景。
   - **模型对评估结果的依赖**：目前的评估结果依赖于当前可用的两个LLM（GPT-4和PaLM 2），然而模型间的对比并不具备可推广性，因为这些模型的参数差异较大，无法进行直接比较。建议未来增加更多模型的测试，并在不同大小、架构的模型上进行广泛评估。
   - **指令冲突问题**：论文提到多个指令组合时可能会产生冲突，尽管已经通过few-shot方法减少了这种情况，但在未来的工作中，建议更系统地处理指令冲突，可能通过优化指令生成逻辑或引入冲突检测机制来解决这一问题。

3. 基于论文的内容和研究结果，提出的创新点或研究路径：
   - **引入多模态指令评估**：当前的IFEval仅限于文本生成任务，可以扩展到多模态任务，如图像生成、视频描述等。未来的研究可以开发基于多模态模型的指令跟随评估方法。
   - **面向复杂任务的动态评估**：基于固定指令的评估方法难以覆盖诸如逻辑推理、编程等复杂任务。可以开发一种动态生成指令的评估框架，适应更复杂和开放的问题场景。
   - **基于人类互动的实时评估**：将IFEval扩展至与人类用户的互动评估，设计可模拟真实应用场景的交互指令（如实时问答系统、对话系统的指令跟随能力评估）。

4. 为新的研究路径制定的研究方案：
   - **多模态指令评估研究方案**：
     - 研究方法：扩展当前的IFEval，设计多模态可验证指令，例如“生成一张包含特定元素的图片”或“描述视频中的具体场景”等。
     - 步骤：
       1. 扩展指令库，包含视觉、音频、文本等多种模态的指令。
       2. 采集多模态任务的数据集，包含视频、图像、音频等多样化输入。
       3. 设计自动化验证程序，通过图像识别、视频分析等技术验证模型是否遵循了多模态指令。
       4. 在多个多模态LLM（如GPT-4、CLIP、DALL·E等）上进行测试。
     - 期望成果：开发出适用于多模态任务的指令跟随评估标准，并且能够为未来的多模态生成任务提供更全面的评估框架。

   - **动态指令生成的复杂任务评估方案**：
     - 研究方法：设计一种能够动态生成指令的评估框架，适应复杂任务场景，如编程、逻辑推理等。每次生成的指令不仅能验证结果，还能评估模型的过程和推理逻辑。
     - 步骤：
       1. 创建动态指令生成器，基于任务复杂度和场景自动生成多步骤指令。
       2. 设计任务库，涵盖逻辑推理、编程、复杂决策等多种任务类型。
       3. 通过多层次的验证标准，评估模型是否正确遵循了复杂指令，包括过程验证（如步骤顺序）和结果验证。
       4. 使用不同复杂度的LLM进行测试，分析模型在复杂任务中的表现差异。
     - 期望成果：构建一个能够生成多步复杂指令的评估框架，提升对复杂任务场景中LLM性能的评估能力。

   - **基于人机交互的实时评估方案**：
     - 研究方法：设计一个实时交互系统，评估LLM在与人类互动过程中遵循指令的能力。通过模拟用户的真实需求场景，评估LLM对指令的反应和执行情况。
     - 步骤：
       1. 设计一个模拟真实交互场景的数据集，包括实时问答、对话系统等应用场景。
       2. 开发实时反馈和评分系统，用户在交互中可以给出即时反馈，评估模型的指令执行情况。
       3. 通过定量和定性分析评估LLM在不同交互场景中的表现。
       4. 对比不同模型在实时交互中的表现，确定其指令跟随能力的差异。
     - 期望成果：开发出一种能够在实时交互中评估模型指令跟随能力的系统，适用于对话机器人、虚拟助手等应用场景的模型评估。

:::

# Others

[arXiv-2023] Instruction-Following Evaluation for Large Language Models

> | Resource | Info |
> | :---        |:--- |
> | Paper       | http://arxiv.org/abs/2404.11018 |
> | Code & Data | / |
> | Public      | arXiv |
> | Date        | 2024.09.11 |


# Summary Overview

最新扩展的上下文窗口使我们能够使用数百或数千个示例来研究 ICL ，这就是 many-shot 机制。从 few-shot 到 many-shot，观察到了 LLM 在各种生成任务上的显著增加。但是可用的手工生成的示例的缺乏限制了 many-shot，为了减少这种限制，作者设计了两类方法：1. Reinforced ICL 使用模型生成的示例进行替代。2. Unsupervised ICL 完全删除了利用仅保留了问题。两类方法都对于 many-shot 有积极的影响。

# Main Content

Many-shot learning - 使用大量的示例进行 ICL，可以更好地指定任务，减少对微调的需求，并可能使 LLM 更具通用性和适应性。

使用成百上千的 shot 学习效果明显优于使用少量 shot 的ICL (Figure 1)。

![image.png](/static/img/a391290bb2465d82920a80aed8c4b024.image.png)

通常只有当 shot 数量达到数十万个 token 时，才能实现最高性能 (Figure 2)。我们发现，在 MATH、GPQA 和 Big-Bench Hard 等问题解决任务中，reinforced ICL 和 unsupervised ICL 比使用手工生成的 few-shot 效果更有效。

![image.png](/static/img/332c6105ea2d2d6ec6bf81cee7c2416e.image.png)

**Contributions**:
- **Scaling ICL**: 使用了 Gemini 1.5 Pro 系统地评估了 ICL 在各种任务不同规模的上下文示例下的性能。结果表明，从 few-shot 到 many-shot 时，性能会出现大幅提升。
- **Reforced and Unsupervised ICL**: 发现使用模型生成的理由或是只使用问题能够减少对于手工生成数据的依赖。
- **Analysing ICL**: many-shot ICL 能够克服预训练偏差，性能与微调相当，并且可以学习非 NLP 任务，而 few-shot ICL 在这些任务中则是十分的吃力。还揭示了 next-token prediction loss 可能并不能很好地预测 ICL 的性能。

为了获得可靠的结果，作者使用不同的随机数种子对每个 K-shot prompt进行多次随机抽样，并报告平均性能以及单个 seed 的性能。为了确保使用 K-shot prompt 逐步提供更多信息，设置中的 任何 K-shot 都包括少于 K-shot 提示中的所有上下文示例。

**Scaling In-Context Learning**

- Machine Translation**

使用 chrF2++ 对测试集中的前 150 个句子进行了性能评估，chrF2++ 是基于生成译文和参考译文之间字符和单词 n-gram 重合度的标准指标。

![image.png](/static/img/714c6ea26d3b37ac104cf6f3ec04cdef.image.png)

- Planning: Logistics Domain

![image.png](/static/img/7e67eae3a5ff20a4abadd7b05baea7a6.image.png)

- Reward Modelling with Many-Shot ICL: Learning Code Verifiers

![image.png](/static/img/e3b97f57e7180c65e15086a6a55228ec.image.png)

**Many-shot learning without Human-Written Rationales**

- Reinforced ICL
- Unsupervised ICL: 由以下部分组成 1.前言，如"You will be provided questions similar to the ones below." 2.未解决的输入或问题列表 3. zero-shot instruction 或带有所需输出格式输出的 few-shot prompt

当 LLM 已经掌握了解决任务所需的知识时，在提示中插入任何可以缩小任务所需知识范围的信息都会有所帮助。

![image.png](/static/img/619c405fecd92ebfbef14abb686bd20e.image.png)

Singh 等人发现，根据 MATH 数据生成的模型数据集微调的模型，可提高 GSM8K 的测试性能，而 GSM8K 的问题分布与 MATH 不同。

![image.png](/static/img/ccbda1298a7330187394dcd93c0a6e78.image.png)

![image.png](/static/img/cf9e1151fcef1d42fb6e25410eb936f4.image.png)

如 Figure 9 所示，在几乎所有的任务中，Reinforced ICL 都大大优于 Unsupervised ICL，而 Unsupervised ICL 又优于标准的 3-shot CoT prompt。

**Analyzing Many-Shot ICL**

![image.png](/static/img/8731b575b9bf3de034900a583ca12030.image.png)

这表明，在 shot 较少的 ICL 中，模型难以克服预先训练中存在的偏差。然而，随着 shot 的增加，翻转标签和抽象标签的性能显著提高，接近默认标签的性能。

![image.png](/static/img/8187bf086caa19695b2dc6af8fbbca69.image.png)

![image.png](/static/img/db90ff91ba92ffa6acf28f7a54eeda3c.image.png)

具体来说，我们对来自 MATH 训练拆分的固定 50 个上下文示例的十种不同随机排序进行了评估，并对保留的 MATH500 测试集进行了性能评估。

![image.png](/static/img/9d2221b86fb68571bc88cf163a566127.image.png)

相比之下，多镜头 ICL 不需要任何训练，但推理成本较大，而 KV 缓存可以大幅降低推理成本。

![image.png](/static/img/0960f01e7006bd8cc03aaf16797b6fdd.image.png)

此外，多镜头性能很可能成为评估长语境模型质量的一个有价值的指标，超越了流行的大海捞针测试。

# 🤖ChatGPT

:::info{title="ChapGPT"}
1. 论文的创新之处与独特性：
   - **扩展的上下文窗口与多示例学习**：该论文通过扩展LLM的上下文窗口，首次在自然语言处理中深入研究了多示例情境学习（Many-Shot In-Context Learning，ICL）的性能。传统的少示例ICL仅处理少量输入-输出示例，而多示例ICL则能够处理成百上千个示例，大幅提高了模型在生成任务和判别任务中的表现。 
   - **克服预训练偏差**：论文发现，多示例ICL在任务复杂度增加时，能够显著克服预训练模型的偏差，这是在少示例学习中难以做到的。论文指出，ICL在数学问题解决和推理任务中的表现与模型微调（fine-tuning）相当。
   - **模型生成推理链与无监督学习**：引入了强化ICL和无监督ICL两种新方式，以应对多示例学习对人类生成数据的高需求。通过使用模型生成的推理链替代人类推理链，论文成功在一些复杂推理任务中取得了显著效果。
   
2. 论文中存在的问题及改进建议：
   - **上下文窗口对性能的影响**：尽管论文指出了多示例ICL的显著优势，但部分任务在示例数目增加后，性能反而有所下降（例如MATH任务）。这表明仅增加示例数未必总是带来性能提升。改进建议是进一步研究ICL中示例顺序和不同示例组合对性能的影响，优化示例的选择和排序策略。
   - **负对数似然与实际性能不一致**：论文指出，负对数似然（NLL）并不能很好地预测ICL在复杂问题上的表现。这揭示了当前评估指标的局限性。建议进一步探索更适合评估ICL的指标，例如多样性生成、推理深度等。
   - **对模型生成推理链的信任问题**：虽然模型生成推理链在强化ICL中取得了较好的表现，但可能出现假阳性问题，即错误的推理链导致正确的最终答案。建议结合更多质量控制机制，例如在推理链生成过程中引入多样性采样或通过其他验证模型进行交叉验证。

3. 基于论文的内容和研究结果，提出的创新点或研究路径：
   - **研究点1：多任务多示例ICL的示例排序策略优化**：研究示例在多任务环境下的排序与组合如何影响ICL性能。
   - **研究点2：跨领域知识迁移下的多示例ICL**：探索多示例ICL在跨领域任务中的迁移能力，尤其是在完全未知领域中的表现。
   - **研究点3：自适应示例选择与自我监督的结合**：开发自适应的示例选择策略，并结合自我监督机制，减少对高质量人类生成数据的依赖。

4. 为新的研究路径制定的研究方案：
   - **研究点1：多任务多示例ICL的示例排序策略优化**
     - **研究方法**：在多任务（如翻译、推理、分类）环境下，使用强化学习方法训练示例选择器，以确定最优示例排序策略。通过示例的排列组合来评估不同任务下的ICL性能。
     - **步骤**：
       1. 收集多任务数据集，确保每个任务有足够的示例。
       2. 训练一个强化学习代理，针对每个任务学习最优的示例排序。
       3. 比较不同排序策略对模型在每个任务上的表现。
       4. 优化示例选择机制，使其适应不同任务需求。
     - **期望成果**：提高多任务情境下的ICL表现，并且减少示例排序对模型性能的负面影响。

   - **研究点2：跨领域知识迁移下的多示例ICL**
     - **研究方法**：设计实验，验证ICL在跨领域任务中的泛化能力，尤其是在预训练数据中没有覆盖的任务上。使用不同领域的数据进行训练，研究模型如何迁移学习。
     - **步骤**：
       1. 从不同领域（如医学、法律、工程等）收集任务数据。
       2. 设计实验，分别在同领域和跨领域设置下进行ICL训练。
       3. 评估模型在跨领域任务中的迁移性能，并分析影响迁移效果的因素。
     - **期望成果**：验证多示例ICL在跨领域任务上的迁移能力，提出改进的多示例ICL模型，提升其跨领域泛化能力。

   - **研究点3：自适应示例选择与自我监督的结合**
     - **研究方法**：开发一个基于自我监督的示例选择机制，使得模型能够自动筛选出对任务最有帮助的示例，并不断自我改进。结合强化ICL，减少对人类生成数据的依赖。
     - **步骤**：
       1. 设计自我监督机制，允许模型生成并验证自身推理链。
       2. 训练模型学习哪些示例对任务完成最为关键，并动态调整示例选择。
       3. 在多种任务上测试自适应示例选择的效果，并对比传统的固定示例选择方法。
     - **期望成果**：减少多示例ICL对人工生成数据的依赖，提高任务推理的质量与效率。

:::

# Others

[arXiv-2024] Many-Shot In-Context Learning

> | Resource | Info |
> | :---        |:--- |
> | Paper       | http://arxiv.org/abs/2406.04093 |
> | Code & Data | https://github.com/openai/sparse_autoencoder |
> | Public      | arXiv |
> | Date        | 2024.09.04 |


# Summary Overview

Openai 关于 SAE 的一篇工作。提出了 TopK SAE。Sparse autoencoder (SAE) 提供了一种具有前景的无监督方法，通过从神经网络层中重建激活来提取具有可解释性特征。同时还介绍了一些新指标来评估特征质量。

# Main Content

**三个问题：**
- whether certain hypothesized features were recovered
- whether downstream effects are sparse
- whether features can be explained with both high precision and recall

## Methods:

**Setup**

Wec choose a layer near the end of the network, which should contain many features without being specialized for next-token predictions.

Evaluation: sparsity $L_0$, reconstruction meansquared error (MSE)

**Baseline: ReLU autoencoder**

$$\begin{aligned}&z=\text{ReLU}(W_\text{enc}(x-b_\text{pre})+b_\text{enc})\\&\hat{x}=W_\text{dec}z+b_\text{pre}\end{aligned}$$

The training loss is defined by $\mathcal{L}=\left|\left|x-\hat{x}\right|\right|^2_2+\lambda\left|\left|z\right|\right|_1$, where $\left|\left|x-\hat{x}\right|\right|_2^2$ is reconstruction MSE. $$\left|\left|z\right|\right|_1$$ is an $L_1$ penalty promoting sparsity in latent activations $z$, and $\lambda$ is a hyperarameter that need to be tuned.

**TopK activation function**

Using an activation function (TopK) that only keeps the $k$ largest latents, zeroning the rest.

$$z=\mathrm{TopK}(W_{\mathrm{enc}}(x-b_{\mathrm{pre}}))$$

The training loss is simply $\mathcal{L}=\left|\left|x-\hat{x}\right|\right|_2^2$. No penalty anymore!

Using $k$-sparse autoencoder has a number of benefits:
- In remove the need for the $L_1$ penalty. $L_1$ is an imperfect approximation of $L_0$, and it introduces a bias of shrinking all positive activations towward zero.
- It enables setting the $L_0$ directly, as opposed to tuning an $L_1$ coefficient $\lambda$, enabling simpler model comparison and rapid iteration. It can also be used in combination with arbitrary activation functions
- It empirically outperforms baseline ReLU autoencoders on the sparsity-reconstruction frontier, and this gap increases with scale.
- It increases monosemanticity of random activating examples by effectively clamping small activations to zero.

![image.png](/static/img/8ed27dd1baf6f6a2285912ad81c635e3.image.png)

**Preventing dead latents**

We find two important ingredients for preventing dead latents:
- intialize the encoder to the transpose of the decoder
- use an auxiliary loss that models reconstruction errorusing the top-$k_\mathrm{aux}$ dead latents.

Using thesetechniques, even in our largest (16 million latent autoencoder only 7% of latents are dead).

![image.png](/static/img/d082ba25e8413c644daddfb4b1128860.image.png)

![image.png](/static/img/23a013600b6d3fc4637ad76b3ad81751.image.png)

## Evaluation

**Following metrics**:

- **Downstream loss:** How good is the language model loss if the residual stream latent is replaced with the autoencoder reconstruction of that latent?
- **Probe loss:** Do autoencoders recover features that we believe they might have?
- **Explainability:** Are there simple explanations that are both necessary and sufficient for the activation of the autoencoder latent?
- **Ablation sparsity:** Does ablating individual latents have a sparse effect on downstream logits?

**Finding simple explaination for features**

Neuron to Graph (N2G), a substantially less expressive but much cheaper method that outputs explanations in the form of collections of n-grams with wildcards.

![image.png](/static/img/3bfc26a1a2793b560713e06f973715c9.image.png)

![image.png](/static/img/476d4a9163a3b3f169b37abc7cd8e94d.image.png)

![image.png](/static/img/42480bc7ab3a835f220ed7cc9056c4b0.image.png)

## Limitations and Future Directions

- TopK forces every token to use exactly $k$ latents, which is likely suboptimal. Ideally we would constrain $\mathbb{E}\left[L_0\right]$ rather than $L_0$.
- The optimization can likely be greatly improved, for example with learning rate scheduling, better optimizers, and better aux losses for preventing dead latents.
- Much more could be done to understand what metrics best track relevance to downstream applications, and to study those applications themselves. Applications include: finding vectors for steering behavior, doing anomaly detection, identifying circuits, and more.
- We're excited about work in the direction of combining MoE and autoencoders, which would substantially improve the asymptotic cost of autoencoder training, and enable much larger autoencoders.
- A large fraction of the random activations of features we find, especially in GPT-4, are not yet adequately monosemantic. We believe that with improved techniques and greater scale this is potentially surmountable.
- Our probe based metric is quite noisy, which could be improved by having a greater breadth of tasks and higher quality tasks.
- While we use N2G for its computational efficiency, it is not only able to capture very simple patterns. We believe there is a lot of room for improvement in terms of more expressive explanation methods that are also cheap enough to simulate to estimate explanation precision.
- A context length of 64 tokens is potentially too few tokens to exhibit the most interesting behaviors of GPT-4.

## Related Work

Malllat 和 Zhang 引入了在超完整字典上的稀疏编码。Olshausen 和 Field 通过无监督的方法从数据中学习字典改善了此方法。这种方法在图像处理中影响极大。之后 Hinton 和 Salakhutdinov 提出了自动编码器结构以降低维度。结合这些概念，人们开发了 SAE 以训练具有稀疏先验的自动编码器，例如使用$L_1$惩罚来提取稀疏特征。Makhzani 和 Frey 通过引入 K-Sparse autoencoder，使用 TopK 激活功能而不是 $L_1$惩罚 来完善这一概念。 Makelov 等人使用度量标准来自动评估编码器，该指标可以测量从先前发现的回路中恢复特征。

最近，将 SAE 用于语言模型，并且有多种 SAE 在开源的小模型上进行训练。Mark 等人发现SAE 可以从语言模型中发现回路。Wright 和 Sharkey 指出 SAE 会受到 $L_1$惩罚激活缩小。Taggart 和 Rajamanoharan 建议使用不同的激活来解决 SAE 中的激活收缩问题。Braun 等人建议在下游 KL 上训练 SAE，而不是重建 MSE。

Kaplan 等人研究了语言模型的缩放定律，这些定律发掘了损失如何随各种超参数而变化。Clark 等人使用双线性你和探索与稀疏有关的缩放定律。Lindsey 等人研究了专门针对自动编码器的缩放定律，将损失定义为重建和稀疏性的特定平衡（而不是简单地重建）。


# 🤖ChatGPT

:::info{title="ChapGPT"}

### 1. 论文的创新之处与独特性：

这篇论文提出并系统评估了稀疏自动编码器在处理大型语言模型激活数据中的应用，特别是在GPT-4的残差流中。这项工作具有几个创新之处：
- **使用k-稀疏自动编码器**：论文采用了k-稀疏自动编码器，这种方法通过限制活跃的潜在变量数量直接控制稀疏性，简化了超参数调优，显著提高了稀疏性和重构精度之间的平衡。
- **解决了死神经元问题**：通过引入初始化和辅助损失，显著减少了在大规模训练中出现的死神经元问题，使得在极大规模的稀疏自动编码器中（如16百万潜在变量的模型）依然能够保持高效的训练。
- **提出了新的评价指标**：为评估稀疏自动编码器的特征质量，论文引入了一系列新的评价指标，包括对假设特征的恢复、激活模式的可解释性和下游影响的稀疏性，推进了对模型可解释性和特征解读的研究。
- **稀疏自动编码器的扩展性**：论文展示了稀疏自动编码器在极大规模条件下的扩展性，成功在GPT-4的40亿个token上训练了具有16百万潜在变量的自动编码器。

这些创新不仅在稀疏性和可解释性方面进行了探索，还推动了稀疏自动编码器的规模扩展能力研究。

### 2. 论文中存在的问题及改进建议：

虽然论文具有许多创新点，但仍有一些问题值得改进：
- **稀疏性的固定限制**：使用TopK方法时，每个token的稀疏性L0是固定的，这可能导致在某些情况下的表达能力不足。改进建议是引入更加灵活的稀疏性约束，例如平均稀疏性约束（E[L0]），允许在特定情况下动态调整稀疏性。
- **优化过程的改进空间**：论文提到优化过程（如学习率调度和更好的优化器）可以显著改善性能。建议未来的工作中探索自适应优化器（如RMSProp或AdamW）以及多步学习率衰减方案，提升收敛速度和性能。
- **更多的特征解释**：虽然引入了N2G（Neuron to Graph）解释方法来捕捉激活的n-gram模式，但其表达能力有限，建议引入更具表现力的特征解释方法，如基于深度生成模型的特征重构或图模型方法，增强解释的精度和通用性。

### 3. 基于论文的内容和研究结果，提出的创新点或研究路径：

1. **自适应稀疏性自动编码器**：探索一种可以动态调整稀疏性L0的自动编码器，允许模型根据输入数据自适应调整稀疏性水平，优化模型在不同复杂度输入下的表现。
2. **多尺度特征表示的稀疏自动编码器**：设计一个基于多尺度特征表示的自动编码器，能够捕捉不同抽象层次的特征，同时保持较高的稀疏性，提升模型对复杂数据的解读能力。
3. **结合稀疏自动编码器的混合专家模型（MoE）**：研究如何将稀疏自动编码器与混合专家模型结合，利用MoE模型的高效计算特性，在提升自动编码器表达能力的同时减少训练和推理时的计算成本。

### 4. 为新的研究路径制定的研究方案：

#### 研究路径1：自适应稀疏性自动编码器
- **研究方法**：设计一种自适应TopK机制，通过引入一种稀疏性控制模块（如基于LSTM或变分推断的方法），使得模型能够根据输入自适应调整每层的稀疏性L0。模型训练过程中，动态调整每个token的激活神经元数量，确保稀疏性与数据复杂度匹配。
- **步骤**：
  1. 设计自适应稀疏性模块，允许模型动态调整稀疏性L0。
  2. 在GPT-2和GPT-4的激活数据上进行实验，比较固定L0和自适应L0的效果。
  3. 评估模型在重构精度、稀疏性控制和下游任务性能上的表现。
- **期望成果**：自适应稀疏性模型在处理复杂输入时，能够表现出更高的灵活性，同时保持稀疏性控制，提升整体模型的表达能力。

#### 研究路径2：多尺度特征表示的稀疏自动编码器
- **研究方法**：设计一种多尺度自动编码器结构，利用不同规模的潜在表示来捕捉不同抽象层次的特征，结合稀疏性约束，使得模型能够以稀疏的方式表示多尺度特征。
- **步骤**：
  1. 设计一种多尺度的稀疏自动编码器架构，每一层潜在空间表示不同尺度的特征。
  2. 在GPT-4激活数据上进行实验，验证模型对不同层次特征的捕捉能力。
  3. 通过下游任务评估模型的泛化能力和可解释性。
- **期望成果**：多尺度稀疏自动编码器可以有效捕捉语言模型中的不同层次特征，提升模型对复杂语言现象的解析能力，并增强可解释性。

#### 研究路径3：结合稀疏自动编码器的混合专家模型（MoE）
- **研究方法**：将稀疏自动编码器与MoE模型结合，利用MoE模型的专家模块选择机制，实现稀疏自动编码器在处理复杂任务时的高效计算，同时减少训练和推理时的计算成本。
- **步骤**：
  1. 设计一个结合稀疏自动编码器和MoE的模型框架，在每个输入token上选择特定的专家模块进行激活。
  2. 在GPT-4数据上验证该框架的性能，重点评估模型的计算效率和下游任务表现。
  3. 对比单一稀疏自动编码器和混合专家模型的性能差异。
- **期望成果**：结合MoE的稀疏自动编码器在处理大规模语言模型数据时能够显著减少计算资源需求，并保持或提升模型的解读能力和下游任务性能。
:::

# Others

[arXiv-2024] Scaling and evaluating sparse autoencoders

# The Llama 3 Herd of Models



# 1 Introduction

**Two main stages:**

1. a pre-training stage in which the model is trained at massive scale using straightforward tasks such as next-word prediction or captioning
2. a post-training stage in which the model is tuned to follow instructions, align with human preferences, and improve specific capabilities (for example, coding and reasoning)

We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2.

**Post-training procedure**: supervised finetuning (**SFT**), rejection sampling (**RS**) and direct preference optimization (**DPO**).


![image.png](/static/img/c7775a46d5d3073ca90f280502032b53.image.png)

# 2 General Overview

![image.png](/static/img/2970a0dd5e2243584204960738fcf2a2.image.png)

# 3 Pre-Training

**Pre-training involves:**
1. the curation and filtering of a large-scale training corpus
2. the development of a model architecture and corresponding scaling laws for determining model size
3. the development of techniques for efficient pre-training at large scale
4. the development of a pre-training recipe

**Web Data Curation:**
- PII and safety filtering
- De-duplication
    - URL-level de-duplication
    - Document-level de-duplication
    - Line-level de-duplication
- Heuristic filtering
    - Use duplicated n-gram coverage ratio to remove lines that consist of repeated content such as logging or error messages. Those lines could be very long and unique, hence cannot be filtered by line-dedup.
    - Use "dirty word" counting to filter out adult websites that are not covered by domain block lists.
    - Use a token-distribution Kullback-Leibler divergence to filter out documents containing excessive numbers of outliner tokens compared to the training corpus distribution. 

**Determining the Data Mix**
- **Knowledge classification.** We develop a classifier to catagorize the types of information contained in our web data to more effectively determine a data mix. We use this classifier to downsample data categories that are over-represented on the web, for example, arts and entertainiment.
- **Scaling laws for data mix.** To determine the best data mix, we perform scaling law experiments in which we train several small models on a data mix and use that to predict the preformance of a large model on that mix. We repeat this process multiple time for different data mixes to select a new data mix candidate. Subsequently, we train a larger model on this candidate data mix and evaluate the performance of that model on several key benchmarks.
- **Data mix summary.** Our final data mix contains roughly 50% of tokens corresponding to general knowledge, 25% of mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens.

![image.png](/static/img/2e845286176ca42ea315d5490a626680.image.png)

**Scaling Laws Issuses**:
1. Existing scaling laws typically predict only next-token prediction loss rather than specific benchmark performance.
2. Scaling laws can be noisy and un reliable because they are developed based on pre-training runs conducted with small compute budgets.
3. **Two-stage methodology**
    - We first establish a correlation between the compute-optimal model's negative log-likelihood on downstream tasks and the training FLOPs.
    - Next, we correlate the negative log-likelihood on downstream tasks with task accuracy, utlizing both the scaling law models and older models trained with higher compute FLOPs. In this step, we specifically leverage the Llama 2 family of models.

![image.png](/static/img/7ef36f51afc1945d40283e53c10565fe.image.png)

To do so, we assume a power-law relation between compute budget, C, and the optimal number of training tokens, $N^*(C)$:

$$
N^*(C)=AC^\alpha
$$

We fit $A$ and $\alpha$ using the data from Figure 2. We find that $(\alpha, A)=(0.53,0.29)$.

Pre-train Llama 3 405B consists of three main stages:
1. initial pre-training
2. long-context pre-training
3. annealing

# 4 Post-Training

![image.png](/static/img/38096c6415d98abeb4f2171b8c976a74.image.png)

The backbone of our post-training strategy is **a reward model** and **a language model**. We first train a reward model on top of the pre-trained checkpoint using human-annotated preference data. We the finetune pre-trained checkpoints with supervised finetuning (SFT), and further align the checkpoints with Direct Preference Optimization (DPO). This process is illustrated in Figure 7. Unless otherwise noted, our modeling procedure applies to Llama 3 405B, and we refer to Llama 3 405B as Llama 3 for simplicity.

We also explored on-policy algorithms such as DPO, but found that DPO required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval.

- Masking out formatting tokens in DPO loss
- Regularization with NLL loss

**Preference Data**

![image.png](/static/img/4cc2b9a0d1c29a0127a49b070e95561a.image.png)

We deploy multiple models for annotation after each round and sample two responses from two different models for each user prompt. We ask annotators to rate the strength of their preference by categorizing it into one of four levels, based on how much more they prefer the chosen response over the rejected one: significantly better, better, slightly better, or marginally better. Consenquently, a portion of our preference data has three responses ranked ($\textit{edited}>\textit{chosen}>\textit{rejected}$)

**SFT Data**

![image.png](/static/img/e48b7d4c3e5535cb1dd9b48080a2c995.image.png)

Our finetuning data is largely comprised of the following sources:
- Prompts from our human annotation collection with rejection-sampled repsonses
- Synthetic data targeting specific capabilities
- Small amounts of human-curated data

**Data Processing and Quality Control**

- Data cleaning
- Data pruning
    - Topic classification
    - Quality scoring
    - Difficulty scoring
    - Senmantic deduplication

**Code**

Here, we present our work on improving these coding capabilities via training a code expert, generating synthetic data for SFT, improving formatting with system prompt steering, and creating quality filters to remove bad samples from our training data.

> Continued pre-training on domain-specific data has been shown to be effective for improving performance in a specifc domain.

During development, we identified key issues in code generation, including difficulty in following instructions, code syntax errors, incorrect code generation, and difficulty in fixing bugs.

> Howerver, our initial experiments revealed that training Llama 3 405B on its own generated data is not helpful (and can even degrade performance).

- Problem description generation
- Solution generation
- Correctness analysis
    - Static analysis
    - Unit test generation and execution
- Error feedback and iterative self-correction
- Fine-tuning and iterative improvement

![image.png](/static/img/acdb24c38e8082bea59ff61db18bc08b.image.png)

**Math and Reasoning**

Several challenges:
- Lack of prompts
- Lack of ground truth chain of thought
- Incorrect intermediate steps
- Teaching models to use external tools
- Discrepancy between training and inference

Following methodologies;
- Addressing the lack of prompts
- Augmenting training data with step-wise reasoning traces
- Filtering incorrect reasoning traces
- Interleaving code and text reasoning
- Learning from feedback and mistakes

**Factuality**

We follow the principle that post-training should align the model to "know what it knows" rather than add knowledge. Our primary approach involves generating data that aligns model generations with subsets of factual data. **To achieve this, we develop a knowledge probing technique that takes advantage of Llama 3's in-context abilities.** This data generation process involves the following procedure:
1. **Extract a data snippet** from the pre-training data.
2. **Generate a factual question** about these snippets (context) by prompting Llama 3
3. **Sample responses** from Llama 3 to the question
4. **Score the correctness** of the generations using the original context as a reference and Llama 3 as a judge
5. **Score the informativeness** of the generations using Llama 3 as a judge
6. **Generate a refusal** for responses which are consistently informative and incorrect across the generations, using Llama 3

We use data generated from the knowledge probe to encourage the model to only answer questions which it has knowledge about, and refuse answering those questions that it is unsure about. Further, pre-training data is not always factually consistent or correct. We therefore also collect a limited set of labeled factuality data that deals with sensitive topics where factually contradictory or incorrect statements are prevalent.

# 5 Results

# 6 Inference

# 7 Vision Experiments

# 8 Speech Experiments

# 9 Related Work

# 10 Conclusion