The benefits of providing good examples.
If you give an LLM some examples of the output you are looking for, the LLM will learn to generate that output. Each example is a “shot” or a chance to learn the output. Even small models can learn to generate a wide variety of outputs if given enough examples.
Example of a few shot prompt template:
Your task is to generate a list of 3 names of people who are famous for their work in the following field:
Science Fiction
[
"Isaac Asimov",
"Arthur C. Clarke",
"Ray Bradbury",
]
Space Exploration
[
"Neil Armstrong",
"Buzz Aldrin",
"Yuri Gagarin",
]
{insert new topic}This is a static few shot prompt with 2 examples.
Dynamic few shot prompting is a technique that allows you to dynamically change the examples you give to the LLM.
Here is a diagram of the process:
graph TD
A[User Input] --> B[Look up relevant examples]
B --> C[Template for few shot prompt]
C --> D[LLM]
D --> E[Output]graph TD
A[User Input] --> B[Look up relevant examples]
B --> C[Template for few shot prompt]
C --> D[LLM]
D --> E[Output]
Here is an example of a dynamic few shot prompt template:
Your task is to answer user questions.
{example question 1}
{example answer 1}
{example question 2}
{example answer 2}
{example question 3}
{example answer 3}
{user question}And the trick is to find the most relevant examples for the user question.
Dynamic few shot prompting is a type of RAG (Retrieval Augmented Generation) where we retrieve the most relevant examples from a knowledge base of examples. Ideally, the knowledge base is a collection of user input and gold-standard answers that would be useful examples for the LLM to learn from.
In practice, we can embed the user question and the examples and then use a similarity search to find the most relevant example user input. Note that we want to find example input that is similar to the user input, not the example output.
The key to dynamic few shot prompting is to have a good knowledge base with a wide variety of examples that are similar to the expected user input. Even small LLMs can interpolate between examples to generate a good answer. If your knowledge base doesn’t have coverage of the expected user input, the LLM will not be able to generate a good answer.
Curating a knowledge base can be a lot of work. Human experts are needed to review the examples and determine if they are correct and relevant.
A knowledge base of examples will make it easy to calculate the similarity between the user input and the examples. The similarity score can indicate if the user input is not covered by the examples. Implementing a threshold for minimum similarity can ensure that the LLM refuses to answer questions that are not covered by the examples. Logging below threshold user inputs can help the curation process by identifying when the knowledge base is missing important examples that users care about.
Context Windows are getting larger, and it’s tempting to stuff all the examples into the prompt. But this has some drawbacks. Prompt tokens cost money and time to load into the LLMs KV cache and will slow down the time to first token. Prompt caching can help with this but comes with its own limitations.
Bad or irrelevant examples can confuse the LLM and cause it to generate incorrect answers which is harder to troubleshoot in a monolithic prompt.
Dynamic few shot prompting allows you to identify which examples are most important across all of your examples. Leave-one-out validation can help identify which examples are most important and where knowledge base coverage is lacking.
Dynamic few shot prompting is useful for tasks where the LLM is not trained on the task, perhaps because the data is proprietary or the task is not common enough to be in the training data.