Lab 4b: Tree-of-Thought#
This notebook introduces the Tree of Thoughts prompting strategy, that guides generative models for text to generate, evaluate, expand on, and decide among multiple solutions. The process is similar to how humans solve problems by evaluating various potential solutions before deciding on the most promising one.
About This Lab#
Throughout this lab, you will encounter two types of interactive elements:
No coding is needed for an activity. You try to understand a concept,
answer questions, or run a code cell.
Challenges are where you test your understanding by implementing something new or taking a short quiz.
Please work through this notebook from top to bottom to avoid errors due to missing code or context.
Table of Contents#
1. Import libraries#
Let’s start by installing all required packages as specified in the requirements.txt file and importing several libraries.
%%capture
!pip install -q -r ../requirements.txt
import sys
sys.path.append('..')
import boto3
import time
import re
import random
import numpy as np
import textwrap
from collections import Counter
from langchain_core.prompts import PromptTemplate
from igraph import Graph
import plotly.graph_objects as go
from IPython.display import Markdown, display
%load_ext autoreload
%autoreload 2
from mlu_utils.utils import *
2. Set up Bedrock for inference#
To get started, set up Bedrock and instantiate an active bedrock-runtime to query LLMs.
Let us define a custom function to run inference with Bedrock-hosted models. The code below leverages LangChain’s Bedrock integration and allows to use Bedrock-hosted models.
Next, use Bedrock for inference to test everything works as expected:
# Example model prompting
MODEL = "mistral.mistral-7b-instruct-v0:2"
TEMP = 0.8
SAMPLES = 2
greeting_prompt = "<s>[INST]How are you doing?[/INST]"
Markdown(generate_outputs(greeting_prompt, MODEL, TEMP, max_tokens=128, n=5)[0])
I’m just a computer program, so I don’t have feelings or the ability to do things in the same way that a living being does. I’m here to help answer any questions you might have to the best of my ability. Is there a specific topic you’d like to know more about? I’ll do my best to provide you with accurate and up-to-date information. If you have any questions, feel free to ask!
3. Tree of Thoughts (ToT)#
The Tree of Thoughts (ToT) framework was proposed in 2023 in the following two papers:
Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Yao et al. (2023)
Large Language Model Guided Tree-of-Thought, Long (2023)
ToT is motivated by the difficulties that LLMs prompted with simple techniques encounter when trying to solve complex tasks that require exploration or strategic lookahead. ToT generalizes over Chain of Thought (CoT) and encourages exploration over “thoughts” that serve as intermediate steps for general problem solving with LLMs. CoT prompting samples thoughts sequentially, but ToT prompting follows a tree-branching technique. With the ToT technique, the LLM can consider multiple paths instead of one sequential path. ToT is an especially effective method for tasks that involve important initial decisions, strategies for the future, and exploration of multiple solutions.
Below you will see how to use a Bedrock-hosted LLM to solve a task using the ToT technique. You will compare the results with those given by standard and CoT prompts.
3.1 Creative writing task#
We will build on the creating writing exercise shown in Yao et al. which prompts the LLM to write a coherent passage of a few paragraphs constrained to some fixed sentences. For baselines they use zero-shot standard and CoT prompts. While the former prompts the LLM to directly generate a coherent passage given input constraints, the latter prompts to first make a brief plan then write the passage, i.e. the plan serves as the intermediate thought step.
For ToT we will build a tree with depth 2 (and only 1 intermediate thought step). The LLM is prompted to generate k plans and vote for the most promising one, then similarly generate k passages based on the best plan to finally vote for the best one. A simple zero-shot vote prompt is used to sample j votes at both steps. The LLM will be tasked to write a customer press release.
# Amazon Local Gift Cards PR
title = "Amazon introduces Local Gift Cards"
start = "Gift cards are consistently among the most sought after gifts year after year."
mid = '''"Happy Anniversary"'''
end = '''"And we'll continue to work towards our vision of allowing our customers to find and purchase any gift card they need."'''
input_data = (title, start, mid, end)
3.2 Prompts and custom class to run the creative writing task#
For this exercise we will be using several prompts, that we keep in a separate file prompts.py.
# We load all prompts from a separate file prompts.py
from mlu_utils.prompts import (
standard_prompt_template,
cot_prompt_template,
plan_prompt_template,
vote_plan_prompt_template,
vote_passage_prompt_template,
score_coherency_prompt_template,
compare_prompt_template,
)
To facilitate running experiments, we have created a custom class that implements ToT and allows to generate outputs with standard, CoT, and ToT prompting techniques.
The code below implements all needed methods. You don’t need to understand every detail. Later in the notebook we will use the class step by step to run the ToT approach.
BASELINES = ["standard", "cot"]
VOTES = ["vote_plan", "vote_passage"]
PASSAGE_STR = "Passage:"
PLAN_STR = "Plan:"
TEMPLATES = {
"standard": standard_prompt_template,
"cot": cot_prompt_template,
"vote_plan": vote_plan_prompt_template,
"vote_passage": vote_passage_prompt_template,
}
class TextTaskToT:
"""
Class to run ToT for creative writing
"""
def __init__(self, input_data, model, temperature):
self.title, self.start, self.mid, self.end = input_data
self.model = model
self.temperature = temperature
self.votes_outputs = {}
self.best_choice = {}
def wrap_prompt(self, method, extra_args=None):
"""
Returns different prompts used in this task
Supported methods: standard, cot, plan, vote_plan, vote_passage
"""
if method in BASELINES:
prompt = PromptTemplate.from_template(TEMPLATES[method]).format(
title=self.title, start=self.start, mid=self.mid, end=self.end
)
if method == "plan":
prompt = PromptTemplate.from_template(plan_prompt_template).format(
plan=extra_args,
title=self.title,
start=self.start,
mid=self.mid,
end=self.end,
)
if method in VOTES:
enumerated_choices = ""
for i, choice in enumerate(extra_args, 1):
enumerated_choices += f"Choice {i}: \n{choice}\n\n"
prompt = PromptTemplate.from_template(TEMPLATES[method]).format(
enumerated_choices=enumerated_choices.strip()
)
return prompt
def generate_tot_thoughts(self, n_branches):
"""
Generate n_branches thoughts to start the tree of thoughts
"""
# CoT prompt to generate the branches for ToT
prompt = self.wrap_prompt("cot")
# Stop generating in "Passage:" since we want the plans only
thoughts = generate_outputs(
prompt,
self.model,
self.temperature,
stop_sequences=[PASSAGE_STR],
n=n_branches,
)
# Ensure that the thoughts are clean of "Passage:" and "Plan:" markers
thoughts = [self.sanitize_passage(t, PLAN_STR, position=1) for t in thoughts]
thoughts = [self.sanitize_passage(t, PASSAGE_STR, position=0) for t in thoughts]
return thoughts
def vote_tot_alternatives(self, method, choices, n_rounds_vote):
"""
Choose best alternative by voting (majority vote)
"""
# method is "vote_plan" to choose among thoughts or "vote_passage" to choose among passages
vote_prompt = self.wrap_prompt(method, choices)
# Store votes outputs for later inspection
self.votes_outputs[method] = generate_outputs(
vote_prompt, self.model, self.temperature, n=n_rounds_vote
)
votes = [self.extract_vote(vote_out) for vote_out in self.votes_outputs[method]]
votes_ctr = Counter(votes)
print(votes_ctr)
majority_vote = votes_ctr.most_common(1)[0][0]
if majority_vote is None:
majority_vote = 0
# Majority vote determines best choice
print(f"choices :: {choices}")
self.best_choice[method] = choices[majority_vote]
return self.best_choice[method], majority_vote
def make_tot_passage(self, n_branches, n_rounds_vote):
"""
Run ToT method end-to-end
"""
print(f"--- Generating {n_branches} plans...")
self.thoughts = self.generate_tot_thoughts(n_branches)
print(f"--- Voting for most promising plan in {n_rounds_vote} rounds...")
best_plan, best_plan_index = self.vote_tot_alternatives(
"vote_plan", self.thoughts, n_rounds_vote
)
print(f"--- Generating {n_branches} passages from most promising plan...")
prompt_plan = self.wrap_prompt("plan", best_plan)
passages = generate_outputs(
prompt_plan, self.model, self.temperature, n=n_branches
)
print(f"--- Voting for most coherent passage in {n_rounds_vote} rounds...")
best_passage, best_package_index = self.vote_tot_alternatives(
"vote_passage", passages, n_rounds_vote
)
best_passage = self.sanitize_passage(best_passage, PASSAGE_STR, position=1)
print("Done.\n")
return best_passage
def make_passages(self, method, n=1, n_branches=5, n_rounds_vote=3):
"""
Wrapper to generate passages with the three methods
"""
if method in BASELINES:
prompt = ttt.wrap_prompt(method)
passages = generate_outputs(prompt, self.model, self.temperature, n=n)
passages = [
self.sanitize_passage(p, PASSAGE_STR, position=1) for p in passages
]
if method == "tot":
passages = []
for i in range(n):
passages.append(self.make_tot_passage(n_branches, n_rounds_vote))
return passages
@staticmethod
def sanitize_passage(passage, string_to_clean, position):
"""
Util function to remove unwanted markers in text
"""
# position is 0 if we keep text before, 1 if we keep text after
if string_to_clean in passage:
passage = passage.split(string_to_clean)[position].strip()
return passage
@staticmethod
def extract_vote(vote_output: str) -> int:
"""
Parser to extract the best choice when the LLM is voting
"""
pattern = r".*best choice is (\d+).*"
match = re.match(pattern, vote_output, re.DOTALL)
vote = None
if match:
vote = int(match.groups()[0]) - 1
return vote
3.3 Baseline 1: Standard prompting#
We will use Mistral model hosted in Bedrock to generate passages. Let’s start by generating a passage with the standard prompt.
MODEL = "mistral.mistral-7b-instruct-v0:2"
TEMP = 0.75
ttt = TextTaskToT(input_data, MODEL, TEMP)
display(Markdown(ttt.wrap_prompt("standard")))
display(Markdown("---"))
standard_passage = ttt.make_passages("standard")[0]
display(Markdown(standard_passage))
Human: Write a passage of 4 paragraphs.
The first line must contain only this title: Amazon introduces Local Gift Cards
The first paragraph must start with sentence: Gift cards are consistently among the most sought after gifts year after year.
The second paragram must contain the words: “Happy Anniversary”
The last paragraph must end with sentence: “And we’ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need.”
Assistant:
Title: Amazon Introduces Local Gift Cards
Gift cards are consistently among the most sought after gifts year after year. They offer flexibility, convenience, and the ability to let the recipient choose something they truly desire. Amazon, the global e-commerce giant, has recognized this trend and has announced the introduction of Local Gift Cards. These cards, tied to specific regions or communities, will cater to the unique needs and preferences of local businesses and consumers.
As we celebrate “Happy Anniversary” of another successful year in business, Amazon is expanding its horizons. The new Local Gift Cards initiative is a testament to Amazon’s commitment to serving its diverse customer base. By offering these locally-focused gift cards, Amazon aims to bridge the gap between online shopping and local businesses. It’s a win-win situation: customers get to enjoy the benefits of shopping on Amazon, while local businesses gain a new platform to reach a wider audience.
The Local Gift Cards will be available for a variety of local businesses, from grocery stores to boutiques, restaurants to service providers. These cards can be purchased online through Amazon, making the process of buying and gifting as seamless as ever. Once purchased, the cards can be sent directly to the recipient, or they can be used by the buyer themselves. This not only makes gift-giving more convenient but also supports local businesses, contributing to the community’s economic growth.
As we look towards the future, Amazon’s vision remains clear: to be the go-to platform for all your shopping needs. And we’ll continue to work towards this vision. With the introduction of Local Gift Cards, Amazon is not just expanding its product offerings but also strengthening its commitment to local communities. We’re excited to see how this initiative will benefit both our customers and local businesses, and we’ll keep pushing the boundaries to make shopping more convenient, more personal, and more rewarding for everyone.
3.4 Baseline 2: Chain-of-thought prompting#
Let’s continue with a CoT prompt that asks the LLM to first produce a plan (“thought”) before writing the passage.
display(Markdown(ttt.wrap_prompt("cot")))
display(Markdown("---"))
cot_passage = ttt.make_passages("cot")[0]
display(Markdown(cot_passage))
Human: Write a coherent passage of 4 paragraphs.
The first line must contain only this title: Amazon introduces Local Gift Cards
The first paragraph must start with sentence: Gift cards are consistently among the most sought after gifts year after year.
The second paragram must contain the words: “Happy Anniversary”
The last paragraph must end with sentence: “And we’ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need.”
Make a plan, then write the 4 paragraphs according to the plan.
Your output should have the following format.
Plan: Your plan here.
Passage: Your passage here.
Assistant:
Gift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something they truly desire. Amazon, the global e-commerce leader, has recognized this trend and is taking it a step further.
On the occasion of our “Happy Anniversary”, we are excited to introduce Amazon’s new offering - “Local Gift Cards”. These gift cards are not just limited to popular brands or stores. Instead, they represent a wide array of local businesses, making them a unique and thoughtful gift option.
The benefits of “Local Gift Cards” are manifold. For those who appreciate the charm of local businesses and the unique products they offer, these gift cards provide an opportunity to support their favorite establishments, even from the comfort of their homes. For businesses, this partnership with Amazon can lead to increased visibility and sales.
At Amazon, we believe in the power of choice. And we’ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need, be it for a local bakery, a favorite clothing store, or a renowned tech brand. So, whether it’s a birthday, a graduation, or just because, Amazon’s “Local Gift Cards” are here to make your gifting experience more personal and diverse.
3.5 Tree-of-thought prompting#
Finally let’s use Tree of Thoughts to improve on this passage generation. The system will undergo the following steps:
produce
kplans as “thoughts”vote for the most promising one
generate
npassages using the most promising thoughtvote for the most coherent passage
Let’s show a step by step example using k=3 branches for the number of thoughts.
First we generate the 3 thoughts.
We then ask the model to vote for the most promising using a simple vote prompt (see
vote_plan_prompt_templateinprompts.py).Next we generate multiple passages from the best plan.
Finally the system is again tasked with voting for the “best” passage, understood as the most coherent.
The following cell takes 1-2 minutes to run.
# Generate 3 thoughts
thoughts = ttt.generate_tot_thoughts(n_branches=3)
# Vote for the best plan
best_plan, best_plan_index = ttt.vote_tot_alternatives("vote_plan", thoughts, n_rounds_vote=3)
# Generate passages with the best plan
prompt_plan = ttt.wrap_prompt("plan", best_plan)
candidate_passages = generate_outputs(prompt_plan, ttt.model, ttt.temperature, n=3)
# Vote for the best package
best_passage, best_passage_index = ttt.vote_tot_alternatives(
"vote_passage", candidate_passages, n_rounds_vote=3
)
Markdown(best_passage)
Counter({None: 3})
choices :: ['Amazon introduces Local Gift Cards - Gift cards continue to be popular gifts - Discussing the "Happy Anniversary" aspect - Conclusion on Amazon\'s vision', 'Title: Amazon introduces Local Gift Cards\nParagraph 1: Discuss the popularity of gift cards.\nParagraph 2: Introduce Amazon\'s new "Local Gift Cards" feature, relating it to a specific occasion like "Happy Anniversary".\nParagraph 3: Describe the benefits and convenience of the new feature.\nParagraph 4: Conclude with a statement about Amazon\'s ongoing commitment to expand their gift card offerings.', 'Amazon introduces Local Gift Cards - This is the title of the passage.\nGift cards are consistently among the most sought after gifts year after year. In recognition of this trend, Amazon has announced a new feature.\n(Paragraph 1)\n\nHappy Anniversary to another successful year of gifting! Amazon is delighted to introduce a new addition to its array of offerings - Local Gift Cards.\n(Paragraph 2)\n\nThese Local Gift Cards are a thoughtful way to celebrate special occasions such as "Happy Anniversaries" or to express gratitude and appreciation to loved ones and colleagues. They come in various denominations and can be used at a wide range of local businesses.\n\nAs we move forward, Amazon is committed to expanding this offering. We understand the importance of supporting local businesses and communities. With Local Gift Cards, we aim to make it easier for our customers to contribute to their local economy while also providing them with a diverse range of gifting options. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need, be it local or global, for any occasion.\n(Paragraph 3 and 4)']
Counter({0: 2, 1: 1})
choices :: ['\n\nPlan:\nAmazon introduces Local Gift Cards - Gift cards continue to be popular gifts - Discussing the "Happy Anniversary" aspect - Conclusion on Amazon\'s vision.\n\nPassage:\n\nGift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something they truly love. Amazon, the global e-commerce giant, has recognized this trend and has recently introduced a new feature - Local Gift Cards. These are digital gift cards that can be used at specific local businesses or chains, expanding Amazon\'s gifting options significantly.\n\nThe joy of giving and receiving gifts is universally acknowledged. One special occasion that holds a significant place in this context is "Happy Anniversary." Anniversaries are milestones in a relationship, marking the passage of time and the strengthening of bonds. Traditional anniversary gifts like paper, cotton, or gold have their charm, but the convenience and adaptability of a gift card make it a popular choice for many. With Amazon\'s Local Gift Cards, you can now choose to celebrate this special day with a gift that is both practical and thoughtful.\n\nAmazon\'s introduction of Local Gift Cards is a testament to their customer-centric approach. They understand that every customer has unique needs and preferences. By offering a diverse range of gift cards, they aim to cater to a wider audience. Furthermore, the availability of Local Gift Cards adds a personal touch to gifting, allowing you to support local businesses while making your loved ones feel special on their anniversary.\n\nAnd we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need. Whether it\'s for a birthday, a wedding, a graduation, or an anniversary, Amazon is committed to providing a vast selection of gift cards from a wide range of local and global brands. This not only simplifies the gifting process but also ensures that you always have the perfect gift at your fingertips.', '\n\nPlan:\nAmazon introduces Local Gift Cards - Gift cards continue to be popular gifts - Discussing the "Happy Anniversary" aspect - Conclusion on Amazon\'s vision.\n\nPassage:\n\nGift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something that truly suits their taste and preferences. Recently, Amazon, the global leader in e-commerce, has introduced a new addition to its extensive gift card offerings - Local Gift Cards.\n\nThe "Happy Anniversary" aspect of gift-giving is a significant one. It\'s a time to celebrate the milestones in a relationship, be it personal or professional. Anniversary gifts are often symbolic of the journey so far and a testament to the commitment and love shared between two people. Amazon\'s Local Gift Cards aim to make this special occasion even more memorable. These gift cards can be redeemed at local businesses, adding a personal touch to the gift. Whether it\'s a favorite restaurant, a local boutique, or a popular home improvement store, Amazon\'s Local Gift Cards provide the flexibility to choose a gift that resonates deeply with the recipient.\n\nAmazon\'s introduction of Local Gift Cards is a testament to its customer-centric approach. By offering a wide range of gift cards, Amazon is catering to the diverse needs and preferences of its customers. The ability to purchase Local Gift Cards online, along with the convenience of having them delivered directly to the recipient, makes the gift-giving process seamless and hassle-free. This is particularly beneficial during special occasions like anniversaries, when time and convenience are crucial.\n\nAnd we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need. Whether it\'s a local business or a global brand, Amazon is committed to being your one-stop shop for all your gifting needs. So, whether it\'s a "Happy Anniversary" gift, a birthday present, or a token of appreciation, Amazon\'s Local Gift Cards offer a convenient and thoughtful solution.', '\n\nPlan:\nAmazon introduces Local Gift Cards - Gift cards continue to be popular gifts - Discussing the "Happy Anniversary" aspect - Conclusion on Amazon\'s vision.\n\nPassage:\n\nGift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something that truly suits their tastes and needs. Recently, Amazon has introduced a new addition to its gift card offerings - Local Gift Cards. These cards can be used at specific local businesses, adding a unique and personal touch to the gift.\n\nThe "Happy Anniversary" aspect of gift giving is a significant one. It\'s a time to celebrate the milestones in a relationship, whether it\'s a marriage anniversary, a work anniversary, or the anniversary of a friendship. Local Gift Cards from Amazon offer an excellent solution for this. Imagine being able to give your loved one an anniversary gift that not only shows your thoughtfulness but also supports their favorite local business. It\'s a win-win situation.\n\nAmazon\'s vision is to be the go-to destination for all things shopping. This includes gift cards. By introducing Local Gift Cards, Amazon is taking a step closer to realizing this vision. It\'s not just about selling products anymore; it\'s about providing solutions that make gifting easier and more personal. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need, be it a national chain or a local business.\n\nPassage:\n\nGift cards are a staple in modern gift-giving, with their versatility and convenience making them a popular choice. Amazon\'s latest innovation in this realm is the introduction of Local Gift Cards. These cards, which can be used at specific local businesses, add a new dimension to the gift-giving experience.\n\nThe significance of the "Happy Anniversary" occasion in gift giving cannot be overstated. It\'s a time to commemorate important milestones in relationships, and a thoughtful, personal gift is a great way to celebrate. Amazon\'s Local Gift Cards offer a unique solution for this. Imagine the delight on your loved one\'s face when they receive an anniversary gift that not only shows your thoughtfulness but also supports their favorite local business.\n\nAmazon\'s vision is to be the one-stop shop for all your shopping needs. With the introduction of Local Gift Cards, they\'re inching closer to this goal. It\'s not just about selling products anymore; it\'s about providing solutions that make gifting easier and more personal. And Amazon\'s commitment to this vision is unwavering. They\'ll continue to work towards allowing their customers to find and purchase any gift card they need, be it for a national chain or a local business.\n\nPassage:\n\nGift cards have become a staple in modern gift-giving, offering the flexibility and convenience that makes them a popular choice. Amazon\'s latest offering in this sphere is the introduction of Local Gift Cards. These cards, which can be used at specific local businesses, add a new dimension to the gift-giving experience.\n\nThe "Happy Anniversary" milestone in relationships is a time for celebration. A thoughtful, personal gift is a great way to mark this occasion. Amazon\'s Local Gift Cards offer a unique solution for this. Imagine the joy on your loved one\'s face when they receive an anniversary gift that not only shows your thoughtfulness but also supports their favorite local business.\n\nAmazon\'s vision is to be the go-to destination for all your shopping needs. With the introduction of Local Gift Cards, they\'re taking a significant step towards this goal. It\'s not just about selling products anymore; it\'s about providing solutions that make gifting easier and more personal. And Amazon\'s commitment to this vision is unwavering. They\'ll continue to work towards allowing their customers to find and purchase any gift card they need, be it for a national chain or a local business.']
Plan: Amazon introduces Local Gift Cards - Gift cards continue to be popular gifts - Discussing the “Happy Anniversary” aspect - Conclusion on Amazon’s vision.
Passage:
Gift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something they truly love. Amazon, the global e-commerce giant, has recognized this trend and has recently introduced a new feature - Local Gift Cards. These are digital gift cards that can be used at specific local businesses or chains, expanding Amazon’s gifting options significantly.
The joy of giving and receiving gifts is universally acknowledged. One special occasion that holds a significant place in this context is “Happy Anniversary.” Anniversaries are milestones in a relationship, marking the passage of time and the strengthening of bonds. Traditional anniversary gifts like paper, cotton, or gold have their charm, but the convenience and adaptability of a gift card make it a popular choice for many. With Amazon’s Local Gift Cards, you can now choose to celebrate this special day with a gift that is both practical and thoughtful.
Amazon’s introduction of Local Gift Cards is a testament to their customer-centric approach. They understand that every customer has unique needs and preferences. By offering a diverse range of gift cards, they aim to cater to a wider audience. Furthermore, the availability of Local Gift Cards adds a personal touch to gifting, allowing you to support local businesses while making your loved ones feel special on their anniversary.
And we’ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need. Whether it’s for a birthday, a wedding, a graduation, or an anniversary, Amazon is committed to providing a vast selection of gift cards from a wide range of local and global brands. This not only simplifies the gifting process but also ensures that you always have the perfect gift at your fingertips.
candidate_passages
['\n\nPlan:\nAmazon introduces Local Gift Cards - Gift cards continue to be popular gifts - Discussing the "Happy Anniversary" aspect - Conclusion on Amazon\'s vision.\n\nPassage:\n\nGift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something they truly love. Amazon, the global e-commerce giant, has recognized this trend and has recently introduced a new feature - Local Gift Cards. These are digital gift cards that can be used at specific local businesses or chains, expanding Amazon\'s gifting options significantly.\n\nThe joy of giving and receiving gifts is universally acknowledged. One special occasion that holds a significant place in this context is "Happy Anniversary." Anniversaries are milestones in a relationship, marking the passage of time and the strengthening of bonds. Traditional anniversary gifts like paper, cotton, or gold have their charm, but the convenience and adaptability of a gift card make it a popular choice for many. With Amazon\'s Local Gift Cards, you can now choose to celebrate this special day with a gift that is both practical and thoughtful.\n\nAmazon\'s introduction of Local Gift Cards is a testament to their customer-centric approach. They understand that every customer has unique needs and preferences. By offering a diverse range of gift cards, they aim to cater to a wider audience. Furthermore, the availability of Local Gift Cards adds a personal touch to gifting, allowing you to support local businesses while making your loved ones feel special on their anniversary.\n\nAnd we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need. Whether it\'s for a birthday, a wedding, a graduation, or an anniversary, Amazon is committed to providing a vast selection of gift cards from a wide range of local and global brands. This not only simplifies the gifting process but also ensures that you always have the perfect gift at your fingertips.',
'\n\nPlan:\nAmazon introduces Local Gift Cards - Gift cards continue to be popular gifts - Discussing the "Happy Anniversary" aspect - Conclusion on Amazon\'s vision.\n\nPassage:\n\nGift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something that truly suits their taste and preferences. Recently, Amazon, the global leader in e-commerce, has introduced a new addition to its extensive gift card offerings - Local Gift Cards.\n\nThe "Happy Anniversary" aspect of gift-giving is a significant one. It\'s a time to celebrate the milestones in a relationship, be it personal or professional. Anniversary gifts are often symbolic of the journey so far and a testament to the commitment and love shared between two people. Amazon\'s Local Gift Cards aim to make this special occasion even more memorable. These gift cards can be redeemed at local businesses, adding a personal touch to the gift. Whether it\'s a favorite restaurant, a local boutique, or a popular home improvement store, Amazon\'s Local Gift Cards provide the flexibility to choose a gift that resonates deeply with the recipient.\n\nAmazon\'s introduction of Local Gift Cards is a testament to its customer-centric approach. By offering a wide range of gift cards, Amazon is catering to the diverse needs and preferences of its customers. The ability to purchase Local Gift Cards online, along with the convenience of having them delivered directly to the recipient, makes the gift-giving process seamless and hassle-free. This is particularly beneficial during special occasions like anniversaries, when time and convenience are crucial.\n\nAnd we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need. Whether it\'s a local business or a global brand, Amazon is committed to being your one-stop shop for all your gifting needs. So, whether it\'s a "Happy Anniversary" gift, a birthday present, or a token of appreciation, Amazon\'s Local Gift Cards offer a convenient and thoughtful solution.',
'\n\nPlan:\nAmazon introduces Local Gift Cards - Gift cards continue to be popular gifts - Discussing the "Happy Anniversary" aspect - Conclusion on Amazon\'s vision.\n\nPassage:\n\nGift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something that truly suits their tastes and needs. Recently, Amazon has introduced a new addition to its gift card offerings - Local Gift Cards. These cards can be used at specific local businesses, adding a unique and personal touch to the gift.\n\nThe "Happy Anniversary" aspect of gift giving is a significant one. It\'s a time to celebrate the milestones in a relationship, whether it\'s a marriage anniversary, a work anniversary, or the anniversary of a friendship. Local Gift Cards from Amazon offer an excellent solution for this. Imagine being able to give your loved one an anniversary gift that not only shows your thoughtfulness but also supports their favorite local business. It\'s a win-win situation.\n\nAmazon\'s vision is to be the go-to destination for all things shopping. This includes gift cards. By introducing Local Gift Cards, Amazon is taking a step closer to realizing this vision. It\'s not just about selling products anymore; it\'s about providing solutions that make gifting easier and more personal. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need, be it a national chain or a local business.\n\nPassage:\n\nGift cards are a staple in modern gift-giving, with their versatility and convenience making them a popular choice. Amazon\'s latest innovation in this realm is the introduction of Local Gift Cards. These cards, which can be used at specific local businesses, add a new dimension to the gift-giving experience.\n\nThe significance of the "Happy Anniversary" occasion in gift giving cannot be overstated. It\'s a time to commemorate important milestones in relationships, and a thoughtful, personal gift is a great way to celebrate. Amazon\'s Local Gift Cards offer a unique solution for this. Imagine the delight on your loved one\'s face when they receive an anniversary gift that not only shows your thoughtfulness but also supports their favorite local business.\n\nAmazon\'s vision is to be the one-stop shop for all your shopping needs. With the introduction of Local Gift Cards, they\'re inching closer to this goal. It\'s not just about selling products anymore; it\'s about providing solutions that make gifting easier and more personal. And Amazon\'s commitment to this vision is unwavering. They\'ll continue to work towards allowing their customers to find and purchase any gift card they need, be it for a national chain or a local business.\n\nPassage:\n\nGift cards have become a staple in modern gift-giving, offering the flexibility and convenience that makes them a popular choice. Amazon\'s latest offering in this sphere is the introduction of Local Gift Cards. These cards, which can be used at specific local businesses, add a new dimension to the gift-giving experience.\n\nThe "Happy Anniversary" milestone in relationships is a time for celebration. A thoughtful, personal gift is a great way to mark this occasion. Amazon\'s Local Gift Cards offer a unique solution for this. Imagine the joy on your loved one\'s face when they receive an anniversary gift that not only shows your thoughtfulness but also supports their favorite local business.\n\nAmazon\'s vision is to be the go-to destination for all your shopping needs. With the introduction of Local Gift Cards, they\'re taking a significant step towards this goal. It\'s not just about selling products anymore; it\'s about providing solutions that make gifting easier and more personal. And Amazon\'s commitment to this vision is unwavering. They\'ll continue to work towards allowing their customers to find and purchase any gift card they need, be it for a national chain or a local business.']
Visualize ToT logic#
Let’s plot a tree to help us visualize the steps followed by the ToT method. We will use igraph and plotly for that.
You don’t need to understand every detail of the plotting code. Here’s some plotly documentation if you’re interested in using the library yourself.
def flatten(matrix):
""" Function to flatten a list of lists """
return [item for row in matrix for item in row]
def wrap_text(text, width=40):
""" Function to wrap text into a fixed size box for display """
text = "\n".join(
[
"\n".join(
textwrap.wrap(
line, width, break_long_words=False, replace_whitespace=False
)
)
for line in text.splitlines()
if line.strip() != ""
]
)
return text.replace("\n", "<br>")
# Hard-coded to this example with k=3 thoughts
nr_vertices = 7
best_plan_index_tree = 1 + best_plan_index
best_passage_index_tree = 4 + best_passage_index
edges = [(0, 1), (0, 2), (0, 3), (best_plan_index_tree, 4), (best_plan_index_tree, 5), (best_plan_index_tree, 6)]
G = Graph(edges=edges)
lay = G.layout('rt', root=[0])
# Coordinates of nodes and edges
Xn, Yn = zip(*lay.coords)
Yn = [-y for y in Yn]
Xe = flatten([(lay.coords[edge[0]][0], lay.coords[edge[1]][0], None) for edge in edges])
Ye = flatten([(-lay.coords[edge[0]][1], -lay.coords[edge[1]][1], None) for edge in edges])
# Put the proper labels into the nodes of the tree
labels = [
"Tree-of-Thoughts root",
wrap_text(thoughts[0]),
wrap_text(thoughts[1]),
wrap_text(thoughts[2]),
wrap_text(candidate_passages[0].split("\nPlan")[1], width=80),
wrap_text(candidate_passages[1].split("\nPlan")[1], width=80),
wrap_text(candidate_passages[2].split("\nPlan")[1], width=80)
]
# Assemble everything and plot
lo = go.Layout(
autosize=False, width=1200, height=400,
xaxis=go.layout.XAxis(linecolor="white", linewidth=1, mirror=True, showgrid=False, showticklabels=False),
yaxis=go.layout.YAxis(linecolor="white", linewidth=1, mirror=True, showgrid=False, showticklabels=False),
margin=go.layout.Margin(l=0, r=0, b=0, t=50, pad=0),
)
fig = go.Figure(layout=lo)
fig.update_layout(plot_bgcolor="white", showlegend=False, title="Tree-of-thoughts with k=3 and depth=2")
# Add edges
fig.add_trace(
go.Scatter(x=Xe, y=Ye, mode="lines", line=dict(color="lightgray", width=2), hoverinfo="none",)
)
# Add nodes
fig.add_trace(
go.Scatter(x=Xn, y=Yn, mode="markers", name="",
marker=dict(
symbol="circle-dot",
size=18,
color="royalblue",
line=dict(color="royalblue", width=1),
),
text=labels, hoverinfo="text", opacity=0.8,
)
)
# Update with best alternatives
bestXn = [Xn[best_plan_index_tree], Xn[best_passage_index_tree]]
bestYn = [Yn[best_plan_index_tree], Yn[best_passage_index_tree]]
bestlabels = [labels[best_plan_index_tree], labels[best_passage_index_tree]]
fig.add_trace(
go.Scatter(x=bestXn, y=bestYn, mode="markers", name="",
marker=dict(
symbol="circle-dot",
size=18,
color="seagreen",
line=dict(color="seagreen", width=1),
),
text=bestlabels, hoverinfo="text", opacity=0.8,
)
)
Try it Yourself!#
You can inspect the reasons why the system voted for one or the other thoughts and passages.
Try looking into ttt.votes_outputs. You will find keys vote_plan and vote_passage containing the analysis that yielded the best options. Majority vote decided the output. Do you agree with the decision made by the system?
############## CODE HERE ####################
############## END OF CODE ##################
Run ToT end-to-end#
We can now generate a ToT passage using the wrapper function. We will use k=5 for the number of thoughts and j=3 for the number of times we ask the LLM to vote in both levels of the tree.
try:
tot_passage = ttt.make_passages("tot", n_branches=5, n_rounds_vote=3)[0]
display(Markdown(tot_passage))
except ClientError as e:
error_code = e.response['Error']['Code']
# Handle throttling errors
if error_code in ['ThrottlingException']:
#raise Exception(f"ThrottlingException. Wait some minutes and try again: {str(e)}")
print(f"\n\nThrottlingException. Wait some minutes and try again.")
else:
# Re-raise non-throttling errors
raise
--- Generating 5 plans...
--- Voting for most promising plan in 3 rounds...
Counter({None: 3})
choices :: ['Amazon introduces Local Gift Cards - This is the title of the passage.\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature.\n(Paragraph 1)\nIn celebration of a "Happy Anniversary" milestone, Amazon has announced the availability of Local Gift Cards.\n(Paragraph 2)\nThese Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative aims to strengthen the bond between Amazon and local businesses, as well as provide customers with more diverse gift options.\n(Paragraph 3)\nAmazon\'s commitment to customer satisfaction has led them to expand their gift card offerings. By introducing Local Gift Cards, Amazon is now able to cater to a wider range of customer preferences, making it easier for customers to find the perfect gift for any occasion.\n(Paragraph 4)\nAs we move forward, Amazon plans to expand the selection of Local Gift Cards to cover more areas and businesses. We believe that this feature will be well-received by our customers, and we\'re excited to continue this journey towards making shopping more convenient and personalized. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need.\n(Paragraph 5)\nIn conclusion, Amazon\'s introduction of Local Gift Cards is a significant step towards enhancing the shopping experience for customers. By offering Local Gift Cards, Amazon is not only catering to the growing demand for diverse gift options but also strengthening its relationship with local businesses. We look forward to the continued growth and success of this initiative.', 'Amazon introduces Local Gift Cards - This title sets the context for the passage.\nGift cards are consistently among the most sought after gifts year after year. - The first paragraph establishes the relevance of gift cards.\nDiscuss the introduction of "Happy Anniversary" Local Gift Cards. - The second paragraph focuses on a specific type of local gift card.\nExplain the benefits of these local gift cards and Amazon\'s future plans. - The third paragraph discusses the advantages and Amazon\'s intentions.\nConclude with a statement about continuing efforts to expand gift card offerings. - The fourth paragraph reiterates Amazon\'s commitment.', 'Amazon introduces Local Gift Cards - This is the title of the passage.\nGift cards are consistently among the most sought after gifts year after year. In recognition of this fact, Amazon has announced a new feature.\nParagraph 2: Amazon introduces Local Gift Cards for special occasions like "Happy Anniversaries" and other milestones.\nParagraph 3: This new feature, Amazon Local Gift Cards, allows customers to purchase gift cards tailored to specific local businesses. This not only makes gift-giving more personal but also supports local economies.\nParagraph 4: As we move forward, Amazon is committed to expanding its range of gift card offerings. With "Happy Anniversaries" being just the beginning, we\'re excited to explore how this feature can be used for other special occasions and events. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need.', 'Amazon introduces Local Gift Cards - This title will start the first line of the first paragraph. The second paragraph will be about the popularity of gift cards and the significance of "Happy Anniversary". The third paragraph will discuss the new Local Gift Cards offering. The fourth paragraph will conclude with a forward-looking statement.', 'Title: Amazon introduces Local Gift Cards\nParagraph 1: Discuss the popularity of gift cards.\nParagraph 2: Introduce the concept of Local Gift Cards with a "Happy Anniversary" example.\nParagraph 3: Explain the benefits and convenience of Local Gift Cards.\nParagraph 4: Conclude with future plans and vision.']
--- Generating 5 passages from most promising plan...
--- Voting for most coherent passage in 3 rounds...
Counter({None: 2, 2: 1})
choices :: ['\n\nPlan:\nAmazon introduces Local Gift Cards - This is the title of the passage.\n\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature.\n\nPassage:\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature. In celebration of a "Happy Anniversary" milestone, Amazon has announced the availability of Local Gift Cards. These Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative aims to strengthen the bond between Amazon and local businesses, as well as provide customers with more diverse gift options.\n\nParagraph 1:\nAmazon, the global e-commerce giant, is marking a significant milestone with the announcement of a new feature - Local Gift Cards. This move comes in response to the ever-growing popularity of gift cards as a preferred gift option.\n\nParagraph 2:\nIn a bid to celebrate their "Happy Anniversary," Amazon has introduced Local Gift Cards. These cards are tailored to cater to specific regions or communities, allowing customers to support local businesses, restaurants, or services. By offering Local Gift Cards, Amazon is not only enhancing its relationship with local businesses but also providing customers with a wider range of gift options that resonate with their community.\n\nParagraph 3:\nAmazon\'s commitment to customer satisfaction has led them to expand their gift card offerings. With the introduction of Local Gift Cards, Amazon is now able to cater to a wider range of customer preferences. This feature allows customers to find the perfect gift for any occasion, making their shopping experience more personalized and convenient.\n\nParagraph 4:\nAs we move forward, Amazon plans to expand the selection of Local Gift Cards to cover more areas and businesses. We believe that this feature will be well-received by our customers, and we\'re excited to continue this journey towards making shopping more convenient and personalized. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need, be it a Local Gift Card or a gift card from a major retailer.\n\nParagraph 5:\nIn conclusion, Amazon\'s introduction of Local Gift Cards is a significant step towards enhancing the shopping experience for customers. By offering Local Gift Cards, Amazon is not only catering to the growing demand for diverse gift options but also strengthening its relationship with local businesses. We look forward to the continued growth and success of this initiative, as we strive to make every shopping experience a memorable one.', '\n\nPlan:\nAmazon introduces Local Gift Cards - This is the title of the passage.\n\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature.\n\nPassage:\nAmazon introduces Local Gift Cards\n\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its commitment to customer satisfaction, has made an exciting announcement. In celebration of a "Happy Anniversary" milestone, Amazon has announced the availability of Local Gift Cards. These Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative aims to strengthen the bond between Amazon and local businesses, as well as provide customers with more diverse gift options.\n\nPassage:\nAmazon\'s Local Gift Cards - A New Way to Celebrate\n\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature to celebrate a "Happy Anniversary" milestone. The e-commerce giant has announced the availability of Local Gift Cards. These Local Gift Cards are a thoughtful addition to Amazon\'s gift card offerings, designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative not only enhances the shopping experience for customers but also strengthens Amazon\'s relationship with local businesses.\n\nPassage:\nAmazon\'s Local Gift Cards: A Win-Win Initiative\n\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its commitment to customer satisfaction, has taken a significant step forward. In celebration of a "Happy Anniversary" milestone, Amazon has announced the availability of Local Gift Cards. These Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative not only provides customers with more diverse gift options but also strengthens Amazon\'s relationship with local businesses. Amazon plans to expand the selection of Local Gift Cards to cover more areas and businesses in the future. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need.', '\n\nPlan:\nAmazon introduces Local Gift Cards - This is the title of the passage.\n\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature.\n\nPassage:\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature. In celebration of a "Happy Anniversary" milestone, Amazon has announced the availability of Local Gift Cards. These Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative aims to strengthen the bond between Amazon and local businesses, as well as provide customers with more diverse gift options. Amazon\'s commitment to customer satisfaction has led them to expand their gift card offerings. By introducing Local Gift Cards, Amazon is now able to cater to a wider range of customer preferences, making it easier for customers to find the perfect gift for any occasion. As we move forward, Amazon plans to expand the selection of Local Gift Cards to cover more areas and businesses. We believe that this feature will be well-received by our customers, and we\'re excited to continue this journey towards making shopping more convenient and personalized. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need.\n\nIn conclusion, Amazon\'s introduction of Local Gift Cards is a significant step towards enhancing the shopping experience for customers. By offering Local Gift Cards, Amazon is not only catering to the growing demand for diverse gift options but also strengthening its relationship with local businesses. We look forward to the continued growth and success of this initiative.', '\n\nPlan:\nAmazon introduces Local Gift Cards - This is the title of the passage.\n\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature.\n\nPassage:\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature - Amazon introduces Local Gift Cards. In celebration of a "Happy Anniversary" milestone, Amazon has announced the availability of Local Gift Cards. These Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative aims to strengthen the bond between Amazon and local businesses, as well as provide customers with more diverse gift options.\n\nPassage:\nAmazon\'s commitment to customer satisfaction has led them to expand their gift card offerings. By introducing Local Gift Cards, Amazon is now able to cater to a wider range of customer preferences. These cards are a thoughtful and practical gift for any occasion, especially during the "Happy Anniversary" milestones or other special events. They add a personal touch to the gift-giving experience and allow customers to support their local businesses.\n\nPassage:\nAs we move forward, Amazon plans to expand the selection of Local Gift Cards to cover more areas and businesses. We believe that this feature will be well-received by our customers, and we\'re excited to continue this journey towards making shopping more convenient and personalized. By offering Local Gift Cards, Amazon is not only catering to the growing demand for diverse gift options but also strengthening its relationship with local businesses. In conclusion, Amazon\'s introduction of Local Gift Cards is a significant step towards enhancing the shopping experience for customers. We look forward to the continued growth and success of this initiative. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need.', '\n\nPlan:\nAmazon introduces Local Gift Cards - This is the title of the passage.\n\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature.\n\nPassage:\nGift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature - Amazon introduces Local Gift Cards. In celebration of a "Happy Anniversary" milestone, Amazon has announced the availability of Local Gift Cards. These Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative aims to strengthen the bond between Amazon and local businesses, as well as provide customers with more diverse gift options. Amazon\'s commitment to customer satisfaction has led them to expand their gift card offerings. By introducing Local Gift Cards, Amazon is now able to cater to a wider range of customer preferences, making it easier for customers to find the perfect gift for any occasion. As we move forward, Amazon plans to expand the selection of Local Gift Cards to cover more areas and businesses. We believe that this feature will be well-received by our customers, and we\'re excited to continue this journey towards making shopping more convenient and personalized. In conclusion, Amazon\'s introduction of Local Gift Cards is a significant step towards enhancing the shopping experience for customers. By offering Local Gift Cards, Amazon is not only catering to the growing demand for diverse gift options but also strengthening its relationship with local businesses. We look forward to the continued growth and success of this initiative. And we\'ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need.']
Done.
Gift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature. In celebration of a “Happy Anniversary” milestone, Amazon has announced the availability of Local Gift Cards. These Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative aims to strengthen the bond between Amazon and local businesses, as well as provide customers with more diverse gift options.
Paragraph 1: Amazon, the global e-commerce giant, is marking a significant milestone with the announcement of a new feature - Local Gift Cards. This move comes in response to the ever-growing popularity of gift cards as a preferred gift option.
Paragraph 2: In a bid to celebrate their “Happy Anniversary,” Amazon has introduced Local Gift Cards. These cards are tailored to cater to specific regions or communities, allowing customers to support local businesses, restaurants, or services. By offering Local Gift Cards, Amazon is not only enhancing its relationship with local businesses but also providing customers with a wider range of gift options that resonate with their community.
Paragraph 3: Amazon’s commitment to customer satisfaction has led them to expand their gift card offerings. With the introduction of Local Gift Cards, Amazon is now able to cater to a wider range of customer preferences. This feature allows customers to find the perfect gift for any occasion, making their shopping experience more personalized and convenient.
Paragraph 4: As we move forward, Amazon plans to expand the selection of Local Gift Cards to cover more areas and businesses. We believe that this feature will be well-received by our customers, and we’re excited to continue this journey towards making shopping more convenient and personalized. And we’ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need, be it a Local Gift Card or a gift card from a major retailer.
Paragraph 5: In conclusion, Amazon’s introduction of Local Gift Cards is a significant step towards enhancing the shopping experience for customers. By offering Local Gift Cards, Amazon is not only catering to the growing demand for diverse gift options but also strengthening its relationship with local businesses. We look forward to the continued growth and success of this initiative, as we strive to make every shopping experience a memorable one.
4. Evaluation of results#
A key aspect of LLM usage is the evaluation of its results. In this example we will use a second LLM, Amazon Nova Lite model, to assess the coherency of pairs of passages. Following Yao et al. we implement two methods:
ask the evaluator LLM to provide coherency score from 1 to 10
ask the evaluator LLM to compare two passages and assess which one is more coherent.
To faciliate experimentation, we provide a class for evaluation below that implements both methods.
class EvaluateToT:
"""
Class to evaluate ToT against baselines
"""
def __init__(self, model, temperature):
self.model = model
self.temperature = temperature
def compare_passages(self, passage1, passage2, n_rounds=1):
"""
Ask LLM to compare passages and decide which one is more coherent
"""
compare_prompt = PromptTemplate.from_template(compare_prompt_template).format(
passage1=passage1, passage2=passage2
)
comparisons = generate_outputs(
compare_prompt, self.model, self.temperature, n=n_rounds
)
return comparisons
def score_passages(self, passage, n_rounds=1):
"""
Ask LLM to score the coherency of a passage from 1 to 10
"""
score_coherency_prompt = PromptTemplate.from_template(
score_coherency_prompt_template
).format(passage=passage)
scoring = generate_outputs(
score_coherency_prompt, self.model, self.temperature, n=n_rounds
)
return scoring
Below are the passages generated by the three methods:
passages = {"standard": standard_passage, "cot": cot_passage, "tot": tot_passage}
for method, passage in passages.items():
display(Markdown(f"### Passage with {method} prompting:"))
display(Markdown(passage))
display(Markdown("---"))
Passage with standard prompting:
Title: Amazon Introduces Local Gift Cards
Gift cards are consistently among the most sought after gifts year after year. They offer flexibility, convenience, and the ability to let the recipient choose something they truly desire. Amazon, the global e-commerce giant, has recognized this trend and has announced the introduction of Local Gift Cards. These cards, tied to specific regions or communities, will cater to the unique needs and preferences of local businesses and consumers.
As we celebrate “Happy Anniversary” of another successful year in business, Amazon is expanding its horizons. The new Local Gift Cards initiative is a testament to Amazon’s commitment to serving its diverse customer base. By offering these locally-focused gift cards, Amazon aims to bridge the gap between online shopping and local businesses. It’s a win-win situation: customers get to enjoy the benefits of shopping on Amazon, while local businesses gain a new platform to reach a wider audience.
The Local Gift Cards will be available for a variety of local businesses, from grocery stores to boutiques, restaurants to service providers. These cards can be purchased online through Amazon, making the process of buying and gifting as seamless as ever. Once purchased, the cards can be sent directly to the recipient, or they can be used by the buyer themselves. This not only makes gift-giving more convenient but also supports local businesses, contributing to the community’s economic growth.
As we look towards the future, Amazon’s vision remains clear: to be the go-to platform for all your shopping needs. And we’ll continue to work towards this vision. With the introduction of Local Gift Cards, Amazon is not just expanding its product offerings but also strengthening its commitment to local communities. We’re excited to see how this initiative will benefit both our customers and local businesses, and we’ll keep pushing the boundaries to make shopping more convenient, more personal, and more rewarding for everyone.
Passage with cot prompting:
Gift cards are consistently among the most sought after gifts year after year. They offer flexibility and convenience, allowing the recipient to choose something they truly desire. Amazon, the global e-commerce leader, has recognized this trend and is taking it a step further.
On the occasion of our “Happy Anniversary”, we are excited to introduce Amazon’s new offering - “Local Gift Cards”. These gift cards are not just limited to popular brands or stores. Instead, they represent a wide array of local businesses, making them a unique and thoughtful gift option.
The benefits of “Local Gift Cards” are manifold. For those who appreciate the charm of local businesses and the unique products they offer, these gift cards provide an opportunity to support their favorite establishments, even from the comfort of their homes. For businesses, this partnership with Amazon can lead to increased visibility and sales.
At Amazon, we believe in the power of choice. And we’ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need, be it for a local bakery, a favorite clothing store, or a renowned tech brand. So, whether it’s a birthday, a graduation, or just because, Amazon’s “Local Gift Cards” are here to make your gifting experience more personal and diverse.
Passage with tot prompting:
Gift cards are consistently among the most sought after gifts year after year. Amazon, in its continuous effort to meet customer needs, has introduced a new feature. In celebration of a “Happy Anniversary” milestone, Amazon has announced the availability of Local Gift Cards. These Local Gift Cards are designed to cater to specific regions or communities. They can be used at local businesses, restaurants, or services within the specified area. This initiative aims to strengthen the bond between Amazon and local businesses, as well as provide customers with more diverse gift options.
Paragraph 1: Amazon, the global e-commerce giant, is marking a significant milestone with the announcement of a new feature - Local Gift Cards. This move comes in response to the ever-growing popularity of gift cards as a preferred gift option.
Paragraph 2: In a bid to celebrate their “Happy Anniversary,” Amazon has introduced Local Gift Cards. These cards are tailored to cater to specific regions or communities, allowing customers to support local businesses, restaurants, or services. By offering Local Gift Cards, Amazon is not only enhancing its relationship with local businesses but also providing customers with a wider range of gift options that resonate with their community.
Paragraph 3: Amazon’s commitment to customer satisfaction has led them to expand their gift card offerings. With the introduction of Local Gift Cards, Amazon is now able to cater to a wider range of customer preferences. This feature allows customers to find the perfect gift for any occasion, making their shopping experience more personalized and convenient.
Paragraph 4: As we move forward, Amazon plans to expand the selection of Local Gift Cards to cover more areas and businesses. We believe that this feature will be well-received by our customers, and we’re excited to continue this journey towards making shopping more convenient and personalized. And we’ll continue to work towards our vision of allowing our customers to find and purchase any gift card they need, be it a Local Gift Card or a gift card from a major retailer.
Paragraph 5: In conclusion, Amazon’s introduction of Local Gift Cards is a significant step towards enhancing the shopping experience for customers. By offering Local Gift Cards, Amazon is not only catering to the growing demand for diverse gift options but also strengthening its relationship with local businesses. We look forward to the continued growth and success of this initiative, as we strive to make every shopping experience a memorable one.
Evaluating coherence via LLM scoring.#
Let’s instantiate the evaluator with the Nova Lite model. For the first evaluation task, we prompt the model to score the coherency of each passage.
MODEL_EVAL = "amazon.nova-lite-v1:0"
TEMP_EVAL = 0.75
ett = EvaluateToT(MODEL_EVAL, TEMP_EVAL)
for method in passages.keys():
display(Markdown(f"### {method}"))
display(Markdown((ett.score_passages(passages[method])[0])))
display(Markdown(("---")))
standard
The passage provided is clear, well-structured, and effectively communicates the idea behind Amazon’s introduction of Local Gift Cards. Let’s break down the key points that contribute to its coherency:
Title: The title “Amazon Introduces Local Gift Cards” is relevant and gives a precise idea about the content of the passage.
Introduction: The passage starts by discussing the popularity of gift cards, setting the stage for the introduction of Amazon’s new offering. It smoothly transitions into the announcement of the Local Gift Cards.
Flow of Information: The passage logically flows from the general concept of gift cards to the specifics of Amazon’s new initiative. Each paragraph builds on the previous one, maintaining a consistent theme.
Purpose and Benefits: The passage clearly explains the purpose behind Amazon’s new Local Gift Cards and the benefits for both customers and local businesses. It addresses how the cards cater to local needs and the convenience they offer.
Future Vision: The passage concludes by reiterating Amazon’s broader vision and commitment to supporting local communities, which ties back to the initial discussion on the popularity and utility of gift cards.
Language and Tone: The language used is professional and consistent, maintaining a positive and forward-looking tone throughout.
Given these points, the passage demonstrates strong coherency in its structure, flow, and message. Therefore, I would rate the coherency score as:
Thus the coherency score is 9.
cot
The passage is well-structured and maintains a high level of coherency throughout. The introduction clearly presents the context and relevance of gift cards, smoothly transitioning into Amazon’s new offering. Each sentence logically follows the previous one, maintaining a clear and consistent theme.
The passage effectively explains the concept of “Local Gift Cards,” their benefits, and their potential impact on both consumers and local businesses. The concluding sentences reinforce Amazon’s commitment to choice and diversity in gift options, rounding out the message neatly.
Thus, the coherency score is 9.
tot
The passage provided is coherent and logically structured, with each paragraph building on the previous one to elaborate on Amazon’s new feature of Local Gift Cards. Here’s a breakdown of its coherency:
Introduction: The passage starts with a clear statement about the popularity of gift cards and introduces Amazon’s new feature—Local Gift Cards. This sets the stage for the subsequent paragraphs.
Paragraph 1: It provides background on Amazon’s new feature and connects it to the popularity of gift cards, establishing the context for why this feature is being introduced.
Paragraph 2: This paragraph expands on the specifics of Local Gift Cards, explaining their purpose and benefits. It ties back to the celebration of Amazon’s “Happy Anniversary” and emphasizes the support for local businesses.
Paragraph 3: It continues the theme by discussing Amazon’s commitment to customer satisfaction and how Local Gift Cards cater to a wider range of customer preferences. This adds depth to the idea introduced earlier.
Paragraph 4: This paragraph looks forward, discussing Amazon’s plans to expand the selection of Local Gift Cards. It reiterates the company’s vision and commitment to enhancing the shopping experience.
Conclusion: The final paragraph wraps up the passage by summarizing the key points and expressing optimism about the future of this initiative.
Overall, the passage flows logically from the introduction of the concept to its benefits, future plans, and conclusion. Each paragraph contributes to a coherent narrative.
Thus, the coherency score is 9.
Evaluating coherence via pair comparison.#
For the second evaluation task, we construct pairs of passages and ask the evaluator LLM to choose the more coherent of the two. To double check consistency of the LLM as evaluator, we test the coherency of a passage with itself (it should be equally coherent) and we also flip the order of the evaluated passages (theoretically the LLM should not care about the order, but in practice we observe a certain tendency to prefer the first appearing passage).
pairs = [
("standard", "standard"),
("tot", "tot"),
("standard", "cot"),
("cot", "standard"),
("standard", "tot"),
("tot", "standard"),
("cot", "tot"),
("tot", "cot"),
]
for pair in pairs:
passage1, passage2 = pair
display(Markdown((f"### 1. {passage1}\t2. {passage2}")))
display(Markdown((ett.compare_passages(passages[passage1], passages[passage2])[0])))
display(Markdown(("---")))
1. standard 2. standard
Both Passage 1 and Passage 2 are identical in content and structure. They present the same information about Amazon’s introduction of Local Gift Cards, the benefits of these cards, and Amazon’s commitment to local businesses and communities.
Therefore, the coherence of both passages is the same.
“The two passages are similarly coherent.”
1. tot 2. tot
The two passages are similarly coherent.
Both passages maintain a clear and logical flow of information, effectively communicating the introduction of Amazon’s Local Gift Cards and their benefits. The structure, language, and content of both passages are nearly identical, making it difficult to distinguish a clear difference in coherence between them.
The more coherent passage is neither 1 nor 2; they are similarly coherent.
1. standard 2. cot
Both passages discuss Amazon’s introduction of Local Gift Cards, focusing on their benefits and the positive impact on both customers and local businesses. However, upon closer inspection, Passage 1 appears to be more coherent.
Passage 1 has a clear structure, beginning with an introduction to the popularity of gift cards and the announcement of Amazon’s Local Gift Cards. It follows with specific details about the cards, the reasons behind the initiative, and the potential benefits for both customers and local businesses. The passage also includes a celebratory tone that aligns with the occasion of the company’s anniversary.
Passage 2 also explains the concept of Local Gift Cards and their advantages but lacks some of the detailed context and flow found in Passage 1. It briefly touches on the benefits and vision but does not delve deeply into the reasons behind the initiative or its broader implications. The transition between sentences feels slightly abrupt, and the celebratory tone is less pronounced.
Given these observations, the more coherent passage is:
The more coherent passage is 1.
1. cot 2. standard
Upon comparing the coherency of the two passages, it is evident that both passages are similarly coherent. Both passages effectively communicate the introduction of Amazon’s Local Gift Cards, highlight the benefits of these cards for both consumers and local businesses, and emphasize Amazon’s commitment to enhancing the shopping experience.
Passage 1 follows a logical flow, starting with the popularity of gift cards, introducing Amazon’s recognition of this trend, and then detailing the features and benefits of the new “Local Gift Cards”. It concludes by reiterating Amazon’s vision for personalized and diverse gift options.
Passage 2 also follows a clear structure. It begins by establishing the popularity of gift cards and Amazon’s response with Local Gift Cards. It then discusses the benefits for both consumers and local businesses, and concludes with Amazon’s future vision and commitment to local communities.
Both passages are well-organized, clear, and effectively convey the intended message. Therefore, the conclusion is:
The two passages are similarly coherent.
1. standard 2. tot
After carefully analyzing both passages, it can be observed that Passage 1 is more coherent compared to Passage 2.
Passage 1 has a logical flow, beginning with an introduction to the concept of gift cards and then smoothly transitioning into Amazon’s new initiative. The passage is well-structured, with clear and concise sentences that make it easy to follow. Additionally, it provides relevant details about the benefits of the Local Gift Cards for both customers and local businesses.
On the other hand, Passage 2, while containing similar information, appears to be more fragmented. It consists of several short paragraphs, each focusing on a specific aspect of the Local Gift Cards, which can make it slightly harder to follow. Furthermore, the repetition of certain phrases and ideas across different paragraphs can be seen as unnecessary.
Therefore, based on the analysis, “The more coherent passage is 1”.
1. tot 2. standard
Both passages provide information about Amazon’s introduction of Local Gift Cards and their benefits. However, Passage 1 is more coherent because it is more structured and logically organized. It clearly outlines the purpose of Local Gift Cards, their benefits, and how they support local businesses. The paragraphs flow smoothly from one to the next, making it easy to follow the main points.
Passage 2, while informative, lacks the same level of coherence due to its more fragmented structure. Although it covers similar content, the information is presented in a less organized manner, which can make it slightly harder to follow.
The more coherent passage is 1.
1. cot 2. tot
Upon examining both passages, it’s clear that Passage 1 is more coherent. It presents a more linear and seamless narrative, providing concise and clear information about the introduction and benefits of Amazon’s “Local Gift Cards.” The structure flows naturally, allowing the reader to easily follow the information presented.
Passage 2, while also informative, is less coherent due to its segmented approach. It is divided into multiple paragraphs, each reiterating similar points without adding new, substantial information. This repetition can be somewhat redundant and disrupts the flow of the passage.
The more coherent passage is 1.
1. tot 2. cot
Upon comparing the coherency of the two passages, it appears that Passage 1 is more coherent.
Passage 1 has a clear and consistent structure. It begins with a general statement about the popularity of gift cards and then seamlessly transitions into Amazon’s new feature, the Local Gift Cards. Each paragraph logically builds upon the previous one, providing detailed information about the initiative and its benefits. The passage uses clear and concise language, which makes it easy to follow.
Passage 2, while still coherent, has some issues with flow and structure. For example, it starts by discussing the popularity of gift cards but then abruptly shifts to introduce Amazon’s new offering. Additionally, the language is slightly more promotional and less informative compared to Passage 1. While it conveys the necessary information, it does so in a less structured and somewhat less engaging manner.
Therefore, based on the overall clarity, structure, and flow:
The more coherent passage is 1.
Conclusion#
We have implemented the Tree-of-Thoughts prompting approach as introduced in Yao et al. (2023) and have applied it to the same type of creative writing task showcased there (the other 2 are solving a mini crossword and the mathematical reasoning challenge Game of 24). For creative writing, the generation of plans that serve as thoughts allow the system to consider several paths and choose the most promising one.
Notice that due to the variability in the generated outputs, and the fact that LLMs as evaluators tend to prefer the choice presented as first option, it is not always possible to observe a clear superiority of the ToT approach with respect to the baselines. The ToT paper presents results from multiple runs using OpenAI models and are able to demonstrate that ToT produces improved results than standard and CoT prompting.
Try it Yourself!#
Well done on completing the lab. Now it's time for you to get creative.
Try changing the input data to prompt the model to produce passages based on different data. You can also modify the model id and temperature settings to explore the variability of the results.
############## CODE HERE ####################
############## END OF CODE ##################
5. Quiz Questions#
Well done on completing the lab! Now, it’s time for a brief knowledge assessment.
Try it Yourself!#
Answer the following questions to test your understanding of tree-of-thought prompting.
import sys
sys.path.append('..')
from mlu_utils.quiz_questions import *
lab4b_question1.display()