LLM Agents#
The true sign of intelligence is not knowledge but imagination - Albert Einstein
We introduce the capabilities of LLM agents using chatbots with memory, retrieval-augmented generation (RAG), and multi-agent collaboration. Employing Microsoft’s Phi-4-mini model, a ChromaDB vector store and sentence embedding models via Ollama, we demonstrate how LLMs can be enhanced to improve factual accuracy and support long-context reasoning. The final application centers around measuring the value of corporate philanthropy, illustrating how agents can ground their reasoning in retrieved knowledge and engage in multi-role dialogue to develop actionable plans.
# By: Terence Lim, 2020-2025 (terence-lim.github.io)
import torch
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig, logging
import io
import soundfile
import matplotlib.pyplot as plt
from PIL import Image
from tqdm import tqdm
from pprint import pprint
import warnings
import textwrap
def wrap(text):
"""Helper to wrap text for pretty printing"""
return "\n".join([textwrap.fill(s, width=80) for s in text.split('\n')])
torch.random.manual_seed(0)
logging.set_verbosity_error()
# display gpu and memory
gpu_stats = torch.cuda.get_device_properties(0)
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.")
GPU = NVIDIA GeForce RTX 3080 Laptop GPU. Max memory = 15.739 GB.
Microsoft Phi-4 model#
The Microsoft Phi-4 family is designed for compactness and strong reasoning across modalities. Released in February 2024, Phi-4-multimodal is a 5.6B parameter model which processes audio, vision, and language simultaneously within the same representation space. Its capabilities include:
Instruction following: Instruct-tuned for multi-turn conversation, summarization, Q&A
Math and code reasoning: Fine-tuned on mathematical and coding data
Multilingual: Supports multiple languages with cross-lingual reasoning
Vision: integrates a visual encoder that converts image inputs into embeddings, supporting tasks such as
OCR (Optical Character Recognition)
chart and table understanding
image-based reasoning (e.g., figures, diagrams)
Audio Capabilities: a specialized audio encoder for speech inputs, supporting:
Automatic Speech Recognition (ASR)
Multilingual speech translation
https://azure.microsoft.com/en-us/blog/empowering-innovation-the-next-generation-of-the-phi-family/
# load and configures Phi-4-mini-instruct using HuggingFace
model_path = "microsoft/Phi-4-mini-instruct"
model_path = "microsoft/Phi-4-multimodal-instruct"
with warnings.catch_warnings(): # ignore small sample warnings
warnings.simplefilter("ignore")
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
print(processor.tokenizer)
model = AutoModelForCausalLM.from_pretrained(
model_path,
trust_remote_code=True,
torch_dtype='auto',
_attn_implementation='flash_attention_2',
# _attn_implementation='eager', # 'flash_attention_2',
).cuda()
GPT2TokenizerFast(name_or_path='microsoft/Phi-4-multimodal-instruct', vocab_size=200019, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=False, added_tokens_decoder={
199999: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
200010: AddedToken("<|endoftext10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
200011: AddedToken("<|endoftext11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
200018: AddedToken("<|endofprompt|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
200019: AddedToken("<|assistant|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
200020: AddedToken("<|end|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
200021: AddedToken("<|user|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
200022: AddedToken("<|system|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
200023: AddedToken("<|tool|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=False),
200024: AddedToken("<|/tool|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=False),
200025: AddedToken("<|tool_call|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=False),
200026: AddedToken("<|/tool_call|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=False),
200027: AddedToken("<|tool_response|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=False),
200028: AddedToken("<|tag|>", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
}
)
Memory usage
def cuda_memory():
"""Show GPU memory free"""
if torch.cuda.is_available():
device = torch.device('cuda')
total_memory = torch.cuda.get_device_properties(device).total_memory
reserved_memory = torch.cuda.memory_reserved(device)
allocated_memory = torch.cuda.memory_allocated(device)
free_memory = total_memory - reserved_memory
print(f"Total memory: {total_memory / (1024**3):.2f} GB")
print(f"Reserved memory: {reserved_memory / (1024**3):.2f} GB")
print(f"Allocated memory: {allocated_memory / (1024**3):.2f} GB")
print(f"Free memory: {free_memory / (1024**3):.2f} GB")
cuda_memory()
Total memory: 15.74 GB
Reserved memory: 10.45 GB
Allocated memory: 10.42 GB
Free memory: 5.29 GB
Generate model response
generation_args = {
"max_new_tokens": 512,
#"return_full_text": False,
"temperature": 0.6, # nonzero so that chat responses can vary slightly
"do_sample": True, # False,
}
generation_config = GenerationConfig.from_pretrained(model_path,
'generation_config.json')
Helpers to format content to prompt for response
def create_content(query, image=None, audio=None):
"""Formats a message content string"""
image_prompt = '' if image is None else '<|image_1|>'
audio_prompt = '' if audio is None else '<|audio_1|>'
prompt = f'{image_prompt}{audio_prompt}{query}'
return prompt
def create_prompt(query, image=None, audio=None):
"""Generate a single prompt for inference response"""
user_prompt = '<|user|>'
assistant_prompt = '<|assistant|>'
prompt_suffix = '<|end|>'
content = create_content(query, image=image, audio=audio)
prompt = f'{user_prompt}{content}{prompt_suffix}{assistant_prompt}'
return prompt
def pipe(query, image=None, audio=None, verbose=False, **kwargs):
"""Pipeline to format and send query, and generate and decode response"""
if isinstance(query, list): # query input is given as a list of messages
prompt = processor.tokenizer.apply_chat_template(query,
tokenize=False,
add_generation_prompt=True)
# remove last <|endoftext|> if it is there, which is used for training, not inference.
# For training, make sure to add <|endoftext|> in the end.
if prompt.endswith('<|endoftext|>'):
prompt = prompt.rstrip('<|endoftext|>')
elif isinstance(query, str): # query input is given as a string
prompt = create_prompt(query, image=image, audio=audio)
else:
raise Exception('Invalid prompt format, must be str or list of messages')
if verbose:
print(prompt)
inputs = processor(prompt, images=image, audios=audio, return_tensors='pt').to('cuda:0')
with warnings.catch_warnings(): # ignore small sample warnings
warnings.simplefilter("ignore")
generate_ids = model.generate(
**inputs,
generation_config=generation_config,
# https://huggingface.co/microsoft/Phi-4-multimodal-instruct/discussions/46
num_logits_to_keep=1,
**kwargs,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1] :]
response = processor.batch_decode(
generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
return response
Chatbot#
A chatbot simulates ongoing dialogue between a user and an LLM. By appending previous user inputs and model responses into the prompt, the chatbot maintains conversational memory. This enables it to handle follow-up questions, build on earlier answers, and maintain topic continuity over multiple turns. During a chat session, the prompt is iteratively expanded to include each previous exchange. This accumulated memory allows the model to generate responses that take prior context into account.
Simulate user queries
# Simulate a sequence of user queries to be processed
sequence_user_queries = [
"What are the challenges to measuring the value of corporate philanthropy?",
"Please suggest some solutions.",
"Thank you, please concisely summarize into an action plan."
]
Memory#
During a chat session, the prompt is iteratively expanded to include each previous exchange. This accumulated memory allows the model to generate responses that take prior context into account. The final response is displayed, along with the complete prompt history used in that turn—demonstrating how memory accumulation influences generation.
# Prompt grows over time with user and assistant turns.
memory = [] # memory of past AI-assistant turns
for turn in range(len(sequence_user_queries)):
# Add initial user query
query = sequence_user_queries[0]
messages = [{"role": "user", "content": query}]
# 3. Loop through past turns of AI response and user prompts
for response, query in zip(memory, sequence_user_queries[1:len(memory)+1]):
# Add AI assistant's response
messages.append({"role": "assistant", "content": response})
# Add user's next query
messages.append({"role": "user", "content": query})
# 4. Generate response and save to memory
output = pipe(messages, **generation_args)
memory.append(output)
print(f'----------- {turn=} -----------')
print('QUERY:', query)
print('RESPONSE:')
print(wrap(memory[-1]))
print()
33%|███▎ | 1/3 [00:21<00:42, 21.09s/it]
----------- turn=0 -----------
QUERY: What are the challenges to measuring the value of corporate philanthropy?
RESPONSE:
Measuring the value of corporate philanthropy presents several challenges,
including:
1. Lack of standardized metrics: There is no universally accepted way to measure
the impact of corporate philanthropy. Different organizations may use different
metrics, making it difficult to compare and assess the value of their
philanthropic efforts.
2. Difficulty in quantifying social impact: Corporate philanthropy often aims to
address complex social issues, making it challenging to quantify the impact.
Social impact can be long-term and may not be immediately visible, requiring a
long-term perspective to assess the value of corporate philanthropy.
3. Lack of transparency: Corporate philanthropy can be seen as a way for
companies to boost their public image, rather than a genuine effort to make a
positive impact. This lack of transparency can make it difficult to assess the
true value of corporate philanthropy.
4. Difficulty in attributing outcomes: It can be challenging to attribute
specific outcomes to corporate philanthropy, as social issues are often
influenced by multiple factors. This makes it difficult to determine the direct
impact of corporate philanthropy on a particular issue.
5. Conflicts of interest: Corporate philanthropy can sometimes be driven by a
company's strategic interests, rather than a genuine desire to make a positive
impact. This can create conflicts of interest and make it difficult to assess
the true value of corporate philanthropy.
6. Lack of accountability: Without clear guidelines and accountability
mechanisms, it can be challenging to ensure that corporate philanthropy is
effective and achieves its intended goals. This can result in wasted resources
and missed opportunities to make a positive impact.
7. Limited resources: Corporate philanthropy is often limited by a company's
financial resources. This can make it difficult to measure the value of
corporate philanthropy, as it may not be possible to address all the social
issues that a company wants to support.
Overall, measuring the value of corporate philanthropy requires a comprehensive
approach that takes into account the complex nature of social issues and the
various factors that influence corporate philanthropy. It is essential for
companies to be transparent, accountable, and focused on creating a positive
social impact.
67%|██████▋ | 2/3 [00:43<00:21, 21.72s/it]
----------- turn=1 -----------
QUERY: Please suggest some solutions.
RESPONSE:
To address the challenges of measuring the value of corporate philanthropy, the
following solutions can be considered:
1. Develop standardized metrics: Establishing universally accepted metrics and
guidelines for measuring the impact of corporate philanthropy can help
organizations compare and assess their efforts more effectively. These metrics
could include both quantitative and qualitative measures, such as the number of
people served, the amount of money donated, and the long-term impact on
communities.
2. Focus on social impact: Companies should prioritize measuring the social
impact of their philanthropic efforts, rather than just financial outcomes. This
can be done by setting clear goals and objectives, tracking progress over time,
and using both quantitative and qualitative data to assess the impact on the
communities and individuals they serve.
3. Increase transparency: Companies should be transparent about their
philanthropic efforts, including the reasons behind their giving, the selection
process for grants, and the outcomes achieved. This can help build trust with
stakeholders and ensure that corporate philanthropy is genuinely focused on
making a positive impact.
4. Improve attribution methods: To better attribute outcomes to corporate
philanthropy, companies can use a combination of data analysis, case studies,
and stakeholder feedback. This can help identify the direct and indirect impact
of their efforts and better understand how their contributions are making a
difference.
5. Address conflicts of interest: Companies should be mindful of potential
conflicts of interest and ensure that their philanthropic efforts are not solely
driven by a desire to boost their public image. This can be achieved by
involving external stakeholders, such as nonprofit partners or community
leaders, in the decision-making process.
6. Establish accountability mechanisms: Companies should establish clear
accountability mechanisms, such as third-party evaluations or independent
audits, to ensure that their philanthropic efforts are effective and achieving
their intended goals. This can help build trust with stakeholders and provide a
basis for continuous improvement.
7. Leverage partnerships: Companies can collaborate with nonprofit
organizations, community groups, and other stakeholders to maximize the impact
of their philanthropic contributions. By leveraging the expertise and resources
of others, companies can address complex social issues more effectively and
ensure that their efforts are making a meaningful difference.
By implementing these solutions, companies can improve their ability to measure
the value of corporate philanthropy and demonstrate the positive social impact
of their efforts.
100%|██████████| 3/3 [00:59<00:00, 19.80s/it]
----------- turn=2 -----------
QUERY: Thank you, please concisely summarize into an action plan.
RESPONSE:
Action Plan for Measuring the Value of Corporate Philanthropy:
1. Develop standardized metrics:
- Establish universal guidelines and metrics for measuring the impact of
corporate philanthropy.
- Include both quantitative and qualitative measures, such as the number of
people served, amount of money donated, and long-term community impact.
2. Focus on social impact:
- Set clear goals and objectives for philanthropic efforts.
- Track progress over time using both quantitative and qualitative data.
- Assess the impact on communities and individuals served.
3. Increase transparency:
- Be transparent about philanthropic efforts, including selection processes,
outcomes, and reasons behind giving.
- Share information with stakeholders to build trust and demonstrate genuine
impact.
4. Improve attribution methods:
- Use data analysis, case studies, and stakeholder feedback to better
attribute outcomes to corporate philanthropy.
- Identify direct and indirect impacts of philanthropic efforts.
5. Address conflicts of interest:
- Involve external stakeholders, such as nonprofit partners or community
leaders, in decision-making processes.
- Ensure philanthropic efforts are not solely driven by a desire to boost
public image.
6. Establish accountability mechanisms:
- Implement third-party evaluations, independent audits, or other
accountability mechanisms to ensure effectiveness and achievement of goals.
- Use findings to continuously improve philanthropic efforts.
7. Leverage partnerships:
- Collaborate with nonprofit organizations, community groups, and other
stakeholders to maximize impact.
- Leverage their expertise and resources to address complex social issues
more effectively.
By following this action plan, companies can improve their ability to measure
the value of corporate philanthropy and demonstrate the positive social impact
of their efforts.
Show the final response from the last turn of the Chatbot
print('FINAL ANSWER:')
print(wrap(output))
FINAL ANSWER:
Action Plan for Measuring the Value of Corporate Philanthropy:
1. Develop standardized metrics:
- Establish universal guidelines and metrics for measuring the impact of
corporate philanthropy.
- Include both quantitative and qualitative measures, such as the number of
people served, amount of money donated, and long-term community impact.
2. Focus on social impact:
- Set clear goals and objectives for philanthropic efforts.
- Track progress over time using both quantitative and qualitative data.
- Assess the impact on communities and individuals served.
3. Increase transparency:
- Be transparent about philanthropic efforts, including selection processes,
outcomes, and reasons behind giving.
- Share information with stakeholders to build trust and demonstrate genuine
impact.
4. Improve attribution methods:
- Use data analysis, case studies, and stakeholder feedback to better
attribute outcomes to corporate philanthropy.
- Identify direct and indirect impacts of philanthropic efforts.
5. Address conflicts of interest:
- Involve external stakeholders, such as nonprofit partners or community
leaders, in decision-making processes.
- Ensure philanthropic efforts are not solely driven by a desire to boost
public image.
6. Establish accountability mechanisms:
- Implement third-party evaluations, independent audits, or other
accountability mechanisms to ensure effectiveness and achievement of goals.
- Use findings to continuously improve philanthropic efforts.
7. Leverage partnerships:
- Collaborate with nonprofit organizations, community groups, and other
stakeholders to maximize impact.
- Leverage their expertise and resources to address complex social issues
more effectively.
By following this action plan, companies can improve their ability to measure
the value of corporate philanthropy and demonstrate the positive social impact
of their efforts.
Retrieval-augmented generation#
One major challenge in standard LLMs is hallucination, where the model produces text that sounds believable but is actually false or unsupported by facts. Because LLMs are trained to model word sequences rather than factual accuracy, they can sometimes generate convincing yet misleading statements—especially when no clear answer exists in their training data. For example, Hicks et al. (2024) argue that LLMs are “bullshit machines” that prioritize fluent language over truth. Similarly, Mahowald et al. (2023) caution against assuming that skill with language equates to actual reasoning or understanding.
RAG addresses this issue by anchoring the model’s responses in retrieved documents, making the output more factually reliable. In a common implementation known as prompt-based in-context retrieval, the system first retrieves information from an external knowledge base, and then it generates a response based on that retrieved content. The LLM is prompted with the relevant document and instructed to answer solely based on that information, similar to reading a reference article before answering a question, allowing for better factual accuracy and handling of long-context tasks. RAG is increasingly used in several practical domains:
Search-Augmented LLMs improve accuracy in fact-checking and open-domain Q&A tasks.
Enterprise Applications enable better handling of legal, financial, and medical documents.
Personalized AI Assistants can use retrieval to support domain-specific queries using private or proprietary knowledge bases (e.g., for customer service).
For evaluating closed-domain question-answering (QA) LLMs, benchmarks such as SQuAD (Stanford Question Answering Dataset) HotpotQA are used, focusing on a system’s ability to answer questions within specific documents, including tasks like complex document comprehension and multi-hop reasoning. This task differs from open-domain QA, where LLMs are expected to draw knowledge from a vast, unrestricted knowledge base.
More advanced data structures have also been employed for RAG, such as RAPTOR by Sarthi et al (2024), which organizes data and recursive summaries in a tree structure, integrating information across lengthy documents for better performance on complex, multi-step reasoning tasks
The LangChain framework offers flexible tools for loading and preprocessing input documents of many formats. A textbook is read and divided into chunks of roughly 1,000 characters with some overlap. These chunks provide manageable input sizes for retrieval while maintaining continuity. The text itself outlines how companies can measure philanthropic value through social, business, and investor-oriented metrics, and forms the knowledge base for grounding agent responses.
# LangChain to loads and split documents into chunks for retrieval
from langchain_community.document_loaders import TextLoader, UnstructuredMarkdownLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Loads markdown file (MVCP.md) and split into overlapping chunks
loader = TextLoader('assets/MVCP.md')
document = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunked_documents = text_splitter.split_documents(document)
print('Number of chunks:', len(chunked_documents))
print(wrap(str(chunked_documents[:2])))
Number of chunks: 196
[Document(metadata={'source': 'assets/MVCP.md'}, page_content='MEASURING THE
VALUE OF CORPORATE PHILANTHROPY: SOCIAL IMPACT, BUSINESS BENEFITS, AND INVESTOR
RETURNS\nby\nTerence Lim, Ph.D.\nReport Author and Manager, Standards and
Measurement,\nCommittee Encouraging Corporate Philanthropy\n(through the
2008–2009 Goldman Sachs Public Service Program)\n\nHow to measure the value and
results of corporate philanthropy remains\none of corporate giving
professionals’ greatest challenges. Social and\nbusiness benefits are often
long-term or intangible, which make\nsystematic measurement complex. And yet:
Corporate philanthropy faces\nincreasing pressures to show it is as
strategic,cost-effective, and value-enhancing\nas possible. The industry faces a
critical need to assess current practices and\nmeasurement trends, clarify the
demands practitioners face for impact evidence,\nand identify the most promising
steps forward in order to make progress on these\nchallenges.'),
Document(metadata={'source': 'assets/MVCP.md'}, page_content='This report aims
to meet that need, by providing the corporate\nphilanthropic community with a
review of recent measurement studies, models,\nand evidence drawn from
complementary business disciplines as well as the social\nsector. Rather than
present an other compendium of narrative accounts and case\nstudies, we endeavor
to generalize the most valuable concepts and to recognize\nthe strengths and
limitations of various measurement approaches. In conjunction\nwith the
annotated references that follow, the analysis herein should provide
an\nexcellent starting point for companies wishing to adapt current
methodologies in\nthe field to their own corporate giving programs.\nTo realize
meaningful benefits, philanthropy cannot be treated as just\nanother "check in
the box," but rather must be executed no less professionally,\nproactively, and
strategically than other core business activities. Our hope is\nthat this work
will enlighten giving professionals, CEOs, and the investor')]
ChromaDB vector database#
ChromaDB is an open-source vector store used to store and retrieve text embeddings. It enables fast and efficient semantic search, allowing the system to find the most relevant document chunks based on a given query.
For compatibility, ChromaDB requires SQLite version 3.35 or higher. If the system uses an older version, apply this workaround patch: Install pysqlite3-binary, then enter the following three lines in sqlite_version.py to swap the packages.
# kludge to hack ChromaDB to use lower version of SQLite
with open('sqlite_version.py', 'w') as f:
f.write('''__import__('pysqlite3')
import sys
sys.modules['sqlite3'] = sys.modules.pop('pysqlite3')
'''.strip())
import sqlite_version
# ChromaDB vector store to store and retrieve document embeddings.
import chromadb
client = chromadb.Client()
try:
client.delete_collection(name="docs")
except:
pass
#collection = client.get_or_create_collection(name="docs")
collection = client.create_collection(name="docs")
The sentence embeddings for both documents and queries are generated using the mxbai-embed-large model served via Ollama. This embedding model, which reached state-of-the-art performance in April 2024, was trained on over 700 million high-quality sentence pairs and fine-tuned on 30 million triplets. These embeddings are stored in ChromaDB to support rapid document retrieval.
# Ollama for generating sentence embeddings using mxbai-embed-large model
# !ollama pull mxbai-embed-large
import ollama
# store chunk embeddings in a vector database
for i, d in enumerate(chunked_documents):
response = ollama.embeddings(model="mxbai-embed-large", prompt=d.page_content)
embedding = response["embedding"]
collection.add(
ids=[str(i)],
embeddings=[embedding],
documents=[d.page_content],
)
Retrieval#
The retrieval pipeline consists of four main steps:
A user query is written in natural language.
The query is embedded using the same model as the document chunks.
The top 5 most semantically similar chunks are retrieved from the ChromaDB vector store.
A new prompt is constructed by combining the user’s question with the retrieved content for generation.
# A sample question prompt
query = "What are the challenges to measuring the value of corporate philanthropy?"
Embed the prompt query
# generate an embedding for the query
response = ollama.embeddings(
prompt=query,
model="mxbai-embed-large"
)
len(response['embedding']) # vector length
1024
Retrieve the 5 most similar document chunks from the vector database
# ses ChromaDB to find top 5 most similar chunks
results = collection.query(
query_embeddings=[response["embedding"]],
n_results=5,
)
for data in results['documents'][0]:
print(textwrap.fill(data))
MEASURING THE VALUE OF CORPORATE PHILANTHROPY: SOCIAL IMPACT, BUSINESS
BENEFITS, AND INVESTOR RETURNS by Terence Lim, Ph.D. Report Author and
Manager, Standards and Measurement, Committee Encouraging Corporate
Philanthropy (through the 2008–2009 Goldman Sachs Public Service
Program) How to measure the value and results of corporate
philanthropy remains one of corporate giving professionals’ greatest
challenges. Social and business benefits are often long-term or
intangible, which make systematic measurement complex. And yet:
Corporate philanthropy faces increasing pressures to show it is as
strategic,cost-effective, and value-enhancing as possible. The
industry faces a critical need to assess current practices and
measurement trends, clarify the demands practitioners face for impact
evidence, and identify the most promising steps forward in order to
make progress on these challenges.
detailed insights into the related measurement process which can help
demonstrate understanding of what drives long term business success
quality of management and superior potential to create financial
value. The value of corporate philanthropy is measurable; as with many
elements of business, however, it cannot always be measured as
precisely as we would like. What gets measured gets managed goes the
old adage; indeed measurement plays a crucial role in enabling
companies to reach their full potential both philanthropically and as
more successful and sustainable enterprises overall.
This report aims to meet that need, by providing the corporate
philanthropic community with a review of recent measurement studies,
models, and evidence drawn from complementary business disciplines as
well as the social sector. Rather than present an other compendium of
narrative accounts and case studies, we endeavor to generalize the
most valuable concepts and to recognize the strengths and limitations
of various measurement approaches. In conjunction with the annotated
references that follow, the analysis herein should provide an
excellent starting point for companies wishing to adapt current
methodologies in the field to their own corporate giving programs. To
realize meaningful benefits, philanthropy cannot be treated as just
another "check in the box," but rather must be executed no less
professionally, proactively, and strategically than other core
business activities. Our hope is that this work will enlighten giving
professionals, CEOs, and the investor
Corporate philanthropy is as vital as ever to business and society but
it faces steep pressures to demonstrate that it is also cost effective
and aligned with corporate needs. Indeed many corporate giving
professionals cite measurement as their primary management challenge.
Social and business benefits are often long-term, intangible or both
and a systematic measurement of these results can be complex. Social
change takes time. The missions and intervention strategies involved
are diverse. For these reasons, the field of corporate philanthropy
has been unable to determine a shared definition or method of
measurement for social impact. Similarly, the financial value of
enhancing intangibles such as a company's reputational and human
capital cannot be measured directly and may not be converted into
tangible bottom line profits in the near term. Corporate givers and
grant recipients often useless formal anecdotal methods to convey
impact. While stories
### Summary The attractiveness of these ROI methods for calculating
corporate philanthropy's social returns is in bringing businesslike
quantitative frameworks to evaluating and comparing the effectiveness
of diverse social programs, and aggregating their social impact.
However these sophisticated methodologies place heavy demands on data
collection assumptions and value judgments underlying the analysis.
Funders must assemble data and calculations on the program's monetary
benefits and make subjective judgments on the relative value of
different types of social changes. Corporate funders need to be
knowledgeable and thoughtful about these limitations and typically
should not rely solely on ROI when evaluating grants. Proponents of
these methods note that the benefits of ROI analysis lie more in
encouraging funders to lay bare the assumptions and trade-offs that
may already be involved in their grant making decisions.
Generation#
Construct the prompt combining the query and supporting context retrieved. Then send to the LLM, which generates a grounded response.
# assembles relevant chunks into a prompt alongside the original question
prompt = f"""
Only use relevant information in the following text delimited by triple quotes:
'''{data}.'''
Respond to this prompt: {query}."""
# send a conversation-style chat-template prompt incorporating the retrieved text
messages = [
{"role": "user", "content": prompt},
]
output = pipe(messages, **generation_args)
print('QUERY:', query)
print('RESPONSE:')
print(wrap(output))
QUERY: What are the challenges to measuring the value of corporate philanthropy?
RESPONSE:
The challenges to measuring the value of corporate philanthropy include:
1. Social and business benefits are often long-term or intangible, making
systematic measurement complex.
2. Corporate philanthropy faces increasing pressures to show it is as strategic,
cost-effective, and value-enhancing as possible.
3. The industry faces a critical need to assess current practices and
measurement trends.
4. There is a need to clarify the demands practitioners face for impact
evidence.
5. Identifying the most promising steps forward to make progress on these
challenges.
6. Demonstrating understanding of what drives long-term business success and
quality of management.
7. Measuring the financial value of enhancing intangibles such as a company's
reputational and human capital can be complex and may not be directly
convertible into tangible bottom-line profits in the near term.
8. The measurement of social impact lacks a shared definition or method, and
social change takes time.
9. Missions and intervention strategies in corporate philanthropy are diverse.
10. The use of sophisticated ROI methodologies for evaluating and comparing the
effectiveness of diverse social programs places heavy demands on data collection
assumptions and value judgments underlying the analysis.
Multi-agents#
In multi-agent workflows, multiple LLM agents play different roles and collaborate to accomplish a shared objective. In this demonstration, three role-playing agents work together to simulate human-like decision-making processes:
A Chief Executive Officer (CEO) asks strategic questions about measuring philanthropic value.
A Chief Giving Officer (CGO) synthesizes responses into an evolving plan.
An AI Assistant (nicknamed CorGi) uses RAG to answer the CEO’s questions using only retrieved text.
The agents interact with each other over several turns, collaboratively building and refining a plan grounded in the corporate philanthropy document.
Tool calling#
Traditional LLMs are limited in that they cannot retrieve live data or execute real-world functions unless explicitly programmed. Without tools, they may also make computational errors or respond inconsistently to ambiguous prompts. Recent advances focus on enabling tool use, where LLMs can:
Make function calls.
Use APIs to gather real-time data.
Perform calculations or database lookups.
The main approaches include:
Allow the model to generate function calls instead of plain text.
Zero-Shot Tool Use: Models learn to use tools on-the-fly via structured prompts.
Fine-Tuned Tool Use: Models are trained to recognize and call specific functions during generation.
AI Assistant agent:
The AI Research Assistant agent responds to queries by leveraging RAG. Its prompt instructs it to only use retrieved documents to answer and to reply “I don’t know” if the information isn’t found in the text. This strict constraint helps minimize hallucination and ensures that all responses are grounded in the knowledge base.
# AI assistant agent answers using only retrieved text chunks
def rag_agent(query, n_results=5):
"""Role of a RAG-based question-answering agent"""
response = ollama.embeddings(prompt=query, model="mxbai-embed-large")
results = collection.query(query_embeddings=[response["embedding"]],
n_results=n_results)
text = "\n".join(results['documents'][0])
rag_prompt=f"""
You are a helpful AI research assistant.
Use only the information in the text to succintly answer the question.
Reply "I don't know" if the information is not found in the text.
Text: {text}.
Question: {query}"""
# generate a response combining the prompt and data
output = pipe(rag_prompt, **generation_args)
return output, results['ids'][0]
Reply to a query using grounded knowledge in the document
pprint(rag_agent("What is corporate philanthropy?"))
('Corporate philanthropy refers to the charitable giving and activities of a '
"company, often aimed at enhancing the company's reputation, addressing "
'employee concerns, and potentially contributing to business value through '
'innovation, market knowledge, and development of new technologies.',
['102', '1', '87', '140', '142'])
Query about a concept not present in the document
pprint(rag_agent("What is corporate greed?"))
('The text does not provide information about what corporate greed is.',
['137', '140', '136', '130', '117'])
Role playing#
Chief Giving Officer agent:
This agent summarizes CorGi’s outputs into a coherent strategic plan that can be reviewed by leadership.
# CGO (Chief Giving Officer) agent summarizes answers into a growing plan
def cgo_agent(text):
"""Role of an information-summarizing Chief Giving Officer"""
cgo_prompt = f"""
You are the chief giving officer of a company.
You want to describe an action plan for measuring the value
of your company's corporate philanthropy program.
Use bullet points to summarize the plan using only the
relevant and distinct information in the text.
Text: {text}""".strip()
output = pipe(cgo_prompt, **generation_args)
return output
Chief Executive Officer agent:
This agent reads the current version of the plan and generates new questions to improve or expand it.
# CEO (Chief Executive Officer) agent asks new questions to improve the corporate giving plan
def ceo_agent(text, verbose=False):
"""Role as an inquisitive Chief Executive Officer"""
ceo_prompt = f"""
You are the chief executive officer of a company.
You want to know about the plan for measuring the value
of your company's corporate philanthropy program.
Please ask a simple and general question for an idea to
improve the your company's plan that is different than the text.
Text: {text}""".strip()
output = pipe(ceo_prompt, verbose=verbose, **generation_args)
return output
Over multiple turns, this role-playing setup allows the system to iteratively build a more comprehensive and data-backed plan.
# iterate over 10 turns to co-develop a full plan
memory = '* Measure the amount of monetary donations.'
docs = []
for turn in range(10):
print(f"Turn {turn+1}...")
question = ceo_agent(memory)
print('CEO >>>', wrap(question))
print('\n------------------------------\n')
answer, doc = rag_agent(question)
docs.extend(doc) # for diagnostics: track the RAG chunks retrieved
print('Corgi-AI >>>', wrap(answer))
print('\n------------------------------\n')
memory = cgo_agent("\n".join([memory, answer]))
print('CGO >>>', wrap(memory))
print('\n------------------------------')
Turn 1...
CEO >>> How might we also consider non-monetary contributions, such as volunteer hours
or in-kind donations, in evaluating the overall impact of our corporate
philanthropy program?
------------------------------
Corgi-AI >>> I don't know
------------------------------
CGO >>> - Measure the amount of monetary donations.
- Track the number of volunteer hours contributed.
- Assess the impact of the donations through beneficiary feedback.
- Evaluate the visibility and reach of the philanthropy program through media
coverage and social media engagement.
- Analyze the correlation between philanthropic activities and company
reputation.
- Compare the company’s philanthropic efforts with industry standards.
- Review the alignment of the philanthropy program with the company’s mission
and values.
- Monitor the long-term outcomes of the philanthropic initiatives.
- Collect data on the satisfaction levels of the stakeholders involved in the
program.
- Assess the effectiveness of the philanthropic program in achieving the
company’s social responsibility goals.
- Evaluate the cost-effectiveness of the philanthropy program in relation to the
benefits received.
------------------------------
Turn 2...
CEO >>> How can we incorporate community engagement and local stakeholder input into the
evaluation of our company's philanthropy program?
------------------------------
Corgi-AI >>> I don't know
------------------------------
CGO >>> - Measure the amount of monetary donations.
- Track the number of volunteer hours contributed.
- Assess the impact of the donations through beneficiary feedback.
- Evaluate the visibility and reach of the philanthropy program through media
coverage and social media engagement.
- Analyze the correlation between philanthropic activities and company
reputation.
- Compare the company’s philanthropic efforts with industry standards.
- Review the alignment of the philanthropy program with the company’s mission
and values.
- Monitor the long-term outcomes of the philanthropic initiatives.
- Collect data on the satisfaction levels of the stakeholders involved in the
program.
- Assess the effectiveness of the philanthropy program in achieving the
company’s social responsibility goals.
- Evaluate the cost-effectiveness of the philanthropy program in relation to the
benefits received.
------------------------------
Turn 3...
CEO >>> Considering the diverse facets of our corporate philanthropy program, what
innovative metrics could we implement to better understand its cultural and
employee engagement impact within our organization?
------------------------------
Corgi-AI >>> To better understand the cultural and employee engagement impact within the
organization, the company could implement metrics such as internal surveys to
assess employee needs and identity, measure the extent to which philanthropic
programs are meeting these needs, and evaluate the relative importance different
employee segments attach to intrinsic needs. Additionally, the company could
analyze the benefits of philanthropy in terms of employee engagement, customer
loyalty, reputation capital, and market opportunities, and identify key
intermediate outcomes that could yield desired business behaviors and benefits.
These metrics could be developed by leveraging models and evidence from related
business disciplines.
------------------------------
CGO >>> - Measure the amount of monetary donations.
- Track the number of volunteer hours contributed.
- Assess the impact of the donations through beneficiary feedback.
- Evaluate the visibility and reach of the philanthropy program through media
coverage and social media engagement.
- Analyze the correlation between philanthropic activities and company
reputation.
- Review the alignment of the philanthropy program with the company’s mission
and values.
- Monitor the long-term outcomes of the philanthropic initiatives.
- Collect data on the satisfaction levels of the stakeholders involved in the
program.
- Assess the effectiveness of the philanthropy program in achieving the
company’s social responsibility goals.
- Evaluate the cost-effectiveness of the philanthropy program in relation to the
benefits received.
- Implement internal surveys to assess employee needs and identity.
- Measure the extent to which philanthropic programs are meeting these needs.
- Evaluate the relative importance different employee segments attach to
intrinsic needs.
- Analyze the benefits of philanthropy in terms of employee engagement, customer
loyalty, reputation capital, and market opportunities.
- Identify key intermediate outcomes that could yield desired business behaviors
and benefits.
------------------------------
Turn 4...
CEO >>> How can we integrate community engagement and environmental sustainability into
our corporate philanthropy program to further align with our company's values
and enhance our overall impact?
------------------------------
Corgi-AI >>> I don't know.
------------------------------
CGO >>> - Measure the amount of monetary donations.
- Track the number of volunteer hours contributed.
- Assess the impact of the donations through beneficiary feedback.
- Evaluate the visibility and reach of the philanthropy program through media
coverage and social media engagement.
- Analyze the correlation between philanthropic activities and company
reputation.
- Review the alignment of the philanthropy program with the company’s mission
and values.
- Monitor the long-term outcomes of the philanthropic initiatives.
- Collect data on the satisfaction levels of the stakeholders involved in the
program.
- Assess the effectiveness of the philanthropy program in achieving the
company’s social responsibility goals.
- Evaluate the cost-effectiveness of the philanthropy program in relation to the
benefits received.
- Measure the extent to which philanthropic programs are meeting employee needs.
- Evaluate the relative importance different employee segments attach to
intrinsic needs.
- Analyze the benefits of philanthropy in terms of employee engagement, customer
loyalty, reputation capital, and market opportunities.
- Identify key intermediate outcomes that could yield desired business behaviors
and benefits.
------------------------------
Turn 5...
CEO >>> How can we integrate employee feedback and participation to further enhance the
effectiveness of our corporate philanthropy program?
------------------------------
Corgi-AI >>> To further enhance the effectiveness of the corporate philanthropy program,
integrating employee feedback and participation can be achieved by using
internal surveys to assess the extent to which the philanthropic program meets
employee needs and creates a greater sense of identity between employee and
employer. This assessment should consider the relative importance that different
employee segments attach to different intrinsic needs. Additionally, improving
employees' sense of status, prestige, belonging within the work group and
organization, and emotional rewards inherent in their work can be achieved
through corporate philanthropic initiatives. These initiatives can also help
employee recruitment, as evidenced by the 2004 corporate community involvement
survey by Deloitte LLP, which found that 72% of employed Americans trying to
decide between two jobs offering the same location job description pay and
benefits would choose to work for the company that also supports charitable
causes. By systematically measuring the impact of corporate philanthropy,
companies can provide data-based evidence of its positive effects and make a
more persuasive case for why companies should engage in philanthropic causes.
------------------------------
CGO >>> - Measure the amount of monetary donations
- Track the number of volunteer hours contributed
- Assess the impact of the donations through beneficiary feedback
- Evaluate the visibility and reach of the philanthropy program through media
coverage and social media engagement
- Analyze the correlation between philanthropic activities and company
reputation
- Review the alignment of the philanthropy program with the company’s mission
and values
- Monitor the long-term outcomes of the philanthropic initiatives
- Collect data on the satisfaction levels of the stakeholders involved in the
program
- Assess the effectiveness of the philanthropy program in achieving the
company’s social responsibility goals
- Evaluate the cost-effectiveness of the philanthropy program in relation to the
benefits received
- Measure the extent to which philanthropic programs are meeting employee needs
- Evaluate the relative importance different employee segments attach to
intrinsic needs
- Analyze the benefits of philanthropy in terms of employee engagement, customer
loyalty, reputation capital, and market opportunities
- Identify key intermediate outcomes that could yield desired business behaviors
and benefits
- Use internal surveys to assess the extent to which the philanthropic program
meets employee needs and creates a greater sense of identity between employee
and employer
- Consider the relative importance that different employee segments attach to
different intrinsic needs
- Improve employees' sense of status, prestige, belonging within the work group
and organization, and emotional rewards inherent in their work through corporate
philanthropic initiatives
- Leverage the positive impact of corporate philanthropy on employee recruitment
- Systematically measure the impact of corporate philanthropy to provide data-
based evidence of its positive effects and make a more persuasive case for why
companies should engage in philanthropic causes.
------------------------------
Turn 6...
CEO >>> How can we incorporate the perspectives of the communities we aim to serve into
the evaluation of our corporate philanthropy program's effectiveness?
------------------------------
Corgi-AI >>> The text does not provide specific information on how to incorporate the
perspectives of the communities served into the evaluation of a corporate
philanthropy program's effectiveness.
------------------------------
CGO >>> - Measure the amount of monetary donations
- Track the number of volunteer hours contributed
- Assess the impact of donations through beneficiary feedback
- Evaluate the visibility and reach of the philanthropy program through media
coverage and social media engagement
- Analyze the correlation between philanthropic activities and company
reputation
- Review the alignment of the philanthropy program with the company’s mission
and values
- Monitor the long-term outcomes of the philanthropic initiatives
- Collect data on the satisfaction levels of the stakeholders involved in the
program
- Assess the effectiveness of the philanthropy program in achieving the
company’s social responsibility goals
- Evaluate the cost-effectiveness of the philanthropy program in relation to the
benefits received
- Measure the extent to which philanthropic programs are meeting employee needs
- Evaluate the relative importance different employee segments attach to
intrinsic needs
- Analyze the benefits of philanthropy in terms of employee engagement, customer
loyalty, reputation capital, and market opportunities
- Identify key intermediate outcomes that could yield desired business behaviors
and benefits
- Use internal surveys to assess the extent to which the philanthropic program
meets employee needs and creates a greater sense of identity between employee
and employer
- Leverage the positive impact of corporate philanthropy on employee recruitment
- Systematically measure the impact of corporate philanthropy to provide data-
based evidence of its positive effects and make a more persuasive case for why
companies should engage in philanthropic causes.
------------------------------
Turn 7...
CEO >>> How can we integrate the evaluation of employee well-being and satisfaction as a
key performance indicator in our corporate philanthropy program's success
measurement?
------------------------------
Corgi-AI >>> To integrate the evaluation of employee well-being and satisfaction as a key
performance indicator in the corporate philanthropy program's success
measurement, we can use internal surveys to assess the extent to which the
philanthropic program is meeting employee needs and creating a greater sense of
identity between employee and employer. This assessment should take into account
the relative importance that different employee segments attach to different
intrinsic needs.
------------------------------
CGO >>> - Measure the amount of monetary donations
- Track the number of volunteer hours contributed
- Assess the impact of donations through beneficiary feedback
- Evaluate the visibility and reach of the philanthropy program through media
coverage and social media engagement
- Analyze the correlation between philanthropic activities and company
reputation
- Review the alignment of the philanthropy program with the company’s mission
and values
- Monitor the long-term outcomes of the philanthropic initiatives
- Collect data on the satisfaction levels of the stakeholders involved in the
program
- Assess the effectiveness of the philanthropy program in achieving the
company’s social responsibility goals
- Evaluate the cost-effectiveness of the philanthropy program in relation to the
benefits received
- Measure the extent to which philanthropic programs are meeting employee needs
- Evaluate the relative importance different employee segments attach to
intrinsic needs
- Analyze the benefits of philanthropy in terms of employee engagement, customer
loyalty, reputation capital, and market opportunities
- Identify key intermediate outcomes that could yield desired business behaviors
and benefits
- Use internal surveys to assess the extent to which the philanthropic program
meets employee needs and creating a greater sense of identity between employee
and employer
- Leverage the positive impact of corporate philanthropy on employee recruitment
- Integrate the evaluation of employee well-being and satisfaction as a key
performance indicator in the corporate philanthropy program's success
measurement.
------------------------------
Turn 8...
CEO >>> How can we incorporate a system to measure the long-term social and
environmental impact of our corporate philanthropy initiatives on the
communities and ecosystems we support?
------------------------------
Corgi-AI >>> To incorporate a system to measure the long-term social and environmental impact
of corporate philanthropy initiatives, we can review recent measurement studies,
models, and evidence from complementary business disciplines and the social
sector. We can generalize the most valuable concepts and recognize the strengths
and limitations of various measurement approaches. We can also consult resources
like the TRASI database, which identifies 150 different approaches currently
used to measure the social impact of programs. Additionally, we can use social
performance and financial performance literature to understand the link between
corporate social performance and financial performance. Finally, we can adapt
current methodologies in the field to our own corporate giving programs.
------------------------------
CGO >>> - Measure the amount of monetary donations
- Track the number of volunteer hours contributed
- Assess the impact of donations through beneficiary feedback
- Evaluate the visibility and reach of the philanthropy program through media
coverage and social media engagement
- Analyze the correlation between philanthropic activities and company
reputation
- Review the alignment of the philanthropy program with the company’s mission
and values
- Monitor the long-term outcomes of the philanthropic initiatives
- Collect data on the satisfaction levels of the stakeholders involved in the
program
- Assess the effectiveness of the philanthropy program in achieving the
company’s social responsibility goals
- Evaluate the cost-effectiveness of the philanthropy program in relation to the
benefits received
- Measure the extent to which philanthropic programs are meeting employee needs
- Evaluate the relative importance different employee segments attach to
intrinsic needs
- Analyze the benefits of philanthropy in terms of employee engagement, customer
loyalty, reputation capital, and market opportunities
- Identify key intermediate outcomes that could yield desired business behaviors
and benefits
- Use internal surveys to assess the extent to which the philanthropic program
meets employee needs and creating a greater sense of identity between employee
and employer
- Leverage the positive impact of corporate philanthropy on employee recruitment
- Integrate the evaluation of employee well-being and satisfaction as a key
performance indicator in the corporate philanthropy program's success
measurement.
- Incorporate a system to measure the long-term social and environmental impact
of corporate philanthropy initiatives
- Review recent measurement studies, models, and evidence from complementary
business disciplines and the social sector
- Generalize the most valuable concepts and recognize the strengths and
limitations of various measurement approaches
- Consult resources like the TRASI database, which identifies 150 different
approaches currently used to measure the social impact of programs
- Use social performance and financial performance literature to understand the
link between corporate social performance and financial performance
- Adapt current methodologies in the field to our own corporate giving programs.
------------------------------
Turn 9...
CEO >>> How can we integrate employee perspectives more deeply into our corporate
philanthropy program's success measurement to ensure alignment with our
company's culture and values while also fostering employee engagement and
satisfaction?
------------------------------
Corgi-AI >>> To integrate employee perspectives more deeply into our corporate philanthropy
program's success measurement, we should consider the following steps:
1. Conduct internal surveys to assess the extent to which the philanthropic
program is meeting employee needs and creating a greater sense of identity
between employee and employer.
2. Take into account the relative importance that different employee segments
attach to different intrinsic needs.
3. Analyze the conversations and analyses summarized above to identify potential
areas for improvement and alignment with our company's culture and values.
4. Consider complementary disciplines such as human resources, marketing, risk
management, and capital budgeting to improve measurement methods and identify
long-term financial benefits.
5. Document success in attracting top candidates based on their exposure to the
company's corporate philanthropy program.
By following these steps, we can better understand and measure the impact of our
corporate philanthropy program on employee engagement, satisfaction, and
alignment with our company's culture and values.
------------------------------
CGO >>> - Measure the monetary donations and volunteer hours.
- Assess the impact through beneficiary feedback, media coverage, social media
engagement, and company reputation.
- Review the alignment with the company's mission and values, long-term
outcomes, and stakeholder satisfaction.
- Analyze the philanthropy program's effectiveness in achieving social
responsibility goals, cost-effectiveness, and meeting employee needs.
- Evaluate the importance of philanthropy for different employee segments,
benefits in terms of engagement, loyalty, reputation, and market opportunities.
- Identify key intermediate outcomes for business behaviors and benefits.
- Use internal surveys to assess meeting employee needs and creating a sense of
identity.
- Leverage philanthropy's impact on employee recruitment.
- Integrate employee well-being and satisfaction as KPIs.
- Measure the long-term social and environmental impact.
- Review measurement studies, models, and evidence from complementary
disciplines.
- Consult resources like the TRASI database for social impact measurement
approaches.
- Understand the link between corporate social and financial performance.
- Adapt current methodologies to our corporate giving programs.
- Conduct internal surveys for employee needs and identity.
- Consider intrinsic needs of different employee segments.
- Analyze conversations and analyses for improvement and alignment.
- Include complementary disciplines for measurement and long-term benefits.
- Document success in attracting top candidates through the program.
------------------------------
Turn 10...
CEO >>> How can we incorporate a more holistic approach to measure the intangible
benefits, such as employee morale and community relations, in addition to the
tangible outcomes of our corporate philanthropy program?
------------------------------
Corgi-AI >>> To incorporate a more holistic approach to measure the intangible benefits of
corporate philanthropy, such as employee morale and community relations, in
addition to tangible outcomes, we should consider the following strategies:
1. Develop clear strategies by which philanthropic initiatives contribute
towards strategic business needs, such as improved employee engagement, customer
loyalty, reputational risk, and growth opportunities. This involves
understanding the mechanisms by which these business benefits are expected to be
achieved.
2. Utilize complementary disciplines such as human resources, marketing, risk
management, and capital budgeting to improve measurement methods. This includes
applying modeling approaches for valuing future cash flows, analyzing scenarios,
and calibrating expected monetary profits linked to the behaviors of loyal
customers and engaged employees.
3. Conduct measurement studies, models, and gather evidence from complementary
business disciplines and the social sector. This can provide a starting point
for adapting current methodologies in the field to corporate giving programs.
4. Recognize the strengths and limitations of various measurement approaches and
generalize the most valuable concepts from recent measurement studies and
evidence.
5. Treat philanthropy as a professional, proactive, and strategic activity, not
just another "check in the box." This involves executing corporate philanthropy
with the same level of professionalism and strategic planning as other core
business activities.
By implementing these strategies, companies can better articulate and quantify
the long-term financial benefits and intangible assets created by philanthropic
initiatives, thereby making a more persuasive business case for their investment
in corporate philanthropy.
------------------------------
CGO >>> - Develop strategies linking philanthropic initiatives to strategic business
needs (employee engagement, customer loyalty, reputational risk, growth
opportunities).
- Utilize complementary disciplines (human resources, marketing, risk
management, capital budgeting) for improved measurement methods.
- Conduct measurement studies, models, and gather evidence from complementary
business disciplines and the social sector.
- Recognize strengths and limitations of measurement approaches, applying
valuable concepts from recent studies.
- Treat philanthropy as a professional, strategic activity.
- Articulate and quantify long-term financial benefits and intangible assets
created by philanthropy.
- Leverage internal surveys to assess employee needs, identity, and benefits
(engagement, loyalty, reputation, market opportunities).
- Evaluate philanthropy's impact on employee recruitment and well-being.
- Measure long-term social and environmental impact.
- Integrate employee well-being and satisfaction as KPIs.
- Document success in attracting top candidates through the program.
- Consult resources like the TRASI database for social impact measurement
approaches.
- Understand the link between corporate social and financial performance.
- Adapt current methodologies to corporate giving programs.
- Analyze conversations and analyses for improvement and alignment.
------------------------------
# Display the final turn
print("FINAL ANSWER")
print(wrap(memory))
FINAL ANSWER
- Develop strategies linking philanthropic initiatives to strategic business
needs (employee engagement, customer loyalty, reputational risk, growth
opportunities).
- Utilize complementary disciplines (human resources, marketing, risk
management, capital budgeting) for improved measurement methods.
- Conduct measurement studies, models, and gather evidence from complementary
business disciplines and the social sector.
- Recognize strengths and limitations of measurement approaches, applying
valuable concepts from recent studies.
- Treat philanthropy as a professional, strategic activity.
- Articulate and quantify long-term financial benefits and intangible assets
created by philanthropy.
- Leverage internal surveys to assess employee needs, identity, and benefits
(engagement, loyalty, reputation, market opportunities).
- Evaluate philanthropy's impact on employee recruitment and well-being.
- Measure long-term social and environmental impact.
- Integrate employee well-being and satisfaction as KPIs.
- Document success in attracting top candidates through the program.
- Consult resources like the TRASI database for social impact measurement
approaches.
- Understand the link between corporate social and financial performance.
- Adapt current methodologies to corporate giving programs.
- Analyze conversations and analyses for improvement and alignment.
For attribution, show the source reference documents which supported the responses.
# source document IDs of text chunks retrieved are tracked for attribution
print("Source documents reference id's:")
print(set(docs))
Source documents reference id's:
{'87', '190', '95', '84', '5', '12', '195', '86', '97', '0', '8', '102', '103', '4', '88', '151', '152', '1', '9'}
Multimodal#
Multi-lingual#
LLM agents built on the Phi-4-multimodal model can take advantage of its multilingual capabilities, supporting over 20 languages including English, French, German, Hindi, Italian, Portuguese, Spanish, and Thai. To evaluate language-specific performance, benchmarks and leaderboards based on MMLU and other open datasets are available. For example:
https://huggingface.co/spaces/uonlp/open_multilingual_llm_leaderboard
# Translate english to french
french = pipe(f"Please translate to French: {memory}", max_new_tokens=512)
print(wrap(french))
- Développer des stratégies reliant les initiatives philanthropiques aux besoins
stratégiques des affaires (engagement des employés, fidélité des clients, risque
de réputation, opportunités de croissance).
- Utiliser des disciplines complémentaires (ressources humaines, marketing,
gestion des risques, budgétisation des capitaux) pour des méthodes de mesure
améliorées.
- Conduire des études de mesure, des modèles et rassembler des preuves des
disciplines commerciales complémentaires et du secteur social.
- Reconnaître les forces et les limites des approches de mesure, en appliquant
des concepts précieux des études récentes.
- Traiter la philanthropie comme une activité professionnelle et stratégique.
- Articuler et quantifier les avantages financiers à long terme et les actifs
intangibles créés par la philanthropie.
- Utiliser des enquêtes internes pour évaluer les besoins, l'identité et les
avantages des employés (engagement, fidélité, réputation, opportunités de
marché).
- Évaluer l'impact de la philanthropie sur la recrutement des employés et leur
bien-être.
- Mesurer l'impact à long terme social et environnemental.
- Intégrer le bien-être et la satisfaction des employés comme indicateurs de
performance (KPIs).
- Documenter le succès dans l'attraction de candidats exceptionnels grâce au
programme.
- Consulter des ressources comme la base de données TRASI pour les approches de
mesure de l'impact social.
- Comprendre la relation entre la performance sociale et financière de
l'entreprise.
- Adapter les méthodologies actuelles aux programmes de dons d'entreprise.
- Analyser les conversations et les analyses pour l'amélioration et
l'alignement.
# Translate back to english
print(wrap(pipe(f"Please translate to English: {french}", max_new_tokens=512)))
- Develop strategies that link philanthropic initiatives to the strategic
business needs (employee engagement, customer loyalty, reputation risk, growth
opportunities).
- Use complementary disciplines (human resources, marketing, risk management,
capital budgeting) for improved measurement methods.
- Conduct measurement studies, models, and gather evidence of the complementary
commercial disciplines and the social sector.
- Recognize the strengths and limitations of measurement approaches, applying
valuable concepts from recent studies.
- Treat philanthropy as a professional and strategic activity.
- Articulate and quantify the long-term financial benefits and intangible assets
created by philanthropy.
- Use internal surveys to evaluate the needs, identity, and benefits of
employees (engagement, loyalty, reputation, market opportunities).
- Evaluate the impact of philanthropy on employee recruitment and well-being.
- Measure the long-term social and environmental impact.
- Integrate employee well-being and satisfaction as performance indicators
(KPIs).
- Document the success in attracting exceptional candidates through the program.
- Consult resources like the TRASI database for social impact measurement
approaches.
- Understand the relationship between the social performance and financial
performance of the company.
- Adapt current methodologies to corporate donation programs.
- Analyze conversations and analyses for improvement and alignment.
Vision language#
Perform optical character recognition (OCR) and chart and table interpretation of figure 6 from the MVCP document.
Multimodal evaluation of agents for image and video analysis can be performed with more comprehensive benchmarks and datasets, such as:
bradyfu/awesome-multimodal-large-language-models
# Load and display a test image
image = Image.open("assets/lim-fig6.png")
plt.imshow(image)
plt.axis('off')
(np.float64(-0.5), np.float64(953.5), np.float64(785.5), np.float64(-0.5))
print(wrap(pipe("Describe what is presented in this image", image=image, max_new_tokens=512)))
The image is a diagram titled "Figure 6: A Framework for Measuring Employee
Engagement and Corporate Philanthropy." It is divided into several sections,
each with a specific focus. The top left section is labeled "CORPORATE
PHILANTHROPY ACTIVITIES" and includes examples such as grants and employee
volunteer programs. The top right section is labeled "BUSINESS IMPACT" and lists
outcomes like increased output, sales, and profitability. Below these sections,
there are three main columns. The first column on the left is labeled "EMPLOYEE
NEEDS FULFILLED" and includes points such as self-enhancement, work-life
integration, reputational shield, bridge to company, and collective self-esteem.
The middle column is labeled "INTERMEDIATE OUTCOME TO BE TARGETED AND MEASURED"
and includes "EMPLOYEE ATTITUDES" and "Sense of organizational identification."
The third column on the right is labeled "JOB-RELATED BEHAVIORS" and lists
outcomes like reduced absenteeism, retention, efficiency, co-operative
behaviors, work effort, and advocacy. At the bottom, there is a section labeled
"OTHER MODERATING FACTORS" which includes extrinsic incentives, employee
characteristics, and employee perception of HR practices, work environment,
management, and company capabilities. The source of the diagram is cited as
adapted from Bhattacharya, C. B., Sen, S., & Korschun, D. (2008) and Bartel, C.
(2001).
Probing the model’s image reasoning capabilities:
# Load and display a test image
image = Image.open("assets/MSDTRPL_EC013-H-2020-1602198302.webp")
plt.imshow(image)
plt.axis('off')
(np.float64(-0.5), np.float64(999.5), np.float64(562.5), np.float64(-0.5))
print(wrap(pipe("What is shown in this image?", image=image, max_new_tokens=512)))
The image depicts a large crowd of people, many of whom are holding up their
hands. The crowd appears to be in a state of excitement or celebration, with
some individuals raising their hands in the air. The people are dressed in a
variety of clothing, including suits and casual attire. The background is filled
with more people, suggesting that this is a public event or gathering.
print(wrap(pipe("Identify the movie that this scene is taken from, and explain your reasoning",
image=image, max_new_tokens=128)))
This scene is from the movie 'Forrest Gump.' The reasoning behind this
identification includes the distinctive style of the crowd, the casual yet
significant attire of the individuals, and the overall chaotic yet focused
atmosphere typical of a pivotal moment in the film. The scene captures a moment
of intense public reaction, which is a recurring theme in the movie.
print(wrap(pipe("Please give your best estimate of the number of people in this image",
image=image, max_new_tokens=32)))
There are approximately 30 people visible in the image.
print(wrap(pipe("Briefly recount a specific instance of corporate philanthropy in the plot of the movie Trading Places?",
max_new_tokens=512)))
In the movie "Trading Places," a notable instance of corporate philanthropy
occurs when Louis Winthorpe III, played by Eddie Murphy, and Billy Ray
Valentine, played by Dan Akroyd, are swapped due to a bet. Louis, who is a
successful commodities trader, is given a chance to live a life of luxury and
privilege, while Billy, a struggling street trader, is given Louis' life.
During this period, Louis, who is now living a life of luxury, decides to use
his newfound wealth to help those in need. He donates a significant amount of
money to a local homeless shelter, which is a clear act of corporate
philanthropy. This act not only helps the shelter but also serves as a turning
point in Louis' character development, as he begins to understand the value of
giving back to the community.
This instance of corporate philanthropy in the movie highlights the importance
of using wealth and resources to help those in need, and it also serves as a
reminder that even those who are successful can make a positive impact on
society.
Audio#
Finally, perform automatic speech recognition (ASR) and speech translation on an audio input.
Rigorous open benchmark datasets for evaluation are also available, e.g.
huggingface/open_asr_leaderboard
# Perform speech-to-text on an audio file
url = "assets/pork-bellies!.mp3"
audio, samplerate = soundfile.read(url)
response = pipe("Transcribe the audio to text",
audio=[(audio, samplerate)], max_new_tokens=512)
print(wrap(response))
Pork bellies! I have a hunch something very exciting is going to happen in the
pork belly market this morning.
from playsound import playsound
playsound(url) # play the original sound file
References:
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal†, Heinrich Küttler, Mike Lewis†, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela, 2021, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Danqi Chen, Adam Fisch, Jason Weston, Antoine Bordes. (2017), “Reading Wikipedia to Answer Open-Domain Questions.”
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham (2023), “In-Context Retrieval-Augmented Language Models.”
Kyle Mahowald and Anna A. Ivanova and Idan A. Blank and Nancy Kanwisher and Joshua B. Tenenbaum and Evelina Fedorenko, 2024, Dissociating language and thought in large language models, https://arxiv.org/abs/2301.06627
Hicks, Michael Townsen, Humphries, James and Slater, Joe, 2024, “ChatGPT is bullshit”, Ethics and Information Technology Volume 26, Issue 2, https://doi.org/10.1007/s10676-024-09775-5
Parth Sarthi and Salman Abdullah and Aditi Tuli and Shubh Khanna and Anna Goldie and Christopher D. Manning, 2024, RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval, https://arxiv.org/abs/2401.18059
Lim, Terence, 2010, Measuring the value of corporate philanthropy: Social impact, business benefits, and investor returns. published by CECP.