Episode 15: Week 2 of Building Multi-Agent Applications - A Bootcamp
I have recently joined a mentorship program for building Multi-Agent LLM Applications and will be sharing my learnings in a series of detailed blog posts over the next couple of weeks.
Check out Week 1 post.
Updates from Week 2:
Product Requirement Document:
💡 We learned how to create a Product Requirements Document (PRD) while building an Agentic Application. This post highlights the key factors to consider when scoping out such a project.
Automated Data Scientist:
💡 The default project in this bootcamp is to build an “Automated Data Scientist” that can generate, train, and evaluate machine learning models for a given user problem with as little human intervention as possible (and potentially beat humans at Kaggle competitions! 😉) This week’s post uses the default project as an example to demonstrate how to create a PRD.
CrewAI:
💡 I played around with CrewAI to build a rudimentary multi-agent system that can create research reports based on internal and external data. I have used the International Passenger Survey data collected by the Office of National Statistics - UK’s national statistical institute and taught the model to present findings following Expedia Group’s yearly travel reports as a gold standard. In my opinion, this use case is an excellent starting point because it allows us to compare the system’s response with actual reports published by human researchers, helping us identify the patterns that agents can pick up or miss compared to humans. Demo at the end of the blog post!
Demo Day:
💡 On Sunday, we had demo day 🚀 to present our PRDs, with Megan Risdal and Yerzat Marat as adjudicators. Lots of fun projects. My team and I are working on a multi-agent system that can act as a BI assistant for performing routine BI tasks, thereby unblocking data analysts for more complex projects (More on that in coming weeks!)
Let’s get right in!
The Basics:
What are some of the traditional software design priciples that can be applied to agentic applications? [1]
- Domain-Driven Design (DDD)
- Service-Oriented Architecture (SOA)
- Microservices architecture
These principles behave differently in agentic applications. Instead of pre-programmed routing between one agent (service) to another, agentic applications have probablistic routing between different agents i.e. an agent can consult other agents only if it thinks it would help provide the requested service more efficiently. This not only enhances the functionality of each component but also introduces new capabilities. These agents (formerly services) can communicate with each other via not only traditional methods like JSON and DSL, but also through natural language!
What is domain-driven design and why is it important while building agentic applications? [1]
When you start building an agentic product for a client, it is important to break down the domain of the business into bounded contexts - well-defined areas of the business which have independent responsibilities and for each one of them, you can set the criteria for success separately. These components allow non-technical and technical stakeholders to have a common vocabulary while discussing the project.
Example:
You are building an agentic workflow for an online retail-store. In this case, you can broadly divide the business into these departments:
- Customer service
- Inventory management
- Order processing
- Content management
In order to transform the existing workflows into agentic, it is important to define them separately for each department, understand how they function currently and then decide the success metrics for the new agentic workflow to be evaluated on, for each of these departments separately.
What is microservices architecture in service oriented architecture? [2]
Mapping each domain to a service comes under service oriented architecture. Breaking those services down into smaller services with dedicated functionalities is called building a microservices based architecture. This is the secret sauce of building modular, scalable software systems!
Now how can this be applied to agentic applications?
In the paper by Boye Niu et al. “Flow: A Modular Approach to Automated Agentic Workflow Generation”, they discuss the following principles:
- Agentic theory focuses on creating intelligent agents that can act independently, reason, and make decisions within defined environments.
- Designing modular agents that can autonomously perform tasks like reasoning, planning, and executing actions.
- In microservices-based service-oriented architecture (SOA), the system is broken into small, independent services, each handling a specific task (like a module for data processing, another for storage, etc.).
- Think of Flow as an agent-based version of microservices: instead of small services, you have intelligent agents that interact to complete workflows.
- The services are no longer interacting passively with the environment and doing as they are told, they have now become active and can intelligently change the course of the workflow as needed.
This paper is a good example of how to measure and present improvements in efficiency of workflows before and after using agents.
The Bootcamp
The key to building a good agentic product is to NOT start with the fanciest thing that an agent can do, rather with the most impactful task in the current workflow that it can automate, thereby unblocking humans for more complex tasks.
A product requirements document (PRD) should have the following components:
High Level Summary
This should give a sense of what the product specifically does. It should be as concrete as possible for the target audience it is being built for.
Example:
An intelligent agent that can solve routine data science requests made by different departments within an organization: It can generate, train, and evaluate machine learning models for a given user problem.
- Target audience: Sales team in real estate
- Target audience: Marketing departments and so on …
Business Objective
This section deals with how this product will save/make money.
- Save Money:
- Hiring data scientists costs a lot of money. Average salary for a data scientist in the US is around $120-150k.
- Having an agentic data scientist can be a cost-effective alternative.
- Unblock data scientist for more complex tasks:
- When the agents take care of run-of-the-mill requests, data scientists can focus on more complex tasks e.g. gathering extremely contextualized information from stakeholders, come up with new experiments and models to launch and make further improvements in the company’s product/operations. This will increase their productivity and increase their capacity.
- Unblock sales and business development teams to create dirty POCs and quick demos faster for the end user.
- Remove marketing teams’ dependency on DS teams for quick analysis and modeling related to their experiments.
- Win Kaggle competitions.
- To enable founders to quickly prototype several different ideas/pipeline without depending on a technical teams.
While planning this section, it helps to know the target audience for your product. The examples above are just to show how this product can be developed for different stakeholders i.e. data scientist, sales departments etc. but in the end, for a particular product, we have to narrow our scope to one set of audience/objective and decide whether we are building to augment or replace the existing workflows. Ideally, we should always start with augmenting the existing workflows rather than replacing them, starting off with the target audience that can be benefited the most from the new workflow and improve it over several iterations.
User’s Current Workflow
This section deals with how things are being run now without the agentic workflow.
Example:
- Data Scientists create JIRA tickets based on the requests coming from different departments.
- For each project, a DS needs to gather requirements, collect data, clean it and set up the model needed to perform the task.
- After deployment, the data scientists also need to monitor the model, collect results, analyze and present the findings. This can involve collaboration with data engineers and analysts as well.
Assumptions
We are assuming that the new agentic system is going to abide by certain rules that exist currently.
Example:
- System will handle increased complexity and data volume without significant performance degradation.
- System will fail with humility if it is unable to answer correctly and defer to an actual human data scientist for validating its findings when needed.
- The tool will seamlessly integrate with multiple data sources but the tool will minimize human intervention for routine DS processes.
- User roles and access control will be there in case someone is trying to access information they are not allowed to access.
Success Criteria
How will the customer who is buying this product make sure that its launch was a success.
- Efficiency improvement in the current flow:
- can be measured by the JIRA tickets closed by agentic data scientist vs traditional data scientists in the team.
- Accuracy metrics of the model developed by automated data scientist vs made by traditional data scientists.
Dependencies
- Data integration:
- Can the agentic product be integrated with any data source?
- Data tools:
- What deployment and visualization tools does the company currently use? Can the output of the automated data scientist automatically use those tools or need the assistance of the data scientist to do that.
Demo Time
This week I played with Crew.ai. It is an AI agent building and orchestration platform that helps to build agentic workflows. A crew in CrewAI represents a collaborative group of agents working together to achieve a set of tasks. This workflow consists of three agents - lead_research_analyst, chief_policy_advisor, presenter to come up with a policy report to improve UK’s travel statistics. It learns from Expedia’s yearly reports to understand how to give trend updates but uses UK’s data and searches the web to build the report. This is a good place to start because we already have a human created report and we can compare the report generated by AI, which has access to more tools and data, to see what new things it looked at which the humans ignored!
References:
[1] LLM Agents, Part 3 - Multi-Agent LLM Products: A Design Pattern Perspective
[2] Flow: A Modular Approach to Automated Agentic Workflow Generation
[4] Git repo - crewAI-examples
[3] Git repo - Youtube-trend-analysis using crewai
[4] Git repo - multi-agent-analysts