The first wave of generative artificial intelligence (GenAI) solutions has already achieved considerable success in companies, particularly in the area of coding assistants and in increasing the efficiency of existing SaaS products. However, these applications only show a small glimpse of what is possible with large language models (LLMs). The real strength of this technology is now unfolding in the second generation of AI-powered applications: agent-based systems that build on the solid foundation of LLMs and take their capabilities to a new level.
Unlike traditional AI chatbots or simple copilots for SaaS products, AI agents leverage the full range of LLM capabilities. They can not only generate text, but also solve complex problems (almost) independently. An AI agent is an LLM-powered system that pursues specific goals within defined autonomy boundaries and uses various tools.
An example illustrates the possibilities: Imagine that an LLM receives the documentation for an API that can retrieve current stock prices. With this information, the LLM can independently create a script that calls this API for a specific stock price. If the system is allowed to execute this script, it becomes a tool for end users to retrieve any stock prices.
An agent rarely acts alone
So, would it be possible to develop a single, comprehensive application that could solve all of a company’s problems? The short answer is no. Although LLMs are capable of generalization, the constraints of the enterprise environment require a relatively narrow scope for each individual application. This is the only way for the company to ensure consistent performance and control access to data and tools.
This imaginary “super application” sounds convenient, but it would require full access to all company data and tools, from the most mundane to the most sensitive. Just as an employee should only have access to the data and tools necessary for their work, an agent-based application’s access must be limited to what it needs to perform its function.
How many such AI agents might a large company need? A rough estimate: A large company with ten departments, each with five core functions, could benefit from five specialized applications per function. For example, the sales department in sales operations could use one agent to:
- Research target customers
- Verify compliance with the sales process
- Analyze the sales pipeline
- Summarize customer meetings
- Support follow-up activities
This would result in a total of 250 applications — a realistic forecast for large organizations.
Build or buy: The art of making the right decision
With hundreds of new applications available, companies are faced with a “build or buy” decision. Software providers are already bringing corresponding applications to market. At the same time, advanced organizations are developing their first production-ready agent-based applications in-house.
Ready-made AI agents offer decisive advantages: Once implemented, they provide turnkey performance, are supported by professional software developers and enable rapid deployment. The downside of such off-the-shelf AI agents is challenges such as the often complex integration into existing enterprise systems, governance issues in tracking the models used and, last but not least, the fact that competitors with identical solutions can achieve the same performance.
On the other hand, self-developed, customized AI agents can be precisely adapted to the specific business context and thus offer the potential for real differentiation in the market. Companies retain full control and transparency over their applications and gain independence from external software, AI and cloud providers. However, these advantages come with their own challenges: The necessary development skills are not sufficiently available in many organizations, and as the number of self-developed applications grows, the complexity of monitoring and maintenance increases significantly.
Most companies will therefore purchase some applications and develop others themselves. It should be noted that while AI agents can improve the efficiency of operational processes, if these efficiency gains keep pace with those of competitors, the competitive position will not improve.
Developing customized, agent-based applications, on the other hand, enables a company to create capabilities that its competitors do not have. Given the costs and complexity involved, companies will focus their internal development efforts on those parts of their business that will benefit most from strong competitive differentiation, typically their core business.
Overcoming the complexity threshold
Every organization has a maximum number of applications that it can develop, monitor and maintain using its current practices. This is the organization’s “complexity threshold.”
As the company develops more and more agent-based applications, the complexity of monitoring and maintaining them increases until, at some point, maximum complexity is reached and no further applications can be developed — even if this would benefit the business.
The company must find a way to raise its complexity threshold in order to develop more applications. This requires standardizing and structuring the development of these applications.
A new architecture paradigm for agent-based applications
Over the years, organizations have used various architectural paradigms, from monolithic applications to service-oriented architectures to microservices. Today’s infrastructures for building agent-based applications are mostly monolithic and use frameworks such as LangChain. While this approach is suitable for developing initial prototypes, it reflects the relative immaturity of agent-based application design in the enterprise environment.
A new architectural paradigm is therefore needed to create and maintain many agent-based applications. One implementation is a mesh architecture for LLMs and the associated components required to create agents in the enterprise. It provides abstraction layers that group different components into uniform object types.
Such a multi-layer architecture could include the following components:
- Base models: The trained AI models with their basic mathematical weights.
- Data layer: Divided into unstructured and structured data.
- Service layer: Includes the services required for model operation as well as data access services.
- Orchestration layer: This is where prompts, agents and tools are brought together.
- Application layer: The usable applications with their user interfaces.
The key advantage of such an architecture lies in abstraction: It creates standardized interfaces between the different layers, allowing a single service to be replaced without having to adapt other components. This decoupling enables companies to increase their complexity threshold.
Kurt Muehmel is the head of AI strategy at Dataiku. He is a creative and analytical executive with 15+ years of experience and foundational expertise in the Enterprise AI space and, more broadly, B2B SaaS go-to-market strategy and tactics.
