Due to the evolving nature of AI and the fear of missing out (FOMO), generative AI initiatives are often imposed from the top down, and business leaders can tend to get too excited about this revolutionary technology. But when companies rush to build and deploy, they often face all the typical problems that arise with other technology implementations. AI is complex and requires specialized expertise, meaning some organizations quickly find themselves overwhelmed.

In fact, Forrester predicts that almost three quarters of organizations this attempt to create AI agents in-house will fail.

“The challenge is that these architectures are convoluted, requiring multiple models, RAG (recovery augmented generation) stacks, advanced data architectures and specialized expertise. » write Forrester analysts Jayesh Chaurasia and Sudha Maheshwari.

So how can businesses choose when to adopt third-party models, open source tools, or create custom, fine-tuned models in-house? The experts intervene.

AI architecture is much more complex than businesses think

Organizations that attempt to create agents themselves often have difficulty augmented recovery generation (RAG) and vector databases, Forrester principal analyst Rowan Curran told VentureBeat. It can be difficult to get accurate results on time, and organizations don’t always understand the process (or importance) of reclassification, which helps ensure the model is working with the highest quality data .

For example, a user might enter 10,000 documents and the model might return the 100 most relevant to the task at hand, Curran pointed out. But short pop-ups limit what can be used for reclassification. So, for example, a human user may have to use judgment and choose 10 documents, thereby reducing the accuracy of the model.

Curran noted that building and optimizing RAG systems can take 6 to 8 weeks. For example, the first iteration might have an accuracy rate of 55% before any adjustments; the second version could have 70% and the final deployment will ideally be closer to 100%.

Developers need to understand the availability (and quality) of data and know how to reclassify, iterate, evaluate, and anchor a model (i.e., match model results to relevant, verifiable sources). Additionally, raising or lowering the temperature determines how creative a model is — but some organizations are “really limited” in creativity, which limits things, Curran said.

“It feels like there’s a simple button around this stuff,” he noted. “There just aren’t any.”

A lot of human effort is required to build AI systems, Curran said, emphasizing the importance of testing, validation and ongoing support. All of this requires dedicated resources.

“It can be complex to successfully deploy an AI agent,” acknowledged Naveen Rao, vice president of AI at Data bricks and founder and former CEO of MosaicAI. Enterprises need access to various large language models (LLMs) and also have the ability to govern and monitor not only the agents and models, but also the underlying data and tools. “This is not a simple problem and, as time passes, AI systems will come under increasing scrutiny over what data is accessed and how it is accessed. »

Factors to Consider When Exploring AI Agents

When considering options for deploy AI agents — third-party, open source or custom — businesses should take a controlled, tactical approach, experts advise.

Start by considering several important questions and factors, recommended Andreas Welsch, founder and chief AI strategist of a consulting firm. Intelligence briefing. These include:

• Where does your team spend the majority of their time?
• Which tasks or steps in this process take the most time?
• How complex are these tasks? Do they involve accessible IT systems and data?
• What would make your business faster or more profitable? And can you (and how) measure reference points?

It’s also important to consider existing licenses and subscriptions, Welsch emphasized. Talk with software salespeople to understand if your company already has access to agent features and, if so, what it would take to use them (like add-ons or higher-level subscriptions).

From there, look for opportunities in a sales role. For example: “Where is your team spending time on multiple manual steps that cannot be described in code? » Later, when exploring agents, discover their potential and “sort out” any gaps.

Also be sure to empower and educate teams by showing them how agents can help them in their work. “And don’t be afraid to mention agent limitations as well,” Welsch said. “This will help you manage expectations.”

Build a strategy, adopt a transversal approach

When developing an enterprise AI strategy, it’s important to take a cross-functional approach, Curran emphasized. Successful organizations involve multiple departments in this process, including business leadership, software development and data science teams, user experience managers, and others.

Develop a roadmap based on the company’s core principles and goals, he advised. “What are our goals as an organization and how will AI enable us to achieve these goals?” »

This can be difficult, probably because technology evolves so quickly, Curran acknowledged. “There is no set of best practices, no frameworks,” he said. Few developers have experience with post-release integrations and DevOps when it comes to AI agents. “The skills needed to build these things haven’t really been developed and quantified in a widespread way.”

As a result, organizations struggle to launch AI projects (of any kind), and many end up turning to a consulting firm or one of their existing technology providers who have the resources and capabilities needed to build on their technology stacks. Ultimately, organizations will be more successful when they work closely with their partners.

“Third-party providers will likely have the bandwidth to keep up with the latest technologies and architectures needed for this implementation,” Curran said.

This is not to say that it is impossible to create custom agents in-house; quite the contrary, he noted. For example, if a company has a strong internal development team and RAG and machine learning (ML) architecture, it can use it to create its own agentic AI. That’s also true if “your data is well governed, documented and labeled” and you don’t have a “giant mess” of API strategy, he emphasized.

Regardless, businesses must consider ongoing post-deployment needs in their AI strategies from the outset.

“There is no free lunch after deployment,” Curran said. “All of these systems require some type of post-launch maintenance and support, ongoing tweaks and adjustments to keep them accurate and make them more accurate over time.”