How to Build an AI Agent With Semantic Router and LLM Tools

Your bag-of-docs representation isn’t helpful for humans, don’t assume it’s any good for agents. Think carefully about how you structure your context to underscore the relationships between parts of it, and make extraction as simple as possible. We’ve found that taking the final prompt sent to the model—with all of the context construction, and meta-prompting, and RAG results—putting it on a blank page and just reading it, really helps you rethink your context. We have found redundancy, self-contradictory language, and poor formatting using this method.

There are many more advanced examples out there it can be an amazing way to lower the technical barrier for people to gain insights from complicated data. Vincent is also a former post-doc at Cambridge University, and the building llm from scratch National Institute of Statistical Sciences (NISS). He published in Journal of Number Theory,  Journal of the Royal Statistical Society (Series B), and IEEE Transactions on Pattern Analysis and Machine Intelligence.

OpenAI Assistant Concepts

The 400M parameter DistilBART is another great option—when fine-tuned on open source data, it could identify hallucinations with an ROC-AUC of 0.84, surpassing most LLMs at less than 5% of latency and cost. Sometimes, our carefully crafted prompts work superbly with one model but fall flat with another. This can happen when we’re switching between various model providers, as well as when we upgrade across versions of the same model. By examining a sample of these logs daily, we can quickly identify and adapt to new patterns or failure modes. When we spot a new issue, we can immediately write an assertion or eval around it.

When working on a new application, it’s tempting to use the biggest, most powerful model available. But once we’ve established that the task is technically feasible, it’s worth experimenting if a smaller model can achieve comparable results. The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates.

To enhance the user experience, we set up a router that intelligently determines whether the query is related to flights, baggage or other conversational tasks like jokes or poems. This function fetches flight data from the AeroAPI and converts UTC times to the local time zones of the departure and arrival airports, which acts as the context to the LLM in providing real-time information about the flight schedules. OpenAI will generate embeddings for our queries, while ChromaDB will store and retrieve the embeddings for contextual data such as baggage policies. When an output fails the criteria, the text is amended by a feedback loop.

Such a high level of energy consumption has significant environmental effects as well. Now, your agent is aware of the world changing around it and can act accordingly. I like to have a metadata JSON object in my instructions that keeps relevant dynamic context. This allows me to pass in data while being less verbose and in a format that the LLM understands really well. To learn more about NVIDIA’s collaboration with businesses and developers in India, watch the replay of company founder and CEO Jensen Huang’s fireside chat at the NVIDIA AI Summit. India’s top global systems integrators are also offering NVIDIA NeMo-accelerated solutions to their customers.

The novel LiGO (Linear Growth Operator) approach we will discuss is setting a new benchmark. Bloomberg is a global leader in business and financial information, delivering trusted data, news, and insights that bring transparency, efficiency, and fairness to markets. The company helps connect influential communities across the global financial ecosystem via reliable technology solutions that enable our customers to make more informed decisions and foster better collaboration. Tech Mahindra, an Indian IT services and consulting company, is the first to use the Nemotron Hindi NIM microservice to develop an AI model called Indus 2.0, which is focused on Hindi and dozens of its dialects. Indus 2.0 harnesses Tech Mahindra’s high-quality fine-tuning data to further boost model accuracy, unlocking opportunities for clients in banking, education, healthcare and other industries to deliver localized services. Generative AI Insights provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss the challenges and opportunities of generative artificial intelligence.

Changes to the Way Enterprises Are Building and Buying Generative AI

Saving ‘Facts’ is only half of the story if we are hoping to be able to reuse previous LLM responses. Code generation cost and performance can be improved by implementing some sort of memory where information from previous identical requests can be retrieved, eliminating the requirement for repeat LLM calls. Solutions such as memgpt work with frameworks like autogen and offer a neat way of doing this. Accessing data directly through APIs means the data doesn’t have to be in a database and opens up a huge world of publically available datasets, but there is a catch.

Since LLaMa was licensed for research use only, a number of new providers have stepped in to train alternative base models (e.g., Together, Mosaic, Falcon, Mistral). Contextual data for LLM apps includes text documents, PDFs, and even structured formats like CSV or SQL tables. Data-loading and transformation solutions for this data vary widely across developers we spoke with. Some also use document loaders built into orchestration frameworks like LangChain (powered by Unstructured) and LlamaIndex (powered by Llama Hub). We believe this piece of the stack is relatively underdeveloped, though, and there’s an opportunity for data-replication solutions purpose-built for LLM apps. In this post, we’re sharing a reference architecture for the emerging LLM app stack.

Among the topics debated was whether the most fruitful approach for domain-specific generative AI in the legal industry was to build a legal large language model (LLM) from scratch or to fine-tune existing models to focus on legal work. Shreya Shankar is an ML engineer and PhD student in computer science at UC Berkeley. Jason Liu is a distinguished machine learning consultant known for leading teams to successfully ship AI products.

In reality, building machine learning or AI products requires a broad array of specialized roles. It can start as simple as the basics of prompt engineering, where techniques like n-shot prompting and CoT help condition the model toward the desired output. Folks who have the knowledge can also educate about the more technical aspects, such as how LLMs are autoregressive in nature. In other words, while input tokens are processed in parallel, output tokens are generated sequentially. As a result, latency is more a function of output length than input length—this is a key consideration when designing UXes and setting performance expectations. Finally, during product/project planning, set aside time for building evals and running multiple experiments.

I need answers that I can integrate in my articles and documentation, coming from trustworthy sources. Many times, all I need are relevant keywords or articles that I had forgotten, was unaware of, or did not know were related to my specific topic of interest. “We’ll definitely work with different providers and different models,” she says.

• Many patterns do something along these lines, passing the output of function calling back to the LLM.
• Data is saved after each section, allowing continuation in a new session if needed.
• In conclusion, while the allure of owning a bespoke LLM, like a fine-tuned version of ChatGPT, can be enticing, it is paramount for businesses to consider the feasibility, cost, and possible complications of such endeavours.
• If you’re not looking at different models, you’re missing the boat.” So RAG allows enterprises to separate their proprietary data from the model itself, making it much easier to swap models in and out as better models are released.
• They must process billions of parameters and learn complex patterns from massive textual data.

NeMo Curator uses NVIDIA RAPIDS libraries to accelerate data processing pipelines on multi-node GPU systems, lowering processing time and total cost of ownership. It also provides pre-built pipelines and building blocks for synthetic data generation, data filtering, classification and deduplication to process high-quality data. The approach is optimised to address task-specific requirements and industry nuances.

India Enterprises Serve Over a Billion Local Language Speakers Using LLMs Built With NVIDIA AI

Naturally, this has inspired many to ask how to get their hands on their ‘own LLM’, or sometimes more ambitiously, their ‘own ChatGPT’. Enterprises want a chatbot that is equipped with knowledge of information from their company’s documentation and data. At Netguru we specialize in designing, building, shipping and scaling beautiful, usable products with blazing-fast efficiency. Twenty-five years later, Andrej Karpathy took his first demo ride in a Waymo.

Nonetheless, rigorous and thoughtful evals are critical—it’s no coincidence that technical leaders at OpenAI work on evaluation and give feedback on individual evals. Additionally, keeping a short list of recent outputs can help prevent redundancy. In our recommended products example, by instructing the LLM to avoid suggesting items from this recent list, or by rejecting and resampling outputs that are similar to recent suggestions, we can further diversify the responses.

A common anti-pattern/code smell in software is the “God Object,” where we have a single class or function that does everything. The PositionWiseFeedForward class extends PyTorch’s nn.Module and implements a position-wise feed-forward network. The class initializes with two linear transformation layers and a ReLU activation function. The forward method applies these transformations and activation function sequentially to compute the output. This process enables the model to consider the position of input elements while making predictions. In the rapidly evolving world of generative AI, making the right choice requires understanding not just the available models but also how each aligns with your unique business goals.

For a customer-facing chatbot offering medical or financial advice, we’ll need a very high bar for safety and accuracy. But for less critical applications, such as a recommender system, or internal-facing applications like content classification or summarization, excessively strict requirements only slow progress without adding much value. Fortunately, many model providers offer the option to “pin” specific model versions (e.g., gpt-4-turbo-1106). This enables us to use a specific version of the model weights, ensuring they remain unchanged. By our calculations, we estimate that the model API (including fine-tuning) market ended 2023 around $1.5–2B run-rate revenue, including spend on OpenAI models via Azure.

For example, some private equity firms are experimenting with LLMs to analyze market trends and patterns, manage documents and automate some functions. The following four-step analysis can assist an organization in deciding whether to build its own LLM or work with a partner to facilitate an LLM implementation. Some examples of these are summarization evals, where we only have to consider ChatGPT App the input document to evaluate the summary on factual consistency and relevance. If the summary scores poorly on these metrics, we can choose not to display it to the user, effectively using the eval as a guardrail. Similarly, reference-free translation evals can assess the quality of a translation without needing a human-translated reference, again allowing us to use it as a guardrail.

This approach not only helps identify potential weaknesses, but also provides a useful source of production samples that can be converted into evals. Nonetheless, while fine-tuning can be effective, it comes with significant costs. We have to annotate fine-tuning data, finetune and evaluate models, and eventually self-host them. If prompting gets you 90% of the way there, then fine-tuning may not be worth the investment. However, if we do decide to fine-tune, to reduce the cost of collecting human annotated data, we can generate and finetune on synthetic data, or bootstrap on open-source data. To get the most juice out of them, we need to think beyond a single prompt and embrace workflows.

In response to this growing complexity in the LLM market, this article aims to summarise the five primary options available to businesses. I will be posting a set of follow-up blog posts detailing the technical implementation of Data Recipes as we work through user testing at DataKind. I will certainly leverage pre-crawled data in the future, for instance from CommonCrawl.org. However, it is critical for me to be able to reconstruct any underlying taxonomy.

In interviews, nearly 60% of AI leaders noted that they were interested in increasing open source usage or switching when fine-tuned open source models roughly matched performance of closed-source models. In 2024 and onwards, then, enterprises expect a significant shift of usage towards open source, with some expressly targeting a 50/50 split—up from the 80% closed/20% open split in 2023. In the table below drawn from survey data, enterprise leaders reported a number of models in testing, which is a leading indicator of the models that will be used to push workloads to production. For production use cases, OpenAI still has dominant market share, as expected. Over the past couple months, we’ve spoken with dozens of Fortune 500 and top enterprise leaders,2 and surveyed 70 more, to understand how they’re using, buying, and budgeting for generative AI.

Then return that same message back to the user, but this time, coming from that live thread. For the model, I chose the gpt-4-turbo-preview model so that we can add function calling in part 2 of this series. You could use gpt-3.5-turbo if you want to save a few fractions of a penny while giving yourself a migraine of pure frustration down the line when we implement tools.

Otherwise, you won’t know whether your prompt engineering is sufficient or when your fine-tuned model is ready to replace the base model. They required an incredible ChatGPT amount of safe-guarding and defensive engineering and remain hard to predict. You can foun additiona information about ai customer service and artificial intelligence and NLP. Additionally, when tightly scoped, these applications can be wildly useful.

The high-cost of collecting data and training a model is minimized—prompt engineering costs little more than human time. Position your team so that everyone is taught the basics of prompt engineering. This encourages everyone to experiment and leads to diverse ideas from across the organization. When deciding on the language model and level of scrutiny of an application, consider the use case and audience.

Cost of Building Large Language Models

Beyond LLM APIs, fine-tuning our specific tasks can also help increase performance. This is particularly relevant as we rely on components like large language models (LLMs) that we don’t train ourselves and that can change without our knowledge. A common source of errors in traditional machine learning pipelines is train-serve skew. This happens when the data used in training differs from what the model encounters in production. Although we can use LLMs without training or fine-tuning, hence there’s no training set, a similar issue arises with development-prod data skew.

This involves standard data cleaning tasks — such as removing duplicates and noise, and handling missing data — as well as labeling data to improve its utility for specific tasks, such as sentiment analysis. Depending on the task’s scope, this stage can also include augmenting the data set with synthetic data. Bryan Bischof is the Head of AI at Hex, where he leads the team of engineers building Magic – the data science and analytics copilot.

Despite their popularity, LLM models like GPT, Llama, and PaLM are only appropriate for downstream tasks (such as question answering and summarization) with few-shot prompting or additional fine-tuning. Although foundational models can function well in a wider context, they lack the industry or business-specific domain expertise necessary to be useful in most applications. Achieving great results in downstream tasks does not mean it will also have domain awareness for your specific industry. In some cases, they are trained on smaller datasets than commercial models.

This of course will not work well for massive data volumes, but it’s at least limiting ingestion based on user demand rather than trying to ingest an entire remote dataset. Another interesting aspect of this architecture is that it captures specific data analysis requirements and the frequency these are requested by users. This can be used to invest in more heavily utilized recipes bringing benefits to end users. For example, if a recipe for generating a humanitarian response situation report is accessed frequently, the recipe code for that report can improved proactively. By capturing data analysis requests from users and making these highly visible in the system, transparency is increased.

To sustain a competitive edge in the long run, you need to think beyond models and consider what will set your product apart. Paul Krill is an editor at large at InfoWorld, focusing on coverage of application development (desktop and mobile) and core web technologies such as Java. McKinsey tried to speed up writing evaluations by feeding transcripts of evaluation interviews to an LLM. But without fine-tuning or grounding it in the organization’s data, it was a complete failure, according to Lamarre. “The LLM didn’t have any context about the different roles, what kind of work we do, or how we evaluate people,” he says. Building generative AI applications powered by LLMs requires meticulous planning and execution to ensure high performance, security and ethical standards.

Similarly, any updates to failure mode definitions should be reflected in the evaluation criteria. These “vibe checks” are signals of bad outputs; code and assertions operationalize them. Finally, this attitude must be socialized, for example by adding review or annotation of inputs and outputs to your on-call rotation. Enterprise leaders are currently mostly measuring ROI by increased productivity generated by AI. While they are relying on NPS and customer satisfaction as good proxy metrics, they’re also looking for more tangible ways to measure returns, such as revenue generation, savings, efficiency, and accuracy gains, depending on their use case. In the near term, leaders are still rolling out this tech and figuring out the best metrics to use to quantify returns, but over the next 2 to 3 years ROI will be increasingly important.

Hosting companies like Replicate are already adding tooling to make these models easier for software developers to consume. There’s a growing belief among developers that smaller, fine-tuned models can reach state-of-the-art accuracy in narrow use cases. Looking ahead, most of the open source vector database companies are developing cloud offerings.

Techniques such as Automate-CoT can help automate this process by using the LLM itself to create chain-of-thought examples from a small labeled dataset. Interpretable rationale queries require LLMs to not only understand factual content but also apply domain-specific rules. These rationales might not be present in the LLM’s pre-training data but they are also not hard to find in the knowledge corpus. Building effective data-augmented LLM applications requires careful consideration of several factors.

An excellent example of this synergy is the enhancement of KeyBERT with KeyLLM for keyword extraction. If you, your team or your company design and deploy AI architecture, data pipelines or algorithms, having great diagrams to illustrate the workflow is a must. It will resonate well with busy professionals such as CTOs, with little time and possibly — like myself — limited experience playing with tools such as Canvas or Mermaid. For example, a customer service chatbot might need to integrate documented guidelines on handling returns or refunds with the context provided by a customer’s complaint. For example, techniques like Interleaving Retrieval with Chain-of-Thought (IRCoT) and Retrieval Augmented Thought (RAT) use chain-of-thought prompting to guide the retrieval process based on previously recalled information. Singapore is not starting from scratch in building the region’s first LLM.

The Nemotron Hindi model has 4 billion parameters and is derived from Nemotron-4 15B, a 15-billion parameter multilingual language model developed by NVIDIA. The business problem, quality of readily available data, and number of experts and AI engineers involved all impact the length and quality of the project. Because the process relies on trial and error, it’s an inherently longer time before the solution is ready for use.