Exposing legacy systems to LLMs through APIs

Why legacy systems matter for AI

The most valuable enterprise data rarely sits in a modern SaaS tool. It lives in a 15-year-old ERP, an on-premise SQL Server, or a mainframe that still runs billing. An AI assistant that cannot reach those systems can only answer generic questions.

Replacing those platforms to enable AI is slow and risky. The practical alternative is an integration layer: a set of APIs that expose exactly what the model needs, and nothing more.

Design tools around business actions

Modern LLMs call "tools", which are functions with a name, a description and typed parameters. The quality of those tools determines how reliably the model uses them.

Resist exposing generic endpoints such as "run query" or "call SOAP method". Instead, model each tool on a task a person actually performs. A narrow tool is easier for the model to choose correctly and easier for you to secure.

• get_order_status(order_id) instead of a raw SELECT on the orders table
• list_open_invoices(customer_id, limit) with pagination built in
• create_return_request(order_id, reason) that validates business rules first

The adapter layer

Between the tool and the legacy system sits an adapter. It translates clean JSON requests into whatever the old system expects: SQL, SOAP, flat files or screen automation. It also translates responses back into compact, well-named fields.

This layer is where data transformation pipelines earn their keep. Normalize dates and currencies, map cryptic status codes to readable labels, and strip fields the model does not need. Smaller, cleaner responses improve accuracy and reduce token cost.

Treat the adapter as a real product. Document it with OpenAPI, version it, and give internal teams an SDK. The same API that serves the model often becomes the cleanest way for new applications to reach the old system.

Safety and governance

Assume the model will occasionally pick the wrong tool or pass the wrong argument. Your API must stay safe when that happens.

• Authenticate as the end user, not as a shared service account, so existing permissions apply.
• Keep read and write tools separate, and require explicit human confirmation for writes.
• Enforce rate limits and timeouts so a looping agent cannot overload a fragile system.
• Log every tool call with the user, arguments and result for audit and debugging.

Combining live APIs with retrieval

Not every question needs a live call. Policies, manuals and historical reports work well in a retrieval index; current balances and order status should come from the API. A good orchestration layer decides which path to use for each request. We describe common shapes in LLM orchestration patterns for production.

If you are deciding how much knowledge to index versus fetch, start with RAG vs. fine-tuning for enterprise data.

Where to start

Pick one workflow with a clear owner and measurable volume, such as order status inquiries. Build three to five read-only tools, ship to a small group, and review the logs weekly. Add write actions only after the read path is trusted.

This is the core of our enterprise AI integration work: modernizing access to legacy systems without rewriting them.