Evaluating AI Solutions: The Questions You Should Be Asking Every Vendor

Companies are being sold a lot of AI right now (a LOT). The demos are compelling, the decks are polished, and the proof-of-concept results are impressive enough to move a conversation into procurement. The pressure to act is real; pressure from boards and investors and immense competitive anxiety about who is moving faster.

What is less real in most of those conversations is a clear framework for how to decide how good actually looks. How do business teams decide if the AI demo they are seeing will solve their real, everyday problems.

I have spent the past several years working on AI solutions for commercial teams and guiding enterprise organizations through AI transformation; redesigning the operating models, organizational structures, and decision processes that determine whether an AI investment delivers lasting value or joins the graveyard of abandoned pilots. The pattern I see most consistently is organizations buying a cool idea that lacks strategic direction and effective execution.

This is the framework I would want in the hands of every Commercial and Technology leader walking into a vendor meeting. The vendors who have built something real will welcome these questions. The ones who have not will struggle with them.

On Data: Where Is the System Getting Its Context?

When this system produces an output, where did the reasoning come from?

There are two different answers. The first: the system is reasoning from general knowledge—what the underlying model learned about airlines and revenue management broadly. The second: the system is reasoning from your data—your booking history, your market performance, your competitive context, your commercial strategy.

These produce fundamentally different outputs. General knowledge produces results that are plausible and often directionally correct. Your data produces outputs specific to your network, calibrated to your strategic thought patterns, and actionable in ways generic outputs are not.

Ask directly: when the system flags an opportunity or surfaces a recommendation, what data is it reasoning over? Is it yours or industry-generic? How current is it? How does it get there, who maintains that connection, and what happens to the outputs when the pipeline has a problem?

Vendors who have done this work will answer with specificity. Those whose system is primarily a well-packaged general model will pivot back to the demo.

On Agent Design: Is This an Agent?

The word “agentic” is doing a lot of work in vendor marketing right now, and it is worth understanding what it means before evaluating whether a given system qualifies.

An agent does something. It has a goal, access to tools and data, and it takes actions to achieve that goal. A reporting layer with a conversational interface is not an agent. A dashboard that surfaces alerts is not an agent. Both can be valuable, but they are different products, and buying one when you thought you were buying the other is an expensive mistake.

Ask: what can this system do versus what does it surface for a human to do? When it identifies a market that needs attention, does it hand that finding to an analyst with the context required to act, or does it produce an alert indistinguishable from the fifty others already in the queue?

Neither answer is wrong. Human-in-the-loop design is often the correct architecture for high-stakes commercial decisions, but you should know which one you are buying and why the vendor made that choice.

On Workflow Integration: What Happens After the Output?

This is the question that separates deployments that stick from ones that get abandoned.

An AI system that produces excellent outputs but does not connect to the systems where teams execute will not be used. Not because analysts are resistant, but because friction accumulates until the system becomes optional—and optional systems get deprioritized when things get busy.

Ask how the system’s outputs connect to action. Is there an audit trail—a record of what the system surfaced, what action was taken, and what the outcome was? Can you measure, at the end of a quarter, whether recommendations were followed and whether following them improved performance?

If you cannot measure whether the AI improved your decisions, you cannot know whether your investment delivered value. The vendors who have thought seriously about this will have a point of view and a roadmap. The ones who have not will tell you the outputs speak for themselves. They do not. Outcomes do.

On Trust and Explainability: Can You See the Reasoning?

Commercial decisions carry real financial consequences. An analyst acting on an AI recommendation needs to understand why the system thought what it thought—to evaluate the recommendation intelligently, and to learn from the outcome afterward.

Ask whether the system can show its reasoning. Not a confidence score—the actual reasoning: here’s the data, here’s where it lives, here’s why this market needs attention. A system that produces recommendations without explanation is asking analysts to trust it blindly. That is not a reasonable ask for decisions that affect revenue, and it does not build analyst capability over time.

Also ask whether analysts can override the system’s recommendations and whether the system learns from those overrides rather than optimizing around them. The best commercial AI systems treat analyst judgment as signal, not interference.

On Transformation: Is the Vendor Selling a Tool or Building a Capability?

This is the question most organizations do not think to ask until they are twelve months into a deployment that has not moved.

Deploying an AI tool and transforming how a commercial organization operates are not the same project. The tool is the easy part. The hard part is the operating model change: who owns the AI outputs, how decision authority shifts, how analysts develop the skills to evaluate and act on AI reasoning, and how leadership measures success in a world where the system is doing work that used to be done by people.

The vendors who understand this will engage with those organizational questions, not just the technical ones. They will have a point of view on change management, on how to sequence the deployment to build trust before expanding scope, and on what a healthy human-AI workflow actually looks like in a commercial team.

The Question Underneath All of These

Every question in this framework asks the same thing: does this vendor understand the problem well enough to have built something that solves it?

The vendors who have will answer with specificity, acknowledge the hard parts honestly, and point to production deployments where the answers held up under real conditions. The ones who have not will redirect to capability and talk about roadmap.

The bar for this category should be high. Set it. Ask the questions. The vendors who clear them are the ones worth the conversation.