LLM User Beware: The Dangers of Relying on AI for Vendor Shortlists – A Heads Up from the CEO of servicePath™

  • Home
  • Blog Post
  • LLM User Beware: The Dangers of Relying on AI for Vendor Shortlists – A Heads Up from the CEO of servicePath™
Discover the risks of relying on AI-generated vendor shortlists. Learn how LLM bias can misrepresent top CPQ vendors, including servicePath™, and why conducting your own LLM audit is essential for accurate vendor evaluation.

Introduction

As the CEO of servicePath™, I’ve always believed in the transformative power of technology. We’ve worked hard to establish ourselves as leaders in the CPQ space—recognized by top global analysts like Gartner, IDC, and Forrester, as well as on platforms like G2 and Capterra. Yet, I’ve recently witnessed a troubling trend that compels me to share a candid heads-up.

The Risk of AI-Generated Vendor Shortlists

In today’s fast-paced digital environment, many buyers rely on large language models (LLMs) such as ChatGPT, Perplexity, Gemini, Co-Pilot, Grok, Kimi, DeepSeek, and Qwen 2.5 to quickly build vendor shortlists. These systems promise to deliver curated vendor lists in minutes, and at first glance, that seems ideal. However, my recent experiences have revealed that these AI tools can and do misrepresent market reality—even when a vendor is among the best.

The CPQ Conundrum: LLMs Miss Key Vendors

Despite our extensive industry recognition—including being the only CPQ vendor placed in the Visionary quadrant of Gartner’s 2025 Magic Quadrant—we’ve found that many LLMs consistently overlook top-tier vendors in their outputs.

Omission Despite Clear Credentials

When I recently asked an LLM to list the best-of-breed CPQ/Quote-to-Cash vendors recognized by major analysts, the system listed Salesforce, Oracle, SAP, Conga, Tacton, and PROS. Even though I explicitly mentioned our strong credentials—including coverage by IDC, Forrester, and Gartner—our name wasn’t mentioned until I pressed further. This forced omission is unacceptable.

Refusal to Acknowledge Visionary Status

In some cases, tools like Perplexity even refused to acknowledge that servicePath™ is recognized as a Visionary in the 2025 Magic Quadrant. Despite enhancing the prompts with terms like “Gartner Visionary,” “IDC CPQ report,” and “Forrester,” the AI still didn’t surface us.

Varied Results Across LLMs

Out of the seven LLMs we tested—Grok, Kimi, DeepSeek, Qwen, along with ChatGPT, Perplexity, and Gemini—we came out strong and balanced in two of the most recent models, Kimi and DeepSeek, likely benefiting from crawling more recent data. Grok provided the best data collection statement, noting, “For the absolute latest Visionary (if a 2025 report exists beyond my current data), you’d need to check Gartner directly, as positions evolve.” In contrast, other LLMs even implied that servicePath™—and even some of our competitors—did not exist in these analyst reports when queried further. We know this is simply not the truth.

Understanding User Positivity Bias in AI Outputs

Beyond the inherent challenges of AI data retrieval, there’s another critical factor at play—the user positivity bias towards LLM outputs. Today, everyone is enamored with the power and benefits these LLMs bring to the table—significant gains in productivity and time savings, to name a few. However, this enthusiasm can blur our traditional, critical assessment of the underlying data and results.

The Risks of Blindly Trusting AI

  • Uncritical Acceptance of AI Results: Users often accept vendor lists at face value without scrutinizing their accuracy.
  • Overlooking Data Gaps: AI-generated lists may omit key vendors validated by analysts.
  • Risk of Misrepresentation: Depending solely on AI-curated vendor lists can lead to a skewed perception of market reality.

Who We Serve: Tech-Enabled Enterprises in a Rapidly Changing Market

At servicePath™, we focus on enterprise customers—tech-enabled businesses selling complex solutions in evolving markets. These companies have traditionally relied on trusted third-party reports, references, and reviews to create their target vendor lists. However, the rise of LLMs introduces a dangerous disconnect by amplifying user positivity bias and limiting visibility into best-fit solutions.

A Call to Action: Conduct Your Own LLM Audit

We initially had a positive bias towards these models, trusting that their rapid access to data would yield the best vendor lists. Our experience has taught us otherwise.

Why You Need an LLM Audit

  • Identify Bias and Data Gaps: Ensure AI is capturing the full spectrum of vendors.
  • Ensure Comprehensive Vendor Coverage: Validate AI-generated lists against analyst reports and review sites.
  • Champion Transparency: An independent LLM audit protects against costly oversights.

While I believe LLMs will evolve, this is a long journey. Businesses must remain vigilant and maintain a strong marketing presence across multiple channels rather than relying solely on third-party AI aggregators.

Inviting the Community to Join the Conversation

Transparency and open dialogue drive innovation. I invite industry experts, analysts, and thought leaders to review and discuss this perspective with me.

Key Discussion Points:

  • How can we ensure top vendors—like servicePath™—are accurately represented?
  • What additional measures can vendors and buyers take to improve AI-generated vendor lists?

A virtual roundtable discussion is coming soon, and I encourage all interested parties to participate. Let’s set a new standard for transparency and excellence in vendor evaluation.

Conclusion

LLMs have revolutionized information access, but they are not infallible. The CPQ conundrum—where even industry-recognized leaders are underrepresented—serves as a cautionary tale for vendors and buyers.

At servicePath™, we understand the stakes. Buyers and sellers alike must be proactive to avoid the pitfalls of relying solely on AI-generated vendor lists. Let’s ignite a conversation that protects vendors and empowers buyers to make truly informed decisions.

Take Control of Your Vendor Selection Process

Don’t let the allure of AI and its positivity bias lead you to make uninformed decisions. At servicePath™, we’ve seen how LLM-generated vendor lists can overlook top-tier solutions—even ones backed by reputable analyst reports. I urge you to:

  • Audit Your AI Outputs: Verify vendor lists against trusted sources to ensure no key players are missed.
  • Engage in Critical Evaluation: Don’t accept results at face value—challenge and scrutinize the data.
  • Join the Conversation: Participate in our upcoming virtual roundtable where industry experts, analysts, and enterprise buyers will discuss best practices and strategies for transparent vendor evaluation.

Take charge of your vendor selection today.

Join the discussion using #CPQLeadership and #AITransparency. I look forward to your insights.

In this article

You may be interested in these articles next