A frontier model like ChatGPT or Claude is good at reading. Give it a SOC 2 Type II report, and it’ll summarize the auditor's opinion and list the exceptions the auditor found. It will also pull out the complementary user entity controls, which are the controls the vendor expects you to run on your side.
Ask it to map the report to NIST CSF 2.0 or to ISO 27001, and it’ll make a reasonable first pass. It drafts follow-up questions for the vendor faster than you can type them.
The results are convincing, and they arrive fast. One analyst with a stack of reports and a good prompt can clear a review backlog far quicker than before. But what a frontier model works with is the document in front of it, the prompt you wrote, and the model as it is that day. A vendor decision needs evidence from outside the conversation too.
A SOC 2 report is the vendor’s account of its controls, tested by an auditor the vendor engaged, for the systems the vendor put in scope. It’s good evidence, with limits that matter when you make a decision:
A SOC 2 report also answers only part of a framework-based assessment. In UpGuard’s analysis, a single SOC 2 report provides enough evidence, on average, to complete 61% of the checks in a NIST CSF 2.0 or ISO 27001:2022-aligned AI-powered Security Profile assessment.
The remaining checks need evidence from somewhere else: the vendor’s answers to a focused questionnaire, other security documentation, and what you can observe about the vendor from the outside.
Once the evidence comes from several sources, it rarely agrees. A questionnaire answer says one thing, a policy says another, and a newer piece of evidence says something else. A chat tool typically looks at the documents you give it in a single conversation and produces one answer.
UpGuard Vendor Risk’s Security Profile assesses all of a vendor’s evidence against the same checks and shows which sources support each check, which contradict it, and where gaps remain. You then decide how to weigh the conflict.
Seeing a conflict matters most when an auditor or a customer asks how you know a vendor meets your requirements. “The SOC 2 said so” covers part of the answer. It doesn’t cover what the report left out, or what you found that contradicted it.

Some of the most useful evidence about a vendor never appears in its documents. It’s what anyone can observe about the vendor’s internet-facing systems today, such as whether exposed services run software with known vulnerabilities, or whether its email domains can be spoofed.
No model can collect this from a document. It takes infrastructure that continuously scans the internet. UpGuard scans every monitored vendor’s domains and IP addresses at least once a day and runs 414 security checks against them.
The results are a security rating calculated with a published method: each failed check lowers the rating based on its severity, and weaker domains and IP addresses carry more weight, so a few exposed systems can’t hide behind many well-run ones.
For an analyst, the rating is a starting point. It tells you where to look before any questionnaire comes back, and it gives you something to test the documents against.
If a vendor’s SOC 2 report describes strict patching and its exposed servers run end-of-life software, you have a specific question to ask. And because scanning continues after you approve the vendor, you're not relying on a report whose period ended months ago.
Frontier models have known limitations that matter when the output needs to be managed as a governed vendor decision:
None of these limits matter as much when reading a single report. They become more apparent when you compare results across vendors or repeat them at reassessment, and the biggest risk there is how a control gets judged.
Take a control such as "the vendor has an incident response plan." It bundles several separate claims together: Is the plan documented? Has it been tested? Does it name who leads the response? Does it set a timeframe for notifying customers?
A policy can satisfy one of these and say nothing about the rest. If the control gets one overall verdict, you can't tell which part the evidence supported.
Breaking the control into specific checks solves that problem, but only if the checks are defined in advance. If the model decides them fresh each time, two vendors marked "met" may have been tested against different things. And when a vendor's result changes at reassessment, you can't tell whether the vendor changed or the questions did.
Whichever AI tool you use, four questions tell you whether you can rely on its output:
A frontier model can cite passages if you ask it to. The other three need a fixed assessment structure and a single, secure system that records decisions, which a chat session isn't designed to provide.
Vendor Risk’s Security Profile breaks each control into a defined set of checks and assesses every vendor against the same ones. The AI marks each check as met, unmet, or needing more evidence, and cites the source text behind each conclusion.
Because each check makes one narrow claim, an analyst can quickly see whether the cited passage supports it. Analysts can reject a citation, override a finding, comment on a result, or remove a document from the AI's review, and those decisions stay with the assessment. The AI does the reading, and the analyst makes the call.
Fixed checks and recorded decisions also show how rarely analysts disagree with the AI. Fewer than 0.2% of Security Profile AI check findings are rejected in published risk assessments. A chat tool can’t produce this number because there’s no fixed set of checks to count against and no record of what analysts rejected across their sessions.
Vendor decisions get revisited. An auditor samples your approvals, or a vendor has an incident and the board wants to know what you knew.
The answer has to be a record: the evidence you read, the findings, any risk you accepted and why, and who signed off. A chat history records a conversation, and it lives in the account of whoever ran it. It can’t show how a finding changed between assessments or who approved the vendor.
UpGuard Vendor Risk keeps each assessment's evidence, vendor communication, and findings together. Check history shows how each result changed over time and which evidence sat behind each version, so a reassessment can pick up from what’s changed since the last one.
When you accept a risk, Vendor Risk records it as a waiver with a justification and any compensating controls. Waivers can require approval and can expire, so accepted risks come back for review. "Why did we approve this vendor?" becomes a quick lookup.
If your team already reviews SOC 2 reports with a general AI tool, here’s a fair test. Pick three vendors you’ve assessed that way, run the same three through Vendor Risk, and compare:
Both approaches will be fast. The Security Profile analyzes security documents in minutes and generates a risk assessment report in under 60 seconds. For widely used vendors, you might not need to ask for anything. UpGuard has over 2,100 pre-sourced evidence documents covering the 250 most monitored vendors.
AI has changed how fast a vendor review can move, and that's worth keeping. A SOC 2 report that used to take an afternoon by hand now takes minutes with AI, and a review backlog clears far faster. No team should give that up.
The harder question is whether you can stand behind the approval later. That requires evidence beyond the vendor's own documents, consistent checks across vendors and reassessments, findings analysts can trace back and understand, and an audit trail that outlasts whoever ran the review. Any technology used to support a vendor approval, whether frontier models, a third-party risk management platform, or something your team has built, needs to meet that standard. If it can't, it shouldn't be the basis for a defensible decision, and you’ll have a hard time defending it at your next audit.
Start a free Vendor Risk trial and run three of your vendors side by side.