We helped a Kissimmee property management group avoid a bad AI vendor. Here's what you need to ask.
Why This Matters
I’ve seen too many local businesses sign AI contracts that look good on paper but turn into nightmares. A six-month subscription for a chatbot that leaks guest data. A contract with a three-year lock-in and no exit clause. A vendor that claims SOC2 compliance but can’t produce a report.
I’m writing this because I want Orlando and Central Florida businesses to avoid those traps. This checklist is based on real vendor evaluations I’ve done with small and mid-market clients. Use it before you sign anything.
Callout: Three out of five AI vendors I reviewed last quarter had vague data handling policies. Don’t assume anything.
The 17 Questions
1. Where does my training data go?
Ask if your data is used to train the vendor’s models. If yes, that means your proprietary business data could end up in a competitor’s output. Many vendors include this in fine print. I recommend getting it in writing that your data is not used for training.
2. Do you have SOC2 Type II?
SOC2 Type II is the gold standard for data security. Ask for the report, not just a certificate. I’ve had vendors claim SOC2 but only had a Type I, which is a point-in-time snapshot, not ongoing.
3. What is the exit clause?
Can you cancel with 30 days notice? Or are you locked in for a year? Look for automatic renewal clauses. I’ve seen contracts that auto-renew at double the price.
4. How do you price? Any hidden fees?
Pricing traps: per-seat licensing that scales badly, API call overage charges, data storage fees. Ask for a total cost of ownership over 12 and 24 months.
5. Can I switch models without rebuilding?
Model lock-in is real. If you use a specific LLM (like GPT-4), can you swap to another (like Claude or Llama) later? Or are you stuck because of custom integrations?
6. Can I see the prompts being used?
Prompt visibility matters for debugging and compliance. Some vendors hide thier prompts. I prefer vendors that let you view and edit prompts.
7. Where is my data stored (data residency)?
If your data must stay in the US (or Florida), ask about server locations. Some vendors route data through Europe or Asia, which can violate compliance.
8. How do you handle kids’ data?
If your business deals with children (e.g., a tutoring center), ask about COPPA compliance. This is a legal minefield.
9. What is your uptime SLA?
Look for 99.9% uptime or higher. And ask about credits if they miss it.
10. Can I get a sandbox or trial environment?
Test before you commit. A vendor that won’t let you test is a red flag.
11. Who has access to my data internally?
Some vendors let employees view customer data. Ask about access controls and background checks.
12. What happens if you get acquired?
Change of control clauses can break your contract. Ask for protection if the vendor is sold.
13. Can I audit your security?
Some vendors allow third-party audits. If not, ask why.
14. How do you handle data breaches?
Ask for a breach notification timeline. 24 hours is ideal. 72 hours is common, but anything longer is risky.
15. What is your model’s accuracy on my use case?
Don’t accept generic benchmarks. Ask for performance metrics on similar tasks to yours.
16. Can I export my data in a standard format?
Data portability is key. Can you get your prompts, logs, and fine-tuned models out in CSV or JSON.
17. Do you have a reference customer in my industry?
Talk to someone who uses the tool for a similar purpose. A hotel management company using AI for guest communication is a good reference for a property group. This type of referral occured naturally with vendors we trust.
Real Story: Kissimmee Property Group
A Kissimmee short-term rental property management group with 60 doors came to me after they signed a contract with a flashy AI vendor. The vendor promised automated guest messaging, dynamic pricing, and maintenance scheduling. But six months in, the system started mixing up guest names and sending maintenance requests to wrong contractors. The vendor’s support took days to respond, alot of delays meant frustrated staff. The group was locked into a three-year contract with a $10,000 early termination fee.
We helped them negotiate a partial exit and switch to a simpler, more reliable system. The new setup uses a plain-language chatbot built on a local server (no cloud data leakage) with a fixed monthly fee. no lock-in. They now ask every vendor the questions above before signing anything.
Callout: That group saved $18,000 a year by switching to a vendor that passed our checklist.
Next Steps
Download the checklist, just write down these 17 questions and use it for your next vendor meeting. If you want help evaluating a specific contract, I offer a fixed-fee AI vendor due diligence assessment. Contact me to learn more.
Also check out our AI readiness assessment to see if you’re ready for an AI tool at all. And read about customer service AI assessment for tips on evaluating chatbots.
Comparison
| Checklist Item | Vendor A (Pass) | Vendor B (Fail) |
|---|---|---|
| Data not used for training | Yes, in contract | No, fine print allows it |
| SOC2 Type II | Current report | Only Type I (old) |
| Exit clause | 30-day notice, no fee | 3-year lock-in, $10k fee |
| Pricing transparency | Fixed monthly fee | Per-seat + overage charges |
| Model portability | Supports multiple LLMs | Locked to proprietary model |
Three out of five AI vendors I reviewed last quarter had vague data handling policies.
That Kissimmee group saved $18,000 a year by switching to a vendor that passed our checklist.
Frequently asked questions
What is the most important question on this checklist?
Question 1: where does your training data go? If your data is used to train the vendor's models, you lose control. I've seen local businesses leak proprietary data this way.
Do I need SOC2 if I'm a small business?
Yes, if you handle customer data. Even a small property management group with 60 doors has guest names, addresses, and payment info. SOC2 Type II is the minimum standard.
How long does a vendor due diligence check take?
I can usually complete a thorough review in 2-3 hours if the vendor cooperates. If they don't share documentation, that's a red flag.
Can I use this checklist for free AI tools?
Free tools often have worse data handling. For example, some free chatbots train on your data and sell it. Avoid them for business use unless you've confirmed the policy.
What if my vendor is a large company like Microsoft or OpenAI?
Even large vendors need scrutiny. Microsoft's M365 Copilot has specific data handling rules you should review. We cover that in our <a href='/copilot-m365-ai-assessment/'>Copilot assessment</a>.
Do you offer a template contract add-on for these questions?
Yes, I can help you draft a data protection addendum. <a href='/contact/'>Contact me</a> for details.
Ready to talk it through?
Send a one-line description of what you are trying to do. I will reply within one business day with a plain-English next step. Email or use the form →