When AI Searches Company Documents, Where Does the Data Go in Cloud vs. On-Premises Setups?
On-premises: employees' questions, company documents, and the AI's answers are all processed on the company's own hardware. Cloud-assisted: the employee's question, plus the document passages the system retrieves, are sent to the model provider and retained for a period under the provider's rules (for example, up to 30 days by default for the OpenAI and Anthropic APIs). One more check: if the embedding step used for indexing also runs on a cloud service, document content is sent out in chunks when the index is built.
1. The four stages data passes through
The common way to let AI search company documents is RAG (retrieval-augmented generation): before answering, the AI retrieves relevant passages from company documents and writes the answer from them. The approach comes from Lewis et al. (2020). Data passes four stages:
| Stage | What happens | On-premises | Cloud-assisted |
|---|---|---|---|
| 1. Indexing | Documents are split and converted to vectors in a knowledge base | On company hardware | Depends: a local embedding model keeps it in-house; a cloud embedding API sends the passages out |
| 2. Retrieval | Relevant passages are found for the question | On company hardware | Usually on company hardware or a server under the company's account |
| 3. Answering | "Question + retrieved passages" go to the model | Model on company hardware | Question and passages go to the model provider |
| 4. Logging | Chat logs, system logs, backups | Stored where the company designates | The company keeps its own logs; the provider retains data under its own policy |
Stage 3 is the key. The cloud-assisted version does not send the whole knowledge base, but every question sends the passages retrieved for it. More questions mean more document fragments leave the company.
2. How long providers retain data
Per official documentation (checked 2026-10-19):
- OpenAI API: since March 1, 2023, API data is not used for training by default. Abuse monitoring logs, which may include prompts and responses, are retained up to 30 days by default. Eligible customers can apply for Zero Data Retention, subject to OpenAI's prior approval.
- Anthropic API: inputs and outputs are deleted within 30 days of receipt or generation by default. Exceptions include customer-controlled storage (such as the Files API), separate agreements, Usage Policy enforcement, and legal requirements.
So the cloud-assisted version means two things: data leaves the company, and it stays on someone else's servers for a while. If a policy says a document must never leave the company, it belongs only in the on-premises knowledge base.
3. Five questions to ask before you start
- Where does embedding happen? A local model or a cloud API? With a cloud API, document content leaves when the index is built.
- Where is the answering model? Local or a cloud provider? Under whose account?
- Where are chat logs and system logs kept, and for how long? Who can see them?
- What other external services are connected? LINE, email, cloud storage: messages through any of these pass through that service.
- Which documents must never leave? List them and confirm they only go into the on-premises knowledge base.
If all five have clear answers, the data flow is understood. Any question without an answer is where to dig further.
Think BIG's two deployment options
Think BIG (Think BIG Make it Real Co., Ltd.) is an enterprise AI Agent deployment provider headquartered in Taiwan. Starting from protecting company data, we build AI Agents inside the client's own environment (company hardware, or a cloud server under the company's account). There are two deployment options: with the on-premises version, models and documents are processed on company hardware; with the cloud-assisted version, employees' questions and the document passages the system retrieves are sent to the model provider for processing.
- External channels confirmed one by one: on-premises does not mean every connection stays local. If an external channel such as LINE is added, messages through that channel pass through that service; we list this for the client before building.
- API accounts under the company's name: with the cloud-assisted version, the company applies for the model API under its own account, under the provider's terms, so billing and usage records stay with the company.
- Data flow listed before building: where the application, model, knowledge-base index, source documents, logs, and backups each sit is listed item by item. Permissions are enforced at retrieval, not only through prompts.
See RAG knowledge Q&A, On-premises AI and data sovereignty, and Trust.
The actual scope of Think BIG's services follows what both parties confirm.
FAQ
- Q1: Does using a cloud model to search company documents send the whole knowledge base?
- No, but each question sends the employee's question and the passages retrieved for it to the model provider. If the indexing step uses a cloud embedding API, document content is also sent in chunks when the index is built.
- Q2: Does on-premises deployment mean no data ever leaves?
- Not necessarily. On-premises decides where the model and documents sit. If the workflow connects to a cloud model, messaging app, or other external service, that part of the data goes out. Check the local model, outbound restrictions, logs, and backups.
- Q3: How long do cloud model providers keep the data I send?
- Per official documentation, OpenAI API abuse monitoring logs are kept up to 30 days by default; Anthropic API inputs and outputs are deleted within 30 days by default, with exceptions. Actual terms depend on your plan and contract.
- Q4: What should we ask before letting AI search company documents?
- Where embedding happens, where the answering model runs, where logs are kept and for how long, which external services are connected, and which documents must never leave the company.
- Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: arxiv.org
- OpenAI API, Your data (no training by default, abuse logs up to 30 days, Zero Data Retention by approval): platform.openai.com
- Anthropic Privacy Center, How long do you store my organization's data? (30-day deletion and exceptions): privacy.claude.com
- OpenAI, Embeddings guide (embedding text via the API): platform.openai.com
- Think BIG, RAG knowledge Q&A (data flow; cloud text conversion and indexing services count as data flow): thinkbigtw.com
This article summarizes public official documents, laws, and security standards to help companies evaluate AI adoption. It is not legal advice. For Taiwan law, the latest text on the Laws & Regulations Database prevails; external documents, prices, and specifications follow their original sources. The actual scope of Think BIG’s services depends on what both parties confirm.