RAG Upload Flow (Admin)

This flow shows how documents, PDFs, and images are processed and added to vector stores in the admin panel.

Key Steps

Upload file in Admin → RAG.
PDFs: extract JSON via Responses API, then vectorize JSON.
Text/docs: upload directly, then vectorize the file.
Images: upload to S3, build JSON metadata, vectorize metadata.
Store vector IDs in DB for the tenant.
Chat uses fileSearch tool over tenant vector stores.
Delete removes vector store file, deletes disk file, then removes DB record.

Mermaid Flow

flowchart TD A["Admin Panel: Upload Document or Image"] --> B{File Type?} B -->|PDF| C["Save PDF to uploads/tenant"] C --> D["Upload PDF to OpenAI Files (purpose: user_data)"] D --> E["Responses API: extract structured JSON"] E --> F["Write JSON to temp file"] F --> G["Upload JSON to OpenAI Files (purpose: assistants)"] G --> H["Create or get tenant vector store"] H --> I["Add JSON file to vector store"] I --> J["Update DB: tenant_files + vectorStoreId + jsonFileId + jsonContent"] B -->|"TXT, MD, DOC, DOCX, XLS, XLSX, HTML, JSON"| K["Save file to uploads/tenant"] K --> L["Upload file to OpenAI Files (purpose: assistants)"] L --> M["Create or get tenant vector store"] M --> N["Add file to vector store"] N --> O["Update DB: tenant_files + vectorStoreId + openaiFileId"] B -->|Image| P["Upload image to S3"] P --> Q["Build image metadata JSON"] Q --> R["Upload JSON to OpenAI Files (purpose: assistants)"] R --> S["Create or get image vector store"] S --> T["Add JSON file to image vector store"] T --> U["Update DB: tenant_images + imageVectorStoreId + openaiFileId"] V["Chat Request"] --> W["Load tenant config + vector store IDs"] W --> X["Attach fileSearch tools"] X --> Y["Agent answers using vector search results"] D1["Admin Panel: Delete File"] --> D2["DELETE /api/admin/tenant/{tenantId}/files/{fileId}"] D2 --> D3["Remove file from vector store (openaiFileId/jsonFileId)"] D3 --> D4["Delete physical file from disk (filePath/imagePath)"] D4 --> D5["Delete DB record from tenant_files"]