
Implementation Guide: Triage patient portal messages, answer routine questions, escalate urgent concerns
Step-by-step implementation guide for deploying AI to triage patient portal messages, answer routine questions, escalate urgent concerns for Healthcare clients.
Hardware Procurement
Next-Generation Firewall
$400–$500 MSP cost / $700–$900 suggested resale (includes initial config)
HIPAA-required network perimeter security with application-layer filtering, TLS inspection, and the ability to whitelist AI vendor API endpoints while blocking unauthorized PHI exfiltration. Required for HIPAA technical safeguard compliance if practice lacks a compliant firewall.
Managed PoE Network Switch
$400 MSP cost / $600 suggested resale
VLAN-capable managed switch to segment clinical network traffic from guest/IoT networks, a HIPAA technical safeguard requirement. PoE ports support VoIP phones if voice triage channel is added later.
Wireless Access Point (HIPAA-compliant)
$150/unit MSP cost / $250/unit suggested resale
Secure wireless coverage for staff tablets and mobile devices used to receive AI escalation alerts. Supports VLAN tagging for network segmentation between clinical and guest wireless.
UPS Battery Backup
$550 MSP cost / $800 suggested resale
Protects firewall, switch, and internet connectivity from power interruptions, ensuring AI triage and escalation alerts remain operational during outages. Critical for practices where urgent message escalation is life-safety relevant.
Software Procurement
Sully.ai AI Healthcare Platform (Medical Practices - Primary Recommendation)
~$79/provider/month MSP cost (Pro tier ~$1,580/mo for 20 clinicians) / Suggested resale: $129–$149/provider/month
Primary AI triage platform for medical practices. Includes AI Receptionist (inbound call handling, scheduling, FAQs), AI Nurse (intake, symptom collection, triage routing), AI Scribe, and EHR integration. SOC 2 Type II, HIPAA, HITRUST, and ISO 27001 certified. Sub-second response times via LPU architecture. License type: SaaS per-provider monthly.
DeepCura AI Platform (Alternative for Budget-Conscious Medical Practices)
$129/provider/month (all features included) / Suggested resale: $179–$229/provider/month
All-in-one AI platform including scribe, receptionist, billing, and triage functions. Best value for solo to mid-size practices. Includes EHR integration with Epic, eClinicalWorks, OptiMantra, Athena, and Veradigm. HIPAA and CASA certified. 12 pre-built call templates and 15+ callable AI functions.
Arini Dental AI Receptionist (Dental Practices - Primary Recommendation)
~$249/month per practice MSP cost / Suggested resale: $349–$449/month
Purpose-built dental AI receptionist with native integration to Dentrix, Eaglesoft, Open Dental, Curve, and Dentrix Ascend. Handles appointment booking, patient inquiry routing, cancellation waitlist management, and triage of dental concerns. HIPAA compliant with real-time schedule optimization.
Klara Patient Communication Platform
Custom pricing, typically $300–$800/month per practice / Suggested resale: quote + 25% markup
Secure patient messaging hub that serves as the communication layer between patients and the AI triage system. Features two-way messaging, intelligent message routing, telemedicine with virtual waiting rooms, and integrations with athenaOne, eClinicalWorks, AdvancedMD, Greenway, ModMed, Nextech, and Veradigm. No app download required for patients.
ThreoAI by Synthreo (MSP White-Label Option)
MSP partner pricing varies; typical MSP resale price $500–$2,500/client/month
White-label AI assistant platform purpose-built for MSPs. Supports GPT (Azure ZDR), Claude, Gemini, Mistral, and LLaMA models with custom domains, per-tenant configuration, zero data retention by default, and ConnectWise Marketplace integration. SOC 2, HIPAA, and GDPR compliant. Use this if you want to build a branded AI triage service rather than reselling a third-party platform.
Microsoft 365 Business Premium (with Compliance Center)
$22/user/month MSP cost / $30–$35/user/month suggested resale
Provides secure email for escalation notifications, Microsoft Teams for internal clinical escalation channels, Azure AD for SSO/MFA to AI platforms, and Microsoft Purview compliance tools for HIPAA audit logging. Most practices already have this; verify license tier supports required compliance features.
DNSFilter (DNS Security)
$1.15/user/month MSP cost / $3–$5/user/month suggested resale
DNS-layer security that prevents PHI exfiltration to unauthorized endpoints, blocks malicious domains, and provides content filtering. HIPAA-supportive network control. Integrates with RMM tools for centralized MSP management.
CRMBridge API (Dental PMS Integration Layer)
Free tier: 100K API calls/month; paid tiers from ~$50/month / Suggested resale: bundled into dental AI service
Provides a unified REST API to integrate with Dentrix, Eaglesoft, Open Dental, and 27+ dental PMS systems. HIPAA-compliant real-time patient data sync. Essential middleware if using a custom-built or white-label AI solution instead of Arini for dental practices.
Compliancy Group (HIPAA Compliance Platform)
~$300–$400/month for MSP partner program / Resale: bundled into compliance service at $500/month per practice
Managed HIPAA compliance platform for generating risk assessments, managing BAAs, tracking employee training, documenting policies and procedures, and providing HIPAA Seal of Compliance. Streamlines the compliance overhead that is mandatory for this AI deployment.
Prerequisites
- Active EHR system (medical) or PMS (dental) with API access enabled. For medical: athenahealth, Epic, eClinicalWorks, AdvancedMD, Veradigm, or ModMed. For dental: Dentrix, Eaglesoft, Open Dental, Curve, or Dentrix Ascend.
- Patient portal already deployed and actively used by patients for messaging (minimum 30% patient adoption recommended before AI triage adds value).
- Business-grade internet connection: minimum 50 Mbps symmetric; fiber preferred. Verify uptime SLA with ISP—AI triage is time-sensitive for urgent escalations.
- HIPAA-compliant network infrastructure: next-gen firewall with TLS 1.2+ inspection, VLAN segmentation between clinical and guest networks, DNS filtering. If missing, include hardware procurement items above.
- Designated Privacy Officer / HIPAA Compliance Officer at the practice (required by HIPAA; may be the office manager or a provider).
- Signed Business Associate Agreement (BAA) between the practice and the MSP. The MSP must have its own HIPAA compliance program in place before handling PHI.
- Administrative credentials for the EHR/PMS system with sufficient privileges to enable API integrations, create webhook endpoints, and configure message routing.
- Identified clinical champion: at least one licensed provider (MD, DO, DDS, DMD, PA, NP) who will collaborate on triage rule definition, validate AI response templates, and serve as the clinical escalation reviewer.
- Microsoft 365 Business Premium or equivalent for SSO/MFA, secure email, and Teams-based escalation channels. Azure AD tenant configured for the practice.
- Current HIPAA risk assessment completed within the past 12 months. If not available, this must be conducted as Phase 1 of the project (add 2 weeks to timeline).
- Staff availability for training: minimum 2-hour initial training session for all portal-managing staff, plus 1-hour session for providers on AI review workflows.
- For dental practices using Arini: active PMS login credentials and admin access to the scheduling module for real-time calendar integration.
Installation Steps
Step 1: HIPAA Compliance Foundation & BAA Execution
Before any technology is deployed, establish the complete HIPAA compliance framework for this AI implementation. This is non-negotiable—deploying AI that processes PHI without proper BAAs and risk assessment exposes both the MSP and the practice to significant liability (fines up to $50,000 per violation). Execute BAAs with every vendor in the solution stack: AI platform vendor (Sully.ai, DeepCura, or Arini), communication platform (Klara if used), DNS filtering provider, and any cloud infrastructure providers. Conduct or update the practice's HIPAA risk assessment to include AI-specific risks: model hallucination, PHI exposure in AI training data, unauthorized access to AI admin consoles, and AI decision audit trail gaps.
BAA tracking template and vendor compliance verification reference
Do not proceed to Step 2 until ALL BAAs are fully executed and filed. BAA execution with AI vendors typically takes 3–10 business days. Sully.ai and DeepCura include BAAs in their standard onboarding; Klara requires a separate request. If using ThreoAI/Synthreo for white-label, their BAA covers the platform but you must execute a separate BAA between your MSP and the practice. Keep copies of all BAAs for a minimum of 6 years per HIPAA retention requirements.
Step 2: Network Security Hardening & HIPAA Technical Safeguards
Configure the practice's network infrastructure to meet HIPAA technical safeguard requirements and securely support cloud AI platform connectivity. This includes firewall configuration, VLAN segmentation, DNS filtering, TLS enforcement, and whitelisting of AI vendor API endpoints. If the practice lacks a compliant firewall, install the Fortinet FortiGate 40F from the hardware procurement list.
# FortiGate 40F Initial Configuration (via CLI after initial wizard)
config system interface
edit port1
set alias 'WAN'
set mode dhcp
set allowaccess ping https ssh
next
edit port2
set alias 'Clinical-LAN'
set ip 10.10.10.1 255.255.255.0
set allowaccess ping https ssh
set device-identification enable
next
edit port3
set alias 'Guest-WiFi'
set ip 10.10.20.1 255.255.255.0
set allowaccess ping
next
end
# Create VLAN for clinical traffic isolation
config system interface
edit 'VLAN-Clinical'
set vdom root
set ip 10.10.30.1 255.255.255.0
set allowaccess ping https
set interface port2
set vlanid 30
next
end
# Enable TLS 1.2+ enforcement (block TLS 1.0/1.1)
config firewall ssl-ssh-profile
edit 'HIPAA-SSL-Profile'
config https
set ports 443
set status deep-inspection
set unsupported-ssl-version block
end
set ssl-exemption-log enable
set ssl-anomaly-log enable
config ssl
set inspect-all deep-inspection
set min-allowed-ssl-version tls-1.2
end
next
end
# Whitelist AI vendor API endpoints
config firewall address
edit 'Sully-AI-API'
set type fqdn
set fqdn 'api.sully.ai'
next
edit 'DeepCura-API'
set type fqdn
set fqdn 'api.deepcura.ai'
next
edit 'Arini-API'
set type fqdn
set fqdn 'api.arini.ai'
next
edit 'Klara-API'
set type fqdn
set fqdn 'api.klara.com'
next
endIf the practice already has a compliant firewall (SonicWall TZ, Meraki MX, etc.), adapt these configurations to that platform. The key requirements are: (1) TLS 1.2+ enforcement, (2) VLAN segmentation between clinical and guest networks, (3) AI vendor endpoint whitelisting, (4) DNS filtering to prevent PHI exfiltration, and (5) full logging enabled for HIPAA audit trail. Document all firewall rules in the practice's HIPAA technical safeguard documentation. Test connectivity to AI vendor endpoints after firewall changes before proceeding.
Step 3: Identity & Access Management Configuration
Configure Single Sign-On (SSO) and Multi-Factor Authentication (MFA) for all AI platform access using Azure AD (Entra ID). Create role-based access control groups that map to clinical workflows: AI Admin (MSP technicians), Clinical Reviewer (providers who approve AI-drafted responses), Office Manager (triage rule configuration), and Read-Only Auditor (HIPAA compliance monitoring). This ensures every access to the AI platform handling PHI is authenticated, authorized, and logged.
# PowerShell: Export Azure AD sign-in logs for compliance
Install-Module AzureADPreview
Connect-AzureAD
Get-AzureADAuditSignInLogs -Filter "appDisplayName eq 'Sully AI Triage'" -Top 1000If the AI vendor does not support SAML/OIDC SSO (some smaller vendors use email/password only), enforce MFA using the vendor's built-in MFA settings and document this as a compensating control in the HIPAA risk assessment. Sully.ai and DeepCura both support SSO integration. For Arini, confirm SSO support during onboarding—if unavailable, enforce strong passwords (16+ chars) and vendor-native MFA. All AI platform credentials must be unique per user—no shared accounts.
Step 4: EHR/PMS API Integration Setup
Enable and configure the bidirectional API connection between the AI triage platform and the practice's EHR (medical) or PMS (dental). This is the most variable step in the deployment—integration complexity depends heavily on which EHR/PMS the practice uses. The goal is to allow the AI agent to: (1) receive inbound patient messages, (2) access relevant patient context (demographics, recent visits, medication list), (3) create tasks/tickets for clinical review, (4) write approved responses back to the patient, and (5) update appointment schedules.
MEDICAL: athenahealth API Setup
# test token and patient document endpoint connectivity
# athenahealth API - Test connectivity
curl -X POST 'https://api.platform.athenahealth.com/oauth2/v1/token' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&scope=athena/service/Athenanet.MDP.*'
# Verify patient message endpoint access
curl -X GET 'https://api.platform.athenahealth.com/v1/{practiceid}/patients/{patientid}/documents' \
-H 'Authorization: Bearer YOUR_ACCESS_TOKEN'MEDICAL: Epic (via SMART on FHIR)
# Epic FHIR - Test patient communication endpoint
curl -X GET 'https://fhir.{epic-instance}.org/api/FHIR/R4/Communication?patient={patientId}' \
-H 'Authorization: Bearer YOUR_ACCESS_TOKEN' \
-H 'Accept: application/fhir+json'DENTAL: Arini Native PMS Integration
DENTAL: CRMBridge Alternative (for custom/white-label builds)
curl -X GET 'https://api.crmbridge.com/v1/patients?search=Smith' \
-H 'Authorization: Bearer YOUR_CRMBRIDGE_API_KEY' \
-H 'Content-Type: application/json'EHR integration is the highest-variability step. athenahealth has the best-documented public API and fastest approval process (~5-10 business days). Epic requires App Orchard review which can take 2–4 weeks—start this in Phase 1 if the practice uses Epic. For eClinicalWorks, work through the vendor's partner integration team. For dental PMS, Arini's native integrations are the fastest path—their bridge agent installs in under 30 minutes. If using a custom/white-label solution for dental, CRMBridge provides a unified API layer across 27+ dental PMS systems. ALWAYS test API connectivity in a sandbox/dev environment before connecting to production patient data.
Step 5: AI Triage Platform Provisioning & Configuration
Set up the AI triage platform account, configure the practice profile, define message categories, and establish the initial triage classification rules. This step transforms the generic AI platform into a practice-specific triage engine. Work directly with the clinical champion identified in prerequisites to validate all triage rules and response templates.
- === Sully.ai Setup (Medical Practices) ===
- Navigate to https://app.sully.ai and create organization account
- Add practice details: name, NPI, specialty, address, phone, timezone
- Add providers: full name, credentials, NPI, specialties, schedule
- Enable AI Receptionist module: Settings > AI Modules > Receptionist > Enable
- Enable AI Nurse (Triage) module: Settings > AI Modules > Nurse > Enable
- Configure EHR connection (from Step 4 API credentials)
- Define triage categories (see Custom AI Components section for full taxonomy)
- === DeepCura Setup (Alternative Medical) ===
- Navigate to https://app.deepcura.ai and create account
- Connect EHR via link-based integration (no API keys needed for basic)
- Select call/message templates from 12 pre-built options
- Customize triage escalation rules
- === Arini Setup (Dental) ===
- Navigate to https://app.arini.ai and create practice account
- Connect PMS (completed in Step 4)
- Configure business hours and appointment types
- Set up AI greeting and response personality
- Define dental-specific triage rules (pain, swelling, trauma = urgent)
- Configure waitlist management for cancellation auto-fill
- === Common Configuration for All Platforms ===
- Configure escalation notification channels: Microsoft Teams webhook for clinical escalation channel, SMS/email alerts for on-call provider, office phone for true emergencies
- === Microsoft Teams Incoming Webhook Setup ===
- In Teams, navigate to the 'Clinical Escalations' channel
- Click '...' > Connectors > Incoming Webhook
- Name the webhook: 'AI Triage Escalation'
- Copy webhook URL and paste into AI platform's notification settings
https://outlook.office.com/webhook/GUID/IncomingWebhook/GUIDSpend at least 2 hours in a working session with the clinical champion during this step. They must review and approve every auto-response template, every triage category definition, and every escalation threshold. Document their approval in writing—this is both a compliance requirement and a liability safeguard. The clinical champion should sign off on the triage rule matrix before any patient messages are processed by the AI. For Sully.ai, the initial setup wizard is comprehensive and guides through most configuration; budget 2–3 hours for complete setup. For Arini (dental), the setup is faster—typically under 1 hour for basic configuration.
Step 6: Triage Rule Engine & Escalation Workflow Configuration
Configure the detailed triage classification rules, urgency scoring, auto-response templates, and escalation workflows. This is the clinical heart of the system—the rules that determine which messages get auto-responded, which get queued for clinician review, and which trigger immediate emergency escalation. All rules must be validated by the clinical champion and documented in the practice's clinical protocols.
Triage category configuration reference — emergency keywords, urgency levels, auto-response templates, dental-specific rules, and after-hours behavior
CRITICAL SAFETY NOTE: The emergency category must ALWAYS include a directive to call 911. The AI must never attempt to provide clinical advice for emergency-level concerns. The auto-response for emergencies should be hard-coded and not editable by AI—use a static template. For dental practices, knocked-out permanent teeth require same-day response (reimplantation within 1 hour has best prognosis), so this falls in the emergency category even though it may not seem life-threatening. Review the triage rules monthly for the first 3 months, then quarterly. Document every rule change with clinical champion sign-off.
Step 7: AI Response Template Library Configuration
Create and configure the library of AI response templates that the system uses to auto-respond to routine inquiries and draft responses for clinical review. Templates must be written in plain language (6th-grade reading level per health literacy best practices), approved by the clinical champion, and tagged with appropriate metadata for audit trails. Each template should include the practice name, a clear disclaimer about AI-assisted communication, and appropriate next-step instructions.
# JSON format for API-based platforms. Store in AI platform's template
# library or custom knowledge base.
// Example: Appointment Request Auto-Response
{
"template_id": "ADMIN-APPT-001",
"category": "administrative",
"subcategory": "appointment_request",
"requires_clinician_review": false,
"response_template": "Hello {patient_first_name}, thank you for reaching out to {practice_name}. We'd be happy to help you schedule an appointment.\n\nBased on your request, here are available times:\n{available_slots}\n\nYou can also schedule directly through your patient portal at {portal_url} or call us at {practice_phone}.\n\nThis message was prepared with AI assistance and reviewed by our team.\n\nBest regards,\n{practice_name}",
"variables": ["patient_first_name", "practice_name", "available_slots", "portal_url", "practice_phone"],
"approved_by": "Dr. Smith",
"approved_date": "2025-01-15",
"last_reviewed": "2025-01-15"
}
// Example: Prescription Refill Acknowledgment
{
"template_id": "ADMIN-REFILL-001",
"category": "administrative",
"subcategory": "prescription_refill",
"requires_clinician_review": false,
"response_template": "Hello {patient_first_name}, we received your prescription refill request for {medication_name}. Your request has been placed in the refill queue and will be reviewed by your provider within {refill_sla} business days.\n\nIf you need the medication urgently, please call our office at {practice_phone}.\n\nThis message was prepared with AI assistance.\n\n{practice_name}",
"variables": ["patient_first_name", "medication_name", "refill_sla", "practice_phone", "practice_name"]
}
// Example: Clinical Question — AI Draft for Clinician Review
{
"template_id": "CLINICAL-DRAFT-001",
"category": "clinical",
"subcategory": "general_question",
"requires_clinician_review": true,
"ai_draft_instructions": "Draft a response using the patient's medical history from EHR context. Use plain language at 6th-grade reading level. Do NOT provide a diagnosis. Recommend follow-up if appropriate. Include standard disclaimer.",
"disclaimer_footer": "This response was drafted with AI assistance and reviewed by {reviewing_provider}. It is not a substitute for an in-person evaluation. If your symptoms change or worsen, please contact our office immediately or call 911 for emergencies."
}Templates should be stored in a version-controlled document (SharePoint or the AI platform's native template system) with full change history. Every template must include the AI disclosure notice—Texas, California, Colorado, and other states are passing laws requiring disclosure of AI interaction. Even where not yet legally required, it is a best practice and builds patient trust. Review all templates quarterly with the clinical champion. For dental practices, create dental-specific templates covering post-procedure care FAQs, insurance coverage explanations, and treatment plan questions. Maintain a separate 'Denied Templates' log documenting any auto-response scenarios the clinical champion explicitly rejected.
AI Disclosure Notice (required by a growing number of state laws): Include on ALL AI-generated or AI-assisted messages: 'This message was prepared with the assistance of AI technology and reviewed by our clinical team.'
Step 8: Pilot Deployment & Shadow Mode Testing
Deploy the AI triage system in 'shadow mode' where it processes all incoming patient messages but does NOT send auto-responses or take autonomous action. Instead, the AI's classification and draft responses are logged alongside the actual human staff responses for comparison. This 2–4 week pilot phase allows validation of triage accuracy, identification of edge cases, and building clinician confidence before going live. Configure the system to run in parallel with existing workflows.
Enable Shadow/Audit Mode
- Sully.ai: Settings > Deployment Mode > 'Shadow Mode' / 'Audit Only'
- DeepCura: Settings > AI Behavior > 'Draft Only - No Auto-Send'
- Arini: Settings > Testing Mode > Enable
Create Monitoring Dashboard
Set up a daily review process tracking the following metrics:
Daily Shadow Mode Review Spreadsheet
Create in SharePoint or Google Sheets with the following columns:
- Date
- Message ID
- Patient ID (de-identified for tracking)
- AI Category
- Human Category
- Match?
- AI Draft Response
- Human Response
- Clinician Quality Score (1–5)
- Notes
Weekly Metrics Review Meeting
Attendees: MSP tech, clinical champion, office manager
Go-Live Criteria
All of the following criteria must be met before proceeding to live deployment:
The shadow mode phase is non-negotiable for clinical safety. Do NOT skip this step to save time. A 2-week minimum is recommended; 4 weeks is ideal. During this phase, the existing manual workflow continues unchanged—staff respond to all messages as they normally would. The AI processes the same messages in the background. Focus especially on false negatives (urgent messages the AI classified as routine)—these are the patient safety risks. If the false negative rate on urgent/emergency categories exceeds 5% after tuning, do not go live until it is resolved. Document all shadow mode findings in the HIPAA risk assessment as part of the AI system validation evidence.
Step 9: Staff Training & Change Management
Conduct comprehensive training for all practice staff who interact with the AI triage system. This includes providers (who review and approve AI-drafted clinical responses), clinical staff (who monitor the triage queue and handle escalations), and administrative staff (who manage scheduling and billing messages that the AI routes to them). Training must cover both the technical operation of the system and the clinical governance framework.
- Session 1: All Staff Overview (2 hours) — What the AI triage system does and doesn't do; HIPAA implications and staff responsibilities; AI disclosure requirements to patients; Demo of message flow: patient sends message → AI classifies → action taken; How to identify and report AI errors; Emergency override procedures (how to manually escalate); Q&A
- Session 2: Providers/Clinicians (1.5 hours) — AI draft response review workflow; How to approve, edit, or reject AI drafts; Clinician override of AI classification; Understanding AI confidence scores; Clinical liability considerations; Triage rule modification request process; Hands-on practice with review queue
- Session 3: Office Managers/Admin Staff (1 hour) — Administrative message routing workflows; AI-assisted scheduling: how it works, how to correct errors; Monitoring dashboard walkthrough; Escalation notification management; Reporting AI issues to MSP
- Session 4: On-Call Providers (30 minutes) — Emergency escalation notification flow; How urgent alerts arrive (SMS, Teams, phone); Acknowledging receipt of escalation; After-hours AI behavior and limitations
Change management is often harder than the technology. Common staff concerns: 'Will AI replace my job?' (No—it handles routine messages so staff can focus on complex cases), 'What if the AI makes a mistake?' (Clinician-in-the-loop for all clinical content; emergency safeguards), 'Do patients know it's AI?' (Yes—disclosure on every message). Address these proactively. The clinical champion should co-present training sessions to build peer credibility. Plan for 1–2 weeks of increased support tickets after go-live as staff adapt to new workflows.
Step 10: Phased Go-Live & Production Deployment
Transition from shadow mode to live production in a phased approach. Phase A (Week 1): Enable auto-response for administrative messages only (appointment requests, office info, billing routing). Phase B (Week 2): Enable AI draft responses for routine clinical questions, with mandatory clinician review before sending. Phase C (Week 3+): Enable urgent/emergency escalation automation. Never enable clinical auto-response without clinician review—always maintain the human-in-the-loop for clinical content.
Phased go-live configuration notes, escalation setup, production monitoring, and rollback procedure
The phased approach is essential for patient safety and staff confidence. Administrative auto-responses (Phase A) are low-risk and demonstrate quick value. Clinical draft-and-review (Phase B) is where the biggest time savings for providers occur—studies show 60% of AI draft responses require minimal or no editing. Never enable auto-sending of clinical responses without clinician review—this crosses into potential 'practice of medicine' territory and creates liability. Monitor the system intensively during the first 2 weeks of each phase. Keep the rollback procedure documented and accessible to the office manager so they can disable auto-response immediately if needed, without waiting for the MSP.
Step 11: Compliance Documentation & Audit Trail Verification
Finalize all HIPAA compliance documentation specific to the AI triage deployment. Verify that complete audit trails are being captured, retention policies are configured correctly, and all required policies and procedures are documented. This step produces the documentation package that will withstand a HIPAA audit or OCR investigation.
Required Documentation Package
Audit Trail Test
Select 5 random patient messages from the past week. For each, verify you can produce all of the following elements. If any element is missing, work with the vendor to resolve before sign-off.
Export Audit Logs for Compliance Archive
- Navigate to Reports > Audit Logs in your AI platform
- Select Export and choose your desired Date Range
- Download in CSV or JSON format
- Store exports in an encrypted SharePoint folder with a 6-year retention policy applied
This documentation is not optional—it is your liability shield and the practice's compliance evidence. If a patient files a complaint with the HHS Office for Civil Rights about an AI-generated response, this documentation package is what demonstrates the practice took reasonable precautions. Store all documents in an encrypted, access-controlled SharePoint site with automatic retention policies. Review and update the documentation quarterly. The risk assessment addendum should be a living document updated whenever triage rules change, new message categories are added, or the AI platform is updated.
Custom AI Components
Patient Message Triage Classifier
Type: agent
The core AI agent that receives inbound patient portal messages, analyzes their content using NLP and clinical context, assigns an urgency classification (Emergency, Urgent Clinical, Routine Clinical, Administrative), and routes them to the appropriate workflow. This agent operates as the first-touch processor for all patient messages and must have a clinically-validated decision framework with explicit safety guardrails for emergency detection.
Implementation:
Patient Message Triage Classifier Agent
Architecture
This agent runs within the selected AI platform (Sully.ai, DeepCura, or ThreoAI) and uses a multi-stage classification pipeline:
Stage 1: Safety Screen (Rule-Based, No AI) Hard-coded keyword matching that runs BEFORE the LLM to catch emergencies with zero latency:
EMERGENCY_KEYWORDS = [
'chest pain', 'can\'t breathe', 'difficulty breathing', 'shortness of breath',
'heart attack', 'stroke', 'seizure', 'unconscious', 'unresponsive',
'severe bleeding', 'won\'t stop bleeding', 'suicidal', 'want to die',
'kill myself', 'self-harm', 'overdose', 'poisoning', 'anaphylaxis',
'allergic reaction', 'swelling throat', 'can\'t swallow',
'severe abdominal pain', 'vomiting blood', 'coughing blood',
'head injury', 'loss of consciousness', 'numbness face',
'slurred speech', 'vision loss', 'sudden weakness'
]
DENTAL_EMERGENCY_KEYWORDS = [
'knocked out tooth', 'tooth fell out', 'avulsed', 'jaw broken',
'jaw fracture', 'uncontrolled bleeding mouth', 'can\'t breathe swelling',
'facial swelling airway', 'severe facial trauma'
]
def safety_screen(message_text: str) -> dict:
message_lower = message_text.lower()
for keyword in EMERGENCY_KEYWORDS + DENTAL_EMERGENCY_KEYWORDS:
if keyword in message_lower:
return {
'classification': 'EMERGENCY',
'confidence': 1.0,
'matched_keyword': keyword,
'bypass_llm': True,
'action': 'IMMEDIATE_ESCALATION'
}
return {'bypass_llm': False}Stage 2: LLM Classification (AI-Powered) For messages that pass the safety screen without emergency detection:
TRIAGE_SYSTEM_PROMPT = """
You are a medical message triage assistant for {practice_name}, a {practice_type} practice.
Your role is to classify patient portal messages into one of four categories and extract key information.
You are NOT a doctor. You do NOT provide diagnoses or medical advice.
You classify messages to ensure they reach the right person at the right time.Classification Categories:
EMERGENCY (Immediate escalation required)
Life-threatening symptoms, severe acute conditions, mental health crisis.
Examples: chest pain, difficulty breathing, suicidal ideation, severe allergic reaction, uncontrolled bleeding, stroke symptoms, loss of consciousness.
ACTION: Immediate alert to on-call provider. Auto-response directing patient to call 911.
URGENT_CLINICAL (Same-day clinician review required)
New or worsening symptoms that need prompt attention but are not immediately life-threatening.
- fever > 101°F
- new onset pain
- medication side effects
- post-surgical concerns
- worsening chronic condition
- abnormal test result questions
- injury (non-emergency)
ACTION: Priority queue for clinician review within 2-4 hours.
ROUTINE_CLINICAL (Standard clinician review, 24-48 hours)
General health questions, chronic disease management, wellness inquiries, non-urgent symptom questions.
Examples: 'What does my lab result mean?', diet/exercise questions, medication questions (non-side-effect), preventive care questions, minor symptom inquiries.
ACTION: AI drafts response for clinician review and approval.
ADMINISTRATIVE (Auto-response eligible, no clinician review needed)
Non-clinical requests that can be handled by office staff or automated systems.
Sub-categories:
- APPOINTMENT_REQUEST: Scheduling, rescheduling, cancellation
- PRESCRIPTION_REFILL: Medication refill requests
- REFERRAL_STATUS: Referral tracking inquiries
- BILLING_INSURANCE: Billing questions, insurance verification, payment plans
- RECORDS_REQUEST: Medical records access or transfer requests
- GENERAL_INFO: Office hours, location, provider availability
ACTION: Auto-respond with appropriate template OR route to admin staff.
Output Format:
Return ONLY valid JSON:
{
"classification": "EMERGENCY|URGENT_CLINICAL|ROUTINE_CLINICAL|ADMINISTRATIVE",
"sub_category": "specific sub-category if ADMINISTRATIVE, null otherwise",
"confidence": 0.0-1.0,
"reasoning": "Brief explanation of why this classification was chosen",
"key_entities": {
"symptoms": ["list of symptoms mentioned"],
"medications": ["list of medications mentioned"],
"timeframe": "when symptoms started or when action is needed",
"body_part": "relevant body part if applicable"
},
"suggested_response_template": "template_id to use for response",
"requires_patient_context": true/false
}Safety Rules (NEVER violate):
def classify_message(message_text: str, patient_context: dict = None) -> dict:
# Stage 1: Safety Screen
safety_result = safety_screen(message_text)
if safety_result.get('bypass_llm'):
return safety_result
# Stage 2: LLM Classification
context_prompt = ''
if patient_context:
context_prompt = f"""
Patient Context (from EHR):
- Age: {patient_context.get('age', 'Unknown')}
- Active Conditions: {patient_context.get('conditions', 'None listed')}
- Recent Procedures (last 30 days): {patient_context.get('recent_procedures', 'None')}
- Current Medications: {patient_context.get('medications', 'None listed')}
"""
user_prompt = f"""
{context_prompt}
Patient Message:
\"{message_text}\"
Classify this message according to your instructions.
"""
# Call LLM API (platform-specific)
response = llm_client.chat.completions.create(
model='gpt-5.4', # or platform's default model
messages=[
{'role': 'system', 'content': TRIAGE_SYSTEM_PROMPT},
{'role': 'user', 'content': user_prompt}
],
temperature=0.1, # Low temperature for consistent classification
response_format={'type': 'json_object'}
)
result = json.loads(response.choices[0].message.content)
# Stage 3: Safety Override
# If LLM classified as ROUTINE but confidence < 0.7, upgrade to URGENT
if result['classification'] == 'ROUTINE_CLINICAL' and result['confidence'] < 0.7:
result['classification'] = 'URGENT_CLINICAL'
result['reasoning'] += ' [AUTO-UPGRADED: Low confidence on routine classification]'
return resultConfiguration in Sully.ai / DeepCura:
If using a turnkey platform, this logic is configured through the platform's triage rule builder UI rather than custom code. Map the above categories and rules to the platform's configuration options:
- Sully.ai: AI Nurse module > Triage Rules > Custom Categories
- DeepCura: Message Templates > Triage Configuration
- ThreoAI: Workflows > Classification Agent > System Prompt (paste TRIAGE_SYSTEM_PROMPT above)
Clinical Response Drafter
Type: agent An AI agent that generates draft responses for clinical questions classified as ROUTINE_CLINICAL or URGENT_CLINICAL. The draft is placed in a clinician review queue—it is NEVER sent directly to the patient. The agent uses patient EHR context (medical history, medications, recent visits) to generate contextually relevant responses while strictly avoiding diagnosis or medical advice. All drafts include a standard disclaimer and are written at a 6th-grade reading level.
Implementation:
Clinical Response Drafter Agent
System Prompt
CLINICAL_DRAFT_SYSTEM_PROMPT = """
You are a clinical communication assistant for {practice_name}.
You help draft responses to patient portal messages that will be reviewed and approved
by a licensed healthcare provider before being sent to the patient.Your Role:
- Draft clear, compassionate, accurate responses to patient clinical questions
- Use the patient's EHR context to personalize responses
- Write at a 6th-grade reading level (short sentences, simple words)
- NEVER provide a diagnosis, prognosis, or definitive medical advice
- ALWAYS recommend follow-up when clinically appropriate
- ALWAYS include the standard disclaimer footer
Response Guidelines:
Absolute Prohibitions:
- Do NOT say 'you have [condition]' or 'this is likely [diagnosis]'
- Do NOT recommend starting, stopping, or changing medications
- Do NOT interpret lab results with definitive conclusions
- Do NOT provide prognosis ('you will/won't recover')
- Do NOT contradict established treatment plans without clinician review
- Do NOT use medical jargon without plain-language explanation
Disclaimer Footer (ALWAYS include):
This response was prepared with AI assistance and reviewed by {reviewing_provider_name}. It is intended for informational purposes and does not replace an in-person evaluation. If your symptoms worsen or you have new concerns, please contact our office at {practice_phone} or call 911 for emergencies.
Output Format:
{
"draft_response": "The full response text to be reviewed by the clinician",
"clinician_notes": "Internal notes for the reviewing clinician about this draft\n (e.g., 'Patient asking about metformin side effects - may want\n to discuss at next visit', 'Lab results referenced - please\n verify interpretation before approving')",
"confidence": 0.0-1.0,
"requires_ehr_update": true/false,
"suggested_followup": "appointment|phone_call|none|referral",
"reading_level": "estimated Flesch-Kincaid grade level of response"
}Clinician Review Queue Workflow
SLA Timer Configuration
RESPONSE_SLA = {
'EMERGENCY': {'max_minutes': 5, 'escalation': 'on_call_provider_call'},
'URGENT_CLINICAL': {'max_minutes': 240, 'escalation': 'teams_office_manager'},
'ROUTINE_CLINICAL': {'max_minutes': 1440, 'escalation': 'email_office_manager'},
'ADMINISTRATIVE': {'max_minutes': 60, 'escalation': 'none_auto_respond'}
}
def check_sla_compliance(message_id: str, classification: str, received_at: datetime):
sla = RESPONSE_SLA[classification]
elapsed = (datetime.utcnow() - received_at).total_seconds() / 60
if elapsed > sla['max_minutes']:
trigger_escalation(
message_id=message_id,
escalation_type=sla['escalation'],
elapsed_minutes=elapsed
)
log_sla_breach(message_id, classification, elapsed)Platform-Specific Implementation:
- Sully.ai: Use AI Nurse module with custom response templates. Configure review queue notifications via Settings > Notifications > Clinical Review.
- DeepCura: Use the 'Draft Only' mode for clinical messages. Clinicians review in the DeepCura dashboard or via EHR inbox integration.
- ThreoAI (white-label): Implement the full prompt above as a custom agent workflow. Use the webhook system to push drafts to a Microsoft Teams Adaptive Card for clinician review.
Emergency Escalation Orchestrator
Type: workflow
An automated workflow that activates when a message is classified as EMERGENCY. This is the most safety-critical component of the system. It implements a multi-channel notification cascade to ensure an on-call provider is reached within minutes, sends an immediate safety-focused auto-response to the patient directing them to call 911, and logs every action for audit trail purposes. The workflow uses redundant notification channels (SMS, phone call, Teams, email) with automatic escalation if the primary contact does not acknowledge within defined timeframes.
Implementation:
Emergency Escalation Orchestrator Workflow
Workflow Trigger
Activated when triage classifier returns classification = 'EMERGENCY'
Workflow Definition (Platform-Agnostic)
workflow:
name: emergency_escalation_v1
trigger: message.classified.emergency
timeout_minutes: 30
steps:
- step: 1
name: immediate_patient_response
action: send_patient_message
template: |
IMPORTANT: If you are experiencing a medical emergency,
please call 911 immediately or go to your nearest emergency room.
Your message has been flagged as urgent and our on-call clinical
team has been notified for immediate review. A member of our
team will contact you shortly.
Emergency: Call 911
Poison Control: 1-800-222-1222
Suicide & Crisis Lifeline: 988
- {practice_name}
delay_seconds: 0
log: true
- step: 2
name: primary_provider_sms
action: send_sms
recipient: on_call_provider_phone
message: |
🚨 URGENT PATIENT MESSAGE - {practice_name}
Patient: {patient_name} (DOB: {patient_dob})
Message excerpt: {message_preview_50_chars}
Classification: EMERGENCY
Action required: Review immediately
Reply 'ACK' to acknowledge receipt.
Portal link: {direct_message_link}
delay_seconds: 0
log: true
wait_for_ack: true
ack_timeout_seconds: 300
- step: 3
name: teams_clinical_channel_alert
action: teams_webhook
webhook_url: ${TEAMS_CLINICAL_WEBHOOK_URL}
payload:
type: MessageCard
themeColor: 'FF0000'
title: '🚨 EMERGENCY Patient Message'
sections:
- activityTitle: 'Patient: {patient_name}'
activitySubtitle: 'Received: {timestamp}'
facts:
- name: Classification
value: EMERGENCY
- name: Trigger
value: '{matched_keyword_or_ai_reasoning}'
- name: On-Call Provider
value: '{on_call_provider_name}'
- name: SMS Sent
value: 'Yes - awaiting ACK'
text: 'Message: {message_full_text}'
potentialAction:
- type: OpenUri
name: Open in Portal
targets:
- os: default
uri: '{direct_message_link}'
delay_seconds: 0
log: true
- step: 4
name: check_primary_ack
action: wait_for_condition
condition: sms_ack_received
timeout_seconds: 300
on_timeout: step_5
on_success: step_7
- step: 5
name: secondary_escalation_phone_call
action: automated_phone_call
recipient: on_call_provider_phone
message_tts: |
This is an automated emergency alert from {practice_name}.
A patient message has been classified as an emergency and requires
your immediate review. Please check your SMS and patient portal.
Press 1 to acknowledge.
delay_seconds: 0
log: true
wait_for_dtmf: '1'
dtmf_timeout_seconds: 60
on_timeout: step_6
on_success: step_7
- step: 6
name: tertiary_escalation_backup_provider
action: send_sms
recipient: backup_provider_phone
message: |
🚨 ESCALATION: Primary on-call ({on_call_provider_name})
did not respond to emergency patient message.
Patient: {patient_name} (DOB: {patient_dob})
Original message time: {timestamp}
Please review immediately: {direct_message_link}
Also notifying office manager.
delay_seconds: 0
log: true
parallel:
- action: send_sms
recipient: office_manager_phone
message: '🚨 Emergency escalation: On-call provider not responding. Backup notified. Patient: {patient_name}. See Teams channel.'
- action: send_email
recipient: practice_admin_email
subject: 'CRITICAL: Unacknowledged Emergency Patient Message'
body: 'Full audit trail attached. Immediate action required.'
- step: 7
name: log_resolution
action: create_audit_record
record:
message_id: '{message_id}'
classification: EMERGENCY
patient_response_sent: '{step_1_timestamp}'
provider_notified: '{step_2_timestamp}'
provider_acknowledged: '{ack_timestamp}'
escalation_level_reached: '{highest_escalation_step}'
total_time_to_ack_seconds: '{time_to_ack}'
log: true
configuration:
on_call_schedule:
source: 'ehr_schedule_api' # Pull from EHR on-call calendar
fallback: 'static_rotation' # Manual rotation if API unavailable
static_rotation:
monday: {provider_id: 'DR001', phone: '+1XXXXXXXXXX'}
tuesday: {provider_id: 'DR002', phone: '+1XXXXXXXXXX'}
# ... complete weekly rotation
backup_providers:
- {provider_id: 'DR003', phone: '+1XXXXXXXXXX', role: 'Medical Director'}
office_manager:
phone: '+1XXXXXXXXXX'
email: 'manager@practice.com'Implementation Notes:
- For Sully.ai: Configure via Settings > Escalation Workflows > Emergency Protocol. Map the above steps to Sully's built-in escalation tiers.
- For DeepCura: Use the Urgent Alert configuration. SMS notifications are built-in; add Teams webhook via Settings > Integrations.
- For ThreoAI/custom: Implement the full workflow using the platform's workflow engine or a custom integration using Twilio (SMS/Voice) + Microsoft Graph API (Teams).
- Twilio integration (if building custom): SMS at ~$0.0079/message, Voice at ~$0.013/min. Sign BAA with Twilio before use with PHI.
Test this workflow monthly with a simulated emergency message. Log the test and response times. This is analogous to testing fire alarms—it must work when needed.
Administrative Auto-Responder
Type: agent
An AI agent that automatically responds to administrative (non-clinical) patient messages without requiring clinician review. Handles appointment scheduling requests, prescription refill acknowledgments, referral status inquiries, billing/insurance routing, medical records requests, and general office information queries. Integrates with the EHR scheduling API to offer real-time available appointment slots.
Implementation:
Administrative Auto-Responder Agent
System Prompt
ADMIN_RESPONDER_SYSTEM_PROMPT = """
You are an administrative assistant for {practice_name}.
You help patients with non-clinical requests by providing helpful,
accurate information and taking appropriate actions.
You are NOT a medical professional. You NEVER provide medical advice.
If a patient's message contains ANY clinical concern mixed with an
administrative request, classify it as clinical and do not auto-respond.Your Capabilities:
Practice Information:
- Name: {practice_name}
- Phone: {practice_phone}
- Address: {practice_address}
- Hours: {business_hours}
- Portal URL: {patient_portal_url}
- Providers: {provider_list_with_specialties}
- Accepted Insurance: {insurance_list}
Response Rules:
Available Actions (call via function/tool):
- get_available_appointments(provider_id, date_range, appointment_type)
- create_appointment_request(patient_id, preferred_times, appointment_type, reason)
- submit_refill_request(patient_id, medication_name)
- check_referral_status(patient_id, referral_id)
- route_to_billing(patient_id, message_summary)
- route_to_records(patient_id, request_type)
Function Definitions (for LLM tool calling)
tools = [
{
"type": "function",
"function": {
"name": "get_available_appointments",
"description": "Retrieve available appointment slots from the EHR scheduling system",
"parameters": {
"type": "object",
"properties": {
"provider_id": {
"type": "string",
"description": "Provider ID (or 'any' for any available provider)"
},
"date_range_start": {
"type": "string",
"format": "date",
"description": "Start date for availability search (YYYY-MM-DD)"
},
"date_range_end": {
"type": "string",
"format": "date",
"description": "End date for availability search (YYYY-MM-DD)"
},
"appointment_type": {
"type": "string",
"enum": ["new_patient", "follow_up", "annual_physical",
"sick_visit", "procedure", "consultation",
"cleaning", "exam", "emergency_dental"],
"description": "Type of appointment requested"
}
},
"required": ["date_range_start", "appointment_type"]
}
}
},
{
"type": "function",
"function": {
"name": "submit_refill_request",
"description": "Submit a prescription refill request to the provider refill queue",
"parameters": {
"type": "object",
"properties": {
"patient_id": {"type": "string"},
"medication_name": {"type": "string"},
"pharmacy_name": {"type": "string", "description": "Preferred pharmacy"},
"urgency": {"type": "string", "enum": ["routine", "running_low", "out"]}
},
"required": ["patient_id", "medication_name"]
}
}
},
{
"type": "function",
"function": {
"name": "route_to_department",
"description": "Route a message to a specific department for human handling",
"parameters": {
"type": "object",
"properties": {
"patient_id": {"type": "string"},
"department": {"type": "string", "enum": ["billing", "records", "referrals", "front_desk"]},
"summary": {"type": "string", "description": "Brief summary for the department"}
},
"required": ["patient_id", "department", "summary"]
}
}
}
]EHR API Integration Examples
# athenahealth: Get available appointment slots
def get_available_appointments_athena(provider_id, start_date, end_date, appt_type):
response = requests.get(
f'https://api.platform.athenahealth.com/v1/{PRACTICE_ID}/appointments/open',
headers={'Authorization': f'Bearer {get_athena_token()}'},
params={
'providerid': provider_id,
'departmentid': DEPARTMENT_ID,
'appointmenttypeid': APPT_TYPE_MAP[appt_type],
'startdate': start_date.strftime('%m/%d/%Y'),
'enddate': end_date.strftime('%m/%d/%Y')
}
)
slots = response.json().get('appointments', [])
return format_slots_for_patient(slots)# Open Dental (via CRMBridge): Get available slots
def get_available_appointments_opendental(provider_id, start_date, end_date):
response = requests.get(
f'https://api.crmbridge.com/v1/appointments/available',
headers={'Authorization': f'Bearer {CRMBRIDGE_API_KEY}'},
params={
'provider_id': provider_id,
'start_date': start_date.isoformat(),
'end_date': end_date.isoformat()
}
)
return response.json()Platform Configuration:
- Sully.ai: AI Receptionist module handles administrative auto-responses natively. Configure appointment types and business rules in Settings > Receptionist > Scheduling Rules.
- Arini (dental): Native scheduling auto-response is core functionality. Configure appointment types, provider schedules, and waitlist in Settings > Schedule.
- ThreoAI: Deploy the above prompt and tools as a custom agent. Connect EHR APIs via the platform's integration framework.
Triage Analytics & Quality Dashboard
Type: integration
A monitoring integration that tracks AI triage performance metrics, identifies classification errors, monitors SLA compliance, and generates weekly/monthly reports for the MSP and practice management. Connects to the AI platform's analytics API and presents data in a format suitable for both MSP service reviews and HIPAA compliance audits.
Implementation:
Triage Analytics & Quality Dashboard
Metrics to Track
metrics:
operational:
- total_messages_processed_daily
- messages_by_category:
- emergency_count
- urgent_clinical_count
- routine_clinical_count
- administrative_count
- auto_response_count
- clinician_review_queue_depth
- average_time_to_patient_response_minutes
- average_time_to_clinician_review_minutes
quality:
- clinician_approval_rate_percent # % of AI drafts approved without edits
- clinician_edit_rate_percent # % of AI drafts approved with edits
- clinician_rejection_rate_percent # % of AI drafts rejected entirely
- classification_override_rate # % of messages reclassified by clinicians
- false_negative_emergency_count # CRITICAL: emergencies missed by AI
- false_positive_emergency_count # non-emergencies flagged as emergency
compliance:
- sla_breach_count_by_category
- escalation_acknowledgment_time_avg
- audit_log_completeness_percent
- baa_expiration_alerts
patient_satisfaction:
- response_time_improvement_vs_baseline
- patient_portal_message_volume_trend
- patient_complaint_count_ai_relatedWeekly Report Template (auto-generated)
AI Triage Weekly Report - {practice_name}
Week of {start_date} to {end_date}
Volume Summary
- Total Messages — This Week: {total} | Last Week: {prev_total} | Trend: {trend_arrow}
- Emergency — This Week: {emerg} | Last Week: {prev_emerg} | Trend: {trend}
- Urgent Clinical — This Week: {urgent} | Last Week: {prev_urgent} | Trend: {trend}
- Routine Clinical — This Week: {routine} | Last Week: {prev_routine} | Trend: {trend}
- Administrative — This Week: {admin} | Last Week: {prev_admin} | Trend: {trend}
Performance
- Avg Response Time (Admin): {min} min | Target: <5 min | Status: ✅/❌
- Avg Response Time (Routine): {hrs} hrs | Target: <24 hrs | Status: ✅/❌
- Avg Response Time (Urgent): {min} min | Target: <240 min | Status: ✅/❌
- Emergency Escalation Time: {sec} sec | Target: <60 sec | Status: ✅/❌
- Clinician Approval Rate: {pct}% | Target: >80% | Status: ✅/❌
- SLA Breaches: {count} | Target: 0 | Status: ✅/❌
- False Negative Emergencies: {count} | Target: 0 | Status: ✅/❌
Action Items
{auto_generated_action_items_based_on_metrics}
Implementation via Power BI / Microsoft 365
# Export metrics to SharePoint list for Power BI consumption
import requests
def export_weekly_metrics(metrics: dict):
# Write to SharePoint List via Microsoft Graph API
graph_url = 'https://graph.microsoft.com/v1.0/sites/{site-id}/lists/{list-id}/items'
headers = {
'Authorization': f'Bearer {get_graph_token()}',
'Content-Type': 'application/json'
}
payload = {
'fields': {
'WeekStartDate': metrics['week_start'],
'TotalMessages': metrics['total_messages'],
'EmergencyCount': metrics['emergency_count'],
'UrgentCount': metrics['urgent_count'],
'RoutineCount': metrics['routine_count'],
'AdminCount': metrics['admin_count'],
'AvgResponseTimeAdmin': metrics['avg_response_admin_min'],
'AvgResponseTimeRoutine': metrics['avg_response_routine_hrs'],
'ClinicianApprovalRate': metrics['approval_rate'],
'SLABreaches': metrics['sla_breaches'],
'FalseNegativeEmergencies': metrics['false_neg_emergency']
}
}
requests.post(graph_url, headers=headers, json=payload)Alert Thresholds (trigger MSP notification)
alerts:
critical:
- condition: false_negative_emergency_count > 0
action: immediate_sms_to_msp_lead + practice_clinical_champion
message: 'CRITICAL: AI missed an emergency classification. Immediate review required.'
- condition: escalation_ack_time > 600_seconds
action: sms_to_msp_lead
message: 'Emergency escalation not acknowledged within 10 minutes.'
warning:
- condition: clinician_rejection_rate > 20_percent
action: email_msp_lead
message: 'AI draft quality declining. Review triage rules and prompts.'
- condition: sla_breach_count > 5_per_week
action: email_msp_lead + office_manager
message: 'SLA breaches increasing. Review staffing and queue management.'
- condition: classification_override_rate > 15_percent
action: email_msp_lead
message: 'High override rate suggests triage rules need tuning.'Testing & Validation
- EMERGENCY DETECTION TEST: Send 10 simulated emergency messages through the system (chest pain, suicidal ideation, severe bleeding, difficulty breathing, knocked-out tooth for dental, etc.). Verify ALL 10 are classified as EMERGENCY within 5 seconds. Verify auto-response directing patient to call 911 is sent immediately. Verify on-call provider receives SMS notification within 30 seconds. Zero tolerance for false negatives on this test.
- ESCALATION CASCADE TEST: Send a simulated emergency message and intentionally do NOT acknowledge the primary SMS notification. Verify that the system escalates to phone call after 5 minutes, then to backup provider after 10 minutes. Time each escalation step and verify it matches the configured workflow. Document response times.
- ADMINISTRATIVE AUTO-RESPONSE TEST: Send 20 simulated administrative messages covering all sub-categories (appointment request, refill, billing inquiry, records request, office hours question, insurance question). Verify all receive appropriate auto-responses within 60 seconds. Verify appointment request responses include actual available slots from the EHR. Verify billing and records requests are properly routed to the correct staff queue.
- CLINICAL DRAFT QUALITY TEST: Send 15 simulated clinical questions (medication side effect inquiry, post-procedure concern, lab result question, chronic condition management question, new symptom description). Verify all are classified as ROUTINE_CLINICAL or URGENT_CLINICAL (none as ADMINISTRATIVE). Verify AI draft responses are generated and placed in clinician review queue. Have the clinical champion score each draft 1-5 on accuracy, appropriateness, reading level, and completeness. Target: average score >= 4.0.
- MIXED MESSAGE TEST: Send 10 messages that contain both clinical and administrative content (e.g., 'I need to schedule a follow-up and also my arm has been hurting since the procedure'). Verify the system classifies these as clinical (not administrative) and does NOT auto-respond with scheduling info alone. The clinical concern must be routed for clinician review.
- EHR INTEGRATION VERIFICATION: Verify bidirectional data flow by (1) sending a message and confirming a task/ticket appears in the EHR inbox, (2) having a clinician approve an AI draft and confirming the response appears in the patient's portal message thread, (3) verifying patient context (name, DOB, recent visits, medications) is correctly pulled into the AI's classification context. Check for data accuracy on at least 10 patients.
- HIPAA AUDIT TRAIL TEST: Select 5 random messages processed in the past 24 hours. For each, produce a complete audit trail showing: message receipt timestamp, AI classification, confidence score, action taken, response content, clinician reviewer (if applicable), clinician action, final response sent, and all notification/escalation events. Verify all 5 audit trails are complete with no missing fields. Export audit data and verify it can be produced in under 15 minutes (simulating a compliance audit request).
- AFTER-HOURS BEHAVIOR TEST: Send messages outside business hours covering all categories. Verify: (1) Emergency messages still trigger full escalation to on-call provider, (2) Administrative messages receive 'next business day' acknowledgment, (3) Clinical messages receive 'next business day' acknowledgment with instruction to call 911 if worsening, (4) No auto-scheduling occurs outside business hours unless configured.
- NETWORK SECURITY VALIDATION: Run a vulnerability scan on the practice network using the MSP's standard tools. Verify: (1) TLS 1.2+ is enforced on all AI platform connections, (2) AI vendor API endpoints are whitelisted and accessible, (3) Clinical VLAN is properly segmented from guest network, (4) DNS filtering is blocking unauthorized categories, (5) Firewall logs show encrypted connections to AI vendor IPs only.
- LOAD AND RELIABILITY TEST: Send 50 messages in rapid succession (simulating a Monday morning portal message surge). Verify: (1) All messages are processed without dropping any, (2) Classification accuracy is maintained under load, (3) Auto-responses are sent within acceptable timeframes, (4) The clinician review queue correctly orders messages by priority (emergency > urgent > routine), (5) No duplicate responses are sent.
- PATIENT EXPERIENCE TEST: Have 5 test patients (staff members using test portal accounts) send realistic messages and evaluate the experience from the patient perspective. Assess: (1) Response time perception, (2) Response quality and helpfulness, (3) AI disclosure clarity, (4) Ease of escalating if AI response is insufficient, (5) Overall satisfaction compared to previous manual process. Collect qualitative feedback.
- ROLLBACK PROCEDURE TEST: Simulate a critical system failure by disabling the AI platform. Verify: (1) Office manager can disable auto-response within 2 minutes using documented procedure, (2) All messages fall back to manual staff processing, (3) No messages are lost during the transition, (4) Staff notification of manual mode is sent via Teams within 5 minutes, (5) System can be re-enabled and resume processing without data loss.
Client Handoff
Client Handoff Checklist
Training Delivered (Document attendance with sign-off sheets)
Documentation Package Delivered
Success Criteria Review (review with practice leadership)
Handoff Meeting Agenda
- Walk through 1 week of production metrics together
- Review any open issues or edge cases discovered during go-live
- Confirm all documentation is accessible and understood
- Review ongoing MSP support scope and SLA
- Schedule first monthly optimization review (30 days post go-live)
- Schedule first quarterly compliance audit (90 days post go-live)
- Exchange emergency contact information for critical issues
- Obtain written sign-off from practice owner/manager on project completion
Maintenance
Ongoing MSP Maintenance Responsibilities
Weekly (30-60 minutes)
- Review AI triage performance dashboard: check classification accuracy, auto-response rates, SLA compliance, and escalation metrics
- Review any false negative emergency alerts (investigate and remediate immediately if any occurred)
- Check clinician override rate—if >15%, schedule triage rule tuning session
- Verify EHR/PMS API connectivity is healthy (check for authentication errors or timeout increases)
- Review AI platform vendor status page for any incidents or upcoming maintenance
Monthly (2-3 hours)
- Detailed metrics review with practice office manager (present monthly report)
- Triage rule optimization session: review misclassified messages from the past month with clinical champion, adjust rules and prompts as needed
- Update auto-response templates based on staff feedback and seasonal needs (flu season, holiday hours, etc.)
- Verify audit log integrity and export monthly compliance snapshot
- Test emergency escalation workflow with simulated message (document test results)
- Review and update on-call provider rotation in AI platform
- Check for AI platform software updates; review release notes for breaking changes before applying
- Invoice client for monthly managed service
Quarterly (4-6 hours)
- Comprehensive HIPAA compliance audit: verify all BAAs are current, review access controls, audit log completeness, risk assessment updates
- AI performance deep-dive: analyze classification trends, identify new message categories or edge cases emerging, assess whether additional templates are needed
- Staff refresher training (30 minutes): review any workflow changes, address questions, reinforce error reporting
- Network security review: verify firewall rules, TLS enforcement, DNS filtering, and VLAN segmentation remain properly configured
- Review AI vendor compliance certifications (SOC 2 reports, HIPAA attestations) for any changes
- Strategic review with practice leadership: ROI assessment, expansion opportunities (add voice channel, additional specialties, etc.)
Annually
- Full HIPAA risk assessment update including AI system
- BAA renewal review with all vendors
- Comprehensive staff retraining (all roles)
- Platform version upgrade planning (major releases)
- Contract renewal with client
SLA Considerations
- Critical (Emergency escalation failure, PHI breach, complete system down): 15-minute MSP response, 1-hour resolution target
- High (EHR integration failure, auto-response malfunction, triage misclassification pattern): 1-hour MSP response, 4-hour resolution target
- Medium (Dashboard issues, template updates needed, minor configuration changes): 4-hour MSP response, next business day resolution
- Low (Feature requests, optimization suggestions, reporting questions): Next business day response, scheduled maintenance window
Escalation Path
Model/Prompt Retraining Triggers
- Clinician rejection rate exceeds 20% for 2 consecutive weeks
- New medical service or specialty added at the practice
- Significant change in message volume or category distribution
- AI platform vendor releases new model version
- Regulatory changes affecting AI disclosure or triage requirements
- Post-incident review identifies systematic classification errors
Alternatives
Turnkey Platform: DeepCura (Budget Medical)
Use DeepCura at $129/provider/month as the primary AI platform instead of Sully.ai. DeepCura offers all-in-one functionality including AI scribe, receptionist, billing, and triage in a single flat-rate subscription. It uses a link-based integration approach that bypasses some API complexity, and includes 12 pre-built call templates. Best for solo practitioners or small practices (1-5 providers) who want maximum features at the lowest per-provider cost.
Tradeoffs
- Cost: Lower per-provider cost ($129 vs. $79 for Sully.ai base, but Sully.ai scales better for larger practices).
- Complexity: Slightly simpler setup with link-based integration.
- Capability: Fewer modular options than Sully.ai—all features bundled rather than selectable. Less proven at scale for larger practices.
- Recommend when: budget is the primary constraint, practice has 1-5 providers, and the practice values simplicity over customization.
White-Label MSP Platform: ThreoAI by Synthreo
Instead of reselling a third-party healthcare AI platform, build a branded AI triage service using Synthreo's ThreoAI white-label platform. This gives the MSP full control over branding, pricing, and client experience. ThreoAI supports multiple LLM backends (GPT via Azure ZDR, Claude, Gemini, Mistral, LLaMA), offers per-tenant configuration with zero data retention by default, and is listed on the ConnectWise Marketplace. Requires more MSP effort to configure the clinical triage prompts and workflows, but creates a proprietary service offering.
- Cost: Higher MSP investment in setup and ongoing management; typical client pricing $500–$2,500/month.
- Complexity: Significantly more complex—MSP must build triage prompts, manage LLM configurations, and handle clinical workflow design.
- Capability: Maximum flexibility and customization; can serve multiple healthcare verticals with different configurations.
- Risk: MSP assumes more responsibility for clinical safety of the AI triage logic.
- Recommend when: MSP wants to build a scalable, branded healthcare AI practice serving multiple clients, has AI/ML expertise on staff, and is willing to invest in a long-term service line.
Microsoft Azure Healthcare Agent Service (Health Bot)
Use Microsoft's Azure Healthcare Agent Service (formerly Health Bot) as the AI triage engine. This is a cloud platform providing healthcare-specific agent instances with built-in clinical safeguards, a generative AI-powered healthcare orchestrator, and native Azure AD integration. Free tier allows 3,000 messages and 100 MCUs per month for proof-of-concept. Integrates natively with Microsoft 365 ecosystem and Azure OpenAI Service. Best for practices already heavily invested in the Microsoft ecosystem.
Tradeoffs
- Cost: Free tier for POC; paid tiers scale with usage (generally $0.50–$1.00 per session).
- Complexity: Highest complexity—requires Azure subscription management, custom development of triage scenarios, and more hands-on integration work.
- Capability: Most customizable option with access to the full Azure AI services suite; excellent for building highly specialized triage logic. Requires developer resources (not just IT configuration).
- Recommend when: practice or MSP has Azure development capability, wants maximum control over the AI pipeline, or has unique requirements not met by turnkey platforms.
Communication-Layer Approach: Klara + Manual Triage Enhancement
Instead of deploying a full AI triage agent, implement Klara's patient communication platform with its intelligent message routing (non-AI rule-based triage) to improve message management without AI-generated responses. Klara routes messages to the right staff member/team based on keywords and message type, supports secure two-way messaging, and offers telemedicine integration. AI is used only for routing classification, not for generating patient-facing responses.
Tradeoffs
- Cost: Lower total cost ($300–$800/month) and lower compliance risk since AI is not generating PHI-containing responses.
- Complexity: Much simpler deployment (2–3 weeks); lower HIPAA compliance burden since no AI-generated clinical content.
- Capability: Significantly less automation—staff still write all responses manually; AI only helps sort and route messages.
- Time savings are 30–40% vs. 50–70% with full AI triage.
Recommend when: the practice is risk-averse about AI-generated clinical communications, the regulatory environment is uncertain, or the practice wants to start with routing automation and add AI response generation later as a Phase 2.
Dental-Specific: TrueLark Multi-Channel
For dental practices that want broader channel coverage beyond portal messages, deploy TrueLark (~$199/month) instead of Arini (~$249/month). TrueLark covers phone calls, SMS, and webchat in addition to portal messages, providing a more comprehensive patient communication AI. Slightly less deep PMS integration than Arini but broader channel coverage.
Tradeoffs: Cost: Lower monthly cost ($199 vs. $249). Complexity: Similar deployment complexity. Capability: Broader channel coverage (phone + SMS + webchat) but less specialized PMS integration than Arini's native Dentrix/Eaglesoft/Open Dental connectors. Arini has stronger scheduling intelligence and waitlist management. Recommend when: the dental practice receives significant patient communication via phone and SMS in addition to portal messages, and wants a single platform covering all channels.
Want early access to the full toolkit?