60 min readAutonomous agents

Implementation Guide: Triage patient portal messages, answer routine questions, escalate urgent concerns

Step-by-step implementation guide for deploying AI to triage patient portal messages, answer routine questions, escalate urgent concerns for Healthcare clients.

Hardware Procurement

Next-Generation Firewall

FortinetFortiGate 40F (FG-40F)Qty: 1

$400–$500 MSP cost / $700–$900 suggested resale (includes initial config)

HIPAA-required network perimeter security with application-layer filtering, TLS inspection, and the ability to whitelist AI vendor API endpoints while blocking unauthorized PHI exfiltration. Required for HIPAA technical safeguard compliance if practice lacks a compliant firewall.

Managed PoE Network Switch

UbiquitiUSW-Pro-24-PoE (USW-Pro-24-PoE)Qty: 1

$400 MSP cost / $600 suggested resale

VLAN-capable managed switch to segment clinical network traffic from guest/IoT networks, a HIPAA technical safeguard requirement. PoE ports support VoIP phones if voice triage channel is added later.

Wireless Access Point (HIPAA-compliant)

UbiquitiU6-Pro (U6-Pro)Qty: 2

$150/unit MSP cost / $250/unit suggested resale

Secure wireless coverage for staff tablets and mobile devices used to receive AI escalation alerts. Supports VLAN tagging for network segmentation between clinical and guest wireless.

UPS Battery Backup

APC by Schneider ElectricSMT1500RM2UC Smart-UPS 1500VAQty: 1

$550 MSP cost / $800 suggested resale

Protects firewall, switch, and internet connectivity from power interruptions, ensuring AI triage and escalation alerts remain operational during outages. Critical for practices where urgent message escalation is life-safety relevant.

Software Procurement

~$79/provider/month MSP cost (Pro tier ~$1,580/mo for 20 clinicians) / Suggested resale: $129–$149/provider/month

Primary AI triage platform for medical practices. Includes AI Receptionist (inbound call handling, scheduling, FAQs), AI Nurse (intake, symptom collection, triage routing), AI Scribe, and EHR integration. SOC 2 Type II, HIPAA, HITRUST, and ISO 27001 certified. Sub-second response times via LPU architecture. License type: SaaS per-provider monthly.

DeepCura AI Platform (Alternative for Budget-Conscious Medical Practices)

DeepCuraSaaS per-provider monthlyQty: per provider/month

$129/provider/month (all features included) / Suggested resale: $179–$229/provider/month

All-in-one AI platform including scribe, receptionist, billing, and triage functions. Best value for solo to mid-size practices. Includes EHR integration with Epic, eClinicalWorks, OptiMantra, Athena, and Veradigm. HIPAA and CASA certified. 12 pre-built call templates and 15+ callable AI functions.

Arini Dental AI Receptionist (Dental Practices - Primary Recommendation)

AriniSaaS monthly subscriptionQty: per practice

~$249/month per practice MSP cost / Suggested resale: $349–$449/month

Purpose-built dental AI receptionist with native integration to Dentrix, Eaglesoft, Open Dental, Curve, and Dentrix Ascend. Handles appointment booking, patient inquiry routing, cancellation waitlist management, and triage of dental concerns. HIPAA compliant with real-time schedule optimization.

Klara Patient Communication Platform

Klara (a Hearst Health company)SaaS subscription (quote-based)

Custom pricing, typically $300–$800/month per practice / Suggested resale: quote + 25% markup

Secure patient messaging hub that serves as the communication layer between patients and the AI triage system. Features two-way messaging, intelligent message routing, telemedicine with virtual waiting rooms, and integrations with athenaOne, eClinicalWorks, AdvancedMD, Greenway, ModMed, Nextech, and Veradigm. No app download required for patients.

ThreoAI by Synthreo (MSP White-Label Option)

SynthreoSaaS multi-tenant, MSP white-label

MSP partner pricing varies; typical MSP resale price $500–$2,500/client/month

White-label AI assistant platform purpose-built for MSPs. Supports GPT (Azure ZDR), Claude, Gemini, Mistral, and LLaMA models with custom domains, per-tenant configuration, zero data retention by default, and ConnectWise Marketplace integration. SOC 2, HIPAA, and GDPR compliant. Use this if you want to build a branded AI triage service rather than reselling a third-party platform.

$22/user/month MSP cost / $30–$35/user/month suggested resale

Provides secure email for escalation notifications, Microsoft Teams for internal clinical escalation channels, Azure AD for SSO/MFA to AI platforms, and Microsoft Purview compliance tools for HIPAA audit logging. Most practices already have this; verify license tier supports required compliance features.

DNSFilter (DNS Security)

DNSFilterSaaS per-seat monthly

$1.15/user/month MSP cost / $3–$5/user/month suggested resale

DNS-layer security that prevents PHI exfiltration to unauthorized endpoints, blocks malicious domains, and provides content filtering. HIPAA-supportive network control. Integrates with RMM tools for centralized MSP management.

CRMBridge API (Dental PMS Integration Layer)

CRMBridgeAPI usage-based with free tierQty: Free tier: 100K API calls/month; paid tiers from ~$50/month

Free tier: 100K API calls/month; paid tiers from ~$50/month / Suggested resale: bundled into dental AI service

Provides a unified REST API to integrate with Dentrix, Eaglesoft, Open Dental, and 27+ dental PMS systems. HIPAA-compliant real-time patient data sync. Essential middleware if using a custom-built or white-label AI solution instead of Arini for dental practices.

Compliancy Group (HIPAA Compliance Platform)

Compliancy GroupSaaS annual subscriptionQty: 1

~$300–$400/month for MSP partner program / Resale: bundled into compliance service at $500/month per practice

Managed HIPAA compliance platform for generating risk assessments, managing BAAs, tracking employee training, documenting policies and procedures, and providing HIPAA Seal of Compliance. Streamlines the compliance overhead that is mandatory for this AI deployment.

Prerequisites

  • Active EHR system (medical) or PMS (dental) with API access enabled. For medical: athenahealth, Epic, eClinicalWorks, AdvancedMD, Veradigm, or ModMed. For dental: Dentrix, Eaglesoft, Open Dental, Curve, or Dentrix Ascend.
  • Patient portal already deployed and actively used by patients for messaging (minimum 30% patient adoption recommended before AI triage adds value).
  • Business-grade internet connection: minimum 50 Mbps symmetric; fiber preferred. Verify uptime SLA with ISP—AI triage is time-sensitive for urgent escalations.
  • HIPAA-compliant network infrastructure: next-gen firewall with TLS 1.2+ inspection, VLAN segmentation between clinical and guest networks, DNS filtering. If missing, include hardware procurement items above.
  • Designated Privacy Officer / HIPAA Compliance Officer at the practice (required by HIPAA; may be the office manager or a provider).
  • Signed Business Associate Agreement (BAA) between the practice and the MSP. The MSP must have its own HIPAA compliance program in place before handling PHI.
  • Administrative credentials for the EHR/PMS system with sufficient privileges to enable API integrations, create webhook endpoints, and configure message routing.
  • Identified clinical champion: at least one licensed provider (MD, DO, DDS, DMD, PA, NP) who will collaborate on triage rule definition, validate AI response templates, and serve as the clinical escalation reviewer.
  • Microsoft 365 Business Premium or equivalent for SSO/MFA, secure email, and Teams-based escalation channels. Azure AD tenant configured for the practice.
  • Current HIPAA risk assessment completed within the past 12 months. If not available, this must be conducted as Phase 1 of the project (add 2 weeks to timeline).
  • Staff availability for training: minimum 2-hour initial training session for all portal-managing staff, plus 1-hour session for providers on AI review workflows.
  • For dental practices using Arini: active PMS login credentials and admin access to the scheduling module for real-time calendar integration.

Installation Steps

Step 1: HIPAA Compliance Foundation & BAA Execution

Before any technology is deployed, establish the complete HIPAA compliance framework for this AI implementation. This is non-negotiable—deploying AI that processes PHI without proper BAAs and risk assessment exposes both the MSP and the practice to significant liability (fines up to $50,000 per violation). Execute BAAs with every vendor in the solution stack: AI platform vendor (Sully.ai, DeepCura, or Arini), communication platform (Klara if used), DNS filtering provider, and any cloud infrastructure providers. Conduct or update the practice's HIPAA risk assessment to include AI-specific risks: model hallucination, PHI exposure in AI training data, unauthorized access to AI admin consoles, and AI decision audit trail gaps.

BAA tracking template and vendor compliance verification reference

1
Document all BAAs in a tracking spreadsheet Template: Vendor Name | BAA Execution Date | BAA Expiration | PHI Types Covered | Data Retention Policy | Breach Notification Terms
2
Example BAA tracking entry:
3
Sully.ai | 2025-01-15 | Perpetual until termination | Patient messages, demographics, appointment data | 0-day retention (real-time processing only) | 24-hour notification to covered entity
4
Verify vendor compliance certifications: - Sully.ai: SOC 2 Type II, HIPAA, HITRUST, ISO 27001 - DeepCura: HIPAA, CASA Certified - Arini: HIPAA compliant
5
Download and archive vendor SOC 2 reports and HIPAA attestation letters
Critical

Do not proceed to Step 2 until ALL BAAs are fully executed and filed. BAA execution with AI vendors typically takes 3–10 business days. Sully.ai and DeepCura include BAAs in their standard onboarding; Klara requires a separate request. If using ThreoAI/Synthreo for white-label, their BAA covers the platform but you must execute a separate BAA between your MSP and the practice. Keep copies of all BAAs for a minimum of 6 years per HIPAA retention requirements.

Step 2: Network Security Hardening & HIPAA Technical Safeguards

Configure the practice's network infrastructure to meet HIPAA technical safeguard requirements and securely support cloud AI platform connectivity. This includes firewall configuration, VLAN segmentation, DNS filtering, TLS enforcement, and whitelisting of AI vendor API endpoints. If the practice lacks a compliant firewall, install the Fortinet FortiGate 40F from the hardware procurement list.

FortiGate 40F Initial Configuration (via CLI after initial wizard)
bash
# FortiGate 40F Initial Configuration (via CLI after initial wizard)
config system interface
  edit port1
    set alias 'WAN'
    set mode dhcp
    set allowaccess ping https ssh
  next
  edit port2
    set alias 'Clinical-LAN'
    set ip 10.10.10.1 255.255.255.0
    set allowaccess ping https ssh
    set device-identification enable
  next
  edit port3
    set alias 'Guest-WiFi'
    set ip 10.10.20.1 255.255.255.0
    set allowaccess ping
  next
end

# Create VLAN for clinical traffic isolation
config system interface
  edit 'VLAN-Clinical'
    set vdom root
    set ip 10.10.30.1 255.255.255.0
    set allowaccess ping https
    set interface port2
    set vlanid 30
  next
end

# Enable TLS 1.2+ enforcement (block TLS 1.0/1.1)
config firewall ssl-ssh-profile
  edit 'HIPAA-SSL-Profile'
    config https
      set ports 443
      set status deep-inspection
      set unsupported-ssl-version block
    end
    set ssl-exemption-log enable
    set ssl-anomaly-log enable
    config ssl
      set inspect-all deep-inspection
      set min-allowed-ssl-version tls-1.2
    end
  next
end

# Whitelist AI vendor API endpoints
config firewall address
  edit 'Sully-AI-API'
    set type fqdn
    set fqdn 'api.sully.ai'
  next
  edit 'DeepCura-API'
    set type fqdn
    set fqdn 'api.deepcura.ai'
  next
  edit 'Arini-API'
    set type fqdn
    set fqdn 'api.arini.ai'
  next
  edit 'Klara-API'
    set type fqdn
    set fqdn 'api.klara.com'
  next
end
1
Navigate to https://app.dnsfilter.com and log in to the DNSFilter dashboard
2
Add the practice network's public IP to the DNSFilter dashboard
3
Enable Healthcare category blocking policy
4
Whitelist AI vendor domains in the DNSFilter allow list
Note

If the practice already has a compliant firewall (SonicWall TZ, Meraki MX, etc.), adapt these configurations to that platform. The key requirements are: (1) TLS 1.2+ enforcement, (2) VLAN segmentation between clinical and guest networks, (3) AI vendor endpoint whitelisting, (4) DNS filtering to prevent PHI exfiltration, and (5) full logging enabled for HIPAA audit trail. Document all firewall rules in the practice's HIPAA technical safeguard documentation. Test connectivity to AI vendor endpoints after firewall changes before proceeding.

Step 3: Identity & Access Management Configuration

Configure Single Sign-On (SSO) and Multi-Factor Authentication (MFA) for all AI platform access using Azure AD (Entra ID). Create role-based access control groups that map to clinical workflows: AI Admin (MSP technicians), Clinical Reviewer (providers who approve AI-drafted responses), Office Manager (triage rule configuration), and Read-Only Auditor (HIPAA compliance monitoring). This ensures every access to the AI platform handling PHI is authenticated, authorized, and logged.

1
Create Security Groups in Azure AD Admin Center (https://entra.microsoft.com): SG-AI-Triage-Admins (MSP technicians), SG-AI-Triage-ClinicalReviewers (licensed providers), SG-AI-Triage-OfficeManagers (office staff with triage config access), SG-AI-Triage-Auditors (read-only compliance monitoring)
2
Enable MFA via Conditional Access Policy — Navigate to: Entra ID > Security > Conditional Access > New Policy. Policy Name: 'AI-Triage-MFA-Required'. Assignments > Users: Include SG-AI-Triage-* groups. Cloud Apps: Include the AI platform's Enterprise App (register via SAML/OIDC). Grant: Require MFA + Require compliant device. Session: Sign-in frequency = 8 hours
3
Register AI Platform as Enterprise Application — Navigate to: Entra ID > Enterprise Applications > New Application. Select 'Create your own application' > 'Non-gallery application'. Name: 'Sully AI Triage' (or vendor name). Configure SAML SSO with vendor-provided metadata URL. Map Azure AD groups to vendor RBAC roles
4
Configure Audit Logging — Entra ID > Monitoring > Diagnostic Settings. Enable: Sign-in logs, Audit logs, Provisioning logs. Export to: Log Analytics Workspace (retain for 6+ years per HIPAA)
Export Azure AD sign-in logs for HIPAA compliance reporting
powershell
# PowerShell: Export Azure AD sign-in logs for compliance
Install-Module AzureADPreview
Connect-AzureAD
Get-AzureADAuditSignInLogs -Filter "appDisplayName eq 'Sully AI Triage'" -Top 1000
Note

If the AI vendor does not support SAML/OIDC SSO (some smaller vendors use email/password only), enforce MFA using the vendor's built-in MFA settings and document this as a compensating control in the HIPAA risk assessment. Sully.ai and DeepCura both support SSO integration. For Arini, confirm SSO support during onboarding—if unavailable, enforce strong passwords (16+ chars) and vendor-native MFA. All AI platform credentials must be unique per user—no shared accounts.

Step 4: EHR/PMS API Integration Setup

Enable and configure the bidirectional API connection between the AI triage platform and the practice's EHR (medical) or PMS (dental). This is the most variable step in the deployment—integration complexity depends heavily on which EHR/PMS the practice uses. The goal is to allow the AI agent to: (1) receive inbound patient messages, (2) access relevant patient context (demographics, recent visits, medication list), (3) create tasks/tickets for clinical review, (4) write approved responses back to the patient, and (5) update appointment schedules.

MEDICAL: athenahealth API Setup

1
Register for athenahealth API access at https://developer.athenahealth.com
2
Create an application in the developer portal
3
Request production API key (requires athenahealth review, ~5-10 business days)
4
Configure OAuth 2.0 credentials in the AI platform's integration settings
athenahealth API
bash
# test token and patient document endpoint connectivity

# athenahealth API - Test connectivity
curl -X POST 'https://api.platform.athenahealth.com/oauth2/v1/token' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&scope=athena/service/Athenanet.MDP.*'

# Verify patient message endpoint access
curl -X GET 'https://api.platform.athenahealth.com/v1/{practiceid}/patients/{patientid}/documents' \
  -H 'Authorization: Bearer YOUR_ACCESS_TOKEN'

MEDICAL: Epic (via SMART on FHIR)

1
Register app at https://fhir.epic.com/Developer/Apps
2
Select 'Backend System' app type for server-to-server integration
3
Request required FHIR scopes: Patient.read, Communication.read, Communication.write, Appointment.read, Appointment.write, Task.write
4
Provide vendor's redirect URI and public key for JWT authentication
5
Submit for Epic App Orchard review (allow 2-4 weeks for approval)
Epic FHIR — test patient Communication resource endpoint
bash
# Epic FHIR - Test patient communication endpoint
curl -X GET 'https://fhir.{epic-instance}.org/api/FHIR/R4/Communication?patient={patientId}' \
  -H 'Authorization: Bearer YOUR_ACCESS_TOKEN' \
  -H 'Accept: application/fhir+json'

DENTAL: Arini Native PMS Integration

1
In Arini dashboard, navigate to Settings > Practice Management System
2
Select PMS type: Dentrix | Eaglesoft | Open Dental | Curve | Dentrix Ascend
3
For Dentrix/Eaglesoft: Install Arini bridge agent on PMS server — Download from: https://app.arini.ai/downloads/bridge-agent — Run installer as administrator on the PMS workstation/server — Enter practice API key from Arini dashboard when prompted
4
For Open Dental: Enable API in Open Dental (Setup > Advanced Setup > API) — Generate API key and enter in Arini dashboard
5
For Curve: OAuth connection via Arini dashboard (no local install needed)

DENTAL: CRMBridge Alternative (for custom/white-label builds)

1
Register at https://crmbridge.com and obtain API key
CRMBridge — test patient search endpoint
bash
curl -X GET 'https://api.crmbridge.com/v1/patients?search=Smith' \
  -H 'Authorization: Bearer YOUR_CRMBRIDGE_API_KEY' \
  -H 'Content-Type: application/json'
Note

EHR integration is the highest-variability step. athenahealth has the best-documented public API and fastest approval process (~5-10 business days). Epic requires App Orchard review which can take 2–4 weeks—start this in Phase 1 if the practice uses Epic. For eClinicalWorks, work through the vendor's partner integration team. For dental PMS, Arini's native integrations are the fastest path—their bridge agent installs in under 30 minutes. If using a custom/white-label solution for dental, CRMBridge provides a unified API layer across 27+ dental PMS systems. ALWAYS test API connectivity in a sandbox/dev environment before connecting to production patient data.

Step 5: AI Triage Platform Provisioning & Configuration

Set up the AI triage platform account, configure the practice profile, define message categories, and establish the initial triage classification rules. This step transforms the generic AI platform into a practice-specific triage engine. Work directly with the clinical champion identified in prerequisites to validate all triage rules and response templates.

  • === Sully.ai Setup (Medical Practices) ===
  • Navigate to https://app.sully.ai and create organization account
  • Add practice details: name, NPI, specialty, address, phone, timezone
  • Add providers: full name, credentials, NPI, specialties, schedule
  • Enable AI Receptionist module: Settings > AI Modules > Receptionist > Enable
  • Enable AI Nurse (Triage) module: Settings > AI Modules > Nurse > Enable
  • Configure EHR connection (from Step 4 API credentials)
  • Define triage categories (see Custom AI Components section for full taxonomy)
  • === DeepCura Setup (Alternative Medical) ===
  • Navigate to https://app.deepcura.ai and create account
  • Connect EHR via link-based integration (no API keys needed for basic)
  • Select call/message templates from 12 pre-built options
  • Customize triage escalation rules
  • === Arini Setup (Dental) ===
  • Navigate to https://app.arini.ai and create practice account
  • Connect PMS (completed in Step 4)
  • Configure business hours and appointment types
  • Set up AI greeting and response personality
  • Define dental-specific triage rules (pain, swelling, trauma = urgent)
  • Configure waitlist management for cancellation auto-fill
  • === Common Configuration for All Platforms ===
  • Configure escalation notification channels: Microsoft Teams webhook for clinical escalation channel, SMS/email alerts for on-call provider, office phone for true emergencies
  • === Microsoft Teams Incoming Webhook Setup ===
  • In Teams, navigate to the 'Clinical Escalations' channel
  • Click '...' > Connectors > Incoming Webhook
  • Name the webhook: 'AI Triage Escalation'
  • Copy webhook URL and paste into AI platform's notification settings
Microsoft Teams Incoming Webhook URL format
text
https://outlook.office.com/webhook/GUID/IncomingWebhook/GUID
Note

Spend at least 2 hours in a working session with the clinical champion during this step. They must review and approve every auto-response template, every triage category definition, and every escalation threshold. Document their approval in writing—this is both a compliance requirement and a liability safeguard. The clinical champion should sign off on the triage rule matrix before any patient messages are processed by the AI. For Sully.ai, the initial setup wizard is comprehensive and guides through most configuration; budget 2–3 hours for complete setup. For Arini (dental), the setup is faster—typically under 1 hour for basic configuration.

Step 6: Triage Rule Engine & Escalation Workflow Configuration

Configure the detailed triage classification rules, urgency scoring, auto-response templates, and escalation workflows. This is the clinical heart of the system—the rules that determine which messages get auto-responded, which get queued for clinician review, and which trigger immediate emergency escalation. All rules must be validated by the clinical champion and documented in the practice's clinical protocols.

Triage category configuration reference — emergency keywords, urgency levels, auto-response templates, dental-specific rules, and after-hours behavior

1
=== Configure Triage Categories in AI Platform === Category 1: EMERGENCY (Immediate escalation, no auto-response) Keywords/patterns: chest pain, difficulty breathing, severe bleeding,
2
suicidal thoughts, allergic reaction, loss of consciousness, stroke symptoms
3
Action: Immediate SMS + call to on-call provider + Teams alert Auto-response: 'If you are experiencing a medical emergency, call 911 immediately.
4
Your message has been flagged as urgent and a member of our clinical team
5
has been notified for immediate review.' Category 2: URGENT CLINICAL (Queue for same-day clinician review) Keywords/patterns: fever over 101, new pain, worsening symptoms,
6
medication side effects, post-procedure concerns, abnormal lab questions
7
Action: Priority queue for clinician review + Teams notification Auto-response: 'Thank you for your message. Based on the nature of your
8
concern, a member of our clinical team will review and respond within
9
[2-4 hours]. If your condition worsens, please call our office or 911.' Category 3: ROUTINE CLINICAL (Clinician review, standard queue) Keywords/patterns: general health questions, chronic condition management,
10
non-urgent symptom inquiries, lifestyle/diet questions
11
Action: Draft AI response for clinician review within 24 hours
12
AI drafts response based on practice protocols; clinician approves/edits Category 4: ADMINISTRATIVE - AUTO-RESPOND
13
Sub-categories and auto-actions:
14
4a. Appointment Request → Auto-offer available slots from EHR calendar
15
4b. Prescription Refill → Route to refill queue + confirm receipt
16
4c. Referral Status → Pull status from EHR and auto-respond
17
4d. Insurance/Billing → Route to billing department + confirm receipt
18
4e. Medical Records Request → Provide portal instructions + route to HIM
19
4f. Office Hours/Location → Auto-respond with practice info
20
=== Dental-Specific Triage Rules (Arini) === EMERGENCY: Knocked-out tooth, uncontrolled oral bleeding, jaw fracture,
21
facial swelling restricting airway URGENT: Severe toothache, broken tooth with sharp edges, lost filling/crown,
22
swelling (non-airway), post-extraction dry socket symptoms ROUTINE: Teeth whitening questions, cosmetic consultations, cleaning scheduling ADMINISTRATIVE: Insurance verification, payment plans, appointment changes
23
=== Configure After-Hours Behavior === During business hours: Full triage + auto-response + clinician queue After hours: Emergency escalation to on-call provider; all other categories: acknowledge receipt + 'Our team will respond
24
during the next business day. If this is an emergency, call 911
25
or go to your nearest emergency room.'
Critical

CRITICAL SAFETY NOTE: The emergency category must ALWAYS include a directive to call 911. The AI must never attempt to provide clinical advice for emergency-level concerns. The auto-response for emergencies should be hard-coded and not editable by AI—use a static template. For dental practices, knocked-out permanent teeth require same-day response (reimplantation within 1 hour has best prognosis), so this falls in the emergency category even though it may not seem life-threatening. Review the triage rules monthly for the first 3 months, then quarterly. Document every rule change with clinical champion sign-off.

Step 7: AI Response Template Library Configuration

Create and configure the library of AI response templates that the system uses to auto-respond to routine inquiries and draft responses for clinical review. Templates must be written in plain language (6th-grade reading level per health literacy best practices), approved by the clinical champion, and tagged with appropriate metadata for audit trails. Each template should include the practice name, a clear disclaimer about AI-assisted communication, and appropriate next-step instructions.

Template Structure
json
# JSON format for API-based platforms. Store in AI platform's template
# library or custom knowledge base.

// Example: Appointment Request Auto-Response
{
  "template_id": "ADMIN-APPT-001",
  "category": "administrative",
  "subcategory": "appointment_request",
  "requires_clinician_review": false,
  "response_template": "Hello {patient_first_name}, thank you for reaching out to {practice_name}. We'd be happy to help you schedule an appointment.\n\nBased on your request, here are available times:\n{available_slots}\n\nYou can also schedule directly through your patient portal at {portal_url} or call us at {practice_phone}.\n\nThis message was prepared with AI assistance and reviewed by our team.\n\nBest regards,\n{practice_name}",
  "variables": ["patient_first_name", "practice_name", "available_slots", "portal_url", "practice_phone"],
  "approved_by": "Dr. Smith",
  "approved_date": "2025-01-15",
  "last_reviewed": "2025-01-15"
}

// Example: Prescription Refill Acknowledgment
{
  "template_id": "ADMIN-REFILL-001",
  "category": "administrative",
  "subcategory": "prescription_refill",
  "requires_clinician_review": false,
  "response_template": "Hello {patient_first_name}, we received your prescription refill request for {medication_name}. Your request has been placed in the refill queue and will be reviewed by your provider within {refill_sla} business days.\n\nIf you need the medication urgently, please call our office at {practice_phone}.\n\nThis message was prepared with AI assistance.\n\n{practice_name}",
  "variables": ["patient_first_name", "medication_name", "refill_sla", "practice_phone", "practice_name"]
}

// Example: Clinical Question — AI Draft for Clinician Review
{
  "template_id": "CLINICAL-DRAFT-001",
  "category": "clinical",
  "subcategory": "general_question",
  "requires_clinician_review": true,
  "ai_draft_instructions": "Draft a response using the patient's medical history from EHR context. Use plain language at 6th-grade reading level. Do NOT provide a diagnosis. Recommend follow-up if appropriate. Include standard disclaimer.",
  "disclaimer_footer": "This response was drafted with AI assistance and reviewed by {reviewing_provider}. It is not a substitute for an in-person evaluation. If your symptoms change or worsen, please contact our office immediately or call 911 for emergencies."
}
Note

Templates should be stored in a version-controlled document (SharePoint or the AI platform's native template system) with full change history. Every template must include the AI disclosure notice—Texas, California, Colorado, and other states are passing laws requiring disclosure of AI interaction. Even where not yet legally required, it is a best practice and builds patient trust. Review all templates quarterly with the clinical champion. For dental practices, create dental-specific templates covering post-procedure care FAQs, insurance coverage explanations, and treatment plan questions. Maintain a separate 'Denied Templates' log documenting any auto-response scenarios the clinical champion explicitly rejected.

Note

AI Disclosure Notice (required by a growing number of state laws): Include on ALL AI-generated or AI-assisted messages: 'This message was prepared with the assistance of AI technology and reviewed by our clinical team.'

Step 8: Pilot Deployment & Shadow Mode Testing

Deploy the AI triage system in 'shadow mode' where it processes all incoming patient messages but does NOT send auto-responses or take autonomous action. Instead, the AI's classification and draft responses are logged alongside the actual human staff responses for comparison. This 2–4 week pilot phase allows validation of triage accuracy, identification of edge cases, and building clinician confidence before going live. Configure the system to run in parallel with existing workflows.

Enable Shadow/Audit Mode

  • Sully.ai: Settings > Deployment Mode > 'Shadow Mode' / 'Audit Only'
  • DeepCura: Settings > AI Behavior > 'Draft Only - No Auto-Send'
  • Arini: Settings > Testing Mode > Enable

Create Monitoring Dashboard

Set up a daily review process tracking the following metrics:

1
Total messages processed by AI
2
Classification distribution (Emergency / Urgent / Routine / Admin)
3
Concordance rate: AI classification vs. human staff classification
4
False positive rate: Messages AI flagged as urgent that weren't
5
False negative rate: Messages AI missed that were urgent (CRITICAL METRIC)
6
Draft response quality score (clinician 1–5 rating)

Daily Shadow Mode Review Spreadsheet

Create in SharePoint or Google Sheets with the following columns:

  • Date
  • Message ID
  • Patient ID (de-identified for tracking)
  • AI Category
  • Human Category
  • Match?
  • AI Draft Response
  • Human Response
  • Clinician Quality Score (1–5)
  • Notes

Weekly Metrics Review Meeting

Attendees: MSP tech, clinical champion, office manager

1
Review concordance metrics
2
Review false negative cases in detail (any missed urgent messages)
3
Adjust triage rules based on findings
4
Discuss edge cases and create new rules/templates as needed
5
Go/No-Go assessment for live deployment

Go-Live Criteria

All of the following criteria must be met before proceeding to live deployment:

Warning

The shadow mode phase is non-negotiable for clinical safety. Do NOT skip this step to save time. A 2-week minimum is recommended; 4 weeks is ideal. During this phase, the existing manual workflow continues unchanged—staff respond to all messages as they normally would. The AI processes the same messages in the background. Focus especially on false negatives (urgent messages the AI classified as routine)—these are the patient safety risks. If the false negative rate on urgent/emergency categories exceeds 5% after tuning, do not go live until it is resolved. Document all shadow mode findings in the HIPAA risk assessment as part of the AI system validation evidence.

Step 9: Staff Training & Change Management

Conduct comprehensive training for all practice staff who interact with the AI triage system. This includes providers (who review and approve AI-drafted clinical responses), clinical staff (who monitor the triage queue and handle escalations), and administrative staff (who manage scheduling and billing messages that the AI routes to them). Training must cover both the technical operation of the system and the clinical governance framework.

  • Session 1: All Staff Overview (2 hours) — What the AI triage system does and doesn't do; HIPAA implications and staff responsibilities; AI disclosure requirements to patients; Demo of message flow: patient sends message → AI classifies → action taken; How to identify and report AI errors; Emergency override procedures (how to manually escalate); Q&A
  • Session 2: Providers/Clinicians (1.5 hours) — AI draft response review workflow; How to approve, edit, or reject AI drafts; Clinician override of AI classification; Understanding AI confidence scores; Clinical liability considerations; Triage rule modification request process; Hands-on practice with review queue
  • Session 3: Office Managers/Admin Staff (1 hour) — Administrative message routing workflows; AI-assisted scheduling: how it works, how to correct errors; Monitoring dashboard walkthrough; Escalation notification management; Reporting AI issues to MSP
  • Session 4: On-Call Providers (30 minutes) — Emergency escalation notification flow; How urgent alerts arrive (SMS, Teams, phone); Acknowledging receipt of escalation; After-hours AI behavior and limitations
1
Quick Reference Card (laminated, for each workstation): Triage category definitions with examples; Escalation phone numbers; How to override AI classification; How to report AI errors
2
Full User Guide (PDF, 15–20 pages): Complete workflow documentation with screenshots; FAQ section; Troubleshooting common issues
3
Video Walkthrough (record training sessions): Upload to SharePoint for new hire onboarding
Note

Change management is often harder than the technology. Common staff concerns: 'Will AI replace my job?' (No—it handles routine messages so staff can focus on complex cases), 'What if the AI makes a mistake?' (Clinician-in-the-loop for all clinical content; emergency safeguards), 'Do patients know it's AI?' (Yes—disclosure on every message). Address these proactively. The clinical champion should co-present training sessions to build peer credibility. Plan for 1–2 weeks of increased support tickets after go-live as staff adapt to new workflows.

Step 10: Phased Go-Live & Production Deployment

Transition from shadow mode to live production in a phased approach. Phase A (Week 1): Enable auto-response for administrative messages only (appointment requests, office info, billing routing). Phase B (Week 2): Enable AI draft responses for routine clinical questions, with mandatory clinician review before sending. Phase C (Week 3+): Enable urgent/emergency escalation automation. Never enable clinical auto-response without clinician review—always maintain the human-in-the-loop for clinical content.

Phased go-live configuration notes, escalation setup, production monitoring, and rollback procedure

1
=== Phase A: Administrative Auto-Response (Week 1) ===
2
In AI platform, enable auto-send for categories: - ADMIN-APPT (appointment requests) - ADMIN-INFO (office hours, location, general info) - ADMIN-RECORDS (records request routing)
3
Keep all clinical categories in 'draft only' mode Monitor: volume, patient response rate, error reports === Phase B: Clinical Draft + Review (Week 2) ===
4
Enable AI draft generation for: - CLINICAL-ROUTINE (general health questions) - CLINICAL-CHRONIC (chronic disease management queries) Workflow: AI drafts response → appears in clinician review queue →
5
clinician approves/edits → response sent to patient Monitor: clinician approval rate, edit rate, time-to-response improvement === Phase C: Full Escalation Automation (Week 3+) ===
6
Enable automated urgent/emergency escalation: - EMERGENCY: Immediate SMS + call to on-call provider - URGENT: Priority queue + Teams notification to clinical staff
7
Test escalation with simulated messages before enabling Monitor: escalation volume, response time, false alarm rate
8
=== Production Monitoring Commands ===
9
Set up automated daily report email:
10
Configure in AI platform: Reports > Automated Reports > Daily Summary Recipients: MSP tech lead, office manager, clinical champion Metrics: Total messages, category distribution, auto-response count,
11
average response time, escalation count, error count
12
=== Rollback Procedure (if needed) ===
13
If critical issues arise during any phase: 1. Disable auto-response: AI Platform > Settings > Auto-Response > OFF
14
2. Set all categories to 'Draft Only' mode 3. Notify staff via Teams: 'AI auto-response temporarily disabled;
15
please respond to all messages manually'
16
4. Conduct root cause analysis with MSP and vendor
17
5. Re-enable after fix is validated
Note

The phased approach is essential for patient safety and staff confidence. Administrative auto-responses (Phase A) are low-risk and demonstrate quick value. Clinical draft-and-review (Phase B) is where the biggest time savings for providers occur—studies show 60% of AI draft responses require minimal or no editing. Never enable auto-sending of clinical responses without clinician review—this crosses into potential 'practice of medicine' territory and creates liability. Monitor the system intensively during the first 2 weeks of each phase. Keep the rollback procedure documented and accessible to the office manager so they can disable auto-response immediately if needed, without waiting for the MSP.

Step 11: Compliance Documentation & Audit Trail Verification

Finalize all HIPAA compliance documentation specific to the AI triage deployment. Verify that complete audit trails are being captured, retention policies are configured correctly, and all required policies and procedures are documented. This step produces the documentation package that will withstand a HIPAA audit or OCR investigation.

Required Documentation Package

1
AI-Specific HIPAA Risk Assessment Addendum — Document: risk of PHI exposure via AI processing; risk of AI misclassification (patient safety); risk of AI hallucination in draft responses; mitigations for each risk (clinician review, escalation protocols); residual risk acceptance (signed by practice Privacy Officer)
2
Business Associate Agreement Register — All executed BAAs with AI vendors, cloud providers, communication platforms; annual review schedule for each BAA
3
AI System Configuration Documentation — Triage rules and categories (with clinical champion sign-off); auto-response templates (with approval signatures); escalation workflows and notification channels; integration architecture diagram
4
Staff Training Records — Training attendance sheets; training materials and presentations; competency acknowledgment forms (each staff member signs)
5
Audit Log Verification — Confirm AI platform logs every: message received, classification assigned, response generated, clinician review action, response sent, escalation triggered; confirm log retention is set to minimum 6 years; test log export: can you produce a complete audit trail for any patient message?

Audit Trail Test

Select 5 random patient messages from the past week. For each, verify you can produce all of the following elements. If any element is missing, work with the vendor to resolve before sign-off.

Export Audit Logs for Compliance Archive

  • Navigate to Reports > Audit Logs in your AI platform
  • Select Export and choose your desired Date Range
  • Download in CSV or JSON format
  • Store exports in an encrypted SharePoint folder with a 6-year retention policy applied
Note

This documentation is not optional—it is your liability shield and the practice's compliance evidence. If a patient files a complaint with the HHS Office for Civil Rights about an AI-generated response, this documentation package is what demonstrates the practice took reasonable precautions. Store all documents in an encrypted, access-controlled SharePoint site with automatic retention policies. Review and update the documentation quarterly. The risk assessment addendum should be a living document updated whenever triage rules change, new message categories are added, or the AI platform is updated.

Custom AI Components

Patient Message Triage Classifier

Type: agent

The core AI agent that receives inbound patient portal messages, analyzes their content using NLP and clinical context, assigns an urgency classification (Emergency, Urgent Clinical, Routine Clinical, Administrative), and routes them to the appropriate workflow. This agent operates as the first-touch processor for all patient messages and must have a clinically-validated decision framework with explicit safety guardrails for emergency detection.

Implementation:

Patient Message Triage Classifier Agent

Architecture

This agent runs within the selected AI platform (Sully.ai, DeepCura, or ThreoAI) and uses a multi-stage classification pipeline:

Stage 1: Safety Screen (Rule-Based, No AI) Hard-coded keyword matching that runs BEFORE the LLM to catch emergencies with zero latency:

python
EMERGENCY_KEYWORDS = [
    'chest pain', 'can\'t breathe', 'difficulty breathing', 'shortness of breath',
    'heart attack', 'stroke', 'seizure', 'unconscious', 'unresponsive',
    'severe bleeding', 'won\'t stop bleeding', 'suicidal', 'want to die',
    'kill myself', 'self-harm', 'overdose', 'poisoning', 'anaphylaxis',
    'allergic reaction', 'swelling throat', 'can\'t swallow',
    'severe abdominal pain', 'vomiting blood', 'coughing blood',
    'head injury', 'loss of consciousness', 'numbness face',
    'slurred speech', 'vision loss', 'sudden weakness'
]

DENTAL_EMERGENCY_KEYWORDS = [
    'knocked out tooth', 'tooth fell out', 'avulsed', 'jaw broken',
    'jaw fracture', 'uncontrolled bleeding mouth', 'can\'t breathe swelling',
    'facial swelling airway', 'severe facial trauma'
]

def safety_screen(message_text: str) -> dict:
    message_lower = message_text.lower()
    for keyword in EMERGENCY_KEYWORDS + DENTAL_EMERGENCY_KEYWORDS:
        if keyword in message_lower:
            return {
                'classification': 'EMERGENCY',
                'confidence': 1.0,
                'matched_keyword': keyword,
                'bypass_llm': True,
                'action': 'IMMEDIATE_ESCALATION'
            }
    return {'bypass_llm': False}

Stage 2: LLM Classification (AI-Powered) For messages that pass the safety screen without emergency detection:

python
TRIAGE_SYSTEM_PROMPT = """
You are a medical message triage assistant for {practice_name}, a {practice_type} practice.
Your role is to classify patient portal messages into one of four categories and extract key information.

You are NOT a doctor. You do NOT provide diagnoses or medical advice.
You classify messages to ensure they reach the right person at the right time.

Classification Categories:

EMERGENCY (Immediate escalation required)

Life-threatening symptoms, severe acute conditions, mental health crisis.

Examples: chest pain, difficulty breathing, suicidal ideation, severe allergic reaction, uncontrolled bleeding, stroke symptoms, loss of consciousness.

Critical

ACTION: Immediate alert to on-call provider. Auto-response directing patient to call 911.

URGENT_CLINICAL (Same-day clinician review required)

New or worsening symptoms that need prompt attention but are not immediately life-threatening.

  • fever > 101°F
  • new onset pain
  • medication side effects
  • post-surgical concerns
  • worsening chronic condition
  • abnormal test result questions
  • injury (non-emergency)
Warning

ACTION: Priority queue for clinician review within 2-4 hours.

ROUTINE_CLINICAL (Standard clinician review, 24-48 hours)

General health questions, chronic disease management, wellness inquiries, non-urgent symptom questions.

Examples: 'What does my lab result mean?', diet/exercise questions, medication questions (non-side-effect), preventive care questions, minor symptom inquiries.

Note

ACTION: AI drafts response for clinician review and approval.

ADMINISTRATIVE (Auto-response eligible, no clinician review needed)

Non-clinical requests that can be handled by office staff or automated systems.

Sub-categories:

  • APPOINTMENT_REQUEST: Scheduling, rescheduling, cancellation
  • PRESCRIPTION_REFILL: Medication refill requests
  • REFERRAL_STATUS: Referral tracking inquiries
  • BILLING_INSURANCE: Billing questions, insurance verification, payment plans
  • RECORDS_REQUEST: Medical records access or transfer requests
  • GENERAL_INFO: Office hours, location, provider availability
Note

ACTION: Auto-respond with appropriate template OR route to admin staff.

Output Format:

Return ONLY valid JSON:

json
{
  "classification": "EMERGENCY|URGENT_CLINICAL|ROUTINE_CLINICAL|ADMINISTRATIVE",
  "sub_category": "specific sub-category if ADMINISTRATIVE, null otherwise",
  "confidence": 0.0-1.0,
  "reasoning": "Brief explanation of why this classification was chosen",
  "key_entities": {
    "symptoms": ["list of symptoms mentioned"],
    "medications": ["list of medications mentioned"],
    "timeframe": "when symptoms started or when action is needed",
    "body_part": "relevant body part if applicable"
  },
  "suggested_response_template": "template_id to use for response",
  "requires_patient_context": true/false
}

Safety Rules (NEVER violate):

1
When in doubt between two categories, ALWAYS choose the more urgent one.
2
Any mention of self-harm, suicide, or harm to others = EMERGENCY, no exceptions.
3
Any post-surgical concern within 14 days of procedure = minimum URGENT_CLINICAL.
4
Chest pain, breathing difficulty, or neurological symptoms = EMERGENCY, no exceptions.
5
If the message is ambiguous or you cannot determine intent, classify as URGENT_CLINICAL.
6
Never provide a diagnosis, prognosis, or specific medical advice in your reasoning.
python
def classify_message(message_text: str, patient_context: dict = None) -> dict:
    # Stage 1: Safety Screen
    safety_result = safety_screen(message_text)
    if safety_result.get('bypass_llm'):
        return safety_result
    
    # Stage 2: LLM Classification
    context_prompt = ''
    if patient_context:
        context_prompt = f"""
        Patient Context (from EHR):
        - Age: {patient_context.get('age', 'Unknown')}
        - Active Conditions: {patient_context.get('conditions', 'None listed')}
        - Recent Procedures (last 30 days): {patient_context.get('recent_procedures', 'None')}
        - Current Medications: {patient_context.get('medications', 'None listed')}
        """
    
    user_prompt = f"""
    {context_prompt}
    
    Patient Message:
    \"{message_text}\"
    
    Classify this message according to your instructions.
    """
    
    # Call LLM API (platform-specific)
    response = llm_client.chat.completions.create(
        model='gpt-5.4',  # or platform's default model
        messages=[
            {'role': 'system', 'content': TRIAGE_SYSTEM_PROMPT},
            {'role': 'user', 'content': user_prompt}
        ],
        temperature=0.1,  # Low temperature for consistent classification
        response_format={'type': 'json_object'}
    )
    
    result = json.loads(response.choices[0].message.content)
    
    # Stage 3: Safety Override
    # If LLM classified as ROUTINE but confidence < 0.7, upgrade to URGENT
    if result['classification'] == 'ROUTINE_CLINICAL' and result['confidence'] < 0.7:
        result['classification'] = 'URGENT_CLINICAL'
        result['reasoning'] += ' [AUTO-UPGRADED: Low confidence on routine classification]'
    
    return result

Configuration in Sully.ai / DeepCura:

If using a turnkey platform, this logic is configured through the platform's triage rule builder UI rather than custom code. Map the above categories and rules to the platform's configuration options:

  • Sully.ai: AI Nurse module > Triage Rules > Custom Categories
  • DeepCura: Message Templates > Triage Configuration
  • ThreoAI: Workflows > Classification Agent > System Prompt (paste TRIAGE_SYSTEM_PROMPT above)

Clinical Response Drafter

Type: agent An AI agent that generates draft responses for clinical questions classified as ROUTINE_CLINICAL or URGENT_CLINICAL. The draft is placed in a clinician review queue—it is NEVER sent directly to the patient. The agent uses patient EHR context (medical history, medications, recent visits) to generate contextually relevant responses while strictly avoiding diagnosis or medical advice. All drafts include a standard disclaimer and are written at a 6th-grade reading level.

Implementation:

Clinical Response Drafter Agent

System Prompt

python
CLINICAL_DRAFT_SYSTEM_PROMPT = """
You are a clinical communication assistant for {practice_name}.
You help draft responses to patient portal messages that will be reviewed and approved
by a licensed healthcare provider before being sent to the patient.

Your Role:

  • Draft clear, compassionate, accurate responses to patient clinical questions
  • Use the patient's EHR context to personalize responses
  • Write at a 6th-grade reading level (short sentences, simple words)
  • NEVER provide a diagnosis, prognosis, or definitive medical advice
  • ALWAYS recommend follow-up when clinically appropriate
  • ALWAYS include the standard disclaimer footer

Response Guidelines:

1
Acknowledge the patient's concern with empathy
2
Provide general educational information relevant to their question
3
Reference their specific context (medications, conditions) when helpful
4
Suggest next steps (schedule appointment, continue current plan, etc.)
5
Include when to seek immediate care
6
Add the disclaimer footer

Absolute Prohibitions:

  • Do NOT say 'you have [condition]' or 'this is likely [diagnosis]'
  • Do NOT recommend starting, stopping, or changing medications
  • Do NOT interpret lab results with definitive conclusions
  • Do NOT provide prognosis ('you will/won't recover')
  • Do NOT contradict established treatment plans without clinician review
  • Do NOT use medical jargon without plain-language explanation

This response was prepared with AI assistance and reviewed by {reviewing_provider_name}. It is intended for informational purposes and does not replace an in-person evaluation. If your symptoms worsen or you have new concerns, please contact our office at {practice_phone} or call 911 for emergencies.

Output Format:

json
{
  "draft_response": "The full response text to be reviewed by the clinician",
  "clinician_notes": "Internal notes for the reviewing clinician about this draft\n                       (e.g., 'Patient asking about metformin side effects - may want\n                       to discuss at next visit', 'Lab results referenced - please\n                       verify interpretation before approving')",
  "confidence": 0.0-1.0,
  "requires_ehr_update": true/false,
  "suggested_followup": "appointment|phone_call|none|referral",
  "reading_level": "estimated Flesch-Kincaid grade level of response"
}

Clinician Review Queue Workflow

1
AI generates draft response → stored in review queue database
2
Clinician receives notification (Teams/email/in-app)
3
Clinician opens draft in review interface: a. APPROVE: Draft sent as-is to patient (clinician name added) b. EDIT: Clinician modifies draft → edited version sent to patient c. REJECT: Draft discarded → clinician writes manual response d. ESCALATE: Reclassify to higher urgency (e.g., routine → urgent)
4
All actions logged with timestamp, clinician ID, and action taken
5
If no action within SLA (4hrs urgent / 24hrs routine), re-notify clinician + alert office manager

SLA Timer Configuration

SLA timer configuration and compliance check function
python
RESPONSE_SLA = {
    'EMERGENCY': {'max_minutes': 5, 'escalation': 'on_call_provider_call'},
    'URGENT_CLINICAL': {'max_minutes': 240, 'escalation': 'teams_office_manager'},
    'ROUTINE_CLINICAL': {'max_minutes': 1440, 'escalation': 'email_office_manager'},
    'ADMINISTRATIVE': {'max_minutes': 60, 'escalation': 'none_auto_respond'}
}

def check_sla_compliance(message_id: str, classification: str, received_at: datetime):
    sla = RESPONSE_SLA[classification]
    elapsed = (datetime.utcnow() - received_at).total_seconds() / 60
    
    if elapsed > sla['max_minutes']:
        trigger_escalation(
            message_id=message_id,
            escalation_type=sla['escalation'],
            elapsed_minutes=elapsed
        )
        log_sla_breach(message_id, classification, elapsed)

Platform-Specific Implementation:

  • Sully.ai: Use AI Nurse module with custom response templates. Configure review queue notifications via Settings > Notifications > Clinical Review.
  • DeepCura: Use the 'Draft Only' mode for clinical messages. Clinicians review in the DeepCura dashboard or via EHR inbox integration.
  • ThreoAI (white-label): Implement the full prompt above as a custom agent workflow. Use the webhook system to push drafts to a Microsoft Teams Adaptive Card for clinician review.

Emergency Escalation Orchestrator

Type: workflow

An automated workflow that activates when a message is classified as EMERGENCY. This is the most safety-critical component of the system. It implements a multi-channel notification cascade to ensure an on-call provider is reached within minutes, sends an immediate safety-focused auto-response to the patient directing them to call 911, and logs every action for audit trail purposes. The workflow uses redundant notification channels (SMS, phone call, Teams, email) with automatic escalation if the primary contact does not acknowledge within defined timeframes.

Implementation:

Emergency Escalation Orchestrator Workflow

Workflow Trigger

Activated when triage classifier returns classification = 'EMERGENCY'

Workflow Definition (Platform-Agnostic)

Emergency escalation workflow definition and configuration (platform-agnostic YAML)
yaml
workflow:
  name: emergency_escalation_v1
  trigger: message.classified.emergency
  timeout_minutes: 30
  
  steps:
    - step: 1
      name: immediate_patient_response
      action: send_patient_message
      template: |
        IMPORTANT: If you are experiencing a medical emergency, 
        please call 911 immediately or go to your nearest emergency room.
        
        Your message has been flagged as urgent and our on-call clinical 
        team has been notified for immediate review. A member of our 
        team will contact you shortly.
        
        Emergency: Call 911
        Poison Control: 1-800-222-1222
        Suicide & Crisis Lifeline: 988
        
        - {practice_name}
      delay_seconds: 0
      log: true
      
    - step: 2
      name: primary_provider_sms
      action: send_sms
      recipient: on_call_provider_phone
      message: |
        🚨 URGENT PATIENT MESSAGE - {practice_name}
        Patient: {patient_name} (DOB: {patient_dob})
        Message excerpt: {message_preview_50_chars}
        Classification: EMERGENCY
        Action required: Review immediately
        Reply 'ACK' to acknowledge receipt.
        Portal link: {direct_message_link}
      delay_seconds: 0
      log: true
      wait_for_ack: true
      ack_timeout_seconds: 300
      
    - step: 3
      name: teams_clinical_channel_alert
      action: teams_webhook
      webhook_url: ${TEAMS_CLINICAL_WEBHOOK_URL}
      payload:
        type: MessageCard
        themeColor: 'FF0000'
        title: '🚨 EMERGENCY Patient Message'
        sections:
          - activityTitle: 'Patient: {patient_name}'
            activitySubtitle: 'Received: {timestamp}'
            facts:
              - name: Classification
                value: EMERGENCY
              - name: Trigger
                value: '{matched_keyword_or_ai_reasoning}'
              - name: On-Call Provider
                value: '{on_call_provider_name}'
              - name: SMS Sent
                value: 'Yes - awaiting ACK'
            text: 'Message: {message_full_text}'
        potentialAction:
          - type: OpenUri
            name: Open in Portal
            targets:
              - os: default
                uri: '{direct_message_link}'
      delay_seconds: 0
      log: true
      
    - step: 4
      name: check_primary_ack
      action: wait_for_condition
      condition: sms_ack_received
      timeout_seconds: 300
      on_timeout: step_5
      on_success: step_7
      
    - step: 5
      name: secondary_escalation_phone_call
      action: automated_phone_call
      recipient: on_call_provider_phone
      message_tts: |
        This is an automated emergency alert from {practice_name}. 
        A patient message has been classified as an emergency and requires 
        your immediate review. Please check your SMS and patient portal. 
        Press 1 to acknowledge.
      delay_seconds: 0
      log: true
      wait_for_dtmf: '1'
      dtmf_timeout_seconds: 60
      on_timeout: step_6
      on_success: step_7
      
    - step: 6
      name: tertiary_escalation_backup_provider
      action: send_sms
      recipient: backup_provider_phone
      message: |
        🚨 ESCALATION: Primary on-call ({on_call_provider_name}) 
        did not respond to emergency patient message.
        Patient: {patient_name} (DOB: {patient_dob})
        Original message time: {timestamp}
        Please review immediately: {direct_message_link}
        
        Also notifying office manager.
      delay_seconds: 0
      log: true
      parallel:
        - action: send_sms
          recipient: office_manager_phone
          message: '🚨 Emergency escalation: On-call provider not responding. Backup notified. Patient: {patient_name}. See Teams channel.'
        - action: send_email
          recipient: practice_admin_email
          subject: 'CRITICAL: Unacknowledged Emergency Patient Message'
          body: 'Full audit trail attached. Immediate action required.'
      
    - step: 7
      name: log_resolution
      action: create_audit_record
      record:
        message_id: '{message_id}'
        classification: EMERGENCY
        patient_response_sent: '{step_1_timestamp}'
        provider_notified: '{step_2_timestamp}'
        provider_acknowledged: '{ack_timestamp}'
        escalation_level_reached: '{highest_escalation_step}'
        total_time_to_ack_seconds: '{time_to_ack}'
      log: true

configuration:
  on_call_schedule:
    source: 'ehr_schedule_api'  # Pull from EHR on-call calendar
    fallback: 'static_rotation'  # Manual rotation if API unavailable
    static_rotation:
      monday: {provider_id: 'DR001', phone: '+1XXXXXXXXXX'}
      tuesday: {provider_id: 'DR002', phone: '+1XXXXXXXXXX'}
      # ... complete weekly rotation
  
  backup_providers:
    - {provider_id: 'DR003', phone: '+1XXXXXXXXXX', role: 'Medical Director'}
  
  office_manager:
    phone: '+1XXXXXXXXXX'
    email: 'manager@practice.com'

Implementation Notes:

  • For Sully.ai: Configure via Settings > Escalation Workflows > Emergency Protocol. Map the above steps to Sully's built-in escalation tiers.
  • For DeepCura: Use the Urgent Alert configuration. SMS notifications are built-in; add Teams webhook via Settings > Integrations.
  • For ThreoAI/custom: Implement the full workflow using the platform's workflow engine or a custom integration using Twilio (SMS/Voice) + Microsoft Graph API (Teams).
  • Twilio integration (if building custom): SMS at ~$0.0079/message, Voice at ~$0.013/min. Sign BAA with Twilio before use with PHI.
Critical

Test this workflow monthly with a simulated emergency message. Log the test and response times. This is analogous to testing fire alarms—it must work when needed.

Administrative Auto-Responder

Type: agent

An AI agent that automatically responds to administrative (non-clinical) patient messages without requiring clinician review. Handles appointment scheduling requests, prescription refill acknowledgments, referral status inquiries, billing/insurance routing, medical records requests, and general office information queries. Integrates with the EHR scheduling API to offer real-time available appointment slots.

Implementation:

Administrative Auto-Responder Agent

System Prompt

python
ADMIN_RESPONDER_SYSTEM_PROMPT = """
You are an administrative assistant for {practice_name}.
You help patients with non-clinical requests by providing helpful, 
accurate information and taking appropriate actions.

You are NOT a medical professional. You NEVER provide medical advice.
If a patient's message contains ANY clinical concern mixed with an 
administrative request, classify it as clinical and do not auto-respond.

Your Capabilities:

1
APPOINTMENT SCHEDULING: Check available slots and offer options
2
PRESCRIPTION REFILL: Acknowledge receipt and route to refill queue
3
REFERRAL STATUS: Check and report referral status
4
BILLING/INSURANCE: Acknowledge and route to billing department
5
RECORDS REQUEST: Provide instructions for records access
6
GENERAL INFO: Answer questions about hours, location, providers, services

Practice Information:

  • Name: {practice_name}
  • Phone: {practice_phone}
  • Address: {practice_address}
  • Hours: {business_hours}
  • Portal URL: {patient_portal_url}
  • Providers: {provider_list_with_specialties}
  • Accepted Insurance: {insurance_list}

Response Rules:

1
Always be warm, professional, and helpful
2
Use the patient's first name
3
Keep responses concise (under 150 words when possible)
4
Include specific next steps
5
Provide contact number for follow-up questions
6
Include AI disclosure: 'This message was prepared with AI assistance.'
7
If ANY clinical question is detected, respond ONLY with: 'I want to make sure your health concern gets the right attention. I've forwarded your message to our clinical team for review. They will respond within [SLA timeframe].' And reclassify the message as ROUTINE_CLINICAL or URGENT_CLINICAL.

Available Actions (call via function/tool):

  • get_available_appointments(provider_id, date_range, appointment_type)
  • create_appointment_request(patient_id, preferred_times, appointment_type, reason)
  • submit_refill_request(patient_id, medication_name)
  • check_referral_status(patient_id, referral_id)
  • route_to_billing(patient_id, message_summary)
  • route_to_records(patient_id, request_type)

Function Definitions (for LLM tool calling)

LLM tool definitions for appointment scheduling, prescription refill, and department routing
python
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_available_appointments",
            "description": "Retrieve available appointment slots from the EHR scheduling system",
            "parameters": {
                "type": "object",
                "properties": {
                    "provider_id": {
                        "type": "string",
                        "description": "Provider ID (or 'any' for any available provider)"
                    },
                    "date_range_start": {
                        "type": "string",
                        "format": "date",
                        "description": "Start date for availability search (YYYY-MM-DD)"
                    },
                    "date_range_end": {
                        "type": "string",
                        "format": "date",
                        "description": "End date for availability search (YYYY-MM-DD)"
                    },
                    "appointment_type": {
                        "type": "string",
                        "enum": ["new_patient", "follow_up", "annual_physical",
                                 "sick_visit", "procedure", "consultation",
                                 "cleaning", "exam", "emergency_dental"],
                        "description": "Type of appointment requested"
                    }
                },
                "required": ["date_range_start", "appointment_type"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "submit_refill_request",
            "description": "Submit a prescription refill request to the provider refill queue",
            "parameters": {
                "type": "object",
                "properties": {
                    "patient_id": {"type": "string"},
                    "medication_name": {"type": "string"},
                    "pharmacy_name": {"type": "string", "description": "Preferred pharmacy"},
                    "urgency": {"type": "string", "enum": ["routine", "running_low", "out"]}
                },
                "required": ["patient_id", "medication_name"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "route_to_department",
            "description": "Route a message to a specific department for human handling",
            "parameters": {
                "type": "object",
                "properties": {
                    "patient_id": {"type": "string"},
                    "department": {"type": "string", "enum": ["billing", "records", "referrals", "front_desk"]},
                    "summary": {"type": "string", "description": "Brief summary for the department"}
                },
                "required": ["patient_id", "department", "summary"]
            }
        }
    }
]

EHR API Integration Examples

athenahealth: Get available appointment slots
python
# athenahealth: Get available appointment slots
def get_available_appointments_athena(provider_id, start_date, end_date, appt_type):
    response = requests.get(
        f'https://api.platform.athenahealth.com/v1/{PRACTICE_ID}/appointments/open',
        headers={'Authorization': f'Bearer {get_athena_token()}'},
        params={
            'providerid': provider_id,
            'departmentid': DEPARTMENT_ID,
            'appointmenttypeid': APPT_TYPE_MAP[appt_type],
            'startdate': start_date.strftime('%m/%d/%Y'),
            'enddate': end_date.strftime('%m/%d/%Y')
        }
    )
    slots = response.json().get('appointments', [])
    return format_slots_for_patient(slots)
Open Dental (via CRMBridge): Get available slots
python
# Open Dental (via CRMBridge): Get available slots
def get_available_appointments_opendental(provider_id, start_date, end_date):
    response = requests.get(
        f'https://api.crmbridge.com/v1/appointments/available',
        headers={'Authorization': f'Bearer {CRMBRIDGE_API_KEY}'},
        params={
            'provider_id': provider_id,
            'start_date': start_date.isoformat(),
            'end_date': end_date.isoformat()
        }
    )
    return response.json()

Platform Configuration:

  • Sully.ai: AI Receptionist module handles administrative auto-responses natively. Configure appointment types and business rules in Settings > Receptionist > Scheduling Rules.
  • Arini (dental): Native scheduling auto-response is core functionality. Configure appointment types, provider schedules, and waitlist in Settings > Schedule.
  • ThreoAI: Deploy the above prompt and tools as a custom agent. Connect EHR APIs via the platform's integration framework.

Triage Analytics & Quality Dashboard

Type: integration

A monitoring integration that tracks AI triage performance metrics, identifies classification errors, monitors SLA compliance, and generates weekly/monthly reports for the MSP and practice management. Connects to the AI platform's analytics API and presents data in a format suitable for both MSP service reviews and HIPAA compliance audits.

Implementation:

Triage Analytics & Quality Dashboard

Metrics to Track

Metrics configuration schema covering operational, quality, compliance, and patient satisfaction tracking
yaml
metrics:
  operational:
    - total_messages_processed_daily
    - messages_by_category:
        - emergency_count
        - urgent_clinical_count
        - routine_clinical_count
        - administrative_count
    - auto_response_count
    - clinician_review_queue_depth
    - average_time_to_patient_response_minutes
    - average_time_to_clinician_review_minutes
    
  quality:
    - clinician_approval_rate_percent  # % of AI drafts approved without edits
    - clinician_edit_rate_percent  # % of AI drafts approved with edits
    - clinician_rejection_rate_percent  # % of AI drafts rejected entirely
    - classification_override_rate  # % of messages reclassified by clinicians
    - false_negative_emergency_count  # CRITICAL: emergencies missed by AI
    - false_positive_emergency_count  # non-emergencies flagged as emergency
    
  compliance:
    - sla_breach_count_by_category
    - escalation_acknowledgment_time_avg
    - audit_log_completeness_percent
    - baa_expiration_alerts
    
  patient_satisfaction:
    - response_time_improvement_vs_baseline
    - patient_portal_message_volume_trend
    - patient_complaint_count_ai_related

Weekly Report Template (auto-generated)

AI Triage Weekly Report - {practice_name}

Week of {start_date} to {end_date}

Volume Summary

  • Total Messages — This Week: {total} | Last Week: {prev_total} | Trend: {trend_arrow}
  • Emergency — This Week: {emerg} | Last Week: {prev_emerg} | Trend: {trend}
  • Urgent Clinical — This Week: {urgent} | Last Week: {prev_urgent} | Trend: {trend}
  • Routine Clinical — This Week: {routine} | Last Week: {prev_routine} | Trend: {trend}
  • Administrative — This Week: {admin} | Last Week: {prev_admin} | Trend: {trend}

Performance

  • Avg Response Time (Admin): {min} min | Target: <5 min | Status: ✅/❌
  • Avg Response Time (Routine): {hrs} hrs | Target: <24 hrs | Status: ✅/❌
  • Avg Response Time (Urgent): {min} min | Target: <240 min | Status: ✅/❌
  • Emergency Escalation Time: {sec} sec | Target: <60 sec | Status: ✅/❌
  • Clinician Approval Rate: {pct}% | Target: >80% | Status: ✅/❌
  • SLA Breaches: {count} | Target: 0 | Status: ✅/❌
  • False Negative Emergencies: {count} | Target: 0 | Status: ✅/❌

Action Items

{auto_generated_action_items_based_on_metrics}

Implementation via Power BI / Microsoft 365

Export weekly metrics to SharePoint List via Microsoft Graph API
python
# Export metrics to SharePoint list for Power BI consumption
import requests

def export_weekly_metrics(metrics: dict):
    # Write to SharePoint List via Microsoft Graph API
    graph_url = 'https://graph.microsoft.com/v1.0/sites/{site-id}/lists/{list-id}/items'
    headers = {
        'Authorization': f'Bearer {get_graph_token()}',
        'Content-Type': 'application/json'
    }
    payload = {
        'fields': {
            'WeekStartDate': metrics['week_start'],
            'TotalMessages': metrics['total_messages'],
            'EmergencyCount': metrics['emergency_count'],
            'UrgentCount': metrics['urgent_count'],
            'RoutineCount': metrics['routine_count'],
            'AdminCount': metrics['admin_count'],
            'AvgResponseTimeAdmin': metrics['avg_response_admin_min'],
            'AvgResponseTimeRoutine': metrics['avg_response_routine_hrs'],
            'ClinicianApprovalRate': metrics['approval_rate'],
            'SLABreaches': metrics['sla_breaches'],
            'FalseNegativeEmergencies': metrics['false_neg_emergency']
        }
    }
    requests.post(graph_url, headers=headers, json=payload)
1
Schedule via Azure Logic App or Power Automate flow
2
Trigger: Recurrence - Every Monday at 7:00 AM
3
Action: Call AI platform analytics API → Transform → Write to SharePoint → Email report

Alert Thresholds (trigger MSP notification)

Alert threshold configuration defining critical and warning conditions that trigger MSP notifications
yaml
alerts:
  critical:
    - condition: false_negative_emergency_count > 0
      action: immediate_sms_to_msp_lead + practice_clinical_champion
      message: 'CRITICAL: AI missed an emergency classification. Immediate review required.'
    - condition: escalation_ack_time > 600_seconds
      action: sms_to_msp_lead
      message: 'Emergency escalation not acknowledged within 10 minutes.'
      
  warning:
    - condition: clinician_rejection_rate > 20_percent
      action: email_msp_lead
      message: 'AI draft quality declining. Review triage rules and prompts.'
    - condition: sla_breach_count > 5_per_week
      action: email_msp_lead + office_manager
      message: 'SLA breaches increasing. Review staffing and queue management.'
    - condition: classification_override_rate > 15_percent
      action: email_msp_lead
      message: 'High override rate suggests triage rules need tuning.'

Testing & Validation

  • EMERGENCY DETECTION TEST: Send 10 simulated emergency messages through the system (chest pain, suicidal ideation, severe bleeding, difficulty breathing, knocked-out tooth for dental, etc.). Verify ALL 10 are classified as EMERGENCY within 5 seconds. Verify auto-response directing patient to call 911 is sent immediately. Verify on-call provider receives SMS notification within 30 seconds. Zero tolerance for false negatives on this test.
  • ESCALATION CASCADE TEST: Send a simulated emergency message and intentionally do NOT acknowledge the primary SMS notification. Verify that the system escalates to phone call after 5 minutes, then to backup provider after 10 minutes. Time each escalation step and verify it matches the configured workflow. Document response times.
  • ADMINISTRATIVE AUTO-RESPONSE TEST: Send 20 simulated administrative messages covering all sub-categories (appointment request, refill, billing inquiry, records request, office hours question, insurance question). Verify all receive appropriate auto-responses within 60 seconds. Verify appointment request responses include actual available slots from the EHR. Verify billing and records requests are properly routed to the correct staff queue.
  • CLINICAL DRAFT QUALITY TEST: Send 15 simulated clinical questions (medication side effect inquiry, post-procedure concern, lab result question, chronic condition management question, new symptom description). Verify all are classified as ROUTINE_CLINICAL or URGENT_CLINICAL (none as ADMINISTRATIVE). Verify AI draft responses are generated and placed in clinician review queue. Have the clinical champion score each draft 1-5 on accuracy, appropriateness, reading level, and completeness. Target: average score >= 4.0.
  • MIXED MESSAGE TEST: Send 10 messages that contain both clinical and administrative content (e.g., 'I need to schedule a follow-up and also my arm has been hurting since the procedure'). Verify the system classifies these as clinical (not administrative) and does NOT auto-respond with scheduling info alone. The clinical concern must be routed for clinician review.
  • EHR INTEGRATION VERIFICATION: Verify bidirectional data flow by (1) sending a message and confirming a task/ticket appears in the EHR inbox, (2) having a clinician approve an AI draft and confirming the response appears in the patient's portal message thread, (3) verifying patient context (name, DOB, recent visits, medications) is correctly pulled into the AI's classification context. Check for data accuracy on at least 10 patients.
  • HIPAA AUDIT TRAIL TEST: Select 5 random messages processed in the past 24 hours. For each, produce a complete audit trail showing: message receipt timestamp, AI classification, confidence score, action taken, response content, clinician reviewer (if applicable), clinician action, final response sent, and all notification/escalation events. Verify all 5 audit trails are complete with no missing fields. Export audit data and verify it can be produced in under 15 minutes (simulating a compliance audit request).
  • AFTER-HOURS BEHAVIOR TEST: Send messages outside business hours covering all categories. Verify: (1) Emergency messages still trigger full escalation to on-call provider, (2) Administrative messages receive 'next business day' acknowledgment, (3) Clinical messages receive 'next business day' acknowledgment with instruction to call 911 if worsening, (4) No auto-scheduling occurs outside business hours unless configured.
  • NETWORK SECURITY VALIDATION: Run a vulnerability scan on the practice network using the MSP's standard tools. Verify: (1) TLS 1.2+ is enforced on all AI platform connections, (2) AI vendor API endpoints are whitelisted and accessible, (3) Clinical VLAN is properly segmented from guest network, (4) DNS filtering is blocking unauthorized categories, (5) Firewall logs show encrypted connections to AI vendor IPs only.
  • LOAD AND RELIABILITY TEST: Send 50 messages in rapid succession (simulating a Monday morning portal message surge). Verify: (1) All messages are processed without dropping any, (2) Classification accuracy is maintained under load, (3) Auto-responses are sent within acceptable timeframes, (4) The clinician review queue correctly orders messages by priority (emergency > urgent > routine), (5) No duplicate responses are sent.
  • PATIENT EXPERIENCE TEST: Have 5 test patients (staff members using test portal accounts) send realistic messages and evaluate the experience from the patient perspective. Assess: (1) Response time perception, (2) Response quality and helpfulness, (3) AI disclosure clarity, (4) Ease of escalating if AI response is insufficient, (5) Overall satisfaction compared to previous manual process. Collect qualitative feedback.
  • ROLLBACK PROCEDURE TEST: Simulate a critical system failure by disabling the AI platform. Verify: (1) Office manager can disable auto-response within 2 minutes using documented procedure, (2) All messages fall back to manual staff processing, (3) No messages are lost during the transition, (4) Staff notification of manual mode is sent via Teams within 5 minutes, (5) System can be re-enabled and resume processing without data loss.

Client Handoff

Client Handoff Checklist

Training Delivered (Document attendance with sign-off sheets)

1
All Staff Overview (2 hrs): System capabilities, HIPAA responsibilities, AI disclosure requirements, message flow walkthrough, error reporting procedure
2
Provider/Clinician Training (1.5 hrs): Draft response review workflow, approve/edit/reject process, classification override procedure, clinical liability framework
3
Admin Staff Training (1 hr): Administrative routing monitoring, scheduling integration usage, dashboard walkthrough, common troubleshooting
4
On-Call Provider Training (30 min): Emergency escalation notification flow, acknowledgment procedure, after-hours behavior

Documentation Package Delivered

1
Quick Reference Card (laminated, placed at each workstation): Triage categories, escalation contacts, override procedures, error reporting number
2
Complete User Guide (PDF, 20 pages): Full workflow documentation with screenshots for all user roles
3
Video Recordings: All training sessions recorded and uploaded to practice SharePoint
4
HIPAA Compliance Binder Section: AI-specific risk assessment addendum, all executed BAAs, triage rule approval documentation with clinical champion signatures, audit log access procedures
5
Emergency Procedures Document: Rollback procedure, vendor support contacts, MSP escalation path, after-hours MSP support number

Success Criteria Review (review with practice leadership)

1
Response Time: Average patient message response time reduced by 50%+ for administrative queries (target: <5 minutes vs. previous 4-24 hours)
2
Auto-Resolution Rate: 40-60% of administrative messages fully auto-resolved without human intervention
3
Clinical Draft Acceptance: >80% of AI-drafted clinical responses approved by clinicians with minimal or no edits
4
Zero Emergency Misses: 100% of emergency messages correctly identified and escalated within 60 seconds
5
SLA Compliance: >95% of messages responded to within category-specific SLA timeframes
6
Staff Satisfaction: Staff report reduced message management burden in post-deployment survey
7
Patient Satisfaction: No increase in patient complaints; positive feedback on response speed

Handoff Meeting Agenda

  • Walk through 1 week of production metrics together
  • Review any open issues or edge cases discovered during go-live
  • Confirm all documentation is accessible and understood
  • Review ongoing MSP support scope and SLA
  • Schedule first monthly optimization review (30 days post go-live)
  • Schedule first quarterly compliance audit (90 days post go-live)
  • Exchange emergency contact information for critical issues
  • Obtain written sign-off from practice owner/manager on project completion

Maintenance

Ongoing MSP Maintenance Responsibilities

Weekly (30-60 minutes)

  • Review AI triage performance dashboard: check classification accuracy, auto-response rates, SLA compliance, and escalation metrics
  • Review any false negative emergency alerts (investigate and remediate immediately if any occurred)
  • Check clinician override rate—if >15%, schedule triage rule tuning session
  • Verify EHR/PMS API connectivity is healthy (check for authentication errors or timeout increases)
  • Review AI platform vendor status page for any incidents or upcoming maintenance

Monthly (2-3 hours)

  • Detailed metrics review with practice office manager (present monthly report)
  • Triage rule optimization session: review misclassified messages from the past month with clinical champion, adjust rules and prompts as needed
  • Update auto-response templates based on staff feedback and seasonal needs (flu season, holiday hours, etc.)
  • Verify audit log integrity and export monthly compliance snapshot
  • Test emergency escalation workflow with simulated message (document test results)
  • Review and update on-call provider rotation in AI platform
  • Check for AI platform software updates; review release notes for breaking changes before applying
  • Invoice client for monthly managed service

Quarterly (4-6 hours)

  • Comprehensive HIPAA compliance audit: verify all BAAs are current, review access controls, audit log completeness, risk assessment updates
  • AI performance deep-dive: analyze classification trends, identify new message categories or edge cases emerging, assess whether additional templates are needed
  • Staff refresher training (30 minutes): review any workflow changes, address questions, reinforce error reporting
  • Network security review: verify firewall rules, TLS enforcement, DNS filtering, and VLAN segmentation remain properly configured
  • Review AI vendor compliance certifications (SOC 2 reports, HIPAA attestations) for any changes
  • Strategic review with practice leadership: ROI assessment, expansion opportunities (add voice channel, additional specialties, etc.)

Annually

  • Full HIPAA risk assessment update including AI system
  • BAA renewal review with all vendors
  • Comprehensive staff retraining (all roles)
  • Platform version upgrade planning (major releases)
  • Contract renewal with client

SLA Considerations

  • Critical (Emergency escalation failure, PHI breach, complete system down): 15-minute MSP response, 1-hour resolution target
  • High (EHR integration failure, auto-response malfunction, triage misclassification pattern): 1-hour MSP response, 4-hour resolution target
  • Medium (Dashboard issues, template updates needed, minor configuration changes): 4-hour MSP response, next business day resolution
  • Low (Feature requests, optimization suggestions, reporting questions): Next business day response, scheduled maintenance window

Escalation Path

1
Practice staff reports issue via MSP ticketing system or phone
2
MSP L1 technician assesses severity and attempts resolution
3
If AI platform issue: escalate to vendor support (Sully.ai, DeepCura, Arini support channels)
4
If EHR integration issue: engage EHR vendor support in parallel
5
If HIPAA/compliance concern: escalate to MSP compliance lead immediately
6
If patient safety concern (missed emergency): immediate escalation to MSP leadership + practice clinical champion + vendor. Document incident for potential breach assessment.

Model/Prompt Retraining Triggers

  • Clinician rejection rate exceeds 20% for 2 consecutive weeks
  • New medical service or specialty added at the practice
  • Significant change in message volume or category distribution
  • AI platform vendor releases new model version
  • Regulatory changes affecting AI disclosure or triage requirements
  • Post-incident review identifies systematic classification errors

Alternatives

Turnkey Platform: DeepCura (Budget Medical)

Use DeepCura at $129/provider/month as the primary AI platform instead of Sully.ai. DeepCura offers all-in-one functionality including AI scribe, receptionist, billing, and triage in a single flat-rate subscription. It uses a link-based integration approach that bypasses some API complexity, and includes 12 pre-built call templates. Best for solo practitioners or small practices (1-5 providers) who want maximum features at the lowest per-provider cost.

Tradeoffs

  • Cost: Lower per-provider cost ($129 vs. $79 for Sully.ai base, but Sully.ai scales better for larger practices).
  • Complexity: Slightly simpler setup with link-based integration.
  • Capability: Fewer modular options than Sully.ai—all features bundled rather than selectable. Less proven at scale for larger practices.
  • Recommend when: budget is the primary constraint, practice has 1-5 providers, and the practice values simplicity over customization.

White-Label MSP Platform: ThreoAI by Synthreo

Instead of reselling a third-party healthcare AI platform, build a branded AI triage service using Synthreo's ThreoAI white-label platform. This gives the MSP full control over branding, pricing, and client experience. ThreoAI supports multiple LLM backends (GPT via Azure ZDR, Claude, Gemini, Mistral, LLaMA), offers per-tenant configuration with zero data retention by default, and is listed on the ConnectWise Marketplace. Requires more MSP effort to configure the clinical triage prompts and workflows, but creates a proprietary service offering.

  • Cost: Higher MSP investment in setup and ongoing management; typical client pricing $500–$2,500/month.
  • Complexity: Significantly more complex—MSP must build triage prompts, manage LLM configurations, and handle clinical workflow design.
  • Capability: Maximum flexibility and customization; can serve multiple healthcare verticals with different configurations.
  • Risk: MSP assumes more responsibility for clinical safety of the AI triage logic.
  • Recommend when: MSP wants to build a scalable, branded healthcare AI practice serving multiple clients, has AI/ML expertise on staff, and is willing to invest in a long-term service line.

Microsoft Azure Healthcare Agent Service (Health Bot)

Use Microsoft's Azure Healthcare Agent Service (formerly Health Bot) as the AI triage engine. This is a cloud platform providing healthcare-specific agent instances with built-in clinical safeguards, a generative AI-powered healthcare orchestrator, and native Azure AD integration. Free tier allows 3,000 messages and 100 MCUs per month for proof-of-concept. Integrates natively with Microsoft 365 ecosystem and Azure OpenAI Service. Best for practices already heavily invested in the Microsoft ecosystem.

Tradeoffs

  • Cost: Free tier for POC; paid tiers scale with usage (generally $0.50–$1.00 per session).
  • Complexity: Highest complexity—requires Azure subscription management, custom development of triage scenarios, and more hands-on integration work.
  • Capability: Most customizable option with access to the full Azure AI services suite; excellent for building highly specialized triage logic. Requires developer resources (not just IT configuration).
  • Recommend when: practice or MSP has Azure development capability, wants maximum control over the AI pipeline, or has unique requirements not met by turnkey platforms.

Communication-Layer Approach: Klara + Manual Triage Enhancement

Instead of deploying a full AI triage agent, implement Klara's patient communication platform with its intelligent message routing (non-AI rule-based triage) to improve message management without AI-generated responses. Klara routes messages to the right staff member/team based on keywords and message type, supports secure two-way messaging, and offers telemedicine integration. AI is used only for routing classification, not for generating patient-facing responses.

Tradeoffs

  • Cost: Lower total cost ($300–$800/month) and lower compliance risk since AI is not generating PHI-containing responses.
  • Complexity: Much simpler deployment (2–3 weeks); lower HIPAA compliance burden since no AI-generated clinical content.
  • Capability: Significantly less automation—staff still write all responses manually; AI only helps sort and route messages.
  • Time savings are 30–40% vs. 50–70% with full AI triage.
Note

Recommend when: the practice is risk-averse about AI-generated clinical communications, the regulatory environment is uncertain, or the practice wants to start with routing automation and add AI response generation later as a Phase 2.

Dental-Specific: TrueLark Multi-Channel

For dental practices that want broader channel coverage beyond portal messages, deploy TrueLark (~$199/month) instead of Arini (~$249/month). TrueLark covers phone calls, SMS, and webchat in addition to portal messages, providing a more comprehensive patient communication AI. Slightly less deep PMS integration than Arini but broader channel coverage.

Note

Tradeoffs: Cost: Lower monthly cost ($199 vs. $249). Complexity: Similar deployment complexity. Capability: Broader channel coverage (phone + SMS + webchat) but less specialized PMS integration than Arini's native Dentrix/Eaglesoft/Open Dental connectors. Arini has stronger scheduling intelligence and waitlist management. Recommend when: the dental practice receives significant patient communication via phone and SMS in addition to portal messages, and wants a single platform covering all channels.

Want early access to the full toolkit?