This post documents a deployable automation that reads inbound emails, classifies intent, extracts structured fields, and creates/updates CRM records with auditability. It is designed for revenue and ops teams that want faster response times without adding headcount.
Use case
– Inbound mailbox (sales@, info@, partnerships@)
– Auto-detect lead vs. support vs. vendor vs. spam
– Extract entities: company, contact, intent, ARR, timeline, region, product interest
– Create or update CRM objects
– Route to queues and SLAs with observability
Reference stack
– Ingestion: Gmail API watch + Pub/Sub (or webhook) → FastAPI
– Queue: Celery + Redis (or Cloud Tasks)
– Processing: Python + OpenAI Responses API (function calling) or Claude Tools
– Storage: Postgres (normalized tables + JSONB for raw artifacts)
– CRM: HubSpot or Salesforce REST
– Metrics/Tracing: Prometheus + Grafana; Sentry for errors
– Secrets: AWS Secrets Manager or GCP Secret Manager
High-level architecture
1) Gmail Watch pushes new-message IDs to /webhook/email.
2) FastAPI validates signature, enqueues job with message_id.
3) Worker pulls raw content via Gmail API, normalizes MIME, removes signatures and legal footers.
4) LLM extraction + classification with a constrained schema.
5) Deterministic business rules (routing, dedupe, SLOs).
6) CRM create/update with idempotency keys.
7) Write audit row (inputs, model, outputs, actions).
8) Emit metrics and alerts.
Data model (Postgres)
– emails(id, gmail_id, received_at, subject, sender, to, cc, body_text, body_html, thread_id, raw_json)
– extractions(email_id, model, version, schema_name, json, confidence, tokens_in, tokens_out, cost_usd)
– crm_events(id, email_id, action, object_type, object_id, status, response_json, idempotency_key)
– routes(email_id, intent, queue, priority, sla_minutes)
– eval_labels(email_id, labeler, intent, fields_json, notes, created_at)
Extraction schema (LLM tool/function)
– intent: one of [lead, support, vendor, job_applicant, newsletter, spam]
– entities:
– person.name, person.email
– company.name, company.domain
– interest.products[] (strings)
– budget.annual_amount_usd (number or null)
– timeline: one of [now, quarter, six_months, unknown]
– region: ISO country or null
– summary: 1–2 sentences
– required_action: one of [schedule_demo, send_pricing, forward_support, ignore, route_ops]
– confidence: 0–1
Routing rules (deterministic)
– If intent=lead and timeline in [now, quarter] → queue=sales_inbound, priority=high, SLA=15 min.
– If domain matches customer list and intent=support → escalate to support queue.
– If intent=spam → no CRM write; mark ignored.
Idempotency and dedupe
– Use message_id + thread_id to avoid duplicate processing.
– Before CRM write:
– Search by email domain + person email.
– If exists, update contact and associate with company and existing deal.
– Create new deal only if no open deal in last 45 days and confidence ≥ 0.6.
FastAPI endpoints
– POST /webhook/email: validate provider signature; enqueue Celery task with message_id.
– POST /admin/replay: replay by email_id (requires auth).
– GET /healthz, /metrics.
LLM prompt and constraints
– System: “Return only tool calls. Do not invent values. Use null if missing. Keep currency as USD.”
– Tool schema: the extraction schema above. Reject messages shorter than 6 words as low confidence.
– Safety: strip signatures via rules (look for “–”, “Best,” blocks, legal boilerplate).
– Cost control: short context window; pass subject + first 1,500 chars plain text + minimal thread history.
Evaluation harness
– Sample 200 real (or synthetic) emails with gold labels.
– Metrics:
– Intent accuracy (micro): target ≥ 0.92
– Field F1 (person.email, company.name): ≥ 0.95
– High-stakes fields (budget, timeline) exact-match: ≥ 0.80
– CRM write error rate: ≤ 0.5%
– Average lead time to CRM: ≤ 25s P50, ≤ 90s P95
– Weekly regression: run on main before deploy; block if below thresholds.
Observability
– Metrics:
– emails_ingested_total, by inbox
– extraction_confidence_bucket
– crm_write_latency_seconds
– idempotency_conflicts_total
– llm_cost_usd_total
– Tracing: tag spans with email_id, gmail_id, model, crm_object_id.
– Alerts:
– Spike in spam routed to sales
– Confidence 1% for 5 minutes
Error handling and retries
– Transient: Gmail 429/5xx, CRM 429/5xx → exponential backoff with jitter, max 4 tries.
– Permanent: schema validation fail → store, mark for human review.
– Dead letter: push to “email_triage_dlq”; Slack notify ops with replay link.
– Partial failure: if CRM create succeeds but association fails, retry association only.
Security and compliance
– Least-privilege service accounts for Gmail/CRM.
– Encrypt at rest; redact PII in logs (hash emails).
– Store raw email in S3/GCS with short TTL (e.g., 30 days) if policy allows.
– Model provider: use enterprise endpoint with data retention off.
Cost controls
– Heuristics pre-filter:
– If DKIM fail or sender in blocklist → skip LLM.
– If thread already classified in last 24h → reuse prior result.
– Use small model for spam/intent gate; large model only for leads/support.
– Batch CRM reads (search) and cache domain-to-company mappings.
Deployment notes
– Use Gunicorn/Uvicorn workers with timeout ≥ 60s for rare slow providers.
– Celery autoscale based on queue depth.
– Blue/green deploy with read-only mode for admin tools.
– Run nightly backfill job for any emails stuck without crm_events.
Sample ROI (realistic baseline)
– Inbox volume: 2,500/month; previously 2 FTE hours/day routing.
– After automation:
– Manual routing reduced by ~85% (≈ 28 hours/month saved).
– Lead first-touch from 4h median to 12m median.
– Net-new pipeline lift from faster replies: +6–10% (org dependent).
– LLM + infra cost: ~$85–$160/month at this volume.
Implementation checklist
– Configure Gmail watch; verify webhook signature handling.
– Create Postgres schema and migrations.
– Implement MIME normalize + signature stripping.
– Ship LLM tool schema + eval harness with gold set.
– Build CRM client with search + idempotent create/update.
– Add metrics, Sentry, and Slack alerting.
– Run canary on one inbox for 2 weeks; compare to human labels.
– Roll out to remaining inboxes; set SLAs with owners.
What to ship first (MVP)
– Intent-only classifier → route to queues, no CRM writes.
– Manual one-click “Create in CRM” from admin UI.
– Add extraction and auto-writes after 1 week of eval stability.
Extensions
– Calendar integration: auto-schedule demos if confidence high.
– Account enrichment via Clearbit/Apollo before CRM write.
– Language detection → route to regional teams.
– Thread memory to avoid re-asking model each reply.
If you want the project template (FastAPI app, Celery, schema, eval harness, and HubSpot client), reach out and I’ll publish a repo skeleton with env-var based configuration.
This is an excellent, practical blueprint; building in observability and an audit trail is crucial for making a system like this truly reliable. What is your strategy for handling emails that the LLM fails to classify with high confidence?
Thanks — agree that the “unknown/low confidence” bucket is where reliability is won or lost. Do you route those into a human review queue by default, or do you have a lightweight rules-based fallback (e.g., keyword/domain heuristics) that can make a safe decision for a subset? Also curious if you implement an automatic retry/escalation path (different prompt/model, or reprocess after thread context is fetched), and what confidence thresholds and logging you use to make those handoffs auditable (including tracking false positives/negatives over time).
Our process is to first apply a rules-based fallback, then route any remaining low-confidence items to a human review queue, logging the confidence score at each step for audit.
That makes a lot of sense—rules first for the “safe wins,” then humans for the rest, with confidence logging throughout.
Do you have a defined confidence threshold (or different thresholds by intent type) for when the LLM result is accepted vs. sent to review? And on the rules fallback, what are the main heuristics you’ve found reliably “low risk” (e.g., known vendor domains, unsubscribe/spam markers), and how do you review/tune those rules over time based on false positives/negatives?
Yes, we use dynamic confidence thresholds per intent and tune rules like sender reputation based on weekly human review audits.
Interesting—when you say “dynamic” per-intent thresholds, how are you actually setting them (e.g., based on historical precision/recall per intent, cost of a false positive vs. false negative, volume), and do they auto-adjust or stay as a versioned config you update manually? Also on the weekly audits: what’s the concrete feedback loop—are you logging review outcomes into a dataset and then updating specific rules/thresholds (or retraining/refining prompts), and how do you measure that the changes improved things week over week?
Those audit outcomes feed into a dataset we use to manually tune versioned configs for rules and thresholds, tracking the impact on precision and recall for each intent.
That’s a solid approach—especially keeping the thresholds/rules as a versioned config tied back to labeled audit outcomes.
How do you roll out config changes: do you canary them (e.g., apply to a small slice of traffic) or run a shadow evaluation first before switching over? And when you track precision/recall per intent, are you using any holdout set or lightweight A/B framework to avoid “tuning to last week,” plus any drift monitoring (volume/intent mix shifts, sender-domain shifts) to know when thresholds need another pass?
Great questions; we run a shadow evaluation before switching over and use a holdout set from recent audits to track performance and monitor for drift.