LLMs in E-Commerce Operations — Beyond the Chatbot

The pattern of the last three years: every e-commerce company built a customer-facing shopping chatbot, most quietly shelved it, and meanwhile the unglamorous operational applications — cleaning catalogs, mining reviews, triaging tickets — went from pilot to load-bearing. This post covers the applications with proven unit economics, in enough detail to actually build them.

1. The Value Framework — Where LLMs Win and Lose

One test predicts almost every LLM success and failure in e-commerce operations:

LLMs win where the task is: high-volume + language-shaped + tolerant of review, and lose where it's: precise + numerical + unsupervised.

2. Catalog Enrichment — The Workhorse Use Case

The business problem

Marketplace catalogs are a mess by nature: sellers upload "Nike Air Max 90 White SIZE 42 ORIGINAL!!!" with no attributes, three photos, and a category guess. Bad catalog data silently damages everything downstream — search can't find it, filters can't include it, recommendations can't relate it, and ads can't target it. Catalog quality is invisible on every dashboard, and it caps the performance of every system on that dashboard. Typical impact when attribute coverage improves: filter-usage conversion lifts, search zero-result rates drop, and returns fall (customers knew what they were buying).

The implementation pattern

Attribute extraction with a strict schema, batched over the catalog:

# The pattern: unstructured seller text → validated structured attributes
prompt = """Extract product attributes from this listing.

Listing title: {title}
Seller description: {description}
Category hint: {category}

Return JSON matching exactly this schema:
{
  "brand": string or null,          // null if not stated — NEVER guess
  "color": string or null,          // normalize: "navy", not "dark blue-ish"
  "size": string or null,
  "material": string or null,
  "gender": "men"|"women"|"unisex"|"kids"|null,
  "confidence": "high"|"medium"|"low"
}

Rules:
- Extract ONLY what is stated or unambiguously visible in the text.
- If the text contradicts itself, set the field null and confidence "low".
"""

# Then: validate against the schema (pydantic), route low-confidence
# rows to human review, load the rest. Review queue ≈ 5-15% of volume.python

The three rules that make this production-grade rather than demo-grade:

• "null if not stated — never guess" in the prompt, enforced by spot-audits. The failure mode of LLM extraction is confident invention; explicitly buying nulls is what keeps precision high.
• Schema validation on every row — a response that doesn't parse is a retry, not a warning log.
• Confidence-routed human review — the LLM does 100% of the work on 85% of rows and turns the rest into a fast review queue. That's the economics: reviewers go from writers to checkers, a 5–10x throughput change.

The same pattern covers: description generation (feed attributes in, get consistent copy out — the reverse direction), title normalization, translation/localization at catalog scale (a big deal in bilingual markets), and compliance flagging (restricted-product detection from listing text).

3. Review Mining — Structured Truth from Unstructured Customers

The business problem

Reviews are the largest honest dataset you own — customers documenting product failures, sizing problems, delivery damage, and counterfeit suspicions, in their own words, for free — and in most companies nobody reads them at scale. Traditional sentiment analysis ("3.8 stars, 72% positive") extracts almost none of the value.

The implementation pattern

Aspect-level extraction, aggregated into dashboards that route to owners:

prompt = """Analyze this product review. Extract every distinct claim.

Review: "{review_text}"
Product category: {category}

Return JSON:
{
  "aspects": [
    {
      "aspect": "sizing"|"quality"|"delivery"|"packaging"|
                "authenticity"|"value"|"functionality"|"other",
      "sentiment": "positive"|"negative"|"neutral",
      "claim": string,              // one sentence, the specific claim
      "severity": "info"|"minor"|"major"|"safety"
    }
  ]
}"""

# Aggregate weekly per SKU / brand / seller:
#   % negative on "sizing"  → size-guide fix, PDP note ("runs small")
#   spike in "authenticity" → seller investigation queue
#   any "safety"            → immediate escalation, human reviewpython

What makes this valuable is the routing, not the extraction:

• Returns prevention: "runs small — order a size up" surfaced on the PDP measurably cuts size-related returns, usually the #1 fashion return reason.
• Quality assurance at marketplace scale: authenticity-claim spikes per seller are one of the strongest counterfeit signals available, and they're free.
• Buying decisions: aspect summaries per SKU tell buyers why something sells or returns — input the inventory playbook's prevention loop can use.

4. Semantic Search & Query Understanding

Two distinct upgrades hide under "AI search," with very different costs:

Embedding retrieval (not an LLM at runtime)

Embed products and queries into the same vector space; nearest-neighbor search makes "warm jacket for Iceland" find parkas with zero shared keywords. This is the two-tower/ANN architecture from the ML playbook — embeddings are computed offline, retrieval is milliseconds, cost per query is effectively zero. This is where zero-result rates go to die, and it should be your first search investment.

LLM query understanding (an LLM, used surgically)

For the hard tail of queries — multi-constraint, conversational, or ambiguous — a small/fast LLM parses the query into structured filters before retrieval:

# "gift for my 8 year old nephew who loves space, under 150 AED"
#                     ↓ small LLM, ~50ms, cached
{
  "intent": "gift",
  "recipient_age": 8,
  "themes": ["space", "astronomy", "STEM"],
  "price_max": 150,
  "categories": ["toys", "books", "games"]
}
# → structured filters + embedding retrieval + normal ranking.
# The LLM never touches the 95% of queries like "iphone 15 case".pattern

The architecture principle: LLM at the edges, classical retrieval and ranking in the middle. Route only the queries that need it (long, natural-language, zero-result retries); cache aggressively — query distributions are extremely head-heavy, so a small cache covers a large share of LLM calls.

5. Support Triage — Augment, Don't Replace

The graveyard is full of "AI will handle 80% of tickets" projects. What actually works is the augmentation stack, deployed in this order:

1. Classification & routing (deploy first, zero risk): intent, order-id extraction, urgency, language → right queue, right priority. Pure internal plumbing; errors cost a misroute, not a customer.
2. Agent-assist (the big win): the LLM drafts a reply from the ticket + order data + policy snippets; the human edits and sends. Handle-time drops 20–40% in most published deployments, quality goes up (consistency), and the human remains accountable for every promise made.
3. Full automation (only for the proven-safe tail): WISMO ("where is my order") with verified tracking data, password resets — flows where the answer comes from a system of record and the blast radius of an error is small. Everything else keeps a human in the loop.

6. Product Matching & Competitive Intelligence

Price intelligence lives or dies on product matching — knowing that your "Samsung Galaxy Buds3 Pro Graphite" is competitor X's "Buds 3 Pro (2024) - Grey". Classical fuzzy string matching drowns in this; embeddings + LLM adjudication solves it:

# Stage 1: embed all titles, retrieve top-k candidates per SKU (cheap)
# Stage 2: LLM adjudicates only the candidate pairs (targeted)

prompt = """Are these the same product? Consider model numbers, storage,
color variants, regional versions, and bundle contents.

A: "{title_a}"  (attrs: {attrs_a})
B: "{title_b}"  (attrs: {attrs_b})

Return JSON: {"match": "same"|"variant"|"different",
              "reason": string}"""

# "variant" matters: same model, different color → price-comparable
# with a flag, not silently merged.python

The same two-stage shape (cheap retrieval → LLM adjudication of candidates) powers duplicate-listing detection inside your own marketplace and assortment-gap analysis against competitors ("which of their top-500 sellers do we not carry?") — both previously manual, quarterly, and stale; now automatable weekly.

7. The Cost Math Nobody Shows

The listicles skip this, and it decides everything. The governing variables: tokens per task × tasks × model tier. Rough 2026-era shape, using a cheap-tier model (~$0.10–0.50 per million input tokens):

Three cost rules that keep it that way:

• Use the cheapest model that passes your eval — extraction and classification rarely need frontier models. Build the eval first (next section), then shop downward until it fails.
• Batch APIs for anything offline — catalog and review workloads aren't latency-sensitive; batch pricing is typically ~50% off.
• Cache by input hash — catalogs re-process the same unchanged listings; don't pay twice.

The honest summary: for operational workloads, API costs are a rounding error next to the labor they replace. The real costs are engineering time and the review queue — budget for those instead.

8. Evaluation — The Discipline That Separates Demos from Systems

Every failed LLM project skipped this. The minimum viable evaluation loop:

1. Golden set first: before writing the prompt, hand-label 200–500 examples (extracted attributes, ticket categories, review aspects). This is your exam. Include the ugly cases — mixed languages, contradictory text, sarcasm in reviews.
2. Measure precision and recall per field, not "it looks good." For extraction: exact-match per attribute. For classification: confusion matrix. Set thresholds with the business owner (95% precision on brand? 99% for safety flags?).
3. Version prompts like code — a prompt change reruns the eval before it ships. Silent prompt edits are silent regressions.
4. Continuous audit in production: sample 1–2% of outputs weekly to human review; track the drift. New product categories, new slang in reviews, new ticket types all degrade quietly.
5. Route by confidence: every pattern in this post keeps a human lane for low-confidence outputs. The system's job is to shrink that lane over time, measurably.

9. Where to Start Monday Morning

Ranked by value-to-effort for a typical e-commerce team:

1. Review mining (week 1–2): lowest risk, purely internal, and the insights sell the whole program upward. One prompt, one batch job, one dashboard.
2. Catalog attribute backfill (month 1–2): pick your top category, backfill missing attributes, measure the search/filter conversion delta. This one produces a number leadership understands.
3. Support classification → agent-assist (month 2–3): classification is a safe rehearsal; assist is the payoff.
4. Query understanding for zero-result queries (month 3+): start with search's failure tail where the baseline is "we showed nothing" — impossible to regress.
5. Product matching (when pricing/assortment asks): needs the embedding infra from step 4's retrieval work anyway.

Note what's not on the list: a customer-facing shopping chatbot. Build one after the operational stack works — by then you'll have the catalog quality, the retrieval infra, and the evaluation discipline it needs to not embarrass you.

Sources & Further Reading:
Netguru: LLM Use Cases in E-Commerce • LLM Cascades for Catalog Quality (arXiv) • LLM Semantic Search for Conversational Queries (arXiv) • Bloomreach: LLMs in E-Commerce