Building a Competitor Discovery System: LLMs, Classification, and Guardrails

UX Bench helps users benchmark their site's Core Web Vitals against competitors. For that to work, I needed to automatically identify 5 relevant competitors for any website. A user analyzing walmart.com should see target.com and costco.com, not random e-commerce sites. The system needed to work for thousands of known domains AND handle unknown startups users might analyze.

This article tells two connected stories: Part 1 covers how I built the classification database (5,500 domains with increasingly granular NAICS codes and custom business verticals). Part 2 shows how I designed an intelligent system that handles both known domains (fast lookup) and unknown domains (brand similarity matching), with automatic rollup strategies when granular matches are sparse.

Before diving into the methods, it's important to understand why using NAICS (North American Industry Classification System) codes for domain classification is tricky.

The solution? Layer semantic "business verticals" on top of NAICS codes. More on that later.

Before showing what I actually built, here's a quick look at how classification approaches evolved over the past 30 years. The first three are educational references showing historical context for the waterfall approach I developed.

TF-IDF (Term Frequency-Inverse Document Frequency) weighs words by how important they are to a specific document. "Plumbing" appears often on Home Depot's site but rarely across all sites, making it a strong signal. Combine with cosine similarity to find the most similar NAICS category.

# TF-IDF classification algorithm
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# Pre-compute NAICS code descriptions as TF-IDF vectors
naics_descriptions = {
    "444110": "Home centers, retail hardware stores...",
    "722511": "Full-service restaurants, cafeterias...",
    # ... 1,030 more codes
}

vectorizer = TfidfVectorizer(max_features=5000, ngram_range=(1, 2))
naics_vectors = vectorizer.fit_transform(naics_descriptions.values())

# For each website, compute similarity to all NAICS codes
website_vector = vectorizer.transform([homepage_text])
similarities = cosine_similarity(website_vector, naics_vectors)[0]

# Return top K most similar codes (typically K=40-80)
top_candidates = similarities.argsort()[-40:][::-1]

Modern language models like BART, GPT, and Gemini can classify text without explicit training on your specific categories. Feed them website content and NAICS descriptions, and they'll predict the best match using their pre-trained semantic understanding.

# Zero-shot classification with BART
from transformers import pipeline

classifier = pipeline(
    "zero-shot-classification",
    model="facebook/bart-large-mnli"
)

# Chunk long homepage text (models have token limits)
chunks = split_text_into_chunks(homepage_text, max_tokens=500)

# Classify each chunk against candidate NAICS codes
candidate_labels = [
    "Home improvement retail stores",
    "Full-service restaurants",
    "Software publishing and SaaS"
]

scores = []
for chunk in chunks:
    result = classifier(chunk, candidate_labels)
    scores.append(result['scores'])

# Max-pool across chunks to get final prediction
final_scores = np.max(scores, axis=0)
best_match = candidate_labels[np.argmax(final_scores)]

Few-shot learning provides the LLM with examples of correctly classified domains before asking it to classify a new one. This teaches the model your specific classification standards and edge case handling.

# Few-shot prompting with GPT-4 / Gemini
prompt = """Classify the following website into a NAICS code.

Examples:
- walmart.com → 455110 (Department Stores)
- shopify.com → 541511 (Custom Computer Programming Services)
- mcdonalds.com → 722513 (Limited-Service Restaurants)

Now classify this website:
Domain: {domain}
Content: {homepage_excerpt}

NAICS Code:"""

response = llm.generate(
    prompt.format(domain=domain, homepage_excerpt=text[:1000])
)
predicted_code = extract_naics_code(response)

The breakthrough: use embeddings to retrieve the most semantically similar already-classified domains, then use those as few-shot examples. Instead of random examples, you get examples that are actually relevant to the domain you're classifying.

# Embedding-based example retrieval
from sentence_transformers import SentenceTransformer

# Pre-compute embeddings for all classified domains
model = SentenceTransformer('all-MiniLM-L6-v2')
classified_domains = load_classified_domains()  # 4,790 domains
domain_embeddings = model.encode([d.text for d in classified_domains])

# For new domain, find most similar examples
new_domain_embedding = model.encode(new_domain_text)
similarities = cosine_similarity([new_domain_embedding], domain_embeddings)[0]
top_5_examples = similarities.argsort()[-5:][::-1]

# Use those examples in few-shot prompt
examples_text = "\n".join([
    f"- {classified_domains[i].domain} → {classified_domains[i].naics}"
    for i in top_5_examples
])

prompt = f"""Classify this website using these similar examples:
{examples_text}

Domain: {new_domain}
Content: {new_domain_text}
NAICS Code:"""

After classifying 4,790 companies with 89% accuracy, I discovered a fatal flaw: NAICS works great for distinguishing restaurants from law firms, but completely fails when everyone in tech gets code 541511 ("Custom Computer Programming Services").

The solution? Add a second classification layer: Business Verticals. These are human-readable, semantic groupings that sit between NAICS codes and individual companies.

# Two-layer classification system
classification = {
    "naics_2digit": "54",  # Professional, Scientific, Technical Services
    "naics_4digit": "5415",  # Computer Systems Design
    "naics_6digit": "541511",  # Custom Computer Programming
    "vertical": "E-commerce Platforms",  # NEW: Semantic vertical
    "confidence": "high"
}

# Now competitors are truly similar:
shopify_competitors = find_competitors(
    naics="541511",
    vertical="E-commerce Platforms"
)
# Returns: WooCommerce, BigCommerce, Wix, Squarespace
# NOT: Google, Salesforce, Oracle

The theoretical best approach: fine-tune a model specifically for NAICS classification using thousands of labeled examples. This would likely achieve 90-95%+ accuracy and fast inference.

After testing all six methods, I didn't pick just one. I built an ensemble that combines the best of each approach, weighted by their strengths.

# Final ensemble formula
final_score = (
    0.55 * zero_shot_llm_score +
    0.25 * naicskit_score +
    0.20 * keyword_score
)

best_naics = candidates[argmax(final_score)]

# Then add vertical layer
vertical = llm_classify_vertical(
    domain_text=text,
    naics_code=best_naics
)