Topic Modelling: A Modern Approach

Introduction #

I’ve already shown how the dsllmpy package can be used to make LLM-powered classification and tagging of entire columns of data easy. Another, juicier, use case I keep coming back to is topic modelling: analysing a collection of documents to identify the common themes, and label individual documents using them.

Classical topic modelling approaches rely on identifying sets of words or terms that occur frequently in documents about the same topic, essentially by clustering the document-word matrix.

The excellent BERTopic package provides a more modern approach to this, mostly based around clustering the document-embedding matrix.

A New Playbook #

However, whenever I’ve had to solve topic modelling problems in practice, I’ve found the best results with a simple, LLM-powered approach:

1. Sample a reasonable number of documents from the data
2. Paste them into a chat with a capable, long-context LLM, and ask it to identify themes, and generate a set of theme labels and definitions. You can use the chat to iterate on and steer this - often, you’ll have some priors about what themes, or at least what kind of themes, you would like to see in the output.
3. Use batched/async LLM calls to classify individual documents using these themes.
4. At this point, you should do a manual review to see if you’re happy with how the documents are being classified. Exporting a sample of 100 or so documents to a spreadsheet and adding a simple pass/fail annotation is often a good enough starting point here.
5. If a lot of your documents are being classified as “Other”, or if there are some themes amount the documents that you weren’t happy about during manual review, go back to your chat (or make manual changes) to expand and iterate your themes.
6. Optionally, if you don’t want to use an LLM to classify all of your data (e.g. to manage cost and latency in production), applying a well-built LLM classifier to a subset of your data is an excellent way of bootstrapping training data that you can then use to train more lightweight classifiers (e.g. using embeddings, bag of words, or even some LLM-generated regular expressions).

In the rest of this post, I work through an example of this approach in action.

Worked Example #

Generating Themes #

# Setup
import os
from typing import Literal

from dotenv import load_dotenv
from pydantic import BaseModel

from dsllmpy import OpenAIClient

load_dotenv()
OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]
llm_client = OpenAIClient(
    api_key=OPENAI_API_KEY,
    model="gpt-5.4-mini",
    max_requests_per_minute=500,
    max_tokens_per_minute=200000,
)

Let’s work with a dataset of stackoverflow questions.

import itertools

import pandas as pd
from datasets import load_dataset
import tiktoken


def get_n_rows_from_hf(dataset: str, n_rows: int, **kwargs):
    ds = load_dataset(dataset, streaming=True, split="train", **kwargs)
    return pd.DataFrame(itertools.islice(ds, n_rows))


data = get_n_rows_from_hf("LDJnr/Puffin", n_rows=1000)
data["query"] = data["conversations"].map(lambda x: x[0]["value"])


# Some long queries. To save time and tokens, let's truncate
tokeniser = tiktoken.encoding_for_model("gpt-5")


def first_n_tokens(text: str, n: int) -> str:
    return tokeniser.decode(tokeniser.encode(text)[:n])


data["truncated_query"] = data["query"].map(lambda x: first_n_tokens(x, n=200))
# data["truncated_query"].map(len).hist()
data.head()

Now, let’s sample 200 documents and prepare them for a nice chat with an LLM. I used the following prompt:

import subprocess

sampled_texts: list[str] = data["truncated_query"].sample(n=200)
concatenated_sampled_texts = "\n---\n".join(sampled_texts)

prompt = (
    """
Below you'll find a collection of queries sent to AI chatbots.
Identify the common themes, and provide a brief description of each.
Format your response as a python dict: {theme: description}.
Keep titles and descriptions concise.

"""
    + concatenated_sampled_texts
)

print(llm_client.count_tokens(prompt), "tokens in total")
# Copy to clipboard to paste into chat (MacOS)
subprocess.run(args="pbcopy", text=True, input=prompt)

14230 tokens in total

This yielded the theme definitions below (I added “Other”).

themes_dict = {
    "STEM Problem Solving": "Requests to solve specific math, physics, chemistry, or biology problems — equations, proofs, calculations, and quantitative analysis.",
    "Scientific Explanation": "Questions asking how natural phenomena work — biochemistry, neuroscience, genetics, ecology, physics — at a conceptual or mechanistic level.",
    "Software Development & Debugging": "Code writing, debugging, architecture design, and technical Q&A spanning multiple languages and frameworks.",
    "Creative Writing & Roleplay": "Requests to write stories, scripts, retell narratives, simulate characters, or engage in imaginative scenarios.",
    "Professional & Business Strategy": "Advice on business planning, marketing, team management, product strategy, and organizational decisions.",
    "Data & AI/ML": "Questions about machine learning methods, MLOps, training models, and AI-powered application design.",
    "Personal Advice & Wellbeing": "Queries about job satisfaction, mental health, relationships, therapy, and self-improvement.",
    "Writing Assistance": "Requests to draft, edit, or reformat documents — emails, articles, social posts, presentations, and outlines.",
    "Food & Recipes": "Meal planning, recipe suggestions, grocery lists, and food preparation questions.",
    "Logical Puzzles & Games": "Brain teasers, game theory, Tower of Hanoi, riddles, and combinatorial problems.",
    "Current Events & Society": "Questions about politics, antitrust law, immigration, social movements, and cultural commentary.",
    "Miscellaneous / Unclear": "Vague, incomplete, or test queries with no discernible intent.",
} | {"Other": "None of the above"}

Classifying Documents #

We can them use dsllmpy to classify our documents against these themes (assuming one theme per document for now)

possible_themes: list[str] = list(themes_dict.keys())
themes_as_text: str = "\n".join(f"- {k}: {v}" for k, v in themes_dict.items())


class Label(BaseModel):
    theme: Literal[*possible_themes]  # type: ignore


# Alternative, for multi-label tagging
class Labels(BaseModel):
    themes: list[Literal[*possible_themes]]  # type: ignore


system_prompt = f"""Classify the chatbot query below into one of the following themes:

{themes_as_text}"""

themes_labels_v1 = await llm_client.call_many(
    system_prompt, data["truncated_query"], response_format=Label
)
data["theme_v1"] = [lbl.theme for lbl in themes_labels_v1]
data["theme_v1"].value_counts()

theme_v1
Scientific Explanation              208
Software Development & Debugging    169
STEM Problem Solving                143
Writing Assistance                  111
Professional & Business Strategy    105
Creative Writing & Roleplay          80
Miscellaneous / Unclear              54
Logical Puzzles & Games              40
Personal Advice & Wellbeing          24
Current Events & Society             24
Data & AI/ML                         19
Food & Recipes                       18
Other                                 5
Name: count, dtype: int64

Refining the Themes #

Five queries have been tagged as “Other”, and 54 as “Miscellaneous / Unclear “. I realise at this point, that these aren’t good labels - we should have one catch-all for “Other”, and another for “No Theme”. Let’s feed just these queries back into a chatbot to see if we can break them into any more themes, and tweak our final set of themes. Again, we can and should steer this output towards themes that are actually useful for the problem we’re trying to solve.

import json

other_themes = ["Miscellaneous / Unclear", "Other"]
other_texts_df = data[data["theme_v1"].isin(other_themes)]
sampled_other_texts = "\n\n---\n\n".join(other_texts_df["truncated_query"].sample(n=50).tolist())

prompt = f"""
We are extracting themes from a set of chatbot queries. 
We have identified the following themes so far:

{json.dumps(themes_dict, indent=0)}

The queries below were flagged as not fitting into any of these themes.
Output a dict, of format {{theme: description}}, containing any new themes to  be added to this collection.
You may update an existing theme's definition by outputting it here, but ensure you match the theme name exactly.

---
{sampled_other_texts}
"""

subprocess.run("pbcopy", text=True, input=prompt)

That yields the additional themes below.

new_themes = {
    "Literature & Book Discussion": "Questions about specific books, authors, literary themes, and stylistic analysis — including requests to discuss, summarize, or analyze published works across genres.",
    "Arts & Music": "Questions about visual art, artists, music genres, specific musicians, and creative media — including analysis, history, and recommendations.",
    "AI Prompt Engineering": "Requests to craft, refine, or structure prompts for generative AI systems such as image generators, chatbots, or language models.",
    "System Prompt / Persona Configuration": "Messages that define a custom AI role, persona, or behavioral ruleset — including operator-style instructions establishing how the assistant should act.",
    "Information Retrieval & Definitions": "Requests to define terms, explain jargon, decode acronyms, or clarify the meaning of unfamiliar words or concepts.",
    "Travel & Local Recommendations": "Queries about destinations, road trips, points of interest, local attractions, and itinerary planning.",
    "Finance & Investing": "Questions about financial metrics, investment analysis, asset valuation, coin or commodity value, and personal finance concepts — excluding licensed financial advice.",
    "Evaluation & Scoring Tasks": "Requests to rate, rank, grade, or compare outputs — such as scoring AI responses, student answers, or creative works against defined criteria.",
    "LLM / Chatbot Meta-Tasks": "Queries that probe or test the AI itself — including jailbreak attempts, model behavior questions, prompt injection, and questions about how language models work.",
    "Education & Tutoring": "Structured learning requests where the user wants to be taught a subject progressively, including Socratic tutoring, curriculum-style lessons, and guided exercises.",
    "Product & Retail Queries": "Questions about specific consumer products, subscriptions, retail services, or store offerings — including pricing, availability, and comparisons.",
    "Civic & Ethical Reasoning": "Scenario-based questions requiring judgment about rules, social norms, fairness, or the right course of action — such as policy compliance, workplace dilemmas, or school conduct situations.",
}
themes_dict_v2 = themes_dict | new_themes
# Keep 'Other', 'Misc' at the end
themes_dict_v2 = {k: v for k, v in themes_dict_v2.items() if k not in other_themes} | {
    k: v for k, v in themes_dict_v2.items() if k in other_themes
}
del themes_dict_v2["Miscellaneous / Unclear"]
themes_dict_v2["None"] = "No identifiable theme"
themes_dict_v2

{'STEM Problem Solving': 'Requests to solve specific math, physics, chemistry, or biology problems — equations, proofs, calculations, and quantitative analysis.',
 'Scientific Explanation': 'Questions asking how natural phenomena work — biochemistry, neuroscience, genetics, ecology, physics — at a conceptual or mechanistic level.',
 'Software Development & Debugging': 'Code writing, debugging, architecture design, and technical Q&A spanning multiple languages and frameworks.',
 'Creative Writing & Roleplay': 'Requests to write stories, scripts, retell narratives, simulate characters, or engage in imaginative scenarios.',
 'Professional & Business Strategy': 'Advice on business planning, marketing, team management, product strategy, and organizational decisions.',
 'Data & AI/ML': 'Questions about machine learning methods, MLOps, training models, and AI-powered application design.',
 'Personal Advice & Wellbeing': 'Queries about job satisfaction, mental health, relationships, therapy, and self-improvement.',
 'Writing Assistance': 'Requests to draft, edit, or reformat documents — emails, articles, social posts, presentations, and outlines.',
 'Food & Recipes': 'Meal planning, recipe suggestions, grocery lists, and food preparation questions.',
 'Logical Puzzles & Games': 'Brain teasers, game theory, Tower of Hanoi, riddles, and combinatorial problems.',
 'Current Events & Society': 'Questions about politics, antitrust law, immigration, social movements, and cultural commentary.',
 'Literature & Book Discussion': 'Questions about specific books, authors, literary themes, and stylistic analysis — including requests to discuss, summarize, or analyze published works across genres.',
 'Arts & Music': 'Questions about visual art, artists, music genres, specific musicians, and creative media — including analysis, history, and recommendations.',
 'AI Prompt Engineering': 'Requests to craft, refine, or structure prompts for generative AI systems such as image generators, chatbots, or language models.',
 'System Prompt / Persona Configuration': 'Messages that define a custom AI role, persona, or behavioral ruleset — including operator-style instructions establishing how the assistant should act.',
 'Information Retrieval & Definitions': 'Requests to define terms, explain jargon, decode acronyms, or clarify the meaning of unfamiliar words or concepts.',
 'Travel & Local Recommendations': 'Queries about destinations, road trips, points of interest, local attractions, and itinerary planning.',
 'Finance & Investing': 'Questions about financial metrics, investment analysis, asset valuation, coin or commodity value, and personal finance concepts — excluding licensed financial advice.',
 'Evaluation & Scoring Tasks': 'Requests to rate, rank, grade, or compare outputs — such as scoring AI responses, student answers, or creative works against defined criteria.',
 'LLM / Chatbot Meta-Tasks': 'Queries that probe or test the AI itself — including jailbreak attempts, model behavior questions, prompt injection, and questions about how language models work.',
 'Education & Tutoring': 'Structured learning requests where the user wants to be taught a subject progressively, including Socratic tutoring, curriculum-style lessons, and guided exercises.',
 'Product & Retail Queries': 'Questions about specific consumer products, subscriptions, retail services, or store offerings — including pricing, availability, and comparisons.',
 'Civic & Ethical Reasoning': 'Scenario-based questions requiring judgment about rules, social norms, fairness, or the right course of action — such as policy compliance, workplace dilemmas, or school conduct situations.',
 'Other': 'None of the above',
 'None': 'No identifiable theme'}

Reclassifying #

We could now reclassify the whole dataset using the new themes, but, to save time and tokens, I’m just going to do it for the themes marked ‘Other’ first time around.

themes_as_text_v2: str = "\n".join(f"- {k}: {v}" for k, v in themes_dict.items())

class Label_v2(BaseModel):
    theme: [*list(themes_dict_v2.keys())]


system_prompt = f"""Classify the chatbot query below into one of the following themes:

{themes_as_text}"""

themes_labels_v2 = await llm_client.call_many(
    system_prompt, other_texts_df["truncated_query"], response_format=Label
)
other_texts_df["theme_v2"] = [lbl.theme for lbl in themes_labels_v2]
other_texts_df["theme_v2"].value_counts()

theme_v2
Other                                        77
Language and Framework Specific Issues       19
Web Development and ASP.NET                   3
Software Architecture and Design Patterns     3
Graphics and Image Rendering                  3
Database Design and Queries                   2
Performance and Optimization                  1
Mobile and Touch Device Development           1
Name: count, dtype: int64

data = data.merge(other_texts_df[["truncated_query", "theme_v2"]], on="truncated_query", how="left")
data["theme_v2"] = data["theme_v2"].combine_first(data["theme_v1"]) # coalescing

In this case, most of our ‘Others’ are still ‘Others’ (I strongly suspect, but haven’t checked, that those 77 cases are from the documents we didn’t use when generating our second set of themes). We could dig further into this, but there’s not much value in identifying themes that come up so rarely.

Human Review #

Now, at this point I would normally do a human review of a sample of these outputs to identify errors and cases where the theme definitions are not aligned with what I need. That’s not very interesting for a blog post, so let’s not, and say we did.

Alternative Classifiers #

LLMs are great, but I’ve we’re sensitive to cost or latency, it may be worth exploring alternative classification methods. Now that we’ve used the LLM to classify 1,000 documents, and used human review to ensure we’re happy with the accuracy of these classifications (we did, didn’t we?), we have ready-made training data for more traditional ML and NLP methods. I finish this post by trying a few of theme here. We’ll have to collapse some of the rare categories to ‘Other’ for this to work.

Let’s try:

• Embeddings + logistic regression
• Bag of words + logistic regression
• Asking an LLM to generate regular expressions

Baseline #

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegressionCV
from sklearn.metrics import f1_score, confusion_matrix, accuracy_score
from sklearn.dummy import DummyClassifier


X_text = data["Text"].reset_index(drop=True)
y = data["theme_v2"].reset_index(drop=True)
label_counts = y.value_counts()
rare_categories = label_counts[label_counts < 10].index.tolist()
y[y.isin(rare_categories)] = "Other"

indices = np.arange(len(y))
train_indices, test_indices = train_test_split(indices, test_size=200, random_state=42)
train_y, test_y = [y[ix].values for ix in [train_indices, test_indices]]

# Use a classifier that just predicts the most common label as a baseline
m_dummy = DummyClassifier().fit(X=train_indices, y=train_y)
pred_from_dummy = m_dummy.predict(X=test_indices)
accuracy_score(test_y, pred_from_dummy)

0.14

Embeddings #

We can use dsllmpy to easily get embeddings from OpenAI.

import dsllmpy

X_embeddings = await llm_client.embed_async(text=X_text, dimensions=256, response_fmt='array')
train_X_embedding, test_X_embedding = [X_embeddings[ix] for ix in [train_indices, test_indices]]
m_embeddings = LogisticRegressionCV().fit(train_X_embedding, train_y)
pred_from_embeddings = m_embeddings.predict(test_X_embedding)
accuracy_score(test_y, pred_from_embeddings)  # Better!

0.64

Bag of Words #

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
X_words = vectorizer.fit_transform(X_text)
train_X_words, test_X_words = [X_words[ix] for ix in [train_indices, test_indices]]
m_words = LogisticRegressionCV().fit(train_X_words, train_y)
pred_from_words = m_words.predict(test_X_words)
accuracy_score(test_y, pred_from_words) # Not amazing

0.48

Regular Expressions #

Finally, something I’ve found works a lot better than I would have expected: you can ask an LLM to write some quick and dirty regexes.

train_texts, test_texts = [data['Text'].iloc[ix] for ix in [train_indices, test_indices]]
# Put 5 examples of each label into an LLM call
prompt = """
Below are a collection of labels we have identified for posts on StackOverlow, with 5 examples of each label.
We want to use regular expressions, in python, to classify future posts using the same labels.
Generate a function that takes the text of a post as an input, and classifies it appropriately.
Do not anchor exclusively on the text of the examples given.
Your solution should generalise to other likely posts that would be given this label. Think semantically.

Don't run or test anything, just give me the code.  
"""
counts = train_y.value_counts()
for label in counts.index:
    if counts[label] > 5:
        examplars = train_texts[train_y == label].sample(5)
        examplar_text= "\n<next example>\n".join([txt.strip() for txt in examplars])
        prompt += f"\n\n<next label>\nLabel: {label}\n\nExamples:\n{examplar_text}"
    else:
        print(f"Skipping label '{label}' due to low frequency")
subprocess.run(args="pbcopy", text=True, input=prompt)
print(prompt[:5000])

Below are a collection of labels we have identified for posts on StackOverlow, with 5 examples of each label.
We want to use regular expressions, in python, to classify future posts using the same labels.
Generate a function that takes the text of a post as an input, and classifies it appropriately.
Do not anchor exclusively on the text of the examples given.
Your solution should generalise to other likely posts that would be given this label. Think semantically.

Don't run or test anything, just give me the code.  


<next label>
Label: Software Architecture and Design Patterns

Examples:
Eliminating coupling out of classes that have strong conceptual bonds with each other

I have the types `Rock`, `Paper`, and `Scissors`.  These are components, or "hands" of the Rock, Paper, Scissors game.  Given two players' hands the game must decide who wins.  How do I solve the problem of storing this chain chart

![Rock, Paper, Scissors chart](https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Rock_paper_scissors.jpg/250px-Rock_paper_scissors.jpg)

without coupling the various hands to each other?  The goal is to allow adding a new hand to the game (Jon Skeet, for instance) without changing any of the others.

I am open to any idea of proxies, but not to large switch statements or duplication of code.  For instance, introducing a new type that manages the chain's comparisons is fine as long as I don't need to change it for every new hand I add.  Then again, if you can rationalize having a proxy that must change for every new hand or when a hand changes, that is welcome as well.

This is sort of a Design 101 problem, but I am curious what solutions people can come up with for this.  Obviously, this problem can easily scale to much larger systems with many more components with any arbitrarily complex relationships among them.  That's why I'm laying a very simple and concrete example to solve.  Any paradigm used, OOP or otherwise, is welcome.
<next example>
Dependency injection in NerdDinner - actually testing your repository or model

Consider a beginner dealing with Dependency Injection. We're analyzing two relevant classes in NerdDinner.

 from the application:
![Repo image](https://imgur.com/Lv0Uz.png)

 from the tests:
![Fakes image](https://imgur.com/TVLV9.png)

They implement different logic, which of course is necessary, as the key idea here is to implement the `IDinnerRepository`, and provide different implementations and private members.

I understand the test is for the controller, but I am concerned that there are two different implementations of the data access logic. Consider any project that uses any kind of ORM, ADO.NET, SubSonic, or whatever flavour of data access you like. Yes, you can setup your fake repository to match the real repo. 

My worry is that over time, implementation details in the real repo change. Perhaps a typo slips in, or some other important implementation detail changes in the . This leads to a potential mismatch of the logic in the Model between the fake and the real repo. The worry is that the implementation of the real repo and test repo get out of sync.



- - -
<next example>
Scalable role based authentication

I am currently designing a role based authentication system for resources where many users have different access rights to it.

A role may be a single user, or a group of roles (so a role is a tree of roles). (see graphic below)

![The other image here](https://yuml.me/diagram/scale:65/class/%5BAdministration%5D-%3E%5BSite1Admins%5D,%20%5BAdministration%5D-%3E%5BSite2Admins%5D,%20%5BSite1Admins%5D-%3E%5BSally%5D,%20%5BSite1Admins%5D-%3E%5BJoe%5D,%20%5BSite2Admins%5D-%3E%5BMike%5D,%20%5BSite2Admins%5D-%3E%5BMax%5D,%20%5BSite2Admins%5D-%3E%5BHarry%5D)

A resource can have multiple authentication properties (like read, write, delete), where each of this is a list of roles allowed to do access the operation. (see graphic below)

![Image goes here](https://yuml.me/diagram/scale:65/class/%5BResource%5D+-%3E%5Bread%5D,%20%5Bread%5D-%3E%5BAdministrators%5D,%20%5Bread%5D-%3E%5BManagement%5D,%20%5Bread%5D-%3E%5BMarketing%5D,%20%5BResource%5D+-%3E%5Bwrite%5D,%20%5Bwrite%5D-%3E%5BAdministrators%5D,%20%5Bwrite%5D-%3E%5BManagement%5D,%20%5BResource%5D+-%3E%5Bdelete%5D,%20%5Bdelete%5D-%3E%5BAdministrators%5D)

The problem is if I want to check if a user has the right to access a property, i have to traverse n trees in worst case (where n is the number of roles assigned to an property).

So for example to check if 'Max' may read the property I might have to check the Marketing, Management and Administration trees if they contain 'Max'.


---



Do you know of any algorithm or alternative approach which removes the quite expensive tree searches while maintaining the role system or something equally powerful.

The perfect case would be some lookup like O(log(n)) for n roles.

Thanks,
Fionn
<next example>
PRISM and WCF - Do they play nice?

Ok,

this is a more general "ugly critter

That produces the following (after some back and forth to get Opus to simplify it’s solution)…

import re

# Ordered most-specific first: the first pattern that matches wins.
RULES = [
    ("Mobile and Touch Device Development",
     r"iphone|ipad|\bios\b|android|uikit|uitableview|uiview|objective-c|"
     r"mpmovieplayer|touch[- ]?screen|on-?screen keyboard|mobile (app|device)"),

    ("Browser Compatibility",
     r"\bie ?[6-9]\b|internet explorer|quirks mode|cross[- ]?browser|"
     r"(works|renders).{0,40}\bbut\b.{0,40}(firefox|chrome|safari|opera|\bie\b)|"
     r"(firefox|chrome|safari|opera).{0,60}(but|whereas|however).{0,60}\bie\b"),

    ("WPF/Windows Forms UI Layout",
     r"\bwpf\b|\bxaml\b|winforms|windows forms|controltemplate|datatemplate|"
     r"gridviewcolumn|stackpanel|textblock|group ?box|anchor|dock|"
     r"when the (window|form) is maximi[sz]ed"),

    ("Database Design and Queries",
     r"\bsql\b|\bt-?sql\b|stored proc|\bschema\b|foreign key|primary key|"
     r"\bjoin\b|order by|group by|nhibernate|\borm\b|entity framework|"
     r"lookup table|audit (table|trail)|\bquer(y|ies|ying)\b"),

    ("Software Architecture and Design Patterns",
     r"design pattern|repository pattern|dependency injection|"
     r"\b(factory|singleton|observer|strategy|decorator|facade) pattern\b|"
     r"loosely? coupl|separation of concerns|object model|value object|"
     r"best practice.{0,40}(class|design|architect)|\bre-?use\b.{0,30}class"),

    ("Graphics and Image Rendering",
     r"\bgdi\+?|quartz|core graphics|opengl|\bpil\b|bitmap|drawstring|"
     r"anti-?alias|alpha (channel|transparen)|render(ing)? (text|the image)|"
     r"draw (a |the )?(text|string|circle|ellipse|gradient|shadow)|pixel font"),

    ("CSS and HTML Styling Issues",
     r"\bcss\b|stylesheet|\bdivs?\b|float: ?(left|right)|display: ?(block|inline|table)|"
     r"position: ?(absolute|relative)|overflow: ?(hidden|auto)|z-index|"
     r"margin|padding|rounded corner|unwanted space|<(div|ul|span|table)\b"),

    ("Third-party Library Integration",
     r"jquery|\$\(|jcarousel|mootools|prototype\.js|\byui\b|ext-?js|dojo|"
     r"resharper|telerik|\bplugin\b|third[- ]?party|which (library|component|tool)"),

    ("Web Development and ASP.NET",
     r"asp\.?net|\.aspx|\bmvc\b|maproute|viewstate|postback|web\.config|\biis\b|"
     r"sharepoint|web ?part|\bmoss\b|\bwss\b|linq to sql|gridview|web service|\bftp\b"),

    ("Language and Framework Specific Issues",
     r"\bcl\.exe\b|compil(er|ing)|\bdll\b|assembly attribute|com interface|"
     r"p/?invoke|interop|activex|extendscript|why (does|is|doesn'?t).{0,60}(work|return)|"
     r"explain (the )?(math|how this)"),
]

PATTERNS = [(label, re.compile(p, re.I)) for label, p in RULES]


def classify_post(text):
    """Return the best-matching label for a StackOverflow post, else 'Other'."""
    if not text:
        return "Other"
    for label, pattern in PATTERNS:
        if pattern.search(text):
            return label
    return "Other"

pred_from_regex = [classify_post(text) for text in test_texts]
accuracy_score(test_y, pred_from_regex) # Actually worse than TF-IDF, but worth a shot

0.435

SKILL.md #

Of course, it is 2026, so you’re not going to want to actually do any of this work yourself, are you? Well, don’t worry, I’ve got you: check out the SKILL.md that accompanies this post, feed it to your Agent of choice, and let it do the work.