Notebook: Dynamic Classifier / treehousePairwise.ipynb

The goal: take a new query in a chat stream and decide which of two categories it's closer to, A or B. Also report how certain that decision is, and keep a history of every call.

For a quick test I needed two categories that are clearly different, so I took the first 500 rows of SQuAD and used contexts (long passages) as A and questions (short ones) as B. That gave me 74 unique contexts and 499 questions. I used 40 of each, because each category needs the same number of rows.

BASE_MODEL = "all-distilroberta-v1"  # replace with any

ctx_rep = base.encode(train_context, normalize_embeddings=True)   # (40, 768)
que_rep = base.encode(train_question, normalize_embeddings=True)  # (40, 768)

Choosing a similarity score

I tried three scores side by side before picking a default.

Method 1: Euclidean. sentence-transformers returns the negative distance, so I checked the diagonal by hand:

np.linalg.norm(ctx_rep[0] - que_rep[0])  # 1.0468777  → matrix entry -1.0469 ✅
np.linalg.norm(ctx_rep[1] - que_rep[1])  # 1.1283697  → matrix entry -1.1284 ✅

Method 2: Manhattan. Same shape, much bigger numbers (around −22 to −32), so it's harder to read at a glance.

Method 3: Cosine. For normalised vectors this is the same as the dot product. That makes it the cheapest one, so it became the default.

⚠️ Self note (kept from the notebook): base.similarity always returns tensors, whatever type you pass in. In an app, handle the device properly, "for the sake of Jesus Christ".

The predict step

def predict(sample_A, sample_B, new_query):
    base.similarity_fn_name = "dot"
    inputs = base.encode([new_query], normalize_embeddings=True)
    A = base.similarity(inputs, sample_A).numpy()
    B = base.similarity(inputs, sample_B).numpy()
    pair_scores = abs(np.array([A.mean(axis=-1), B.mean(axis=-1)])).flatten()
    pair_scores /= pair_scores.sum()
    # omega: how far the query is from its closest match in the winning category
    winner = A if np.argmax(pair_scores) == 0 else B
    diff_omega = np.round(abs(1 - winner).min(), 5)
    return diff_omega, pair_scores

Query	Omega	A (context)	B (question)
a training question	0.0	0.467	0.533
a training context	0.0	0.579	0.421
"what do you mean?" (unseen)	0.885	0.046	0.954

Omega is 0 for anything the classifier has already seen, and it rises for new queries. The unseen short query lands confidently in "question", and its omega of 0.885 says it's a long way from any training example. One line isn't enough to test with, but it's a start.

Treehouse

The final class wraps all of that in a SentenceTransformer subclass. You give it lists A and B up front, it embeds them once, and every predict call appends {certainity, scores} to self.history so I can look back at how it behaved.

I wrote this for the dialogue side of Tom: a cheap, fast first pass that tags what kind of thing the user just said before a bigger model decides how to answer.