NLP Track — Overview & live demo

MDIG2026 Summer School · Language in the Taboo Balance Corpus

Dyads play Taboo — one person (the clue-giver) describes a hidden target word without saying it or its five taboo words, while the partner (the guesser) guesses — sometimes on a balance board, sometimes on the ground. That makes the corpus a natural laboratory for spoken-language NLP, and this track walks the whole arc:

  audio  ─▶  transcripts  ─▶  meaning            ─▶  coordination        ─▶  themes
 (stereo)   (Whisper ASR)    (semantic homing)     (lexical alignment)     (topic modeling)

This notebook is the front door: it runs a compact live demo of each stage on a few trials, then points you to the deep-dive notebooks. Everything is CPU-only and multilingual.

The deep-dive notebooks

#	Notebook	What it adds
0	00_dataset_preview	spaCy + a semantic network of the game words (no audio)
1	01_audio_to_transcripts	Whisper ASR → role-labeled transcripts (the shared input)
2	02_semantic_homing	does the clue-giver close in on the target’s meaning?
3	03_lexical_alignment	do the two partners’ words converge?
4	04_multimodal_bonus	language × gesture × board sway on one clock
5	05_topic_modeling	recurring clue-giving themes

Setup — create the `mdig-nlp` conda environment (once)

Run these in a terminal from this folder (Datasets/BalanceCorpus/scripts/nlp_taboo):

conda env create -f environment.yml       # creates the "mdig-nlp" env (Python 3.11, all deps)
conda activate mdig-nlp
python -m spacy download en_core_web_sm    # only Notebook 0 needs this
python -m ipykernel install --user --name mdig-nlp   # registers the Jupyter kernel

Then, in the notebook, select the mdig-nlp kernel (kernel picker, top-right) before running any cells.

No conda? A plain venv works too: python -m venv .venv && .venv\Scripts\activate (Windows) or source .venv/bin/activate (macOS/Linux), then pip install -r requirements.txt.

First run downloads two ~0.5 GB models (Whisper small + a multilingual sentence encoder), once.

0 · Locate the corpus

import os, certifi
if not os.path.exists(os.environ.get("SSL_CERT_FILE", "")):   # heal broken Anaconda cert path
    os.environ["SSL_CERT_FILE"] = certifi.where(); os.environ["REQUESTS_CA_BUNDLE"] = certifi.where()
def find_corpus(start=None):
    d = os.path.abspath(start or os.getcwd())
    for _ in range(8):
        if os.path.isfile(os.path.join(d, "metadata.csv")) and os.path.isdir(os.path.join(d, "audios")):
            return d
        parent = os.path.dirname(d)
        if parent == d: break
        d = parent
    raise FileNotFoundError("Could not find the BalanceCorpus root (metadata.csv + audios/). "
                            "Open this notebook from inside the cloned repository.")
CORPUS = find_corpus()
TRANSCRIPTS = os.path.join(CORPUS, "scripts", "nlp_taboo", "transcripts")
print("Corpus root:", CORPUS)

1 · The dataset in one cell

metadata.csv drives everything: per trial, the target word, its five taboos, who the clue-giver was, and the board/ground condition.

import pandas as pd
meta = pd.read_csv(os.path.join(CORPUS, "metadata.csv"), encoding="utf-8-sig")
meta["pair_id"] = meta["pair_id"].astype(str)
import glob
wavs = glob.glob(os.path.join(CORPUS, "audios", "*", "*.wav"))
n_audio = len(wavs)
# condition is encoded in the filename stem (..._<target>_<condition>.wav); count only what's on disk
on_disk_cond = pd.Series([os.path.basename(w).rsplit(".", 1)[0].split("_")[7] for w in wavs]).value_counts().to_dict()
print(f"{meta.target_word.nunique()} unique target words · {n_audio} trials with audio on disk · "
      f"conditions on disk: {on_disk_cond}")
meta[["pair_id","target_word","taboo_1","taboo_2","taboo_3","clue_giver_condition"]].head(4)

2 · Stage 1 — Audio → transcripts (live)

Each trial’s .wav is stereo with one speaker per channel (separate lavalier mics), so we get speaker separation for free. We recover which channel is which speaker from the data (the clue-giver talks more), remove the faint crosstalk between mics, and transcribe with Whisper. Here we demo a few trials; Notebook 1 does all 80.

import wave, numpy as np, librosa

def parse_stem(p):
    q = os.path.basename(p).rsplit(".", 1)[0].split("_")
    return dict(pair_id=f"{q[0]}_{q[1]}", trial_number=int(q[2]), target_word=q[6],
                condition=q[7], stem=os.path.basename(p).rsplit(".", 1)[0])

rows = []
for w in sorted(glob.glob(os.path.join(CORPUS, "audios", "*", "*.wav"))):
    r = parse_stem(w); r["audio_path"] = w
    m = meta[(meta.pair_id == r["pair_id"]) & (meta.trial_number == r["trial_number"])]
    if len(m) != 1: continue
    m = m.iloc[0]
    r.update(clue_giver_id=int(m.clue_giver_id), participant_1_id=int(m.participant_1_id),
             participant_2_id=int(m.participant_2_id))
    rows.append(r)
trials = pd.DataFrame(rows)

DEMO = trials.groupby("condition", group_keys=False).head(2).reset_index(drop=True)  # 2 board + 2 ground
print(f"{len(trials)} trials available; live-transcribing {len(DEMO)} for the demo.")

# --- recover channel->speaker mapping (clue-giver = louder channel), per pair ---
def channels(p):
    w = wave.open(p, "rb"); sr = w.getframerate()
    raw = np.frombuffer(w.readframes(w.getnframes()), dtype=np.int16).astype(np.float64)
    return raw[0::2], raw[1::2], sr

def venergy(x, sr, frame=0.05, pct=60):
    n = max(int(sr*frame), 1); fr = x[:len(x)//n*n].reshape(-1, n)
    e = np.sqrt((fr**2).mean(1)); k = e >= np.percentile(e, pct)
    return float(e[k].mean()) if k.any() else float(e.mean())

def calibrate(pdf):
    louder_left, cg_is_p1 = [], []
    for _, r in pdf.iterrows():
        L, R, sr = channels(r.audio_path)
        louder_left.append(venergy(L, sr) > venergy(R, sr))
        cg_is_p1.append(r.clue_giver_id == r.participant_1_id)
    a = np.mean(np.array(louder_left) == np.array(cg_is_p1))
    return {"L":"participant_1","R":"participant_2"} if a >= .5 else {"L":"participant_2","R":"participant_1"}

pair_map = {p: calibrate(g) for p, g in DEMO.groupby("pair_id")}

# --- load Whisper + transcribe with crosstalk removal ---
try:
    from faster_whisper import WhisperModel
except ModuleNotFoundError:
    raise ModuleNotFoundError("faster-whisper is not installed — see the Setup note at the top.")
asr = WhisperModel("small", device="cpu", compute_type="int8")

def load16k(p):
    y, _ = librosa.load(p, sr=16000, mono=False)
    if y.ndim == 1: y = np.vstack([y, y])
    return [np.ascontiguousarray(y[c], dtype=np.float32) for c in (0, 1)]

def rms(a, s, e, sr=16000):
    seg = a[int(s*sr):int(e*sr)]; return float(np.sqrt(np.mean(seg**2))) if len(seg) else 0.0

drows = []
for _, r in DEMO.iterrows():
    ch = load16k(r.audio_path); m = pair_map[r.pair_id]
    for c, side in [(0, "L"), (1, "R")]:
        who = m[side]; pid = r.participant_1_id if who == "participant_1" else r.participant_2_id
        role = "clue_giver" if pid == r.clue_giver_id else "guesser"
        segs, _ = asr.transcribe(ch[c], language="en", vad_filter=True, beam_size=1)
        for s in segs:
            if s.text.strip() and rms(ch[c], s.start, s.end) >= rms(ch[1-c], s.start, s.end):
                drows.append(dict(trial_id=r.stem, pair_id=r.pair_id, trial_number=r.trial_number,
                                  condition=r.condition, target_word=r.target_word, role=role,
                                  start=round(s.start,2), end=round(s.end,2), text=s.text.strip()))
demo_tx = pd.DataFrame(drows)

# show one clue-giver transcript
cg = demo_tx[demo_tx.role == "clue_giver"]
if len(cg):
    t0 = cg.trial_id.iloc[0]; tgt = demo_tx[demo_tx.trial_id == t0].target_word.iloc[0]
    print(f"clue-giver on '{tgt}':")
    for _, s in cg[cg.trial_id == t0].sort_values("start").iterrows():
        print(f"  [{s.start:5.1f}] {s.text}")

3 · Which transcripts do the analyses use?

The demo transcribed a few trials. If you have already run Notebook 1 (all 80 trials), the analyses below automatically use that full set for meaningful numbers; otherwise they fall back to the handful we just transcribed.

full = os.path.join(TRANSCRIPTS, "transcripts_all.csv")
if os.path.exists(full):
    tx = pd.read_csv(full, encoding="utf-8")
    print(f"Using the FULL corpus: {tx.trial_id.nunique()} trials (from Notebook 1).")
else:
    tx = demo_tx
    print(f"Using the {tx.trial_id.nunique()} demo trials. Run Notebook 1 for the full picture.")

try:
    from sentence_transformers import SentenceTransformer
except ModuleNotFoundError:
    raise ModuleNotFoundError("sentence-transformers is not installed — see the Setup note at the top.")
EMB = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
def embed(t): return EMB.encode(list(t), normalize_embeddings=True, show_progress_bar=False)

4 · Stage 2 — Semantic homing

Two things at a glance: (a) are the taboo words really the target’s near neighbours? and (b) does the clue-giver’s running speech close in on the target over a trial? → deep dive in Notebook 2.

import matplotlib.pyplot as plt
# (a) taboo vs random neighbours (metadata only)
words = meta.drop_duplicates("target_word"); tset = words.target_word.tolist()
tv = {w: v for w, v in zip(tset, embed(tset))}
ts, rs = [], []; rng = np.random.default_rng(0)
for _, row in words.iterrows():
    t = tv[row.target_word]
    ts += [float(a @ t) for a in embed([str(row[f"taboo_{i}"]) for i in range(1, 6)])]
    rs += [float(tv[w] @ t) for w in rng.choice([x for x in tset if x != row.target_word], 5, replace=False)]
_verdict = "much closer, so the game is well designed" if np.mean(ts) > np.mean(rs) else "NOT closer than random"
print(f"(a) taboo cosine {np.mean(ts):.3f}  vs  random {np.mean(rs):.3f}  — taboos are {_verdict}.")

# (b) homing curve for one trial
def clue(tid): return tx[(tx.trial_id == tid) & (tx.role == "clue_giver")].sort_values("start")
cand = [t for t in tx.trial_id.unique() if len(clue(t)) >= 3]
if cand:
    tid = cand[0]; tgt = tx[tx.trial_id == tid].target_word.iloc[0]; tvec = embed([tgt])[0]
    cum, T, S = "", [], []
    for _, s in clue(tid).iterrows():
        cum = (cum + " " + str(s.text)).strip(); T.append(s.end); S.append(float(embed([cum])[0] @ tvec))
    plt.figure(figsize=(7, 3)); plt.plot(T, S, "-o", color="#c44")
    plt.xlabel("time in trial (s)"); plt.ylabel(f"cosine to '{tgt}'")
    plt.title(f"(b) Semantic homing · {tid[:34]}"); plt.tight_layout(); plt.show()
else:
    print("(b) need a trial with >=3 clue-giver segments for a homing curve.")

5 · Stage 3 — Lexical alignment

Do partners reuse each other’s words? We measure content-word overlap between adjacent cross-speaker turns. → deep dive in Notebook 3.

import re
STOP = set(("a an the of to in on at for and or but is are was were be been am i you he she it we "
            "they me him her us them my your this that these those de het een van en of maar").split())
def toks(s): return [w for w in re.findall(r"[a-zA-Zà-ÿ']+", str(s).lower()) if w not in STOP]
def turns(tid):
    d = tx[tx.trial_id == tid].sort_values("start")
    return [(r.role, toks(r.text)) for _, r in d.iterrows() if toks(r.text)]

reuse = []
for tid in tx.trial_id.unique():
    seq = turns(tid)
    for i in range(1, len(seq)):
        if seq[i][0] != seq[i-1][0]:
            c, p = set(seq[i][1]), set(seq[i-1][1])
            if c: reuse.append(len(c & p) / len(c))
print(f"content-word reuse from partner's previous turn: {np.mean(reuse):.3f}"
      if reuse else "not enough cross-speaker turns yet — run Notebook 1 on more trials.")

6 · Stage 4 — Language meets the body

The corpus exists to study postural load. The board’s motion sensor shows the manipulation starkly — and it’s the channel Notebook 4 lines up against the language. → deep dive in Notebook 4.

g = pd.read_csv(os.path.join(CORPUS, "gyroscope.csv"))
AS = [c for c in g.columns if c.startswith(("AsX", "AsY", "AsZ"))]
g["mag"] = np.sqrt((g[AS] ** 2).sum(axis=1))
cond = {(str(r.pair_id), int(r.trial_number)): r.clue_giver_condition for _, r in meta.iterrows()}
g["cond"] = [cond.get((str(a), int(b))) for a, b in zip(g.group_name, g.trial_number)]
sway = g.groupby("cond")["mag"].mean().round(2).to_dict()
print(f"mean board sway (deg/s): {sway}")
print("On the ground there's no board to balance -> ~0; on the board the body works constantly.")

7 · Stage 5 — Clue-giving themes

Group the clue-giver utterances by meaning and label each group with its distinctive words. → deep dive in Notebook 5.

from sklearn.cluster import KMeans
from sklearn.feature_extraction.text import CountVectorizer
docs = tx[tx.role == "clue_giver"].text.astype(str)
docs = docs[docs.str.split().str.len() >= 2].tolist()
if len(docs) >= 6:
    Xt = embed(docs)
    K = max(2, min(5, len(docs) // 8))
    lab = KMeans(n_clusters=K, n_init=10, random_state=0).fit_predict(Xt)
    joined = [" ".join(d for d, l in zip(docs, lab) if l == c) for c in range(K)]
    cv = CountVectorizer(stop_words="english", min_df=1)
    C = cv.fit_transform(joined).toarray().astype(float); voc = np.array(cv.get_feature_names_out())
    tf = C / np.maximum(C.sum(1, keepdims=True), 1); A = C.sum() / C.shape[0]; f = C.sum(0)
    ct = tf * np.log(1 + A / np.maximum(f, 1))
    for c in range(K):
        print(f"topic {c}: " + ", ".join(voc[np.argsort(-ct[c])[:6]]))
else:
    print("Need more clue utterances for topics — run Notebook 1 on more trials.")

Where to go next

You’ve seen the whole arc — audio → transcripts → meaning → coordination → themes — on a few trials. For the real analyses (all 80 trials, the figures, the board-vs-ground contrasts, the caveats), open the deep-dive notebooks in order:

Notebook 1 — run it once with MAX_TRIALS = None to build the full transcripts.
Then 2, 3, 4, 5 — each reads those transcripts.
Notebook 0 needs no audio at all — a nice warm-up or poster visual.

Caveat for all of it: only two dyads currently have media (80 trials), so treat every number here as exploratory — read the shapes, not the p-values. The same code scales to the full corpus.