Overview
A human life is a sequence of interconnected events: jobs held, hospital admissions, places lived, and incomes earned. These events happen at different times, with different durations, and at different levels of granularity. Studying such sequences typically means aggregating them into simpler features: average income, number of hospital admissions, highest education, and marital status at time of survey. That aggregation is where information is lost.
This workshop shows how methods from natural language processing, specifically embeddings and transformer architectures, can enrich life-course studies and how one can use these techniques for exploratory data analysis. We apply transformers not to text but to sequences of life events.
We start hands-on with language-based transformers, then turn to life-event sequences: how to adapt transformers to life events, how to obtain dense representations (embeddings), how to visualise the resulting embedding space, and how to use these representations for downstream tasks. Finally, we look at methods for interpreting transformer predictions.
The scope. The workshop covers encoder-only, BERT-style masked modelling, and it does not cover autoregressive LLMs or prompting.
What you will leave with
- Introduction to the NLP concepts: tokenization, self-attention, transformers
- Insight into representation learning and embeddings
- Hands-on work with transformer models (BERT-style) on English-language text
- A full pipeline for life-course data processing and analysis: tabular events → sequences → tokens → encoder → representation
- Overview of methods to visualize and interpret token and sequence representations
- How to audit a prediction: trivial baselines, ablations, leakage probes and attribution
Schedule
Two sessions with a break.
| Block | Session | Duration |
|---|---|---|
| 01 | Natural Language Processing and Transformers Tokenisation, embeddings, self-attention and masked language modelling, worked on a pretrained English BERT. Includes one attention head computed by hand, away from the keyboard. | 1 hour 45 min |
| Break | 15 min | |
| 02 | A Person-Sequence Model, End to End Tokenising synthetic health records, masked prediction over event sequences, person-level embeddings, and honest baselines, including where a sequence model beats them and where it does not. | 1 hour 45 min |
| Buffer and discussion | 15 min |
Prepare
Assumed background
- Experience with Python and packages such as
pandasornumpy(comfortable reading and modifying data-manipulation code). - Familiarity with Jupyter notebooks or Google Colab.
- Experience with data processing and basic stats.
Explicitly not assumed
- PyTorch
torch, or any deep-learning framework. - Hugging Face
transformers. - Any prior exposure to embeddings, representation learning or NLP.
There are no from-scratch coding exercises. Notebook work is run-only or narrow fill-in-the-blank, and every blank is about a modelling decision rather than framework boilerplate.
What to bring
- A laptop with a browser and a code editor (you can also run notebooks in Google Colab).
- A pen and paper: one full segment happens away from the keyboard.
Before the workshop
If you want to prepare ahead of the workshop:
- You can find the environment specification in the GitHub repository. You can use
uv(preferred) orcondato install it. - You can find the preparation notebook (that downloads the necessary data and checks the environment): 0-Preparation.
Please run the preparation notebook before you arrive. It downloads roughly 700 MB of models and data, and conference wifi will not enjoy a hundred people doing that at once, and the first exercise starts twenty-five minutes in. You do not need a local environment: Colab is enough, and everything needed on the day is linked from this page.
Materials
Published here ahead of the session.
- SlidesPDF
- NotebooksSee below
- Pen-and-paper exerciseGitHub
- Pen-and-paper solutionsGitHub
- Pen-and-paper solutionsGitHub
- Environment setupGitHub
Notebooks
Open in Colab: nothing to install. The GitHub link shows the source and the saved outputs without running anything.
- 0 · Preparation Colab · GitHub
- A1 · Introduction to BERT Colab · GitHub
- A2 · What embeddings capture Colab · GitHub
- A3 · Probing for truthfulness Colab · GitHub
- B1 · From tables to sequences Colab · GitHub
- B2 · Pretraining and masked events Colab · GitHub
- B3 · The event embedding space Colab · GitHub
- B4 · Predicting severe disease Colab · GitHub
Run 0 · Preparation first. The rest assume the model and the data are already downloaded.
Data
In the first hands-on session, we will use a small dataset of true and false statements: Trilemma of Truth dataset.
During the second hands-on session, we will use a Synthea synthetic patient population. Synthea is a rule-based simulator for electronic health records (EHR). No real patient data is used and none is required to participate.
Both the pretrained event model and the patient data are published on the Hugging Face Hub, so the notebooks download them and nothing needs to be built on the day:
- carlomarxx/synthea-bert: the pretrained BERT-style model over event sequences (~4 MB).
- carlomarxx/synthea-workshop-data : 50,000 synthetic patients, with the vocabulary and labels (~19 MB).
Background reading
Not required. The workshop is self-contained.
- Using sequences of life-events to predict human lives
- CAREER: A Foundation Model for Labor Sequence Data
- BEHRT: Transformer for Electronic Health Records
- GETNext: Trajectory Flow Map Enhanced Transformer for Next POI Recommendation
- MAPLE: Mobile App Prediction Leveraging Large Language Model Embeddings
- Foresight — a generative pretrained transformer for modelling patient timelines
- Learning the natural history of human disease with generative transformers
Organizers
- Germans Savcisens Northeastern University Workshop Lead
- Charlie Rahal University of Oxford Conference Organizer (Senior TPC Member)
- Zerui Tian University of Oxford Workshops Chair
Questions about the workshop: germans@savcisens.com.