Fairness in rankings
Rankings shape hiring, admissions, and recommendation systems — but most fairness frameworks ignore that real candidate pools are small and finite, and that selecting one candidate changes the odds for everyone else.
The gap in existing fairness frameworks
When organisations rank candidates for jobs, university places, or loans, a growing literature asks whether the resulting rankings are fair to minority groups. Most existing fairness measures assume independence: each candidate's selection is treated like an independent coin flip with a fixed probability. But selections from real pools are not independent — once a candidate is picked, the composition of the pool changes, and so do the probabilities for everyone else.
For small pools or for top-k selections that are large relative to pool size — exactly the cases that matter most in practice — this independence assumption produces misleading fairness assessments. This is why we replace the binomial distribution used in past methods with a hypergeometric model that accounts for the finite pool and the changing probabilities as candidates are selected.
Binomial and hyper... what?
A simple silly analogy helps illustrate the difference. In standard Russian roulette, the cylinder is spun before every pull — so the probability of a bullet is always 1-in-6, regardless of what happened before. Each round is independent, with a fixed probability. That is the binomial world.
Now what if the cylinder is only spun once at the start, and then you pull the trigger repeatedly until you get a bullet?
The scene that inspired the analogy — Squid Game, season 2
Now in this second version, the probability of a bullet changes with every pull. The first pull has a 1-in-6 chance, but if you survive, the next pull is now 1-in-5, then 1-in-4, and so on. This is the hypergeometric world — where selections are not independent, and probabilities shift as the pool is depleted.
Data
The framework is evaluated on synthetic ranking scenarios and a real-world university admissions dataset. Synthetic experiments systematically vary pool size, group proportions, and selection fraction to map where the finite-pool correction changes conclusions relative to the binomial baseline — especially when the protected group is small or when a large share of candidates is selected.
The empirical case study uses anonymised student records from an elite university that expanded low-income enrolment via a government scholarship programme, shared by Prof. María José Álvarez-Rivabulla. Three student cohorts are analysed with GPA-based rankings and socioeconomic status and gender as protected attributes (data not publicly available). The interactive Ranks of Disparity tool allows practitioners to apply the framework to their own ranking data.
hyperFA*IR
We introduce hyperFA*IR, a framework built on the hypergeometric distribution — the natural model for sampling without replacement from a finite pool with known group sizes. The framework lets us:
(i) evaluate whether an observed ranking is statistically consistent with fair sampling, accounting for the constrained nature of the candidate pool; (ii) enforce fairness during ranking generation in a way that is statistically grounded rather than relying on rigid quotas; and (iii) reason about uncertainty — distinguishing genuine bias from outcomes that look unfair but are well within the range of random variation.
Findings
The hypergeometric correction changes conclusions most in two regimes: when the protected group is small relative to the total pool, and when a large fraction of the pool is selected (large top-k). In both cases the binomial model's independence assumption is furthest from reality.
The empirical case study — a Latin American university that expanded low-income enrolment via a government scholarship programme — illustrates both failure modes. Under the binomial model, gender and socioeconomic disparities in certain GPA-based rankings appeared either more or less statistically significant than under the hypergeometric model, with the direction depending on pool composition. Some apparent disparities disappear; some hidden ones surface.
Beyond auditing, hyperFA*IR enables fair generation: producing rankings that satisfy statistical fairness constraints while accounting for pool depletion, without imposing rigid numerical quotas.
Try it yourself
Together with Diego Baptista Theuerkauf and Liuhuaying Yang (thank you both!) we built Ranks of Disparity — an interactive tool that walks through the intuition for finite-pool fairness and lets you run the hyperFA*IR test on your own ranking data. Upload your numbers and see immediately whether your ranking is statistically consistent with fair sampling — and by how much it falls inside or outside the hypergeometric confidence corridor.
Open Ranks of Disparity →