Belltake / bIQ

What is bIQ?

bIQ, Belltake IQ, is our measure of reasoning ability. It comes from short, wordless puzzles, it is scored the way modern intelligence tests are, and it is never shown to anyone, you included. It exists for one thing: showing how people of different reasoning ability react to the same post.

I.

An estimate that narrows.

Nobody can be measured from one puzzle. A bIQ starts wide and narrows with every answer. A right answer to a hard puzzle says more than a right answer to an easy one, and each new puzzle is pitched where it will tell the most, the way adaptive tests have worked since the 1970s.

Fig. 1. One person’s estimate after each of twelve answers (filled: right, open: wrong), with the range it is 95% sure of. An illustration.

II.

Puzzles with a pedigree.

Seventeen kinds of puzzle across three kinds of reasoning: spatial (turning and folding shapes in your head), pattern finding (series, matrices, rules) and logic (working out what must follow, and planning ahead). Each is a version of a task that established tests have used for decades (Raven, 1938; Shepard & Metzler, 1971; Ekstrom et al., 1976; Wechsler, 2008), rebuilt so it can be generated fresh. No two people see the same puzzle, and a separate program checks that every one has exactly one right answer before anyone sees it.

  1. 1938 Progressive MatricesJ. C. Raven
  2. 1971 Mental rotationR. N. Shepard and J. Metzler
  3. 1976 Kit of Factor-Referenced TestsR. B. Ekstrom and colleagues, ETS
  4. 1982 Tower of LondonT. Shallice
  5. 2008 Figure Weights, WAIS-IVD. Wechsler
  6. 2014 ICAR, in the public domainD. M. Condon and W. Revelle
  7. Now bIQ: all of them, made freshBelltake
Fig. 2. The tests bIQ’s puzzles descend from.
The other thirteen kinds, and the tests they are closest to

Spatial: turning and folding shapes in your head

Pattern finding: series, matrices and rules

Logic: working out what must follow, and planning ahead

III.

Scored like a real test.

Answers are scored with item response theory, the statistics behind modern adaptive testing (Lord, 1980; Embretson & Reise, 2000): every answer is weighed by how hard and how telling its puzzle is, with an allowance for lucky guesses (Birnbaum, 1968).

Then each person is compared with people their own age, because reasoning ability rises into early adulthood and declines later in life (Hartshorne & Germine, 2015), and every major intelligence test norms by age for the same reason. The result sits on the familiar scale: 100 in the middle, 15 points to a standard deviation.

Fig. 4. Raw results differ by age (20, 45, 75). Compared within age, everyone stands on the same scale. An illustration.

IV.

Never shown. Only counted.

No one sees a bIQ: not other people, not you, and not us in the app. You never learn whether an answer was right, or how hard a puzzle was. Nothing to brag about, nothing to fake, and nothing to feel bad about.

bIQ only places your reactions into a few broad groups. A split appears only once enough people have answered, small groups are left out, and every count carries a little random noise, so no one person can ever be picked out of it. Splits correct for measurement error across everyone at once (Mislevy, 1991) instead of trusting any single estimate.

V.

Can the curve be rigged?

A split is only worth something if nobody can tilt it. There are two ways to try, and lines of defence against both.

Fig. 5. Answers that don’t look like honest effort are set aside before a split is drawn. An illustration.

Cheating up.

Nobody sees a bIQ, so there is nothing to show off. The temptation is principle: someone who suspects they score low might cheat to pull the high end of a chart toward what they believe. So there are several lines of defence. It starts with blocking screenshots of puzzles, and with puzzles that can’t be captured efficiently in a screenshot or a description, of the kinds AI has proven worse at than people. Behind that sit statistical protections on the server, harder to get around and kept undisclosed, because describing them would help defeat them.

Failing on purpose.

The same trick in reverse: making accounts that answer wrong or at random, then reacting against what they believe, to make the other side look worse. Failing on purpose leaves its own statistical traces, and sign-ups made in bulk are screened. Answers that don’t look like honest effort, in either direction, are set aside before any split is drawn.

Verified splits.

Next comes ID verification, done securely through a trusted provider: one person, one account, and the details on the ID checked as valid. Belltake keeps only the fact that you are verified and a code that stops a second account, never a copy of the ID. Once a split has enough verified people, it can be shown for verified accounts alone.

VI.

Under scrutiny.

The objections a psychometrician, a journalist or a sceptic would raise, answered plainly.

Is bIQ an IQ test?
No. It is built the way intelligence tests are, with the same kinds of puzzles, item response theory and an age-normed scale with a mean of 100 and a standard deviation of 15. But it is taken online, a few puzzles at a time, and normed on Belltake’s own users rather than a representative national sample. So it is an IQ-style measure for group statistics, not a clinical IQ, and nobody is ever shown one.
Can a few puzzles here and there really measure intelligence?
Each answer adds information, and the estimate tightens as answers add up: in simulation, after about fifty puzzles it is as reliable as short published reasoning tests. Splits never rely on any one person’s estimate either: they use a handful of broad groups and correct for measurement error across everyone at once.
What stops people from cheating?
Nobody sees a score, so there is nothing to show off. The real temptation is to tilt the splits: someone who suspects they score low might cheat to pull the high end of a chart toward what they believe. So there are several lines of defence. It starts with blocking screenshots of puzzles, and with puzzles that can’t be captured efficiently in a screenshot or a description, of the kinds AI has proven worse at than people. Every puzzle is generated fresh, so there is no answer key to leak. Behind all that sit statistical protections on the server, harder to get around and kept undisclosed, because describing them would help defeat them. Answers that don’t look like honest effort quietly drop out of the statistics.
Couldn’t people fail on purpose to make the other side look bad?
That is the same trick in reverse: accounts that answer wrong or at random, then react against what they believe. Failing on purpose leaves its own statistical traces, and those answers are set aside too. Sign-ups made in bulk are screened, and ID verification is coming: one person, one account, and splits that can be shown for verified people alone once there are enough of them.
Is it fair to everyone?
Everyone gets the same kinds of puzzles, the same rules and the same scoring. Age is the only thing bIQ is compared within, as with the WAIS, the Stanford-Binet and Raven’s. Differences between groups are reported as they are, never adjusted. The puzzles use no words, but no test is free of culture, so country splits are best read with that in mind.
Does a split mean that smart people believe X?
No. A split describes the Belltake users who answered, compared with people their own age. It shows how people of different reasoning ability reacted to a post, not what is true, and not what everyone thinks.
How will you know it measures what it should?
A validation study will compare bIQ with an established, recently normed reasoning test and the public-domain ICAR-16, in a stratified adult sample. Until then bIQ is a measure relative to Belltake’s users, and every page that uses it says so.

VII.

What it’s built on.

  1. AERA, APA & NCME (2014). Standards for Educational and Psychological Testing. American Educational Research Association.
  2. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick, Statistical Theories of Mental Test Scores. Addison-Wesley.
  3. Carpenter, P. A., Just, M. A., & Shell, P. (1990). What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices Test. Psychological Review, 97(3), 404-431.
  4. Carroll, J. B. (1993). Human Cognitive Abilities: A Survey of Factor-Analytic Studies. Cambridge University Press.
  5. Condon, D. M., & Revelle, W. (2014). The International Cognitive Ability Resource: Development and initial validation of a public-domain measure. Intelligence, 43, 52-64.
  6. Ekstrom, R. B., French, J. W., Harman, H. H., & Dermen, D. (1976). Manual for Kit of Factor-Referenced Cognitive Tests. Educational Testing Service.
  7. Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. Lawrence Erlbaum.
  8. Hartshorne, J. K., & Germine, L. T. (2015). When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span. Psychological Science, 26(4), 433-443.
  9. Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum.
  10. Mislevy, R. J. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56(2), 177-196.
  11. Raven, J. C. (1938). Progressive Matrices: A Perceptual Test of Intelligence. H. K. Lewis.
  12. Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171(3972), 701-703.
  13. van der Linden, W. J., & Glas, C. A. W. (Eds.) (2010). Elements of Adaptive Testing. Springer.
  14. Wechsler, D. (2008). Wechsler Adult Intelligence Scale, Fourth Edition. Pearson.

Belltake keeps a full technical manual, written to the AERA, APA and NCME Standards, with every method, simulation and limitation. It opens to everyone with the site.

See it at work.

Belltake opens by invitation, in waves. Put your name down and be there when the first splits come in.

Join the waitlist