October 19, 2009

Simpson’s Paradox In The Slush Pile

Filed under: Books,Mathematics,Writing,Writing/Science Fiction — anilm @ 11:45 pm

In 1973, a time of pitchforks, flaming bras and napalm, the University of Berkeley received a total of 12,763 graduate program applications. 8,442 apps were from men, and the remaining 4,321 apps were from womenLet me forestall impertinent questions by quickly adding that before Al Gore invented the net, gender used to be a simple binary affair: you were either male or female; there were no in-between’s, no undecided’s, no none-of-the-above’s and no you-tell-me’s.. Of this hopeful lot, Berkeley admitted 3,738 men and 1,494 women. In other words, about 44% of the men were admitted compared with 35% of the women. A nine-percent difference. A woman applying to Berkeley’s grad programs had a 9% less chance of being admitted than a male. That, as statisticians like to joke, smelled of Fisher.

Assuming that women candidates were as qualified as the male candidates, what could explain why the discrepancy in admission rates was so large? Sexism, of course. There was no use pointing out that Berkeley was in effing California, not Louisiana. No use pointing out that 1,494 women had been admitted. Token women didn’t count. Women twice as good as men didn’t count twofold. Men as gentle as temple cows didn’t count. What counted was the 9%.

Eugene Hammel, the male Associate Dean of Graduate Studies, had the bright idea of asking Peter Bickel, a male statistics prof who was on the board of the Grad Council at Berkeley, to analyze the admissions data. The result of that analysis by Bickel, Hammel and O’Connell is now a statistical classic. They showed that on a department by department basis, if there was a bias, it was a slight one in favor of admitting women over men.

How was this possible? How could it be that at a departmental level, women were as likely, if not slightly more likely, to be admitted, but the admission rates for women were 9% lower than that of men? Was it… Could it be… Could it really be just… ARITHMETIC!!!

Yes. While the odds of admission did favor women on a department-by-department basis, the admission standards of different departments were not all the same. Some departments, say, Physics, had notoriously high standards. Other departments, say, Sociology, had notoriously low ones. What Bickel and gang showed was that women were applying in greater proportion to the more difficult programs rather than the easy ones, and so were getting rejected at higher rates. Men, strategically unambitious as always, were much more spread across the departments, and hence their slight disadvantage in odds was offset by the fact that more of them had sent their sweet nothings to the floozy departments. It was a tale with a statistical villain.

The villain’s name is Simpson’s Paradox. It is a statistical paradox that often arises when we calculate averages over aggregates. It sometimes happens that a statement may be true of every mixed subgroup (“Compared with men, women have a slightly higher odds of getting admitted to engineering/humanities/sciences/architecture/…”), but when you aggregate over all the groups, the statement turns false (“women have a significantly lower odds of getting admitted”). Simpson’s Paradox– that is, the potential for the paradox– plagues mixtures, heterogeneity, population studies of all kinds. It is perhaps the closest thing there is to the problem of evil in statistics.

So how is all this relevant to science fiction? Well, say there’s this fantasy world with two groups (genders) of writers (candidates): West and Other. Both groups have more or less the same distribution of talent. There are fewer Other writers than Western ones, and some chaps belong to both groups, but never mind that. Writers send in their stories to SF&F outlets (departments). Each outlet’s acceptance (admission) procedure is decided by an Editor. Not all the outlets are equally easy. Even though most outlets are in the West, the Others have a slightly better chance on a per outlet basis (because editors in this world act to encourage new voices). However, it turns out that the Others mostly apply to the harder-to-get-into outlets. Why? Well, these are the well-known ones, and if you’re in Pune, India, why send a story to the Vampire Gnome Anthology, when for the same time and postal expense and much greater potential benefit, you could send it to the New Yorker? And so it happens that there are great differences between acceptance rates for Westerners as compared to the Others. In this speculative world– hypothetical liberal world– Simpson’s Paradox, not racism, is the villain.

That world may not be our world. In our world, we have editors like William Sanders. But it also has editors who were willing to take chances with my writing, some of it truly godawful. So it’s hard to be sure. I’m going to give it the benefit of the doubt. Besides, I’d take doubt any day over the certainties of pitchforks, flaming bras and napalm.

September 15, 2008

Evolution Of The Obvious: The Foster-Kokko Model Of Superstition

Filed under: Mathematics,Science — anilm @ 11:03 pm

Social evolutionists Kevin Foster and Hanna Kokko, in their recent paper in The Proceedings Of The Royal Society,  set themselves the following problem:

“…under what conditions might a tendency for performing behaviours that incorrectly assign cause and effect be adaptive from an individual fitness point of view?”

It’s puzzling why the authors think there is anything to explain. Is superstitious reasoning an inheritable, evolutionary feature? Take woodcutting. Amateurs will work wood in incorrect and erroneous ways. That’s not to say there isn’t an efficient and systematic way to work wood that can be taught and encouraged. Do we really need an evolutionary explanation why we evolved to have babies who don’t know the difference between an adze and a maul?

Foster and Kokko’s real motivation is revealed, I think, in an earlier paragraph:

“In a world increasingly dominated by science, superstitious and indeed religious thinking typically take a back seat in academic affairs. However, superstitions play a central role in many small-scale societies, and indeed remain prevalent in the popular culture of all societies. Why is this? Can science rationalize this seemingly most irrational aspect of human behaviour?”

Needless to say, the authors’ rationalization is that superstitious reasoning may have some adaptive value. It’s a curiously Victorian attitude to human cognition; as if irrationality were somehow taboo, and town and manor had to be reassured that the phenomenon only appeared to be irrational.

One of my irrational habits, while reading papers on evolutionary models, is to substitute the key word– in this case, “superstitious reasoning”– with something else, say, “a fondness for weevils.” I’m glad to report that applying the technique to this paper produced an equally cogent explanation of why weevil-lovers roam the planet Earth.


August 30, 2006

Say, lovely woman, the number of bees: Bhaskara’s Lilavati

Filed under: Books,History,Mathematics — anilm @ 6:15 am

"Having bowed to the deity, whose head is like an elephant; whose feet are adorned by gods; who, when called to mind, relieves his votaries from embarrassment; and bestows happiness on his worshipers; I propound this easy process of computation, delightful by its elegance, perspicuous with words concise, soft and correct, and pleasing to the learned."

So begins Bhaskara‘s Lilavati in Henry T. Colebrooke‘s classic 1817 translation. There’s an earlier Bhaskara, also famous, so Lilavati‘s Bhaskara is sometimes referred to as Bhaskara II or Bhaskaracharya. He was born in 1114 A.D. in Vijayapura (modern day Bijapur), India. 

It was a quiet time. The quiet, that is, of a hurricane’s eye. In 1114 A.D., Angkor Wat was still an idea searching for its stone, but its future builder, King Suryavarman II, was already an year old. Genghis Khan was just fifty odd years away. The West hadn’t rediscovered Euclid and Aristotle yet, but Abelard of Bath and his students were in Syria, poring over the Arabic texts that would eventually ignite the Western renaissance. The Muslims had gained a foothold in Gujarat, and in the 13th through 15th centuries, they were to reinvigorate the subcontinent. And way north of Vijayapura, about 400 miles from Delhi, the last of the magnificent Khajuraho temples were being built.

Why did Bhaskara write the Lilavati? Simple. To teach duffers. As he concludes in the Bijaganita (one of his six works):

"A morsel of tuition conveys knowledge to a comprehensive mind; and having reached it, expands of its own impulse. As oil poured upon water, as a secret entrusted to the vile, as alms bestowed upon the worthy, however little, so does knowledge infused into a wise mind spread by intrinsic force….What is there unknown to the intelligent? Therefore for the dull alone it is set forth."

Almost 600 years later, in his Decline and Fall, Gibbons was even more pessimistic:

"The power of instruction is seldom of much efficacy except in those happy dispositions where it is almost superfluous."

And more recently, the late Richard Feynman– by all accounts, a great teacher– cites Gibbons with glum relish in the preface to his celebrated Lectures.

The Lilavati is a collection of worked out examples in algebra and geometry. The level of mathematics ranges between high school algebra and freshman pre-Calculus. In its time, it represented the height of 12th-century mathematics. The problems are generally  addressed to one Lilavati, traditionally taken to be either his wife or his daughter. Tradition is as good a reason as any because there’s no other reason to support the claim. To modern ears, the phrasing of some of the problems is decidedly odd. Problem 2.2.16 begins with:

"Beautiful and dear Lilavati, whose eyes are like a fawn’s…."

Then there’s Problem 3.1.49 which begins:

"Pretty girl, with tremulous eyes, if thou know the correct method of inversion… "

And how can I overlook Problem 3.5.68?

"The square root of half the number of a swarm of bees is gone to a shrub of jasmine; and so are eight-ninths of the whole swarm: a female is buzzing to one remaining male that is humming within a lotus in which he is confined, having allured to it by its fragrance at night. Say, lovely woman, the number of bees."

Colebrooke, thorough as always, notes that the "jasmine" referred to is the "jasminum grandiflorum." And Ganesa, in his Buddhivilasini (1545 AD), supplies some context: "the lotus being open at night and closed in the day, the bee might be caught in it."

Indeed. The good professor’s concern is not misplaced. The hazards of being a bee are many. For poor Colebrooke, the text must have made many a warm Calcutta night even warmer.


October 6, 2005

Caste, Closure & Contagion

Filed under: Culture,History,Mathematics,Philosophy — anilm @ 8:48 am

In 1857, Maria Mitchell (1818-1889), the first female American astronomer, went on a tour of Europe.

She was already very famousMariamitchell. At 29, she’d discovered a comet (after calculating its expected position), was elected to the American Academy of Arts & Sciences a year later, and was the first female professor of the United States. These bland biographical details obscure a more interesting story. It was an age where women were considered too stupid to be entrusted with the vote. It was an age where a Thomas Huxley could argue that women were not worthy of membership in learned societies because they were, ipso facto, amateurs. It was an age where the American Academy felt it necessary to mention in their 1848 report that they’d decided to grant her membership "in spite of her being a woman" [1]. The next admission of a female to that august assembly would be in 1943!

In Europe, she met everybody who was somebody: Roget, Babbage, Humboldt, Somerville, George Airy, Leverrier, Stokes, Struve, Herschel … the list is a who’s who of 19th century Arts & Science. Her diary reveals a keen mind with a droll sense of humor [13]. For example, she comments that:

Thus far England has impressed me seriously; I cannot imagine how it has ever earned the name of ‘Merrie England.’

She demolishes the ever-reliably pompus William Whewell with:

An Englishmen is proud; a Cambridge man is the proudest of Englishmen; and Dr. Whewell, the proudest of Cambridge men.

She comments about a statue of Dr. Johnson that, "It must be like him, for it is exceedingly ugly."

Then there’s her entry on November 2, 1858. She commented that:

It was hard for me to become accustomed to English ideas of caste. I heard Professor Sedgwick say that Miss Herschel, the daughter of Sir John and niece to Caroline, married a Gordon. ‘Such a great match for her!’ he added; and when I asked what match could be great for a daughter of the Herschels, I was told that she had married one of the queen’s household, and was asked to sit in the presence of the queen!

"When I hear a missionary tell that the pariah caste sit on the ground, the peasant caste lift themselves by the thickness of a leaf, and the next rank by the thickness of a stalk, it seems to me that the heathen has reached a high state of civilization — precisely that which Victoria has reached when she permits a Herschel to sit in her presence!

This particular entry got me thinking. It’s a natural and intuitive analogy. However, Ms. Mitchell made two errors. First, though the caste system is related to class, race and religion, it’s a very distinct entity. Second, though the caste system establishes a pecking order, it’s not a hierarchy. I will attempt to justify both statements.


September 7, 2005

The God of Small Things

Filed under: Mathematics — anilm @ 5:52 pm

Goodstein’s remarkable theorem is almost completely unknown outside of formal logic. Roger Penrose sees deep consequences in the fact that Goodstein’s mind was able to discover it. Penrose’s claim, as I understand it, is that human minds can derive theorems that a Turing machine cannot, and therefore  we’re not — or rather, Goodstein is not — a Turing machine. So what’s this theorem about?

Note: 2^x means 2 multiplied with itself x times. Thus 2^3 = 2 * 2 * 2 = 8.

1. Take any positive number. [We'll use the number 33 as a running example.]
2. Write the number in base-2 representation. [33 =  2^5 + 1]
3. Express everything — even the the powers — in base 2. [33 = 2^5 + 1 = 2^(2^2 + 1) + 1]
(This is called the hereditary representation of a number. )

4. Increase the base by 1.  [33 = 2^(2^2 + 1) + 1 --> 3^(3^3 + 1) + 1 = 22876792454962.]
5. Decrease the whole number by 1. [3^(3^3 + 1) + 1 -->  3^(3^3 + 1) = 22876792454961.]
6. Repeat steps 4 and 5 alternatively.

What do you think will happen? One would think that the number would get larger and larger and larger, since base increases have much more impact than the tiny reductions by 1. In the running example, the number went from 33 to 22876792454962 after step 4.

But R. L. Goodstein showed that the small constant decreases by 1 will eventually make the sequence converge to … ZERO!

Initially, the increases in base sends the numbers exploding, but the decrements in step 5 take their fatal toll and eventually converges the sequence to zero. It is death by a googol of pinpricks.

Amazing, right? Here’s an even more amazing fact. J. Paris and L. Kirby showed that Goodstein’s theorem cannot be proved using Peano’s axioms (roughly, basic arithmetic). In short, it’s an example of a Godel statement.

So how did Goodstein prove it? He used a stronger form of induction known as transfinite induction.

Goodstein’s theorem: a mathematical version of the God of Small Things, so to speak. Why did Arundhati Roy take so many pages?


Eric W. Weisstein.  "Goodstein’s Theorem." From MathWorld–A
Wolfram Web Resource.  http://mathworld.wolfram.com/GoodsteinsTheorem.html

Goodstein, R. L. "On the Restricted Ordinal Theorem." J. Symb. Logic 9,
33-41, 1944.