According to the IIT Kanpur Website, there has been a “A devastating and rather harsh exposé of the
'scientific temper' (or the lack of it) shown by members of the IIT Council.
'JEE 2013: An Open Letter to Prof. Barua' by Prof. Dheeraj Sanghi, IIT Kanpur.”
Elsewhere in the
same website we have “A very strong response by Prof. Dheeraj Sanghi,
IIT Kanpur to the claims made by those defending the IIT Council proposal.”
Harsh and strong: I agree.
I have no desire to engage in argument regarding my motives and my
behaviour. I only wish to state that I reject all allegations of lying. I have
defended the proposal because I think it is the best alternative under the
present circumstances. I was not responsible for delaying the Aptitude
Test. In fact not only me, but the IITG
Senate wanted an Aptitiude test (see the
IITG Senate resolution of Apr 25) . It should come in later years. I have no hidden agenda and I do not have any
“irrestible urge to manage other IITs” (ridiculous! way beyond decency!).
I forgive Dheeraj for his trespasses for he knows not what
……
But I would like to focus on the meat of the proposal:
On the ISI Report and Percentile Ranks
Dheeraj Sanghi has stated that
They gave a report which said that
more studies needed to be done with data from more boards for more years.
This had two problems. One, MHRD would have taken a long time to get all this
data. ….
He has obviously not read the report or has not understood
its contents.
The ISI report made the following assumptions (the report is
available
here):
2
Assumptions needed for comparability of different board scores
The following assumptions would
have to be made in order to make the aggregate
scores of different boards
comparable.
•
Aggregate scores are expected to increase from less meritorious to more
meritorious students in any
particular subject
•
Merit distribution is the same in all boards.
The first assumption is that Boards awards marks according
to merit.
This has
been challenged by many with respect to State Boards without any analysis of any data (not sure it is even possible to
do any analysis as merit cannot be established objectively: it has to be
something society by and large agrees upon), but by anecdotal evidences of
corruption, fraud etc.
The second assumption is that meritorious students are
unformily distributed across all Boards (I have used the argument of the law of
large numbers in relation to the
population base of Boards (and not the size of the Boards) to argue in
favour of this).
This has
been challenged by some on the basis of the varying sizes of Boards, but again without any analysis of any data (again,
not sure it is even possible to do any analysis).
The ISI report then goes on to state (bold mine):
3
Stability of board scores
Under the above assumptions, the percentile ranks of students in different board
examinations become directly comparable. It would be of interest to
observe how the
raw aggregate scores relate to the
percentile ranks, and how these relationships vary
from year to year as well as across
different boards.
There is therefore no
need for any more analysis of data of other Boards to establish this assertion.
I throw an open challenge to anyone to refute this assertion. It is so simple,
what is there to refute? Any classs IX student should be able to understand.
Unfortunately, many well respected IIT faculty have failed to understand this.
Maybe they have not read the ISI report (the full report is enclosed in another
post).
Now the ISI report does talk about analysing the data of
other Boards, Why? First of all they repeat the above assertion again in
section 4 (bold mine):
4
Criterion for selection
Under the two assumptions mentioned
in Section 2, the percentile ranks of the
students computed from aggregate scores are comparable across different
boards and
years. Any monotone transformation of the percentile ranks is also
appropriate for
comparison, as long as the same
transformation is used across different
boards and
years. Let us now consider a few
such transformations.
They then go on to consider a transformation (bold mine):
Any of the curves in the first
figure is a monotone function of the percentile rank. One
can use any one of them, say CBSE
2007, as standard. If the same transformation of
percentile ranks is used for other
boards and years, then the resulting modified score
of any student of any board in any
year can be regarded as the aggregate score, which
could have been obtained by that
student if he/she had appeared for the CBSE
examinations in 2007. Thus, the
transformed scores provide a common basis for
comparison.
A feature of such a transformation
is that, after this transformation, the scores are not
evenly distributed throughout the
available range of scores. In particular, when the
scale of the CBSE 2007 aggregate
score is used, less than 5% of the
students have
scores in the range of 90% to 100%
of the maximum score. On the other hand, more
than 10% of the students (spanning
over the percentile range of 50 to 62) have scores
squeezed in the narrow range
of 65% to 70% of maximum score. This
would lead to
a loss of discriminating power in
that percentile range, particularly if the board scores
are used only as a component in a
weighted selection criterion involving multiple
components.
For maximal discrimination over the requisite range of percentile
ranks, it is
imperative that the scores have the uniform distribution over that
range. This may be
achieved if the percentile ranks themselves are used as scores. If
there is a threshold
percentile, say 75%, then the
available range is maximally utilized by using the
following linear transformation of
the percentile rank:
(Percentile Rank of Student -75 /
(100-75) ) * 100 -- (1)
According to this scale, a student
with percentile rank 75 receives the score 0, a
student with percentile rank 90
receives 60, and the topper receives 100. Similar
computations can be done for other
choices of the threshold percentile.
Then comes the recommendations, which has caused some
confusion as some eminent folks seem to have read only the recommendations and
not the rest of the report.
5
Recommendations
(a) The above analysis regarding
stability of board scores should be carried out
for all the boards over a longer
period of time.
(b) If the reported stability of
the board scores is found to hold generally,
then a
transformed percentile rank with a
suitable cut-off, as described in (1), may be
used as a score representing
performance in the board examination, for the
purpose of admission to tertiary
education.
(c) The different boards should be
asked to indicate the percentile rank of each
student in the mark sheet.
(d) In order to prepare a formal
and reliable basis for selection at the tertiary level,
educational institutions at that
level, including the IITs, should be asked to
provide to the HRD ministry a
statement of marks obtained by each graduating
student, together with the
student’s score in the admission test of that
institution (if any), the board
score at the class XII level and the name of the
board.
Now why is the analysis mentioned in (a) above
required? Because of recommendation (b)!
A transformation is recommended only if the analysis of (a) is done. But if
there is to be no transformation but the percentile ranks themselves are used
as scores, then there is no need to analyse any further data, as the two
assumptions are there. One may point out that since ISI did not propose this,
there must be a problem with using percentile ranks as scores. I think they wanted better discrimination
through some transformation and so they only recommended some transformation. I
confess I am not able to give a clear answer to this. But I am confident that
what has been proposed is sound (see below).
Now to the formula in the proposal. The only difference is that ISI had suggested
a cut-off and had recommended that a suitable
cut-off be used, but the proposal
uses no cut-off. Why was this done? This was done because with reservations,
any cut-off could adversely affect the
filling up of reserved seats. Further,
while a cut-off would improve the level of discrimination, it was felt that since the proposal was
likely to meet some resistance, it is better to reduce the discrimination, and let the exams be the
discriminating components. So, there was no “Barua formula”, and there was nothing sinister about the proposal.
The “formula” itself, which was not given by ISI (they might have felt that
they would be insulting the readers of their reports if they did so – in
hindsight, they should have done so!), is a standard one that can be found in
any text book on Statistics. I cannot be given credit for this ( a case of
reverse plagiarism?).
