Wednesday, 27 June 2012

On Percentile Ranking


Some folks who have not been following the earlier threads of discussion have misinterpreted my last post. That post was not a defence of the percentile ranking system of comparing Board marks. It was merely to refute the assertion of Dheeraj (and others, by the way) that “They gave a report which said that more studies needed to be done with data from more boards for more years.
This had two problems. One, MHRD would have taken a long time to get all this data. ….”
I had argued that the ISI report had not stated that more studies would need to be done before one could say that percentile ranking can be used for comparison.
Now,  the claim is that if the two assumptions are made, then, percentile ranking can be used for comparison. Actually there is only one assumption. The first assumption,  to quote the ISI report (not my words!), “Aggregate scores are expected to increase from less meritorious to more meritorious students in any particular subject”, is a mere technical one, which is saying that marks are not given randomly. The student who does better, gets more marks.   Clearly, if this is questioned, Board marks cannot be used for anything. It is the second assumption that is the basis for asserting that we can use percentile ranks for comparison. And that is, “Merit distribution is the same in all boards.” .  Some are asking, what is merit? How do you define it?  I cannot do much more but to suggest synonyms for merit in this context: innate ability, intelligence.
Some have questioned the above assumption. Fine. After all, it is an assumption. It is an assumption that cannot be “proved”. One argument against it has been that some of the Boards are so small, that this assumption will surely not hold. I have tried to argue that it is the size of the base population that the Board represents, that should be looked at. The numbers will not be so small then. There may be an exception or two (a Sanskrit Board exists, I am told), but this will introduce very few errors, if any.
The point I wish to make in this post is the following: if Board results across the 42 Boards of the country are to be compared in any form, THE ONLY REASONABLE way is to use percentile ranks. Nobody has suggested , to the best of my knowledge, any other method in any of the comments, posts, articles, etc. that have come out in the last three months.
I have also tried to argue that it is reasonable to do so because the errors introduced by this method, if any, are no different than the errors introduced in other  stages of any admission process: question paper setting, evaluation (even machine evaluation), tie-breaking rules, state of mind of the candidate on the exam day, the health of the candidate on the exam day, and so on.
I have also argued that, because of the bunching effect, at least for the “good” students, the difference in marks due to Board results will be very small. The candidate at the bottom of the top 10% will be 5 marks (with 50% weightage) away from the top candidate, and probably 3-4 marks away from the last candidate who qualifies on the basis of Board marks alone. So he has a very good chance to make this up in the exams. In fact, as opposed to using any cut-off, EVERYONE has a chance., no matter what his Board marks are.
There has been some objections of combining percentile ranks with marks of the exams, stating that such combinations are not valid. Why not? It is only for the purpose of ranking. THE IIT SYSTEM HAS BEEN COMBINING PERCENTILE RANKS and MARKS FOR YEARS. M. Tech admission rules (framed by MHRD!) requires that 70% weightage be given to GATE scores and 30% to tests / interviews conducted. Till a few years back, GATE scores were available only in percentile form. We used to combine the percentile with the marks obtained in a  local test to rank the students. Almost always, the local test decided the ranking, as the students had percentiles in the range of 95-99. So, a candidate had to do well in GATE (to get within the 95-99 range ) but he had to be “intelligent” to get into the system! This is exactly what is being proposed in here, except that we are not restricting candidates to the 95-99 range, but we are allowing everyone to compete.


Saturday, 23 June 2012

Response to Dheeraj Sanghi's Open Letter


According to the IIT Kanpur Website, there has been a “A devastating and rather harsh expos√© of the 'scientific temper' (or the lack of it) shown by members of the IIT Council. 'JEE 2013: An Open Letter to Prof. Barua' by Prof. Dheeraj Sanghi, IIT Kanpur.”
Elsewhere in the same website we have “A very strong response by Prof. Dheeraj Sanghi, IIT Kanpur to the claims made by those defending the IIT Council proposal.”

Harsh and strong: I agree.  I have no desire to engage in argument regarding my motives and my behaviour. I only wish to state that I reject all allegations of lying. I have defended the proposal because I think it is the best alternative under the present circumstances. I was not responsible for delaying the Aptitude Test.  In fact not only me, but the IITG Senate wanted an  Aptitiude test (see the IITG Senate resolution of Apr 25) . It should come in later years.  I have no hidden agenda and I do not have any “irrestible urge to manage other IITs” (ridiculous! way beyond decency!).
I forgive Dheeraj for his trespasses for he knows not what ……

But I would like to focus on the meat of the proposal:


 On the ISI Report and Percentile Ranks

 
Dheeraj Sanghi has stated that
They gave a report which said that more studies needed to be done with data from more boards for more years.
This had two problems. One, MHRD would have taken a long time to get all this data. ….

He has obviously not read the report or has not understood its contents.
The ISI report made the following assumptions (the report is available here):

2  Assumptions needed for comparability of different board scores
The following assumptions would have to be made in order to make the aggregate
scores of different boards comparable.
  Aggregate scores are expected to increase from less meritorious to more
meritorious students in any particular subject 
  Merit distribution is the same in all boards.

The first assumption is that Boards awards marks according to merit.
               This has been challenged by many  with respect to  State Boards without any analysis of any data (not sure it is even possible to do any analysis as merit cannot be established objectively: it has to be something society by and large agrees upon), but by anecdotal evidences of corruption, fraud etc.
The second assumption is that meritorious students are unformily distributed across all Boards (I have used the argument of the law of large numbers in relation to the population base of Boards (and not the size of the Boards) to argue in favour of this).
               This has been challenged by some on the basis of the varying sizes of Boards, but again without any analysis of any data (again, not sure it is even possible to do any analysis).

The ISI report then goes on to state (bold mine):

3  Stability of board scores
Under the above assumptions, the percentile ranks of students  in different board
examinations become directly comparable. It would be of interest to observe how the
raw aggregate scores relate to the percentile ranks, and how these relationships vary
from year to year as well as across different boards.

There is therefore no need for any more analysis of data of other Boards to establish this assertion. I throw an open challenge to anyone to refute this assertion. It is so simple, what is there to refute? Any classs IX student should be able to understand. Unfortunately, many well respected IIT faculty have failed to understand this. Maybe they have not read the ISI report (the full report is enclosed in another post).

Now the ISI report does talk about analysing the data of other Boards, Why? First of all they repeat the above assertion again in section 4 (bold mine):

4  Criterion for selection
Under the two assumptions mentioned  in Section 2, the percentile ranks of the
students computed from aggregate scores are comparable across different boards and
years. Any monotone transformation of the percentile ranks is also appropriate for
comparison, as long as the same transformation is used across  different boards and
years. Let us now consider a few such transformations.

They then go on to consider a transformation (bold mine):

Any of the curves in the first figure is a monotone function of the percentile rank. One
can use any one of them, say CBSE 2007, as standard. If the same transformation of
percentile ranks is used for other boards and years, then the resulting modified score
of any student of any board in any year can be regarded as the aggregate score, which
could have been obtained by that student if he/she had appeared for the CBSE
examinations in 2007. Thus, the transformed scores provide a common basis for
comparison.

A feature of such a transformation is that, after this transformation, the scores are not
evenly distributed throughout the available range of scores. In particular, when the
scale of the CBSE 2007 aggregate score is used,  less than 5% of  the  students have
scores in the range of 90% to 100% of the maximum score. On the other hand, more
than 10% of the students (spanning over the percentile range of 50 to 62) have scores
squeezed in the narrow range of   65% to 70% of maximum score. This would lead to
a loss of discriminating power in that percentile range, particularly if the board scores
are used only as a component in a weighted selection criterion involving multiple
components.
For maximal discrimination over the requisite range of percentile ranks, it is
imperative that the scores have the uniform distribution over that range. This may be
achieved if the percentile ranks themselves are used as scores. If there is a  threshold
percentile, say 75%, then the available range is maximally utilized by using the
following linear transformation of the percentile rank:

(Percentile Rank of Student -75 / (100-75) )  * 100  -- (1)


According to this scale, a student with percentile rank 75 receives the score 0, a
student with percentile rank 90 receives 60, and the topper receives 100. Similar
computations can be done for other choices of the threshold percentile.

Then comes the recommendations, which has caused some confusion as some eminent folks seem to have read only the recommendations and not the rest of the report.

5  Recommendations
(a) The above analysis regarding stability of board scores should be carried out
for all the boards over a longer period of time.
(b) If the reported stability of the board scores is found to hold generally,  then a
transformed percentile rank with a suitable cut-off, as described in (1), may be
used as a score representing performance in the board examination, for the
purpose of admission to tertiary education.
(c) The different boards should be asked to indicate  the  percentile rank of each
student in the mark sheet.
(d) In order to prepare a formal and reliable basis for selection at the tertiary level,
educational institutions at that level, including the IITs, should be asked to
provide to the HRD ministry a statement of marks obtained by each graduating
student, together with the student’s score in the admission test of that
institution (if any), the board score at the class XII level and the name of the
board.

Now why is the analysis mentioned in (a) above required?  Because of recommendation (b)! A transformation is recommended only if the analysis of (a) is done. But if there is to be no transformation but the percentile ranks themselves are used as scores, then there is no need to analyse any further data, as the two assumptions are there. One may point out that since ISI did not propose this, there must be a problem with using percentile ranks as scores.  I think they wanted better discrimination through some transformation and so they only recommended some transformation. I confess I am not able to give a clear answer to this. But I am confident that what has been proposed is sound (see below).



Now to the formula in the proposal.  The only difference is that ISI had suggested a cut-off and had recommended that a suitable cut-off  be used, but the proposal uses no cut-off. Why was this done? This was done because with reservations, any cut-off  could adversely affect the filling up of reserved seats.  Further, while a cut-off would improve the level of discrimination,  it was felt that since the proposal was likely to meet some resistance, it is better to reduce the  discrimination, and let the exams be the discriminating components. So, there was no “Barua formula”, and  there was nothing sinister about the proposal. The “formula” itself, which was not given by ISI (they might have felt that they would be insulting the readers of their reports if they did so – in hindsight, they should have done so!), is a standard one that can be found in any text book on Statistics. I cannot be given credit for this ( a case of reverse plagiarism?).



Saturday, 16 June 2012

IIT JEE 2011 Report


The JEE 2011 report is here. Comments:
1.      Most of the data is voluntary and unverified, so please  be careful while interpeting the data. We think  that  income information is particularly suspect.
2.      About 45% of the students who qualify are from CBSE. The  percentage appearing is similar. Why is it that with about 12% of the class XII students, CBSE applicants are so high?  Food for thought!
3.      Look at the “origin” of the qualified candidates.  Andhra Pradesh and Rajasthan with 2693 and 1931 successful candidates top the list. No surprise as the report says: “These figures are consistent with the data of JEE 2010 and with available data to show that a large number of JEE coaching centres operate out of these two states.”
4.      To further emphasise the point: city wise data – “Jaipur (read Kota)  leads with 1458 candidates followed by Hyderabad with 1307 candidates.” 

Friday, 15 June 2012

IITK's Actions and the IIT Act


IIT Kanpur’s Actions Re: the IIT Council’s decision and their implications vis-√†-vis the IIT Act

·         Ordinance 3.2 of IIT Kanpur reads:

3.2 The Admission of Indian Nationals to the B. Tech., B. Tech.-M. Tech. (Dual Degree) and M.Sc. (Integrated) Programmes shall be made once a year on the basis of the Joint
Entrance Examination (JEE) conducted jointly by all the IITs.

·         So, their resolution to conduct their own Entrance Examination goes against Ordinance 3.2, which states that the JEE will be “conducted jointly by all IITs”.
·         The Senate will therefore have to amend this Ordinance.
·         This amendment, by clauses 29(2) and 29(3) of the IIT Act, has to go to the Board of IITK, and the Board can cancel or modify it.
·         Now, the issue is, even if the IITK Board approves the modification, is it valid?
·         Clause 33(2)(b) apparently gives powers to the IIT Council to set policies for “matters of common interest”.
·         IITK is apparently arguing that Clause 33(2)(a) deals with issues academic, and there the role of the Council is to “advise on matters relating to”. So the Council has only an advisory role in this matter and Clause 33(2)(b) does not apply to matters academic.
·         There is now a question of what is meant by “advise”. The following definitions are from the “World English Dictionary”

1. to offer advice (to a person or persons); counsel: he advised the king ; to advise caution ; he advised her to leave 2. formal  to inform or notify 3. obsolete chiefly , or  ( US ) to consult or discuss

·         The “formal” definition is “to inform or notify”.
·         Further, clause 37 states that “If any difficulty arises in giving effect to the provisions of this Act” then the Central Government  can “make such provision or give such direction not inconsistent with the purposes of this Act, as appears to it to be necessary or expedient for removing the difficulty.”
·         Finally, as per clause 9 of the IIT Act, the Visitor may appoint persons to “to review the work and Progress of any Institute and to hold inquiries into the affairs thereof and to report thereon” and to “take such action and issue such directions as he considers necessary in respect of any of the matters dealt with in the report and the Institute shall be bound to comply with such directions.”
·         I am not a lawyer, so these are just a layman’s reading of the provisions of the IIT Act and its implications in the current matter.