Grading

I wish we will all pass and then we can start studying for Level II :)
 
Micholien wrote:Modified Angoff is quite interesting! this is crazy though, i feel like it works backward to what it should be… the more easy question u get the more points u get! lets say u spend 4 minutes on a tough question and get i right but they expect only 30% of the body of candidates to have it right then u get only 30% credit for it while like, lets say a stupid definition that you havent memorized you take a guess and you are wrong but because it was an “easy question” and they expect 90% of people to get it right (then you miss a question worth 3 times as much).
To summarize in a scenario described above: its like instead of 50-50 you end up 25% (30%/120%) despite knowing something more complicated than others!
I don’t think you understand Angoff quite right.
Let’s say there are three questions on the exam, A, B, C. A is really easy so the standard setters believe that 90% of minimally qualified CFA’s will get it. The 90% is called the Angoff weight. B is harder so the Angoff weight is 50%. C is really hard so the Angoff weight is 10%. If eaach question is worth 3 points, the passing score on the exam would be 3*0.9 + 3*0.5 + 3*0.1 = 4.5.
Now you take the test and Angoff has nothing to do with how the test is scored. If you get questions B, C correct, you get 6 points and pass the exam (which has a passing score of 4.5). So anyway, if you get a hard question right, that’s really good because that hard question didn’t raise the passing score by much but you got full credit for it so your score jumps a lot. The flipside is that missing easy questions is really bad.
I also completely disagree with that snippet that says that Angoff was used for Level 1 starting in 1996. CFAI (actually AIMR then) was pretty hush-hush about how they determiined the minimum passing score then but I didn’t hear any mention of Angoff until many years after that. I’m pretty sure I would have heard about it from someone.
In some past life, I was an empirical scientist. The whole basis for empirical science is that you don’t know anything except that which can be observed. Then a bunch of psychometricians come along (a group I have very little respect for since I was one of them once) and says that the way to score an exam is to have some expert tell us what percent of minimally qualified caandidates can answer a particular question. That’s the antithesis of empirical science. Who cares what he thinks? Let a bunch of people take the exam and assume that the populations that take the exams every year are about the same. Then differences in scoring each year are due to differences in the exam not the peole taking the exam. Then curve the exam accordingly. I am completely, 100% convinced that my way makes more sense than their way. In fact, I think Angoff is inconsistent with pretty much our whole epistemology of science and it is something that only a psychometrician could like (the problem with psychometricians is that they are pseudo-quants - even the most well-respected psychometricans in the country know far less math and stats than a run of the mill quant at a hedge fund or investment bank)
Edit: BTW - What do you think about an Angoff weight below 33% on a multiple choice exam with 3 choices?
 
JoeyDVivre wrote: I’m not sure you do want the passing score released. If it’s a 58 or something, you could lose some credibility if people think that kind of score is just too low.
Great point. My roommate and I went to the same B School with the same major and took the same classes. He couldn’t figure out how a 3 option multiple choice exam that only required roughly 70% could be difficult. I guess you have to experience it to know for yourself.
 
Yeah, the 3-option thing came in a few years back based on some really silly logic. I couldn’t and still can’t understand why they did it. No matter what a bunch of psychometricians say, it seems intuitive to most people that a multiple choice test with 4 options is harder to pass by luck than one with 3 options. I think the exam lost some credibility by that change and what it gained was really trivial (it’s easier to write questions as if that’s a huge concern - send me the a copy of the test, give me two days and I will come up with a plausible but incorrect answer for every question).
 
turbolt]
JoeyDVivre[/I said:
wrote: I’m not sure you do want the passing score released. If it’s a 58 or something, you could lose some credibility if people think that kind of score is just too low.
Possible, but I think that would just be more indicative of the difficulty of the test itself. If only about 40% of the people taking the exam can get the “58”, or other low passing score, then I would think that says it’s a pretty difficult exam and not necessarily lose cred.
 
JoeyDVivre wrote:Yeah, the 3-option thing came in a few years back based on some really silly logic. I couldn’t and still can’t understand why they did it. No matter what a bunch of psychometricians say, it seems intuitive to most people that a multiple choice test with 4 options is harder to pass by luck than one with 3 options. I think the exam lost some credibility by that change and what it gained was really trivial (it’s easier to write questions as if that’s a huge concern - send me the a copy of the test, give me two days and I will come up with a plausible but incorrect answer for every question).
Yeah, what was the basic logic? Are you familiar with the basis of why they went from 4 to 3?
 
thanks you SIR for the clarification! You right, I misunderstood the concept. It is a bit more reassuring the way you presented it. Hopefully I understood the material and exam questions better than the modified angoff!
 
I respectfully disagree that with 3 choices it is easier..yeah probability wise yes..But, if you look at the questions, you can see that with 4 choices, with some understanding of the question and reading, you will end up with looking at 2 choices to decide the right one, because with 4 choices one choice will have the kind of same wrong explanation as with the first one you eliminated, making it a two choice decision. So, my experience (limited compared to Joe’s for sure) is 3 and 4 is almost the same difficulty…
 
Oyster wrote:
Your explanation sounds like the unsubstantiated theory wherein someone claims that “if I suffered, then I want them to ensure that you suffer, too.” With that same notion, I (Level I Candidate) could also argue that the test was much simpler 10 years ago, when IFRS wasn’t around. I am sure the psychometric specialists took the whole GAAP vs. IFRS argument into account when going from four to three answer choices. The additional differences that Candidates have to commit to memory more than make up for the “lost fourth” choice.
BTW, having “just” three choices doesn’t make the test “simpler.” Also, do you have any data that shows that test lost its credibility following the change?
I guess I am being anal about this is because such developments (going from four to three choices, IFRS vs. GAAP, etc.) are bound to come into existence. If anything, I’d expect the current Charterholders, Candidates, and Members to embrace such changes, thereby, maintaining the value of the designation. The last thing I want is for the material and the exams to remain static for eternity. I could also argue that going from four to three choices saves time and energy - the question architects have to worry about three choices, not four. Between the additional material and the added productivity, I can see how three choices came into existence. Of course, this doesn’t justify/explain how the Institute markets such changes… maybe they can provide better explanations to convince the general audience of any impending radical changes.

First off, I didn’t suffer that much taking CFA exams. And I’ve spent way more time helping other people pass their CFA exams than I ever did passing mine, so the notion that I somehow want other people to suffer as I did is just totally off-base.
I do care about the credibility of the CFA credential. All I have is anecdotal data, but if you share with people that you might have to get a 58% on a multiple choice exam with only 3 choices people think that sounds pretty easy, even if that might not be the case. The not totally flawed logic goes…OK, you get 33% by chance alone then to pass you only need to know an additional 25% of the material so that means you really only need to get a 25% to pass this exam. In just about any body of knowledge getting the first 25% is pretty easy. That means that all a CFA charter shows is that someone learned the first 25% of material that is surely not the most difficult material in finance.
I hate that kind of argument and hate having to rebut it. Why does CFAI do that to all of us when they could just add a fourth multiple choice option and make the exam like SAT’s, GRE’s, GMAT’s, LSAT’s, Series [blah] exams, (and probably a bunch of other exams I have never looked at)? Saving effort in test construction doesn’t move me at all. Constructing this test is just not that hard and CFAI gets tons of money for it.
There is no way that moving from four choices to three “maintains the value of the designation”. There is just not one thinking person on the planet who thinks that makes the exam harder or more relevant. I completely support having an ever-changing curriculum that responds to new developments. You should know that adding things like IFRS vs GAAP also involves getting rid of things (e.g., pooled interest vs purchase accounting for mergers). I personally think that the CFA program needs to get with the times and put more stuff on the exams about credit derivatives and credit in general. You can reasonably be a CFA charterholder and know next to nothing about how CDS function for example when the CDS market is two or three times the size of the US GDP. That seems significantly more important to me than some of the FSA minutia.
 
If one incorrect answer choice is eliminated from each question and all else is equal, candidate scores will become more random. This is a completely logical conclusion.
 
I don’t think you just need to be sure about 25% questions as suggested by Oyster.
Suppose X is the percentage of questions you need to be sure about and 70% is the passing rate, we can develop the following equation
0.7 = X + (1-X)/3 [you have 1/3 chance only among the question you are not sure about]
0.7 = X + 0.333 – 0.333X
0.7 – 0.333 = 0.777X
0.367 = 0.777X
X = 0.367/0.777 = 47%
You need to be sure about roughly half of the questions.
However, if you increase the options the figure jumps to 60% and it makes the test difficult for sure.
There are many questions where you are trapped to select wrong answers and this biases the chance factor.
 
Back
Top