Posts Tagged ‘Examinations’
USMLE – Designing The Ultimate Questions
Shot courtesy crystaljingsr @ Flickr (Creative Commons Attribution, Non-Commercial License)
There are strategies that examiners can employ to frame questions that are designed to stump you on an exam such as the USMLE. Many of these strategies are listed out in the Kaplan Qbook and I’m sure this stuff will be familiar to many. My favorite techniques are the ‘multi-step’ and the ‘bait-and-switch’.
The Multi-Step
Drawing on principles of probability theory, examiners will often frame questions that require you to know multiple facts and concepts to get the answer right. As a crude example:
“This inherited disease exclusive to females is associated with acquired microcephaly and the medical management includes __________________.”
Such a question would be re-framed as a clinical scenario (an outpatient visit) with other relevant clinical data such as a pedigree chart. To get the answer right, you would need:
- Knowledge of how to interpret pedigree charts and identify that the disease manifests exclusively in females.
- Knowledge of Mendelian inheritance patterns of genetic diseases.
- Knowledge of conditions that might be associated with acquired microcephaly.
- Knowledge of medical management options for such patients.
Now taken individually, each of these steps – 1, 2, 3 and 4 – has a probability of 50% that you could get it right purely by random guessing. Combined together however, which is what is necessary to get the answer, the probability would be 50% * 50% * 50% * 50% = 6.25% [combined probability of independent events]. So now you know why they actually prefer multi-step questions over one or two-liners! 🙂 Notice that this doesn’t necessarily have anything to do with testing your intelligence as some might think. It’s just being able to recollect hard facts and then being able to put them together. They aren’t asking you to prove a math theorem or calculate the trajectory of a space satellite 😛 !
The Bait-and-Switch
Another strategy is to riddle the question with chock-full of irrelevant data. You could have paragraph after paragraph describing demographic characteristics, anthropometric data, and ‘bait’ data that’s planted there to persuade you to think along certain lines and as you grind yourself to ponder over these things you are suddenly presented with an entirely unrelated sentence at the very end, asking a completely unrelated question! Imagine being presented with the multi-step question above with one added fly in the ointment. As you finally finish the half-page length question, it ends with ‘<insert-similar-disease> is associated with the loss of this enzyme and/or body part: _______________’. Very tricky! Questions like these give flashbacks and dejavu of days from 2nd year med school, when that patient with a neck lump begins by giving you his demographic and occupational history. As an inexperienced med student you immediately begin thinking: ‘hmmm..okay, could the lump be related to his occupation? …hmm…’. But wait! You haven’t even finished the physical exam yet, let alone the investigations. As medics progress along their careers they tend to phase out this kind of analysis in favor of more refined ‘heuristics’ as Harrison’s puts it. A senior medic will often wait to formulate opinions until the investigations are done and will focus on triaging problems and asking if management options are going to change them. The keyword here is ‘triage’. Just as a patient’s clinical information in a real office visit is filled with much irrelevant data, so too are many USMLE questions. That’s not to say that demographic data, etc. are irrelevant under all conditions. Certainly, an occupational history of being employed at an asbestos factory would be relevant in a case that looks like a respiratory disorder. If the case looks like a respiratory disorder, but the question mentions an occupational history of being employed as an office clerk, then this is less likely to be relevant to the case. Similarly if it’s a case that overwhelmingly looks like an acute abdomen, then a stray symptom of foot pain is less likely to be relevant. Get my point? That is why many recommend reading the last sentence or two of a USMLE question before reading the entire thing. It helps you establish what exactly is the main problem that needs to be addressed.
Hope readers have found the above discussion interesting :). Adios for now!
—
Copyright © Firas MR. All rights reserved.
Does Changing Your Anwer In The Exam Help?
The Monty Hall Paradox
One of the 3 doors hides a car. The other two hide a goat each. In search of a new car, the player picks a door, say 1. The game host then opens one of the other doors, say 3, to reveal a goat and offers to let the player pick door 2 instead of door 1. Is there an advantage if the the player decides to switch? (Courtesy: Wikipedia)
Hola amigos! Yes, I’m back! It’s been eons and I’m sure many of you may have been wondering why I was MIA. Let’s just say it was academia as usual.
This post is unique as it’s probably the first where I’ve actually learned something from contributors and feedback. A very critical audience and pure awesome discussion. The main thrust was going to be an analysis of the question, “If you had to pick an answer in an MCQ randomly, does changing your answer alter the probabilities to success?” and it was my hope to use decision trees to attack the question. I first learned about decision trees and decision analysis in Dr. Harvey Motulsky’s great book, “Intuitive Biostatistics“. I do highly recommend his book. As I pondered over the question, I drew a decision tree that I extrapolated from his book. Thanks to initial feedback from BrownSandokan (my venerable computer scientist friend from yore :P) and Dr. Motulsky himself, who was so kind as to write back to just a random reader, it turned out that my diagram was wrong and so was the original analysis. The problem with the original tree (that I’m going to maintain for other readers to see and reflect on here) was that the tree in the book is specifically for a math (or rather logic) problem called the Monty Hall Paradox. You can read more about it here. As you can see, the Monty Hall Paradox is a special kind of unequal conditional probability problem, in which knowing something for sure, influences the probabilities of your guesstimates. It’s a very interesting problem, and has bewildered thousands of people, me included. When it was originally circulated in a popular magazine, “nearly 1000 PhDs” (cf. Wikipedia) wrote back to say that the solution put forth was wrong, prompting numerous psychoanalytical studies to understand human behavior. A decision tree for such a problem is conceptually different from a decision tree for our question and so my original analysis was incorrect.
So what the heck are decision trees anyway? They are basically conceptual tools that help you make the right decisions given a couple of known probabilities. You draw a line to represent a decision, and explicitly label it with a corresponding probability. To find the final probability for a number of decisions (or lines) in sequence, you multiply or add their individual probabilities. It takes skill and a critical mind to build a correct tree, as I learned. But once you have a tree in front of you, its easier to see the whole picture.
Let’s just ignore decision trees completely for the moment and think in the usual sense. How good an idea is it to change an answer on an MCQ exam such as the USMLE? The Kaplan lecture notes will tell you that your chances of being correct are better off if you don’t. Let’s analyze this. If every question has 1 correct option and 4 incorrect options (the total number of options being 5), then any single try on a random choice gives you a probability of 20% for the correct choice and 80% for the incorrect choice. The odds are higher that on any given attempt, you’ll get the answer wrong. If your choice was correct the first time, it still doesn’t change these basic odds. You are still likely to pick the incorrect choice 80% of the time. Borrowing from the concept of “regression towards the mean” (repeated measurements of something, yield values closer to said thing’s mean), we can apply the same reasoning to this problem. Since the outcomes in question are categorical (binomial to be exact), the measure of central tendency used is the Mode (defined as the most commonly or frequently occurring thing in a series). In a categorical series – cat, dog, dog, dog, cat – the mode is ‘dog’. Since the Mode in this case happens to be the category “incorrect”, if you pick a random answer and repeat this multiple times, you are more likely to pick an incorrect answer! See, it all make sense 🙂 ! It’s not voodoo after all 😀 !
Coming back to decision analysis, just as there’s a way to prove the solution to the Monty Hall Paradox using decision trees, there’s also a way to prove our point on the MCQ problem using decision trees. While I study to polish my understanding of decision trees, building them for either of these problems will be a work in progress. And when I’ve figured it all out, I’ll put them up here. A decision tree for the Monty Hall Paradox can be accessed here.
To end this post, I’m going to complicate our main question a little bit and leave it out in the void. What if on your initial attempt you have no idea which of the answers is correct or incorrect but on your second attempt, your mind suddenly focuses on a structure flaw in one or more of the options? Assuming that an option with a structure flaw can’t be correct, wouldn’t this be akin to Monty showing the goat? One possible structure flaw, could be an option that doesn’t make grammatical sense when combined with the stem of the question. Does that mean you should switch? Leave your comments below!
Hope you’ve found this post interesting. Adios for now!
Copyright © Firas MR. All rights reserved.
Readability grades for this post:
Flesch reading ease score: 72.4
Automated readability index: 7.8
Flesch-Kincaid grade level: 7.3
Coleman-Liau index: 8.5
Gunning fog index: 11.4
SMOG index: 10.7
Intuitive Biostatistics, by Harvey Motulsky
Powered by ScribeFire.
How Examinations And Diagnostic Tests Are Similar
“You are more than a score”, or so the saying goes. But how much of that comes out as an emotional outburst as opposed to objective and rational thinking? Let’s try to see why the above is totally true, scientifically speaking.
In medicine, we’ve learned a lot about diagnostic tests, right? In fact everything investigative in nature can be considered a diagnostic test. Be it a screening exam for cervical cancer, that blood test for glucose, an X-ray for a broken arm, or your palpating hand feeling for that enlarged liver. Heck, even an entire research study could be considered a diagnostic test. The ‘null hypothesis’ technique often used in analytical research studies is nothing more than a diagnostic test of sorts.
When considering the dynamics of a diagnostic test, a fundamental underlying principle is that we separate what is observed via the test from the actual truth. In the case of tangible phenomena like death, disease and disability, it is quite easy to distinguish the actual truth from what the test predicts. Because of this, you have terms like ‘false positives’, ‘false negatives’ and the like. A pregnancy test for example could be positive, but you could easily compare that prediction to the actual outcome (pregnancy vs. non-pregnancy) and say that this particular test has got such and such false positive rates. More or less, all tests have the following attributes in this regard:-
- Sensitivity
- Specificity
- Positive Predictive Value
- Negative Predictive Value
- Validity/Accuracy
- Reliability/Precision
We ought to think about examinations such as the USMLE, etc. in this manner as well. Why? Well, because they are investigations too! Think of them as X-rays to diagnose your intelligence or whatever, if that metaphor helps. And as a consequence, notions about false positives, false negatives and all of the other things on that list also apply to them. Being the abstract intangible thing intelligence is, it is impossible to know its true value. And because there’s no way to compare prediction versus truth, it is impossible to say for sure what the false positive or negative rates (or any item on that list) for an exam are. And that’s why, ‘you are more than a score’ 
! Statistically speaking, examinations are just so lame 
!
Do send in your comments!
—
Readability grades for this post:
Kincaid: 9.3
ARI: 9.8
Coleman-Liau: 12.9
Flesch Index: 58.0/100
Fog Index: 13.2
Lix: 42.9 = school year 7
SMOG-Grading: 12.0
—
Powered by BlogJet and Ubuntu Linux 7.04
–
Copyright © 2006 – 2008 Firas MR. All rights reserved.
When Treatments Kill
It truly amazes me how soon writers block can set in. As you can probably see, my enterprise hasn’t exactly seen a lot of throughput. LOL 😀 . Okay enough of microprocessor terminology and let’s get on to something really cool 🙂 .
Doctors are quite peculiar in the fact that they strive to kill their own profession, at least indirectly. You could say the same for police officers, firefighters and their like. If there weren’t disease, crime or fire incidents, each of these groups would have achieved their missions and would have wiped out the very purpose of their existence. In our never ending struggle with disease, we are prone to treating people. EBM has taught us that that might not necessarily be a good thing. End-of-Life care and palliative medicine have totally transformed our thinking about the very definition of the word treatment. Treatments may very well be characterized by lack of interventions. For instance, CPR (Cardiopulmonary Resuscitation) no longer is viewed as something absolutely necessary. Through EBM, we’ve come to realize that the overall success rate of CPR is a meager ~15%.¹ To many of us that sounds surprizing, doesn’t it? We also now have clearer statistical evidence on which patient groups have better vs. worse success rates. Given these statistical insights, it is perfectly reasonable in certain instances for people to be given the choice of a DNR (Do Not Resuscitate) order in their treatment plans. The risks of broken ribs, fat embolism and other complications of CPR outweigh the benefits in such cases. Similarly, maintaining full nutrition may not be that good an idea, again if it’s not contrary to the specifics of a given case (eg. the patient’s choice, etc.) . It has been found that the mild ketosis during the starved state can very well induce a sense of comfort in painful end-of-life conditions.¹ So if the patient requests not to be tube-fed, you’re not only obligated to respect this request from an ethical standpoint but from a scientific perspective as well. The list of interventions that could be withdrawn in palliative care goes on, but I don’t really want to focus on that here. In most of these situations, the primary reason for not intervening isn’t because intervening is likely to accelerate death.
Nor is this post’s intention to bring to your attention, side-effects of medications/interventions that might eventually kill. No, we are talking about entirely different beasts here.
There are rare cases when the situation at hand isn’t palliative in nature or one that has a side-effect angle to it. A couple of unique instances actually wherein, the act of intervening itself will in fact worsen a patient’s condition and likely result in death. These go in line with the medical myths we discussed in my last post. Notice how these beasts baffle your instincts. So without further ado, some of these include²:
- Infantile Botulism – an infectious process – yet antibiotics worsen the case and are contraindicated.
- Hemolytic Uremic Syndrome due to Shigella – an infectious process – yet again, antibiotics worsen symptoms and are contraindicated.
- Thrombotic Thrombocytopenic Purpura – a situation where there’s a platelet decline – yet platelet transfusions are contraindicated.
Note that these aren’t the only ones, so do watch out for others! It’ll do you and the patient a lot good!
—
References:-
- Current Medical Diagnosis & Treatment, Chapter. Palliative Care and Pain Management by Michael W. Rabow, MD; Steven Z. Pantilat, MD
- Kaplan Medical, Lecture Notes for the USMLE Step 1
–
Copyright © 2006 – 2008 Firas MR. All rights reserved.
My Dominant Hemisphere 