What good are clinical trials?

[Gavjenks]

In general scientific research, we use a confidence interval of 95%, that is, we call a relationship "significant" if we are 95% sure that the data aren't just a result of noise.

My problem I would like to talk about is that when it comes to medical outcomes, 95% is often grossly inappropriate as a responsible confidence interval, and yet clinical trials are designed around it still. Rendering them in most cases almost completely useless for actually assisting me in making good medical decisions.

Example #1: Athlete's foot cream
Athlete's foot is annoying, but it is a pretty minorly bad thing.
Death is a really really majorly bad thing, however. Even if there's a very tiny chance of it happening due to using the cream, I'm not going to want to use the cream for its minor benefit, right? And by "tiny chance" I mean that I expect there to be MUCH MORE confident than a p < 0.05 level that the athlete's foot cream is not going to cause, say, kidney failure. If there's even a 1/1000 chance that there exists a relationship between athlete's foot cream and sudden death, given the data, I probably would decline to use that medication.
But the problem is, nobody has any way whatsoever of guaranteeing me that degree of certainty. Because they don't run tens of thousands of people in foot cream clinical trials. They maybe run a few hundred usually. Enough to satisfy a power analysis that operates on a 0.05 level ONLY.

(and in case you think this is a frivolous example, ketoconazole fairly common antifungal drugs were announced last year to be pretty solidly linked to total liver and adrenal failure, and possibly birth defects, and only to be used orally as a last resort in severe deadly fungal infections, long after being released and used widely. And I didn't even KNOW that when I started writing this example half an hour ago!)

Example #2: Flu vaccines
Some numbers (all from the CDC) for people under 65 years of age (most of these numbers aren't known for people over 65 is why I restrict it):
% of people who get the flu when unvaccinated = roughly 15%
% of THOSE people who would have gotten the flu but don't, due to being vaccinated = roughly 60% in average seasons (i.e. in simpler terms ~40% of people it "doesn't take")
So far = roughly 9% likelihood that a given vaccine actually ends up saving you from anything
Then, the likelihood of a person with symptoms being hospitalized = roughly 0.1% (although this is only ballpark guessed by experts)
So far = roughly 0.009% likelihood that a given vaccine saves you a trip to the hospital.
Then, percent of those people who die = nobody knows. The CDC doesn't even know how many people die of the flu, not even within an order of magnitude of precision. But even the most liberal estimates would be something like 1% of hospitalized people.
So far = at most, about 0.00009% likelihood that a given vaccine will save your life. I.e. something like 1/1,000,000, (a number with truly horrible precision to it, though).

Okay, so if I want to decide whether to take a vaccine or not, I also want to know how likely a vaccine is to kill me itself. Is it higher or lower than 1/1,000,000?

Well, thanks to clinical trials for flu vaccines only involving a few hundred people, the answer is: nobody has a remotely vague idea. All they can tell you is that it is probably less than about 1/100 to kill you (likely much much lower, but that's what the data can GUARANTEE). (Note: this is all for death only, but the numbers are similarly fuzzy / almost completely unknown for any other severe complication or side effect you could name)

So how do I make an educated decision whether it helps me more than it hurts me? I can't. I have to flip a coin. So what good is the clinical trial doing me? And why am I being told it's a great thing?



And so on and so forth for pretty much every MINOR drug or medical procedure out there. Major procedures like heart transplants are actually much easier to make educated decisions on, because the deadly outcomes either way are so much more common, that we have plenty of data to get the precision we need. It's the minor ones that nobody has any clue about when you get down to it.

So is it even ethical to "prescribe" such minor things, or promote them? Versus, for example, simply saying "Here are our best known odds. You have permission to get this done if you want, but you have to follow your own intuition." That would be the HONEST way...

 

[Designer]

I just finished a book an this.

Tainted Truth - The Manipulation of fact in America by Cynthia Crossen

The author analyzes surveys, studies, and clinical trials, and points out that we consumers and our doctors need to understand how they are produce, who paid for them and how they are affected by bias of the researchers and the drug companies (who often pay for the research).

Another thing we need to understand is the mathematics of statistics, in order to have a better grasp on the risks involved.

 

[mmaria]

I was thinking about vaccines just yesterday.

Too bad my English is not that good to say more about the subject but I'm interested to see what others think.

My thoughts are similar to Cynthia's (mentioned here in a nutshell by Designer, didn't read the book or anything) and being a highly sensitive person ( I'm allergic to lots of chemical ingredients in cremes, gels, medical products and else f.e.) colored my opinion on this subject

 

[runnah]

Well take your flu vaccine example. You can't just say 1 in every 100,000 dies of the flu. If you use a sample group of healthy young people the number would be closer to 1 in a million, but if you use a sample group of elderly people you'd get a much higher ratio.

 

[runnah]

I was thinking about vaccines just yesterday.

Too bad my English is not that good to say more about the subject but I'm interested to see what others think.

My thoughts are similar to Cynthia's (mentioned here in a nutshell by Designer, didn't read the book or anything) and being a highly sensitive person ( I'm allergic to lots of chemical ingredients in cremes, gels, medical products and else f.e.) colored my opinion on this subject

The risks greatly outweigh the harm on the vaccine issue. Google images of smallpox, measles etc... it will make you thankful for modern science.

 

[kathyt]

Here is my thought on vaccines. It is real quick. If vaccines did not exist, then most of us would not be here right now reading my comment on TPF this very second. The end.

 

[runnah]

Here are is thought on vaccines. It is real quick. If vaccines did not exist, then most of us would not be here right now reading my comment on TPF this very second. The end.

Well the last hot blonde that told me about vaccines said they were bad so I don't know what to think.

 

[kathyt]

Here are is thought on vaccines. It is real quick. If vaccines did not exist, then most of us would not be here right now reading my comment on TPF this very second. The end.

Well the last hot blonde that told me about vaccines said they were bad so I don't know what to think.

Let ME give you your vaccines. Bend over.

 

[mmaria]

....and the thread goes in another direction...

 

[runnah]

....and the thread goes in another direction...

I was being serious until you and Kathy sidetracked me.

Anyways. Statistics are way more complicated than people assume. Like I said before saying "1 in a 100" doesn't really tell people very much. You have to ask questions about sample size, sample make up, testing procedures such as were they double blind, triple blind, and were the final results peer reviewed. More often than not clinical studies are BS and are used more as marketing speak than actual scientific study.

 

[mmaria]

....and the thread goes in another direction...

I was being serious until you and Kathy sidetracked me.

What have I done!?

Anyways. I don't believe in statistics that is presented to us, consumers. I'll almost always and no matter in what field that research took place, assume worse regarding the data that's been showed to us. Medicine and medical products are almost the same as any other products there is when it comes to marketing and selling.

 

[TreeofLifeStairs]

I agree that there are many clinical trials that are done with bias, but a double blind peer reviewed study does carry weight in my mind.

Another thought is weighing the side effects. I see commercials for medications that will treat something trivial (your athletes foot example) and then they will rattle off the possible side effects at the end and the side effects will often be far worse than the original ailment. The flip side is the person that is dieing of cancer and there is a drug that has been shown to have promise but the patient cannot take it because it hasn't been fully tested. They care very little about the side effects at that point. They just want the chance that they'll live.

Maybe this is the point of the OP.

 

[Gavjenks]

I agree that there are many clinical trials that are done with bias, but a double blind peer reviewed study does carry weight in my mind.

My argument is not about bias. For purposes of this thread, I am assuming that all clinical trials are done by perfect science robots who never make methodological errors and have zero political or personal opinions, etc. etc. My argument is purely based on the extremely low numbers of participants compared to the number you would need to actually conclude the information that any normal person would be interested in / find useful for real decisions.

In short, for rare events like death or mutilation (either vaccines causing them OR vaccines preventing them), the data is so sparse that it might as well be a couple of anecdotes from your neighbors. Almost literally.

Another thought is weighing the side effects. I see commercials for medications that will treat something trivial (your athletes foot example) and then they will rattle off the possible side effects at the end and the side effects will often be far worse than the original ailment. The flip side is the person that is dieing of cancer and there is a drug that has been shown to have promise but the patient cannot take it because it hasn't been fully tested. They care very little about the side effects at that point. They just want the chance that they'll live.

Maybe this is the point of the OP.

I'm not sure I have much of a problem with people weighing death more heavily than side effects. There's not necessarily a "right" answer to how you relatively weigh one versus another versus quality versus time, etc.

I just would want people to have more accurate numbers available to them to make their own decisions, that's all.

Well take your flu vaccine example. You can't just say 1 in every 100,000 dies of the flu. If you use a sample group of healthy young people the number would be closer to 1 in a million, but if you use a sample group of elderly people you'd get a much higher ratio.

1) Nobody knows how effective the flu vaccine is on elderly people reliably. It seems to vary wildly from particular strain to strain. All they really seem to know is that in general it is some amount less effective in the elderly on average.

2) It is a mistake to just assume that the elderly suffer more from flu by default. Swine flu in particular appears to be LESS of a risk for seniors than for middle aged adults, and the few people who absolutely died of that this year were disproportionately not senior citizens.

3) Seniors are prone to already having various complications when they get the flu, so it's really hard to even know though whether they die of the flu. If some dude is already dying of AIDS for instance, and gets the flu on a compromised immune system, and then also gets pneumonia, and then dies, how do you report that? The answer is three different people report it 3 different ways and it's a huge mess.



Bottom line: almost every step of the way, we don't have the data to conclude anything particularly strong or useful right now. For seniors we have even less data, because well there are fewer of them, and we also run even smaller studies on them, and they seem to be more variable.

Here is my thought on vaccines. It is real quick. If vaccines did not exist, then most of us would not be here right now reading my comment on TPF this very second. The end.

The public opinion on, for example, measles immediately prior to the vaccine being introduced was NOT abject terror. People were not dropping dead in droves in the street. It was more along the lines of a typical sitcom plot device to keep a kid home from school.

This is because by the 1960's, roughly 99.9% of the death rate of measles had already been eradicated without the usage of any vaccines. Nor is this an atypical case. Presumably the vast majority of the elimination of endemic measles was because of:
-Soap and water treatment improvements
-Fever reducers / NSAIDs
-Better life support and symptom care in general
-Better hospital access in general (roads and frequency)
-Better hospital conditions and antibiotics to fight secondary infections
etc.

Vaccines clearly helped spike downward deaths after introduction, and they do certainly work, but they may well have been redundant, and didn't necessarily save that many people. By no means is it reasonable to say "most of us wouldn't be here." Unless maybe you're thinking of 1960 BC, not 1960 AD...



Same goes for pertussis



and scarlet fever, which is especially interesting since we now have zero deaths most years from it, and a vaccine was in fact never nationally used. (in wales, England, or the US)

 

[kathyt]

Every single drug I give has potential side effects listed that can be as long as my arm. Usually the benefits of the drug are greater than the "potential" side effects for a doctor to prescribe the medication in the first place. The bold lettered side effects are the ones that the greatest number of patients generally report. These are the ones we care the most about, and monitor the closest. As a consumer, I would look at the pros and the cons of the medication to determine if it is truly worth the risk of the potential side effects. Just like anything else, one must do their research and determine which pathway is right for them. Sometimes medications are not always the best option, and sometimes it is the only option.

 

[kathyt]

I agree that there are many clinical trials that are done with bias, but a double blind peer reviewed study does carry weight in my mind.

My argument is not about bias. For purposes of this thread, I am assuming that all clinical trials are done by perfect science robots who never make methodological errors and have zero political or personal opinions, etc. etc. My argument is purely based on the extremely low numbers of participants compared to the number you would need to actually conclude the information that any normal person would be interested in / find useful for real decisions.

In short, for rare events like death or mutilation (either vaccines causing them OR vaccines preventing them), the data is so sparse that it might as well be a couple of anecdotes from your neighbors. Almost literally.

Another thought is weighing the side effects. I see commercials for medications that will treat something trivial (your athletes foot example) and then they will rattle off the possible side effects at the end and the side effects will often be far worse than the original ailment. The flip side is the person that is dieing of cancer and there is a drug that has been shown to have promise but the patient cannot take it because it hasn't been fully tested. They care very little about the side effects at that point. They just want the chance that they'll live.

Maybe this is the point of the OP.

I'm not sure I have much of a problem with people weighing death more heavily than side effects. There's not necessarily a "right" answer to how you relatively weigh one versus another versus quality versus time, etc.

I just would want people to have more accurate numbers available to them to make their own decisions, that's all.

Well take your flu vaccine example. You can't just say 1 in every 100,000 dies of the flu. If you use a sample group of healthy young people the number would be closer to 1 in a million, but if you use a sample group of elderly people you'd get a much higher ratio.

1) Nobody knows how effective the flu vaccine is on elderly people reliably. It seems to vary wildly from particular strain to strain. All they really seem to know is that in general it is some amount less effective in the elderly on average.

2) It is a mistake to just assume that the elderly suffer more from flu by default. Swine flu in particular appears to be LESS of a risk for seniors than for middle aged adults, and the few people who absolutely died of that this year were disproportionately not senior citizens.

3) Seniors are prone to already having various complications when they get the flu, so it's really hard to even know though whether they die of the flu. If some dude is already dying of AIDS for instance, and gets the flu on a compromised immune system, and then also gets pneumonia, and then dies, how do you report that? The answer is three different people report it 3 different ways and it's a huge mess.



Bottom line: almost every step of the way, we don't have the data to conclude anything particularly strong or useful right now. For seniors we have even less data, because well there are fewer of them, and we also run even smaller studies on them, and they seem to be more variable.

Here is my thought on vaccines. It is real quick. If vaccines did not exist, then most of us would not be here right now reading my comment on TPF this very second. The end.

The public opinion on, for example, measles immediately prior to the vaccine being introduced was NOT abject terror. People were not dropping dead in droves in the street. It was more along the lines of a typical sitcom plot device to keep a kid home from school.

This is because by the 1960's, roughly 99.9% of the death rate of measles had already been eradicated without the usage of any vaccines. Nor is this an atypical case. Presumably the vast majority of the elimination of endemic measles was because of:
-Soap and water treatment improvements
-Fever reducers / NSAIDs
-Better life support and symptom care in general
-Better hospital access in general (roads and frequency)
etc.

Vaccines clearly helped spike downward deaths after introduction, and they do certainly work, but they may well have been redundant, and didn't necessarily save that many people. By no means is it reasonable to say "most of us wouldn't be here." Unless maybe you're thinking of 1960 BC, not 1960 AD...

View attachment 69425

Same goes for pertussis

View attachment 69426

and scarlet fever, which is especially interesting since we now have zero deaths most years from it, and a vaccine was in fact never nationally used. (in wales, England, or the US)

View attachment 69427

Okay. Don't get vaccinated then. That is your choice, and by all means please use your data as a reference. I was just giving my opinion, and I am sure you will drag this out till kingdom come. I don't have time for that.