Friday, March 24, 2006

On how not to apply mathematical models

Over at Aetiology, there's a furious war going on in the comments between Tara and an AIDS-HIV denialist. One of the things that the denialist has referenced is an essay by Professor Rebecca Culshaw, a mathematician who has worked on mathematical models of AIDS and HIV transmission, and who has become an HIV denialist.

I'll leave the science debate over at Aetiology, where it belongs. But there's definitely a mathematical aspect to this. Professor Culshaw lends her authority as a mathematician to the HIV denialist folks. Does her math support what she's saying?

Alas, no.

Professor Culshaw is not a bad mathematician - quite the opposite. What I can read of her publications shows very solid mathematical work, done extremely well.

The problem is that when she tries to apply the mathematics to the science of epidemiology, she fails miserably, and the reason why is mathematical.

Let me first present an excerpt of her essay. (I suggest reading the thing in it's entirety; it's an interesting piece of writing, even if I think she's wrong.)

Over the past ten years, my attitude toward HIV and AIDS has undergone a dramatic shift. This shift was catalyzed by the work I did as a graduate student, analyzing mathematical models of HIV and the immune system. As a mathematician, I found virtually every model I studied to be unrealistic. The biological assumptions on which the models were based varied from author to author, and this made no sense to me. It was around this time, too, that I became increasingly perplexed by the stories I heard about long-term survivors. From my admittedly inexpert viewpoint, the major thing they all had in common – other than HIV – was that they lived extremely healthy lifestyles. Part of me was becoming suspicious that being HIV-positive didn’t necessarily mean you would ever get AIDS.

By a rather curious twist of fate, it was on my way to a conference to present the results of a model of HIV that I had proposed together with my advisor, that I came across an article by Dr. David Rasnick about AIDS and the corruption of modern science. As I sat on the airplane reading this story, in which he said "the more I examined HIV, the less it made sense that this largely inactive, barely detectable virus could cause such devastation," everything he wrote started making sense to me in a way that the currently accepted model did not. I didn’t have anywhere near all the information, but my instincts told me that what he said seemed to fit.

Over the past ten years, I nevertheless continued my research into mathematical models of HIV infection, all the while keeping an ear open for dissenting voices. By now, I have read hundreds of articles on HIV and AIDS, many from the dissident point of view but far, far more from that of the establishment, which unequivocally promotes the idea that HIV causes AIDS and that the case is closed. In that time, I even published four papers on HIV (from a modeling perspective). I justified my contributions to a theory I wasn’t convinced of by telling myself these were purely theoretical, mathematical constructs, never to be applied in the real world. I suppose, in some sense also, I wanted to keep an open mind.

Reading this, you would conclude that what she had studied in her research was mathematical models of HIV transmission between people. You'd be wrong. Her work is on the transmission of HIV between infected and non-infected cells in culture. For example, here's one abstract of a paper available on the web:

We consider a two-dimensional model of cell-to-cell spread of HIV-1 in tissue cultures, assuming that infection is spread directly from infected cells to healthy cells and neglecting the effects of free virus. The intracellular incubation period is modeled by a gamma distribution and the model is a system of two differential equations with distributed delay, which includes the differential equations model with a discrete delay and the ordinary differential equations model as special cases.We study the stability in all three types of models. It is shown that the ODE model is globally stable while both delay models exhibit Hopf bifurcations by using the (average) delay as a bifurcation parameter. The results indicate that, differing from the cell-to-free virus spread models, the cell-to-cell spread models can produce infective oscillations in typical tissue culture parameter regimes and the latently infected cells are instrumental in sustaining the infection. Our delayed cell-to-cell models may be applicable to study other types of viral infections such as human T-cell leukaemia virus type 1 (HTLV-1)

And there lies the problem. Mathematical models are highly dependent on the details of their formulation and their parameters. Her work on modeling viral behavior is not modeling viral behavior of HIV in a human host; it's modeling viral behavior in a cell culture. That's a useful thing to understand: it helps us understand how HIV attacks and spreads between cells. But it doesn't really tell us anything directly about how HIV should spread between cells in a dramatically different environment. Viruses don't behave the same in culture and in a complete host. (For example, see this paper, describing how differently Hepatitis-A behaves in culture from in a host.)

She's trying to take a mathematical model created to specifically model one environment, and to apply that model to a different environment, without modifying the model for the new environment.

You can't do that. It doesn't work. A good mathematical model of HIV infection in a human host is not the same as a good mathematical model of HIV infection in a cell culture. You can't use the math that you derived for describing a cell culture, and apply it to the new environment without modifying the model to account for the differences between the environments. That's just bad math. Professor Culshaw appears to be a good enough mathematician that she should know better.

• One statement of hers really caught my attention

"It was around this time, too, that I became increasingly perplexed by the stories I heard about long-term survivors. From my admittedly inexpert viewpoint, the major thing they all had in common – other than HIV – was that they lived extremely healthy lifestyles."

Did she have any information on people who lived healthy lifestyles but died anyway?

By  ma3rk, at 3:11 PM

• Isn't this more or less a simple case of physics envy? i.e. she is expecting a mathematical model in biology to be something more than an approximation of reality that is too complex to fully capture? Physics revolves around mathmatical models that more or less are reality. While biology by comparison will always be muddied up with intrinsic levels of uncertainty and statistical inference, which is what makes it a much more fertile ground for quacks like the AIDS denialists.

By  steve, at 3:22 PM

• From the Introduction of the paper you cited:

"There is precedent for studying in vitro cell-to-cell spread of HIV-1 (as well
as that of other viruses) since many features are easier to determine experimen-
tally in tissue cultures than in, for example, a more complex medium such as the
bloodstream."

It seems the authors were well aware of the distinction you pointed out.

It sounds from her essay like Dr. Culshaw's disquiet over HIV modelling came from seeing that there was no consensus about how the dynamics of HIV should be modelled, nor were any of the models well grounded in studying the virus' dynamics in vivo.

These may or may not be legitimate complaints (I have no idea), but in any case they hardly justify a leap to "HIV doesn't cause AIDS". In her essay she doesn't present much of a case for the latter, mainly she recycles the arguments of David Rasnick from his article in Spin Magazine.

By  Eric Wallace, at 1:08 AM

• Yes, yes, yes! Great article, thank you.

A model is a well-crafted lie designed to represent some facet of reality. No model ever adequately reflects the true complexity of interactions, for a variety of er, sometimes-random reasons.

That we can observe stuff and quantify our observations, then create formulae and models doesn't mean that we necessarily understand more than we did before, or that we understand it better (GIGO factor, blah-blah-blah).

Models always need to be tested with new, different sets of subjects. One of the intrinsic and important factors in a model is the environment. As someone who's studied behaviour, I can tell you that the data of a model tested in anything other than field conditions (a Skinner box or petri dish) will have a lot of noise in it for what it lacks.

This is the paradox of behaviour. Field conditions lack controlled conditions and are extremely hard to have control agents (including consistency over long periods of time, as one cannot establish trends in a single year). Lab conditions may have some factors controlled, but lack a number of obvious and not-so-obvious factors that affect living things. (You try getting into the Umwelt of a butterfly!)

The Harvard Law of Behavior says that under carefully controlled conditions, an organism will do whatever it damn well pleases. This is why we have to have so many subjects to have anything like statistical significance, especially when dealing with humans, because of their complexity.

Cells also have complex behaviour, as pointed out previously in the in vivo versus in vitro testing. Once again, our conclusions are limited by what is present and is NOT present in the given arena.

andrea

By  andrea, at 6:27 PM

• Steve said "Physics revolves around mathmatical models that more or less are reality".
Gack! even Quantum Field Theories as mostly used are perturbation models, that is, an approximations, and we can only calculate a few terms of expansion, and they only work at low energy. We have to go to lattices or some other definitely false, but useful model for more. Plus all the sold state physics folks are laughing at thinking their models are very close to reality. Sorry, I'm just saying that even 'hard" physics models are definitely not simple, and you cannot do what the article discusses there either.

By  Markk, at 4:57 PM

• Markk said: "Steve said "Physics revolves around mathmatical models that more or less are reality".
Gack! even Quantum Field Theories as mostly used are perturbation models, that is, an approximations, and we can only calculate a few terms of expansion, and they only work at low energy."

Well at least the math is real enough in physics that you can build a bridge with it. Thats more than you can say for biology or economics.

By  steve, at 5:33 PM

• That's true - biology is of minor use in building a bridge. Economics is slightly more useful, in actually getting it done.

Steve, I don't mean to be personal, or hurtful, or cruel, but it's 2006. After years of Usenet and the blogosphere, physics bragging just isn't endearing anymore.

By  Barry, at 7:05 PM

• Barry said... Steve, I don't mean to be personal, or hurtful, or cruel, but it's 2006. After years of Usenet and the blogosphere, physics bragging just isn't endearing anymore.

Don't give yourself too much power there Barry. If you think discussing how mathematics has a precision and relevance in physics that is unmatched by other sciences like biology, or social disciplines like economics is passe then you are entitled to your opinion. My background is in finance and I see the results of "physics envy" all too much.

By  steve, at 7:33 PM

• Dr Culshaw makes herself a fraction more clear here Why I Quit HIV: The Aftermath

To clarify an issue that has caused some confusion, it was not the mathematical models themselves that caused me to doubt HIV, but rather the scientific literature on which the models are based.

So she is not really using mathematical arguments to argue against the causal role of HIV in AIDS. She talks about molecular biology and antibody tests etc where she has no expertise. The information she uses in these arguments appears to have been gleaned from reading "dissident" websites.

ie: There is good reason to believe the antibody tests are flawed as well. The two types of tests routinely used are the ELISA and the Western Blot (WB). The current testing protocol is to "verify" a positive ELISA with the "more specific" WB (which has actually been banned from diagnostic use in the UK because it is so unreliable).

The western blot test has not been banned in the UK. This is just a "factoid" that has been circulating amongst "dissidents".

By  Chris Noble, at 7:08 AM

• "Isn't this more or less a simple case of physics envy?"

Steve, you're comparing apples to oranges. Physics and biology often ask different kinds of questions. The study of variation is a central theme in biology. Other sciences often ignore variation as "noise". Obvious, trying to model "noise" is going to make models more complex.

I think a good physical analog of many biology studies would be to ask a physicist to take a single atom or particle and describe its movements for the last million years.

Of course, as markk pointed out, physical modeling can get hairy when they do look at the "noise".

• Yes I know all this. My point was that perhaps Dr. Culshaw could not make these distinctions - expecting a biological model of an extremely complex phenomena to work as well as, say, Kepler's Laws. However, after reading her Lew Rockwell pieces, ISTM that she is simply rehashing the same old tired anti-HIV arguments

By  steve, at 2:48 PM