Risk - Part I

Risk, one of the central issues in research ethics, is one of the most misunderstood and misapplied terms in the entire field of human research protections. This will be a two-part discussion of risk. This first part will deal with the misunderstanding with regard to the evaluation of risk in research. The second part will misunderstandings about the role of risk in human research protections.

Most researchers and most IRB members confuse risk and harm. Risk is not harm, risk is the possibility of harm. The evaluation of risk requires evaluation of both the magnitude and the likelihood of possible harm. To quote the Belmont Report, “The term "risk" refers to a possibility that harm may occur. However, when expressions such as "small risk" or "high risk" are used, they usually refer (often ambiguously) both to the chance (probability) of experiencing a harm and the severity (magnitude) of the envisioned harm.” The federal regulations do not define risk; however, in defining “minimal risk” the regulations state, “Minimal risk means that the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.”

Most protocols merely list the possible harms that might occur in the research. For example, a research protocol might list hair loss as a risk. This tells us nothing about the risks of this research. In order to evaluate the risk of hair loss, we need to know the severity of the hair loss and the likelihood of the hair loss occurring. A remote chance of slight hair loss is very different from the likely loss of all of one’s hair. IRBs must determine whether the risks are reasonable in relation to the benefits of the research. Most people would consider a high likelihood of total hair loss a reasonable risk in an oncology study but not in an acne study.

In addition to IRBs needing this information, subjects need this information to decide whether to participate in the research. A remote chance of slight hair loss may be a minor risk to most people, but to a teenage girl, the research would probably have to have significant benefit to justify that risk. The subjects must be provided information on both the magnitude and the likelihood of the possible harms in order to give true informed consent. This information must be provided to the subjects in language that they can understand and evaluate in terms of their own lives and values. As with the protocols, most consent forms merely list possible harms without provide real information about the risks of research.

This concern is not limited to biomedical research. The same requirement applies to social/behavioral research. Much social/behavioral research is “minimal risk,” but to make that determination, the IRB must examine both the magnitude of the possible harm and the likelihood that the harm will occur. Consider, for example, a survey on AIDS and substance abuse. While the magnitude of the possible harm if the information was inappropriately divulged is great, in order to evaluate the risk the IRB must also evaluate the likelihood of that occurring. If the confidentiality procedures were sufficient to ensure that there is an extremely small likelihood of the information being divulged, then the research might be considered a minimal risk study.

So, IRBs need to have information about both the magnitude of the possible harm and the likelihood of the harm occurring in order to determine if the risks are reasonable in relation to the benefits. If the IRB does not have enough information on both of these, it cannot approve the research.

Answering Questions

Before I get into the topic of this blog, I want to apologize, once again, for my sporadic posting. Our consulting firm (HRP Consulting Group) has been undergoing considerable growth and change over the past year and I have not had the time to devote to keeping up with blogging. I want to assure any readers that I have left that I am now committed to maintaining this blog. In fact, I have asked my colleagues in the firm to nag me on a regular basis about this! In this blog I will continue to share my thoughts and comments on the HRPP field, but, from time to time, I may turn this space over to my colleagues (Cheryl Savini and Nick Slack) when they have interesting thoughts they would like to share.

And now for my topic of the day – answering questions. I just got back from a very exciting PRIM&R meeting. While at the meeting, I conducted a few workshops where there were lots of questions asked and I was asked lots of questions in the hallways and at our booth. This caused me to think about what it means to answer questions in our field, and I thought I’d share my thoughts about this.

First of all, I do not believe that there are any “right” answers to questions in our field. Nothing in human research protections is black or white; it is all gray. Everything is open to interpretation, even the regulations. And the only wrong answers are those that are clearly in violation of the regulations. So, when I answer a question, my answer is my opinion, based my years of experience and my interpretation of the regulations and best practice. For example, I was asked several questions about “quorum” at IRB meetings. The regulations at 45 CFR 46.108(b)/21CFR 56.108(c) say: “Except when an expedited review procedure is used, [each IRB shall] review proposed research at convened meetings at which a majority of the members of the IRB are present, including at least one member whose primary concerns are in nonscientific areas. In order for the research to be approved, it shall receive the approval of a majority of those members present at the meeting.” This may seem straight forward, but there is plenty that is open to interpretation. What is a majority? What is a nonscientist?

My answer will reflect my opinion on how best to meet the spirit of the regulations and implement practical procedures for ensuring the maintenance of a quorum. So, I would suggest a “simple majority” which is more than half of the members (for example, if there are 9 members 5 would be a majority). A nonscientist should be someone who clearly has no scientific training or experience; if there is any doubt, then that person is not a nonscientist. And, institutions should not rely solely on the unaffiliated member to be the nonscientist since it is sometimes hard to get the unaffiliated member to a meeting. Could someone else come up with a different answer? Certainly! Is my answer the “right” one? There is no one “right” answer but my answer will not be a “wrong” answer. If I give you an answer to a question, you should see it as my expert opinion and weigh how well it works for you and your circumstances.

Second, there is almost always more than one way to accomplish regulatory and ethical goals in our field. What works at one institution may not work at another. People should get as many opinions as possible on a given issue and choose the one that works best for them. For example, in a study on heroin use, subjects may reveal information about other people (dealers, fellow users, etc.). Under the regulations these people may be considered subjects because identifiable, private information is being obtained. How is the IRB to deal with these “secondary” subjects? One institution may require informed consent from these people. Another may require that they be de-identified and another might have the IRB waive the requirement for informed consent if the confidentiality procedures are sufficient to say that is a minimal risk study. When I am answering a question I will always try to give options and, even if I don’t, there are always other options available.

Finally, the federal regulations are the “floor,” not the “ceiling.” They set minimum standards and specify the least that you can do, not what you should do. A good human research protections program should not be based only on compliance with regulations, but also based on best practice. Institutions have been working with these regulations for almost 40 years and have figured out good ways to implement them in ways that protect subjects while continuing to facilitate research. For someone to say, “This is what it says in the regulations and that’s what you should do,” ignores best practice and the need for higher standards. And, once you get into the area of best practice, you leave the realm of right and wrong.

So, am I saying that my answers are not useful because they are not definitive or “right”? No, I think my knowledge of the regulations and my over 30 years of experience in the field give some weight to my answers. However, don’t expect me to tell you what is “right”.

OHRP Guidance on Exemption Determinations

OHRP has come up with new guidance on exemption determinations (http://www.hhs.gov/ohrp/policy/exempt_res_det.html).

Although OHRP continues to recommend that investigators not be given the authority to independently make these determinations, they are more explicit in indicating that, under the regulations, institutions can adopt a policy that allows investigators to make their own exemption determinations. OHRP even gives examples of procedures which allow investigators to determine that their research is exempt, begin the research and then have the determination reviewed by an appropriate institutional office.

However, allowing investigators to make exemption determinations increases the likelihood that non-exempt research would be inadvertently conducted without IRB approval. This not only puts subjects at risk, but also puts the institution and the investigator at risk of non-compliance.

This guidance will likely be interpreted by investigators as blanket permission by OHRP for investigators to decide that their research is exempt without needing any other review. Institutions are advised to come up with a clear justification as to why investigator exemption determinations are not in the best interest of subjects, the institution or even the investigators.

While no one is questioning the ethics or integrity of investigators, they should not make these determinations for the following reasons:

1) No one can be totally objective about their own work. While we all try to be objective, human nature makes difficult to see all of the implications of work that is very familiar to us. Also, we tend to underestimate the risks in very familiar activities. So, every research activity needs an independent, objective review to ensure that the investigator has not overlooked some important implications for subjects in the research they are proposing.

2) Although it is assumed that exempt research has minimal risk for subjects, that is not necessarily the case. Although it may be less likely in this type of research than other types, exempt research such as interviews and surveys can harm subjects and the determination of risk, especially social and psychological risks, is more difficult than it may seem. The individuals making the exemption determinations must have sufficient knowledge and experience to evaluate the risks.

3) The exemption categories in the regulations are not as clear-cut as they may seem. Making an accurate determination with regard to whether research fits in a given category requires some training and experience with exemptions. Investigators generally do not have that level of experience.

We believe that investigators should not have the authority to determine whether their own research is exempt. That being said, it is incumbent on institutions that do not allow investigators to make exemption determinations to come up with determination procedures that are minimally burdensome and expeditious. Exempt research is generally innocuous and often time sensitive.

The exemption determinations do not have to be made by the IRB or even the IRB office, but the individual(s) making the determinations must be knowledgeable about the regulations and ethical principles. The institution must provide some sort of oversight over the process to ensure that accurate determinations are being made.

The process for requesting an exemption determination should not require investigators to fill out extensive forms. The reviewer does not need all of information that is needed for non-exempt research, only enough to make an accurate determination.

The process should also be as quick as possible. These are determinations that can be made in days, not weeks and often on the spot. Whoever is making the determinations should be readily available and provide quick turnaround.
Finally, a word to the wise with regard to OHRP’s examples of investigator determination policies , OHRP considers the conduct of non-exempt research without IRB approval as non-compliance. If, under OHRP’s examples, an investigator makes an exemption determination, begins the research and then an institutional review determines that it is not exempt, the conduct of that research prior to the institutional review would be non-compliance and would have to be reported to ORHP. We have confirmed this with OHRP.

Substantive IRB Review

With all of the attention and increased oversight of IRBs, it is essential that organizations ensure that all IRB review is "substantive". In the past, and all to often even now, IRBs have used what could be called the "sniff test". You know when you have a container of milk in the fridge and you don't know how long it's been there, you pick it up, smell it and, if it smells OK, you drink it. Well, that's how a lot of IRBs review protocols. They look at the research and, if it's good science and they are getting consent, they approve it. That's not substantive review.

The regulations have always been very specific about what constitutes substantive review. Section 111 of the regulations (in both the Common Rule and FDA regulations) details the Criteria for IRB Approval of Research. Section 111 states: "In order to approve research covered by this policy the IRB shall determine that all of the following requirements are satisfied". The criteria in Section 111 include: risks are minimized, risks are reasonable in relation to benefits, subject selection is equitable, informed consent is appropriately obtained, informed consent is appropriately documented, data monitored to ensure subject safety, privacy and confidentiality are protected, and additional protections are in place for vulnerable subjects. Substantive IRB review means that each and every one of the criteria are addressed by the IRB for every review - initial review, continuing review, and review of modifications.

In addition to the 111 criteria, there are also other required determinations in the regulations, such as the criteria for waiver of consent and the use of vulnerable popultions. These too, when applicable, must be addressed by the IRB.

Most IRBs that do more than the "sniff test" sort of address the 111 criteria and other required determinations, but don't do it in a systematic way. The best way to ensure that the required determinations have been addressed is to use a review checklist which includes all of the required determinations. Many IRBs have review checklists but don't really use them. Individual reviewers should use the checklists when doing their review and the full IRB should use the checklist to guide it's discussion of the research.

Of course, as they say, "It's not good enough to do the right thing, you must also be able to demonstrate to others that you have done the right thing." It's not enough for IRBs to address the required determinations, they must also adquately document that they have addressed them. For full IRB review, the minutes should document that the IRB addressed the 111 criteria for each protocol and the other required determinations as applicable. For expedtied review, there should be a review form that documents that they are addressed and is signed by the reviewer.

By ensuring that the IRB is doing a substantive review and adequately documenting that it has been done for every review, an organization not only fulfills it's responsiblity under the regulations, but also best protects itself from outside criticism or worse.

OHRP Proposes New IRB Rule

ORHP has issued an Advanced Notice of Proposed Rule Making to enable OHRP to hold institutional review boards (IRB) and the institutions or organizations operating the IRBs directly accountable for meeting certain regulatory requirements of the DHHS regulations for the protection of human subjects. (http://edocket.access.gpo.gov/2009/E9-4628.htm) Comments on this proposal are due by June 3, 2009.

If adopted, this proposal could change the relationship between research institutions and external IRBs. Is this a good thing or not? Although I think that external IRBs should be held accountable for compliance, I am concerned that this will make some of the problems concerning external IRBs worse. One problem with external IRBs is what was illustrated with the GAO sting. Research organizations just pick an IRB without making sure that they meet appropriate standards. If the organizations are no longer responsible for IRB compliance, this will increase the likelihood that organizations will just pick an IRB that is fast and easy. Second, organizations that use an external IRB often do not realize that they need to have an HRPP and oversee the research. They think they have IRB approval so that's all they need to do. This new rule could increase that tendency.

So, I think that holding external IRBs accountable is fine, but OHRP should continue to hold the research organizations accountable for IRB compliance as well.

Accountability: Preserving Public Trust

Human research protection programs primarily focus on ethics and regulatory compliance. However, those are not the only functions of HRPPs; they also serve to preserve public trust in research. We must remember that there is no right to conduct research. Research, especially research involving human subjects, is a privilege granted to the research community by the public. That privilege is granted based on the trust that the research will be conducted according to appropriate ethical principles and regulations. Anything that erodes that trust can result in that privilege being restricted or withdrawn. When the public has serious concerns about research it puts pressure on Congress and federal agencies to further regulate research, which is how we got to the current regulatory situation in this country.

So, how do HRPPs serve to maintain the public trust in human research? Through accountability; that is, being able to demonstrate that organizations and investigators are conducting research according to ethical and regulatory standards. It is not good enough just to do the right thing; we must be able to demonstrate to others that we are doing the right thing!

Accountability begins with investigators and institutions maintaining appropriate documentation. If you examine the federal regulations you will see that, although part of the regulations address the implementation of ethical principles (informed consent, risk/benefit, etc.), the rest is about documentation (consent forms, minutes, etc.). This is because the regulations, when they were drafted, were designed to restore public trust in human research.

The next level of accountability involves oversight by government agencies. The federal human subject regulations were first imposed back in the 70s because the public trust in research had eroded. When public trust eroded again in the mid-90s, federal oversight increased. By issuing regulations and overseeing the compliance with those regulations, the government assures the public that human research is being conducted with appropriate safeguards for the rights and welfare of subjects.

Finally, accountability involves accreditation. Organizations must be able to demonstrate that they have an HRPP that meets appropriate standards. An independent accrediting body, currently AAHRPP, develops HRPP standards and assesses organizations as to how well their HRPP meets those standards. Accreditation involves more than just compliance with regulations and appropriate documentation; it also involves the evaluation of the effectiveness of the HRPP through a site visit and interviews. As more organizations become accredited, the public can trust that research at those institutions at least meets AAHRPP standards.

Many people in our field see accountability as just so much bureaucratic nonsense and fail to appreciate the important role it plays in ensuring that important research can continue to be done. Maintaining the public's trust in research is as important as the ethical conduct of research for, without that trust, the conduct of research will be even more difficult.

IRB Stings & Accreditation

The IRB world has been rocked by the recent "stings" conducted by the Congressional Government Accountability Office (GAO) to investigate IRBs. This investigation was conducted at the behest of Congress because of concern over the adequacy of IRB review. Those who are not familiar with the results of the investigation should review the GAO and testimony from the hearings:
http://energycommerce.house.gov/index.php?option=com_content&task=view&id=1552

Although the focus of the hearings was on Independent IRBs and most of the attention has been on Coast IRB, the concerns raised by this investigation and these hearings apply to the whole field of human research protections. The best response to this investigation is accreditation. Accreditation not only raises the standards for what constitutes adequate IRB review, it provide helps ensure that organizations are meeting those standards. How would accreditation have addressed the GAO findings in this investigation?

1) Due diligence to verify information

The GAO faked 510(k) status of the device and the qualifications of PI. AAHRPP expects IRBs to obtain documentation of INDs, IDEs and 510(k) status. AAHRPP also expects organizations to have an ongoing quality assurance program, which should include audits and site visits.

2) Inadequate information to make determinations

The GAO selected IRBs that "...had less burdensome initial paperwork requirements than other IRBs for protocol submission" and "...designed our protocol so that it would contain vague information about certain aspects of our proposed study." AAHRPP expects detailed information in the protocol on all aspects of human subject protections, including subject populations, subject recruitment, consent process and safety monitoring. If the protocol requirements had been that detailed, the GAO could not have provided the information.

3) Lack of substantive IRB review

The GAO raised concerns that one IRB approved the protocol even though there were safety concerns. AAHRPP expects that all IRB review will be substantive and will address all of the regulatory criteria for approval for each review. This must be documented in the minutes with protocol-specific justification that the criteria have been met.

4) Inadequate federal oversight

The GAO was able to register a fake IRB and get an assurance for a fake medical device company. Beyond assuring that the minimum regulatory requirements have been met, OHRP is unable to review the quality of IRBs or research institutions. Accreditation will provide the accountability to help ensure that there is adequate IRB of human research.

The best protection that institutions and IRBs have against the kind of investigation conducted by the GAO (and, for that matter, against litigation) is accreditation. Not only will AAHRPP accreditation help ensure adequate IRB review, but organizations will be able to say, "We're doing it right and you don't have to take our word for it!"

Posts from Newsletter

We have sent out three newsletters so far (they are available in the Newsletter Archive: http://archive.constantcontact.com/fs039/1102478511527/archive/1102495502466.html). In the newsletter we have included articles and discussions which should be of general interest to the HRPP community so we will be copying them here on the blog. This will also allow ongoing discussion on these topics.

Doing More With Less

As we all know, we are living in difficult times. Those of us in the human research protections field are faced with the challenge of maintaining an appropriate level of protection with diminishing resources. Here are some ways that institutions can "do more with less":

Increase Efficiency

• Review policies and procedures to evaluate efficiency
• Track the "life of a protocol" to look for bottle-necks and unnecessary steps that can be eliminated without compromising the quality of IRB review
• Automate processes to save steps and time

Reduce Waste, Save Time

• Make full use of the flexibiltiy available in the regulations. No need to take up IRB time for research that is exempt or eligible for expedited review
• Poorly prepared protocols waste staff, IRB and investigator time
  
• Improve forms to ensure that IRB receives all the necessary information
• Improve investigator education and guidance to obtain better protocols

Cope with Staff Reductions

• Computerized IRB System
• Equivalent of at least .5 FTE
• Outsource IRB functions
• Quality Assurance
• Education and Training
• Webinars
  
• Full IRB Administration
• Use an external IRB

HRPA Newsletter

For those of you who are reading this blog, I want to apologize for not keeping it up. In this busy life, it's just something that slips through the cracks. I will try to do better in the future. In the meantime, I want to refer you to the HRPA Newsletter. It not only talks about what our consulting firm is up to, it will also provide articles of interest to people in the human research protections field, a Q&A section where we answer questions sent to us and news relating to HRPPs.