Evaluation of impact of adult male circumcision programs on HIV incidence and prevalence: current research, gaps in knowledge and recommendations for additional research


Kim S., Mishra S., and Miller V.  Evaluation Of Impact Of Adult Male Circumcision Programs On HIV Incidence And Prevalence: Current Research, Gaps In Knowledge And Recommendations For Additional Research.  Ann Forum Collab HIV Res. 2009;11(1):1-14.


Collaboration with The Bill & Melinda Gates Foundation, World Health Organization, and UNAIDS.  Meeting held in Washington, DC.  October 8, 2008.

Evaluation of impact of adult male circumcision programs on HIV incidence and prevalence: current research, gaps in knowledge and recommendations for additional research   1


Background        3

Meeting Objectives       4

I. Policymaker’s needs for supporting MC programs  4

II. KAIS highlights & Impact Evaluation in Kenya   5

III. Updates on current and on-going MC program activities  5

IV. Mathematical modelling approaches for MC impact analysis  8

V. Future studies necessary for Evaluating MC impact  9

Conclusion        11

Appendix A: Planning Committee    13

Appendix B:  List of Participants    14

Appendix C:   Agenda        15

References       16

















This project was a partnership between the Forum for Collaborative HIV Research, the World Health Organization, UNAIDS, and The Bill & Melinda Gates Foundation.  Special thanks go to Renee Ridzon (Bill and Melinda Gates Foundation), Tim Farley, Kim Eva Dickson (World Health Organization), and Cate Hankins (UNAIDS) for their vision and guidance in the planning and execution of the meeting. We gratefully acknowledge the support from the Gates Foundation, which made the Washington DC roundtable possible.


Many thanks go to all the roundtable participants for their active participation via presentations and discussions. We thank Abigail Wilkes and Ruth Kanthula for their expert assistance in gathering background materials. Special thanks to Linda Onaga and Sarah Kim from the Forum for Collaborative HIV Research, and Kaisa Sakrison from the Bill and Melinda Gates Foundation for their help in coordinating this meeting.


The Forum for Collaborative HIV Research is an independent public/private partnership that receives core operational funding from government agencies and industry.




The reduced risk of HIV infection due to male circumcision potentially provides benefit at three different levels: the individual; the population of circumcised men (or the direct effect); the population of the partners of circumcised men, the individuals in their sexual networks and eventually the population at large (or the indirect effects). The direct effect has been demonstrated in the three randomized controlled trials[1-3]; the indirect effect, which is more difficult to ascertain, is the topic of this report.


Prior to this roundtable discussion, UNAIDS sponsored a series of three meetings with mathematical modellers, epidemiologists, statisticians, policymakers and other public health officials to discuss mathematical modelling approaches to the measurement of potential impact on the HIV epidemic and cost-benefit of adult male circumcision (MC)[4]. Modelling efforts can be useful to estimate cost and the potential impact of MC at the population level[5-12]. A decision maker’s tool (the Decision-Maker’s Programme Planning Tool, or DMPPT, developed by the Futures Institute in collaboration with UNAIDS and described in [13]) has been designed for

this purpose (see below, Section III). This tool has been applied in Botswana[13] and Namibia, and several other countries are considering its use. Although different models started off with different assumptions in different contexts, the meetings produced the following consensus points with regards to model predictions: 


Predicting the impact of MC on the HIV epidemic and its cost-effectiveness using mathematical models and specially designed decision maker tools, although clearly extremely useful and necessary for program planning, does not replace actually measuring the impact as scale-up programs are being launched. Measuring the impact of MC at the population or community level will be crucial for several reasons, including gaining donor support as well as assisting policymakers in making decisions to support MC scale-up activities. As we move from research to programs, the question posed is: Is it possible (or even necessary) to know or capture the impact on the HIV epidemic specifically attributable to MC programs? And if so, what are the potential data sources suitable for such an analysis?


Though there are some examples of high-level detail demographic surveillance, as in Rakai, Uganda, it is not clear whether data from these surveys will convince policymakers about the positive impact of MC. One of the reasons is that results from sites that performed the earlier randomized controlled trials, like Rakai, Uganda, may not be considered generalizable. The biologic fundamentals underlying the efficacy of MC are consistent across different sites and programs. The differences that may affect the population level effectiveness of MC are contextual, such as the specific characteristics of sexual networks, the practice of condom use, and the underlying prevalence of sexually transmitted diseases. These differences and their role in modifying the impact of MC will need to be evaluated across the different sites of MC scale-up programs and impact assessments will need to be gathered from sites other than the original randomized controlled trial sites. Implementation researchers and program developers will need to ascertain what additional information is needed to gain funding to sustain MC programs long-term. They will also need to have knowledge of barriers and solutions for overcoming these barriers to MC in different populations. Many are concerned that though the efficaciousness of MC might be proved, a lag in response-time for uptake of MC programs will follow in much the same pattern as the prevention of mother to child HIV transmission (PMTCT) in some parts of the world. 


On Wednesday, October 8, 2008 a roundtable discussion was held with the goal to address the emerging issues in MC program rollout.  Key stakeholders included principal investigators, members of academia, non-governmental and governmental agency representatives, and sponsors.


Meeting Objectives


The purpose of this roundtable discussion was to discuss the best methods for evaluating the potential impact of MC programs in various settings at the population level by:


1)                Reviewing existing and planned research that will provide information on the impact of adult MC programs

2)                Identifying additional potential sources of information useful for assessing the impact of MC

3)                Discussing the potential approaches to measuring the impact of MC programs at the population level

4)                Developing a standardized information collection framework to facilitate comparison across regions and countries

5)                Discussing whether existing information sources are adequate to inform policy and investment decisions for MC programs in other countries

6)                Discussing if gaps in knowledge exist, how they can be filled and what the relevant timelines and approximate costs are.


I. Policymaker’s needs for supporting MC programs


Currently several African countries are considering adopting or scaling up MC as an HIV prevention intervention. Among them, Namibia, Kenya, and Botswana have made significant strides in recent years. 


Namibia has currently drafted a policy for MC with hopes that it will pass through the MOH and move onto the cabinet by early 2009. One of Namibia’s primary concerns about MC is behavioral risk compensation. As a result the nation wants to devise a strong and clear communication message to the population about the benefits as well as the limitations of MC.  Namibia has a low circumcision rate to begin with; therefore lack of community acceptability will be an important barrier to overcome. Factors in overcoming this barrier will include the use of medical MC among the groups who do perform traditional MC, as well as proper training and incorporation of traditional circumcisers from the community. Another barrier is the lack of a lower cadre of nurse practitioners or medical officers to reduce the burden on medical doctors. Therefore, incorporation of traditional circumcisers will definitely be necessary to initiate a national MC program for those communities practicing traditional MC. Interestingly, studies on community acceptability of medical MC show a general approval of this method of MC delivery, whereas in other nations traditional circumcision would have to be the main method of delivery. Namibia will also have to determine the feasibility of large-scale MC long-term using currently available facilities and resources. As in any other country, Namibia hopes to offer MC as part of a comprehensive HIV prevention package in their nation.


Kenya, on the other hand, is in the beginning process of MC scale-up activities. Population level impact is of greatest value for the Kenyan government and Ministries of Health (MOH) at this point in time. After initial political instability and controversy about MC, much of the groundwork for MC has been completed in terms of establishing accountability for policymakers, gaining political support, and creating a clinical training manual. The burden is now on clinicians and researchers to provide evidence for policymakers that MC is in fact effective at reducing HIV incidence at the population level. In July 2008, the CDC in collaboration with the Kenyan MOH conducted a nationwide survey that collected population data on the HIV/AIDS epidemic in Kenya. The survey was entitled the Kenya AIDS Indicator Survey (KAIS)[14] and its conclusions and results are described below (see “KAIS highlights and Kenya Impact Evaluation”). 


In order to scale-up MC activities, the Kenya MOH faces a few challenges and considerations. First, like every other country considering MC scale-up, Kenya wants to implement the most cost-effective approach for MC. Part of this will entail further defining the target population. For now, the broad definition for the target population of MC interventions includes males 15-49 years of age as the most at-risk for HIV infection, but further qualification of this target population is necessary in order to focus on those sub-populations that could provide the greatest impact at the population level.


No matter the population, for the purposes of assessing population level impact, it will be necessary to choose a static population in which to measure HIV incidence so that any impact on HIV prevalence can be linked to MC programs rather than to confounders. Behavioral studies will also be important for monitoring and evaluating MC programs in Kenya. A key concern is the possibility of behavioral risk compensation in men who undergo MC. The possibility of adopting infant circumcision programs is being considered as well, and this will depend on cultural acceptability.  Kenya will ultimately need to determine the optimal service delivery method of MC. The MOH is weighing outreach-based approaches versus mobile community approaches, as well as a physician versus a nurse-based approach. Operations research is underway to explore the possibility of non-physician clinicians, including nurses, to provide quality MC with a low rate of adverse events, but training activities are under way in order to explore this option further. 


Rwanda has been working for MC scale-up in their country as well, focusing more on neonatal MC. Current efforts are focusing on training providers, projecting costs, planning for a sustainable national program, and integrating MC into the national efforts for a comprehensive HIV prevention program. At this point in time, Rwanda is fully prepared for a full-scale adult, adolescent, and neonatal MC program. 


Botswana is using a national surgery framework in order to scale-up MC among HIV-infected males 0-49 years of age. They hope to target 80% of eligible men by 2013. In order to achieve this goal 100,000 circumcisions will have to be performed per year, as HIV prevalence in Botswana is 20%. Modelling will be an important resource for Botswana, allowing policy makers to factor in the key drivers of the HIV epidemic in Botswana, including the high HIV prevalence in mining and northern areas (49%), and the high prevalence of multiple concurrent partnerships. MC program plans also include task shifting to a lower cadre of midwives to perform MC on neonates, especially since Botswana has very few physicians (about 1 physician per district). Botswana faces many challenges to MC scale-up including developing a preceptorship program, recruiting medical staff from abroad (particularly medical doctors), performing a baseline study in order to measure and later evaluate behavior change, as well as the need to tackle opposition from those reluctant to adopt MC based on some evidence of a negative effect of MC due to re-initiation of sex before complete wound healing. 


Among all of the countries considering scale-up, an important consideration will be the acceptability and uptake of MC in the various communities and how this affects task shifting to lower cadre health care providers, as well as training and providing quality MC with few adverse events.  Before any country decides to scale-up MC activities, groundwork must include providing guidance and pilot programs for countries, quality assurance of the procedure which includes training all staff and knowledge of adverse events, and most importantly having a comprehensive prevention package in place of which MC will be a part.


II. KAIS highlights & Impact Evaluation in Kenya


In Kenya, the MOH in conjunction with the US CDC and several other institutions published the Kenya AIDS Indicator Survey (KAIS 2007) in 2008[14]. The survey highlighted some important considerations necessary for scale-up of MC programs nationwide, including trends in the Kenyan epidemic, costing, and the use of modelling to measure impact for policymakers and other key stakeholders. The KAIS 2007 is also useful for tracking changes in HIV prevalence since the Kenya Department of Health Services survey in 2003.


KAIS was a 2-stage cluster sample design that covered 8 provinces with 415 clusters surveyed. Overall, 91% of the eligible population participated in interviews and 80% followed through with a blood draw. The survey design was different from a DHS survey in several ways. First, it extended the age group to include 15 year olds. Second, it asked new questions including the HIV status and disclosure for self and partners and service uptake for testing, prevention of mother-to-child transmission, cotrimoxazole, and antiretroviral therapy. A venous blood draw was used to determine HIV prevalence and a CD4 count for those infected with HIV was obtained. Additional tests included diagnosis of HSV-2 and syphilis infection. Test results were returned to the survey participants and serum samples were kept with the goal of developing a national repository.


The results indicated that 1.4 million (7.4%) Kenyans aged 15-64 are HIV infected, which represented a 1.1% increase in prevalence in Kenya since 2003. Of great interest was the marked increase in prevalence among men in rural areas, from 3.6% to 5.7%. This unusual trend has implications for future surveillance efforts and should be of interest to researchers. The survey results also point to regional variability in the epidemic, as Nyanza province has the highest prevalence and the lowest rate of MC (below 50% compared to over 80% in the 7 other provinces). HIV prevalence was 3 times higher in uncircumcised males, confirming the need for MC program scale-up. The mixed epidemic situation in Kenya will pose challenges to the Kenyan MOH as it attempts to create national programs which will include MC scale-up as well as care and treatment scale-up. 


In order to plan future interventions, cost-effectiveness is of primary importance. Kenya is currently comparing a facility-based model versus a high-volume/mobile model. Impact data for developing optimal cost-effective programs targeting specific provinces and districts is necessary as Kenya has a mixed HIV epidemic (such as Nyanza province which has a particularly high HIV prevalence and low MC or coastal areas with an accelerating HIV epidemic). Impact data will also be important for convincing members of the donor community about the efficacy of MC in Kenya.


Unfortunately, KAIS 2007 data does not allow for sub-analyses below the provincial level. Over-sampling may be utilized for scale-up uses. The Kenya MOH and the CDC are currently collaborating with Dr. Robert Bailey (University of Illinois at Chicago) on gathering population-level data, as well as within the DHS in the Nyanza province. 


III. Updates on current and on-going MC program activities


Kisumu, Kenya


The Universities of Nairobi, Illinois, and Manitoba (UNIM) trial is continuing. Approximately 1700 young men are still enrolled in the post-trial cohort, and they are being tested for HIV and STIs every 6 months. The study also includes a behavioral questionnaire and the young men will receive counseling. Preliminary results from the 42-month follow-up show a protective effect of MC that increases over time. Other research activities focus on immune activation in seroconverters vs. controls, HPV studies, foreskin immunohistochemistry studies and an intensive qualitative study on risk compensation. The latter study includes men in the cohort and their partners, as well as men in the community.


The UNIM trial site functions as a MC Training Reference Center, working together with the MC Task Force on providing service, communications and monitoring and evaluation, all in support of the MOH in Kenya. Training is being done through the MC Consortium. UNIM utilizes an expert training team (consisting of nurses, sterilizers and clinicians) to train others: mobile training teams, outreach teams, and mobile service provision teams to deliver services in villages. Mobile training teams will train MOH staff as well as church-based ministries at 19 different facilities, 16 of which will become research sites. Outreach teams will provide MC services at smaller health facilities. Mobile service provision teams will consist of double the usual MC team, with drivers multi-tasking and training counselors.  


Services will be provided within 5 different districts offering comprehensive services including counseling and testing (CT) at fixed sites or at home visits. There will be follow-up with mobile services so that MC services can be provided immediately following home CT. Informed consent and post-op/follow-up counseling including information about wound care and a recommended 6 week abstinence period will also be provided to participants. Moreover, couple’s counseling will also be offered as part of a family approach to MC, as evidence from the UNIM collaboration has shown an increase in partner uptake of MC (e.g. several wives have gone into surgery with their husbands). 

Three additional studies are currently underway in conjunction with MC rollout activities. First, a monitoring and evaluation study will monitor MC uptake and adverse events associated with MC using a passive and an active reporting system. The sensitivity of these two approaches will be compared. The passive system will be clinic-based and collect data on adverse events and management of cases with a sample size of about 7000 men. In the active system, research assistants will go to the households of men undergoing MC and follow them up with a genital exam, collect information on adverse events, note if the men went to a clinic other than the one where they received their circumcision, and administer other follow-up questions. The team will assess factors that facilitate or provide barriers to MC uptake, gather data on time to resumption to sexual activity, and assess satisfaction with MC and the outcome, including factors such as appearance, sexual satisfaction, facility experiences, and partner acceptance.


Second, a prospective longitudinal observation study on behavior risk compensation of circumcised men, the Sexual Health Attitudes Behavior Study (SHABS), will enroll 3200 uncircumcised and circumcised men (1600 in each arm) who will be followed every 6 months for 2 years. The study will not only assess changes in sexual behavior but also changes in perceptions of HIV risk, sexual function and satisfaction of circumcised males and of a sample of their primary partners, both before and after MC, as well as over time, compared to their uncircumcised counterparts. Along with the men, a subset of primary partners will also be surveyed before and after MC.


Third, a series of three cross-sectional surveys will be administered to determine MC impact in the Kisumu municipality.  The Male Circumcision Impact Study (CIRCIS) will conduct surveys every two years (years 1, 3, and 5) from a random household sample of 2000 men and women. The surveys will include questionnaires about knowledge and beliefs of MC and HIV risk and sexual risk behaviors. An HIV test and genital exam will be performed each time as well. The survey aims to measure changes in MC prevalence following MC scale-up as well as behavioral changes and perceived risk.


Other research planned for Kenya includes a neonatal circumcision program, text messaging for post-op adherence, and wound healing in HIV-infected and HIV-uninfected men. The latter study will examine the healing and keratinization processes and was inspired by a similar one currently being conducted in Rakai, Uganda. The aim is to elucidate which factors are involved in wound-healing, including the keratinization process that is thought to perhaps be more crucial in HIV transmission prevention than the density of Langerhans cells. This is an important study because of the different surgical methods used at these sites. The neonatal circumcision program, currently seeking funding, will take place at provincial neonatal hospitals and will include follow-up and evaluation for uptake and safety. The text-messaging program involves sending males who undergo MC a series of 15 text messages over 42 days post-op.


In Kisumu, ongoing and new research is funded through the Male Circumcision Consortium (MCC), sponsored by the Bill & Melinda Gates Foundation. The MCC is led by Family Health International and two core partners—the University of Illinois, Chicago and EngenderHealth—and local partners including the Ministries of Health, the Nyanza Reproductive Health Society and others.  In addition to the studies being implemented through University of Chicago, Illinois, one study examines the feasibility of the private sector to support expanded MC services.  Research is also being conducted to explore trained, non-physician clinicians' ability to provide MC services through fixed facilities and outreach services with a complication rate comparable to that of surgeons.


Orange Farm, South Africa


In South Africa, the Orange Farm phase 4 study, designed in 2005 has been underway since 2007. This is an implementation strategy study, and the goal is to offer free and safe MC to the Orange Farm community of about 200,000 people (i.e. 40,000 MC over 2 years), and to monitor the community for MC effects. The study utilizes communication and information strategies including community meetings, home visits, local radio stations, motorized loud speakers, and a plane-driven banner. In terms of service provision, one surgical facility currently offers fast and high quality MC and several local general practitioners provide MC in private settings. Candidates for MC undergo informed consent, voluntary counseling and testing (VCT), surgery, and follow-up. Of note, Orange Farm incorporates traditional circumcisers who provide all MC services other than the actual surgery. Results will include uptake; effects on knowledge, attitude, and behavior regarding MC; the effect on HIV prevalence among men and women; the impact of MC on sexual behavior at the community and city level; and the effect of VCT.


Results from the baseline analysis of the study have already been analyzed[15]. This five-year study, composed of three cross-sectional surveys given at baseline and at years 3 and 5, measures MC uptake, knowledge, attitude, and behavior; sexual behavior, and HIV prevalence among the 2000 men and women ages 15-49 years who are enrolled in the study. The questionnaire is coupled with a genital examination and collection of biologic samples (blood and urine) for analysis. Results from baseline indicate that 67.5% of those uncircumcised plan on being circumcised in the future, with 92.6% saying they would go to a general practitioner or hospital for MC. This indicates a shift in culture and highlights the need to facilitate this kind of uptake. Of the 82.1% of males who said they would go to a doctor for MC if it were offered free of charge, 511, or 72.1% actually received MC (59.1% uptake among uncircumcised men). 


Concerning knowledge, attitude and behavior, 90% of men and 81.7% of women disagreed that men who are circumcised do not need to use condoms; 92.6% of men and 89.1% of women agree that circumcised men can still become infected with HIV; whereas 17.5% of men and 15.9% of women agree with the statement that circumcision protects fully against HIV. Thus, some discrepancy in knowledge, attitude and behavior still exists. Moreover, following the administration of the baseline survey, it became apparent that uncertainty about what being circumcised meant existed, as many were confusing MC counseling with actual foreskin removal. Thus, visual aids are now being used during surveys to clarify circumcision status[15].


Baseline information also showed that many have a good understanding of condom use, HIV infection, and the efficacy of MC. This shows that within this population, people are receiving information about MC from other sources apart from the Orange Farm study. It is clear, however, that the population still needs more information about HIV and MC.


Rakai, Uganda


The Rakai site, staffed by Ugandan physicians and nurses, will serve as a service and a training site for WHO and PEPFAR. Funding for services in the Rakai cohort comes from multiple sources, as the NIH has funded one trial of HIV-uninfected men and the Bill & Melinda Gates Foundation has funded one for HIV-infected men and their partners. Service MC is funded through PEPFAR. The current HIV prevalence in the Rakai community is approximately 11%, with an incidence of approximately 1.2/100 patient-years. The Rakai program encompasses four research areas: epidemiological, clinical, operational and basic science. The advantages of the Rakai site include the depth of knowledge regarding the epidemic including HIV status, sexually transmitted diseases, antiretroviral treatment status, sexual networks, etc. The Rakai site is also ideally suited for partner studies.


The epidemiological research program is based on community-level surveillance of both men and women regarding MC acceptance, satisfaction with MC, behavior changes, and HIV incidence in men and women. Surveys are being conducted on men and their partners who have received MC during trials or as a service. Longitudinal data available include: trends in HIV prevalence, incidence, antiretroviral therapy use, VCT use, condom use, sexual networks, number of acts of intercourse with each partner, STI symptoms/serology in both partners, HIV viral load in HIV+ index partners, and CD4 levels. This cohort may offer the possibility to dissect the contribution of the various interventions in reducing the incidence and/or prevalence of HIV infection.


The clinical research topics addressed include post-MC healing times and the process of keratinization in both HIV-infected and uninfected men, as well as the effect of MC on HIV viral load in HIV-infected men.


In terms of operations research, the Rakai team is comparing two surgical methods of MC – the sleeve and the dorsal slit methods. Outcomes for analysis include adverse events, healing time, and time needed for surgery. Also, MCs performed by trained medical or clinical officers will be compared. In addition to the outcome measures listed above, this part of the study will also compare the number of training surgeries required for competency.


Much still needs to be understood in terms of the basic science of MC. Researchers in Rakai will investigate foreskin immunohistochemistry, including parameters such as inflammation and cell-associated HIV. Collaboration between the Universities of Toronto and Karolinska has been established, with the goal to look at foreskin immune response in highly exposed HIV-uninfected men, unexposed uninfected men and HIV-infected men. Genital immune factors will also be looked at in female partners, in association with the male MC, HIV and STI status. Finally, associations of MC with HSV-2, STIs and HIV in men and female partners will be analyzed. Planned studies include the effects of having or not having a foreskin on viral load and on viral diversity in the male/female genital biome by HIV status.


To date, Rakai has 3 years of follow up data available. These preliminary results indicate a reduced acquisition of HSV-2 among circumcised men, with a Cox HR estimate of 0.733 (0.489-0.977)[16]. Differences in healing time were observed for HIV-infected and uninfected men (73.0% vs. 83.2% respectively at 30 days; 92.7% vs. 95.8% at 6 weeks)[17] The results also confirm the increasing efficacy of MC as time from MC increases. Thus the incidence of HIV infection per 100 patient-years (incidence/100 py) among circumcised men is 1.2 between months 0-6 post-circumcision; 0.4 between months 6-12; 0.3 between months 12-24 and 0.0 between months 24-36 post circumcision. The incidence/100 py among controls ranges between 1.2 and 1.8 during the same time periods. Adverse events decrease as the number of surgeries increases (from 8.77% for 0-19 surgeries to 2.00% after 100 surgeries). The time required for surgery (a measure of competency) also decreased significantly as the number of surgeries increases[18]. Of note, adverse events were higher among those who initiated sex before healing was certified, with a risk ratio of 1.65 (1.16-2.32). Thus, in order to control the quality of MC, some propose that a smaller group of trained health professionals, rather than a more expansive cadre, should perform MC procedures. This sentiment was found in Kisumu, Kenya as well. One of the advantages to this strategy is the progressive reduction in surgery time and reduced likelihood of adverse events with the greater number of surgeries performed. Therefore “expert circumcisers” are more favorable for large scale-up of MC.


Male circumcision provides benefits to women: significant reductions in genital ulcer disease (PRR 0.76; CI 0.60-0.97), trichomonas (PRR 0.53; CI 0.33-0.85), bacterial vaginosis (PRR 0.80; CI 0.71-0.89) and severe bacterial vaginosis (PRR 0.31; CI 0.18-0.54) were recorded[19]. Circumcision was associated with a significant homogenization of microbiota in men. This included a reduction in the heterogeneity of organisms post circumcision, and more importantly, a reduction in the pro-inflammatory anaerobic organisms. This effect may explain the protective effect of MC. The investigators will continue these studies by analyzing change in the flora of the female partners of the circumcised men pre- and post-circumcision.


PEPFAR activities


Currently PEPFAR is funding several MC programs in Africa. In addition to Rakai, Uganda, PEPFAR is conducting service delivery evaluations of neonatal MC in Zambia and Botswana, where safety and acceptability among providers and parents are being reviewed. Two programs are currently under external review. The first proposes to use DHS data in Kenya to determine the impact on HIV incidence in a well-defined community in addition to service-delivery and risk compensation evaluation among MC men and their partners. The second, in Botswana measures MC impact using AIDS Indicator Survey (AIS) data. Baseline data for 25,000 samples along with district level HIV incidence and prevalence is available. PEPFAR would also like to use MOH/AIS data in 5 years to study MC service delivery impact, with considerations to over-sample or use a stepped-wedge model design.


Encouragingly, countries still lacking political support for MC programming are proceeding with plans to initiate MC programming. Some countries are promoting MC and providing technical assistance to countries needing it. One barrier however is the difficulty of getting non-research determinations approved by the CDC. 



IV. Mathematical modelling approaches for MC impact analysis


Mathematical modelling is the primary method for projecting the impact of MC at the population level. In addition, modelling can also serve as a base from which the research and academic community will be able to ask better questions.  For policymakers, models will be important for providing support for the planning of MC rollout. 


In country use for planning purposes is one application of mathematical modelling. The UNAIDS Decision-Makers’ Programme Planning Tool (DMPPT) [13] was developed for this purpose. Proponents of the tool highlight major benefits of using models for program planning. First, modelling provides decision makers with cost estimates linked to estimates of impact at the population level (e.g. cost per HIV infection averted). The tool is useful for allowing decision makers to understand the costs and impact of different options for MC service delivery. Practically, the DMPPT allows for the convenient creation of cost spreadsheets that include unit costs, user fees, annual demand creation costs, and the proportion of different delivery modes. The software then allows one to alter variables such as time and pace of scale-up, details about the target population such as age, gender, ARV use, and other parameters, in order to project impact over time.


Modelling can also indicate the potential for a secondary impact (indirect effect) of MC interventions on the incidence of HIV in women. This impact is important for policymakers making decisions at the population level, as the beneficial effects on women and the population at large increase in importance with time. The UNAIDS-led modelling consensus work underscored that women will benefit indirectly as HIV prevalence in men falls as a result for male circumcision[4].


As noted above, both the direct effects of MC on HIV incidence among circumcised men, and the indirect effects of MC scale-up on women and the population at large, are of interest to policymakers and impact evaluators. The indirect effects will take substantially longer to become evident at the population level than the direct effect on reducing HIV incidence in circumcised men. During the interim period, a mix of direct and indirect effects would be apparent. Models do highlight the potential for indirect benefit (in terms of HIV infection) to women[4] with higher service coverage leading to more marked effects. Models will not be able to resolve, from trends in prevalence and/or incidence among populations, what portion of benefit is directly attributable to the circumcision intervention and what proportion is attributable to other interventions and the natural evolution of the epidemic.


Since MC clearly shows the direct benefit to circumcised men, is it necessary to demonstrate the indirect effect? It could be argued that, since the indirect effect is driven by the direct effect on circumcised men, the direct effect is in essence a surrogate marker for the indirect effect. Demonstrating an indirect effect on HIV incidence is important because of the substantial size of the population standing to benefit indirectly: all women whose male partners become circumcised, as well as their entire sexual networks, and eventually, entire communities and populations where MC programs exist. Although policy makers would not consider the rollout and scale-up of MC programs if the direct effects were not clearly evident, the predicted much larger indirect impact is what will drive the considerable investment needed in the scale-up of programs.


As stated earlier, the epidemiological context in which MC programs are launched is important. The relevant context includes factors such as antiretroviral therapy use, condom use, sexual networks and sexually transmitted diseases, and will affect the scale of the predicted indirect effects. The fact that various models in different settings, taking into account the different contexts, show similar results is thus reassuring.


It will be important, however, at this stage of the scale-up efforts and implementation research, that studies not depend solely on models for providing primary evidence about the effectiveness of programs; but rather that models are used as part of a comprehensive evidence package, especially when baseline data are unavailable. These models should also be updated with data as it becomes available throughout the studies. 



V. Future studies necessary for Evaluating MC impact


General Considerations


When considering moving forward with planning future research activities related to evaluation and impact assessment of MC, it is important to distinguish between monitoring and evaluation versus impact assessment itself. Data on specific populations and/or communities will be needed in order for nations to be able to make recommendations for MC programs. Questions surrounding community effects should be prioritized in order to help decision makers make intelligent choices about where other studies need to be implemented to supplement the data from current and/or past studies. It will also be important to integrate qualitative components to impact evaluation, on top of quantitative measurements. Important impact questions include those regarding cost-effectiveness as well as the institutionalization and sustainability of programs that are currently donor supported once MC scale-up begins. Efforts should be made to develop multi-partner, multi-country evaluation frameworks which will be beneficial for clarifying which key studies are needed for determining MC impact.


There are several considerations for determining how impact will be assessed and measured in high-risk populations. One consideration is the fact that the high-risk populations originally evaluated are now aging, and it is unknown how this will affect data evaluation and how they will contribute to HIV prevalence within the population. Also, tracking the in/out migration will be essential for determining impact evaluation, particularly in rural areas such as in Kenya.  Migration, particularly that of men, plays a large role in the HIV epidemic in many regions.


Before beginning any new studies, it would be valuable for the research community to evaluate what studies are present and what evidence is flowing from them, as well as identifying any gaps in information.


Cross-sectional vs. longitudinal studies


Among the various types of study currently being conducted are cross-sectional, serial cross-sectional, and longitudinal cohort studies; however when determining long-term impact, many believe that longitudinal cohort studies would be the most useful.


Following the demonstration of efficacy through randomized controlled trials, cohort studies are beneficial because they have the added benefit of including other interventions and programs, thus the value of cohort studies extends beyond analysis of MC programs. A large number of longitudinal studies may not be necessary: a few longitudinal studies that are rich in data will contribute significantly for evaluating the long-term impact of MC. 


Currently the only longitudinal study being conducted is in Rakai, Uganda. Meeting participants recommended support for additional longitudinal cohort studies. These should include studies in areas with high HIV prevalence and low current circumcision rates, such as South Africa. Additional studies could take the form of the prospective cohort study proposed by the CDC in Kenya. The proposed study in Nyanza province would expand home-based testing to the whole Gem area. A mobile MC team would spend 2 weeks in the Gem area and standardized follow-up would be performed in line with MOH guidelines. The objective of this longitudinal study would be to compare service delivery models (facility-based versus mobile-facility), adverse events, acceptability and feasibility of MC, consumer satisfaction with services, cost and cost-effectiveness, as well as sexual behavior and risk compensation among men and women in the communities with high MC prevalence. Plans are to collect data at baseline, 18, and 36 months and HIV incidence will be recorded at months 18 and 36 using serologic testing. Researchers are particularly interested in collecting data to see if MC (as an HIV service) could have a negative impact on people’s perception and uptake of other services offered beyond HIV/AIDS, since the services will be administered by DSS. In Nyanza, among a population of 78,000 about 80% are uncircumcised men within a population with a high rate of undiagnosed HIV. The expected uptake of MC over 3 years is 50%.    


Prospective follow-up in cohorts of different settings (based on HIV prevalence) could help decipher the effects of different variables. A network of cohorts with comparable data across regions would be beneficial for the research community.  Other potential sources of cohorts are occupation-based cohorts (e.g. mine workers in South Africa), and geographic cohorts (e.g. the population of Likoma island on Lake Malawi).  The step-wedge design in the setting of rollout would allow analysis of the impact over time.


Specific research-based cohorts are also of interest. Recently, the NIH has requested and received proposals for research programs that will design and offer comprehensive HIV prevention packages for specific target populations in a clinical research setting, and that will perform pilot studies to determine community acceptability of these programs[20]. These studies, of which several have been funded, could contribute additional information regarding MC interventions in the setting of cohort studies. PrEP (pre-exposure prophylaxis) and Microbicide studies are another potential source of research-based cohorts as many of these studies have high enrollment and are already actively recruiting in a variety of settings.


The group also considered the transformation of the Orange Farm study in South Africa, a serial cross-sectional study, into a longitudinal study. The advantages would be having another longitudinal study in a population with a particularly high HIV incidence rate (whereas Kenya and Uganda both have moderate prevalence). Also, the Rakai study, though a cohort study, remains in a very controlled environment, whereas the South Africa study would take place in a more truly observational setting. The main disadvantage of converting the Orange Farm study into a longitudinal cohort study is that it is already well underway towards achieving its original purposes as a serial cross-sectional study, none of which are longitudinal in nature, including:


1)       Collecting information on MC uptake amongst individuals and infer those results at the community level

2)       Reproducing the effect of MC status on HIV-infection in the context of the community rather than within a trial, i.e. observing real MC rollout at the province level

3)       Collecting the same information on women as that from circumcised men, hopefully within 2 years of study initiation in order to show that women with newly circumcised partners do not have a much higher prevalence than other women.


There is also great value in being able to perform demographic surveillance using population-based longitudinal cohorts for impact assessment. Networks of demographic surveillance sites could be utilized and overlapped with multiple countries such as Uganda, Kenya, and South Africa. These demographic surveillance based studies are in fact “open cohorts”; however immigration within the population will need to be included in the power calculations. In terms of MC impact, information regarding risk compensation, behavioral disinhibition and ecologic evidence will need to be collected. 


Research on MC effect on HSV-2, HPV and interactions with HIV


HSV-2 and HPV infections (or co-infections) play a very major role in the HIV epidemic and meeting participants considered further studies an area of high priority. All three trial sites are currently contributing data and ongoing analysis to this issue[16, 21-28].


Studying the interaction between HSV-2 and HIV needs to take into consideration the difficulties in determining a temporal association. This is a problem in cross-sectional studies, but even in prospective studies, the higher infectivity of HSV-2 – making it more likely to be acquired first – presents difficulties[21]. Another issue is that not all studies have looked at HSV-2 serology as well as clinical symptoms such as genital ulcers. This may be important, because genital ulcer disease (GUD) has been demonstrated to increase HIV acquisition, thus herpetic ulceration could play a role in this manner[21]. In currently available data on MC, HSV-2 acquisition and HIV risk present a complex picture. As the Orange Farm study demonstrated, HSV-2 infection has a substantial impact on HIV acquisition and is responsible for 25% of new cases[27]. Mathematical modelling also indicates a protective effect of MC on acquisition of HSV-2[24], yet the protective effect of MC against HIV acquisition was independent of HSV-2 serostatus[27]. These findings illustrate the need to more closely investigate the role of herpetic ulcers. Data from Kisumu indicate that HSV-2 seroincidence does not differ between circumcised and uncircumcised men, however, the incidence of GUD was 2 times greater in uncircumcised men[21]. This raises the question: do the clinical manifestations of HSV-2 infection differ in circumcised or uncircumcised men? In Rakai, maximum protection against GUD appears to have occurred in HSV-2-negative participants, indicating that MC protected against traumatic ulcers rather than herpetic ulcers.  This highlights the need to include both HSV-2 seroincidence as well as GUD in participants and their partners involved in the study. A longitudinal study will help in analyzing HSV-2 in the context of HIV, but it will be important for all sites to monitor HSV-2 incidence during post-trial analysis in order to gain sufficient data on MC protection against HSV-2. 


The impact of MC on HSV and HPV is also of interest to investigators. The Orange Farm study demonstrated reduced prevalence of urethral high-risk HPV in men after MC[29]. This raises the question of how MC may affect female carriage of HPV, which could in turn affect a woman’s susceptibility for HIV infection.


In settings where the infrastructure for conducting a longitudinal study is not in place, an enhanced DHS survey system could be utilized instead. This could be a rather facile process done by adding a physical evaluation, HSV-2 serology, and an MC questionnaire to DHS surveys.  Physical examinations are an important element of MC studies in any case, in addition to verifying STIs since self-reporting is not always a very reliable source of information.


Additional considerations


There is the potential to use antenatal sentinel surveillance for obtaining country level data on MC. One drawback will be that due to the small sample of these surveillance surveys, the community level impact of MC will not be able to be determined.


There was a general recognition for the need to work with families and incorporate infant circumcision as part of the larger picture for MC scale-up. In addition to the benefit by infant MC itself, the promotion of MC among infants may also help promote MC in young adults.


Because of the various limitations to each study, as well as the vital importance of each study, information on MC programs must be collected and evaluated from a holistic viewpoint that includes all the data from these and future studies.




The general recommendation from this roundtable discussion was that sites where MC programs are currently underway or are currently being set up should continue with such efforts. Program monitoring and evaluation will be critical for determining MC impact accurately. Models will not only be useful for informing these studies but results can also be compared to the projections made by models. Realistically, it will probably take at least 2 years for a population level effect to be seen, and any signal will be at the low end. As time progresses, the population level effect will become more evident. Behavioral risk compensation will be an important factor influencing policymakers about MC scale-up. Policymakers could also be swayed by evidence of the protection due to MC against co-infections. Many studies are incorporating herpes prevention and future studies can be planned once data becomes available. 


It is evident that rigorous and intensive studies will be necessary for determining the impact of MC at the population level in the future. Cohort studies will be useful for this, and have the added benefit of being useful for observing more than MC impact. Additional longitudinal cohort studies are needed and resources will be required for this. Regardless of whether the impact of MC at the population level can be accurately measured, the benefit of MC on the individual level is indisputable. There was a consensus that more meetings such as this roundtable will be required once results from current studies accumulate in order to continue the discussion of MC impact.



















Appendix A:        Planning Committee


Renee Ridzon, M.D.

Bill & Melinda Gates Foundation


Veronica Miller, Ph.D.

Forum for Collaborative HIV Research


Kim Eva Dickson, M.D.

World Health Organization


Tim Farley, Ph.D.

World Health Organization


Catherine Hankins, M.D., C.C.F.P, M.Sc., F.R.C.P.C.






Appendix B:  List of Participants


Nelago Tukondjeni Amagulu, M.D.

Ministry of Health and Social Services, Namibia


Bertran Auvert, M.D., Ph.D.

University of Versailles


Robert Bailey, Ph.D., M.P.H.

University of Illinois at Chicago


Naomi Bock, M.D., M.S.

Global AIDS Program, CDC


Lori Bollinger, Ph.D.

Futures Institute


Rebecca Bunnell, Ph.D.

CDC – Kenya


Peter Cherutich, M.P.H., MBChB

Ministry of Heatlth, Kenya


Kim Eva Dickson, M.D.

World Health Organization


Tim Farley, Ph.D.

World Health Organization


Ron Gray, M.D.

Johns Hopkins Bloomberg School of Public Health


Emily Gumkowski



Timothy Hallett, Ph.D.

Imperial College London


Catherine Hankins, M.D., C.C.F.P, M.Sc., F.R.C.P.C.



Reinhard Kaiser, M.D., M.P.H.

CDC – Kenya


Sarah Kim, B.S.

Forum for Collaborative HIV Research


Timothy Mastro, M.D., F.A.C.P

Family Health International


Veronica Miller, Ph.D.

Forum for Collaborative HIV Research


S. Janelle Mingus, M.P.H.



Shrimant Mishra, M.D., M.P.H.

Forum for Collaborative HIV Research


Luke Nkinsi, M.D., M.P.H.

Bill & Melinda Gates Foundation


Niloo Ratnayake

National AIDS Control Commission


Jason Reed, M.D., M.P.H.

Global AIDS Program, CDC


Renee Ridzon, M.D.

Bill & Melinda Gates Foundation


Monica Ruiz, Ph.D., M.P.H.

Forum for Collaborative HIV Research


Kim Shaffer, M.P.H.



Maria Wawer, M.D.

Johns Hopkins Bloomberg School of Public Health


Carolyn Williams, Ph.D.






Philip Setel, M.D., Ph.D.

Bill & Melinda Gates Foundation


Appendix C:   Agenda


Forum for Collaborative HIV Research

Evaluation of impact of adult male circumcision programs on HIV incidence and prevalence: current research, gaps in knowledge and recommendations for additional research

Oseola McCarthy Room

Bill and Melinda Gates Foundation                                           

1300 I Street, NW Suite 200 East

Washington, DC 20005

October 7-8, 2008

Sponsored jointly by the Bill & Melinda Gates Foundation, Forum for Collaborative HIV Research, UNAIDS and WHO

Meeting Objectives:

*Review existing and planned research that will provide information on the impact of adult MC programs

*Identify additional potential sources of information on the impact of MC

*Discuss the potential approaches to measuring the impact of MC programs at the populations level

*Develop a standardized information collection framework to facilitate comparison across regions and countries

*Discuss whether existing information sources are adequate to inform policy and investment decisions for MC programs in other countries

*If gaps exist, how can they be filled, what are the relevant time lines and approximate costs?

8:30AM - 9:00AM

Continental Breakfast



9:00AM - 9:15AM

Welcome and Introductions

Veronica Miller,

Renee Ridzon


9:15AM - 9:45AM

Meeting Goals and Objectives

Tim Farley, Cate Hankins


9:45AM - 10:30AM

Policy maker's needs: what information is required to allow scale up in country

Dr. Amagulu, Namibia

Dr. Cherutich, Kenya

Dr. Ratnayake, Rwanda


Moderated by   

Cate Hankins

10:30AM - 11:00AM

Review of KAIS:

Potential framework for assessing impact of new interventions?

Rebecca Bunnell

Moderated by

Tim Farley

11:00AM - 11:15AM




11:15AM - 11:30AM

Coffee Break



11:30AM - 11:45AM

Review of currently ongoing work: summary of background paper

Shrimant Mishra, MPH

Moderated by        Kim Dickson

11:45AM - 1:00PM

Updates on currently ongoing and planned work: additional potential sources of information and Discussion

Robert Bailey, Naomi Bock, Bertran Auvert, Maria Wawer; all


1:00PM - 1:30PM




1:30PM - 2:15PM

Measuring the impact of male circumcision at the population or community level

Tim Hallett; Lori Bollinger; Philip Setel

Moderated by Caroline Williams

2:15PM - 2:45PM

Common Methodological Framework

All participants


2:45PM - 3:30PM

What additional studies are needed?

All participants

Moderated by Veronica Miller

3:30PM - 3:50PM

What are the relevant timelines and costs?

All participants

Moderated by

Renee Ridzon

3:50PM - 4:00PM

Final comments and next steps

Cate Hankins, Tim Farley













1.             Auvert B, Taljaard D, Lagarde E, Sobngwi-Tambekou J, Sitta R, Puren A. Randomized, controlled intervention trial of male circumcision for reduction of HIV infection risk: the ANRS 1265 Trial. PLoS Med 2005,2:e298.

2.             Bailey RC, Moses S, Parker CB, Agot K, Maclean I, Krieger JN, et al. Male circumcision for HIV prevention in young men in Kisumu, Kenya: a randomised controlled trial. Lancet 2007,369:643-656.

3.             Gray RH, Kigozi G, Serwadda D, Makumbi F, Watya S, Nalugoda F, et al. Male circumcision for HIV prevention in men in Rakai, Uganda: a randomised trial. Lancet 2007,369:657-666.

4.             UNAIDS/WHO SACEMA Expert Group on Modeling the Impact and Cost of Male Circumcision for HIV Prevention. Male circumcision for HIV prevention in high HIV prevalence settings: What can mathematical modeling contribute to informed decision making? PLOS 2009,in press.

5.             Hallett TB, Singh K, Smith JA, White RG, Abu-Raddad LJ, Garnett GP. Understanding the impact of male circumcision interventions on the spread of HIV in southern Africa. PLoS ONE 2008,3:e2212.

6.             Williams BG, Lloyd-Smith JO, Gouws E, Hankins C, Getz WM, Hargrove J, et al. The potential impact of male circumcision on HIV in Sub-Saharan Africa. PLoS Med 2006,3:e262.

7.             Auvert B, Marseille E, Korenromp EL, Lloyd-Smith J, Sitta R, Taljaard D, et al. Estimating the resources needed and savings anticipated from roll-out of adult male circumcision in Sub-Saharan Africa. PLoS ONE 2008,3:e2679.

8.             Gray RH, Li X, Kigozi G, Serwadda D, Nalugoda F, Watya S, et al. The impact of male circumcision on HIV incidence and cost per infection prevented: a stochastic simulation model from Rakai, Uganda. Aids 2007,21:845-850.

9.             Kahn JG, Marseille E, Auvert B. Cost-effectiveness of male circumcision for HIV prevention in a South African setting. PLoS Med 2006,3:e517.

10.           Londish GJ, Murray JM. Significant reduction in HIV prevalence according to male circumcision intervention in sub-Saharan Africa. Int J Epidemiol 2008,37:1246-1253.

11.           Nagelkerke NJ, Moses S, de Vlas SJ, Bailey RC. Modelling the public health impact of male circumcision for HIV prevention in high prevalence areas in Africa. BMC Infect Dis 2007,7:16.

12.           Podder CN, Sharomi O, Gumel AB, Moses S. To cut or not to cut: a modeling approach for assessing the role of male circumcision in HIV control. Bull Math Biol 2007,69:2447-2466.

13.           Bollinger LA, Stover J, Musuka G, Fidzani B, Moeti T, Busang L. The cost and impact of male circumcision on HIV/AIDS in Botswana. J Int AIDS Soc 2009,12:7.

14.           Kenya AIDS Indicator Survey. 2007,http://www.aidskenya.org/public_site/webroot/cache/article/file/KAIS__Preliminary_Report.pdf.

15.           Auvert B. Phase 4 Orange Farm Study. FCHR Website 2008,http://www.hivforum.org/storage/hivforum/documents/MC2/bertran%20v10.pdf.

16.           Tobian AA, Serwadda D, Quinn TC, Kigozi G, Gravitt PE, Laeyendecker O, et al. Male circumcision for the prevention of HSV-2 and HPV infections and syphilis. N Engl J Med 2009,360:1298-1309.

17.           Kigozi G, Gray RH, Wawer MJ, Serwadda D, Makumbi F, Watya S, et al. The safety of adult male circumcision in HIV-infected and uninfected men in Rakai, Uganda. PLoS Med 2008,5:e116.

18.           Kiggundu V, Watya S, Kigozi G, Serwadda D, Nalugoda F, Buwembo D, et al. The number of procedures required to achieve optimal competency with male circumcision: findings from a randomized trial in Rakai, Uganda. BJU Int 2009.

19.           Gray RH, Kigozi G, Serwadda D, Makumbi F, Nalugoda F, Watya S, et al. The effects of male circumcision on female partners' genital tract symptoms and vaginal infections in a randomized trial in Rakai, Uganda. Am J Obstet Gynecol 2009,200:42 e41-47.

20.           National Institutes of Health. RFA-AI-08-044. http://grants.nih.gov/grants/guide/rfa-files/RFA-AI-08-044.html 2008.

21.           Bailey RC, Mehta SD. Circumcision's place in the vicious cycle involving herpes simplex virus type 2 and HIV. J Infect Dis 2009,199:923-925.

22.           Gray RH, Wawer MJ, Polis CB, Kigozi G, Serwadda D. Male Circumcision and Prevention of HIV and Sexually Transmitted Infections. Curr Infect Dis Rep 2008,10:121-127.

23.           Gray RH, Wawer MJ, Serwadda D, Kigozi G. The role of male circumcision in the prevention of human papillomavirus and HIV infection. J Infect Dis 2009,199:1-3.

24.           Mahiane SG, Legeai C, Taljaard D, Latouche A, Puren A, Peillon A, et al. Transmission probabilities of HIV and herpes simplex virus type 2, effect of male circumcision and interaction: a longitudinal study in a township of South Africa. Aids 2009,23:377-383.

25.           Mehta SD, Moses S, Agot K, Agingu W, Parker C, Ndinya-Achola JO, Bailey RC. Herpes simplex virus type 2 infection among young uncircumcised men in Kisumu, Kenya. Sex Transm Infect 2008,84:42-48.

26.           Mehta SD, Moses S, Ndinya-Achola JO, Agot K, Maclean I, Bailey RC. Identification of novel risks for nonulcerative sexually transmitted infections among young men in Kisumu, Kenya. Sex Transm Dis 2007,34:892-899.

27.           Sobngwi-Tambekou J, Taljaard D, Lissouba P, Zarca K, Puren A, Lagarde E, Auvert B. Effect of HSV-2 serostatus on acquisition of HIV by young men: results of a longitudinal study in Orange Farm, South Africa. J Infect Dis 2009,199:958-964.

28.           Tobian AA, Ssempijja V, Kigozi G, Oliver AE, Serwadda D, Makumbi F, et al. Incident HIV and herpes simplex virus type 2 infection among men in Rakai, Uganda. Aids 2009.

29.           Auvert B, Sobngwi-Tambekou J, Cutler E, Nieuwoudt M, Lissouba P, Puren A, Taljaard D. Effect of male circumcision on the prevalence of high-risk human papillomavirus in young men: results of a randomized controlled trial conducted in Orange Farm, South Africa. J Infect Dis 2009,199:14-19.