Phase 2 Propsal
Leadership: Paul Wesson, Meghan Morris, Thomas Hoffmann, Maria Glymour and YOU!
1. Rationale. Sampling is a cross-cutting challenge in multiple research domains (including health disparities, genomics, disease-specific research, health care, and precision medicine), and a foundational epidemiologic topic. This initiative will have a broad appeal to departmental and university research and training efforts.
2. The Idea. This year-long - four-quarter block - initiative will establish DEB as a knowledge hub for sampling methods to improve population science and health disparities research. Each block will cover a pressing challenge as it relates to sampling and population science, and methodological approaches to address those challenges. Month 1 of each block will include a 2-4 hour themed workshop led by a topic expert (e.g., external academic or industry, DEB member) paired with a DEB faculty expert; with the goal of establishing foundational knowledge and encourage intra/extra-departmental collaboration. Two ninety-minute monthly sessions complete the block; held in a journal club format with a DEB facilitator (paid) with the goal of applied methods/theory connecting public health implications of the topic introduced during the workshop. We envision this culminating in a workgroup forming with a core group of DEB members engaged in collaborative implication (e.g., theoretical papers, applied research projects, commentary/op-ed articles, department courses, grants). The year-long series will increase all attendee’s overall knowledge of sampling methods and strengthen expertise necessary to establish our department as a hub for modern sampling methods for population research. The tentative curriculum is as follows: (Block 1) complex sampling designs for representative samples, (Block 2) sampling for hard-to-reach populations, (Block 3) techniques to estimate the “dark figure”, i.e., assessing the magnitude of unreported data points (selection bias) for hard to reach populations, including estimating population size, and (Block 4) internal and external validity concerns as they relate to pooled data sources, including tools to address inconsistent sampling strategies fielded for different population subgroups such as racial/ethnic minorities.
3. Importance. Sampling is a core study design component and the origins of the analytic sample shape analytic decisions, interpretation, and public health relevance of our research. Many of the technical concepts that arise in sampling relate directly to broader analytic ideas (e.g., the Horvitz Thompson estimator as the basis for inverse probability weighting and marginal structural models). Notwithstanding debates about the importance of representative sampling1, substantial evidence documents the difference between individuals who participate in health studies and the general population about whom we often hope to draw inferences.2,4,5,6,7 At the same time, systematic approaches to transporting effect estimates from one sample to another are being popularized8,9. Many of the statistical tools we use are built from the foundational assumption that our data set comes from a simple random sample. An important contribution from this series will be to highlight and explore this foundational assumption so that we are all better informed on when it is appropriate (and inappropriate) to use certain statistical tools. As far as we know there is not another “hub” focused on sampling let alone cutting edge sampling methods with a methodological and conceptual focus.
4. General Approach. 4 domains/blocks, each 3-months, will include multiple activities allowing for varying participation levels/types, to enhance the value of the initiative for people across DEB. Proposed activities include journal clubs, case studies from diverse areas of research, workgroup participation for new methods/conceptual application, and workshop attendance and problem sets.
Each block will launch with a kickoff in-depth training led by a visiting speaker (either academic or non-academic), paired with a DEB faculty member (if desired) with expertise in that domain. These kickoff workshops will be accessible to DEB trainees, faculty, and staff, and the greater UCSF community, regardless of population research area. The next two months will be 90-min journal-club sessions led by a DEB member. Sessions will have food and the session leader will be paid for facilitating. The journal club sessions will focus on applying the sampling challenge (or workshop topic) for public health research and health disparities research, along with applications of the sampling methodology. In addition to a journal article, participants will be encouraged to submit an issue from their own research. The facilitator can group these issues to inform a set of discussion questions/prompts for the journal club session. We hope this will allow attendees to deepen content area knowledge and apply learnings quickly to their work. These journal club sessions can foster collaboration and provide a platform for members to form workgroups leading to subsequent projects (e.g., manuscripts, commentaries, grants, courses, seminars). Each block will conclude with a meeting to discuss potential spin-off ideas to pursue. The project RA will be charged with compiling the readings (the official journal articles and others raised during discussions), slides and presentations used in discussions, and summary of key ideas from each journal club to feed into the teaching modules (see section 3b). The four blocks are:
(1) Complex sampling designs for representative samples will address design and analysis considerations related to clustering and stratification. We begin with this because it is broadly relevant to nearly everyone in health research, as nearly all major surveillance data sets (e.g., NHANES) use a complex sample design and methods such as inverse probability weighting draw on the same ideas. These designs are especially relevant for disparities research since moving beyond simple random sampling is typically essential to support research on smaller population size groups. This block will address statistical issues when analyzing clustered data.
(2) Sampling for hard-to-reach populations, such as Time-Location Sampling (TLS) and Respondent-Driven Sampling (RDS). Traditional sampling methods rely on an existing sampling frame to draw a representative sample of the target population to generate generalizable knowledge. Many populations of public health importance are socially marginalized (and criminalized) to the extent that an existing sampling frame is infeasible (if not also unethical), e.g., sexual and gender minorities, people who inject drugs, and (in some contexts) women seeking abortion services. This block will discuss recent developments in sampling approaches for hard-to-reach populations, debates concerning the representativeness of these samples, and statistical approaches to derive inference to the underlying target population. We will also discuss how applications of these modern sampling approaches have contributed to health disparities.
(3) Techniques to estimate the dark figure, i.e. selection bias, refers to data points missing from our analyses that, when incorporated, could change our understanding of the situation. In this block, we will examine methods to quantify this figure, particularly in reference to the “denominator problem”, i.e., what is the size of the population at risk (an essential quantity in order to estimate the burden of disease in any population). We will discuss the strengths and limitations of these methods (e.g. capture-recapture/multiple systems estimation), common applications (e.g. addressing undercounts), and analytic options.
(4) Internal and external validity concerns as they relate to pooled data sources, including tools to address inconsistent sampling strategies fielded for different population subgroups such as racial/ethnic minorities. This will include statistical tools for transporting effect estimates to new populations.
5. Anticipated Outcomes.
5.a. Participant Learning objectives include: (1) Understand the purpose of sampling tools (e.g., stratified/ clustered designs, capture/recapture methods, time-location sampling) in population health research. (2) Understand how sample designs affect data utility and generalizability for population research. (3) Evaluate the pros and cons of different types of sampling methods used in population research for various research topics. (4) Learn how sampling should guide analytic decisions and statistical analysis appropriate for responding to select sampling challenges (e.g., transporting effect estimates to new populations). (5) Gain insights about health disparities research and opportunities to overcome sampling challenges through collegial discussions
Table 1. Short-term goals inform the structure, content, and operation of the initiative.
Short-Term Goal | Evaluation Metric(s) |
1. Strengthen relationships with experts outside DEB; increase professional network | Number of outside department speakers and attendees; Number of workgroups formed and number of members/workgroup |
2. Provide resources to help DEB members address sampling challenges in scholarly activities (e.g. grants, manuscripts) | Create, circulate and post citable resources, by topic block |
3. Strengthen the department’s expertise in modern sampling methods for population research | Number of attendees (by workshop, seminar, workgroup) |
4. Increase research rigor | Number of scholarly products (e.g., papers, grants) that apply sampling methods resources |
5.b. Anticipated subsequent outcomes include grants, education/training, core services.
By investing in this “One Big Idea” our department can enhance the quality of future external grants, develop new future training opportunities, and increase the visibility and marketability of our department.
(1) Expertise in sampling methods and approaches to overcome sampling challenges can strengthen any grant application. Examples of funding domains include: (a) understanding disparities in Alzheimer’s disease and related disorders (ADRD) given these studies are often based on highly selected samples from clinical populations leading to a biased picture of the biological processes culminating in ADRD and the drivers of racial disparities in ADRD; (b) Scaling successful interventions to communities with high representation by URM. Methods of transportability are an area of active development and hold promise for solving the well documented problems with the gap between efficacy and effectiveness and non-replication of successful interventions. (c) Grants to increase recruitment/participation in clinical trials and cohort studies to improve the representation of communities traditionally under-sampled. (d) Precision medicine/precision public health initiatives would be strengthened by the incorporation of expertise in sampling methodology to address the premise that individuals have heterogeneous treatment responses.
(2) Training in modern sampling methods is a current gap across UCSF’s educational curriculum. The expertise and content generated by this initiative could efficiently translate to a course, potentially offered online, to fill this gap. Our minimum training goals are to develop teaching modules corresponding with each of the four blocks, generate a list of readings, a recording of the workshop, and problem sets. These resources could be directly applicable to existing courses, e.g., Epi265 (epi methods 3 / chronic disease epi, where sampling is introduced), in the PhD seminar, Bio202 (Intro to Big Data, where public data sets with complex sample designs, such as NHANES, are commonly used), and in BIO215 (where concepts such as IPW, central to classical survey sampling, are used extensively). Other courses may also find these useful. If there is substantial demand, this could be developed into a stand-alone short course offered as part of core epi methods or the Data Science track.
(3) The combination of increasing all member’s knowledge in modern sampling methods, investing existing expert faculty, and providing a set of free educational seminars to the UCSF community will increase the visibility of our department’s valuable skills. Doing so may allow: new consulting services, co-investigator roles on grants, and core services in sampling to complement existing services focused on recruitment.
(4) Our proposal is strategically aligned with DEB’s educational mission. First, it will provide a series of lectures and workshops to further develop expertise in this research area. Second, it will support our goals to recruit excellent junior faculty by providing a resource and center of expertise for those who wish to take advantage of it, e.g., those writing independent grants.
6. Budget. Our $94,800 budget is detailed below. This increase from our initial submission includes additional support from a trainee, and compensation for meeting facilitation to ensure the project proceeds smoothly.
Budget item | Item Amount | Total |
Faculty salary support for Morris, Wesson, and Hoffmann (Glymour in-kind) | $5,000x3 | $15,000 |
Faculty compensation for leading each content block | $10,000x4 blocks | $40,000 |
Journal club facilitation | $500x10 | $5,000 |
External workshop speakers for workshop honorarium | $1,000x4 | $4,000 |
Workshop leaders travel/hotel (assuming one is local) | $2,000x3 | $6,000 |
Food for workshops and journal clubs | $400x12 | $4,800 |
PhD or Postdoc to direct to support content development and organization | Half stipend | $20,000 |
Total | | $94,800 |
7. Timeline. We will organize speakers in 9/19, offer blocks 1/20-12/20, and intense grant-writing starting 6/20.
8. References available here: https://docs.google.com/document/d/1pgqhjGAht5jKOEciAwAPAjV_lHu5KKxR80IYE_xbSag/edit?usp=sharing
Comments
Dear Steve - Thanks for
Dear Steve - Thanks for getting us started with an outstanding idea!
Dear everyone else in DEB. - What do you think of Steve’s idea? Give him some feedback to help improve this concept.
Here’s my question to everyone - What would make you attend or not attend such retreats? The science? The location? The people? The food? the prospect of specific f/u after the retreat? How do we lower the barriers to participation and keep the quality high?
Love this idea and I don't
Love this idea and I don't need a faraway place just slightly remote from MH. Probably half a day - maybe include breakfast but end before lunch. Not sure how to start this, though, would be good to have a proposal for the funding but then have specific assignments to some participants to fertilize the ideas before the retreats.
To me, to really be a retreat
To me, to really be a retreat it would need to be more than 1/2 or even1 day. Really getting away and having down time for free-ranging conversations (perhaps during walks along the beach or in the trees) would make this different enough from usual day-to-day work to be appealing to me. But it would need to be organized by some one good at such things.
I agree that focused time is
I agree that focused time is invaluable to difficult, serious work. I also suspect Tom is right - we will need more than a day to do anything notable. If it's <4 hours, the advantages of short commute time and reliable wifi in MH2500 probably offset the disadvantages of being too close to all the normal demands. Ideally if we were doing a grant, we could get substantial writing and thinking about prelim studies done together during such a retreat. This might be especially valuable for program projects or that kind of sprawling project that needs integration of a lot of disparate parts.
Great idea. I also agree with
Great idea. I also agree with Tom and Maria's comments. What about a DEB competion of teams who have an idea they want to work on and a list of people they want to invite to brainstorm? Winners will go on a 3-4 day retreat to flash the project out and submit an application for funding.
We'll need some electronic
We'll need some electronic device police to confiscate them as you walk in ;-)
Perhaps the other big ideas
Perhaps the other big ideas that seem to be pointing towards eventual program or center grants (e.g. PUMA, the global diabetes program) might adopt this mini-retreat idea as a strategy for accelerating applications for sustainable funding sources within the one-year timeline. The junior faculty/new collaborations idea could also incorporate the mini-retreat idea.
Great idea Steve! And some
Great idea Steve! And some good suggestions about how to implement this. I agree that some of the other Big Ideas could benefit from such a retreat to help accelerate proposals, etc. Designing this in a novel way to focus and engage participants would be great.
Nice idea. I would be
Nice idea. I would be interested in attending one on digital/mHealth... which then makes the collection of devices at the door kind of hard... ! ;)
To Chuck's point - I observed at a high school recently that before class, the teacher collected all phones/devices in cubby containers, then returned them 5 min before the end of class.
What would attract me to attend one? the people, the topic, the location, and food. I understand John's lab does something like this annually...?? Maybe he has further tips on how to make it fun and productive?
I think the last line, about having to report back in is key, and we should have some structure or plans on what tangible product will come out of the retreat. While I like the part about getting away and brainstorming, I think it's important to have some a priori planned HW that will be turned in and reported back. I think some of us work better when there is something to turn in...
To answer KBD, I very much
To answer KBD, I very much enjoyed last year's DEB retreat so am content to entrust you with the format. For me, once a year is enough for these oppotunities to connect, brainstorm, and relax together.
I am more interested in collaborations that jump-start super-productive research. Check out my proposal of "platform studies." I'd love your good ideas.
Great idea! Could take a
Great idea! Could take a quick survey for those interested in each retreat or overall as to specifics that would increase attendance.
I like the idea of a faculty
I like the idea of a faculty retreat as well. For me, it would be better as a whole day (and perhaps overnight if we could afford it) experience to really get people relaxed and focused on each other. I agree that we need some sort of draconian measure to get people away from their screens. Key point for me, however, is that over the years I have noticed that there is not much interdepartmental collaboration on research projects. We tend to work with colleagues outside the DEB and even outside UCSF. If the retreat could be organized to stimulate more intradepartmental collaborations (e.g., program projects perhaps), I think it would be great for scientific as well as social reasons.
Still like this one. My only
Still like this one. My only additional comment is that even with an overnight or day and a half it would be important to select ONE topic to focus on. Time will pass quickly and having enough of it to produce a good product would be important. KBD's questions about operations and process still are important considerations.
Bob
Love the idea of retreats.
Love the idea of retreats. Similarly to a few of the other proposals, I get a bit stuck on funding a proposal to fund other proposals. I do wonder if this might be a target of its own big idea open proposal or say a "Retreats" open proposal where folks could pitch their ideas? Feels like this should be something we are just planning to do yearly as a department.