| Metric | Data is By | Quartile 1 | Median | Quartile 3 |
| # of eSource builds per Study | Study | 2 | 6 | 12 |
| % of visits that are signed off by PI | Visit | n/a | 46% | n/a |
| # of internal comments left per completed visit | Visit | 1 | 1 | 2 |
This blog post is the fourth in a series on site performance KPIs based on an analysis CRIO performed for the Site Accreditation and Standards Institute (SASI), a nonprofit organization that serves as a quality accreditation organization for clinical research sites. Click here to learn more about how this analysis was performed.
The protocol-guided source preparation process, as elaborated in this Tufts study, and the source data collection process, form the backbone of a site’s operations. These processes consume most of the site’s time and resources, are the basis for nearly all the data that goes into the EDC, and are the root of PI oversight.
Below are three critical facts about the entire site source process. While these have been anecdotally described, we can now – with our unique eSource-enabled metrics – finally begin to quantify these elements of the process.
- The source build process is iterative and dynamic, with more versions than protocol versions, due to protocol and eCRF ambiguity
Sites often update the source template mid-study. Our data shows a median of 6 versions per study, with top-quartile studies having 12, and low-quartile studies having 2. A clear driver of this is that protocol amendments drive the need for new source templates.
However, protocol amendments alone don’t explain the multiple source versions a site creates. A Tufts study shows that the average number of protocol amendments is now 3.3 per study, but as CRIO’s data demonstrates, the median study has nearly twice as many source versions. Many of these source template iterations represent modifications of the template within the same protocol version, where the site modifies the template by adding clarifying information or questions. Often, these modifications stem from the fact that the site has realized they did not properly interpret a protocol the first time through, or they receive clarifying information from the sponsor or CRO on a certain procedure.
“Having source written strictly based on the protocol may differ from how you actually collect the data – e.g., the act of data collection might identify the need for new clarifications or reminders. Each alert or reminder may require a separate version depending on when your team determines the need for them,” said Bryn Berkowitz, CRIO’s Sponsor Project Manager and former lead coordinator at an orthopedic site. Bryn goes on to explain that at her site, they would often do a mock walk-through of a subject to anticipate these changes.
Not only does the act of operationalizing the protocol itself surface a need to modify the template, but the sites often act in an information vacuum, with lack of full transparency into the EDC data requirements. “The eCRF guidelines are not always clear; sometimes there are dropdown choices that appear in the EDC that weren’t fully transparent in the guidelines. I’ve even had situations where the CRF guidelines were issued after we started screening,” said Claire Biesecker, CRIO’s Site Enablement Specialist and former research coordinator at a large urology practice.
Check out the video below where Greg Manning, Vice President of Operations at Atlas Clinical Research and Aneesh Vaze, Managing Director at Clinical Research Philadelphia, discuss the source template preparation process and the factors that drive both the difficulty of preparing source document templates and those that drive the need for revisions.
This lack of transparency and the amount of redundant work sites do designing and revising source templates is one of the biggest pain points that CRIO’s Central eSource offering addresses. In Central eSource, CRIO’s expert team of source designers work collaboratively and intensively with the sponsor or CRO, and in harmony with the eCRF design, to build a comprehensive, high quality template that generates consistent quality for the participating sites. Sites can then use CRIO’s local template configuration to layer in their site-specific requirements. Without Central eSource, the quality of the data entering the EDC can vary widely, if for no other reason than that sites interpreted the protocol differently and designed the templates accordingly.
- The source is where the PI documents his or her oversight
CRIO has a feature where sites can enable PI signature on completed source visits. Sites can turn this on or off at the study or visit level. Our data shows that on average, 46% of completed visits in CRIO have a PI signature affixed to them.
This includes all types of visits, including “visits” that are technically part of a bigger Scheduled Visit (e.g., a site might break out the Xray component of V1 to a separate V1a), phone visits, and ad hoc unscheduled visits. Many sites choose to turn off the PI signature requirement for these less important visits, where critical clinical information is not collected.
In the majority of cases, the PI is signing the visit on the same or the next day following completion. This PI signature evidences the PI’s review of the source, and often contains a progress note documenting the PI’s clinical observations, such as the patient’s willingness to participate, the patient’s medical history or eligibility, or safety observations.
These contemporaneous PI signatures are critical to documenting active PI oversight. CRAs and auditors routinely look at the site’s source records for evidence of PI oversight as opposed to the PI signature in the EDC, which is usually perfunctory and many months after the fact.
“The lack of a PI signature on a major observation such as an AE would be a serious red flag for any CRA or auditor,” said Paul Neff, CRIO’s Site Enablement Specialist and a former site manager at a large network. By contrast, “the CRAs view the PI signature in the EDC as a checklist item, and no one would care if it took an extra week or two to obtain their signature there.”
- The source is a collaborative process among site personnel
Site source isn’t just straightforward data such as weight; they often contain observations that require a certain level of clinical judgment, such as eligibility, the assessment of the severity of an Adverse Event, or the proper description or diagnosis of a condition. These data points are often done collaboratively, and to facilitate intra-site collaboration, CRIO has created a workflow whereby a site member can send a “comment” another member about a data point.
Site staff often use this feature to flag data that is still pending – e.g., the coordinator might append to a blank value on “Dose” a comment like “The patient can’t recall the exact dose and will call me back when she gets home.” Sub-Investigators and Coordinators often use this feature to flag something for the PI to review, such as an Adverse Event or a Diagnosis.
CRIO’s analysis shows that the median and bottom-quartile visit has 1 internal comment, while top-quartile visits have 2. In fact, 94% of visits have at least 1 comment. 12% have 4 or more comments, and 2% have 10 or more. This indicates that the remote collaboration feature in source creation is a critical feature for an eSource platform.
An intra-site source data communication workflow is just one of many features CRIO custom-built for sites in its eSource platform. This is one of the many features that differentiate eSource from an EDC system, which have been designed primarily for Data Management. It underscores the importance of sponsors working with eSource providers who have a proven track record of broad adoption by clinical research sites themselves.