Real World Testing Results 2023 (RWT)
General Information
| Developer Name | CloudCraft Software |
| Product Name | CloudCraft Software |
| Version Number | 9.0 |
| Certified Health IT Edition | 2015 Cures Update |
| Product List (CHPL) ID | 15.04.04.3071.Clou.09.01.1.221227 |
| Real World Testing Public URL | https://cloudcraftsoftware.com/realworldtesting.html |
Changes to Original Plan
| Summary of Change | Reason | Impact |
|---|---|---|
| Test data instead of production data was used in testing (b)(1) Transitions of Care. | Providers are still largely dependent on the fax machine for transferring records. There was only one instance of a production user performing a TOC electronically. | While the single electronic TOC
was demonstrably successful,
we chose to perform several
more TOCs using synthetic data
in order to fully capture the
system’s functionality.
We will also be implementing strategic policies and training plans to encourage more providers to adopt the electronic method, as it is faster, more secure, and more accurate. |
| CCDAs to ACOs versus qrda files | The ACO specifications are not ingesting qrda files, only CCDAs | Since the ACOs are monitoring care gaps and assisting our customer in closing those gaps through daily huddle sheets, the customers are not using these features in CloudCraft in 2023. |
| Pre-emptive CQMs for providers to pre plan for patients on the calendar in order to close care gaps. | There is a latency for ACOs to ingest data and provide daily huddle sheets and analytics. | CloudCraft, in 2023, shall highlight patient CQMs real time during the visit. This will remind doctors to cover these CQMs at the point of care, and auto-order procedures that close these gaps through Healthcare Maintenance features. |
| Serve as an audit against the ACOs data. | Provide analytics that compare with the ACO analytics for auditing purposes | Sending data to the ACOs for ingesting is fairly a black box. We know what we sent, but the analytic output needs to be audited. |
| Verified CQMs in Cypress rather than with live data. | See above for current implementation with ACOs. | Test completed with test data rather than production data. |
| Verified Health Care Surveys against NIST tool. | The functionality has not yet been adopted widely in production. | Results reflect test data rather than production data. |
| Altered the test procedure for Health Care Surveys from what is presented on the RWT plan. | The planned steps were based on a previous understanding of f7; the actual testing was in line with the current ONC standard. | The steps completed during testing do not match what was on the RWT plan. |
| Verified eCR files are created via trigger codes and are compliant to standard. | Electronic case reporting is not currently being used by trading partners. | Test completed with test data rather than production data. |
Summary of Testing Methods and Key Findings
|
Accurate and secure transmission of patient health information is paramount to ensuring interoperability
between different health IT solutions. We chose to demonstrate this capability in CloudCraft by
performing transitions-of-care for a number of patients directly from internal provider to external
provider, by bulk exporting a batch of C-CDA files containing patient data, transmitting public health
data to appropriate registries, and accessing patient information via FHIR API. In each scenario, we
confirmed that the data transmitted and/or exported was accurate, up to date, and in conformance to
ONC standards. Though our trading partners are not currently transmitting Electronic Case Reporting functionality to public health registries, we have ensured that the functionality is in place in preparation for future adoption. There is a patient check out event which automatically checks against a table of “trigger codes” composed of diagnoses, test results, and lab orders, and if the trigger event results in a positive match of any of these codes, an eCR document is automatically created. This document has been verified as compliant with the latest standards. Additionally, future plans include utilizing the eCR Now FHIR® application to deliver data to the CDC. Several patients were successfully accessed via the app MyLinks utilizing the Dynamic FHIR API. All patient information was thoroughly inspected to ensure the data shown in MyLinks was identical to the original data in CloudCraft. No discrepancies or missing data were found. The below criteria were tested: § 170.315(b)(1) Transitions of care § 170.315(b)(2) Clinical information reconciliation and incorporation § 170.315(b)(6) Data export § 170.315(c)(1) - Clinical quality measures (CQMs) — record and export § 170.315(c)(2) - Clinical quality measures (CQMs) — import and calculate § 170.315(c)(3) - Clinical quality measures (CQMs) — report § 170.315(f)(1) Transmission to immunization registries § 170.315(f)(5) Transmission to public health agencies — electronic case reporting § 170.315(f)(7) Transmission to public health agencies — health care surveys § 170.315(g)(7) Application access— patient selection § 170.315(g)(9) Application access— all data request § 170.315(g)(10) Standardized API for patient and population services § 170.315(h)(1) Direct Project |
Care setting(s) that were tested for real world interoperability
| Ambulatory |
Voluntary updates to the standards and implementation specifications approved through SVAP (if applicable)
| N/A |
Metrics and Outcomes
| Measurement/Metric | Associated Criteria | Outcomes | Relied Upon Software |
|---|---|---|---|
| 80% or more of outbound TOC's successfully received by HISP | (b)(1) Transition of Care (h)(1) Direct Project: from the Electronic Exchange Category |
75% successfully received (172/256) | ConnectEHR |
| 75% or more of trading partner's TOC C-CDAs are successfully incorporated by SUT. | (b)(1) Transition of Care (b)(2) Clinical information reconciliation and incorporation (h)(1) Direct Project: from the Electronic Exchange Category |
93% successfully incorporated (574/616) | ConnectEHR |
| Work towards 10% of patients where a referral is initiated complete the referral loop using DIRECT. | (b)(1) Transition of Care (h)(1) Direct Project: from the Electronic Exchange Category |
100% completed referral loop (7/7) | ConnectEHR |
| 100% of Exports ran timely, or a user friendly error will be provided. | (b)(6) - Data export | 100% of exports ran at the configured time | ConnectEHR |
| C-CDA count matches actual patient count for requested selection criteria. | (b)(6) - Data export | All patients were requested for export, and all C-CDAs were generated | ConnectEHR |
100% matching data
elements in
CQMsolution vs EHR.
This will be confirmed
by visual validation of
the following data:
|
(c)(1) - Clinical quality measures (CQMs) — record and export | All data for all test patients matched what was received in CQMsolution. | CQMsolution |
| 100% discrepancy will be identified thru Quarterly Reconciliation of measures calculation between ACO calculations and SUT calculations | (c)(2) - Clinical quality
measures (CQMs) —
import and calculate (c)(3) - Clinical quality measures (CQMs) — report |
Calculation of measures in CQMsolution matched the Cypress calculation when the test patient QRDA files were uploaded to Cypress. | CQMsolution |
| 100% correct immunization history records successfully received in EHR confirmed by visual validation. | (f)(1) Transmission to immunization registries | 100% completed | ConnectEHR |
| Successful Transmission to Public Health Registry will be reviewed for ACK & NAK to ensure 100% successful transmission. | (f)(1) Transmission to immunization registries | 100% completed | ConnectEHR |
| 100% correct immunization records successfully posted to registry confirmed by visual validation. | (f)(1) Transmission to immunization registries | 100% completed | ConnectEHR |
| 100% of eCR
messages successfully
received and processed
by public health agency
based on either: a) Logging into agency web site and validating, or b) Using a report provided by agency |
(f)(5) Transmission to public health agencies — electronic case reporting | 100% of eCR
messages triggered
appropriately and are
valid according to
current standards. Metric adjusted due to pending production adoption by trading partners. |
ConnectEHR |
| 100% of providers configuring a survey will have the ability to configure in the EHR. | (f)(7) Transmission to public health agencies — health care surveys | ||
| 100% of the survey questions shall be available for management analytics. | (f)(7) Transmission to public health agencies — health care surveys | ||
| 100% of all non-aggregated survey data shall be aggregated on a supervisory board portal, for board oversight. | (f)(7) Transmission to public health agencies — health care surveys | ||
| Patient is able to retrieve FHIR API data from PHR app for 100% of encounters. | (g)(7) Application
access— patient
selection (g)(9) Application access— all data request |
100% data retrieved. | ConnectEHR Dynamic FHIR MyLinks |
| In 100% of encounters from Step #1, PHR data matches data from EHR. This will be confirmed by visual validation of CCDS. | (g)(7) Application
access— patient
selection (g)(9) Application access— all data request |
All information matched. | ConnectEHR Dynamic FHIR MyLinks |
| 100 percent of encounters where Patient is able to retrieve FHIR API data from PHR app. | (g)(10) Standardized API for patient and population services | ConnectEHR Dynamic FHIR |
|
| 100 percent of encounters
from Step #1 where Patient’s
PHR data matches data from
EHR. This will be done by
visual validation of the
following FHIR resources: a. Demographics b. Problems c. Medications d. Allergies |
(g)(10) Standardized API for patient and population services | ||
| 100 percent of encounters where Provider is able to retrieve FHIR API data from app. | (g)(10) Standardized API for patient and population services | ||
| 100 percent of encounters
from Step #3 where data for
randomly-selected patients as
presented in app matches
data from EHR. This will be
done by visual validation of
the following FHIR resources: a. Demographics b. Problems c. Medications d. Allergies |
(g)(10) Standardized API for patient and population services |
Key Milestones
| Key Milestone | Care Setting | Date/Timeframe |
|---|---|---|
|
Ambulatory | May 2023 |
|
Ambulatory | June 2023 |
| Recipient uses scorecard to grade C-CDA | Ambulatory | July 2023 |
|
Ambulatory | August 2023 |
|
Ambulatory | September 2023 |
| Calculate and compile metrics | Ambulatory | September 2023 |
|
Ambulatory | July 2023 |
| Use the Edge Test Tool to check validity of output file(s) | Ambulatory | July 2023 |
| Calculate and compile metrics | Ambulatory | July 2023 |
|
Ambulatory | July 2023 |
| Validate that immunization interface is functioning as expected | Ambulatory | July 2023 |
| Verify immunization data was received in registry. | Ambulatory | July 2023 |
| Verify immunization data was received in EHR. | Ambulatory | July 2023 |
| Calculate and compile metrics | Ambulatory | July 2023 |
| eCR messages were successfully received and processed by public health agency. | Ambulatory | July 2023 |
| Functioning eCR interface to public health agency | Ambulatory | July 2023 |
| Calculate and compile metrics | Ambulatory | July 2023 |
|
Ambulatory | August 2023 |
| The file should upload and be accepted by the environment without error. | Ambulatory | August 2023 |
| All populations of all measures should match. | Ambulatory | August 2023 |
| Calculate and compile metrics | Ambulatory | August 2023 |
|
Ambulatory | August 2023 |
| Encounter is created and visually confirmed | Ambulatory | August 2023 |
|
Ambulatory | August 2023 |
| Visually validate Assessment, Plan of Treatment and Health Concerns narrative text | Ambulatory | August 2023 |
| Calculate and compile metrics | Ambulatory | August 2023 |
|
Ambulatory | July 2023 |
| Responses are converted to HL7 CDA documents for transmission. | Ambulatory | July 2023 |
| Calculate and compile metrics | Ambulatory | July 2023 |