Appendices

Data Quality Indicator (DQI) Excel-based Tool

This DQI tool is freely available for download at http://www.icbdsr.org/data-quality-indicators-tool/. The tool is an Excel file, where a programme can enter its surveillance data and the DQI are automatically calculated. This tool is especially useful for programmes in low-resource settings (e.g. hospital-based with short follow-up, typically until discharge from birth hospital).

The DQI tool focuses on four key surveillance processes: (1) ascertainment, (2) description, (3) coding and (4) classification. A total of 40 quality indicators covering key surveillance processes are included in the Excel tool.

(1) Ascertainment DQI

There are two types of ascertainment DQI: threshold values and expected ordering of birth prevalence.

Threshold values:

• These values compare the observed prevalence of specific birth defects with a “threshold prevalence”, below which under-ascertainment is likely.
• The thresholds are based on the 2003–2012 annual ICBDSR reports for oral clefts, neural tube defects, abdominal wall defects, hypospadias, limb deficiencies and microtia/anotia.
• For the distribution of prevalence in ICBDSR programmes, the first quartile is arbitrarily defined as the threshold value.

In the following bar chart, the ICBDSR programmes are ordered by prevalence of oral clefts. The threshold value is the 1st quartile or 25th percentile.

Prevalence of oral clefts per 10 000 births

Expected ordering of birth prevalence:

Based on the published literature, it is expected that some subtypes are more frequent than others. For example:

• Cleft lip and palate > cleft palate > cleft lip
• Spina bifida > anencephaly > encephalocele
• Ratio of spina bifida/anencephaly = 1.00–1.33 (acceptable range)
• Gastroschisis > omphalocele
• Gastroschisis among mothers under 20 years old > 20 years old and above
• Transverse limb defects > preaxial defects > postaxial defects
• Upper limb reduction defects > lower limb reduction defects
• Postaxial polydactyly > preaxial polydactyly
• Down syndrome among mothers 35 years and older > under 35 years old.

(2) Description DQI

Description DQI indicate how accurate and complete the description and documentation of birth defects are by the local staff using pre-established criteria. For example, some components of accurate and complete description are:

• Oral clefts: Laterality and extension
• Spina bifida: Level of lesion and skin coverage
• Encephalocele: Site (i.e. occipital, temporal, frontal, etc.)
• Abdominal wall defects: Localization, relation to umbilical cord, membrane covered
• Hypospadias: Degree specified (first, third; if programme includes first degree, assess separately)
• Limb reduction defects: Site (limb and segment) and type (preaxial, postaxial, etc.)
• Feet deformities: Rotation and position
• Congenital heart defects: Type
• Microtia: Degree (second, third, fourth – anotia)

(3) Coding DQI

The coding DQI evaluate if there are errors in the coding process. Some examples include:

• Medial cleft lip miscoded as a typical cleft
• Cleft lip and palate coded using two codes (Q35 & Q36)
• Hypospadias coded as ambiguous genitalia (as the only diagnosis)
• Gastroschisis coded as a ruptured omphalocele.

(4) Classification DQI

Classification DQI evaluate the distribution as isolated, multiples (multiple congenital anomalies [MCA]), and syndromes, for those birth defects whose typical distribution is well known from the literature. For example:

• Percentage syndromic and MCA cases in cleft palate > in cleft lip with or without cleft palate
• Percentage syndromic and MCA cases in encephalocele > in spina bifida
• Omphalocele: Percentage of MCA and syndromic cases > isolated
• Gastroschisis: Percentage isolated > 80%
• Hypospadias: Percentage isolated > 80%

These DQI of ascertainment, coding, classification can be proportions, sentinel or ratios.

• Proportion-based DQI: Expressed as percentage. The numerator is the number of cases with an accurate description of a birth defect, and the denominator is the total number of cases with that birth defect. For example, the proportion of cleft lip cases with laterality specified.
• Sentinel DQI: A sentinel indicator identifies the occurrence of events that are intrinsically (always) undesirable, and that should trigger further analysis and investigation. These undesirable events likely reflect a deficiency in data quality (for example, very low prevalence of a birth defect might be due to under-ascertainment). Sentinel indicators were marked as “achieved” or “not achieved”.
• Ratio DQI: Quotient between two prevalences. For example, the ratio of spina bifida/anencephaly.

The DQI tool includes the following birth defects:

• Oral clefts
• Neural tube defects
• Omphalocele
• Gastroschisis
• Hypospadias
• Limb deficiencies
• Talipes equinovarus
• Critical congenital heart defects
• Microtia/anotia
• Polydactyly
• Down syndrome

The DQI for oral clefts included in the Excel tool are shown below.

Note: In the rightmost column of this and the following tables, the codes refer to the number of cases with that code. For example, “Q35.x + Q36.x + Q37.x” means “number of cases coded Q 35.x + number of cases coded Q36.x + etc.”

 Top of Page

The first column lists selected birth defects. The second column describes the data quality indicator. The third column is for the process being evaluated (ascertainment, description, coding, classification). For oral clefts, the table includes DQI for the ascertainment, description and coding processes. In the fourth column, the type of DQI is specified: sentinel or proportion of ratio. In the fifth column, there is a general definition of the DQI. In the last column, the DQI is defined operationally, using ICD-10 codes.

The first indicator is for total oral clefts. It shows a minimum prevalence indicator, evaluating the ascertainment process. The type is sentinel, because the result can be “achieved” or “not achieved.” The definition is a threshold; that is, achieving this indicator means that prevalence of total oral clefts for a programme should be higher than 11.1 per 10 000 births. If prevalence is below that threshold, under-ascertainment is possible.

The third indicator shown focuses on cleft lip. It is a description DQI since it evaluates the proportion of cleft lip cases with specified laterality. The type is proportion, and it is expressed as a percentage.

The next example shows DQI for neural tube defects (NTD).

There are three ascertainment indicators and two description indicators. The ascertainment indicators include a minimum prevalence of NTD of 5.7 per 10 000 births. The prevalence of spina bifida should be highest, followed by the prevalence of anencephaly, and then prevalence of encephalocele.

The last row shows the prevalence ratio spina bifida/anencephaly indicator, which is an ascertainment indicator to evaluate under-ascertainment of anencephaly. Also, the indicators for spina bifida evaluate how accurate the description is regarding level and skin coverage.

Using the Excel DQI tool

The Excel DQI tool allows the systematic evaluation of the quality of data obtained through surveillance of birth defects. It is available for download at http://www.icbdsr.org/data-quality-indicators-tool/.

The Excel DQI tool has four worksheets:

1. Frequently asked questions
2. Input dashboard
3. DQI report
4. DQI definitions.

The first tab of the Excel DQI tool contains frequently asked questions about the tool and its use.

Input Dashboard

The second tab contains the worksheet for inputting data. This is the only tab where the user is asked to enter information.

The user has to enter the total number of cases, classified by clinical presentation (isolated, multiple, syndromes); the number of births by maternal age groups; and the number of cases by specified descriptions for selected conditions.

Table 1 allows the user to enter the number of cases and clinical classification (isolated, multiple, syndromes) of selected birth defects. These data are used to generate the ascertainment DQI and the classification DQI.

Table 2 allows the user to enter the number of births and Down syndrome cases by maternal age groups. These data are used to compute prevalence for the ascertainment DQI.

Table 3 allows the user to enter number of cases of selected birth defects with description specified; for example, oral cleft cases by laterality. These data are used to calculate description and coding DQI.

The following image shows “Table 1. Clinical classification and prevalence of selected birth defects.”

 Top of Page

This table has two types of cells. The green cells are for entering data. The grey cells (with the division by zero error: #¡DIV/0!) will automatically calculate the results once data are entered in the green cells from this table as well as from Table 2.

The following is “Table 2. Number of births and Down syndrome cases by maternal age” of the input dashboard.

For each maternal age group, the table computes and compares the observed number of cases with Down syndrome (live births plus elective terminations of pregnancy [ETOP]), adjusted for spontaneous fetal losses that would have occurred if the pregnancy had been allowed to continue against the expected number of cases, based on the best available published data (the expected number is thought to depend exclusively on maternal age).

If the user does not have ETOP information, the user can leave the cells of the ETOP columns empty. This information is used to evaluate the DQI observed/expected ratio of Down syndrome. The result of this DQI will be shown in the DQI Report Worksheet of the tool.

Once the user inputs data in Table 1 and Table 2, the cells show the results for clinical classification and prevalence per 10 000 (see image).

The following is “Table 3. Number of cases with description specified” in the input Dashboard. The data are used to compute the description DQI and the coding DQI, shown in the DQI Report Worksheet.

 Top of Page

For example, oral clefts description DQI focus on laterality and extension (hard and/or soft palate). For NTDs description DQI, an accurate description should include details on skin coverage and lesion level of spina bifida, as well as localization of encephalocele.

DQI Report Worksheet

The third tab in the Excel tool is the DQI Report Worksheet, which has two tables: Table 1 “Summary of Data Quality Indicators, by process” and Table 2 “Report of Data Quality Indicators”.

Table 1 is a summary with the total number of indicators by process, and the proportion marked as achieved.

Description indicators are defined as “achieved” when the proportion of cases with a good description is greater than 80%.

Table 2 “Report of Data Quality Indicators” is the main table of the DQI Report Worksheet. This table shows the DQI results for each specific anomaly. The following image presents Table 2 before data are entered.

The table includes information on the birth defect, DQI, surveillance process evaluated, and indicator definition. Once the user completes the Input Dashboard worksheet, then the “blank” and error (#DIV/0!) cells will show the outcome for each DQI.

To illustrate the use of the Excel DQI tool, data from the National Network of Congenital Anomalies of Argentina (RENAC) are presented (see table on p. 234). The outcome column shows the results for each indicator. The achieved indicators are marked in green. The indicators that were not achieved are marked in red. For proportion indicators, “not achieved” are those with a result lower than 80%. The last column shows the actual data used for each DQI. This is useful to know for how much the DQI was not reached.

DQI Definitions Worksheet

The fourth tab presents the Definitions Worksheet that includes the type of indicator (sentinel, proportion or ratios) and the operational definition with the codes and operations used to compute the DQI.

Interpretation of the results

The basic goal of DQI is to make explicit and visible the metrics of data quality. This is a starting point: Programme staff and data users can use these metrics to continuously track “what is going on?” Asking “what is going on?” is the fundamental question of quality assessment that needs to be answered before quality improvement can start.

In general, DQI should be interpreted as indicators that show where further investigation is needed:

• Data quality is often but not always the driver: The variation in certain DQI might in part reflect true variation over time or by geography. In interpreting DQI, ideally use specific reference or population data.
  • Example: Hypospadias prevalence in Latin America is likely lower than in Europe (57, 58)

• DQI make the data more visible and are helpful to:
  •  document and track the quality of the programme
  •  target and support quality improvement efforts
  • compare different sites or data sources within a programme
  • interpret variation between programmes
  • enhance the information from the programme data when reporting to public and stakeholders.

The DQI should be interpreted in the context of the programme methodology.

• Example: Congenital heart defects (CHD)
  • If a programme only ascertains cases diagnosed up to discharge from the birthing hospital, then a fraction (considerable for some heart defects) would be missed, and the associated prevalence would be spuriously low.
  • Also, if a paediatric cardiologist is not available for diagnosis/case review/coding, then this lack of clinical input could be reflected in the CHD-related DQI.

DQI findings can suggest and support practical actions to improve data quality.

For example, a programme might develop:

• regular, timely, constructive feedback to hospitals and providers
• meetings with staff at local hospitals to discuss, learn, train
• tools such as checklists to promote the systematic description of all key features of the specific case or anomaly.

After the interventions, use the DQI to track the impact of these quality improvement interventions. Finally, quality control and quality improvement are different but complementary.

 Top of Page