1.	Introduction
This document defines the data validation, editing, and estimation techniques required to participate in the California market as an MDMA.
2.	Required data validation tests
Data validation checks are designed to identify things that can go wrong at the meter/recorder and cause the data collected not to reflect actual usage.  
The following tests are required for kWh and kVARh interval data and are described in section 3 of this document:

Test | Purpose
Time check of meter reading device/system | Check for time drift of meter reading device/system outside CPUC standard
Time check of meter | Check for time drift of meter clock outside CPUC standard

Pulse overflow check

Check for the following:
•	improper scaling factor in meter 
•	improperly sized transformer
•	hardware problem

Sum check

Check for the following in combination meter/recorder installations: 
•	crossed channels between meter and recorder
•	pulse relay problems  
Check for the following for all installations: 
•	invalid PT and CT ratios, 
•	invalid meter constants

Spike check

Check for the following for all installations:
•	transmission error 
•	spike resulting from meter test.  
Note that a spike can also occur after an outage - in this case the data is valid, but may or may not be used for peak billing depending on the tariff and company policy.

High/low check

Check for the following in all installations: 
•	dropped phases 
•	inaccurate meter constants
•	energy diversion
•	fast/slow meters
Also check for the following in combination meter/recorder installations:
•	erratic pulse input to recorder

Meter identification check

Check for the following:
•	that the meter ID was reported correctly
•	the meter has not been changed out
•	the data is being reported for the correct meter

kVARh check (for kWh data only if corresponding kVARh data available)

Check that:
•	kWh channels are correctly mapped to kVARh channel
•	meter is operating correctly

Test mode check

Check for:
•	Data collected when meter was in test mode that represents test load rather than actual usage

The following tests are required for monthly data validation for kWh and kVARh where required.  They are discussed in section 5 of this document.
Test | Purpose
Time check of meter reading device/system (applies to devices/systems collecting TOU data only) | Check for time drift of meter reading device/system outside CPUC standard
Time check of meter (applies to meters collecting TOU data only) | Check for time drift of meter clock outside CPUC standard

High/low usage check

Check for the following:
•	misread
•	fast/slow meter
•	broken meter
•	incorrect multipliers
•	energy diversion
•	dropped phases

High/low demand check (applies to demand readings only)

Check for the following:
•	misread
•	fast/slow meter
•	broken meter
•	incorrect multipliers
•	energy diversion
•	dropped phases

Time-of-use check (applies to TOU data only)

Check for the following:
•	misread
•	fast/slow meter
•	broken meter
•	incorrect multipliers
•	energy diversion
•	dropped phases

Zero consumption for active meters

Check for the following:
•	energy diversion
•	meter doesn’t register

Usage for inactive meters | Check for unauthorized usage at a site for which there is no customer with financial responsibility

Number of dials on meter

Check for:
•	wrong meter
•	misread

Dial decimal quantity

Check for:
•	wrong meter
•	misread

Meter identification

Check for the following:
•	that the meter ID was reported correctly
•	the meter has not been changed out
•	the data is being reported for the correct meter

3.	Interval data collection and validation rules
These rules apply to both kWh and kVARh data, depending on the data required by the meter’s tariff(s).  General MDMA and MSP business practices should ensure that the meter is programmed correctly for the required revenue data, and that the MDMA is set up appropriately.  This includes information such as interval size, meter constants, and what quantity is recorded by what channel.  These VEE rules do not require or describe how the MDMA verifies that the meter is programmed correctly. 
Note that if interval data is read more often then required for billing, checks need to be performed at different times in the process.  Some must be done as the data is read from the meter; some can be done anytime between when the data is collected from the meter and the end of the cycle, and others have to be done on a billing period basis at the end of the billing cycle.  They are broken out that way in this description.
3.1.	Time check of  meter reading device/system - ensure that the collection device is synchronized to the national time standard before data collection begins
3.2.	Collect data
3.3.	As data is collected, 
3.3.1.	Check meter identification - verify the meter’s identification matches what is expected
3.3.2.	Do time tolerance check of meter.  Note that depending on the communication technology used, network latency must be taken into account.
3.3.2.1.	If time in meter is within 3 minutes of time standard, the data has passed Time Tolerance check.  (Note that if the meter is within 3 minutes of the standard, the time in the meter can optionally be corrected.)
3.3.2.2.	If time in meter is off more than 3 minutes but <= 75 minutes, and the meter does not have the expected number of intervals, prorate data.  (Note: meter time must be corrected.  If the meter fails the time check for three consecutive months, the meter must be physically inspected/tested)  The data to be prorated includes all intervals from the last time the meter’s time was determined to be good (ie, within the 3 minute tolerance) and when it was discovered that the meter’s time was off more than 3 minutes and the meter was re-synched.  Refer to Appendix A for examples of how to prorate data.
3.3.2.2.1.	Compare the number of intervals provided by the meter during the time period in question with the actual number of intervals expected.  If the two numbers are the same (ie, the meter has the correct number of intervals), do not prorate the data. The intervals are good.
3.3.2.2.2.	If the meter does not have the correct number of intervals, 
3.3.2.2.2.1.	Refer to Appendix A for a description of how to prorate the data.  Mark intervals as estimated.
3.3.2.3.	If time in meter is off more than 75 minutes, all intervals between the last time you verified the meter’s time was good (ie, within 3 minute tolerance) and when you discovered the meter’s time was off more than 75 minutes need to be estimated.  (Note: meter time must be corrected.  If the meter fails the time check for three consecutive months, the meter must be physically inspected/tested.)
3.4.	Either as data is collected or prior to publishing on MDMA Server…
3.4.1.	Perform pulse overflow check
3.4.1.1.	Inspect each interval for this condition.  If a pulse overflow occurs, need to mark meter for physical meter test/maintenance. The interval that has a pulse overflow needs to be estimated. Continue VEE process.
3.4.2.	Perform test mode check
3.4.2.1.	If MDMA determines meter was in test mode, the MDMA must ensure that the customer is not billed for the test load.  Zero data reported in intervals while the meter is being tested are left as zero and considered good data.  If the meter is inadvertently left in test mode, the data must be estimated.
3.4.3.	Perform sum check
This check may be done against either consumption or pulse data, provided the data scaling is consistent throughout the period.
3.4.3.1.	Sum the intervals for the time between the start and stop meter readings.  
3.4.3.2.	Compare sum of the intervals of with difference between start and stop readings (taking into account possible rollover between start and stop readings - for example, if the meter reading rolls over at 99999, the start reading was 99968 and the stop reading was 00294, the difference would be 326).  Note that the sum and the readings must be in the same units for the comparison.
3.4.3.3.	If difference is <= two meter multipliers (meter multiplier = ctr x vtr, where ctr is current transformer ratio and vtr is voltage transformer ratio) - data has passed Sum Check. 
3.4.3.4.	If difference is greater than two meter multipliers, optionally, rerun the sum check, taking into account the differences in time between the time of the start read and the start of the first interval, and the time of the stop read and the end of the last interval.
3.4.3.4.1.	Calculate a new “start reading” just to be used in this check by doing the following:
3.4.3.4.1.1.	Calculate the percentage of an interval that has elapsed between the begin time of the first interval and the time of the start reading.  For example, if the first interval in an hourly interval data stream starts at 3 pm, and the start reading was performed at 3:20, this percentage would be (20 minutes / 60 minutes) = 33%.
3.4.3.4.1.2.	Calculate the new start reads by subtracting the percentage times the usage in the first interval.  For example, if the start reading was 55555 and the usage during the first interval was 45, then adjust the start reading to 55540.
3.4.3.4.2.	Calculate an allowable margin of error by doing the following:
3.4.3.4.2.1.	Calculate the percentage of an interval that has elapsed between the end time of the last good interval and the time of the stop reading.
3.4.3.4.2.2.	Multiply the percentage by the usage in the last interval in the data stream.
3.4.3.4.2.3.	Add two multipliers to the value calculated in the previous step.
3.4.3.4.2.4.	Compare the sum of the intervals with the difference between the adjusted start reading and the stop reading (taking into account possible rollover between start and stop readings).
3.4.3.4.3.	If difference is within the allowable margin calculated in step 3.4.2.3.2 - data has passed Sum Check
3.4.3.4.4.	If difference is greater than the margin, and if some intervals are missing or incomplete, optionally prorate the difference between the adjusted start read and the stop read by the percentage of good intervals and rerun the sum check.
3.4.3.4.4.1.	Count the number of good intervals in the data stream.
3.4.3.4.4.2.	Calculate the percentage of good intervals by dividing the count from the previous step by the number of intervals elapsed between start time and stop time.
3.4.3.4.4.3.	Multiply the percentage by the difference between the adjusted start reading and the stop reading.
3.4.3.4.4.4.	Compare the new difference with the sum of the usage in the good intervals.
3.4.3.4.4.5.	If the difference is within the allowable margin calculated in step 3.4.2.3.2 - good intervals have passed Sum Check.
3.4.3.4.5.	If the difference is greater than the allowable margin, optionally reread meter.
3.4.3.4.5.1.	Redo sumcheck test from original start meter reading to new stop meter reading
3.4.3.4.5.2.	If the difference is within two meter multipliers - data has passed Sum Check
3.4.3.4.5.3.	If the difference is greater than two meter multipliers - kick out for manual inspection.  
3.4.3.4.6.	Other checks may be performed, based on the technology used, to verify that the interval data is an accurate representation of usage as measured by the meter readings.
3.4.3.4.7.	If sumcheck still not resolved, perform manual inspection of data
3.4.3.4.7.1.	Manually inspect the data.  If the data seems reasonable, it may be considered verified.
3.4.3.4.7.2.	Manual inspection is performed to verify meter and pulse multipliers.  If  a multiplier was wrong, recalculate the intervals using the new multiplier and redo the sum check.  If data now passes the sum check, data has passed the sum check.  
3.4.3.4.7.3.	  If multipliers are ok, check for meter changeout between the start and stop meter readings.  If the meter was changed out, redo the sum check for each meter independently.  If sum check passes, data has passed sum check.
3.4.3.4.7.4.	If multiplier was ok, do physical meter test/inspection.  If meter tests ok, data has passed sum check.  If meter does not test ok, data needs to be estimated.  If unable to visit site and perform meter test prior to posting the data, the data needs to be estimated.
3.4.3.4.8.	 If interval data is available, but there are no meter readings to perform the sum check with, manually inspect the data.  Data that appears to be reasonable (compared with historical data) can be considered to be verified.  Any data that does not appear to be reasonable needs to be estimated
3.4.4.	Spike check
This check may be done against either consumption or pulse data, provided the data scaling is consistent throughout the period.
3.4.4.1.	For each 24 hour period, determine highest and third highest peaks.  (Normally the 24 hour period is from midnight to midnight.  If the data is at the beginning and doesn’t start at midnight, use enough data from the next day of data to get 24 hours of data.  If the data is at the end and doesn’t stop at midnight, use enough data from the next to last day of data to get 24 hours.)  A spike check threshold of 10 pulses is used to eliminate false spikes for meters with very low usage.  If the highest peak is non-zero but less than or equal to the spike check threshold (10 pulses), the data passes the spike check.  If the highest peak is greater than the spike check threshold and (highest peak - third highest peak)/third highest peak> 1.8, a spike exists - the highest interval fails the spike check
3.4.4.2.	If interval fails spike check, optionally reread meter.
3.4.4.3.	If you get different value from reread, redo spike check.
3.4.4.4.	If value is the same on reread or you cannot reread the meter, kick it out for manual analysis 
3.4.4.5.	Manual analysis looks for similar patterns on other similar days.  May also optionally involve calling the customer to check for unusual conditions at the time of the spike.  If there is a regular pattern of failing this check, the customer may be an irregular usage customer.  See section on Irregular Usage Customers for additional information.
3.4.4.6.	If no similar pattern or other legitimate reason for spike is found, the highest interval needs to be estimated.  If similar pattern or other reason for legitimate spike is found, data can be considered to be verified after  having failed the spike check 
3.4.5.	Additional check if interval kVARh data is available
See section on Irregular Usage Customers for additional information.  This check may be done against either consumption or pulse data, provided the data scaling is consistent throughout the period.

Note that if multiple kWh channels map to a single kVARh channel, or multiple kVARh channels map to single kWh channel, the appropriate channels must be totaled prior to this check.  
3.4.5.1.	If there are any kWh intervals with zero consumption, check the corresponding kVARh interval.  
3.4.5.2.	If the corresponding kVARh interval is also zero or less than or equal to the kVARh check threshold of 4 pulses, the kWh data is considered to have passed the kVARh check.  (A kVARh check threshold is used to eliminate false errors for meters with very low usage.)
3.4.5.3.	If the corresponding kVARh interval is not zero, the kWh interval needs to be estimated.  If multiple kWh channels map to a single kVARh channel, the MDMA may optionally investigate the situation to determine if the problem can be directly traced to specific kWh channels.  If this is the case, only data for those channels needs to be estimated.  If the problem is not determined to be attributed to specific channels, all channels need to be estimated.  The MDMA may also optionally investigate to determine if this data represents actual customer usage, in which case the data is verified.  If there is a regular pattern of failing this check, the customer may be an irregular usage customer.  See section on Irregular Usage Customers for additional information.
3.5.	On the billing cycle for the meter…..
3.5.1.	High/low check
Note that for 1/1/98, this check may be done on good and verified data only (ie, not including any estimated intervals) to check the raw data, or after validation and estimation (including good, verified, and estimated intervals) to verify the estimated data, or both.  All parties should work toward doing this check both times in the future.  

This check is done against consumption data, not pulses.
3.5.1.1.	If last year’s data is available, calculate average daily usage for same billing month last year; use summed VEE or historical billing interval data if available, if not use VEE or historical billing usage (ie, difference between register readings)
3.5.1.2.	Else if last year’s data is not available, calculate average daily usage for the previous billing month; use summed VEE or historical billing interval data if available, if not use VEE or historical billing usage (ie, difference between register readings).
3.5.1.3.	Else if previous billing month not available - don’t do this check!
3.5.1.4.	Calculate average daily usage for this billing month using either summed VEE data (if test includes estimated data) or summed good and verified data (if test does not include estimated data).  If there were any spikes in the data, don’t include those intervals in the average - prorate the daily average accordingly.
3.5.1.5.	If the difference between the two daily averages is greater than 50% of the historical daily average, flag data (all the intervals in the month) as failing high/low average check and kick out for manual analysis.  Else data has passed High/Low Check.  
If (|historical daily average - this month’s daily average)| <= .5* historical daily average) then data has passed the High/Low Check.  
If (|historical daily average - this month’s daily average)| > .5* historical daily average) then data (all intervals in the month) is flagged as failing high/low average check and kicked out for manual analysis. 

3.5.1.6.	If data is flagged for manual analysis - look at the recent history of that meter.  If monthly usage has been on a trend in the appropriate direction and this seems reasonable, data is considered to be verified.  If not, optionally check with customer for changed usage patterns.  If changed usage patterns match change in data, data is considered to be verified.  If no change in usage patterns reported by customer, perform physical meter test/inspection.  If a problem is found with the meter, the data needs to be estimated.  (Note: if the problem existed prior to this billing period, previously posted data will need to be adjusted and reposted.)   Also check to see if some of the data looks reasonable; reasonable looking data is considered verified.  (For example, this may happen if a meter fails sometime during the month - the data at the beginning of the month may be ok, while the data after the failure may be obviously bad.)  If there is a regular pattern of failing this check, the customer may be an irregular usage customer.  See section on Irregular Usage Customers for additional information.
3.5.1.7.	If the meter is investigated and found to be accurate, the data is considered verified.
3.5.1.8.	If data fails high/low average check, suspect data needs to be estimated.
3.5.2.	If data has passed all appropriate validation tests without manual verification, data has been validated.  If data failed one or more validation tests but was manually verified, the specific tests the data failed and the fact that it was manually verified must be recorded on an interval level.  If data did not pass all appropriate validation tests or was not manually verified, the specific tests the data failed must be recorded on an interval level, and the data needs to be estimated.  If an interval fails more than one test, all failed tests must be recorded.
3.5.3.	Estimate data needing estimation; see section 4.  Note that for each interval that is estimated, the MDMA must record the estimation algorithm used.  The estimation algorithm will be reported when the data is posted when it is supported by CMEP.  Interval data estimation algorithms include:
•	less than 2 hours (4.1) 
•	greater than 2 hours (4.2) - not scaled based on usage
•	greater than 2 hours (4.2) - scaled based on usage
•	time-drifted intervals prorated (Appendix A  - method 1) 
•	time-drifted intervals prorated (Appendix A  - method 2) 
•	intervals adjusted (4.3) 
•	intervals manually estimated (4.2.8) 
•	intervals estimated due to meter interval programmed incorrectly (4.4) 
•	Load profile template used (4.2.7)
3.5.4.	Optionally rerun checks after estimation to help verify estimated readings.  
3.6.	Irregular Usage Interval Data
An irregular usage customer is one whose usage pattern does not follow normal usage patterns and consistently fails the spike check, kVARh check, or high/low usage check.  A customer may be determined to be an irregular usage customer by an MDMA if the MDMA verifies that:
1.	the customer data fails the standard validation check for three consecutive months, and that the data represents the actual customer usage, OR
2.	the MDMA is notified by the customer’s previous MDMA.
The data used to determine a customer is an irregular usage customer could be data collected by the MDMA, or historical data provided by the previous ESP or MDMA.  An MDMA may modify the spike check and/or high/low usage check, and skip the kVARh check if an irregular usage customer consistently fails the check.  The MDMA must notify both the customer’s ESP and UDC of the customer’s irregular usage status and what modified tests will be performed.  
The goal of the modified checks is to automate the manual procedures the MDMA would perform to verify this is the customer’s normal usage pattern.  An MDMA may use a variation of the spike check or high/low usage check based on the actual usage pattern.  Note that the MDMA may not skip the spike check or high/low usage check.  If the data passes the modified test, the data is validated.
•	Possible modifications for the spike test include modifying the spike check threshold value (180%) or the pulse threshold value (10 pulses).
•	Possible modifications for the high/low check include changing the percentages (+/- 50%), using the year’s average instead of one billing period’s average, or comparing to the minimum and maximum values for the past year.
For some customers, irregular usage patterns are symptomatic of the business and will always be present, such as co-gen customers.  For other customers, irregular usage patterns may be a temporary condition, such as when a factory adds a second shift and fails the high/low usage check for the first 12 months.  The MDMA must determine whether a customer is a permanent or temporary irregular usage customers.  Temporary irregular usage customers must be reviewed annually to determine if they are still irregular usage customers or should be returned to the normal checks.

4.	Interval Data Estimation Rules
Estimate intervals needing estimation using the following estimation rules….
4.1.	If section of data needing estimation is 2 hours or less in length…
Use the point to point linear interpolation method to estimate the data where there is a gap in the data.  If the gap occurs in the middle of the data, the “first point” is the last good interval before the gap, and the “second point” is the first good interval after the gap.

If the gap occurs at the beginning of the span, use the last interval from the historical data as the first point, if the historical data is available and the interval data is not estimated. Otherwise, use the second point (the first good interval after the gap) as the first point - this will cause the load to be estimated as a flat load. 

If the gap occurs at the end of the span, use the first point (the last good interval before the gap) as the second point (- this will cause the load to be estimated as a flat load.
4.2.	If section of data needing estimation is more than 2 contiguous hours…

Note that no data from days in which there is a power failure can be used as reference data for estimation.  Power failures can cause irregular usage patterns, resulting in data that is not typical for the customer.

Note that the days chosen to provide reference data in the subsequent steps may optionally be “partial” days, as long as the intervals being used as reference data are not themselves estimated and the reference days do not include a power failure.  This situation could occur in one of two ways:
a)	The use of reference days that contain estimated data for estimation as long as the period of time that you are going to use for your estimate does not contain any estimated data.  For example, if you are trying to estimate 3/18/98 from 2-5 PM, and 3/11/98 was estimated from 10-11 AM but not from 2-5 PM, 3/11/98 would be an acceptable day to include.
b)	The use of partial days for estimation, where data collection began at some time other than midnight.  For example, this could be encountered on the install date for a given meter/recorder.
	
4.2.1.	Determine a daily profile
4.2.1.1.	Find the three “same weekdays” with valid data (ie, contain no estimated data during the intervals needing estimation or power failures) closest in time to the day with the data needing estimation.  Reference days are chosen first from the present billing period; historical data is only used if enough days are not available in the present billing period.  Historical data up to 90 days prior to the day needing estimation may be used.  If two potential reference days are equidistant from the day requiring estimation, use the earlier day first.  Note that days from the current billing period may be a later date than the day needing estimation.  A “same weekday” means the same day of week.  Note that same weekdays from the same billing period are used first, and then same weekdays from historical data are used if necessary.  If the day with data needing estimation is a holiday, the “same weekdays” are holidays, not the same day of week.  If not enough historical holidays exist in the current billing period or  previous 90 day period, use Sundays. Note that a standard list of holidays will be used across the board, regardless of the tariff or service territory the meter is in.  The historical data used is that data immediately preceding the usage period, regardless of seasonal crossover.
For example, if June 2, 1998, needed estimation, and the billing period was from June 1 through June 30, the reference days used would be June 9, 16, and 23, provided they contained valid data.

If June 16, 1998, needed estimation, and the billing period was from June 1 through June 30, the reference days used would be June 2, 9, and 23, provided they contained valid data.  If June 9 contained estimated data, June 2, 23, and 30 would be used.
Calculate the average daily profile using the three selected days.

4.2.1.2.	If only two same weekdays are available from the current billing period and 90 days historical data, calculate the average daily profile using the two selected days.  
4.2.1.3.	If only one same weekday is available from the current billing period and 90 days historical data, it us used as the daily profile.
4.2.1.4.	If no same weekdays are available in the current billing period and 90 days historical data, look for the three “like” days that are closest chronologically to the day with intervals needing estimation.  A like day means weekday or weekend/holiday.  For example, if the intervals needing estimation were on Tuesday, use Monday, Wednesday, and Thursday.  Only use weekdays with weekdays; only use weekends with weekends; only use holidays or weekends with holidays. Calculate the average daily profile using the three selected days.  
4.2.1.5.	If no similar days are available and there aren’t three “like” days, select the two “like” days that are chronologically closest to the day with intervals needing estimation.  Calculate the average daily profile using the two selected days.  
4.2.1.6.	If no similar days are available and there aren’t two “like” days, select the one “like” day that is chronologically closest to the day with intervals needing estimation.   This day is used as the daily profile.  
4.2.2.	Use the daily profile to estimate the required data
1.	Estimate the data needing estimation by applying the appropriate intervals from the average daily profile to fill the missing intervals.  
2.	If begin and end meter readings are available and are known to be good, they may optionally be used to scale the estimated interval data.  This is done by:
a)	determine the total usage during the time period (end meter reading - start meter reading)
b)	sum together all the good intervals (validated or verified)
c)	subtract the total of the good intervals from the total usage to determine the total estimated usage
d)	sum together all the previously estimated intervals
e)	calculate the scaling factor by dividing the total estimated usage (from step c) with the sum of the estimated intervals (step d)
f)	multiply each estimated interval by the scaling factor
4.2.3.	If there are no similar days or like days, use the load profile for the customer’s class to estimate the data.  Use this month’s usage if available to scale the load profile.  If this month’s usage is not available, use last month’s usage or last year’s usage, whichever is determined to be more reasonable, to scale the load profile,.  Refer to the UDCs’ load profile documentation for more information on applying load profiles.  Note that the load profiles are for hourly data.  If 15 minute data is required, assume a flat load throughout the hour (i.e., each 15 minute interval would have 1/4^th the hourly usage).
4.2.4.	If there is no historical data that can be used, the data must be estimated manually and the process and assumptions documented.  This must be coordinated with the customer’s UDC and ESP within the 5 day window required to report interval data.
4.3.	If the cause of the problem is determined to be a physical problem with the meter or meter installation and the data can be corrected by scaling the intervals and meter readings, the MDMA will be notified of the time period requiring correction and the scaling factor to be applied to each interval in that period.  When the data is posted in the CMEP format, it would be marked as estimated if it had not been previously posted, and marked as adjusted if it had previously been posted.  Data that is corrected in this fashion may be reported in a separate category on the monthly MDMA reports, and not included as part of the % estimated data for the month.  (Details need to be worked out as to how MDMAs will receive this notification.  This has been forwarded to the PSWG.)  Examples of these situations include a meter running slow, a meter running fast, one or two phases dropped, ….
4.4.	Interval in meter doesn’t match tariff or settlement requirements
If the meter programming and the MDMA requirements are inconsistent, the data is calculated as follows:

a)	If the meter is programmed to collect data at a smaller interval than required by its tariffs, and the meter's interval evenly divides into the interval required by the tariff - for example, the meter was programmed to collect 5 minute data, and the tariff requires 15 minute data. Sum the 5 minute intervals into 15 minute intervals on even 15 minute boundaries.  If the data passed all the other Validation tests, it is considered validated.  
b)	If the meter is programmed to collect data at a larger interval than required by the meter’s tariff.  For example, the meter is programmed to collect 60 minute intervals, but the MDMA Server and tariff require 15 minute intervals.  Prorate the data by assuming an even load distribution during the interval.  In this example, the usage in the 60 minute interval would be divided by 4 to estimate the usage in a 15 minute interval.  The data is marked as estimated.  The meter needs to be reprogrammed to the correct interval.
c)	If the meter is programmed to collect data at a smaller interval than required by its tariff, but the meter’s interval doesn't evenly divide into the interval required by the tariff.  For example, the meter is programmed to collect 10 minute intervals, and the tariff requires 15 minute intervals. The data would be estimated and marked as estimated.  To estimate data, all collected intervals that are contained within the required reporting interval are included in the appropriate reporting interval.  Collected intervals that cross the boundaries of required reporting intervals are included proportionally in both reporting intervals.  In this example, if there were three 10 minute intervals containing 10 kWh, 20 kWh, and 30 kWh, the corresponding estimated 15 minute intervals would contain 20 kWh (10 + .5*20) and 40 kWh (.5*20 + 30).   This is similar to the prorating technique discussed in Appendix A.  The meter needs to be reprogrammed to the correct interval.
5.	Monthly Data Validation Rules
5.1.	Time check of Meter Reading Device/System
This test only applies for meter reading devices and systems collecting Time-of-Use data. Time check of  meter reading device/system - ensure that the collection device is synchronized to the national time standard before data collection begins
5.2.	Time tolerance check of meter
The time tolerance check is only required if the meter is collecting Time-of-Use (TOU) data.
5.2.1.	If time in meter is within +/- 5 minutes of the time standard, the data has passed Time Tolerance check.  Note that if the meter is within +/- 5 minutes of the standard, the time in the meter can optionally be corrected.
5.2.2.	If time in meter is off > 5 minutes but <= 30 minutes, the data passes the Time Tolerance check.  The data does not need to be estimated, but the MDMA must record the fact that the meter’s time was off by this amount in case there is a later question about the data.  The meter time may optionally be reset.
5.2.3.	If time in meter is off > 30 minutes, the data fails the time tolerance check and must be estimated.  The time in the meter must be reset.  If the meter fails the time tolerance check after being reset for three consecutive months, the meter must be physically inspected/tested. 
5.3.	Hi/Lo Usage
	The hi/lo usage check validates that the usage (e.g. kWH, kVARH, water, and gas) meter read is within a 40% and 200% tolerance window of the highest daily average usage meter read based on either last year’s usage for the same billing month, the previous three billing months, or the previous billing month.
	If the meter read falls outside of the tolerance window, the meter read must be re-read or re-entered, and it must be flagged to indicate a failure of the hi/lo usage validation.
	The Ho/Lo Usage check validates cumulative consumption (kWh, kVARh).
5.3.1.	Calculate the daily average usage for the present billing period.  
Daily average = (current read - previous read)/(# days between reads)
5.3.2.	Calculate the historical daily average usage,
5.3.2.1.	If there is not at least one month (minimum 27 days) of historical billing data available, skip this test.
5.3.2.2.	If data for the same customer and site is available, calculate the daily average for the same billing period last year.  Use this as the historical billing average usage.  To determine which billing period last year is “the same”, choose the mid-point of this year’s billing period, and find the billing period last year that included the same date.  For example, if the billing period this year was from April 13 to May 13, the mid-point would be April 28.
5.3.2.3.	If data for the previous year’s billing period is not available but there is data for the last three billing periods, calculate the daily average for the last three billing periods and use it as the historical billing average usage.
5.3.2.4.	If neither of the previous two options are available but there is data for the last billing period, calculate the average daily usage for the last billing period and use this as the historical billing average usage.  
5.3.3.	Compare the present billing period average daily usage with the historical billing period average daily usage.  If the present billing period average daily usage is between 40% and 200% of the historical average daily usage, the data passes this test.
5.3.4.	If the present billing period average daily usage is not within 40% to 200% of the historical billing period average daily usage, optionally re-read the meter.  
•	If the reread is essentially at the same time as the original read and a different value is obtained, assume the first reading was a misread and perform the check again with the new reading.  If the same reading was obtained, assume the meter reading is correct; the data failed the Hi/Lo Usage check but is verified. 
•	If the reread is not at the same time as the original read, re-compute the average daily usage using the new reading.  If the new average daily usage is within +/- 20% of the previous average daily usage, the data fails the Hi/Lo usage test but is verified.  If the re-read results in the same value, the data fails the Hi/Lo usage test but is verified..
5.3.5.	Data that fails the Hi/Lo usage test and has not verified may be investigated and manually verified if justification is found.  Else data must be estimated.
5.4.	Hi/Lo Demand
The hi/lo demand check validates the demand (e.g. kW and kVAR) meter read is within a 75% and 125% tolerance window of the highest average demand meter read based on either last year’s demand for the same billing month,  the previous three billing months, or the previous billing month.
If the meter read falls outside of the tolerance window, the meter read must be re-read or re-entered, and it must be flagged to indicate a failure of the hi-lo demand validation.
QUESTION - what if demand is TOU demand?  Do you do this for each “period” (ie, peak, off-peak, etc)?  YES
QUESTION - what if you have historic interval data for this customer, but the customer wasn’t on a demand rate a year ago, so there’s no historical demand?  Would you try to use the historical interval at all?  YES
Note that if demand is TOU demand, this test is performed for each period and each season separately.  Demand is only compared with values from the same period and season.
5.4.1.	Determine the peak demand for this billing period.
5.4.2.	Determine the historical peak demand,
5.4.2.1.	If there is not at least one month (at least 27 days) of historical billing demand data available, skip this test.
5.4.2.2.	If demand data for this customer and site is available for the same billing period last year, use that as the historical peak demand.
5.4.2.3.	If data for the previous year’s billing period is not available but there is data for the last three billing periods, calculate the average peak demand for those billing periods by summing the three demands and dividing the sum by 3.  Use this value as the historical peak demand.
5.4.2.4.	If neither of the previous two options are available but there is data for the last billing period, use the peak demand from the previous billing month as the historical peak demand.  
5.4.3.	Compare the present peak demand with the historical peak demand.  If the present peak demand is between 40% and 200% of the historical peak demand, the data passes this test.
5.4.4.	If the present peak demand is not within 40% to 200% of the historical peak demand, optionally re-read the meter.  
•	If the reread is essentially at the same time as the original read and a different value is obtained, assume the first reading was a misread and perform the check again with the new reading.  If the same reading was obtained, assume the meter reading is correct; the data failed the Hi/Lo Demand check but is verified. 
•	If the reread is not at the same time as the original read and the new demand value is within +/- 20% of the previous demand value, the data fails the Hi/Lo demand test but is verified.  If the re-read results in the same value, the data fails the Hi/Lo usage test but is verified..
5.4.4.	Data that fails the Hi/Lo Demand test and has not been verified may be investigated and manually verified if justification is found.  Else data must be estimated.
5.5.	TOU Usage
	The TOU usage check compares the sum of the current season peak kWH meter readings and the current season total kWH meter reading.
	The difference must be within plus or minus the number of active peaks for the current season.
	If the difference is greater than the tolerance, the MDMA must verify the data (i.e. re-reading, meter test, etc.) and flag the current season total kWH meter read to indicate a failure of the TOU usage validation.
5.5.1.	For the current billing period, calculate the total kWh by summing all the periods (including all seasons).
5.5.2.	Compare the calculated total kWh with the total kWh read from the meter.  If they are within +/- the number of periods summed together, the data passes the test.  If they are not, the data fails the TOU Usage test and must be estimated.  Note that this test must be done in whatever units are read from the meter.  For example, if the meter provides kWh, the kWh values must be summed and compared.  If the meter provides pulses, the pulse values must be summed and compared.
5.6.	Zero Consumption for Active Meters
	The zero consumption check validates that if the meter is an active meter and the current reading is equal to the previous reading.
	If there is zero consumption, the MDMA must verify the meter read (i.e. re-reading, meter test, meter replacement, etc.) and flag the meter read to indicate a failure of the zero consumption validation.
5.6.1.	If the meter is an active meter (ie, is associated with a customer who has financial responsibility), calculate the usage for the present billing month.
5.6.2.	If the usage is greater than 0, the data passes the zero consumption check.
5.6.3.	If the usage is 0, the data failed the zero consumption check.  Verify the meter reading by re-reading the meter and/or testing the meter.  If the reread is the same and the usage is still 0, the data failed the Zero Consumption check but is verified.  If a new, different meter reading is obtained, run all the tests again using the new data.  If 0 usage is determined to be legitimate (for example, a building or equipment that is only used seasonally), mark the data as verified.
5.6.4.	Data that fails the zero consumption check may be manually investigated and verified if justification is found (for example, a building or equipment that is only used seasonally).  If the data is not validated or verified, it must be estimated.
5.7.	Usage for Inactive Meters
	The inactive meter check validates that if the meter is an inactive meter and the current reading is not equal to the previous reading.
	If there is consumption, the MDMA must verify the meter read (i.e. re-reading, meter test, meter replacement, activate meter for new service, etc.) and flag the meter read to indicate a failure of the usage for inactive meters validation.
	An inactive meter is one for which there is no customer with financial responsibility.  This does not apply to non-UDC MDMAs and is not a required check.  
5.8.	Number of Dials on Meter
	The dial quantity check validates that the dial quantity of the meter read is different than the expected dial quantity.
	If the dial quantity is different, the MDMA must do further investigation (i.e. re-reading, correcting meter configuration, etc.) and flag the meter read to indicate that a meter read dial quantity was investigated and corrected.
	This test applies to cumulative consumption only.  It checks that the number of “dials” (digits) reported in the read is consistent with the number of dials (or digits) on the meter display.  This test is performed for both remote and local reads if supported by the meter reading technology.  If the meter reading technology doesn’t support this check, it is not performed.
5.8.1.	Determine the number of digits in the meter reading.
5.8.2.	If the number of digits in the meter reading is consistent with the number of digits/dials on the meter, the data passes this test.  
5.8.3.	If the number of digits is not consistent, re-read the meter to verify that the correct meter is being read and that it has the correct number of digits/dials.  If the re-read provides the same values, the meter reading failed the Number of Dials on Meter check but is verified.  The situation must be investigated and records must be corrected.  If the re-read produces different values, perform the check again with the new values. 
5.9.	Meter Read Dial Decimal Quantity Difference
	The dial decimal quantity check validates that the dial decimal quantity of the meter read is different than the expected dial decimal quantity.
	If the dial decimal quantity is different, the MDMA must do further investigation (i.e. re-reading, correcting meter configuration, etc.) and flag the meter read to indicate that a meter read dial decimal quantity was investigated and corrected.
Note this check is only performed for on-site meter reads, and is not performed for remote meter reads.
5.9.1.	When the meter is read on-site, the meter reader compares the number of decimal places displayed by the meter with the number of decimal places expected.  If they are the same, the reading passes the Meter Read Dial Decimal Quantity check.
5.9.2.	If they are not the same, re-read the meter to verify that the correct meter is being read and that it has the correct number of decimal places.  If the re-read provides the same values, the meter reading failed the Meter Read Dial Decimal Quantity Difference check but is verified.  The situation must be investigated and records must be corrected.  The meter may need to be re-programmed.  If the re-read produces different values, perform the check again with the new values. 
5.10.	Meter Identification
	The external meter identification check validates that the external meter ID of the meter read is not different than expected.
	If the external meter ID is different, the MDMA must do further investigation (i.e. re-reading, correcting meter configuration, etc.) and flag the meter read to indicate that an external meter id mismatch was investigated and corrected.
	There are two types of Meter Identification checks depending on how the meter is read - Internal Meter Identification check and External Meter Identification check.  The following table summarizes when each check is required:
	
Meter Reading Type | Perform External ID Check? | Perform Internal ID Check?
Remote | No | Yes
Optical Port | Yes | Yes
Manual | Yes | No

a)	If the meter, recorder, or other measurement device is read remotely or via its optical port, the Internal Meter Identification check is performed.  This verifies the meter’s internal identification (often its serial number) with the identification expected by the meter reading system.  If they match, the data passes this check.  If they don’t match, the MDMA must investigate why the meter is different than indicated by their records and resolve the inconsistency
b)	If the meter is read locally (including via an optical port), the External Meter Identification Check is performed.  This verifies the Meter ID on the meter nameplate with the Meter ID expected by the meter reading system  If they match, the data passes this check.  If they don’t match, the MDMA must investigate why the meter is different than indicated by their records and resolve the inconsistency.
6.	Monthly Data Estimation Rules
Note that the MDMA must record the estimation algorithm used for each data element that is estimated.  The estimation algorithm will be reported when the data is posted when it is supported by CMEP.  Monthly data estimation algorithms include:
•	Estimation based on previous year’s data
•	Estimation based on previous billing period’s data
•	Estimation based on pro-rated load profile
•	Manual estimation

6.1.	Estimating Usage 
	The MDMA must estimate all missing usage meter reads.
	In order to estimate the missing usage meter read, the MDMA must use last year’s daily average usage.  If that is not available, then the previous month’s daily average usage will be used.  If there is no historical data available, a daily average usage will be calculated utilizing the appropriate Load Profile.
	In order to estimate the current meter read, the daily average will be applied to the previous month’s meter read, projecting the usage on the current scheduled meter read date.
	All estimated usage meter reads will be flagged to indicate that the read is estimated.
	QUESTION - how do you use a Load Profile to calculate average daily usage when there is no historical data?
6.1.1.	Calculate average daily usage to be applied
	If billing data is available from the same customer and same site for the same billing period last year and it is not estimated, calculate the average daily usage for the same billing period last year and use this value as the average daily usage.  To determine which billing period last year is “the same”, choose the mid-point of this year’s billing period, and find the billing period last year that included the same date.  For example, if the billing period this year was from April 13 to May 13, the mid-point would be April 28.  In this case, the estimation algorithm is estimation based on previous year’s data.
6.1.1.1.	If there is no data from the previous year but there is a full billing month’s data (at least 27 days) available from the previous billing month and it is not estimated, calculate the average daily usage for the previous billing month and use this value for the average daily usage.  In this case, the estimation algorithm is estimation based on previous billing period’s data.
6.1.1.2.	If there is less than one month’s historical data available but at least one week’s historical data, use the customer’s load profile to calculate an average daily usage.  Use the historical data to calculate a scaling factor to be applied to the load profile.  In this case, the estimation algorithm is estimation based on prorated load profile.

For example:  Assume the week’s historical data had an average hourly usage of 10 kWh.  Thirty days of usage are to be estimated using the appropriate load profile, or 720 hours.  Calculate the average hourly usage of the load profile for the month by dividing the total usage by the number of hours.  In this case assume the total usage from the load profile was 3600 kWh, which results in an average hourly usage of 3700/720 or 5 kWh.  The ratio of actual load to estimated load is 10 kWh /5 kWh, or 2.  Multiply the estimation obtained from the load profile by 2 to obtain the final estimation.
6.1.1.3.	If there is not at least one week’s historical usage, data must be estimated manually based on any available data, such as similar customers, load profiles, average usage for the customer class, etc.  In this case, the estimation algorithm is manual.
6.1.2.	Calculate the number of days since the last good meter reading available to the end of this billing period.  If the meter is read monthly, this would typically be last month’s meter reading.
6.1.3.	Multiply the average daily usage by the number of days since the last good reading, and add that value to the last good reading to obtain an estimated reading.  This is the estimated meter reading.  Mark the reading as being estimated using the appropriate algorithm.
6.2.	Estimating Demand 
	The MDMA must estimate all missing demand meter reads.
	In order to estimate the missing demand meter read, the MDMA must use last year’s demand.  If that is not available, then the previous month’s demand will be used.  If there is no historical data available, the Load Profile demand will be used.
	All estimated demand meter reads will be flagged to indicate that the read is estimated.
	QUESTION - how do you use a Load Profile to calculate peak demand when there is no historical data or you don’t have the meter reading?
6.2.1.	If a similar demand value is available from the same billing period for the previous year, use the previous year’s demand as the estimate.  In this case, the estimation algorithm is estimation based on previous year’s data.
6.2.2.	If data is not available from the previous year but is available for the previous billing period, use the previous billing period’s demand as the estimate.  In this case, the estimation algorithm is estimation based on previous billing period’s data.
6.2.3.	If there is no historical demand data, demand must be estimated manually based on any available data, such as similar customers, load profiles, average demand for the customer class, etc.  In this case, the estimation algorithm is manual.
6.3.	Estimating TOU Usage
For missing TOU usage data, each period must be estimated separately, using historical data from the same TOU schedule, period, and season as the data requiring estimation.  
6.3.1.	For each period requiring estimation, the following steps are performed.  Note that if there is a season change during the time period requiring estimation, each season needs to be done separately. 
6.3.1.1.	Calculate average daily usage to be applied
6.3.1.1.1.	If billing data is available from the same customer, same site, and same TOU schedule for the same billing period last year, and last year’s data is not estimated, calculate the average daily usage for the same billing period last year and use this value as the average daily usage.  To determine which billing period last year is “the same”, choose the mid-point of this year’s billing period, and find the billing period last year that included the same date.  For example, if the billing period this year was from April 13 to May 13, the mid-point would be April 28.  If a season change occurred during the month, only use data from the appropriate season as reference data.  If necessary, use data from the month before or after the same billing period last year to get at least one week’s worth of data for the season.  In this case, the estimation algorithm is estimation based on previous year’s data.
6.3.1.1.2.	If there is no data from the previous year but there is at least one week’s data available from the previous billing month and it is not estimated, calculate the average daily usage for the previous billing month and use this value for the average daily usage.  If a season change occurred during the month, only use data from the appropriate season as reference data.  If necessary, use data from the month before or after the same billing period last year to get at least one week’s worth of data for the season.  In this case, the estimation algorithm is estimation based on previous billing period’s data.
6.3.1.1.3.	If there is less than one week’s historical data available, data must be estimated manually based on any available data, such as similar customers, load profiles, average usage for the customer class, etc.  In this case, the estimation algorithm is manual.
6.3.1.2.	Calculate the number of days in this period requiring estimation.  
6.3.1.3.	Multiply the average daily usage by the number of days requiring estimation.  This is the estimated usage.  Mark the reading as being estimated using the appropriate algorithm.
6.4.	Estimating TOU Demand Data
For missing TOU demand data, each period must be estimated separately, using historical data from the TOU schedule, period, and season as the data requiring estimation.
6.4.1.	For each period requiring estimation, the following steps are performed.  Note that if there is a season change during the time period requiring estimation, each season needs to be done separately. 
6.4.1.1.	If a similar demand value is available from the same billing period for the previous year, use the previous year’s demand as the estimate.  In this case, the estimation algorithm is estimation based on previous year’s data.
6.4.1.2.	If data is not available from the previous year but at least one week’s data is available for the previous billing period, use the previous billing period’s demand as the estimate.  In this case, the estimation algorithm is estimation based on previous billing period’s data.
6.4.1.3.	If there is less than one week’s historical data available, demand must be estimated manually based on any available data, such as similar customers, load profiles, average demand for the customer class, etc.  In this case, the estimation algorithm is manual.
6.5.	

1. Introduction	
2. Required data validation tests	
3. Interval data collection and validation rules	
4. Interval Data Estimation Rules	
5. Monthly Data Validation Rules	
6. Monthly Data Estimation Rules