1.	Introduction
This document attempts to clarify the required data validation, editing, and estimation techniques required to participate in the California market as an MDMA.
2.	Required data validation tests
Data validation checks are designed to identify things that can go wrong at the meter/recorder and cause the data collected not to reflect actual usage.  The following tests are required for kWh and kVARh interval data:

Test	Purpose	
Time check of meter reading device/system	Check for time drift of meter reading device/system outside CPUC standard	
Time check of meter	Check for time drift of meter clock outside CPUC standard	
Pulse overflow check	Check for the following:
•	improper scaling factor in meter 
•	improperly sized transformer
•	hardware problem	
Sum check	Check for the following in combination meter/recorder installations: 
•	crossed channels between meter and recorder
•	pulse relay problems  
Check for the following for all installations: 
•	invalid PT and CT ratios, 
•	invalid meter constants	
Spike check	Check for the following for all installations:
•	transmission error 
•	spike resulting from meter test.  
Note that a spike can also occur after an outage - in this case the data is valid, but may or may not be used for peak billing depending on the tariff and company policy.	
High/low check	Check for the following in all installations: 
•	dropped phases 
•	inaccurate meter constants
•	energy diversion
•	fast/slow meters
Also check for the following in combination meter/recorder installations:
•	erratic pulse input to recorder	
Meter identification check	Check for the following:
•	that the meter ID was reported correctly
•	the meter has not been changed out
•	the data is being reported for the correct meter	
kVARh check (for kWh data only if corresponding kVARh data available)	Check that:
•	kWh channels are correctly mapped to kVARh channel
•	meter is operating correctly	

The following tests are required for monthly data validation, and would be required to be performed by a non-UDC MDMA when non-interval metering is unbundled or when the UDCs accept actual billing determinants instead of 15 minute interval data.  This will be no later than 1/1/99.
Test	Purpose	
Time check of meter reading device/system (applies to devices/systems collecting TOU data only)	Check for time drift of meter reading device/system outside CPUC standard	
Time check of meter (applies to meters collecting TOU data only)	Check for time drift of meter clock outside CPUC standard	
High/low usage check	Check for the following:
•	misread
•	fast/slow meter
•	broken meter
•	incorrect multipliers
•	energy diversion
•	dropped phases	
High/low demand check (applies to demand readings only)	Check for the following:
•	misread
•	fast/slow meter
•	broken meter
•	incorrect multipliers
•	energy diversion
•	dropped phases	
Time-of-use check (applies to TOU data only)	Check for the following:
•	misread
•	fast/slow meter
•	broken meter
•	incorrect multipliers
•	energy diversion
•	dropped phases	
Zero consumption for active meters	Check for the following:
•	energy diversion
•	meter doesn’t register	
Usage for inactive meters	Check for unauthorized usage at a site for which there is no customer with financial responsibility	
Number of dials on meter	Check for:
•	wrong meter
•	misread	
Dial decimal quantity	Check for:
•	wrong meter
•	misread	
Meter identification	Check for the following:
•	that the meter ID was reported correctly
•	the meter has not been changed out
•	the data is being reported for the correct meter	

3.	Interval data collection and validation rules
These rules apply to both kWh and kVARh data, depending on the data required by the meter’s tariff(s).  General MDMA and MSP business practices should ensure that the meter is programmed correctly for the required revenue data, and that the MDMA is set up appropriately.  This includes information such as interval size, meter constants, and what quantity is recorded by what channel.  These VEE rules do not require or describe how the MDMA verifies that the meter is programmed correctly. 
Note that if interval data is read more often then required for billing, checks need to be performed at different times in the process.  Some must be done as the data is read from the meter; some can be done anytime between when the data is collected from the meter and the end of the cycle, and others have to be done on a billing period basis at the end of the billing cycle.  They are broken out that way in this description.
3.1.	Time check of  meter reading device/system - ensure that the collection device is synchronized to the national time standard before data collection begins
3.2.	Collect data
3.3.	As data is collected, 
3.3.1.	Check meter identification - verify the meter’s identification matches what is expected
3.3.2.	Do time tolerance check of meter.  Note that depending on the communication technology used, network latency must be taken into account.
3.3.2.1.	If time in meter is within 3 minutes of time standard, the data has passed Time Tolerance check.  Note that if the meter is within 3 minutes of the standard, the time in the meter can optionally be corrected.
3.3.2.2.	If time in meter is off more than 3 minutes but <= 75 minutes, prorate data.  (Note: meter time must be corrected.  If the meter fails the time check for three consecutive months, the meter must be physically inspected/tested)  The data to be prorated includes all intervals from the last time the meter’s time was determined to be good (ie, within the 3 minute tolerance) and when it was discovered that the meter’s time was off more than 3 minutes and the meter was re-synched.  Refer to Appendix A for examples of how to prorate data.
3.3.2.2.1.	Compare the number of intervals provided by the meter during the time period in question with the actual number of intervals expected.  If the two numbers are the same (ie, the meter has the correct number of intervals), do not prorate the data. The intervals are good.
3.3.2.2.2.	If the meter does not have the correct number of intervals, 
3.3.2.2.2.1.	Refer to Appendix A for a description of how to prorate the data.  Mark intervals as estimated.
3.3.2.3.	If time in meter is off more than 75 minutes, all intervals between the last time you verified the meter’s time was good (ie, within 3 minute tolerance) and when you discovered the meter’s time was off more than 75 minutes need to be estimated.
3.4.	Either as data is collected or prior to publishing on MDMA Server…
3.4.1.	Perform pulse overflow check
3.4.1.1.	Inspect each interval for this condition.  If a pulse overflow occurs, need to mark meter for physical meter test/maintenance. The interval that has a pulse overflow needs to be estimated. Continue VEE process.
3.4.2.	Perform sum check
This check may be done against either consumption or pulse data, provided the data scaling is consistent throughout the period.
3.4.2.1.	Sum the intervals for the time between the start and stop meter readings.  
3.4.2.2.	Compare sum of with difference between start and stop readings (taking into account possible rollover between start and stop readings - for example, if the meter reading rolls over at 99999, the start reading was 99968 and the stop reading was 00294, the difference would be 326).  Note that the sum and the readings must be in the same units for the comparison.
3.4.2.3.	If difference is <= two meter multipliers (meter multiplier = ctr x vtr, where ctr is current transformer ratio and vtr is voltage transformer ratio) - data has passed Sum Check. 
3.4.2.4.	If difference is greater than two meter multipliers, optionally, rerun the sum check, taking into account the differences in time between the time of the start read and the start of the first interval, and the time of the stop read and the end of the last interval.
3.4.2.4.1.	Adjust the start reading by doing the following:
3.4.2.4.1.1.	Calculate the percentage of an interval that has elapsed between the begin time of the first interval and the time of the start reading.  For example, if the first interval in an hourly interval data stream starts at 3 pm, and the start reading was performed at 3:20, this percentage would be (20 minutes / 60 minutes) = 33%.
3.4.2.4.1.2.	Adjust the start reads by subtracting the percentage times the usage in the first interval.  For example, if the start reading was 55555 and the usage during the first interval was 45, then adjust the start reading to 55540.
3.4.2.4.2.	Calculate an allowable margin of error by doing the following:
3.4.2.4.2.1.	Calculate the percentage of an interval that has elapsed between the end time of the last good interval and the time of the stop reading.
3.4.2.4.2.2.	Multiply the percentage by the usage in the last interval in the data stream.
3.4.2.4.2.3.	Add two multipliers to the value calculated in the previous step.
3.4.2.4.2.4.	Compare the sum of the intervals with the difference between the adjusted start reading and the stop reading (taking into account possible rollover between start and stop readings).
3.4.2.4.3.	If difference is within the allowable margin calculated in step 3.4.2.3.2 - data has passed Sum Check
3.4.2.4.4.	If difference is greater than the margin, and if some intervals are missing or incomplete, optionally prorate the difference between the adjusted start read and the stop read by the percentage of good intervals and rerun the sum check.
3.4.2.4.4.1.	Count the number of good intervals in the data stream.
3.4.2.4.4.2.	Calculate the percentage of good intervals by dividing the count from the previous step by the number of intervals elapsed between start time and stop time.
3.4.2.4.4.3.	Multiply the percentage by the difference between the adjusted start reading and the stop reading.
3.4.2.4.4.4.	Compare the new difference with the sum of the usage in the good intervals.
3.4.2.4.4.5.	If the difference is within the allowable margin calculated in step 3.4.2.3.2 - good intervals have passed Sum Check.
3.4.2.4.5.	If the difference is greater than the allowable margin, optionally reread meter.
3.4.2.4.5.1.	Redo sumcheck test from original start meter reading to new stop meter reading
3.4.2.4.5.2.	If the difference is within two meter multipliers - data has passed Sum Check
3.4.2.4.5.3.	If the difference is greater than two meter multipliers - kick out for manual inspection.  
3.4.2.4.6.	Other checks may be performed, based on the technology used, to verify that the interval data is an accurate representation of usage as measured by the meter readings.
3.4.2.4.7.	If sumcheck still not resolved, perform manual inspection of data
3.4.2.4.7.1.	Manually inspect the data.  If the data seems reasonable, it may be considered verified.
3.4.2.4.7.2.	Manual inspection is performed to verify meter and pulse multipliers.  If  a multiplier was wrong, recalculate the intervals using the new multiplier and redo the sum check.  If data now passes the sum check, data has passed the sum check.  
3.4.2.4.7.3.	  If multipliers are ok, check for meter changeout between the start and stop meter readings.  If the meter was changed out, redo the sum check for each meter independently.  If sum check passes, data has passed sum check.
3.4.2.4.7.4.	If multiplier was ok, do physical meter test/inspection.  If meter tests ok, data has passed sum check.  If meter does not test ok, data needs to be estimated.  If unable to visit site and perform meter test prior to posting the data, the data needs to be estimated.
3.4.2.4.8.	 If interval data is available, but there are no meter readings to perform the sum check with, manually inspect the data.  Data that appears to be reasonable (compared with historical data) can be considered to be verified.  Any data that does not appear to be reasonable needs to be estimated
3.4.3.	Spike check
This check may be done against either consumption or pulse data, provided the data scaling is consistent throughout the period.
3.4.3.1.	For each 24 hour period, determine highest and third highest peaks.  (Normally the 24 hour period is from midnight to midnight.  If the data is at the beginning and doesn’t start at midnight, use enough data from the next day of data to get 24 hours of data.  If the data is at the end and doesn’t stop at midnight, use enough data from the next to last day of data to get 24 hours.)  A spike check threshold of 10 pulses is used to eliminate false spikes for meters with very low usage.  If the highest peak is non-zero but less than or equal to the spike check threshold (10 pulses), the data passes the spike check.  If the highest peak is greater than the spike check threshold and (highest peak - third highest peak)/third highest peak> 1.8, a spike exists - the highest interval fails the spike check
3.4.3.2.	If interval fails spike check, optionally reread meter.
3.4.3.3.	If you get different value from reread, redo spike check.
3.4.3.4.	If value is the same on reread or you cannot reread the meter, kick it out for manual analysis 
3.4.3.5.	Manual analysis looks for similar patterns on other similar days.  May also optionally involve calling the customer to check for unusual conditions at the time of the spike.
3.4.3.6.	If no similar pattern or other legitimate reason for spike is found, the highest interval needs to be estimated.  If similar pattern or other reason for legitimate spike is found, data can be considered to be verified after  having failed the spike check 
3.4.4.	Additional check if interval kVARh data is available
This check may be done against either consumption or pulse data, provided the data scaling is consistent throughout the period.

Note that if multiple kWh channels map to a single kVARh channel, or multiple kVARh channels map to single kWh channel, the appropriate channels must be totaled prior to this check.  If any intervals are determined to require estimation, the MDMA may optionally investigate the situation to determine if the problem can be directly traced to specific kWh channels.  If this is the case, only data for those channels needs to be estimated.  If the problem is not determined to be attributed to specific channels, all channels summed into that interval need to be estimated.
3.4.4.1.	If there are any kWh intervals with zero consumption, check the corresponding kVARh interval.  
3.4.4.2.	If the corresponding kVARh interval is also zero or less than or equal to the kVARh check threshold of 4 pulses, the kWh data is considered to have passed the kVARh check.  (A kVARh check threshold is used to eliminate false errors for meters with very low usage.)
3.4.4.3.	If the corresponding kVARh interval is not zero, the kWh interval needs to be estimated.  If multiple kWh channels map to a single kVARh channel, the MDMA may optionally investigate the situation to determine if the problem can be directly traced to specific kWh channels.  If this is the case, only data for those channels needs to be estimated.  If the problem is not determined to be attributed to specific channels, all channels need to be estimated. 
3.5.	On the billing cycle for the meter…..
3.5.1.	High/low check
Note that for 1/1/98, this check may be done on good and verified data only (ie, not including any estimated intervals), or after validation and estimation (including good, verified, and estimated intervals), or both.  All parties should work toward doing this check both times in the future.

This check is done against consumption data, not pulses.
3.5.1.1.	If last year’s data is available, calculate average daily usage for same billing month last year; use summed VEE or historical billing interval data if available, if not use VEE or historical billing usage (ie, difference between register readings)
3.5.1.2.	Else if last year’s data is not available, calculate average daily usage for the previous billing month; use summed VEE or historical billing interval data if available, if not use VEE or historical billing usage (ie, difference between register readings).
3.5.1.3.	Else if previous billing month not available - don’t do this check!
3.5.1.4.	Calculate average daily usage for this billing month using either summed VEE data (if test includes estimated data) or summed good and verified data (if test does not include estimated data).  If there were any spikes in the data, don’t include those intervals in the average - prorate the daily average accordingly.
3.5.1.5.	If the difference between the two daily averages is greater than 50% of the historical daily average, flag data (all the intervals in the month) as failing high/low average check and kick out for manual analysis.  Else data has passed High/Low Check.  
If (|historical daily average - this month’s daily average)| <= .5* historical daily average) then data has passed the High/Low Check.  
If (|historical daily average - this month’s daily average)| > .5* historical daily average) then data (all intervals in the month) is flagged as failing high/low average check and kicked out for manual analysis. 

3.5.1.6.	If data is flagged for manual analysis - look at the recent history of that meter.  If monthly usage has been on a trend in the appropriate direction and this seems reasonable, data is considered to be verified.  If not, optionally check with customer for changed usage patterns.  If changed usage patterns match change in data, data is considered to be verified.  If no change in usage patterns reported by customer, perform physical meter test/inspection.  If a problem is found with the meter, the data needs to be estimated.  (Note: if the problem existed prior to this billing period, previously posted data will need to be adjusted and reposted.)   Also check to see if some of the data looks reasonable; reasonable looking data is considered verified.  (For example, this may happen if a meter fails sometime during the month - the data at the beginning of the month may be ok, while the data after the failure may be obviously bad.)
3.5.1.7.	If the meter is investigated and found to be accurate, the data is considered verified.
3.5.1.8.	If data fails high/low average check, suspect data needs to be estimated.
3.5.2.	If data has passed all appropriate validation tests without manual verification, data has been validated.  If data failed one or more validation tests but was manually verified, the specific tests the data failed and the fact that it was manually verified must be recorded on an interval level.  If data did not pass all appropriate validation tests or was not manually verified, the specific tests the data failed must be recorded on an interval level, and the data needs to be estimated.  If an interval fails more than one test, all failed tests must be recorded.
3.5.3.	Estimate data needing estimation; see section 4.  Note that for each interval that is estimated, the MDMA must record the estimation algorithm used.  The estimation algorithm will be reported when the data is posted when it is supported by CMEP.  Interval data estimation algorithms include:
•	less than 2 hours (4.1) 
•	greater than 2 hours (4.2)
•	time-drifted intervals prorated (Appendix A) 
•	intervals adjusted (4.3) 
•	intervals manually estimated (4.2.8) 
•	intervals estimated due to meter interval programmed incorrectly (4.4) 
•	Load profile template used (4.2.7)
3.5.4.	Optionally rerun checks after estimation to help verify estimated readings.  For example, this may include doing the sum check again if it is a mechanical meter..
4.	Interval Data Estimation Rules
Estimate intervals needing estimation using the following estimation rules….
4.1.	If section of data needing estimation is 2 hours or less in length…
Use the point to point linear interpolation method to estimate the data where there is a gap in the data.  If the gap occurs in the middle of the data, the “first point” is the last good interval before the gap, and the “second point” is the first good interval after the gap.

If the gap occurs at the beginning of the span, use the last interval from the historical data as the first point, if the historical data is available and the interval data is not estimated. Otherwise, use the second point (the first good interval after the gap) as the first point - this will cause the load to be estimated as a flat load. 

If the gap occurs at the end of the span, use the first point (the last good interval before the gap) as the second point (- this will cause the load to be estimated as a flat load.
4.2.	If section of data needing estimation is more than 2 contiguous hours…
Note that the days chosen to provide reference data in the subsequent steps may optionally be “partial” days, as long as the intervals being used as reference data are not themselves estimated and the reference days do not include a power failure.  This situation could occur in one of two ways:
a)	The use of reference days that contain estimated data for estimation as long as the period of time that you are going to use for your estimate does not contain any estimated data.  For example, if you are trying to estimate 3/18/98 from 2-5 PM, and 3/11/98 was estimated from 10-11 AM but not from 2-5 PM, 3/11/98 would be an acceptable day to include.
b)	The use of partial days for estimation, where data collection began at some time other than midnight.  This would only be encountered on the install date for a given meter/recorder.
	
4.2.1.	Find the three most recent “same weekdays” with valid data (ie, contain no estimated data during the intervals needing estimation or power failures) closest in time to the day with the data needing estimation.  The “same weekdays” can be chosen from the current billing period and historical data up to 90 days prior to the day with the data needing estimation.  Note that days from the current billing period may be a later date than the day needing estimation.  A “same weekday” means the same day of week.  Note that same weekdays from the same billing period are used first, and then same weekdays from historical data are used if necessary.  If the day with data needing estimation is a holiday, the “same weekdays” are holidays, not the same day of week.  If not enough historical holidays exist in the current billing period or  previous 90 day period, use Sundays. Note that a standard list of holidays will be used across the board, regardless of the tariff or service territory the meter is in.  The historical data used is that data immediately preceding the usage period, regardless of seasonal crossover.

Calculate the average daily profile using the three selected days.

Estimate the data needing estimation by applying the appropriate intervals from the average daily profile to fill the missing intervals.  If begin and end meter readings are available and are known to be good, they may optionally be used to scale the estimated interval data.  This is done by:
1.	determine the total usage during the time period (end meter reading - start meter reading)
2.	sum together all the good intervals (validated or verified)
3.	subtract the total of the good intervals from the total usage to determine the total estimated usage
4.	sum together all the estimated intervals
5.	calculate the scaling factor by dividing the total estimated usage (from step 3) with the sum of the estimated intervals (step 4)
6.	multiply each estimated interval by the scaling factor
4.2.2.	If only two same weekdays are available from the current billing period and 90 days historical data, calculate the average daily profile using the two selected days.  Estimate the data needing estimation by applying the appropriate intervals from the average daily profile to fill the missing intervals.
4.2.3.	If only one same weekday is available from the current billing period and 90 days historical data, estimate the data needing estimation by applying the appropriate intervals from the similar day to fill the intervals needing estimation.
4.2.4.	If no same weekdays are available in the current billing period and 90 days historical data, look for the three “like” days that are closest chronologically to the day with intervals needing estimation.  A like day means weekday or weekend/holiday.  For example, if the intervals needing estimation were on Tuesday, use Monday, Wednesday, and Thursday.  Only use weekdays with weekdays; only use weekends with weekends; only use holidays or weekends with holidays. Calculate the average daily profile using the three selected days.  Estimate the data by applying the appropriate intervals from the average daily profile to fill the intervals needing estimation.
4.2.5.	If no similar days are available and there aren’t three “like” days, select the two “like” days that are chronologically closest to the day with intervals needing estimation.  Calculate the average daily profile using the two selected days.  Estimate the data by applying the appropriate intervals from the average daily profile to fill the intervals needing estimation.
4.2.6.	If no similar days are available and there aren’t two “like” days, select the one “like” day that is chronologically closest to the day with intervals needing estimation.   Estimate the data by applying the appropriate intervals from the similar day to fill the intervals needing estimation.
4.2.7.	If there are no similar days or like days, use the load profile for the customer’s class to estimate the data.  Use this month’s usage if available to scale the load profile.  If this month’s usage is not available, use last month’s usage or last year’s usage, whichever is determined to be more reasonable, to scale the load profile,.  Refer to the UDCs’ load profile documentation for more information on applying load profiles.  Note that the load profiles are for hourly data.  If 15 minute data is required, assume a flat load throughout the hour (i.e., each 15 minute interval would have 1/4th the hourly usage).
4.2.8.	If there is no historical data that can be used, the data must be estimated manually and the process and assumptions documented.  This must be coordinated with the customer’s UDC and ESP within the 5 day window required to report interval data.
4.3.	If the cause of the problem is determined to be a physical problem with the meter or meter installation and the data can be corrected by scaling the intervals and meter readings, the MDMA will be notified of the time period requiring correction and the scaling factor to be applied to each interval in that period.  When the data is posted in the CMEP format, it would be marked as estimated if it had not been previously posted, and marked as adjusted if it had previously been posted.  Data that is corrected in this fashion may be reported in a separate category on the monthly MDMA reports, and not included as part of the % estimated data for the month.  (Details need to be worked out as to how MDMAs will receive this notification.  This has been forwarded to the PSWG.)  Examples of these situations include a meter running slow, a meter running fast, one or two phases dropped, ….
4.4.	Interval in meter doesn’t match tariff or settlement requirements
If the meter programming and the MDMA requirements are inconsistent, the data is calculated as follows:

a)	If the meter is programmed to collect data at a smaller interval than required by its tariffs, and the meter's interval evenly divides into the interval required by the tariff - for example, the meter was programmed to collect 5 minute data, and the tariff requires 15 minute data. Sum the 5 minute intervals into 15 minute intervals on even 15 minute boundaries.  If the data passed all the other Validation tests, it is considered validated.  
b)	If the meter is programmed to collect data at a larger interval than required by the meter’s tariff.  For example, the meter is programmed to collect 60 minute intervals, but the MDMA Server and tariff require 15 minute intervals.  Prorate the data by assuming an even load distribution during the interval.  In this example, the usage in the 60 minute interval would be divided by 4 to estimate the usage in a 15 minute interval.  The data is marked as estimated.  The meter needs to be reprogrammed to the correct interval.
c)	If the meter is programmed to collect data at a smaller interval than required by its tariff, but the meter’s interval doesn't evenly divide into the interval required by the tariff.  For example, the meter is programmed to collect 10 minute intervals, and the tariff requires 15 minute intervals. The data would be estimated and marked as estimated.  To estimate data, all collected intervals that are contained within the required reporting interval are included in the appropriate reporting interval.  Collected intervals that cross the boundaries of required reporting intervals are included proportionally in both reporting intervals.  In this example, if there were three 10 minute intervals containing 10 kWh, 20 kWh, and 30 kWh, the corresponding estimated 15 minute intervals would contain 20 kWh (10 + .5*20) and 40 kWh (.5*20 + 30).   This is similar to the prorating technique discussed in Appendix A.  The meter needs to be reprogrammed to the correct interval.
5.	Monthly Data Validation Rules
5.1.	Time check of Meter Reading Device/System
This test only applies for meter reading devices and systems collecting Time-of-Use data. Time check of  meter reading device/system - ensure that the collection device is synchronized to the national time standard before data collection begins
5.2.	Time tolerance check of meter
5.2.1.	If time in meter is within 3 minutes of time standard, the data has passed Time Tolerance check.  Note that if the meter is within 3 minutes of the standard, the time in the meter can optionally be corrected.
5.2.2.	If time in meter is off more than 3 minutes, ????  NEED RULES HERE  (Note: meter time must be corrected.  If the meter fails the time check for three consecutive months, the meter must be physically inspected/tested)  
5.3.	Hi/Lo Usage
	The hi/lo usage check validates that the usage (e.g. kWH, kVARH, water, and gas) meter read is within a 40% and 200% tolerance window of the highest daily average usage meter read based on either last year’s usage for the same billing month, the previous three billing months, or the previous billing month.
	If the meter read falls outside of the tolerance window, the meter read must be re-read or re-entered, and it must be flagged to indicate a failure of the hi/lo usage validation.
	The Ho/Lo Usage check validates cumulative consumption (kWh, kVARh).
5.3.1.	Calculate the daily average usage for the present billing period.  
Daily average = (current read - previous read)/(# days between reads)
5.3.2.	Calculate the historical daily average usage,
5.3.2.1.	If there is not at least one month of historical billing data available, skip this test.
5.3.2.2.	If data for the same customer and site is available, calculate the daily average for the same billing period last year.  Use this as the historical billing average usage.
5.3.2.3.	If data for the previous year’s billing period is not available but there is data for the last three billing periods, calculate the daily average for the last three billing periods and use it as the historical billing average usage.
5.3.2.4.	If neither of the previous two options are available but there is data for the last billing period, calculate the average daily usage for the last billing period and use this as the historical billing average usage.  
5.3.3.	Compare the present billing period average daily usage with the historical billing period average daily usage.  If the present billing period average daily usage is between 40% and 200% of the historical average daily usage, the data passes this test.
5.3.4.	If the present billing period average daily usage is not within 40% to 200% of the historical billing period average daily usage, optionally re-read the meter.  If the reread is essentially at the same time as the original read and a significantly different value is obtained, assume the first reading was a misread and perform the check again with the new reading.  Otherwise re-compute the average daily usage using the new reading.  If the new average daily usage is +/- 20% of the previous average daily usage, the data fails the Ho/Lo usage test but is validated.  If the re-read results in the same value, the data fails the Hi/Lo usage test but is verified..
5.3.5.	Data that fails the Hi/Lo usage test and has not verified may be investigated and manually verified if justification is found.  Else data must be estimated.
5.4.	Hi/Lo Demand
The hi/lo demand check validates the demand (e.g. kW and kVAR) meter read is within a 75% and 125% tolerance window of the highest average demand meter read based on either last year’s demand for the same billing month,  the previous three billing months, or the previous billing month.
If the meter read falls outside of the tolerance window, the meter read must be re-read or re-entered, and it must be flagged to indicate a failure of the hi-lo demand validation.
QUESTION - what if demand is TOU demand?  Do you do this for each “period” (ie, peak, off-peak, etc)?  YES
QUESTION - what if you have historic interval data for this customer, but the customer wasn’t on a demand rate a year ago, so there’s no historical demand?  Would you try to use the historical interval at all?  YES
5.4.1.	Determine the peak demand for this billing period.
5.4.2.	Determine the historical peak demand,
5.4.2.1.	If there is not at least one month of historical billing demand data available, skip this test.
5.4.2.2.	If demand data for this customer and site is available for the same billing period last year, use that as the historical peak demand.
5.4.2.3.	If data for the previous year’s billing period is not available but there is data for the last three billing periods, calculate the average peak demand for those billing periods.  Use this value as the historical peak demand.
5.4.2.4.	If neither of the previous two options are available but there is data for the last billing period, use the peak demand from the previous billing month as the historical peak demand.  
5.4.3.	Compare the present peak demand with the historical peak demand.  If the present peak demand is between 40% and 200% of the historical peak demand, the data passes this test.
5.4.4.	If the present peak demand is not within 40% to 200% of the historical peak demand, optionally re-read the meter.  If the reread is essentially at the same time as the original read and a significantly different value is obtained, assume the first reading was a misread and perform the check again with the new reading.  Otherwise If the new peak demand is +/- 20% of the previous peak demand, the data fails the Hi/Lo Demand test but is validated.  If the re-read results in the same value, the data fails the Hi/Lo Demand test but is verified..
5.4.5.	Data that fails the Hi/Lo Demand test and has not been verified may be investigated and manually verified if justification is found.  Else data must be estimated.
5.5.	TOU Usage
	The TOU usage check compares the sum of the current season peak kWH meter readings and the current season total kWH meter reading.
	The difference must be within plus or minus the number of active peaks for the current season.
	If the difference is greater than the tolerance, the MDMA must verify the data (i.e. re-reading, meter test, etc.) and flag the current season total kWH meter read to indicate a failure of the TOU usage validation.
5.5.1.	For the current season, calculate the total kWh by summing all the periods.
5.5.2.	Compare the calculated total kWh with the total kWh read from the meter.  If they are within +/- the number of periods summed together, the data passes the test.  If they are not, the data fails the TOU Usage test and must be estimated. ?????  Note that this test must be done in whatever units are read from the meter.  For example, if the meter provides kWh, the kWh values must be summed and compared.  If the meter provides pulses, the pulse values must be summed and compared.
5.6.	Zero Consumption for Active Meters
	The zero consumption check validates that if the meter is an active meter and the current reading is equal to the previous reading.
	If there is zero consumption, the MDMA must verify the meter read (i.e. re-reading, meter test, meter replacement, etc.) and flag the meter read to indicate a failure of the zero consumption validation.
5.6.1.	If the meter is an active meter (ie, is associated with a customer who has financial responsibility), calculate the usage for the present billing month.
5.6.2.	If the usage is greater than 0, the data passes the zero consumption check.
5.6.3.	If the usage is 0, mark the data as having failed the zero consumption check.  Optionally, verify the meter reading by re-reading the meter and/or testing the meter.  If a new meter reading is obtained, run all the tests again using the new data.  If 0 usage is determined to be legitimate (for example, a building or equipment that is only used seasonally), mark the data as verified.
5.6.4.	If the data fails the zero consumption check and is not verified, it must be estimated.
5.7.	Usage for Inactive Meters
	The inactive meter check validates that if the meter is an inactive meter and the current reading is not equal to the previous reading.
	If there is consumption, the MDMA must verify the meter read (i.e. re-reading, meter test, meter replacement, activate meter for new service, etc.) and flag the meter read to indicate a failure of the usage for inactive meters validation.
	An inactive meter is one for which there is no customer with financial responsibility.  Initial thoughts are that this does not apply to MDMAs.  This will be revisited at the next meeting.
	QUESTION - who will read an Inactive Meter in the new environment?  This assumes an inactive meter is one which is installed but for which there is no customer.
5.7.1.	If the meter is an inactive meter, compare this meter reading with the previous reading.  If the readings are equal, the reading passes the Usage for Inactive Meters check.
5.7.2.	If the readings are not equal, the MDMA may optionally re-read the meter to verify the current reading.  If the new reading equals the previous reading, the new reading passes the Usage for Inactive Meters check.
5.7.3.	If the readings are still not equal, the MDMA?? must investigate the situation to determine why there is usage on an inactive meter.  Possible reasons could include a bad meter, meter having been replaced but not reported, new service established but not reported,…Take appropriate action based on the results of the investigation.  Flag the reading as having failed the Usage for Inactive Meters check.
5.8.	Number of Dials on Meter
	The dial quantity check validates that the dial quantity of the meter read is different than the expected dial quantity.
	If the dial quantity is different, the MDMA must do further investigation (i.e. re-reading, correcting meter configuration, etc.) and flag the meter read to indicate that a meter read dial quantity was investigated and corrected.
	QUESTION - do you always estimate when this happens?  Is the data really invalid?
	
	This test applies to cumulative consumption only.  It checks that the number of “dials” (digits) reported in the read is consistent with the number of dials (or digits) on the meter display.  This test is performed for both remote and local reads.  Note that all technologies may not support reporting leading zeros.
5.8.1.	Determine the number of digits in the meter reading.
5.8.2.	If the number of digits in the meter reading is consistent with the number of digits/dials on the meter, the data passes this test.  If the meter reading technology reports leading zeros, the number of digits in the meter reading should match the number of digits/dials on the meter.  If the meter reading technology does not support reporting leading zeros, the number of digits in the meter reading should be less than or equal to the number of digits/dials on the meter.
5.8.3.	If the number of digits is not consistent, verify that the correct meter is being read and that it has the correct number of digits/dials.  If the correct meter is being read, the data failed the Number of Dials test, but is verified.  Note that records would need to be changed to accurately reflect the meter in the field.  If the correct meter isn’t being read, read the correct meter and restart the VEE process with the new data.
5.9.	Meter Read Dial Decimal Quantity Difference
	The dial decimal quantity check validates that the dial decimal quantity of the meter read is different than the expected dial decimal quantity.
	If the dial decimal quantity is different, the MDMA must do further investigation (i.e. re-reading, correcting meter configuration, etc.) and flag the meter read to indicate that a meter read dial decimal quantity was investigated and corrected.
	QUESTION - do you always estimate when this happens?  Is the data really invalid?
5.9.1.	When the meter is read on-site, the meter reader compares the number of decimal places displayed by the meter with the number of decimal places expected.  If they are the same, the reading passes the Meter Read Dial Decimal Quantity check.
5.9.2.	If they are not the same, the MDMA must investigate why the meter is different than indicated by their records and resolve the inconsistency.  This may involve reprogramming the meter.  The reading failed the Meter Read Dial Decimal Quantity Difference check.  If the data is determined to be good, it is verified.
5.10.	Meter Identification
	The external meter identification check validates that the external meter ID of the meter read is not different than expected.
	If the external meter ID is different, the MDMA must do further investigation (i.e. re-reading, correcting meter configuration, etc.) and flag the meter read to indicate that an external meter id mismatch was investigated and corrected.
	There are two types of Meter Identification checks depending on how the meter is read.  
a)	If the meter is read remotely or via its optical port, the Internal Meter Identification check is performed.  This verifies the meter’s internal identification (often its serial number) with the identification expected by the meter reading system.  If they match, the data passes this check.  If they don’t match, the MDMA must investigate why the meter is different than indicated by their records and resolve the inconsistency
b)	If the meter is read locally (including via an optical port), the External Meter Identification Check is performed.  This verifies the Meter ID on the meter nameplate with the Meter ID expected by the meter reading system  If they match, the data passes this check.  If they don’t match, the MDMA must investigate why the meter is different than indicated by their records and resolve the inconsistency.
6.	Monthly Data Estimation Rules
6.1.	Missing Usage Meter Read
	The MDMA must estimate all missing usage meter reads.
	In order to estimate the missing usage meter read, the MDMA must use last year’s daily average usage.  If that is not available, then the previous month’s daily average usage will be used.  If there is no historical data available, a daily average usage will be calculated utilizing the appropriate Load Profile.
	In order to estimate the current meter read, the daily average will be applied to the previous month’s meter read, projecting the usage on the current scheduled meter read date.
	All estimated usage meter reads will be flagged to indicate that the read is estimated.
	QUESTION - how do you use a Load Profile to calculate average daily usage when there is no historical data?
6.1.1.	Calculate average daily usage to be applied
6.1.1.1.	If billing data is available from the same customer and same site for the same billing period last year, calculate the average daily usage for the same billing period last year and use this value as the average daily usage.
6.1.1.2.	If there is no data from the previous year but there is data available from the previous billing month, calculate the average daily usage for the previous billing month and use this value for the average daily usage.
6.1.1.3.	If there is no historical data available, use the customer’s load profile to calculate an average daily usage ????? how???
6.1.2.	Calculate the number of days since last month’s meter reading to the end of this billing period.  (Or since the last good meter reading if you read the meter more than once a month???)
6.1.3.	Multiply the average daily usage by the number of days since the last good reading, and add that value to the last good reading.  This is the estimated meter reading.  Mark the reading as being estimated using the Missing Usage Meter Reading algorithm.
6.2.	Missing Demand Meter Read
	The MDMA must estimate all missing demand meter reads.
	In order to estimate the missing demand meter read, the MDMA must use last year’s demand.  If that is not available, then the previous month’s demand will be used.  If there is no historical data available, the Load Profile demand will be used.
	All estimated demand meter reads will be flagged to indicate that the read is estimated.
	QUESTION - how do you use a Load Profile to calculate peak demand when there is no historical data or you don’t have the meter reading?
6.2.1.	If a similar demand value is available from the same billing period for the previous year, use the previous year’s demand as the estimate.
6.2.2.	If data is not available from the previous year but is available for the previous billing period, use the previous billing period’s demand as the estimate.
6.2.3.	If no historical data is available, ????
6.3.	Missing TOU Meter Read??
QUESTION - do we need rules for this?

1. Introduction	
2. Required data validation tests	
3. Interval data collection and validation rules	
4. Interval Data Estimation Rules	
5. Monthly Data Validation Rules	
6. Monthly Data Estimation Rules