Automatic validation of files upload to the Performance Assurance area of the REC Portal has been introduced, being phased in since 28 October 2022.  It allows for files uploaded to the Portal to be checked in real-time against the valid file formats defined in the Performance Assurance Report Catalogue (PARC) schemas.

This functionality has been introduced for three main reasons:

1. To provide instant feedback to Portal Users, reducing the need for back-and-forth communication
2. To help Parties submit compliant files
3. To allow for further automation of the process to deliver a more efficient Code Manager Service.

Types of Validation

Validation can include the following types of check:

1. Is the file type valid? E.g. if the file is required to be a .csv, has a .csv file been uploaded?
2. Are the delimiters correct? E.g. if the file is a .csv, are the delimiters commas?
3. Is the file in a standard, supported encoding? I.e. is the file in either UTF-8 or ANSII encoding?
4. Are the fields provided correct? I.e. have all required fields been provided and are they in the correct order? Are there any additional or missing fields?
5. Has a header row been provided and are the field headers correct (relevant for all files except the ‘1340 Commercial/Residential Outcome File’)? I.e. Is there a header row, do the field header names exactly match the PARC (note this check is case sensitive)?
6. Is the data in the correct format? E.g. have date fields been specified in the required date format? Are numeric fields in a numeric format?
7. Have all mandatory fields been populated? I.e. are there any blank entries in mandatory fields?
8. Have valid values been provided? E.g. in a field where only certain values are allowed, are there any entries that do not match a value in the allowed list (note this check is case sensitive)?

Validation of Energy Theft Files

Validation of Energy Theft files, the 1340 Commercial and Residential Outcome Files, applies slightly differently because the required file structures are fundamentally different. Validation rule #5 above does not apply.

The Energy Theft files do not have a field header row, as per many of the other file schemas detailed in the PARC, instead they have a ‘Header Block’ – a first row of data containing key information about the file itself, which has a different number of columns to the rest of the file. They also contain a ‘Trailer Block’, which is a row of data at the end of the file containing a row count, also with a different number of columns to the rest of the file.

As a result, instead of validation check #5 being applied, the ‘Header Block’ and ‘Trailer Block’ are validated against checks #4 and #6-8 as listed above.

Please note that the 'Record Type' field is not considered as a column within file validation as it purely indicates the type of data contained on the row. In resolving file validation error messages references a specific columns and row number, please take this into consideration.

Rejection Messages

If a file uploaded does not pass validation a rejection message will be displayed referencing the type of error and the location of the error in the file. Note that file validation ceases once an error is found, therefore the error message displays only the details of the first error identified in the file, there may be others.

There are three main types of error message that can be displayed:

• File Submitted is Not Valid – this message occurs when either validation check #1 or #2 fails, for example, when attempting to upload a .xlsx file when only a .csv or .txt file is allowed
• The rejection reason is the file structure is incorrect and may contain missing columns that are ordered incorrectly. Validation failed on row [X] column [Y] – this error message is a result of a one or more of validation checks #3-5 failing. It is an error with the structure of the file. In most cases validation will fail on row 1, as validation will not be able to pass any further than this if the structure is wrong. The column referenced will also usually be the last column of the affected row.
• The rejection reason is the file content is incorrect and may contain data values that are missing, badly formatted or invalid. Validation failed on row [X], column [Y] – this error message is a result of validation one or more of validation checks #6-8 failing. It is an error with the content of the file rather than the structure. The row and column referenced is the location of the error identified.

There is one other error message that could be displayed for Energy Theft files only:

• Block [XYZ] does not exist in the uploaded file – this message occurs when either validation check #4 or #6-8 fail as mentioned in Validation of Energy Theft Files, for example, if a Header Block has not been provided, or the Record Type within the Header Block is '0' instead of '000' you may see this error message.

A small number of parties have also reported receiving an intermittent error message that does not pinpoint the location of the error in the file. The Code Manager is investigating the cause of this error. If you do receive this error message, please report it to the Code Manager via the REC Service Desk for assistance.

What to do if your file is rejected

Please follow the following steps:

1. Review the file against the relevant schema and requirements detailed in the PARC.
2. Review the list of validation checks performed above and check for any instances where they may be failing within the file.
3. Compare the file against the sample files provided on the Portal here for Supplier Parties. 
4. Make any required updates and re-attempt upload. This and the prior steps may need to be repeated multiple times if further error messages are displayed.
5. If after attempting the prior steps you are still unsure on the cause of error, please get in contact with the Code Manager via the REC Service Desk at enquiries@recmanager.co.uk, providing detail of the file you are attempting to submit and the error message received.

Please note that it is the responsibility of each Party to submit a valid file as inline with the requirements detailed in the PARC. The Code Manager is able to provide feedback and help in resolution, but the Code Manager is unable to correct and resubmit the file on behalf of a Party.

FAQ & Common Issues

1. I’m converting files to .csv format from MS Excel and it’s creating issues that cause my files to fail validation

Many of the issues we’ve seen that cause files to be rejected are the result of conversion from an Excel file format to a .csv or .txt file. Excel performs a certain level of autoformatting, which can result in issues such as MPxNs being converted to scientific number format or additional blank columns and/or rows being added.
The best way to avoid this is to avoid using Excel entirely, exporting data directly from your system or database to a .csv or .txt file. However, if that’s not possible, we recommend formatting the MPxN field to numeric format and removing the decimal points before conversion. We also recommend reviewing your file in a text editor (e.g. Notepad) post conversion, to correct any unhelpful autoformatting that has occurred.

2. I was able to submit my file without issue in the past, why is it now being rejected?

Prior to automatic file validation going live, files submitted went through a more manual form of validation post go-live, which may have allowed some discrepancy to the PARC requirements and schemas due to the volume of files received. The new automatic functionality employs a systematic validation to the schemas defined in the PARC, which may pick up errors in your file that were not previously identified.

We can assure Parties that no changes to file submission requirements or file schemas have changed as a result of the File Validation functionality in itself. Any changes to schemas or submission requirements are necessary to be accompanied by a formal PARC change. The change process includes a Party consultation period, so Parties will be informed and provided notice prior to any material change to a data item.   

3. What happens if my file is rejected, and it causes me to miss the file submission deadline?

In the event this occurs, please notify the Code Manager via the Performance Assurance mailbox or the Enquiries Helpdesk detailing the file type in question and the reason for the delay. Please make every effort to resolve the issue and submit a valid file as soon as you can.

4. The Portal is returning an issue referring to the last column on row 1, but everything on the header row looks fine to me

There are a few things worth checking:

• The field names in the header row exactly match that as specified in the PARC. File validation is both case sensitive and spacing sensitive.
• The fields are provided in the same order as in the PARC.
• No additional blank columns have been added to the end of the header row.
• The file is encoded in a standard format (UTF-8 or ANSII). We have seen instances where files provided in less common formats like UTF-8 BOM are rejected. The easiest way to check this is to use a text editor, like Notepad or Notepad++, it is particularly easy to change the encoding of a file in Notepad++. If you are converting to a .csv file from Excel, please make sure to save using the ‘CSV (Comma delimited) (*.csv)’ option to avoid this happening in the first place.

5. The Portal is taking a long time to validate my file, what do I do?

The Portal can take a long time to validate particularly large or complex files. If this is the case, validation will complete in the background and you will receive a notifcation once it has either completed, or it has returned a file validation error message. Please do not feel the need to remain on the webpage until completes if it is taking a long time. 

6. I uploaded my file under the 'Other' File Type dropdown, why does it not appear to have been processed?

Files uploaded to the 'Other' dropdown are not subject to File Validation and are not automatically processed by Code Manager systems. The 'Other' dropdown option should only be used if it has been explicitly requested by the Code Manager, or in the case you are unable to identify why your file is rejected and wish the Code Manager to provide feedback, as per step 5 in What to do if your file is rejected. Once the feedback has been provided and any issues are corrected, Parties are still required to reupload it via the appropriate dropdown option.

Useful Resources

• 12 January 2023 PA Webinar Recording – in which File Validation is discussed as an agenda item
• Performance Assurance Report Catalogue (PARC)
• Performance Assurance Report Catalogue Wiki