The M5 File format is the newest file format and was implemented for production data submission in September 2021. It is a JSON file format with header (called Interrogation File) and detail (called Interrogation Field Data) sections. The most current schema is 2022-04. Refer to www.json.org for information about JSON (JavaScript Object Notation) and to json-schema.org for information about schema used to validate JSON files. This free online validator can be used to manually validate M5 files against the schema. JSON is a standard data interchange format between systems and is human readable.
M5 and I5 software can be used to create files of this type.
File Structure
The file contents are composed of two tables with a one-to-many relationship. The Interrogation File table has one or more Interrogation Field Data associated with it.
Interrogation File Fields
The fields of the Interrogation File provide a global context to the related field data records, such as the location (Site Code) of where the events occurred and how to process the data in PTAGIS (Data Role).
|
Field |
Description |
Domain |
|
Schema |
Identifier and location of the M5 schema that this file should be validated against. |
URL identifying the location of the schema.json file |
|
File Name |
A unique file name associated with the dataset. |
SIT-YYYY-DOY-R-SEQ.json
SIT: 3-character site code YYYY: 4-digit year of the File Created date DOY: Day of Year of the File Created date R: Single character representing the Data Role (see table below for values) SEQ: Left-padded 3-digit integer representing the sequence number |
|
Site Code |
Three-character site code registered with PTAGIS representing the interrogation site where the data was collected. |
Valid interrogation site code registered with PTAGIS |
|
File Created |
Date and time denoting the start time of the monitoring period covered in this file. |
|
|
File Closed |
Date and time denoting the end time of the monitoring period covered in this file. |
|
|
Data Role |
Designates the type of data within this file and where it will be validated, stored and/or loaded into the PTAGIS database. |
|
|
System Description |
Describes the system that created this file, including any version information. |
Free text up to 100 characters. |
Data role describes the purpose of the data file and how PTAGIS should process it.
•Validate: schema validation is performed to verify the file format
•Load: contents of the a valid file will be processed into the database and available for reporting
•Stored: physical file will be stored in the PTAGIS file archive in perpetuity
|
Role Value |
Description |
Validate |
Load |
Store |
|
Primary (P) |
Primary or production dataset available via PTAGIS. |
X |
X |
X |
|
Secondary (S) |
Backup or secondary data from a redundant system. |
X |
|
X |
|
Patch (X) |
Patch data to supplement a production interrogation period. |
X |
X |
X |
|
Test Submission (T) |
For testing submission only. Any data inside of file will not be processed. |
|
|
X |
NOTE: The Data Role is associated as a component of the file name in abbreviated form (e.g. P) and again within the Interrogation File structure in explicit form (e.g. Primary).
Interrogation Field Data
The fields of the Interrogation Field Data describe the detail of an observation of a PIT tag or other message from a transceiver or other equipment.
|
Field |
Description |
Domain |
|
Field Data Type*^# |
Defines the type of data message and related properties. |
|
|
Timestamp*^# |
The local data and time the message was recorded. |
|
|
Field Data*^# |
A PIT tag observation, transceiver diagnostic, environmental or system metadata message recorded from a transceiver, data logger, PLC or other device. The message is recorded as it is transmitted and not parsed in any way. |
Free text up to 500 characters. |
|
Field Source*^# |
Describes the field device or other type of data source for the Field Data. |
|
|
System ID*^# |
A unique identifier registered with PTAGIS that identifies a master controller, transceiver unit or probe. |
Two-character hexadecimal ID |
|
Subsystem ID |
A unique identifier registered with PTAGIS that identifies an antenna or similar subsystem. This field should be specified if using the Tag Field Data Type and the data originates from a multiplexing transceiver where the transceiver ID is different from the antenna IDs. |
Two-character hexadecimal ID |
|
Tag Code^# |
A real-time tag code to be processed as an observation record. This field can be used in place of Field Data if a PIT tag code has already been parsed from the raw transceiver message. If this field exists, the data in Field Data will be ignored |
Valid hexadecimal PIT tag code |
|
Signal Level |
FDX-B tag detection signal level measurement in mV. This field can be used in addition to the Tag Code field when the PIT tag code and signal level have already been parsed from the raw transceiver message. |
Number |
|
Sbyc Action Type# |
The type of SbyC diversion taken with with the detection of this tag code. Only present if the Field Data Type = SbycTag. |
|
|
Sbyc Action Code# |
The SbyC action code of the tag detected.Only present if the Field Data Type = SbycTag. |
Integer |
|
Sbyc Gate Action# |
The SbyC gate action that was triggered with the detection of this tag code. Only present if the Field Data Type = SbycTag. |
|
|
Sbyc Processing Latency |
The latency in milliseconds to complete the SbyC action from when the tag is decoded.Only present if the Field Data Type = SbycTag. |
Number |
* Required if Field Data Type = Raw
^ Required if Field Data Type = Tag
# Required if Field Data Type = SbycTag
|
Type |
Description |
|
Raw |
Designates the message in Field Data is raw unparsed data. |
|
Tag |
Designates that the PIT tag code has already been parsed and is located in the Tag Code field. Data in Field Data will be ignored. |
|
SbycTag |
Designates the PIT tag code was parsed for SbyC operations and is located in the Tag Code field. SbyC fields will also be present. |
|
Event |
Records an event from a source other than a transceiver, such as a sample rate or facility gate status at a juvenile bypass facility. Available in schema 2022-04 and later. |
If creating an M5 file from parsed tag detection records, the Field Data Type should be set to 'Tag' indicating a real-time tag code or to 'SbycTag' indicating the tag was parsed for the purpose of conducting Separation by Code. Additional SbyC metadata will be included in SbycTag records. Most of the properties of Interrogation Field Data are optional (nullable) except where the schema defines required properties based upon the value of Field Data Type. The following examples are valid data and required fields are highlighted based upon the value of Field Data Type:
{
"$schema": "https://ptagis.org/content/schemas/interrogation/2022-04/schema.json",
"fileName": "B2J-2023-091-P-011.json",
"siteCode": "B2J",
"fileCreated": "2023-04-01T10:00:00.1082182-08:00",
"fileClosed": "2023-04-01T11:00:00.0978084-08:00",
"dataRole": "Primary",
"systemDescription": "M5 Monitor Service 2.0.2",
"interrogationFieldData": [
{
"fieldDataType": "Raw",
"timestamp": "2022-09-01T10:00:01-08:00",
"fieldData": "TAG: 08 2020 - 08 - 13T03: 26:16.944 3E7.0000001D08 511 mV",
"fieldSource": "Transceiver",
"systemID": "08",
"subsystemID": "08"
} ,
{
"fieldDataType": "Tag",
"timestamp": "2022-09-01T10:00:01-08:00",
"fieldData": "TAG: 08 2020 - 08 - 13T03: 26:16.944 3E7.0000001D08 511 mV",
"fieldSource": "Transceiver",
"systemID": "08",
"subsystemID": "08",
"tagCode": "3E7.0000001D08",
"signalLevel": 511
} ,
{
"fieldDataType": "SbycTag",
"timestamp": "2022-09-01T10:00:01-08:00",
"fieldData": "TAG: 08 2020 - 08 - 13T03: 26:16.944 3E7.0000001D08 511 mV",
"fieldSource": "Transceiver",
"systemID": "08",
"subsystemID": "08",
"tagCode": "3E7.0000001D08",
"signalLevel": 511,
"sbycActionType": "DefaultAll",
"sbycActionCode": 100,
"sbycGateActionType": "NoOp",
"sbycProcessingLatency": 100
}
{
"fieldDataType": "Event",
"timestamp": "2023-04-01T10:00:00.1105875-08:00",
"fieldData": "Sample Gate Open Status: Started",
"fieldSource": "Plc"
},
]
}
Because the schema only validates which properties are required based on Field Data Type value, the following example is also valid and the server would process this as raw data and ignore the value in Tag Code. Also note the required System ID field has a value of "GG" which the schema returns as valid even though it's an invalid transceiver identifier; this type of validation is handled when the file is processed for loading into the database.
{
"fieldDataType": "Raw",
"timestamp": "2020-01-01T05:00:02-07:00",
"fieldData": "TAG: 08 2020-08-13T03: 26:16.944 3E7.0000001D08 511 mV",
"fieldSource": "Transceiver",
"systemID": "GG",
"subsystemID": "08",
"tagCode": "3E7.0000001D08",
"signalLevel": 511
}
|
Type |
Description |
|
Transceiver |
Designates the message in Field Data originated from a transceiver and should be processed by PTAGIS. For example, a real-time tag detection or diagnostic report. |
|
Plc |
A message from a Programmable Logic Controller (PLC) typically used at mainstem hydro facilities. Messages of this type include metadata such as facility gate disposition or temperature/humidity level monitoring. |
|
Probe |
Output from a sensor probe to measure water depth, temperature or other environmental metadata at an interrogation site. |
|
System |
A system related message to indicate an operational behavior or error condition. Typically, these messages are associated with a real-time software system such as M5. |
|
Extraneous |
Typically identified by a formatting utility (e.g. I5) as extraneous and PTAGIS will ignore during processing. For example, the listing of available commands from a transceiver that was included in a memory download. |
|
Type |
Description |
|
None |
No action was triggered. |
|
DefaultIgnore |
Default action triggered because tag code is a timer or test tag. |
|
DefaultNotFound |
Action code associated with the tag code was not found in lookup table, so the default action was triggered. |
|
DefaultAll |
Divert by default, all tag codes trigger the same action. |
|
Override |
The override gate action was triggered based on the action code assigned to the tag code. |
|
DefaultSkip |
The default action was triggered because the override protocol instructed a skip (e.g. daily or seasonal limit has been reached or the protocol time period has been exceeded). |
|
DefaultNA |
The default gate action was triggered because the action code assigned to the tag code is not applicable at this diversion gate. |
|
Ignore |
No gate action was triggered because the tag is a timer tag. |
|
Duplicate |
No gate action was triggered because the tag code is a duplicate. |
|
StandBy |
No gate action was triggered because the SbyC service is in a stand-by state. |
|
Failed |
Indicates a divert request failed to trigger a PLC bit. |
|
Type |
Description |
|
NoOp |
Indicates the gate should take no action. This is often used to override a gate that opens by default for any PIT tag. |
|
Open |
Indicates the gate should open. |
|
Close |
Indicates the gate should close. |
|
Left |
Indicates the gate should open to the left side. |
|
Right |
Indicates the gate should open to the right side. |
Version: 1.08
Published: 11/12/2024