resistics.time.reader_spam module¶
-
class
resistics.time.reader_spam.TimeReaderSPAM(dataPath: str)[source]¶ Bases:
resistics.time.reader.TimeReaderData reader for SPAM data
SPAM data has the following characteristics:
SPAM raw data is single precision floats with unit Volts.
Getting unscaled samples returns data with unit mV for both the electric and magnetic fields. This is because gain is removed in unscaled samples to ensure consistency when a single recording is made up of multiple data files, each with different gain settings
The start time in XTR files is the time of the first sample in the data
The end time in XTR files is the time of the last sample in the data
In situations where a SPAM dataset is recorded in multiple small files, it is required that the recording is continuous.
Notes
Getting unscaled samples for SPAM data removes the gain rather than return exactly the values in the data files. In cases where there are multiple data files, it is not necessary that they have been recorded with the same gain. Therefore, to ensure consistency when looking at raw data, the gain is removed at the getUnscaledSamples stage rather than getPhysicalSamples, where it would have probably been more appropriate. This means that getUnscaledSamples returns data where all channels are in mV.
The scalings to convert the raw data to mV are stored in the ts_lsb chan header and calculated out as the header files are being read.
Todo
Implement reading of XTRX header files
- Attributes
- recChannelsDict
Channels in each data file
- dtypenp.float32
The data type
- numHeaderFilesint
The number of header files
- numDataFilesint
The number of data files
Methods
setParameters()
Set parameters specific to a data format
getUnscaledSamples(**kwargs)
Get raw, unscaled data
getDataFilesForSamples(startSample, endSample)
Get the data files that contribute to the requested samples so they can be read
getPhysicalSamples(**kwargs)
Get data in physical units
spamHeaders()
Get sections and section headers to be read in for SPAM data
chanDefaults()
Get defaults values for channel headers
readHeader()
Read SPAM header files
readHeaderXTR(headerFile)
Read a XTR header file
readHeaderXTRX(headerFile)
Read a XTRX header files
headersFromRawFile(rawFile, headers)
Read headers from the data files
mergeHeaders(headersList, chanHeadersList)
Merge the headers from all the data files
printDataFileList()
Get data file information as a list of strings
printDataFiles()
Print data file information to terminal
-
chanDefaults(self) → Dict[str, Any][source]¶ Get defaults for channel headers
- Returns
- Dict[str, Any]
Dictionary of headers for channels and default values
-
getDataFilesForSamples(self, startSample: int, endSample: int) → Tuple[List[str], List[List[int]], List[float]][source]¶ Get the data files that have to be read to cover the sample range
- Parameters
- startSampleint
Starting sample of the sample range
- endSamplesint
Ending sample of the sample range
- Returns
- dataFilesToRead
Time data object
-
getPhysicalSamples(self, **kwargs)[source]¶ Get data scaled to physical values
resistics uses field units, meaning physical samples will return the following:
Electrical channels in mV/km
Magnetic channels in mV
To get magnetic fields in nT, calibration needs to be performed
- Parameters
- chansList[str]
List of channels to return if not all are required
- startSampleint
First sample to return
- endSampleint
Last sample to return
- remaveragebool
Remove average from the data
- remzerosbool
Remove zeroes from the data
- remnans: bool
Remove NanNs from the data
- Returns
- TimeData
Time data object
Notes
The method getUnscaledSamples multiplies the raw data by the ts_lsb converting it to mV. Because gain is removed when getting the unscaledSamples and all channel data is in mV, the only calculation that has to be done is to divide by the dipole lengths (east-west spacing and north-south spacing).
To get magnetic fields in nT, they have to be calibrated.
-
getUnscaledSamples(self, **kwargs) → resistics.time.data.TimeData[source]¶ Get raw data from data file, returned in mV
SPAM raw data is single precision float with unit Volts. Calling this applies the ts_lsb calculated when the headers are read. This is because when a recording consists of multiple data files, each channel of each data file might have a different scaling. The only way to make the data consistent is to apply the ts_lsb scaling.
Therefore, this method returns the data in mV for all channels.
- Parameters
- chansList[str], optional
List of channels to return if not all are required
- startSampleint, optional
First sample to return
- endSampleint, optional
Last sample to return
- Returns
- TimeData
Time data object
-
headersFromRawFile(self, rawFile: str, headers: Dict) → None[source]¶ Read headers from the raw data files
Read the headers from the raw file and figure out the data byte offset.
- Parameters
- rawFilestr
The .RAW data file
- headersDict
A headers dictionary
Notes
Open with encoding ISO-8859-1 because it has a value for all bytes unlike other encoding. In particular, want to find number of samples and the size of the header. The extended header is ignored.
-
mergeHeaders(self, headersList: List, chanHeadersList: List) → None[source]¶ Merge headers from all the header files
Checks all the header files to see if there are any gaps and calculates the sample ranges for each file together with the total number of samples. Sets the start and end time of the recording and class variables datetimeStart and datetimeStop.
- Parameters
- headersListList
List of headers from each data file
- chanHeadersList :
List of chan headers from each data file
-
printDataFileList(self) → List[str][source]¶ Information about the data files as a list of strings
- Returns
- List[str]
List of information about the data files
-
readHeader(self) → None[source]¶ Read header files
For SPAM data, the may be more than one header file as data can be split up into smaller files as it is recorded. In that case, the header information should be somehow merged.
All sampling frequencies should be the same
-
readHeaderXTR(self, headerFile: str) → None[source]¶ Read a XTR header file
The raw data for SPAM is in single precision Volts. However, if there are multiple data files for a single recording, each one may have a different gain. Therefore, a scaling has to be calculated for each data file and channel. This scaling will convert all channels to mV.
For the most part, this method only reads recording information. However, it does additionally calculate out the lsb scaling and store it in the ts_lsb channel header. More information is provided in the notes.
- Parameters
- headerFilestr
The XTR header file to read in
Notes
The raw data for SPAM is in single precision floats and record the raw Voltage measurements of the sensors. However, if there are multiple data files for a single continuous recording, each one may have a different gain. Therefore, a scaling has to be calculated for each data file.
For electric channels, the scaling begins with the scaling provided in the header file in the DATA section. This incorporates any gain occuring in the device. This scaling is further amended by a conversion to mV and polarity reversal,
\[scaling = read scaling from DATA section of header file \ scaling = 1000 * scaling , \ scaling = -1000 * scaling , \ ts_lsb = scaling , \]where the reason for the 1000 factor in line 2 is not clear, nor is the polarity reversal. However, this information was provided by people more familiar with the data format.
For magnetic channels, the scaling in the header file DATA section is ignored. This is because it includes a static gain correction, which would be duplicated at the calibration stage. Therefore, this is not included at this point.
\[scaling = -1000 , \ ts_lsb = scaling ,\]This scaling converts the magnetic data from V to mV.