Recap and deeper dive¶
Now that many of the basic features of resistics have been introduced, it is a useful time to recap some theory and consider some lower level resistics API. The DecimationParameters
, WindowParameters
and WindowSelector
classes are central to how resistics processes magnetotelluric data. They are lower level elements of the resistics package but are useful to know and understand, particularly before approaching the Advanced and Cookbook sections.
Recap¶
Ahead of introducing the window selector, it is useful to do a quick recap of how resistics processes magnetotelluric data. Ultimately, the point of processing magnetotelluric data is to understand the electrical properties of the subsurface.
In the standard magnetotelluric setup, the electrical properties of the subsurface are described by the impedance tensor, which relates the input magnetic fields to the output electric fields. This is usually posed in the frequency domain, as shown below. Please see recommended literature if this is unfamiliar.
Here, the dependency on frequency (\(\omega\)) is shown explicitly.
In the field, \(E_{x}\), \(E_{y}\), \(H_{x}\), \(H_{y}\) (and often \(H_{z}\)) are recorded. These are then converted into the frequency domain using the Fourier transform, or in resistics, calculating spectra. To read more about resistics spectra, see Up and running and Viewing spectra.
Resistics then processes the magnetotelluric spectra data to try and estimate the four unknown values of \(Z_{xx}\), \(Z_{yy}\), \(Z_{xy}\), \(Z_{yx}\). But in the above, there are two equations and four unknowns, so somehow, more equations need to be generated. This is normally done by by using cross-powers as shown in the below set of equations,
where \(*\) represents the complex conjugate.
This equations are solved at a set of frequencies called the evaluation frequencies. These are normally chosen on a log scale.
So theoretically, a magnetotelluric processing system could do the following:
Transform full electric and magnetic field time series measurements to frequency domain
Calculate cross-power spectra
Select a set of evaluation frequencies
For each evaluation frequency, perform linear regression to solve for \(Z_{xx}\), \(Z_{yy}\), \(Z_{xy}\), \(Z_{yx}\)
However, the impedance tensor estimates can be improved by providing more equations or by stacking cross-power spectra, especially in the presence of electromagnetic noise. This is usually done by windowing the time series, in which case the processing flow looks like this:
Choose window parameters (window size and overlap size between windows)
Transform each window into frequency domain (calculate spectra)
Calculate cross-power spectra at a set of evaluation frequencies
Stack the cross-power spectra for the different windows
For each evaluation frequency, perform linear regression to solve for \(Z_{xx}\), \(Z_{yy}\), \(Z_{xy}\), \(Z_{yx}\)
There are other variants on this and stacking strategies. It is not necessary for stacking to be performed and instead cross-powers from a set of randomly selected windows can be used. Further, linear regression can be replaced by robust regression.
For the purposes of calculating out the longer period evaluation frequencies, the time data is usually decimated before the process is repeated. Explicitly, this means:
For decimation level in number of decimation levels
Decimate time series by the decimation factor
Choose window parameters for decimation level
Select evaluation frequencies for decimation level
Transform each window into frequency domain (calculate spectra)
Calculate cross-power spectra at the evaluation frequencies
Stack the cross-power spectra for the different windows
For each evaluation frequency, perform linear regression to solve for \(Z_{xx}\), \(Z_{yy}\), \(Z_{xy}\), \(Z_{yx}\)
In the following sections, the resistics approach to decimation parameters, window parameters and window selection will be detailed. It is not necessary to know this, but this information will be a useful pre-cursor to exploring the Advanced and Cookbook sections.
Decimation parameters¶
The DecimationParameters
class holds the decimation information. Decimation parameters can be set using configuration files (see Using configuration files). To see what the DecimationParameters
class holds, consider an example.
1 2 3 4 5 6 7 8 9 10 11 | from datapaths import projectPath, imagePath
from resistics.project.io import loadProject
# load project and configuration file
projData = loadProject(projectPath, configFile="tutorialConfig.ini")
# get decimation parameters
from resistics.project.shortcuts import getDecimationParameters
decimationParameters = getDecimationParameters(4096, projData.config)
decimationParameters.printInfo()
|
The project has been loaded along with the configuration file, which has specified 8 decimation levels and 5 evaluation frequencies per level. The decimation paramters can be printed to the terminal using the parent class printInfo()
method to give:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | 09:50:02 DecimationParams: ####################
09:50:02 DecimationParams: DECIMATIONPARAMS INFO BEGIN
09:50:02 DecimationParams: ####################
09:50:02 DecimationParams: Sampling frequency = 4096.000000
09:50:02 DecimationParams: Number of decimation levels = 8
09:50:02 DecimationParams: Level = 0 sample freq. [Hz] = 4096.000000 sample rate [s] = 0.000244 dec. factor = 0000001 inc. factor = 1
09:50:02 DecimationParams: Level = 1 sample freq. [Hz] = 1024.000000 sample rate [s] = 0.000977 dec. factor = 0000004 inc. factor = 4
09:50:02 DecimationParams: Level = 2 sample freq. [Hz] = 256.000000 sample rate [s] = 0.003906 dec. factor = 0000016 inc. factor = 4
09:50:02 DecimationParams: Level = 3 sample freq. [Hz] = 32.000000 sample rate [s] = 0.031250 dec. factor = 0000128 inc. factor = 8
09:50:02 DecimationParams: Level = 4 sample freq. [Hz] = 8.000000 sample rate [s] = 0.125000 dec. factor = 0000512 inc. factor = 4
09:50:02 DecimationParams: Level = 5 sample freq. [Hz] = 1.000000 sample rate [s] = 1.000000 dec. factor = 0004096 inc. factor = 8
09:50:02 DecimationParams: Level = 6 sample freq. [Hz] = 0.250000 sample rate [s] = 4.000000 dec. factor = 0016384 inc. factor = 4
09:50:02 DecimationParams: Level = 7 sample freq. [Hz] = 0.031250 sample rate [s] = 32.000000 dec. factor = 0131072 inc. factor = 8
09:50:02 DecimationParams: Evaluation frequencies [Hz]
09:50:02 DecimationParams: Level = 0: 1024.00000000, 724.07734394, 512.00000000, 362.03867197, 256.00000000
09:50:02 DecimationParams: Level = 1: 181.01933598, 128.00000000, 90.50966799, 64.00000000, 45.25483400
09:50:02 DecimationParams: Level = 2: 32.00000000, 22.62741700, 16.00000000, 11.31370850, 8.00000000
09:50:02 DecimationParams: Level = 3: 5.65685425, 4.00000000, 2.82842712, 2.00000000, 1.41421356
09:50:02 DecimationParams: Level = 4: 1.00000000, 0.70710678, 0.50000000, 0.35355339, 0.25000000
09:50:02 DecimationParams: Level = 5: 0.17677670, 0.12500000, 0.08838835, 0.06250000, 0.04419417
09:50:02 DecimationParams: Level = 6: 0.03125000, 0.02209709, 0.01562500, 0.01104854, 0.00781250
09:50:02 DecimationParams: Level = 7: 0.00552427, 0.00390625, 0.00276214, 0.00195312, 0.00138107
09:50:02 DecimationParams: ####################
09:50:02 DecimationParams: DECIMATIONPARAMS INFO END
09:50:02 DecimationParams: ####################
|
The information provides all the decimation factors and the sampling frequencies at each decimation level. It also provides the 5 evaluation frequencies for each decimation level.
Window parameters¶
The WindowParameters
class contains information about the windowing parameters. Window parameters can be set using configuration files (see Using configuration files).
Again, consider an example which continues from the decimation paramters example:
13 14 15 16 17 | # get the window parameters
from resistics.project.shortcuts import getWindowParameters
windowParameters = getWindowParameters(decimationParameters, projData.config)
windowParameters.printInfo()
|
The project has been loaded along with the configuration file, which has specified a minimum window size of 256 samples and a minimum overlap of 64 samples. However, resistics is free to use window and overlap sizes greater than these if it makes sense. Windowing parameter information can be printed to the terminal using the printInfo()
method.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | 09:50:02 WindowParams: ####################
09:50:02 WindowParams: WINDOWPARAMS INFO BEGIN
09:50:02 WindowParams: ####################
09:50:02 WindowParams: Number of decimation levels = 8
09:50:02 WindowParams: Level = 0, sample freq. [Hz] = 4096.000000, sample rate [s] = 0.000244
09:50:02 WindowParams: Window size = 2048, window duration [s] = 0.499756
09:50:02 WindowParams: Window overlap = 512, overlap duration [s] = 0.124756
09:50:02 WindowParams: Level = 1, sample freq. [Hz] = 1024.000000, sample rate [s] = 0.000977
09:50:02 WindowParams: Window size = 512, window duration [s] = 0.499023
09:50:02 WindowParams: Window overlap = 128, overlap duration [s] = 0.124023
09:50:02 WindowParams: Level = 2, sample freq. [Hz] = 256.000000, sample rate [s] = 0.003906
09:50:02 WindowParams: Window size = 256, window duration [s] = 0.996094
09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 0.246094
09:50:02 WindowParams: Level = 3, sample freq. [Hz] = 32.000000, sample rate [s] = 0.031250
09:50:02 WindowParams: Window size = 256, window duration [s] = 7.968750
09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 1.968750
09:50:02 WindowParams: Level = 4, sample freq. [Hz] = 8.000000, sample rate [s] = 0.125000
09:50:02 WindowParams: Window size = 256, window duration [s] = 31.875000
09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 7.875000
09:50:02 WindowParams: Level = 5, sample freq. [Hz] = 1.000000, sample rate [s] = 1.000000
09:50:02 WindowParams: Window size = 256, window duration [s] = 255.000000
09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 63.000000
09:50:02 WindowParams: Level = 6, sample freq. [Hz] = 0.250000, sample rate [s] = 4.000000
09:50:02 WindowParams: Window size = 256, window duration [s] = 1020.000000
09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 252.000000
09:50:02 WindowParams: Level = 7, sample freq. [Hz] = 0.031250, sample rate [s] = 32.000000
09:50:02 WindowParams: Window size = 256, window duration [s] = 8160.000000
09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 2016.000000
09:50:02 WindowParams: ####################
09:50:02 WindowParams: WINDOWPARAMS INFO END
09:50:02 WindowParams: ####################
|
In this case, for the 4096 Hz data, the for the lower decimation levels, larger window sizes are used but as the decimation increases, the 256 minimum window sample size and 64 minimum overlap sample size is used.
Window selector¶
The WindowSelector
class is a key component of resistics. Before processing time windows (and their respective spectra) to calculate transfer function estimates, the windows to use have to be selected. The WindowSelector
class is the how windows are selected.
Below are a few scenarios under which windows might have to be selected:
Single site processing, use all the data
Single site processing with masks and/or date and time constraints
Remote reference processing with two sites with masks and/or date and time constraints
Intersite processing with two sites
Intersite processing with a remote reference as a third site
Let’s compare 1 and 2 as simple examples. The others will be introduced in the Advanced section.
Begin by getting a WindowSelector
instance. In this case, the 128 Hz data is being used.
24 25 26 27 28 29 | from resistics.project.shortcuts import getWindowSelector
decimationParameters = getDecimationParameters(128, projData.config)
windowParameters = getWindowParameters(decimationParameters, projData.config)
selector = getWindowSelector(projData, decimationParameters, windowParameters)
selector.printInfo()
|
The WindowSelector
needs to know the DecimationParameters
and WindowParameters
. Information can be printed to the terminal using the printInfo()
method. This gives:
1 2 3 4 5 6 7 8 9 10 | 10:01:53 WindowSelector: ####################
10:01:53 WindowSelector: WINDOWSELECTOR INFO BEGIN
10:01:53 WindowSelector: ####################
10:01:53 WindowSelector: Sampling frequency [Hz] = 128.000000
10:01:53 WindowSelector: Spectra directory = dec8_5
10:01:53 WindowSelector: Sites =
10:01:53 WindowSelector: Site information:
10:01:53 WindowSelector: ####################
10:01:53 WindowSelector: WINDOWSELECTOR INFO END
10:01:53 WindowSelector: ####################
|
Currently, no sites have been specified for the WindowSelector
instance. Therefore, there are yet no windows to select from. The next step then is to specify a list of sites.
26 27 28 | # add a site and print the information to the terminal
selector.setSites(["site1"])
selector.printInfo()
|
Printing information gives the following, which details the recordings and the global windows for those recordings. The global windows are the window numbers beginning from the project reference time. Here there is nothing for decimation level 7 as the time series was not long enough to give sufficient windows after decimation.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 10:03:44 WindowSelector: ####################
10:03:44 WindowSelector: WINDOWSELECTOR INFO BEGIN
10:03:44 WindowSelector: ####################
10:03:44 WindowSelector: Sampling frequency [Hz] = 128.000000
10:03:44 WindowSelector: Spectra directory = dec8_5
10:03:44 WindowSelector: Sites = site1
10:03:44 WindowSelector: Site information:
10:03:44 WindowSelector: Sampling frequency [Hz] = 128.000000
10:03:44 WindowSelector: Site = site1
10:03:44 WindowSelector: Site global index information
10:03:44 WindowSelector: Decimation Level = 0
10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:30:00 - 2012-02-11 23:03:43.992188 Global Indices Range = 27600 - 112948
10:03:44 WindowSelector: Decimation Level = 1
10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:30:00 - 2012-02-11 23:03:43.968750 Global Indices Range = 6900 - 28236
10:03:44 WindowSelector: Decimation Level = 2
10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:30:00 - 2012-02-11 23:03:43.875000 Global Indices Range = 1725 - 7058
10:03:44 WindowSelector: Decimation Level = 3
10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:31:12 - 2012-02-11 23:03:27 Global Indices Range = 216 - 881
10:03:44 WindowSelector: Decimation Level = 4
10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:31:12 - 2012-02-11 23:00:12 Global Indices Range = 54 - 219
10:03:44 WindowSelector: Decimation Level = 5
10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:56:48 - 2012-02-11 22:38:24 Global Indices Range = 7 - 26
10:03:44 WindowSelector: Decimation Level = 6
10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 13:39:12 - 2012-02-11 19:12:00 Global Indices Range = 2 - 5
10:03:44 WindowSelector: Decimation Level = 7
10:03:44 WindowSelector: ####################
10:03:44 WindowSelector: WINDOWSELECTOR INFO END
10:03:44 WindowSelector: ####################
|
Now the shared windows can be calculated using the calcSharedWindows()
method. In this case, only a single site is specified so there are no other sites to share windows with. When more than a single site is specified, the shared windows between the sites will be calculated. This means the times where both sites were recording. This is an essential step for remote reference and intersite processing (see Advanced).
30 31 32 | # calculate shared windows
selector.calcSharedWindows()
selector.printSharedWindows()
|
The shared windows can be printed to the terminal using the printSharedWindows()
method.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | 10:04:19 WindowSelector::shared windows: ####################
10:04:19 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO BEGIN
10:04:19 WindowSelector::shared windows: ####################
10:04:19 WindowSelector::shared windows: Decimation Level = 0
10:04:19 WindowSelector::shared windows: Number of shared windows = 85349
10:04:19 WindowSelector::shared windows: Shared window indices: 27600-112948:1
10:04:19 WindowSelector::shared windows: Number of unmasked windows: 85349
10:04:19 WindowSelector::shared windows: Decimation Level = 1
10:04:19 WindowSelector::shared windows: Number of shared windows = 21337
10:04:19 WindowSelector::shared windows: Shared window indices: 6900-28236:1
10:04:19 WindowSelector::shared windows: Number of unmasked windows: 21337
10:04:19 WindowSelector::shared windows: Decimation Level = 2
10:04:19 WindowSelector::shared windows: Number of shared windows = 5334
10:04:19 WindowSelector::shared windows: Shared window indices: 1725-7058:1
10:04:19 WindowSelector::shared windows: Number of unmasked windows: 5334
10:04:19 WindowSelector::shared windows: Decimation Level = 3
10:04:19 WindowSelector::shared windows: Number of shared windows = 666
10:04:19 WindowSelector::shared windows: Shared window indices: 216-881:1
10:04:19 WindowSelector::shared windows: Number of unmasked windows: 666
10:04:19 WindowSelector::shared windows: Decimation Level = 4
10:04:19 WindowSelector::shared windows: Number of shared windows = 166
10:04:19 WindowSelector::shared windows: Shared window indices: 54-219:1
10:04:19 WindowSelector::shared windows: Number of unmasked windows: 166
10:04:19 WindowSelector::shared windows: Decimation Level = 5
10:04:19 WindowSelector::shared windows: Number of shared windows = 20
10:04:19 WindowSelector::shared windows: Shared window indices: 7-26:1
10:04:19 WindowSelector::shared windows: Number of unmasked windows: 20
10:04:19 WindowSelector::shared windows: Decimation Level = 6
10:04:19 WindowSelector::shared windows: Number of shared windows = 4
10:04:19 WindowSelector::shared windows: Shared window indices: 2-5:1
10:04:19 WindowSelector::shared windows: Number of unmasked windows: 4
10:04:19 WindowSelector::shared windows: Decimation Level = 7
10:04:19 WindowSelector::shared windows: Number of shared windows = 0
10:04:19 WindowSelector::shared windows: Shared window indices:
10:04:19 WindowSelector::shared windows: Number of unmasked windows: 0
10:04:19 WindowSelector::shared windows: NOTE: These are the shared windows at each decimation level. Windows for each evaluation frequency might vary depending on masks
10:04:19 WindowSelector::shared windows: ####################
10:04:19 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO END
10:04:19 WindowSelector::shared windows: ####################
|
The WindowSelector
has functionality to limit windows using date and time constraints. Date and time constraints can be provided using the addDateConstraint()
, addTimeConstraint()
and addDatetimeConstraint()
methods.
Important
Providing a date or time constraint means that only windows which are inside these date/time constraints will be selected. Any windows which are outside the date/time constraint will be discarded.
Below is an example of providing a date constraint using the addDateConstraint()
method. After running the calcSharedWindows()
method, only time windows within this date/time constraint will be selected.
34 35 36 37 38 | # currently using every window, but how might this change with a date or time constraint
selector.addDateConstraint("2012-02-11")
selector.printDatetimeConstraints()
selector.calcSharedWindows()
selector.printSharedWindows()
|
The datetime constraints can be printed to the terminal using the printDatetimeConstraints()
method. The various print methods give the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | 10:11:24 WindowSelector::datetime constraints: ####################
10:11:24 WindowSelector::datetime constraints: WINDOWSELECTOR::DATETIME CONSTRAINTS INFO BEGIN
10:11:24 WindowSelector::datetime constraints: ####################
10:11:24 WindowSelector::datetime constraints: Datetime constraints
10:11:24 WindowSelector::datetime constraints: Decimation Level = 0
10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59
10:11:24 WindowSelector::datetime constraints: Decimation Level = 1
10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59
10:11:24 WindowSelector::datetime constraints: Decimation Level = 2
10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59
10:11:24 WindowSelector::datetime constraints: Decimation Level = 3
10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59
10:11:24 WindowSelector::datetime constraints: Decimation Level = 4
10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59
10:11:24 WindowSelector::datetime constraints: Decimation Level = 5
10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59
10:11:24 WindowSelector::datetime constraints: Decimation Level = 6
10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59
10:11:24 WindowSelector::datetime constraints: Decimation Level = 7
10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59
10:11:24 WindowSelector::datetime constraints: ####################
10:11:24 WindowSelector::datetime constraints: WINDOWSELECTOR::DATETIME CONSTRAINTS INFO END
10:11:24 WindowSelector::datetime constraints: ####################
10:11:24 WindowSelector Info: Decimation level = 0. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 57600 - 115199
10:11:24 WindowSelector Info: Decimation level = 1. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 14400 - 28799
10:11:24 WindowSelector Info: Decimation level = 2. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 3600 - 7199
10:11:24 WindowSelector Info: Decimation level = 3. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 450 - 899
10:11:24 WindowSelector Info: Decimation level = 4. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 113 - 224
10:11:24 WindowSelector Info: Decimation level = 5. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 15 - 28
10:11:24 WindowSelector Info: Decimation level = 6. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 4 - 7
10:11:24 WindowSelector Info: Decimation level = 7. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 1 - 1
10:11:24 WindowSelector::shared windows: ####################
10:11:24 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO BEGIN
10:11:24 WindowSelector::shared windows: ####################
10:11:24 WindowSelector::shared windows: Decimation Level = 0
10:11:24 WindowSelector::shared windows: Number of shared windows = 55349
10:11:24 WindowSelector::shared windows: Shared window indices: 57600-112948:1
10:11:24 WindowSelector::shared windows: Number of unmasked windows: 55349
10:11:24 WindowSelector::shared windows: Decimation Level = 1
10:11:24 WindowSelector::shared windows: Number of shared windows = 13837
10:11:24 WindowSelector::shared windows: Shared window indices: 14400-28236:1
10:11:24 WindowSelector::shared windows: Number of unmasked windows: 13837
10:11:24 WindowSelector::shared windows: Decimation Level = 2
10:11:24 WindowSelector::shared windows: Number of shared windows = 3459
10:11:24 WindowSelector::shared windows: Shared window indices: 3600-7058:1
10:11:24 WindowSelector::shared windows: Number of unmasked windows: 3459
10:11:24 WindowSelector::shared windows: Decimation Level = 3
10:11:24 WindowSelector::shared windows: Number of shared windows = 432
10:11:24 WindowSelector::shared windows: Shared window indices: 450-881:1
10:11:24 WindowSelector::shared windows: Number of unmasked windows: 432
10:11:24 WindowSelector::shared windows: Decimation Level = 4
10:11:24 WindowSelector::shared windows: Number of shared windows = 107
10:11:24 WindowSelector::shared windows: Shared window indices: 113-219:1
10:11:24 WindowSelector::shared windows: Number of unmasked windows: 107
10:11:24 WindowSelector::shared windows: Decimation Level = 5
10:11:24 WindowSelector::shared windows: Number of shared windows = 12
10:11:24 WindowSelector::shared windows: Shared window indices: 15-26:1
10:11:24 WindowSelector::shared windows: Number of unmasked windows: 12
10:11:24 WindowSelector::shared windows: Decimation Level = 6
10:11:24 WindowSelector::shared windows: Number of shared windows = 2
10:11:24 WindowSelector::shared windows: Shared window indices: 4,5
10:11:24 WindowSelector::shared windows: Number of unmasked windows: 2
10:11:24 WindowSelector::shared windows: Decimation Level = 7
10:11:24 WindowSelector::shared windows: Number of shared windows = 0
10:11:24 WindowSelector::shared windows: Shared window indices:
10:11:24 WindowSelector::shared windows: Number of unmasked windows: 0
10:11:24 WindowSelector::shared windows: NOTE: These are the shared windows at each decimation level. Windows for each evaluation frequency might vary depending on masks
10:11:24 WindowSelector::shared windows: ####################
10:11:24 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO END
10:11:24 WindowSelector::shared windows: ####################
|
Comparing the number of shared windows to the previous results, it is clear that the number of shared windows has now reduced. This is due to the application of the date constraint. To reset the date constraint, the resetDatetimeConstraints()
method can be used as in the following example, where the current date/time constraint is cleared and a new one added.
40 41 42 43 44 45 | # reset datetime constraints
selector.resetDatetimeConstraints()
selector.addDatetimeConstraint("2012-02-10 19:00:00", "2012-02-11 08:00:00")
selector.printDatetimeConstraints()
selector.calcSharedWindows()
selector.printSharedWindows()
|
Calculating the shared windows again with this new datetime constraint and printing the information to the terminal gives:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | 10:11:47 WindowSelector::datetime constraints: ####################
10:11:47 WindowSelector::datetime constraints: WINDOWSELECTOR::DATETIME CONSTRAINTS INFO BEGIN
10:11:47 WindowSelector::datetime constraints: ####################
10:11:47 WindowSelector::datetime constraints: Datetime constraints
10:11:47 WindowSelector::datetime constraints: Decimation Level = 0
10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00
10:11:47 WindowSelector::datetime constraints: Decimation Level = 1
10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00
10:11:47 WindowSelector::datetime constraints: Decimation Level = 2
10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00
10:11:47 WindowSelector::datetime constraints: Decimation Level = 3
10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00
10:11:47 WindowSelector::datetime constraints: Decimation Level = 4
10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00
10:11:47 WindowSelector::datetime constraints: Decimation Level = 5
10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00
10:11:47 WindowSelector::datetime constraints: Decimation Level = 6
10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00
10:11:47 WindowSelector::datetime constraints: Decimation Level = 7
10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00
10:11:47 WindowSelector::datetime constraints: ####################
10:11:47 WindowSelector::datetime constraints: WINDOWSELECTOR::DATETIME CONSTRAINTS INFO END
10:11:47 WindowSelector::datetime constraints: ####################
10:11:47 WindowSelector Info: Decimation level = 0. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 45600 - 76799
10:11:47 WindowSelector Info: Decimation level = 1. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 11400 - 19199
10:11:47 WindowSelector Info: Decimation level = 2. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 2850 - 4799
10:11:47 WindowSelector Info: Decimation level = 3. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 357 - 599
10:11:47 WindowSelector Info: Decimation level = 4. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 90 - 149
10:11:47 WindowSelector Info: Decimation level = 5. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 12 - 18
10:11:47 WindowSelector Info: Decimation level = 6. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 3 - 4
10:11:47 WindowSelector Info: Decimation level = 7. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 1 - 1
10:11:47 WindowSelector::shared windows: ####################
10:11:47 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO BEGIN
10:11:47 WindowSelector::shared windows: ####################
10:11:47 WindowSelector::shared windows: Decimation Level = 0
10:11:47 WindowSelector::shared windows: Number of shared windows = 31199
10:11:47 WindowSelector::shared windows: Shared window indices: 45600-76798:1
10:11:47 WindowSelector::shared windows: Number of unmasked windows: 31199
10:11:47 WindowSelector::shared windows: Decimation Level = 1
10:11:47 WindowSelector::shared windows: Number of shared windows = 7799
10:11:47 WindowSelector::shared windows: Shared window indices: 11400-19198:1
10:11:47 WindowSelector::shared windows: Number of unmasked windows: 7799
10:11:47 WindowSelector::shared windows: Decimation Level = 2
10:11:47 WindowSelector::shared windows: Number of shared windows = 1949
10:11:47 WindowSelector::shared windows: Shared window indices: 2850-4798:1
10:11:47 WindowSelector::shared windows: Number of unmasked windows: 1949
10:11:47 WindowSelector::shared windows: Decimation Level = 3
10:11:47 WindowSelector::shared windows: Number of shared windows = 242
10:11:47 WindowSelector::shared windows: Shared window indices: 357-598:1
10:11:47 WindowSelector::shared windows: Number of unmasked windows: 242
10:11:47 WindowSelector::shared windows: Decimation Level = 4
10:11:47 WindowSelector::shared windows: Number of shared windows = 59
10:11:47 WindowSelector::shared windows: Shared window indices: 90-148:1
10:11:47 WindowSelector::shared windows: Number of unmasked windows: 59
10:11:47 WindowSelector::shared windows: Decimation Level = 5
10:11:47 WindowSelector::shared windows: Number of shared windows = 6
10:11:47 WindowSelector::shared windows: Shared window indices: 12-17:1
10:11:47 WindowSelector::shared windows: Number of unmasked windows: 6
10:11:47 WindowSelector::shared windows: Decimation Level = 6
10:11:47 WindowSelector::shared windows: Number of shared windows = 1
10:11:47 WindowSelector::shared windows: Shared window indices: 3
10:11:47 WindowSelector::shared windows: Number of unmasked windows: 1
10:11:47 WindowSelector::shared windows: Decimation Level = 7
10:11:47 WindowSelector::shared windows: Number of shared windows = 0
10:11:47 WindowSelector::shared windows: Shared window indices:
10:11:47 WindowSelector::shared windows: Number of unmasked windows: 0
10:11:47 WindowSelector::shared windows: NOTE: These are the shared windows at each decimation level. Windows for each evaluation frequency might vary depending on masks
10:11:47 WindowSelector::shared windows: ####################
10:11:47 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO END
10:11:47 WindowSelector::shared windows: ####################
|
The number of shared windows has reduced even more. Instead of using a full day (24 hours) of data as in the previous example with the date constraint, now only the evening and night has been specified, totalling 13 hours.
Masks can also be specified in the WindowSelector
using the addWindowMask()
method.
47 48 49 50 51 | # can add a mask
selector.addWindowMask("site1", "coh70_100")
selector.printWindowMasks()
selector.calcSharedWindows()
selector.printSharedWindows()
|
And printing the information to the terminal gives:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | 10:12:14 WindowSelector::window masks: ####################
10:12:14 WindowSelector::window masks: WINDOWSELECTOR::WINDOW MASKS INFO BEGIN
10:12:14 WindowSelector::window masks: ####################
10:12:14 WindowSelector::window masks: Site = site1
10:12:14 WindowSelector::window masks: Mask = coh70_100
10:12:14 WindowSelector::window masks: ####################
10:12:14 WindowSelector::window masks: WINDOWSELECTOR::WINDOW MASKS INFO END
10:12:14 WindowSelector::window masks: ####################
10:12:14 WindowSelector Info: Decimation level = 0. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 45600 - 76799
10:12:14 WindowSelector Info: Decimation level = 1. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 11400 - 19199
10:12:14 WindowSelector Info: Decimation level = 2. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 2850 - 4799
10:12:14 WindowSelector Info: Decimation level = 3. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 357 - 599
10:12:14 WindowSelector Info: Decimation level = 4. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 90 - 149
10:12:14 WindowSelector Info: Decimation level = 5. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 12 - 18
10:12:14 WindowSelector Info: Decimation level = 6. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 3 - 4
10:12:14 WindowSelector Info: Decimation level = 7. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 1 - 1
10:12:14 WindowSelector::shared windows: ####################
10:12:14 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO BEGIN
10:12:14 WindowSelector::shared windows: ####################
10:12:14 WindowSelector::shared windows: Decimation Level = 0
10:12:14 WindowSelector::shared windows: Number of shared windows = 31199
10:12:14 WindowSelector::shared windows: Shared window indices: 45600-76798:1
10:12:14 WindowSelector::shared windows: Number of unmasked windows: 26341
10:12:14 WindowSelector::shared windows: Decimation Level = 1
10:12:14 WindowSelector::shared windows: Number of shared windows = 7799
10:12:14 WindowSelector::shared windows: Shared window indices: 11400-19198:1
10:12:14 WindowSelector::shared windows: Number of unmasked windows: 3673
10:12:14 WindowSelector::shared windows: Decimation Level = 2
10:12:14 WindowSelector::shared windows: Number of shared windows = 1949
10:12:14 WindowSelector::shared windows: Shared window indices: 2850-4798:1
10:12:14 WindowSelector::shared windows: Number of unmasked windows: 403
10:12:14 WindowSelector::shared windows: Decimation Level = 3
10:12:14 WindowSelector::shared windows: Number of shared windows = 242
10:12:14 WindowSelector::shared windows: Shared window indices: 357-598:1
10:12:14 WindowSelector::shared windows: Number of unmasked windows: 138
10:12:14 WindowSelector::shared windows: Decimation Level = 4
10:12:14 WindowSelector::shared windows: Number of shared windows = 59
10:12:14 WindowSelector::shared windows: Shared window indices: 90-148:1
10:12:14 WindowSelector::shared windows: Number of unmasked windows: 53
10:12:14 WindowSelector::shared windows: Decimation Level = 5
10:12:14 WindowSelector::shared windows: Number of shared windows = 6
10:12:14 WindowSelector::shared windows: Shared window indices: 12-17:1
10:12:14 WindowSelector::shared windows: Number of unmasked windows: 3
10:12:14 WindowSelector::shared windows: Decimation Level = 6
10:12:14 WindowSelector::shared windows: Number of shared windows = 1
10:12:14 WindowSelector::shared windows: Shared window indices: 3
10:12:14 WindowSelector::shared windows: Number of unmasked windows: 0
10:12:14 WindowSelector::shared windows: Decimation Level = 7
10:12:14 WindowSelector::shared windows: Number of shared windows = 0
10:12:14 WindowSelector::shared windows: Shared window indices:
10:12:14 WindowSelector::shared windows: Number of unmasked windows: 0
10:12:14 WindowSelector::shared windows: NOTE: These are the shared windows at each decimation level. Windows for each evaluation frequency might vary depending on masks
10:12:14 WindowSelector::shared windows: ####################
10:12:14 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO END
10:12:14 WindowSelector::shared windows: ####################
|
However, there has been no change in the number of shared windows, despite the application of the masks. This is because:
Important
Recall that statistics are calculated for each evaluation frequency. Therefore masks are applied on an evaluation frequency basis rather than decimation level.
The loop below can demonstrate the effect of masks on the number of windows for each evaluation frequency.
53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | # the number of windows for each evaluation frequency will be different when masks are applied
# loop over the evaluation frequencies and print the number of windows that will be processed
# save the information to a file
with open("usingWindowSelector.txt", "w") as f:
numDecimationLevels = decimationParameters.numLevels
numFrequenciesPerLevel = decimationParameters.freqPerLevel
for declevel in range(numDecimationLevels):
evaluationFrequencies = decimationParameters.getEvalFrequenciesForLevel(
declevel
)
for eIdx in range(numFrequenciesPerLevel):
f.write(
"Decimation level = {}, evaluation frequency = {:.3f}, number of windows = {:d}\n".format(
declevel,
evaluationFrequencies[eIdx],
len(selector.getWindowsForFreq(declevel, eIdx)),
)
)
|
The output from this loop is shown below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | Decimation level = 0, evaluation frequency = 32.000, number of windows = 20563
Decimation level = 0, evaluation frequency = 22.627, number of windows = 17550
Decimation level = 0, evaluation frequency = 16.000, number of windows = 17907
Decimation level = 0, evaluation frequency = 11.314, number of windows = 19102
Decimation level = 0, evaluation frequency = 8.000, number of windows = 21295
Decimation level = 1, evaluation frequency = 5.657, number of windows = 2944
Decimation level = 1, evaluation frequency = 4.000, number of windows = 2445
Decimation level = 1, evaluation frequency = 2.828, number of windows = 2267
Decimation level = 1, evaluation frequency = 2.000, number of windows = 2303
Decimation level = 1, evaluation frequency = 1.414, number of windows = 2164
Decimation level = 2, evaluation frequency = 1.000, number of windows = 216
Decimation level = 2, evaluation frequency = 0.707, number of windows = 222
Decimation level = 2, evaluation frequency = 0.500, number of windows = 183
Decimation level = 2, evaluation frequency = 0.354, number of windows = 125
Decimation level = 2, evaluation frequency = 0.250, number of windows = 111
Decimation level = 3, evaluation frequency = 0.177, number of windows = 1
Decimation level = 3, evaluation frequency = 0.125, number of windows = 5
Decimation level = 3, evaluation frequency = 0.088, number of windows = 9
Decimation level = 3, evaluation frequency = 0.062, number of windows = 69
Decimation level = 3, evaluation frequency = 0.044, number of windows = 131
Decimation level = 4, evaluation frequency = 0.031, number of windows = 25
Decimation level = 4, evaluation frequency = 0.022, number of windows = 43
Decimation level = 4, evaluation frequency = 0.016, number of windows = 48
Decimation level = 4, evaluation frequency = 0.011, number of windows = 44
Decimation level = 4, evaluation frequency = 0.008, number of windows = 43
Decimation level = 5, evaluation frequency = 0.006, number of windows = 1
Decimation level = 5, evaluation frequency = 0.004, number of windows = 3
Decimation level = 5, evaluation frequency = 0.003, number of windows = 2
Decimation level = 5, evaluation frequency = 0.002, number of windows = 2
Decimation level = 5, evaluation frequency = 0.001, number of windows = 2
Decimation level = 6, evaluation frequency = 0.001, number of windows = 0
Decimation level = 6, evaluation frequency = 0.001, number of windows = 0
Decimation level = 6, evaluation frequency = 0.000, number of windows = 0
Decimation level = 6, evaluation frequency = 0.000, number of windows = 0
Decimation level = 6, evaluation frequency = 0.000, number of windows = 0
Decimation level = 7, evaluation frequency = 0.000, number of windows = 0
Decimation level = 7, evaluation frequency = 0.000, number of windows = 0
Decimation level = 7, evaluation frequency = 0.000, number of windows = 0
Decimation level = 7, evaluation frequency = 0.000, number of windows = 0
Decimation level = 7, evaluation frequency = 0.000, number of windows = 0
|
Note
To see how to use date and time constraints in the processing of magnetotelluric data, see Date and time constraints.
Complete example script¶
For the purposes of clarity, the complete example script is provided below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | from datapaths import projectPath, imagePath
from resistics.project.io import loadProject
# load project and configuration file
projData = loadProject(projectPath, configFile="tutorialConfig.ini")
# get decimation parameters
from resistics.project.shortcuts import getDecimationParameters
decimationParameters = getDecimationParameters(4096, projData.config)
decimationParameters.printInfo()
# get the window parameters
from resistics.project.shortcuts import getWindowParameters
windowParameters = getWindowParameters(decimationParameters, projData.config)
windowParameters.printInfo()
from resistics.project.shortcuts import getWindowSelector
decimationParameters = getDecimationParameters(128, projData.config)
windowParameters = getWindowParameters(decimationParameters, projData.config)
selector = getWindowSelector(projData, decimationParameters, windowParameters)
selector.printInfo()
# add a site and print the information to the terminal
selector.setSites(["site1"])
selector.printInfo()
# calculate shared windows
selector.calcSharedWindows()
selector.printSharedWindows()
# currently using every window, but how might this change with a date or time constraint
selector.addDateConstraint("2012-02-11")
selector.printDatetimeConstraints()
selector.calcSharedWindows()
selector.printSharedWindows()
# reset datetime constraints
selector.resetDatetimeConstraints()
selector.addDatetimeConstraint("2012-02-10 19:00:00", "2012-02-11 08:00:00")
selector.printDatetimeConstraints()
selector.calcSharedWindows()
selector.printSharedWindows()
# can add a mask
selector.addWindowMask("site1", "coh70_100")
selector.printWindowMasks()
selector.calcSharedWindows()
selector.printSharedWindows()
# the number of windows for each evaluation frequency will be different when masks are applied
# loop over the evaluation frequencies and print the number of windows that will be processed
# save the information to a file
with open("usingWindowSelector.txt", "w") as f:
numDecimationLevels = decimationParameters.numLevels
numFrequenciesPerLevel = decimationParameters.freqPerLevel
for declevel in range(numDecimationLevels):
evaluationFrequencies = decimationParameters.getEvalFrequenciesForLevel(
declevel
)
for eIdx in range(numFrequenciesPerLevel):
f.write(
"Decimation level = {}, evaluation frequency = {:.3f}, number of windows = {:d}\n".format(
declevel,
evaluationFrequencies[eIdx],
len(selector.getWindowsForFreq(declevel, eIdx)),
)
)
|