# Recap and deeper dive¶

Now that many of the basic features of resistics have been introduced, it is a useful time to recap some theory and consider some lower level resistics API. The DecimationParameters, WindowParameters and WindowSelector classes are central to how resistics processes magnetotelluric data. They are lower level elements of the resistics package but are useful to know and understand, particularly before approaching the Advanced and Cookbook sections.

## Recap¶

Ahead of introducing the window selector, it is useful to do a quick recap of how resistics processes magnetotelluric data. Ultimately, the point of processing magnetotelluric data is to understand the electrical properties of the subsurface.

In the standard magnetotelluric setup, the electrical properties of the subsurface are described by the impedance tensor, which relates the input magnetic fields to the output electric fields. This is usually posed in the frequency domain, as shown below. Please see recommended literature if this is unfamiliar.

\begin{eqnarray} E_{x}(\omega) & = & Z_{xx}(\omega) H_{x}(\omega) + Z_{xy}(\omega) H_{y}(\omega) \\ E_{y}(\omega) & = & Z_{yx}(\omega) H_{x}(\omega) + Z_{yy}(\omega) H_{y}(\omega) \end{eqnarray}

Here, the dependency on frequency ($$\omega$$) is shown explicitly.

In the field, $$E_{x}$$, $$E_{y}$$, $$H_{x}$$, $$H_{y}$$ (and often $$H_{z}$$) are recorded. These are then converted into the frequency domain using the Fourier transform, or in resistics, calculating spectra. To read more about resistics spectra, see Up and running and Viewing spectra.

Resistics then processes the magnetotelluric spectra data to try and estimate the four unknown values of $$Z_{xx}$$, $$Z_{yy}$$, $$Z_{xy}$$, $$Z_{yx}$$. But in the above, there are two equations and four unknowns, so somehow, more equations need to be generated. This is normally done by by using cross-powers as shown in the below set of equations,

\begin{eqnarray} <E_{x}(\omega) E_{x}^{*}(\omega)> & = & Z_{xx}(\omega) <H_{x}(\omega) E_{x}^{*}(\omega)> + Z_{xy}(\omega) <H_{y}(\omega) E_{x}^{*}(\omega)> \\ <E_{x}(\omega) E_{y}^{*}(\omega)> & = & Z_{xx}(\omega) <H_{x}(\omega) E_{y}^{*}(\omega)> + Z_{xy}(\omega) <H_{y}(\omega) E_{y}^{*}(\omega)> \\ <E_{x}(\omega) H_{x}^{*}(\omega)> & = & Z_{xx}(\omega) <H_{x}(\omega) H_{x}^{*}(\omega)> + Z_{xy}(\omega) <H_{y}(\omega) H_{x}^{*}(\omega)> \\ <E_{x}(\omega) H_{y}^{*}(\omega)> & = & Z_{xx}(\omega) <H_{x}(\omega) H_{y}^{*}(\omega)> + Z_{xy}(\omega) <H_{y}(\omega) H_{y}^{*}(\omega)> \\ \end{eqnarray}
\begin{eqnarray} <E_{y}(\omega) E_{x}^{*}(\omega)> & = & Z_{yx}(\omega) <H_{x}(\omega) E_{x}^{*}(\omega)> + Z_{yy}(\omega) <H_{y}(\omega) E_{x}^{*}(\omega)> \\ <E_{y}(\omega) E_{y}^{*}(\omega)> & = & Z_{yx}(\omega) <H_{x}(\omega) E_{y}^{*}(\omega)> + Z_{yy}(\omega) <H_{y}(\omega) E_{y}^{*}(\omega)> \\ <E_{y}(\omega) H_{x}^{*}(\omega)> & = & Z_{yx}(\omega) <H_{x}(\omega) H_{x}^{*}(\omega)> + Z_{yy}(\omega) <H_{y}(\omega) H_{x}^{*}(\omega)> \\ <E_{y}(\omega) H_{y}^{*}(\omega)> & = & Z_{yx}(\omega) <H_{x}(\omega) H_{y}^{*}(\omega)> + Z_{yy}(\omega) <H_{y}(\omega) H_{y}^{*}(\omega)> \\ \end{eqnarray}

where $$*$$ represents the complex conjugate.

This equations are solved at a set of frequencies called the evaluation frequencies. These are normally chosen on a log scale.

So theoretically, a magnetotelluric processing system could do the following:

1. Transform full electric and magnetic field time series measurements to frequency domain

2. Calculate cross-power spectra

3. Select a set of evaluation frequencies

4. For each evaluation frequency, perform linear regression to solve for $$Z_{xx}$$, $$Z_{yy}$$, $$Z_{xy}$$, $$Z_{yx}$$

However, the impedance tensor estimates can be improved by providing more equations or by stacking cross-power spectra, especially in the presence of electromagnetic noise. This is usually done by windowing the time series, in which case the processing flow looks like this:

1. Choose window parameters (window size and overlap size between windows)

2. Transform each window into frequency domain (calculate spectra)

3. Calculate cross-power spectra at a set of evaluation frequencies

4. Stack the cross-power spectra for the different windows

5. For each evaluation frequency, perform linear regression to solve for $$Z_{xx}$$, $$Z_{yy}$$, $$Z_{xy}$$, $$Z_{yx}$$

There are other variants on this and stacking strategies. It is not necessary for stacking to be performed and instead cross-powers from a set of randomly selected windows can be used. Further, linear regression can be replaced by robust regression.

For the purposes of calculating out the longer period evaluation frequencies, the time data is usually decimated before the process is repeated. Explicitly, this means:

• For decimation level in number of decimation levels

1. Decimate time series by the decimation factor

2. Choose window parameters for decimation level

3. Select evaluation frequencies for decimation level

4. Transform each window into frequency domain (calculate spectra)

5. Calculate cross-power spectra at the evaluation frequencies

6. Stack the cross-power spectra for the different windows

7. For each evaluation frequency, perform linear regression to solve for $$Z_{xx}$$, $$Z_{yy}$$, $$Z_{xy}$$, $$Z_{yx}$$

In the following sections, the resistics approach to decimation parameters, window parameters and window selection will be detailed. It is not necessary to know this, but this information will be a useful pre-cursor to exploring the Advanced and Cookbook sections.

## Decimation parameters¶

The DecimationParameters class holds the decimation information. Decimation parameters can be set using configuration files (see Using configuration files). To see what the DecimationParameters class holds, consider an example.

  1 2 3 4 5 6 7 8 9 10 11 from datapaths import projectPath, imagePath from resistics.project.io import loadProject # load project and configuration file projData = loadProject(projectPath, configFile="tutorialConfig.ini") # get decimation parameters from resistics.project.shortcuts import getDecimationParameters decimationParameters = getDecimationParameters(4096, projData.config) decimationParameters.printInfo() 

The project has been loaded along with the configuration file, which has specified 8 decimation levels and 5 evaluation frequencies per level. The decimation paramters can be printed to the terminal using the parent class printInfo() method to give:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 09:50:02 DecimationParams: #################### 09:50:02 DecimationParams: DECIMATIONPARAMS INFO BEGIN 09:50:02 DecimationParams: #################### 09:50:02 DecimationParams: Sampling frequency = 4096.000000 09:50:02 DecimationParams: Number of decimation levels = 8 09:50:02 DecimationParams: Level = 0 sample freq. [Hz] = 4096.000000 sample rate [s] = 0.000244 dec. factor = 0000001 inc. factor = 1 09:50:02 DecimationParams: Level = 1 sample freq. [Hz] = 1024.000000 sample rate [s] = 0.000977 dec. factor = 0000004 inc. factor = 4 09:50:02 DecimationParams: Level = 2 sample freq. [Hz] = 256.000000 sample rate [s] = 0.003906 dec. factor = 0000016 inc. factor = 4 09:50:02 DecimationParams: Level = 3 sample freq. [Hz] = 32.000000 sample rate [s] = 0.031250 dec. factor = 0000128 inc. factor = 8 09:50:02 DecimationParams: Level = 4 sample freq. [Hz] = 8.000000 sample rate [s] = 0.125000 dec. factor = 0000512 inc. factor = 4 09:50:02 DecimationParams: Level = 5 sample freq. [Hz] = 1.000000 sample rate [s] = 1.000000 dec. factor = 0004096 inc. factor = 8 09:50:02 DecimationParams: Level = 6 sample freq. [Hz] = 0.250000 sample rate [s] = 4.000000 dec. factor = 0016384 inc. factor = 4 09:50:02 DecimationParams: Level = 7 sample freq. [Hz] = 0.031250 sample rate [s] = 32.000000 dec. factor = 0131072 inc. factor = 8 09:50:02 DecimationParams: Evaluation frequencies [Hz] 09:50:02 DecimationParams: Level = 0: 1024.00000000, 724.07734394, 512.00000000, 362.03867197, 256.00000000 09:50:02 DecimationParams: Level = 1: 181.01933598, 128.00000000, 90.50966799, 64.00000000, 45.25483400 09:50:02 DecimationParams: Level = 2: 32.00000000, 22.62741700, 16.00000000, 11.31370850, 8.00000000 09:50:02 DecimationParams: Level = 3: 5.65685425, 4.00000000, 2.82842712, 2.00000000, 1.41421356 09:50:02 DecimationParams: Level = 4: 1.00000000, 0.70710678, 0.50000000, 0.35355339, 0.25000000 09:50:02 DecimationParams: Level = 5: 0.17677670, 0.12500000, 0.08838835, 0.06250000, 0.04419417 09:50:02 DecimationParams: Level = 6: 0.03125000, 0.02209709, 0.01562500, 0.01104854, 0.00781250 09:50:02 DecimationParams: Level = 7: 0.00552427, 0.00390625, 0.00276214, 0.00195312, 0.00138107 09:50:02 DecimationParams: #################### 09:50:02 DecimationParams: DECIMATIONPARAMS INFO END 09:50:02 DecimationParams: #################### 

The information provides all the decimation factors and the sampling frequencies at each decimation level. It also provides the 5 evaluation frequencies for each decimation level.

## Window parameters¶

The WindowParameters class contains information about the windowing parameters. Window parameters can be set using configuration files (see Using configuration files).

Again, consider an example which continues from the decimation paramters example:

 13 14 15 16 17 # get the window parameters from resistics.project.shortcuts import getWindowParameters windowParameters = getWindowParameters(decimationParameters, projData.config) windowParameters.printInfo() 

The project has been loaded along with the configuration file, which has specified a minimum window size of 256 samples and a minimum overlap of 64 samples. However, resistics is free to use window and overlap sizes greater than these if it makes sense. Windowing parameter information can be printed to the terminal using the printInfo() method.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 09:50:02 WindowParams: #################### 09:50:02 WindowParams: WINDOWPARAMS INFO BEGIN 09:50:02 WindowParams: #################### 09:50:02 WindowParams: Number of decimation levels = 8 09:50:02 WindowParams: Level = 0, sample freq. [Hz] = 4096.000000, sample rate [s] = 0.000244 09:50:02 WindowParams: Window size = 2048, window duration [s] = 0.499756 09:50:02 WindowParams: Window overlap = 512, overlap duration [s] = 0.124756 09:50:02 WindowParams: Level = 1, sample freq. [Hz] = 1024.000000, sample rate [s] = 0.000977 09:50:02 WindowParams: Window size = 512, window duration [s] = 0.499023 09:50:02 WindowParams: Window overlap = 128, overlap duration [s] = 0.124023 09:50:02 WindowParams: Level = 2, sample freq. [Hz] = 256.000000, sample rate [s] = 0.003906 09:50:02 WindowParams: Window size = 256, window duration [s] = 0.996094 09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 0.246094 09:50:02 WindowParams: Level = 3, sample freq. [Hz] = 32.000000, sample rate [s] = 0.031250 09:50:02 WindowParams: Window size = 256, window duration [s] = 7.968750 09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 1.968750 09:50:02 WindowParams: Level = 4, sample freq. [Hz] = 8.000000, sample rate [s] = 0.125000 09:50:02 WindowParams: Window size = 256, window duration [s] = 31.875000 09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 7.875000 09:50:02 WindowParams: Level = 5, sample freq. [Hz] = 1.000000, sample rate [s] = 1.000000 09:50:02 WindowParams: Window size = 256, window duration [s] = 255.000000 09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 63.000000 09:50:02 WindowParams: Level = 6, sample freq. [Hz] = 0.250000, sample rate [s] = 4.000000 09:50:02 WindowParams: Window size = 256, window duration [s] = 1020.000000 09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 252.000000 09:50:02 WindowParams: Level = 7, sample freq. [Hz] = 0.031250, sample rate [s] = 32.000000 09:50:02 WindowParams: Window size = 256, window duration [s] = 8160.000000 09:50:02 WindowParams: Window overlap = 64, overlap duration [s] = 2016.000000 09:50:02 WindowParams: #################### 09:50:02 WindowParams: WINDOWPARAMS INFO END 09:50:02 WindowParams: #################### 

In this case, for the 4096 Hz data, the for the lower decimation levels, larger window sizes are used but as the decimation increases, the 256 minimum window sample size and 64 minimum overlap sample size is used.

## Window selector¶

The WindowSelector class is a key component of resistics. Before processing time windows (and their respective spectra) to calculate transfer function estimates, the windows to use have to be selected. The WindowSelector class is the how windows are selected.

Below are a few scenarios under which windows might have to be selected:

1. Single site processing, use all the data

2. Single site processing with masks and/or date and time constraints

3. Remote reference processing with two sites with masks and/or date and time constraints

4. Intersite processing with two sites

5. Intersite processing with a remote reference as a third site

Let’s compare 1 and 2 as simple examples. The others will be introduced in the Advanced section.

Begin by getting a WindowSelector instance. In this case, the 128 Hz data is being used.

 24 25 26 27 28 29 from resistics.project.shortcuts import getWindowSelector decimationParameters = getDecimationParameters(128, projData.config) windowParameters = getWindowParameters(decimationParameters, projData.config) selector = getWindowSelector(projData, decimationParameters, windowParameters) selector.printInfo() 

The WindowSelector needs to know the DecimationParameters and WindowParameters. Information can be printed to the terminal using the printInfo() method. This gives:

  1 2 3 4 5 6 7 8 9 10 10:01:53 WindowSelector: #################### 10:01:53 WindowSelector: WINDOWSELECTOR INFO BEGIN 10:01:53 WindowSelector: #################### 10:01:53 WindowSelector: Sampling frequency [Hz] = 128.000000 10:01:53 WindowSelector: Spectra directory = dec8_5 10:01:53 WindowSelector: Sites = 10:01:53 WindowSelector: Site information: 10:01:53 WindowSelector: #################### 10:01:53 WindowSelector: WINDOWSELECTOR INFO END 10:01:53 WindowSelector: #################### 

Currently, no sites have been specified for the WindowSelector instance. Therefore, there are yet no windows to select from. The next step then is to specify a list of sites.

 26 27 28 # add a site and print the information to the terminal selector.setSites(["site1"]) selector.printInfo() 

Printing information gives the following, which details the recordings and the global windows for those recordings. The global windows are the window numbers beginning from the project reference time. Here there is nothing for decimation level 7 as the time series was not long enough to give sufficient windows after decimation.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 10:03:44 WindowSelector: #################### 10:03:44 WindowSelector: WINDOWSELECTOR INFO BEGIN 10:03:44 WindowSelector: #################### 10:03:44 WindowSelector: Sampling frequency [Hz] = 128.000000 10:03:44 WindowSelector: Spectra directory = dec8_5 10:03:44 WindowSelector: Sites = site1 10:03:44 WindowSelector: Site information: 10:03:44 WindowSelector: Sampling frequency [Hz] = 128.000000 10:03:44 WindowSelector: Site = site1 10:03:44 WindowSelector: Site global index information 10:03:44 WindowSelector: Decimation Level = 0 10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:30:00 - 2012-02-11 23:03:43.992188 Global Indices Range = 27600 - 112948 10:03:44 WindowSelector: Decimation Level = 1 10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:30:00 - 2012-02-11 23:03:43.968750 Global Indices Range = 6900 - 28236 10:03:44 WindowSelector: Decimation Level = 2 10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:30:00 - 2012-02-11 23:03:43.875000 Global Indices Range = 1725 - 7058 10:03:44 WindowSelector: Decimation Level = 3 10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:31:12 - 2012-02-11 23:03:27 Global Indices Range = 216 - 881 10:03:44 WindowSelector: Decimation Level = 4 10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:31:12 - 2012-02-11 23:00:12 Global Indices Range = 54 - 219 10:03:44 WindowSelector: Decimation Level = 5 10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 11:56:48 - 2012-02-11 22:38:24 Global Indices Range = 7 - 26 10:03:44 WindowSelector: Decimation Level = 6 10:03:44 WindowSelector: Measurement file = meas_2012-02-10_11-30-00 time range = 2012-02-10 13:39:12 - 2012-02-11 19:12:00 Global Indices Range = 2 - 5 10:03:44 WindowSelector: Decimation Level = 7 10:03:44 WindowSelector: #################### 10:03:44 WindowSelector: WINDOWSELECTOR INFO END 10:03:44 WindowSelector: #################### 

Now the shared windows can be calculated using the calcSharedWindows() method. In this case, only a single site is specified so there are no other sites to share windows with. When more than a single site is specified, the shared windows between the sites will be calculated. This means the times where both sites were recording. This is an essential step for remote reference and intersite processing (see Advanced).

 30 31 32 # calculate shared windows selector.calcSharedWindows() selector.printSharedWindows() 

The shared windows can be printed to the terminal using the printSharedWindows() method.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 10:04:19 WindowSelector::shared windows: #################### 10:04:19 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO BEGIN 10:04:19 WindowSelector::shared windows: #################### 10:04:19 WindowSelector::shared windows: Decimation Level = 0 10:04:19 WindowSelector::shared windows: Number of shared windows = 85349 10:04:19 WindowSelector::shared windows: Shared window indices: 27600-112948:1 10:04:19 WindowSelector::shared windows: Number of unmasked windows: 85349 10:04:19 WindowSelector::shared windows: Decimation Level = 1 10:04:19 WindowSelector::shared windows: Number of shared windows = 21337 10:04:19 WindowSelector::shared windows: Shared window indices: 6900-28236:1 10:04:19 WindowSelector::shared windows: Number of unmasked windows: 21337 10:04:19 WindowSelector::shared windows: Decimation Level = 2 10:04:19 WindowSelector::shared windows: Number of shared windows = 5334 10:04:19 WindowSelector::shared windows: Shared window indices: 1725-7058:1 10:04:19 WindowSelector::shared windows: Number of unmasked windows: 5334 10:04:19 WindowSelector::shared windows: Decimation Level = 3 10:04:19 WindowSelector::shared windows: Number of shared windows = 666 10:04:19 WindowSelector::shared windows: Shared window indices: 216-881:1 10:04:19 WindowSelector::shared windows: Number of unmasked windows: 666 10:04:19 WindowSelector::shared windows: Decimation Level = 4 10:04:19 WindowSelector::shared windows: Number of shared windows = 166 10:04:19 WindowSelector::shared windows: Shared window indices: 54-219:1 10:04:19 WindowSelector::shared windows: Number of unmasked windows: 166 10:04:19 WindowSelector::shared windows: Decimation Level = 5 10:04:19 WindowSelector::shared windows: Number of shared windows = 20 10:04:19 WindowSelector::shared windows: Shared window indices: 7-26:1 10:04:19 WindowSelector::shared windows: Number of unmasked windows: 20 10:04:19 WindowSelector::shared windows: Decimation Level = 6 10:04:19 WindowSelector::shared windows: Number of shared windows = 4 10:04:19 WindowSelector::shared windows: Shared window indices: 2-5:1 10:04:19 WindowSelector::shared windows: Number of unmasked windows: 4 10:04:19 WindowSelector::shared windows: Decimation Level = 7 10:04:19 WindowSelector::shared windows: Number of shared windows = 0 10:04:19 WindowSelector::shared windows: Shared window indices: 10:04:19 WindowSelector::shared windows: Number of unmasked windows: 0 10:04:19 WindowSelector::shared windows: NOTE: These are the shared windows at each decimation level. Windows for each evaluation frequency might vary depending on masks 10:04:19 WindowSelector::shared windows: #################### 10:04:19 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO END 10:04:19 WindowSelector::shared windows: #################### 

The WindowSelector has functionality to limit windows using date and time constraints. Date and time constraints can be provided using the addDateConstraint(), addTimeConstraint() and addDatetimeConstraint() methods.

Important

Providing a date or time constraint means that only windows which are inside these date/time constraints will be selected. Any windows which are outside the date/time constraint will be discarded.

Below is an example of providing a date constraint using the addDateConstraint() method. After running the calcSharedWindows() method, only time windows within this date/time constraint will be selected.

 34 35 36 37 38 # currently using every window, but how might this change with a date or time constraint selector.addDateConstraint("2012-02-11") selector.printDatetimeConstraints() selector.calcSharedWindows() selector.printSharedWindows() 

The datetime constraints can be printed to the terminal using the printDatetimeConstraints() method. The various print methods give the following:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 10:11:24 WindowSelector::datetime constraints: #################### 10:11:24 WindowSelector::datetime constraints: WINDOWSELECTOR::DATETIME CONSTRAINTS INFO BEGIN 10:11:24 WindowSelector::datetime constraints: #################### 10:11:24 WindowSelector::datetime constraints: Datetime constraints 10:11:24 WindowSelector::datetime constraints: Decimation Level = 0 10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59 10:11:24 WindowSelector::datetime constraints: Decimation Level = 1 10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59 10:11:24 WindowSelector::datetime constraints: Decimation Level = 2 10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59 10:11:24 WindowSelector::datetime constraints: Decimation Level = 3 10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59 10:11:24 WindowSelector::datetime constraints: Decimation Level = 4 10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59 10:11:24 WindowSelector::datetime constraints: Decimation Level = 5 10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59 10:11:24 WindowSelector::datetime constraints: Decimation Level = 6 10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59 10:11:24 WindowSelector::datetime constraints: Decimation Level = 7 10:11:24 WindowSelector::datetime constraints: Constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59 10:11:24 WindowSelector::datetime constraints: #################### 10:11:24 WindowSelector::datetime constraints: WINDOWSELECTOR::DATETIME CONSTRAINTS INFO END 10:11:24 WindowSelector::datetime constraints: #################### 10:11:24 WindowSelector Info: Decimation level = 0. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 57600 - 115199 10:11:24 WindowSelector Info: Decimation level = 1. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 14400 - 28799 10:11:24 WindowSelector Info: Decimation level = 2. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 3600 - 7199 10:11:24 WindowSelector Info: Decimation level = 3. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 450 - 899 10:11:24 WindowSelector Info: Decimation level = 4. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 113 - 224 10:11:24 WindowSelector Info: Decimation level = 5. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 15 - 28 10:11:24 WindowSelector Info: Decimation level = 6. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 4 - 7 10:11:24 WindowSelector Info: Decimation level = 7. Applying date constraint 2012-02-11 00:00:00 - 2012-02-11 23:59:59, global index constraint 1 - 1 10:11:24 WindowSelector::shared windows: #################### 10:11:24 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO BEGIN 10:11:24 WindowSelector::shared windows: #################### 10:11:24 WindowSelector::shared windows: Decimation Level = 0 10:11:24 WindowSelector::shared windows: Number of shared windows = 55349 10:11:24 WindowSelector::shared windows: Shared window indices: 57600-112948:1 10:11:24 WindowSelector::shared windows: Number of unmasked windows: 55349 10:11:24 WindowSelector::shared windows: Decimation Level = 1 10:11:24 WindowSelector::shared windows: Number of shared windows = 13837 10:11:24 WindowSelector::shared windows: Shared window indices: 14400-28236:1 10:11:24 WindowSelector::shared windows: Number of unmasked windows: 13837 10:11:24 WindowSelector::shared windows: Decimation Level = 2 10:11:24 WindowSelector::shared windows: Number of shared windows = 3459 10:11:24 WindowSelector::shared windows: Shared window indices: 3600-7058:1 10:11:24 WindowSelector::shared windows: Number of unmasked windows: 3459 10:11:24 WindowSelector::shared windows: Decimation Level = 3 10:11:24 WindowSelector::shared windows: Number of shared windows = 432 10:11:24 WindowSelector::shared windows: Shared window indices: 450-881:1 10:11:24 WindowSelector::shared windows: Number of unmasked windows: 432 10:11:24 WindowSelector::shared windows: Decimation Level = 4 10:11:24 WindowSelector::shared windows: Number of shared windows = 107 10:11:24 WindowSelector::shared windows: Shared window indices: 113-219:1 10:11:24 WindowSelector::shared windows: Number of unmasked windows: 107 10:11:24 WindowSelector::shared windows: Decimation Level = 5 10:11:24 WindowSelector::shared windows: Number of shared windows = 12 10:11:24 WindowSelector::shared windows: Shared window indices: 15-26:1 10:11:24 WindowSelector::shared windows: Number of unmasked windows: 12 10:11:24 WindowSelector::shared windows: Decimation Level = 6 10:11:24 WindowSelector::shared windows: Number of shared windows = 2 10:11:24 WindowSelector::shared windows: Shared window indices: 4,5 10:11:24 WindowSelector::shared windows: Number of unmasked windows: 2 10:11:24 WindowSelector::shared windows: Decimation Level = 7 10:11:24 WindowSelector::shared windows: Number of shared windows = 0 10:11:24 WindowSelector::shared windows: Shared window indices: 10:11:24 WindowSelector::shared windows: Number of unmasked windows: 0 10:11:24 WindowSelector::shared windows: NOTE: These are the shared windows at each decimation level. Windows for each evaluation frequency might vary depending on masks 10:11:24 WindowSelector::shared windows: #################### 10:11:24 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO END 10:11:24 WindowSelector::shared windows: #################### 

Comparing the number of shared windows to the previous results, it is clear that the number of shared windows has now reduced. This is due to the application of the date constraint. To reset the date constraint, the resetDatetimeConstraints() method can be used as in the following example, where the current date/time constraint is cleared and a new one added.

 40 41 42 43 44 45 # reset datetime constraints selector.resetDatetimeConstraints() selector.addDatetimeConstraint("2012-02-10 19:00:00", "2012-02-11 08:00:00") selector.printDatetimeConstraints() selector.calcSharedWindows() selector.printSharedWindows() 

Calculating the shared windows again with this new datetime constraint and printing the information to the terminal gives:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 10:11:47 WindowSelector::datetime constraints: #################### 10:11:47 WindowSelector::datetime constraints: WINDOWSELECTOR::DATETIME CONSTRAINTS INFO BEGIN 10:11:47 WindowSelector::datetime constraints: #################### 10:11:47 WindowSelector::datetime constraints: Datetime constraints 10:11:47 WindowSelector::datetime constraints: Decimation Level = 0 10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00 10:11:47 WindowSelector::datetime constraints: Decimation Level = 1 10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00 10:11:47 WindowSelector::datetime constraints: Decimation Level = 2 10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00 10:11:47 WindowSelector::datetime constraints: Decimation Level = 3 10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00 10:11:47 WindowSelector::datetime constraints: Decimation Level = 4 10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00 10:11:47 WindowSelector::datetime constraints: Decimation Level = 5 10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00 10:11:47 WindowSelector::datetime constraints: Decimation Level = 6 10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00 10:11:47 WindowSelector::datetime constraints: Decimation Level = 7 10:11:47 WindowSelector::datetime constraints: Constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00 10:11:47 WindowSelector::datetime constraints: #################### 10:11:47 WindowSelector::datetime constraints: WINDOWSELECTOR::DATETIME CONSTRAINTS INFO END 10:11:47 WindowSelector::datetime constraints: #################### 10:11:47 WindowSelector Info: Decimation level = 0. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 45600 - 76799 10:11:47 WindowSelector Info: Decimation level = 1. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 11400 - 19199 10:11:47 WindowSelector Info: Decimation level = 2. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 2850 - 4799 10:11:47 WindowSelector Info: Decimation level = 3. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 357 - 599 10:11:47 WindowSelector Info: Decimation level = 4. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 90 - 149 10:11:47 WindowSelector Info: Decimation level = 5. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 12 - 18 10:11:47 WindowSelector Info: Decimation level = 6. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 3 - 4 10:11:47 WindowSelector Info: Decimation level = 7. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 1 - 1 10:11:47 WindowSelector::shared windows: #################### 10:11:47 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO BEGIN 10:11:47 WindowSelector::shared windows: #################### 10:11:47 WindowSelector::shared windows: Decimation Level = 0 10:11:47 WindowSelector::shared windows: Number of shared windows = 31199 10:11:47 WindowSelector::shared windows: Shared window indices: 45600-76798:1 10:11:47 WindowSelector::shared windows: Number of unmasked windows: 31199 10:11:47 WindowSelector::shared windows: Decimation Level = 1 10:11:47 WindowSelector::shared windows: Number of shared windows = 7799 10:11:47 WindowSelector::shared windows: Shared window indices: 11400-19198:1 10:11:47 WindowSelector::shared windows: Number of unmasked windows: 7799 10:11:47 WindowSelector::shared windows: Decimation Level = 2 10:11:47 WindowSelector::shared windows: Number of shared windows = 1949 10:11:47 WindowSelector::shared windows: Shared window indices: 2850-4798:1 10:11:47 WindowSelector::shared windows: Number of unmasked windows: 1949 10:11:47 WindowSelector::shared windows: Decimation Level = 3 10:11:47 WindowSelector::shared windows: Number of shared windows = 242 10:11:47 WindowSelector::shared windows: Shared window indices: 357-598:1 10:11:47 WindowSelector::shared windows: Number of unmasked windows: 242 10:11:47 WindowSelector::shared windows: Decimation Level = 4 10:11:47 WindowSelector::shared windows: Number of shared windows = 59 10:11:47 WindowSelector::shared windows: Shared window indices: 90-148:1 10:11:47 WindowSelector::shared windows: Number of unmasked windows: 59 10:11:47 WindowSelector::shared windows: Decimation Level = 5 10:11:47 WindowSelector::shared windows: Number of shared windows = 6 10:11:47 WindowSelector::shared windows: Shared window indices: 12-17:1 10:11:47 WindowSelector::shared windows: Number of unmasked windows: 6 10:11:47 WindowSelector::shared windows: Decimation Level = 6 10:11:47 WindowSelector::shared windows: Number of shared windows = 1 10:11:47 WindowSelector::shared windows: Shared window indices: 3 10:11:47 WindowSelector::shared windows: Number of unmasked windows: 1 10:11:47 WindowSelector::shared windows: Decimation Level = 7 10:11:47 WindowSelector::shared windows: Number of shared windows = 0 10:11:47 WindowSelector::shared windows: Shared window indices: 10:11:47 WindowSelector::shared windows: Number of unmasked windows: 0 10:11:47 WindowSelector::shared windows: NOTE: These are the shared windows at each decimation level. Windows for each evaluation frequency might vary depending on masks 10:11:47 WindowSelector::shared windows: #################### 10:11:47 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO END 10:11:47 WindowSelector::shared windows: #################### 

The number of shared windows has reduced even more. Instead of using a full day (24 hours) of data as in the previous example with the date constraint, now only the evening and night has been specified, totalling 13 hours.

Masks can also be specified in the WindowSelector using the addWindowMask() method.

 47 48 49 50 51 # can add a mask selector.addWindowMask("site1", "coh70_100") selector.printWindowMasks() selector.calcSharedWindows() selector.printSharedWindows() 

And printing the information to the terminal gives:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 10:12:14 WindowSelector::window masks: #################### 10:12:14 WindowSelector::window masks: WINDOWSELECTOR::WINDOW MASKS INFO BEGIN 10:12:14 WindowSelector::window masks: #################### 10:12:14 WindowSelector::window masks: Site = site1 10:12:14 WindowSelector::window masks: Mask = coh70_100 10:12:14 WindowSelector::window masks: #################### 10:12:14 WindowSelector::window masks: WINDOWSELECTOR::WINDOW MASKS INFO END 10:12:14 WindowSelector::window masks: #################### 10:12:14 WindowSelector Info: Decimation level = 0. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 45600 - 76799 10:12:14 WindowSelector Info: Decimation level = 1. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 11400 - 19199 10:12:14 WindowSelector Info: Decimation level = 2. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 2850 - 4799 10:12:14 WindowSelector Info: Decimation level = 3. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 357 - 599 10:12:14 WindowSelector Info: Decimation level = 4. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 90 - 149 10:12:14 WindowSelector Info: Decimation level = 5. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 12 - 18 10:12:14 WindowSelector Info: Decimation level = 6. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 3 - 4 10:12:14 WindowSelector Info: Decimation level = 7. Applying date constraint 2012-02-10 19:00:00 - 2012-02-11 08:00:00, global index constraint 1 - 1 10:12:14 WindowSelector::shared windows: #################### 10:12:14 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO BEGIN 10:12:14 WindowSelector::shared windows: #################### 10:12:14 WindowSelector::shared windows: Decimation Level = 0 10:12:14 WindowSelector::shared windows: Number of shared windows = 31199 10:12:14 WindowSelector::shared windows: Shared window indices: 45600-76798:1 10:12:14 WindowSelector::shared windows: Number of unmasked windows: 26341 10:12:14 WindowSelector::shared windows: Decimation Level = 1 10:12:14 WindowSelector::shared windows: Number of shared windows = 7799 10:12:14 WindowSelector::shared windows: Shared window indices: 11400-19198:1 10:12:14 WindowSelector::shared windows: Number of unmasked windows: 3673 10:12:14 WindowSelector::shared windows: Decimation Level = 2 10:12:14 WindowSelector::shared windows: Number of shared windows = 1949 10:12:14 WindowSelector::shared windows: Shared window indices: 2850-4798:1 10:12:14 WindowSelector::shared windows: Number of unmasked windows: 403 10:12:14 WindowSelector::shared windows: Decimation Level = 3 10:12:14 WindowSelector::shared windows: Number of shared windows = 242 10:12:14 WindowSelector::shared windows: Shared window indices: 357-598:1 10:12:14 WindowSelector::shared windows: Number of unmasked windows: 138 10:12:14 WindowSelector::shared windows: Decimation Level = 4 10:12:14 WindowSelector::shared windows: Number of shared windows = 59 10:12:14 WindowSelector::shared windows: Shared window indices: 90-148:1 10:12:14 WindowSelector::shared windows: Number of unmasked windows: 53 10:12:14 WindowSelector::shared windows: Decimation Level = 5 10:12:14 WindowSelector::shared windows: Number of shared windows = 6 10:12:14 WindowSelector::shared windows: Shared window indices: 12-17:1 10:12:14 WindowSelector::shared windows: Number of unmasked windows: 3 10:12:14 WindowSelector::shared windows: Decimation Level = 6 10:12:14 WindowSelector::shared windows: Number of shared windows = 1 10:12:14 WindowSelector::shared windows: Shared window indices: 3 10:12:14 WindowSelector::shared windows: Number of unmasked windows: 0 10:12:14 WindowSelector::shared windows: Decimation Level = 7 10:12:14 WindowSelector::shared windows: Number of shared windows = 0 10:12:14 WindowSelector::shared windows: Shared window indices: 10:12:14 WindowSelector::shared windows: Number of unmasked windows: 0 10:12:14 WindowSelector::shared windows: NOTE: These are the shared windows at each decimation level. Windows for each evaluation frequency might vary depending on masks 10:12:14 WindowSelector::shared windows: #################### 10:12:14 WindowSelector::shared windows: WINDOWSELECTOR::SHARED WINDOWS INFO END 10:12:14 WindowSelector::shared windows: #################### 

However, there has been no change in the number of shared windows, despite the application of the masks. This is because:

Important

Recall that statistics are calculated for each evaluation frequency. Therefore masks are applied on an evaluation frequency basis rather than decimation level.

The loop below can demonstrate the effect of masks on the number of windows for each evaluation frequency.

 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 # the number of windows for each evaluation frequency will be different when masks are applied # loop over the evaluation frequencies and print the number of windows that will be processed # save the information to a file with open("usingWindowSelector.txt", "w") as f: numDecimationLevels = decimationParameters.numLevels numFrequenciesPerLevel = decimationParameters.freqPerLevel for declevel in range(numDecimationLevels): evaluationFrequencies = decimationParameters.getEvalFrequenciesForLevel( declevel ) for eIdx in range(numFrequenciesPerLevel): f.write( "Decimation level = {}, evaluation frequency = {:.3f}, number of windows = {:d}\n".format( declevel, evaluationFrequencies[eIdx], len(selector.getWindowsForFreq(declevel, eIdx)), ) ) 

The output from this loop is shown below.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 Decimation level = 0, evaluation frequency = 32.000, number of windows = 20563 Decimation level = 0, evaluation frequency = 22.627, number of windows = 17550 Decimation level = 0, evaluation frequency = 16.000, number of windows = 17907 Decimation level = 0, evaluation frequency = 11.314, number of windows = 19102 Decimation level = 0, evaluation frequency = 8.000, number of windows = 21295 Decimation level = 1, evaluation frequency = 5.657, number of windows = 2944 Decimation level = 1, evaluation frequency = 4.000, number of windows = 2445 Decimation level = 1, evaluation frequency = 2.828, number of windows = 2267 Decimation level = 1, evaluation frequency = 2.000, number of windows = 2303 Decimation level = 1, evaluation frequency = 1.414, number of windows = 2164 Decimation level = 2, evaluation frequency = 1.000, number of windows = 216 Decimation level = 2, evaluation frequency = 0.707, number of windows = 222 Decimation level = 2, evaluation frequency = 0.500, number of windows = 183 Decimation level = 2, evaluation frequency = 0.354, number of windows = 125 Decimation level = 2, evaluation frequency = 0.250, number of windows = 111 Decimation level = 3, evaluation frequency = 0.177, number of windows = 1 Decimation level = 3, evaluation frequency = 0.125, number of windows = 5 Decimation level = 3, evaluation frequency = 0.088, number of windows = 9 Decimation level = 3, evaluation frequency = 0.062, number of windows = 69 Decimation level = 3, evaluation frequency = 0.044, number of windows = 131 Decimation level = 4, evaluation frequency = 0.031, number of windows = 25 Decimation level = 4, evaluation frequency = 0.022, number of windows = 43 Decimation level = 4, evaluation frequency = 0.016, number of windows = 48 Decimation level = 4, evaluation frequency = 0.011, number of windows = 44 Decimation level = 4, evaluation frequency = 0.008, number of windows = 43 Decimation level = 5, evaluation frequency = 0.006, number of windows = 1 Decimation level = 5, evaluation frequency = 0.004, number of windows = 3 Decimation level = 5, evaluation frequency = 0.003, number of windows = 2 Decimation level = 5, evaluation frequency = 0.002, number of windows = 2 Decimation level = 5, evaluation frequency = 0.001, number of windows = 2 Decimation level = 6, evaluation frequency = 0.001, number of windows = 0 Decimation level = 6, evaluation frequency = 0.001, number of windows = 0 Decimation level = 6, evaluation frequency = 0.000, number of windows = 0 Decimation level = 6, evaluation frequency = 0.000, number of windows = 0 Decimation level = 6, evaluation frequency = 0.000, number of windows = 0 Decimation level = 7, evaluation frequency = 0.000, number of windows = 0 Decimation level = 7, evaluation frequency = 0.000, number of windows = 0 Decimation level = 7, evaluation frequency = 0.000, number of windows = 0 Decimation level = 7, evaluation frequency = 0.000, number of windows = 0 Decimation level = 7, evaluation frequency = 0.000, number of windows = 0 

Note

To see how to use date and time constraints in the processing of magnetotelluric data, see Date and time constraints.

## Complete example script¶

For the purposes of clarity, the complete example script is provided below.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 from datapaths import projectPath, imagePath from resistics.project.io import loadProject # load project and configuration file projData = loadProject(projectPath, configFile="tutorialConfig.ini") # get decimation parameters from resistics.project.shortcuts import getDecimationParameters decimationParameters = getDecimationParameters(4096, projData.config) decimationParameters.printInfo() # get the window parameters from resistics.project.shortcuts import getWindowParameters windowParameters = getWindowParameters(decimationParameters, projData.config) windowParameters.printInfo() from resistics.project.shortcuts import getWindowSelector decimationParameters = getDecimationParameters(128, projData.config) windowParameters = getWindowParameters(decimationParameters, projData.config) selector = getWindowSelector(projData, decimationParameters, windowParameters) selector.printInfo() # add a site and print the information to the terminal selector.setSites(["site1"]) selector.printInfo() # calculate shared windows selector.calcSharedWindows() selector.printSharedWindows() # currently using every window, but how might this change with a date or time constraint selector.addDateConstraint("2012-02-11") selector.printDatetimeConstraints() selector.calcSharedWindows() selector.printSharedWindows() # reset datetime constraints selector.resetDatetimeConstraints() selector.addDatetimeConstraint("2012-02-10 19:00:00", "2012-02-11 08:00:00") selector.printDatetimeConstraints() selector.calcSharedWindows() selector.printSharedWindows() # can add a mask selector.addWindowMask("site1", "coh70_100") selector.printWindowMasks() selector.calcSharedWindows() selector.printSharedWindows() # the number of windows for each evaluation frequency will be different when masks are applied # loop over the evaluation frequencies and print the number of windows that will be processed # save the information to a file with open("usingWindowSelector.txt", "w") as f: numDecimationLevels = decimationParameters.numLevels numFrequenciesPerLevel = decimationParameters.freqPerLevel for declevel in range(numDecimationLevels): evaluationFrequencies = decimationParameters.getEvalFrequenciesForLevel( declevel ) for eIdx in range(numFrequenciesPerLevel): f.write( "Decimation level = {}, evaluation frequency = {:.3f}, number of windows = {:d}\n".format( declevel, evaluationFrequencies[eIdx], len(selector.getWindowsForFreq(declevel, eIdx)), ) )