Fourier Filtering and Reconstruction

The Fourier Filtering and Reconstruction option in the Process menu or the Process toolbar is an extensive Fourier domain filtering and component isolation procedure. In this option, it is possible to set lower and upper frequency thresholds as well as lower and upper spectrum thresholds, either numerically or graphically. Individual Fourier channels can also be toggled on and off graphically. This option also offers reconstruction of windowed FFTs for isolating very low power components.

This procedure uses an inverse FFT for the reconstruction. The filtered or reconstructed signal will thus contain the same number of data elements as the original or unfiltered signal. To generate any size reconstruction or to reconstruct a specific range of data, a similar (but non-windowed) filtering and reconstruction can be found in the Fourier Interpolation option. There the reconstruction is made directly from the amplitude, frequency, and phase of the continuous sine components rather than by an inverse FFT.

This option presents a dual AutoSignal graph with the frequency domain decomposition in the upper graph and the input (unfiltered) and output (filtered) time domain data in the lower graph.

The graph's toolbar has a button that can toggle on and off and change the proportions of the two graphs in the dialog. The default has the Fourier graph using the upper half of the region, and the output graph using the lower half.

Window

The use of a data window in a Fourier domain filter procedure is not a normal practice. There are two instances, however, where a data tapering window is beneficial. The most common use of data windows is to minimize spectral leakage so that very low power components can be isolated and reconstructed. When no data tapering window is used, low energy components are lost in the background. The use of a data tapering window enhances the dynamic range of the FFT so that even components of very low power can be cleanly distinguished above a noise background.

The other reason is to produce softer filter transitions. When some frequencies are zeroed within an exact n non-windowed FFT, and the reconstructed data are again processed by an exact n non-windowed FFT, there will be true brickwall transitions at the zeroed frequencies. While this is generally desirable, and even the ideal, the transition observed in the subsequent exact n non-windowed FFT can be softened by using a windowed FFT as the basis for the initial processing (of course, the same brickwall occurs if the data window is used for both FFTs).

AutoSignal offers a broad selection of data tapering windows to minimize spectral leakage. The adj field is used to set the spectral width, and thus the dynamic range, of adjustable windows. This field will be disabled for fixed windows.

An appropriate window and adjustable width can be determined by using the Explore Data Tapering Windows option.

In AutoSignal, a windowed FFT is normalized so that its power equals the power of the input data, regardless of the data window used.

When a window is used in this option, an edge% value must be specified. This is a tolerance below which no reconstruction occurs (data tapered to zero at the bounds cannot be reconstructed). The reconstructed data will be shortened in length by the removal of points near the bounds that occurs as a consequence of this edge tolerance. Two windows of particular merit in this option are the Hamming and Blackman windows since they do not fully decay to zero at the bounds of the taper and can usually reconstruct the full data range at the default 1% edge tolerance.

Transform

AutoSignal offers the following FFT algorithms:

· FFT Radix 2

· Prime Factor

· Mixed Radix

· Chirp-Z

· Best Exact N

The Best Exact N composite algorithm is the default. If the data size is a power of 2, the FFT Radix 2 algorithm is used. If not and the size is included in the prime-factor set, then the Prime Factor procedure is used. Otherwise, the Mixed Radix algorithm is used if the largest prime <= 509 and the Chirp-Z is used if the largest prime >= 521. This produces the fastest possible exact n FFT.

Data reconstruction is via an inverse FFT. The same algorithm is used for both the forward and reverse transforms.

Nmin

The initial value will be the data size. To zero pad, enter any value greater than the data size. You may also select from a power of 2 sequence in the drop down box. Note that the actual size of the FFT may be greater than this value if the FFT Radix 2 or Prime Factor algorithm is used.

Increasing the zero padding will increase the number of frequency channels, which may improve the accuracy of isolating components or separating signal and noise for reconstructions.

Plot

The frequency domain information can be plotted in a variety of formats. In the following table, Re is the real component of the FFT at a given frequency, Im is the imaginary component, n is the data set size, dx is the sampling interval, and var is the variance of the data series.

· Magnitude, sqrt(Re*Re+Im*Im)

· Amplitude, 2.0*sqrt(Re*Re+Im*Im)/n

· Phase, sine-based, Pi/2+atan(Im/Re)

· Variance, Power normalized by variance, (Re*Re+Im*Im)/n/var

· dB, decibels, 10.0*log10(Re*Re+Im*Im)

· dB Norm, decibels, normalized to 0 for frequency channel with maximum power

· PSD SumSq, Power as Sum Squared Amplitude, 2.0*(Re*Re+Im*Im)/n

· PSD MeanSq, Power as Mean Squared Amplitude, 2.0*(Re*Re+Im*Im)/n/n

· PSD TimeInt, Power as Time-Integral Squared Amplitude, 2.0*dx*(Re*Re+Im*Im)/n

In an amplitude plot, you see the actual amplitude of sine components. In a normalized decibel plot, the highest peak is at 0dB, a peak at -3dB would have half the power, and a peak at -6dB would have half the amplitude. The PSD TISA (time-integral squared amplitude power) is the actual integral under the curve defined by the square of the raw data.

If the data set size is 512 or lower (257 or lower pts in the frequency spectrum), a bar format is automatically used. Otherwise, the points in the spectrum will be rendered directly with lines connecting the points. You can override either of these point formats using the Modify Point Format option and their respective states will be saved across sessions.

Frequency Thresholds

There are several ways to set the thresholds for Fourier filtering. The first is to define a range of frequency space. This is primarily used to isolate and reconstruct signal components. The Freq1 and Freq2 fields set limits in frequency. The frequency boundaries of this region can either be included (Incl) or excluded (Excl). If a frequency range is included, this means that only the frequencies that fall within this range are used in the data reconstruction. If the region is excluded, only those frequencies that fall outside the region are used for the reconstruction.

The frequencies can be entered numerically. If both fields for frequency are left blank, the full range is assumed for the Incl option, and no range is specified for the Excl option. If only a lower limit is entered, the variable range is assumed to be from this limit and greater for the Incl option, and all values below this limit if the Excl option is set. If only an upper limit is entered, the variable range is assumed to be from this limit and lower for Incl, and all values above this limit for Excl.

It is generally easier to set a frequency range graphically. When the X Sectioning mode is selected, the mouse is used in the upper graph to select the frequency range of interest. Frequencies are toggled off by holding the left mouse button down and sliding right and toggled back on by moving left. Individual frequencies can be toggled on and off by placing the cursor at the point (or top of a bar) and left clicking the mouse.

Spectral Thresholds

Another way to set the thresholds for Fourier filtering involves defining spectral thresholds. This is primarily used to separate signal components by power. It can also be used for noise removal, replicating the functionality in the Fourier Smoothing and Denoising option, or for signal removal, generating a signal-free background for subsequent analysis.

The Spec1 and Spec2 fields set the spectral limits. These can only be entered numerically. If only Spec1 is set, all values at or above this threshold will be used in the reconstruction if Incl is set, and all below this threshold will be used if Excl is checked. If only Spec2 is set, all values at this threshold and below will be used in the reconstruction if Incl is set, and all above this threshold will be used if Excl is set. For this option, the normalized dB values must use the minus sign.

Frequency-Spectral Thresholds

The thresholds for Fourier filtering can also consist of a frequency-spectrum region. This allows signal components to be isolated by both frequency and spectral power. This is the most effective way to isolate and reconstruct a single signal component when some number of components are present.

The Freq1 and Freq2 fields set limits in frequency while the Spec1 and Spec2 fields set the spectral limits. The values can be entered numerically. If only a lower limit field is set, all values at or above this threshold will be used in the reconstruction if Incl is set, and all below this threshold will be used if Excl is checked. If only the upper limit is set, all values at this threshold and below will be used in the reconstruction if Incl is set, and all above this threshold will be used if Excl is set. Again, the normalized dB values must use the minus sign.

It is generally easier to set a frequency-spectrum region graphically. When the XY Sectioning mode is selected, the mouse is used in the upper graph to enclose the frequency-spectral region of interest. To include the region that is to be graphically defined, click and hold down the left mouse button at one corner of the frequency-spectral rectangle in the upper graph and drag the rectangle so that it encloses the region desired. This will automatically enter both the frequency and spectral limits and set the process state to include the FFT channels located within the region.

To exclude the region that is to be graphically defined, the same graphical procedure is followed except that the right rather than the left mouse button is used. In this case, the process state will be set to exclude the FFT channels within the region. This is useful for reconstructing data with a single component, such as line noise, removed.

Note that it is possible to reconstruct only one frequency-spectral region at a time.

The Clear button is used to clear all of the threshold fields.

The Apply button can be used to immediately update the processing rather than wait for the automatic update.

Popup Information

The Toggle Popup Information Window option is used to report estimated noise and power reductions as well as a correlation coefficient between the filtered and unfiltered data.

AutoSignal offers a robust noise estimation procedure that may be of some value for low-frequency signals. A cubic polynomial interpolation is made for each point using the two points to the left and the two to the right (excluding the current point). The difference between the interpolated and signal values is used to generate a measure of the white noise present in the signal. This assumes that the signal can be locally characterized by a smooth cubic interpolant. Also, the signal component(s) should exist only in the lower quarter of the Nyquist range. If a high frequency signal component is present, these estimates of noise will be invalid. The Noise In value reports the estimated white noise in the incoming data, the Noise Out value reports the estimated white noise for the filtered signal. The Noise % is given as the amount of estimated noise remaining after filtration.

The TISA In value reports the TISA power in the incoming data. The TISA Out value is the TISA power for the filtered signal. The TISA % is given as the amount of power remaining after filtration.

The r-squared correlation coefficient is also reported. An rē of 1 is a perfect correlation while a value of 0 means the filtered and unfiltered signals are completely uncorrelated.

List

The List Data option lists the index, time, and output signal in a three column table. The listing uses the AutoSignal text viewer facility.

Copy

The Copy Data to Clipboard option copies the time and output signal values to the clipboard. Formats include full precision binary (for spreadsheets such as Excel) and ASCII (for pasting into text editors).

Save

The Save Data to Disk option writes the time and output values to a supported file format. These formats include ASCII, Excel 97, Excel 95, Lotus WK3, Lotus WK1, SPSS, or Systat.

Production Facility

The AutoSignal Automation facility allows unattended processing of large numbers of data sets. The data sets can be consolidated in an Excel file or acquired using a DLL. The graphs can be exported to an MS Word RTF file, while the processed data can be exported to an Excel 95 or Excel 97 file.

Local Options

A local option changes the data set for the duration of the current procedure only. The main data table is not altered. AutoSignal offers four local options in most of the spectral procedures.

Section the data to isolate specific regions for processing.

Detrend for removing mean or subtracting a least-squares trend model.

Eigendecomposition Filtration for isolating spectral components by signal strength.

The Reset button restores the data to its state when first entering the procedure. Note that if you implement sequential local procedures, all of the revisions are discarded upon reset. If an Automation Session is in progress, the Reset button can be used to terminate the automated processing.

When exiting this procedure with the OK button, an option will be presented to update AutoSignal's main data table with the filtered data.