Fourier Smoothing and Denoising

The Fourier Smoothing and Denoising option in the Process menu or the Process toolbar is a specialized Fourier filtration procedure that sets either a frequency threshold for low pass frequency-domain filtration, or a signal threshold for zeroing all spectral elements below a given dB (normalized) value. This option is used exclusively to remove noise via a Fourier decomposition.

The option presents a dual AutoSignal graph with the frequency domain decomposition in the upper graph and the input and output time domain data in the lower graph.

The graph's toolbar has a button that can toggle on and off and change the proportions of the two graphs in the dialog. The default has the Fourier graph using the upper half of the region, and the output graph using the lower half.

Algorithm

This procedure uses the Best Exact n FFT procedure to create the frequency spectrum in the upper graph. The Best Exact n FFT procedure is also used for the FFT inverse that reconstructs the time domain data from the filtered frequency domain representation. The spectrum is always displayed in a normalized decibel format.

If the data set size is 512 or lower (257 or lower points in the frequency spectrum), a bar format is automatically used. Otherwise, the points in the spectrum will be rendered directly with lines connecting the points. You can override either of these point formats using the Modify Point Format option and their respective states will be saved across sessions.

In this procedure, the main task is to set a threshold, either in frequency or spectral magnitude, whereby the noise that is present in the signal can be removed. When a Frequency threshold is selected, all frequency domain information beyond this specified frequency is zeroed in order to create the output data. It is quite common for a signal to exist at low frequencies, whereas white noise exists across all frequencies. This low-pass filtration preserves the signal and discards the noise within the higher frequencies. The noise that exists within the retained signal frequencies is not filtered.

Unlike time domain filters, a Fourier domain lowpass filter can achieve an absolute truncation at the specified frequency. Whereas discontinuities in time domain data introduce real problems with Fourier analysis, this "cliff" or "brickwall" discontinuity that is produced in the Fourier spectrum of the output data is not usually a concern. In fact, it is usually the ideal in terms of noise reduction and smoothing. If the output data are subsequently transformed into the Fourier domain using an exact-n FFT, it will look exactly like the FFT for the active or retained channels in the upper graph. The spectral magnitudes for the frequencies that had been zeroed will exist at approximately -300dB, the machine precision limit, in a normalized dB plot.

If a dB threshold is selected, the frequency channels are zeroed based on a specified signal threshold. If the signal of interest can be assumed to exist primarily in the highest dB Fourier channels, as is often the case, it is possible that a much higher percentage of the overall channels can be zeroed. This is the only way to threshold a signal with high frequency components within the Fourier domain.

If possible, it is usually a good idea to retain the adjacent sidelobes of each spectral component.

Unlike the frequency thresholding, dB thresholding can result in multiple "cliff" or "brickwall" discontinuity zones in the Fourier spectrum of the output data. This is especially true when multiple components are present. There may be any number of zones at approximately -300dB in a normalized dB spectrum. Again, this is generally the ideal for removing noise.

If you really need to produce soft transitions between the retained and discarded frequencies, you can use the Fourier Filtering and Reconstruction with a data tapering window to implement this same type of frequency or dB thresholding. When this is done, a subsequent transform of the filtered data into the Fourier domain using an exact-n non-windowed FFT will produce the transitions typical of time-domain filter algorithms.

The dB thresholding, in one simple step, accomplishes what could require a number of bandpass filters in the time domain and it does so with brickwall cutoffs. In this basic procedure, the dB threshold is fixed for all components. To filter using a different dB threshold for each component, the Fourier Filtering and Reconstruction must be used.

Thresholding

The Frequency and dB threshold values can be entered numerically or graphically. To enter a frequency graphically left click the mouse on the upper graph at the frequency desired (the dB position is unimportant). To enter a dB value graphically, left click the mouse in the upper graph at the desired dB (the frequency is unimportant). In this option, it is not possible to graphically toggle individual frequency channels on and off.

The Reset Previous button will restore the threshold value last used in this procedure.

Estimated Noise Reduction

AutoSignal offers a robust noise estimation procedure that may be of some value for low-frequency signals. A cubic polynomial interpolation is made for each point using the two points to the left and the two to the right (excluding the current point). The difference between the interpolated and signal values is used to generate a measure of the white noise present in the signal. This assumes that the signal can be locally characterized by a smooth cubic interpolant. Also, the signal component(s) should exist only in the lower quarter of the Nyquist range. If a high frequency signal component is present, these estimates of noise will be invalid.

The In value reports the estimated white noise in the incoming data, the Out value the estimated white noise for the filtered signal. The percent is given as the amount of estimated noise remaining after filtration.

Power Reduction

In this section the In value reports the TISA power in the incoming data. The Out value is the TISA power for the filtered signal. The percent is given as the amount of power remaining after filtration. For most S/N ratios, the reduction in power should be minimal. These values can alert you when signal components are being discarded.

Correlation Coefficient

The r-squared correlation coefficient should also remain high. An rē of 1 is a perfect correlation while a value of 0 means the filtered and unfiltered signals are uncorrelated. Low rē values are also indicative of signal components being lost in the thresholding.

List

The List Data option lists the index, time, and output signal in a three column table. The listing uses the AutoSignal text viewer facility.

Copy

The Copy Data to Clipboard option copies the time and output signal values to the clipboard. Formats include full precision binary (for spreadsheets such as Excel) and ASCII (for pasting into text editors).

Save

The Save Data to Disk option writes the time and output values to a supported file format. These formats include ASCII, Excel 97, Excel 95, Lotus WK3, Lotus WK1, SPSS, or Systat.

Production Facility

The AutoSignal Automation facility allows unattended processing of large numbers of data sets. The data sets can be consolidated in an Excel file or acquired using a DLL. The graphs can be exported to an MS Word RTF file, while the processed data can be exported to an Excel 95 or Excel 97 file.

Local Options

A local option changes the data set for the duration of the current procedure only. The main data table is not altered. AutoSignal offers four local options in most of the spectral procedures.

Section the data to isolate specific regions for processing.

Detrend for removing mean or subtracting a least-squares trend model.

Eigendecomposition Filtration for isolating spectral components by signal strength.

The Reset button restores the data to its state when first entering the procedure. Note that if you implement sequential local procedures, all of the revisions are discarded upon reset. If an Automation Session is in progress, the Reset button can be used to terminate the automated processing.

When exiting this procedure with the OK button, an option will be presented to update AutoSignal's main data table with the denoised data.