Fourier Spectrum of Unevenly Sampled Data

The Fourier Spectrum of Unevenly Sampled Data option in the Spectral menu or the Spectral toolbar generates a Lomb-Scargle periodogram for data with unevenly spaced X values. This procedure was developed by astrophysicists who must often contend with data that are not evenly sampled. The utility of this algorithm is not limited strictly to unevenly spaced data, however. There are also benefits for uniformly sampled data.

This algorithm produces results nearly identical to an FFT, although it is not a traditional Fourier transform, and will not exactly reproduce FFT results. The algorithm essentially defines a second series of abscissa (time) values using a variable offset term in the definition of the PSD. This results in an algorithm that is equivalent to the least-squares fitting of sine curves (at specified frequencies) to the data.

This option is similar to the Fourier Spectrum with Data Window option with an important exception that involves oversampling and spectral length selections.

Data that have been unevenly sampled are not subject to an average Nyquist limitation. That is, spectral information at frequencies higher than the average Nyquist frequency is not automatically aliased to lower frequencies. The reason this is possible is that the uneven sampling trades off a complete information state within the Nyquist interval for an incomplete state, but one where some of the points are spaced much closer than the average sample interval. With unevenly sampled data, you can choose how far to "run out" the spectrum. The AutoSignal implementation allows up to 10x the average Nyquist interval.

The spectrum length is not directly related to the length of the data stream. Further, the spectrum is computed only for real data (positive frequencies), does not include an initial zero frequency, and can be evaluated at any desired frequency set. Although AutoSignal creates a uniformly spaced frequency spectrum similar to the FFT, this is not a requirement for the algorithm.

For more information, see Jeffrey Scargle, Astrophysical Journal, v263, p.835 and William Press and G.B. Rybicki, Astrophysical Journal, v338, p.277.

Algorithm

The original or Direct algorithm can be very slow with large data sets. The Fast algorithm (Press and Rybicki) uses the FFT in a novel way to dramatically speed up computations and should be used for large data sets. The Best Exact n FFT algorithm is used. Both algorithms should yield the same spectrum regardless of data set size and spectrum size, although the Direct procedure may offer slightly greater precision.

Window

AutoSignal offers a broad selection of data tapering windows to minimize spectral leakage. The adjust field is used to set the spectral width, and thus the dynamic range, of adjustable windows. This field will be disabled for fixed windows.

An appropriate window and adjustable width can be determined by using the Explore Data Tapering Windows option.

Only continuous window functions are available in this procedure. For this reason, the Chebyshev and Slepian DPSS windows are not available. The Chebyshev Appr window is a continuous approximation to the Chebyshev window that was designed expressly for this procedure.

Spectrum

The n for the spectrum sets the length of the positive frequencies in the computed spectrum. Unlike the FFT, this does not include an initial zero frequency. You can enter any size desired up to 32768. You may also select from a power of 2 sequence in the drop down box. This is an advantage for the Fast procedure since it uses the FFT.

Increasing this number of evaluated frequencies in the spectrum produces results similar to zero padding an FFT. This can aid in more accurately determining the center frequencies of spectral peaks. As with the FFT, however, this will not change the basic shape of the spectrum.

The End field sets the frequency upper limit of the spectrum. For unevenly sampled data, you can "run out" the spectrum to up to 10x the average Nyquist limit. The initial value will consist of 1x the average Nyquist bound.

AutoSignal creates a uniformly spaced frequency spectrum similar to the FFT.

Plot

The frequency domain information can be plotted in a variety of formats. In the following table, mag2 is the magnitude-squared of the Lomb spectrum (normalized for spectrum size) at a given frequency, n is the data set size, dx is the sampling interval, and var is the variance of the data series.

· Lomb Spec, mag2/n/var

· Magnitude, sqrt(mag2)

· Amplitude, 2.0*sqrt(mag2)/n

· dB, decibels, 10.0*log10(mag2)

· dB Norm, decibels, normalized to 0 for frequency channel with maximum power

· PSD SumSq, Power as Sum Squared Amplitude, 2.0*mag2/n

· PSD MeanSq, Power as Mean Squared Amplitude, 2.0*mag2/n/n

· PSD TimeInt, Power as Time-Integral Squared Amplitude, 2.0*dx*mag2/n

In an amplitude plot, you see the actual amplitude of sine components. In a normalized decibel plot, the highest peak is at 0dB, a peak at -3dB would have half the power, and a peak at -6dB would have half the amplitude. The PSD TISA (time-integral squared amplitude power) is the actual integral under the curve defined by the square of the raw data.

AR(1) Background

AutoSignal offers peak-type critical limits to determine the statistical significance of the largest peak present in the spectrum. These limits are computed for all windows, including those with adjustable parameters. The default background used for this null hypothesis is white (Gaussian or normally distributed) noise, AR(1)=0.0. A lag-1 autoregressive spectrum can also be specified. An AR(1) coefficient greater than 0.0 can often model red noise (where the noise power decreases with increasing frequency).

The AI Expert option sets the AR(1) value to the lag-1 normalized autocorrelation. This should only be used as a preliminary estimate. For a better estimate of the background, use the one of the Fourier Filtering and Reconstruction, Eigendecomposition Filtering and Reconstruction, or Wavelet Filtering and Reconstruction options to remove the spectral components from the signal, leaving only the background. Then use the AR (AutoRegressive) Spectrum option with an order of 1 to fit the AR model. The desired AR(1) coefficient is listed in AR procedure's numeric summary.

The Show Significance Levels option is in the graph's toolbar. This button is used to toggle the significance levels on and off.

Note that the exponential distribution limits traditionally a part of the Lomb-Scargle periodogram are not used in this procedure. AutoSignal implements peak-type critical limits as opposed to the more commonly used confidence limits.

The critical limits for this procedure were generated from Monte Carlo trials that used uniformly spaced abscissae. Although researchers have reported little difference in significance levels between uniformly spaced data and randomly spaced data, you should consider these critical limits approximate when data are not evenly sampled.

Peaks

The spectral peaks are identified by a local maxima detection algorithm. The sig item sets the target number of peaks (signal components) to detect. Up to 50 peaks can be detected. Peaks are ranked by interpolated amplitude. Note that this target signal component count may not be realized as fewer peaks than this target may be detected. Note also that the frequency analysis and linear sinusoidal fits reported in the Numeric Summary use the component count and frequencies from this peak identification.

The wid item sets the bin width tolerance for defining a peak. A peak must exist across this number of FFT bins to be counted. The default is a single bin.

The Display Maxima option is used to step through the options for displaying spectral peak labels: frequencies, spectral magnitudes, both frequencies and spectral magnitudes, or none.

List

The List Data option lists the index, frequency, and the spectral quantity currently plotted. The listing uses the AutoSignal text viewer facility.

Copy

The Copy Data to Clipboard option copies the frequency and the spectral quantity currently plotted to the clipboard. Formats include full precision binary (for spreadsheets such as Excel) and ASCII (for pasting into text editors). You can generally find a Paste As option in most applications if you want specific control over the format imported.

Save

The Save Data to Disk option writes the frequency and the spectral quantity currently plotted to a supported file format. These formats include ASCII, Excel 97, Excel 95, Lotus WK3, Lotus WK1, SPSS, or Systat.

Production Facility

The AutoSignal Automation facility allows unattended processing of large numbers of data sets. The data sets can be consolidated in an Excel file or acquired using a DLL. The numeric summaries and graphs can be exported to an MS Word RTF file, while the extended data summaries or the current spectra can be exported to an Excel 95 or Excel 97 file.

Numeric Summary

The Numeric Summary offers a full FFT report. The report optionally includes a listing of the interpolated spectral peaks, a frequency analysis, and a linear sinusoidal least-squares fit summary.

Non-Linear Optimization

The Non-Linear Optimization offers the means to refine the parameter estimates given in the linear sinusoidal fit that is reported in the Numeric Summary. Constrained least-squares and robust (maximum likelihood) non-linear fitting is available with either sinusoid or damped sinusoid models.

Rich-Text Format Export

The Export Numeric Summary and Graph to RTF File option writes the numeric summary and spectral plot to an RTF file. The numeric portion of the file is based upon the current settings in the Numeric Summary option. The text data will be written to portrait orientation pages. The graph uses the current settings and size of the spectral plot, and is inserted as a Windows Metafile. The graph always uses a landscape orientation. Beyond a certain size, the graph utilizes a full landscape page.

Explore Data Tapering Windows

The Explore Data Tapering Windows option is available to assist with data window selection and adjustment. This option plots a discrete FFT in a problem designed to illustrate the frequency widths of each window as well as the rolloff vs. maximum sidelobe tradeoff. Up to four windows can be inspected simultaneously and key window properties are empirically determined.

Local Options

A local option changes the data set for the duration of the current procedure only. The main data table is not altered. AutoSignal offers four local options in most of the spectral procedures.

Section the data to isolate specific regions for processing.

Detrend for removing mean or subtracting a least-squares trend model.

Fourier Filtration for isolating spectral components by frequency.

Eigendecomposition Filtration for isolating spectral components by signal strength.

The Reset button restores the data to its state when first entering the procedure. Note that if you implement sequential local procedures, all of the revisions are discarded upon reset. If an Automation Session is in progress, the Reset button can be used to terminate the automated processing.