OPEN-SOURCE SCRIPT
Diupdate Weighted Fourier Transform: Spectral Gating & Main Frequency
🙏🏻 This drop has 2 purposes:
1) to inform every1 who'd ever see it that Weighted Fourier Tranform does exist, while being available nowhere online, not even in papers, yet there's nothing incredibly complicated about it, and it can/should be used in certain cases;
2) to show TradingView users how they can use it now in dem endevours, to show em what spectral filtering is, and what can they do with all of it in diy mode.
... so we gonna have 2 sections in the description
Section 1: Weighted Fourier Transform
It's quite easy to include weights in Fourier analysis: you just premultiply each datapoint by its corresponding weight -> feed to direct Fourier Transform, and then divide by weights after inverse Fourier transform. Alternatevely, in direct transform you just multiply contributions of each data point to the real and imaginary parts of the Fourier transform by corresponding weights (in accumulation phase), and in inverse transform you divide by weights instead during the accumulation phase. Everything else stays the same just like in non-weighted version.
If you're from the first target group let's say, you prolly know a thing or deux about how to code & about Fourier Transform, so you can just check [13, 51] lines of code to see the implementation of Weighted Discrete version of Fourier Transform, and port it to to any technology you desire. Pine Script is a developing technology that is incredibly comfortable in use for quant-related tasks and anything involving time series in general. While also using Python for research and C++ for development, every time I can do what I want in Pine Script, I reach for it and never touch matlab, python, R, or anything else.
Weighted version allows you to explicetly include order/time information into the operation, which is essential with every time series, although not widely used in mainstream just as many other obvious and right things. If you think deeply, you'll understand that you can apply a usual non-weighted Fourier to any 2d+ data you can (even if none of these dimensions represent time), because this is a geometric tool in essence. By applying linearly decaying weights inside Fourier transform, you're explicetly saying, "one of these dimensions is Time, and weights represent the order". And obviously you can combine multiple weightings, eg time and another characteristic of each datum, allows you to include another non-spatial dimension in your model.
By doing that, on properly processed (not only stationary but Also centered around zero data), you can get some interesting results that you won't be able to recreate without weights:
^^ A sine wave, centered around zero, period of 16. Gray line made by: DWFT (direct weighted Fourier transform) -> spectral gating -> IWFT (inverse weighted Fourier transform) -> plotting the last value of gated reconstructed data, all applied to expanding window. Look how precisely it follows the original data (the sine wave) with no lag at all. This can't be done by using non-weighted version of Fourier transform.
^^ spectral filtering applied to the whole dataset, calculated on the latest data update
And you should never forget about Fast Fourier Transform, tho it needs recursion...
Section 2: About use cases for quant trading, about this particular implementaion in Pine Script 6 (currently the latest version as of Friday 13, December 2k24).
Given the current state of things, we have certain limits on matrix size on TradingView (and we need big dope matrixes to calculate polynomial regression -> detrend & center our data before Fourier), and recursion is not yet available in Pine Script, so the script works on short datasets only, and requires some time.
A note on detrending. For quality results, Fourier Transform should be applied to not only stationary but also centered around zero data. The rightest way to do detrending of time series
is to fit Cumulative Weighted Moving Polynomial Regression (known as WLSMA in some narrow circles xD) and calculate the deltas between datapoint at time t and this wonderful fit at time t. That's exactly what you see on the main chart of script description: notice the distances between chart and WLSMA, now look lower and see how it matches the distances between zero and purple line in WFT study. Using residuals of one regression fit of the whole dataset makes less sense in time series context, we break some 'time' and order rules in a way, tho not many understand/cares abouit it in mainstream quant industry.
Two ways of using the script:
Spectral Gating aka Spectral filtering. Frequency domain filtering is quite responsive and for a greater computational cost does not introduce a lag the way it works with time-domain filtering. Works this way: direct Fourier transform your data to get frequency & phase info -> compute power spectrum out of it -> zero out all dem freqs that ain't hit your threshold -> inverse Fourier tranform what's left -> repeat at each datapoint plotting the very first value of reconstructed array*. With this you can watch for zero crossings to make appropriate trading decisions.
^^ plot Freq pass to use the script this way, use Level setting to control the intensity of gating. These 3 only available values: -1, 0 and 1, are the general & natural ones.
* if you turn on labels in script's style settings, you see the gray dots perfectly fitting your data. They get recalculated (for the whole dataset) at each update. You call it repainting, this is for analytical & aesthetic purposes. Included for demonstration only.
Finding main/dominant frequency & period. You can use it to set up Length for your other studies, and for analytical purposes simply to understand the periodicity of your data.
^^ plot main frequency/main period to use the script this way. On the screenshot, you can see the script applied to sine wave of period 16, notice how many datapoints it took the algo to figure out the signal's period quite good in expanding window mode
Now what's the next step? You can try applying signal windowing techniques to make it all less data-driven but your ego-driven, make a weighted periodogram or autocorrelogram (check Wiener-Khinchin Theorem), and maybe whole shiny spectrogram?
... you decide, choice is yours,
The butterfly reflect the doors ...
∞
1) to inform every1 who'd ever see it that Weighted Fourier Tranform does exist, while being available nowhere online, not even in papers, yet there's nothing incredibly complicated about it, and it can/should be used in certain cases;
2) to show TradingView users how they can use it now in dem endevours, to show em what spectral filtering is, and what can they do with all of it in diy mode.
... so we gonna have 2 sections in the description
Section 1: Weighted Fourier Transform
It's quite easy to include weights in Fourier analysis: you just premultiply each datapoint by its corresponding weight -> feed to direct Fourier Transform, and then divide by weights after inverse Fourier transform. Alternatevely, in direct transform you just multiply contributions of each data point to the real and imaginary parts of the Fourier transform by corresponding weights (in accumulation phase), and in inverse transform you divide by weights instead during the accumulation phase. Everything else stays the same just like in non-weighted version.
If you're from the first target group let's say, you prolly know a thing or deux about how to code & about Fourier Transform, so you can just check [13, 51] lines of code to see the implementation of Weighted Discrete version of Fourier Transform, and port it to to any technology you desire. Pine Script is a developing technology that is incredibly comfortable in use for quant-related tasks and anything involving time series in general. While also using Python for research and C++ for development, every time I can do what I want in Pine Script, I reach for it and never touch matlab, python, R, or anything else.
Weighted version allows you to explicetly include order/time information into the operation, which is essential with every time series, although not widely used in mainstream just as many other obvious and right things. If you think deeply, you'll understand that you can apply a usual non-weighted Fourier to any 2d+ data you can (even if none of these dimensions represent time), because this is a geometric tool in essence. By applying linearly decaying weights inside Fourier transform, you're explicetly saying, "one of these dimensions is Time, and weights represent the order". And obviously you can combine multiple weightings, eg time and another characteristic of each datum, allows you to include another non-spatial dimension in your model.
By doing that, on properly processed (not only stationary but Also centered around zero data), you can get some interesting results that you won't be able to recreate without weights:
^^ A sine wave, centered around zero, period of 16. Gray line made by: DWFT (direct weighted Fourier transform) -> spectral gating -> IWFT (inverse weighted Fourier transform) -> plotting the last value of gated reconstructed data, all applied to expanding window. Look how precisely it follows the original data (the sine wave) with no lag at all. This can't be done by using non-weighted version of Fourier transform.
^^ spectral filtering applied to the whole dataset, calculated on the latest data update
And you should never forget about Fast Fourier Transform, tho it needs recursion...
Section 2: About use cases for quant trading, about this particular implementaion in Pine Script 6 (currently the latest version as of Friday 13, December 2k24).
Given the current state of things, we have certain limits on matrix size on TradingView (and we need big dope matrixes to calculate polynomial regression -> detrend & center our data before Fourier), and recursion is not yet available in Pine Script, so the script works on short datasets only, and requires some time.
A note on detrending. For quality results, Fourier Transform should be applied to not only stationary but also centered around zero data. The rightest way to do detrending of time series
is to fit Cumulative Weighted Moving Polynomial Regression (known as WLSMA in some narrow circles xD) and calculate the deltas between datapoint at time t and this wonderful fit at time t. That's exactly what you see on the main chart of script description: notice the distances between chart and WLSMA, now look lower and see how it matches the distances between zero and purple line in WFT study. Using residuals of one regression fit of the whole dataset makes less sense in time series context, we break some 'time' and order rules in a way, tho not many understand/cares abouit it in mainstream quant industry.
Two ways of using the script:
Spectral Gating aka Spectral filtering. Frequency domain filtering is quite responsive and for a greater computational cost does not introduce a lag the way it works with time-domain filtering. Works this way: direct Fourier transform your data to get frequency & phase info -> compute power spectrum out of it -> zero out all dem freqs that ain't hit your threshold -> inverse Fourier tranform what's left -> repeat at each datapoint plotting the very first value of reconstructed array*. With this you can watch for zero crossings to make appropriate trading decisions.
^^ plot Freq pass to use the script this way, use Level setting to control the intensity of gating. These 3 only available values: -1, 0 and 1, are the general & natural ones.
* if you turn on labels in script's style settings, you see the gray dots perfectly fitting your data. They get recalculated (for the whole dataset) at each update. You call it repainting, this is for analytical & aesthetic purposes. Included for demonstration only.
Finding main/dominant frequency & period. You can use it to set up Length for your other studies, and for analytical purposes simply to understand the periodicity of your data.
^^ plot main frequency/main period to use the script this way. On the screenshot, you can see the script applied to sine wave of period 16, notice how many datapoints it took the algo to figure out the signal's period quite good in expanding window mode
Now what's the next step? You can try applying signal windowing techniques to make it all less data-driven but your ego-driven, make a weighted periodogram or autocorrelogram (check Wiener-Khinchin Theorem), and maybe whole shiny spectrogram?
... you decide, choice is yours,
The butterfly reflect the doors ...
∞
Catatan Rilis
🙏🏻 This is not an update, it’s a complete remake with entirely different math and improved code. This Version 2 would also present 2 developments of mine:- DTSN Decomposition of chaotic time-series
- Extended Beta(2, 2) Window
What’s new:
- New interval block would allow to select moving windows and expanding windows (including anchored ones, and cumulative one over all data)
- For visualization now we have: plasma colored spectrogram like in any pro DSP software, and a lil periodogram
Notes on use:
- Spectrogram and periodogram would only work with "amplitudes & periods” spectrum. “Powers & frequencies “ spectrum is left for experimentation purposes only: that’s the type of spectrum u wanna use in your EQs when u produce music. So if u do plugins for DAWs, take a note how I made it, might be useful
- Always apply the taper (Extended Beta(2, 2) Window in moving window mode, never apply it in expanding window mode.
- In the “style” setting you can also turn on/off Model and Src plots that would be displayed in the same lower study. Useful sometimes
- the main use case is decomposition, this way you see how different components of the data are moving separately. That can provide weak vs strong signals, and ability to gauge overall market state.
Now the Source part:
DTSN Decomposition
^^ Purple line: Drift, blue & red like: Trend, Maroon line: SIgnal, gray line: Noise.
I made DTSN as general and non-arbitrary as it can be, ready for real world chaotic data where all the keywords related to dynamic & complex systems, alike “feedback” or “self-regulation”, suddenly start to make sense.
DTSN:
- Decomposes data into Drift, Trends, Signal, Noise components (that’s why DTSN)
- No assumptions about particular strong cycles, seasonality etc
- The “Signal” component would contain everything (including possible cycles if there are any, or seasonality etc) that might be non-random and was not included into drift or trends components
- DTSN does not "repaint", no historical values are changed
- Does not mix, and properly separates drift and trends components
- does not need any parameters to tune, all the info is gained implicitly via proper modelling and conscious realization of math entities behind the info flow
The method:
Step 1: De-drift. De-drifted series (data minus moving avg) is formed ‘incrementally’, not in batch way. This allows to clearly separate drift and trend components (otherwise the following regression step would contain both drift and trends).
Moving average now is our “Drift component”.
Step 2: De-trend. Linear regression is put on de-drifted data, and at this stage the residuals get subtracted from regline in batch mode.
Moving regression endpoint now is our “trend component”.
Step 3: De-noise.
- via Fourier, from de-trended data we arrive to amplitudes & periods spectrum (the natural one for real world data, not “signals” per se).
- We then apply linear regression on the spectrum, and obtain residuals from this regression line.
- Next we apply pow2 transform on these residuals, that provides us 2 interesting properties:
-- now data has a probability singularity at mean, meaning it can’t be faithfully modeled as a distribution anymore;
-- now both voids and peaks of the spectrum are treated alike as strong ones. - We calculate mean over these squared residuals, and use it as threshold for spectral gating: we leave only the bins where squared residuals trespassed the threshold.
- This filtered spectrum gets reconstructed back to data space via inverse Fourier transform.
That gives us the “Signal component”. What’s left is noise. Decomposition is now complete.
The most interesting part of it is treating both peaks and voids in the spectrum as the bins worth to be kept as “signal”: white noise doesn't have these voids, its spectrum is flat. As a complex system, the market iterates over equilibrium and imbalance, this can be seen at every angle, including spectrum. All the voids (local equilibriums) want to be filled, all the peaks want to be smoothed.
In order to harness the full power of this decomposition on short lengths moving windows (~16 or less), that brings us to:
Extended Beta(2, 2) Window
^^ the blue is the original EB22W with unit Gamma (scaling factor)
Many have tried to “fix” Fourier’s inability to detect very short cycles especially with short loopback windows by inventing crazy engineering wonders, alike MESA by Burg.
Many were going miles trynna bruteforce smooth and adaptive “filters”. Nature as a complex system loves to converge to true things over time.
I think it’s not about Fourier or optimizations, I think it was always about lack of “the” window, not an empirically found or optimized one, but a natural math entity that exists in the Universe and waits to be identified, and then used & abused ofc. So here we go: Extended Beta(2, 2) Window
I made it out of derivatives and anti-derivatives of PDF function of Beta(2, 2) distribution, which in essence is simply a parabola. This is a hell of a natural choice: this one has symmetric PDF shape, finite support, limited number of derivatives, so it’s naturally tells you what you need to do with it.
The final window construction math can be found in the source code, lines [412, 434].
The resulting window has excellent properties:
- linear phase;
- Spectral leakage is removed
- Recency is encoded, new data have higher weights
- A small amount of negative weights that provide inertia, hence velocity is in the game
- It’s smooth
- Can be made adaptive with external Gamma parameter varying effective window size (Gamma = 1 is the default one that provides linear phase)
- Admissible as a new wavelet
- Its impulse response surprisingly almost exactly fits the response of the famous Jurik Moving Average.
Another use case for the kernel is being an adaptive filter, as presented in Universal Adaptive Tracking Script, or in Market State Tracker script (here the kernel is converted to IIR form, the impulse response match with the original FIR form is almost exact).
...
The “dependencies” block in the code contains all you need to introduce this functionality into your dev
...
// ∞
Catatan Rilis
bug fixSkrip open-source
Dengan semangat TradingView yang sesungguhnya, pembuat skrip ini telah menjadikannya sebagai sumber terbuka, sehingga para trader dapat meninjau dan memverifikasi fungsinya. Salut untuk penulisnya! Meskipun Anda dapat menggunakannya secara gratis, perlu diingat bahwa penerbitan ulang kode ini tunduk pada Tata Tertib kami.
Pernyataan Penyangkalan
Informasi dan publikasi ini tidak dimaksudkan, dan bukan merupakan, saran atau rekomendasi keuangan, investasi, trading, atau jenis lainnya yang diberikan atau didukung oleh TradingView. Baca selengkapnya di Ketentuan Penggunaan.
Skrip open-source
Dengan semangat TradingView yang sesungguhnya, pembuat skrip ini telah menjadikannya sebagai sumber terbuka, sehingga para trader dapat meninjau dan memverifikasi fungsinya. Salut untuk penulisnya! Meskipun Anda dapat menggunakannya secara gratis, perlu diingat bahwa penerbitan ulang kode ini tunduk pada Tata Tertib kami.
Pernyataan Penyangkalan
Informasi dan publikasi ini tidak dimaksudkan, dan bukan merupakan, saran atau rekomendasi keuangan, investasi, trading, atau jenis lainnya yang diberikan atau didukung oleh TradingView. Baca selengkapnya di Ketentuan Penggunaan.