As I understand it, the convolution of stationary deterministic signals in time domain with a Linear Time Invariant Filter has an duelity in the frequency domain which is given by the multiplication of the frequency transform of the signal with the frequency transform of the impulse response of the LTI filter.

My question is, what happens if the signal is a transient? Will the same duelity holds in the frequency domain - that is convolution in time domain ==> multiplication in frequency domain???

Well, I'm having difficulty understanding precisely what you mean, but I think the answer is yes. Transients aren't really anything particularly special. They're just a specific subset of signal types that have certain properties, like being hard to psychoacoustically encode.

Yes. In fact, obviously so . To understand why, remember that the narrower your signal is in the time domain, the wider it will be in the frequency domain and vice versa. Here's a basic example. Consider a FIR filter with only one none-zero coefficient, which will serve as our transient: [1, 0, 0, 0, 0]. Convolving any signal with this will yield the original signal. Fourier transforming this filter yields [1, 1, 1, 1, 1]. Multiplying this with the Fourier transform of any signal and then performing an inverse FT will also yield the original signal, as all frequency components are preserved.

This really holds for all signals, regardless of the type Find a good textbook for proof. In Proakis & Manolakis' "Digital Signal Processing" 3d edition it's on page 297.

Any real-world signal obeys the duality theorem.

Aside from Proakis, Rabiner and Gold, and Cooley/Tukey, you can go all the way back to limits on the Fourier integral determined by the likes of Gauss and LaPlace.

Norman Morrison's book on Fourier Mathematics is also a good place to get a nice grasp on all of this.

the principles of duality in time and frequency domain is the fundamenta theorem independed of type of signal undergoing transformations.
hence nothing wrong with transient alsol

Actually, I have been reading about Adaptive Filtering, that is filtering of non-stationary or slow changing signals. It is noted however, that ; Y(jw) = X(jw) * H(jw) ONLY if the signal reaches it's steady-state. It seems that the treatment of non-stationary signals are divided into 2 parts, one the steady-state analysis in transformed domain and the other, the transient itself in time-domain.

So, I have been trying to find undertand the linkages between Adaptive filtering theory and "classical" digital signal processing techniques. I also realized that in Adaptive filtering, there is very little mention of transform domain processing such as the FFT of a non-stationary signal or the duelity of the convolution - frequency multiplication theorem.

Repeat after me: Adaptive filters are not LTI filters. They are obviously nonlinear and they are not remotely guaranteed to follow the rules of linear filters.

Like Axon said.

If you're going to break a basic principle of linearity, don't expect things to work the same!

Yes, I must admit that Adaptive Filters are non-linear.. but are they non-time invariant as well?

Errr... Um, what else does "adaptive" usually mean?

Now, for a given filter, there's still no problem filtering transients, but now you also need to understand what your adaptation is going to do when the transient starts to drive the adaptation.

Lets say that I am trying to estimate a slow changing signal (desired signal) by adapting the filter coefficients to the error signal. Would the estimated signal always be time-invariant, that is the output delay is a constant, dependant on the desired signal?

Depends on your estimator design partially, but in general, not at all.