Return-Path: X-Spam-DCC: paranoid 1481; Body=2 Fuz1=2 Fuz2=2 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on lipkowski.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DNS_FROM_AHBL_RHSBL, HTML_MESSAGE,PLING_QUERY,SPF_PASS autolearn=no version=3.1.3 Received: from post.thorcom.com (post.thorcom.com [195.171.43.25]) by paranoid.lipkowski.org (8.13.7/8.13.7) with ESMTP id uA3JnWhx029398 for ; Thu, 3 Nov 2016 20:49:32 +0100 Received: from majordom by post.thorcom.com with local (Exim 4.14) id 1c2NxD-0007hs-1s for rs_out_1@blacksheep.org; Thu, 03 Nov 2016 19:45:39 +0000 Received: from [195.171.43.32] (helo=relay1.thorcom.net) by post.thorcom.com with esmtp (Exim 4.14) id 1c2NxC-0007hj-JK for rsgb_lf_group@blacksheep.org; Thu, 03 Nov 2016 19:45:38 +0000 Received: from mail-wm0-x22f.google.com ([2a00:1450:400c:c09::22f]) by relay1.thorcom.net with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.87) (envelope-from ) id 1c2Nx9-0000jE-Aa for rsgb_lf_group@blacksheep.org; Thu, 03 Nov 2016 19:45:37 +0000 Received: by mail-wm0-x22f.google.com with SMTP id a197so8544445wmd.0 for ; Thu, 03 Nov 2016 12:45:35 -0700 (PDT) X-DKIM-Result: Domain=gmail.com Result=Good and Known Domain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=GxaiJlqd9ImZPxbN/BKZHRRS8bmFZVccSvzWqTtpwYU=; b=yAdXkcEhnuueaDEmSnIVK9Hn+/5SWjfbKxr15qZ1WQOB+WJ2H5lwPmXLUJCPyH8kp1 hun/P13SmkGzhlC4d/NqmbPmWPFDzVzRjVUcjf3ZdT6L0faJtEDl7ClrCn5355qQEq7m YPx3yqTJxbo87ugywtiOiCS+G+mE+acC/38HSAI/147RTW+0fpT5mgoNjyAx/Mf+5XnW AMBReYorpvRoCwbfU8Z7LT4E16JeUp2cdQ9Am/CSdLpavaNw97vnzmOt+e0YZ45nDdLW /e+4/kb9S7ZZne4Pi8+T9ZUTECXNx2PSs1rbOnCecWjVxatDln48js7TSqzqZLly7TVF 5+Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=GxaiJlqd9ImZPxbN/BKZHRRS8bmFZVccSvzWqTtpwYU=; b=mw2v3fVZuOUBXdDdfX7AOicID/h7T1MppVNMvpQWIVH82YLKj6T0rPSdCoztfZWiVr j2iyui/CVez9MbS/g0yhdQYgSgx5c9r9zbW3ubfPXKg4scNNGcFuR4PBIixcNgehykEc wVhwsoBmkYnHa9OMq2bDsr/Y8NgYNt621dGtI1/HDLDxxfiRYFX12WEWfaoUkwO4nbfI WAu1fsy5bZE8bWpmWo6xqLAQQai74dGRpOYZhDrT/k1dJGNiBha+ZSWOkNj4ehjo5kvA Yk71lZW1jHbMT0PA382V92mWsyM0iR5s4ggi8q6sEIzbRMp4XrhL8jpiAtQJgPMSkF72 ZDWg== X-Gm-Message-State: ABUngveSJ5XnLlK0scjr+76MIq2UfHfh5axBa8fDhUMxlQBs7nxWeB464aTMXVOn81FaQAa4Pzpa757D/wiIIA== X-Received: by 10.28.150.134 with SMTP id y128mr10770624wmd.6.1478202333564; Thu, 03 Nov 2016 12:45:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.209.74 with HTTP; Thu, 3 Nov 2016 12:45:32 -0700 (PDT) In-Reply-To: <22185768CF114BE0A22668E235A933A2@gnat> References: <581B3A1F.5060609@posteo.de> <581B83FC.6060600@posteo.de> <22185768CF114BE0A22668E235A933A2@gnat> From: Andy Talbot Date: Thu, 3 Nov 2016 19:45:32 +0000 Message-ID: To: rsgb_lf_group@blacksheep.org X-Scan-Signature: caa2bbf6eb61e70e03599bec833fb637 Subject: Re: LF: Smart noise cancelling?!? Content-Type: multipart/alternative; boundary=001a114b4386ff95ef05406acc7d X-SA-Exim-Scanned: Yes Sender: owner-rsgb_lf_group@blacksheep.org Precedence: bulk Reply-To: rsgb_lf_group@blacksheep.org X-Listname: rsgb_lf_group X-SA-Exim-Rcpt-To: rs_out_1@blacksheep.org X-SA-Exim-Scanned: No; SAEximRunCond expanded to false X-Scanned-By: MIMEDefang 2.56 on 10.1.3.11 Content-Length: 11927 Status: O X-Status: X-Keywords: X-UID: 9385 --001a114b4386ff95ef05406acc7d Content-Type: text/plain; charset=UTF-8 But surely, when narrowband filtering is in place - as any narrow band mode will of necessity be doing internally - any wideband non Gaussian or bursty noise when applied to this narrow filter will eventually become Gaussian IN THAT BANDWIDTH We first ealised this in teh original QRSS tests with G3PLX back in the 1990's. 73kHz was full of spikes and 'crud' from teh then existing Decca signals and other stuff. But when Peter examined the output from the narrow filters, (the FFT bins) it lookdd like, and appeared to show itself to be Gaussian. He said it passed the tests for Gaussian noise A mathematician could probably prove that any random non Gaussian signal if filtered sufficiently narrow in comparison to itsnature, would end up Gaussian in the filtered bandwidth. In fact, to end up non-Gaussian, it would have to have definite components repeating at rates very close to the reciprocal of the bandwidth of the filter. Andy G4JNT On 3 November 2016 at 19:24, Alan Melia wrote: > This is an interesting idea. We often said in the early days of LF that LF > "noise" was not Gaussian. > I do remember attending a lecture at work by one of the resident maths > team. (Pitching for customers) He was considering telecom "noise" not > radio. But he demonstrated the standard processes, which assume Gaussian > noise were not effective on his example. He then introduced a "new" > parameter he called "burstiness". He was using Burst Error Rate as a > metric, I think, rather than S/N but the ideas were quite interesting. He > was also investigating different error-detection/correction algorithms > against the "burstness" index.. I am afraid it was mid 1980s and I dont > remember much detail (or even a name to follow for references) A few faces > in the audience were quite blank, one or two were leaning forward and some > sprawled backward as if to say "What's the problem, if its noisy, close the > door" :-)) > > Alan > G3NYK > > ----- Original Message ----- From: "DK7FC" > To: > Sent: Thursday, November 03, 2016 6:37 PM > Subject: Re: LF: Smart noise cancelling?!? > > > Hi Peter, >> >> That sounds interesting and looks convincing. But it only works for >> 'compressed' spectra covering several kHz and hours, right? >> >> As far as i understand this method can't be done for weak signal >> detections because you 'need' all the energy available from the weak signal >> to fill the bin (one pixel) and just throw away what causes a reduction of >> the SNR, so just the stronger QRN bursts are blanked and most of the signal >> is coming through whereas your method selects just a small fraction of the >> incoming energy (?). >> >> My idea/question came from the consideration that different kind of QRN >> has different optimum blanker settings. >> >> Am 02.11.2016 21:25, schrieb Paul Nicholson: >> >>> >>> Even more aggressive sferic blanking raises >>> the decode to Eb/N0 +1.7 dB BER 38.2% >>> S/N 16.10 dB in 25.4 uHz. >>> >> >> So, if propagation changes, the optimal blanker settings will change. So >> they vary all the time. If a post-processing of a transmission/recording >> taking several hours and night/day changes, it could be useful to >> dynamically vary the blanker settings. So these blanker setting levels >> would have to be determined/calculated by the incoming signal and then >> applied in a next step. >> This may be CPU-load intensive, i don't know, it's behind my current >> skills. But the idea is there... >> >> >> 73, Stefan >> >> >> Am 03.11.2016 18:55, schrieb Peter: >> >>> Hi Stefan et al., >>> >>> On 03.11.2016 14:22, DK7FC wrote: >>> >>>> ... >>>> Last night i thought a bit about noise cancelling on LF/VLF. >>>> >>> > ... >>> >>> Running an experimental receiver at VLF/LF for SID-detection (on RPi 3) >>> I chose an almost non-parametric procedure running in frequency domain. It >>> works as follows: >>> Do a windowed FFT, compute the median (by sorting) from the power >>> spectrum. Store all spectra and corresponding median values. Next choose a >>> time period (let's say 100ms), pick the spectrum with the lowest median, >>> plot it, drop all the others. The key is that "Median values" are more >>> robust to outliers compared to other averaging procedures. >>> >>> See what happened when switching from simple averaging to median >>> selection algorithm (~16:50 utc): >>> http://lf-radio.de/cgi-bin/test/show_wf.cgi?date=16-10-02 >>> >>> I know that this won't work in case of searching for coherent signal >>> detection, or would be hard to implement. But using this method I'm >>> detecting such very weak signals from far east like NDI or RTZ on a regular >>> basis. >>> >>> Drawbacks? Yes; it's throwing away a lot of information which may be >>> useful. Another pitfall has to be mentioned: using 1 sec. as a selection >>> window strong time service transmitters nearly vanished since the algorithm >>> will unerringly choose the gaps [^_^]. Therefore I'm using only the >>> spectral part between 15 and 50 kHz for computing the median values. >>> >>> Peter, df3lp >>> >>> >> > > --001a114b4386ff95ef05406acc7d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
But surely, when narrowband filtering is in place - as any= narrow band mode will of necessity be doing internally - any wideband non = Gaussian or bursty noise when applied to this narrow filter will eventually= become Gaussian IN THAT BANDWIDTH

We first ealised this= in teh original QRSS tests with G3PLX back in the 1990's. =C2=A0 73kHz= was full of spikes and 'crud' from teh then existing Decca signals= and other stuff. =C2=A0 But when Peter examined the output from the narrow= filters, (the FFT bins) =C2=A0it lookdd like, and appeared to show itself = to be Gaussian. =C2=A0 He said it passed the tests for Gaussian noise
=

A mathematician could probably prove that any random no= n Gaussian signal if filtered sufficiently narrow in comparison to itsnatur= e, would end up Gaussian in the filtered bandwidth.=C2=A0

In fact, to end up non-Gaussian, it would have to have definite com= ponents repeating at rates very close to the reciprocal of the bandwidth of= the filter.

Andy =C2=A0G4JNT

On 3 November 2016 at 19:2= 4, Alan Melia <alan.melia@btinternet.com> wrote:
=
This is an interesting idea. We often said i= n the early days of LF that LF "noise" was not Gaussian.
I do remember attending a lecture at work by one of the resident maths team= . (Pitching for customers) He was considering telecom "noise" not= radio. But he demonstrated the standard processes, which assume Gaussian n= oise were not effective on his example. He then introduced a "new"= ; parameter he called "burstiness". He was using Burst Error Rate= as a metric, I think, rather than S/N but the ideas were quite interesting= . He was also investigating different error-detection/correction algorithms= against the "burstness" index.. I am afraid it was mid 1980s and= I dont remember much detail (or even a name to follow for references) A fe= w faces in the audience were quite blank, one or two were leaning forward a= nd some sprawled backward as if to say "What's the problem, if its= noisy, close the door" :-))

Alan
G3NYK

----- Original Message ----- From: "DK7FC" <selberdenken@posteo.de> To: <r= sgb_lf_group@blacksheep.org>
Sent: Thursday, November 03, 2016 6:37 PM
Subject: Re: LF: Smart noise cancelling?!?


Hi Peter,

That sounds interesting and looks convincing. But it only works for 'co= mpressed' spectra covering several kHz and hours, right?

As far as i understand this method can't be done for weak signal detect= ions because you 'need' all the energy available from the weak sign= al to fill the bin (one pixel) and just throw away what causes a reduction = of the SNR, so just the stronger QRN bursts are blanked and most of the sig= nal is coming through whereas your method selects just a small fraction of = the incoming energy (?).

My idea/question came from the consideration that different kind of QRN has= different optimum blanker settings.

Am 02.11.2016 21:25, schrieb Paul Nicholson:

Even more aggressive sferic blanking raises
the decode to Eb/N0 +1.7 dB BER 38.2%
S/N 16.10 dB in 25.4 uHz.

So, if propagation changes, the optimal blanker settings will change. So th= ey vary all the time. If a post-processing of a transmission/recording taki= ng several hours and night/day changes, it could be useful to dynamically v= ary the blanker settings. So these blanker setting levels would have to be = determined/calculated by the incoming signal and then applied in a next ste= p.
This may be CPU-load intensive, i don't know, it's behind my curren= t skills. But the idea is there...


73, Stefan


Am 03.11.2016 18:55, schrieb Peter:
Hi Stefan et al.,

On 03.11.2016 14:22, DK7FC wrote:
...
Last night i thought a bit about noise cancelling on LF/VLF.
> ...

Running an experimental receiver at VLF/LF for SID-detection (on RPi 3) I c= hose an almost non-parametric procedure running in frequency domain. It wor= ks as follows:
Do a windowed FFT, compute the median (by sorting) from the power spectrum.= Store all spectra and corresponding median values. Next choose a time peri= od (let's say 100ms), pick the spectrum with the lowest median, plot it= , drop all the others. The key is that "Median values" are more r= obust to outliers compared to other averaging procedures.

See what happened when switching from simple averaging to median selection = algorithm (~16:50 utc):
http://lf-radio.de/cgi-bin/test/show= _wf.cgi?date=3D16-10-02

I know that this won't work in case of searching for coherent signal de= tection, or would be hard to implement. But using this method I'm detec= ting such very weak signals from far east like NDI or RTZ on a regular basi= s.

Drawbacks? Yes; it's throwing away a lot of information which may be us= eful. Another pitfall has to be mentioned: using 1 sec. as a selection wind= ow strong time service transmitters nearly vanished since the algorithm wil= l unerringly choose the gaps [^_^]. Therefore I'm using only the spectr= al part between 15 and 50 kHz for computing the median values.

Peter, df3lp





--001a114b4386ff95ef05406acc7d--