ATSC now possible on magnetic drives

I have found a way to reduce the data rate of ATSC IQ recording from 32 MiB/sec to 14 MiB/sec.

Last night I was taking some recordings of Fox, recording clips of World's Funniest. I was annoyed at only being able to take 1 minute at a time and then having to wait almost two minutes transferring it from RAM to my external hard drive. Taking 1 minute clips spaced 2 minutes apart causes you to miss so much and because of this I looked into reducing the required data bandwidth.

To make a long story short, I found a way to take 8-bit IQ recordings and thus reduce data bandwidth to the point where you can actually take IQ recordings to an ordinary hard drive. Why is this important? Because with hard drives being so cheap versus SSD's, you can buy an 8 TB model like mine and (at least in theory) take recordings of virtually unlimited length.

This new way of recording will take the required data from the 32 MiB/sec in my original work down to 14 MiB/sec, while still yielding good decodes! This should be well within what a magnetic drive can handle, plus it takes less data so your hard drive space goes further.

A few notes/tips:

• Use 7 MHz bandwidth on the SDRPlay. Although I said to use 8 before, 7 has been proven to provide flawless recordings while taking up less data.
• If you can find one, I would suggest you use an older version of SDRSharp. All my recent perfect decodes were recorded in SDRSharp. As I said before, it has superior DC cancellation.
• SDRSharp also has an 8-bit recording option

Some math...

Last night I was limited to 1 minute 12 seconds on a 2-GiB RAM drive. I was using 7 MHz bandwidth at 16 bits per sample. I used a stopwatch app on my phone to time how long it takes Windows to move this onto my external drive, since Windows' reported transfer rate is inaccurate. It took me 102 seconds to transfer. The files were about 1.92 GiB each.

1.92 GiB divided by 102 = about 19 MiB / sec average transfer rate. At 16-bit sampling, this would only allow you to record 4.75 MHz of spectrum, which is obviously not enough for TV. When I calculated this, I saw that if I could switch to 8-bit recordings, my hard drives would be sufficiently fast.

Recall in my last post that I said it would take 16 MiB/sec to do an 8 MHz @ 8 bit recording, which is below the 19 MiB/sec I just calculated. But now that I'm using 7 MHz, it would only take 14 MiB/sec, well below the limit.

But 8-bit recordings didn't work last time. Then it dawned on me: if GNURadio can't natively decode 8-bit ATSC recordings, why not trick it by taking an 8-bit recording and upsampling to 16 bits?

After some more tests I noticed SDRSharp is limited to 2 GiB, which is about 2 minutes. This is a hard limit of the official WAV format, and SDRSharp doesn't support RAW or WAV64. If anyone finds a fix for this please comment below.

How I did it:

Today I took an 8-bit recording in SDRSharp. It was a WAV file, not RAW, so I was able to open it in Audacity and merely save it as a 16-bit WAV. I knew Audacity likes to dither, but that doesn't seem to happen on upsampling. I verified the 16-bit WAV I had just generated by opening it in SDRSharp and, sure enough, it looked like a great signal, so I knew Audacity hadn't messed it up.

Below: live waterfall of WACH Fox.

Below: 8-to-16 bit converted WAV recording

As you can see, there's very little difference.

I fed the 16-bit converted WAV through GNURadio as usual, and waited expectantly. For all I knew, I could get the dreaded "!!! atsc_fs_checker" errors, or simply an empty TS file. Neither happened (see below) but I still had no proof it would play.

To my surprise it actually played! (although it did skip in 2 spots)
(pictured below)

I think the skips were caused by minor sample dropping, so it should work perfectly with a very fast and defragmented hard drive. Of course, PrimoCache would still be a great idea to smooth out any hard drive latency, such as what occurs during recalibration, but only as long as the IQ data rate doesn't exceed your drive's average write speed.

I did more tests on Sunday, but with a RAM drive this time to avoid latency. 8-bit recording is indeed very reliable, not to mention space-efficient, as I was able to fit 2 minutes 33 seconds into my 2-GiB limit. I had been concerned about the reduced dynamic range afforded by 8-bit sampling, but my fears seem to be unfounded.

It would be amazing to leave this recording, latency smoothed out, to my 8 TB drive, to catch shows I'd miss. Below are two ideas for software fixes that would allow us to set up a full software-based DVR.

1. HDSDR supports unlimited file length but has a minimum bit depth of 16. It should be updated to support 8-bit recording,
2. SDR#, on the other hand, supports 8-bit recording but [its native recorder] is limited to 2 GiB. It should be updated to use a format other than WAV. See new recorder for SDRsharp breaks 2GiB limit. With a certain 3rd-party plugin, we can now take 8-bit recordings of unlimited length in Wave RF64 format using SDRsharp.

Thanksgiving ATSC Tests

Yesterday I did some more ATSC experiments.

(above) Finally... the elusive WZRB ION channel 47 comes in. On the right edge is WACH Fox 57.

(I'm using the older copy of SDRSharp I mentioned in my previous post)

Since hard drive recording is spotty, I copied SDRSharp to a RAM drive, since it only records to its own program directory. SDRSharp kept setting the SDRPlay's PPM correction to a crazy value and changing it kept locking up the computer.

SoftPerfect RAM Disk also kept locking up when I tried to unmount disks. I would turn it off in Task Manager and restart it from the Start menu. That caused it to forget the RAM it had previously taken and I would have to reboot. I eventually switched to IMDisk and kept working.

Adding to my troubles, when I tried to decode it, I accidentally used the old flowgraph from my previous post, the one that expects 8-bit samples. Of course, my TS file stayed empty, and it took me a while to realize the issue. (Pictured below)

Even after fixing this, I wasn't able to decode WZRB, even though the TV had no trouble. However, I did manage to get a long successful recording of WRLK PBS.

Pictured below is WAGT NBC. As I write this, I can't get it to come in on my TV.

I've seen this kind of fading in other stations, most notable PBS. This picture was taken with my SDRPlay hooked to a rooftop Yagi. For some reason, a few cases of this fading can be cured by using indoor rabbit ears and rotating them until it goes away. Strangely, I've actually had far better results on PBS with indoor rabbit ears than an outdoor Yagi. I don't yet know why certain portions of the station would fade before reaching me, considering that it leaves the transmitter as a uniform distribution of power over the whole 6 MHz. I know that ATSC doesn't work for moving receivers, but I'm not moving, so why would only certain sections fade as opposed to others?

In summary, trying to decode ATSC is more of a hit-and-miss process since it's hard to know if a station will be readable. In several situations, software-based solutions perform better than embedded hardware, as is the case with DRM radio reception. But from what I've seen, if a TV can't receive a station, GNURadio certainly won't. Sometimes even if a TV is perfectly OK with a certain station GNURadio still can't decode it. I wonder if they'll update GNURadio's blocks for better results? In theory you should be able to decode just as well (if not better) in software as you can with a hardware decoder, since you have more control over processing.

ATSC Update 2

I'd like to share three more aspects of ATSC recording you might not be aware of.

First.

In previous posts I stressed the need for high-speed recording media when recording ATSC IQ samples. A few days ago I proved myself wrong.

I was using an older copy of SDRSharp, one that still works with the SDRPlay. SDRSharp has superior DC cancellation functionality, which it labels "IQ Correction."

I used the Baseband recording option at 16-bit sampling and managed to record 44 seconds to a fast magnetic hard drive before dropping samples. The IQ file yielded a perfect decode. You can download the final TS here: 183mhz_decoded.ts.7z. (70 MiB)

You'll probably still need a RAM drive for longer recordings, but this opens up a lot of possibilities. Someone just needs to write a program that will "buffer" writes to a hard drive to smooth out latency. Ideally this would work by registering a virtual drive and creating a RAM buffer, and having two separate threads, one to accept the write requests and store them in RAM, and another to write them when the drive is ready. That way, SDR programs never see the delay in the hard drives.

(Update) I've found a paid program that should do this. It's called PrimoCache and the concept is called deferred write. It costs $29.95 USD and comes with lifetime free upgrades.

Second,

You've probably noticed that most digital channels offer an EPG (Electronic Program Guide.)

I learned that if you open a valid decoded TS file in MediaInfo you can retrieve the EPG, among other things, such as channel resolution and bitrate.

Click here to read a TXT file of MediaInfo's output for the TS file I offer above. The EPG's for each channel are at the bottom.

(Update 11/24/2015) Interestingly, there is a Start and End UTC near the top. MediaInfo can apparently look at the timestamps embedded in TV channels to find the start and end time of the recording. Note that the recording's Start and End span 38 seconds, 6 seconds less than the 44 seconds of IQ data I took. This implies that the syncing (and de-syncing) periods take off 3 seconds at both the beginning and end.

Third,

During the experiment under "First," I also tried taking 8-bit IQ samples, which cut bandwidth and storage requirements in half from 32 to 16 MiB/sec. I was not able to decode this IQ file, but maybe someone else will figure out a way.

Of course, one big advantage of the SDRPlay over the RTL dongle is the 12-bit sampling. and the loss of dynamic range incurred by downsampling to 8 bits might hamper decoding no matter what you do.



As always, feel free to comment below.

Software to filter through EiBi schedule

One of the important aspects of shortwave listening is identifying the shortwave station you're listening to. The EiBi schedule is an exhaustive list of stations along with their respective frequencies and transmit times. It is my most-used shortwave listening resource.

When I still used analog radios, I would have to wait until the top of the hour and listen for the name or callsign, hoping there was no interference, then go search in Excel to find the frequency. Now that I have SDR, I can accurately pinpoint the frequency. Needless to say, this makes it much easier to find out what the station is. However, usually multiple stations use the same frequency at the same time, and not all stations transmit 24/7. The EiBi schedule gives times in UTC, but it's time-consuming to convert all those UTC time ranges, plus you might make a mistake if you do it in your head (you know, forgetting to "wrap" when you cross 24 or 0.)

Because of all this, I wrote a small Windows program to go through the EiBi CSV-format schedule and populate a list of what's currently on based on your PC's time. You just give it an EiBi *.CSV file and your offset from UTC and it does the rest. I even added an option to show only DRM stations.

I live on the East Coast so I originally hard-coded a timezone of -5, but I updated it recently for international use. Just enter your timezone to replace -5. The program should remember your CSV file and timezone between sessions.

Download an EiBi schedule (get the CSV file)

Download eibiscanner.zip

Unzip the ZIP to a folder and run eibiscanner.exe.
Note: You must unzip everything or it will not run


Feel free to comment on this software.

Intercepting pager messages

Today I heard data bursts on 155.170 MHz and thought they were P25 or MotoTRBO. When they didn't produce any output in DSD+, I went on sigidwiki, chose VHF, and went through NFM signals until I heard one that sounded similar. It turned out to be a pager signal. I downloaded a program called PDW. It can be downloaded here.

I was then able to decode what turned out to be hospital messages. This form of interception is legal, according to what I read on this RTLSDR Blog post. In the post, they wrote, "In most countries it is perfectly legal to receive these messages, as they are plain text unencrypted, but it is illegal to act on the information received. Please respect your local laws."

I can't legally go into detail, but I will say I was able to read some interesting stuff. By law, you must not act on what you hear or divulge it to someone who wasn't present.

929.612 MHz has far more activity than VHF, at least in my area.

I recommend TV rabbit ears in a window with the adjustable parts vertically oriented. This forms a sort of vertical dipole. I've tried pager interception with a rooftop Yagi connected through a TV RF amp and it doesn't seem as good. It could be the RF amp corrupting any signal over 800 MHz, attenuation in the old cable wiring, or maybe TV Yagis just weren't meant for this frequency. Of course, the 150 MHz paging comes in significantly better with the Yagi, but that's understandable because it's near a VHF TV band, something the Yagi is good at, and the RF amp is designed for this frequency.

Extra tip: although HDSDR beats SDR# hands-down when it comes to CPU usage, SDR# seems to have a better FM demodulator, which means higher decoded message accuracy.

Here's where things get mysterious...

While monitoring 155.170 MHz, I heard a voice signal more than once. It seems people sometimes talk between pager bursts. I've embedded a recording of one such instance.

(Update 11/09/2015) I think this is an example of a voice page. Here is a good link to some info on this. Apparently pagers can receive voice messages.