7-zip optimization: sorting by file type

 Recently I was compressing the assets I scraped from a now-deleted Discord server and was able to save almost 600 MiB by having 7-zip sort by file type (the default behavior is to sort by filename) in addition to a few other options. This lets 7-zip take advantage of similarities between files of the same type.

The original data was 6,661,786,104 bytes, with the default compression it was 6,041,019,462 bytes (90.682%), and with "sort by file type" enabled, it was 5,427,406,392 bytes (81.471%). In this case, the optional parameters reduced the size of the output by 613,613,070 bytes (585.19 MiB).

This article was updated on February 11, 2021 because one of the parameters, "tr=on", doesn't exist and causes an error. I changed a few other parameters as well and updated the results in the previous paragraph because the new ones saved an extra 2.15 MiB.

Delivering GPS updates with TV datacasting

In 2011 my family got a Garmin GPS and a few years later I found it fascinating when I heard that it could get traffic data from FM radio. It occurred to me more recently that map updates could be sent over a broadcast with more capacity, such as a TV channel.

As you may know, my UTSC standard supports file transfers. Each packet is only 125,000 bytes, so you can't transfer much in each one, but it's possible to send large files as a split archive.

I set up an experiment to see how quickly I could transfer a full GPS map of South Carolina. Since I'm near the Georgia border I originally included a Georgia map but later decided to just do one state to make it faster.

I downloaded a copy of the OpenStreetMap data for South Carolina. I signed the map with my PGP key because I figured that automatic map updates should be signed and that unsigned ones, or those without a trusted signature, should require user intervention. I added a license file to the folder and then used my UTSC packet muxer to create packets that include a NoCopyrightSounds music compilation, an image that says this is a GPS map update and gives attribution for the map and music, and file transfers consisting of a split 7-Zip archive.

These are the 3 files I transmitted.

My packet muxer reads a JSON file with a list of content and an optional datacasting field. If datacasting is enabled, it finds the content with the highest bitrate and uses 7-Zip to compress the contents of a folder specified in the JSON. It instructs 7-Zip to split the archive into parts just small enough to fit in the least space (packets with the highest content bitrate).

The total content length was 23h23m00s and the file transfer was 104.9 MiB. The packet muxer reported that it had included the files 60 times. By coincidence, each full transfer took 1403 seconds or 23m23s. That means the map can be transmitted 61 times every 24 hours or about once every 23½ minutes.

After the muxer finished, I used my open-source UTSC transmitter (GitHub link) to transmit the output file. I used UTSC channel 2 (center freq 903.265625 MHz). Here's what it looks like in HDSDR.

Unfortunately, I don't know how to demodulate QPSK at the moment so I can't test the signal but I did some verification of the muxer output with a hex editor before I used it.

Multithreaded RRC for Raspberry Pi

In my last post I explained that I couldn't transmit continuous QPSK from my Raspberry Pi 3B because the single-threaded RRC function couldn't keep up with the transmitter. In other words the transmitter was transmitting and consuming the data buffer faster than the RRC function could supply it. This led to a situation in which there were short data bursts separated by slightly longer empty sections.

Tonight I resolved that. I carefully examined the RRC function until I understood how it worked and was sure that nothing depended on previous results, then split it into 4 equal parts, since the Pi has a quad-core CPU. The code uses pthreads in C to run a method that does 1/4 of the work. It populates the start and end indices for each thread, starts all 4, and waits for them to finish. The Pi can now transmit smooth QPSK with just a few sparse dropouts. I was using composite video for this test so it might be fast enough with HDMI which lets the system clocks run faster.

May 2020 Hackerthon

Over the weekend, the owner of a Discord server called The Hive Mind ran a "hackerthon" during which members were supposed to build something and show pictures on Sunday. Since I was already working on a UTSC broadcast utility for LimeSDR's, I decided to finish it if possible and have something to enter if that was all I could do, and then see if I could transmit the same thing from a Raspberry Pi running on a power bank. If you're not familiar with UTSC, it's a TV standard I created in 2017 as a highly reliable and license-free alternative to ATSC.

Before I begin the description, here are the pictures I submitted when the event ended.

One of the first things I did during the hackerthon was to test the range of the LimeSDR Mini at its highest power. I used DATV Express to transmit QPSK at 625 kilosymbols/sec to make it as much like UTSC as possible and walked around outside with a laptop, SDRplay RSP1, and TV rabbit ears. The setup was on the second floor, about 25 feet above ground level, on a table near a window. The transmitting antenna was an adjustable TV dipole with ladder line oriented vertically. I walked until I was about 955 feet away (291 meters) and the signal was roughly 21 dB above the noise.

After I was done, I worked on a QPSK transmitter in Visual Studio. I copied the modulator from Charles Brain's (G4GUO) DATV Express code, which is public domain. I first tried this about a year ago and it worked but stuttered badly. I copied the code from an older version that had an issue with accessing the webcam when I compiled it so I assumed the modulator code was broken as well. It took a while for G4GUO to get back to me the second time I contacted him so I paused the project until a few days ago. That's when I tried again and succeeded. I think the issue was due to using a different buffer size the first time. I made sure to notify him via Twitter that I didn't need his help anymore.

Now that I could transmit QPSK, I wanted to see if I could use my LimeSDR Mini to do it from a Raspberry Pi. I have a Raspberry Pi 3B V1.2 and it was easy to set up the SDR and C++ environments. Since it uses Linux (specifically Raspbian), I didn't need any drivers. All I had to do was download the LimeSuite repo from GitHub and build and install it. When it was done, it left some include and library files in a folder that was easily accessible. I found them and used them with g++ to compile an example I found.

Once I checked that the API was accessible from C++ on the Pi, I copied my code from Visual Studio and removed everything specific to Windows. When I was done, it was able to transmit but had the stuttering issue. I spent hours trying different buffer sizes and noticed something odd: with a small buffer it stuttered rapidly, but with a large one it would transmit smoothly for a few seconds, nothing for slightly longer, and repeat. I tried transmitting pure noise and it didn't stutter so I decided to profile the code. What I found was that the RRC (root-raised cosine) method was too slow on the Pi's CPU. The large buffer size was letting it build up a large array of samples to transmit but the RRC method couldn't keep up, causing gaps longer than what was being transmitted. I decided to create a dummy signal generator instead. This would transmit QPSK with the same bandwidth as UTSC but with no real data. This was done by filling the data buffer with random bytes and using a bool variable called rrcRanOnce. I ran the RRC method once and set the variable to true, and then I had an "if" block that kept it from running again after that. This let me transmit the same RRC-filtered samples in a loop, producing a smooth QPSK signal on the Pi.

On Windows, my code originally had a GUI but I changed to a command-line project because I wanted to print debug information. I added methods to create UTSC packets and do the interleaving and de-interleaving. I wrote the output to a file and verified it in a hex editor. I also added command-line switches so I could use it outside of Visual Studio.

My transmitter is called UTSCTransmitterCli.exe and takes arguments such as an input file and the channel to use. UTSC is meant for the 902-928 MHz band. Each signal takes up about 850 kHz, so there's space for 30 channels. Here's the current UTSC air interface specification.

Intended band: 902-928 MHz
Bandwidth: 843.75 kHz
Modulation: QPSK (or π/4 QPSK)
Symbol rate: 625 kilosymbols/sec
Rolloff: 0.35
Total data rate: 1.25 megabits/sec
FEC: LDPC, 4/5 (250 kilobits/sec)
Usable data rate: 1 megabit/sec

My transmitter uses π/4 QPSK. To achieve this, I duplicated the QPSK symbol array in my code and rotated each one by π/4 radians (45 degrees). Then, for each bit pair, the code checks if the index is odd or even and uses a ternary operator to choose the array that the symbol comes from.

As I said, my app takes an input file and transmits it without any processing. I have an option to create a file with UTSC packets but you can transmit anything you want. If the file is highly random, like if it's compressed, then the signal will be smooth but if not, then there will be patterns based on the content. Here are some examples.

A 7-zip file

A WAV file with music

A Visual Studio 2013 ISO image

Having visible patterns isn't desirable because they could violate power spectral density requirements. Here's what a basic UTSC signal without video would look like without any padding or interleaving.

The empty timeslots with spikes are the empty space for video that is zeroed out. Here's what it looks like when the empty space is filled with random bytes.

Notice the timeslots with peaks. That's from the WAV audio (8 bits, 44.1 kHz). With this method you can see how much of the packet is taken up by sound.

In this case the peaks aren't that bad but even if we were operating within the rules for PSD, there's another problem. If you live in the US then you probably have smart energy meters transmitting in this band. Those can briefly interfere with the signal. If we assume that a burst lasts 20 ms, then it could corrupt 25,000 bits (0.02 seconds * 1250000 bits/second) or 3125 bytes. This could mean the difference between a visible image or blocky colorful garbage, or it could cause bursts of noise in the audio.

To solve this, UTSC uses an interleaver. This is a scrambler that randomly and uniformly rearranges the data bit by bit at the transmitter and restores it at the receiver. If a burst of noise damages the scrambled data, the damage will be spread very evenly over the packet after it's unscrambled, which is not only easier to fix with error correction, but also produces audio that sounds better than bursts of noise if the error correction fails.

Here is the same broadcast from the last picture but with the interleaver enabled.

As I said earlier, the interleaver scrambles bit-by-bit. For example, bit 1633 in a plain packet would end up as bit 32 in an interleaved one, bit 952430 would become bit 33, and so on. Bits 0-31 inclusive are taken by the "UTSC" sync header. I generated a lookup table for this and it's defined in interleaver.h in my project.

Now that I've explained how this works, here's the full UTSC broadcast sequence.

(audio, video, EPG, files, etc.)->[packet muxer]->[FEC generator]->[interleaver]->[transmitter]

I don't have a FEC generator yet so I use random bytes as a placeholder.

Conclusion

I want it to be as easy as possible for people to get started with UTSC so I'm starting what I call the UTSC Ecosystem Project. It's a collection of guides and open-source programs for setting up a station. The goal is for anyone to be able to start with a PC or Raspberry Pi and a LimeSDR and be broadcasting in about 15 minutes instead of needing a week of free time and a PhD in Linux as is the case for too many open source projects.

MP3 player bug (RCA M6204-B)

Tonight I tried to put music on my RCA M6204-B MP3 player but when I tried to play it, it said the music section was empty.

TL;DR If your RCA MP3 player says the music section is empty even though you know it's not, make sure the MUSIC folder has at least 14 audio files. If that fails, split your music into at least 14 subfolders.

Full story:

This issue didn't make sense but I thought maybe I needed to use RCA easyRip. After installing that I had the same problem. I tried putting music directly in the MUSIC folder and a subfolder in MUSIC, making sure it was formatted as FAT32, formatting it in Windows and from its Settings menu, and trying MP3's from various sources. Nothing worked. I found other people online with this issue but most of the results were on fixya.com where they suggested calling a support number.

Fortunately, before clearing my player I had saved everything to a 7-zip file. I deleted everything and extracted the file onto the player and it worked. I thought there was some kind of database that the official RCA app populates when it transfers music, but the manual says you can copy music in Windows Explorer so that didn't make sense, but it was still worth trying. RCA easyRip was able to copy the music and list it when I restarted the program, so it should have worked but it still didn't.

Finally I copied everything from the 7-zip file onto the player again and decided to delete some of the music folders until the player said it was empty. I eventually discovered that the player requires the MUSIC folder to have at least 14 files, or folders with files, before it will recognize any music.

Purse.io review

Recently I became interested in growing grain so I decided to buy 3 types of grain seed on Amazon and try them out. Since I didn't have a credit or debit card at the time I placed the order, I used a platform called Purse.io to buy from Amazon using cryptocurrency. While most people think of Bitcoin when they hear "cryptocurrency", I actually used Bitcoin Cash (BCH) because it has lower fees.

I decided that I wanted wheat, barley, and rye seeds so I found them on Amazon and copy-pasted their URL's into Purse.io. The checkout process was quick and easy. I kept the discount slider at the lowest setting of 5% so it would be shipped immediately. Then I chose Bitcoin Cash from a drop-down menu and was given a QR code. I used the Electron Cash desktop wallet and scanned the QR code with a webcam. This populated the "address" and "amount" fields in the wallet, so all I had to do was quickly verify the info and click Send. Bitcoin Cash is designed to finalize payments in seconds and the website responded quickly once I sent the money.

It cost 98.2089 mBCH (milli-Bitcoin Cash) or 0.0982089 BCH if you don't like the mBCH unit. Bitcoin Cash was about $289 when I placed the order so the equivalent price in USD was $28.57.

I placed the order on Tuesday, October 29, 2019. Purse.io ordered the products from Amazon on Friday, November 1 and the package arrived on Tuesday, November 5.

Some people are concerned that Purse.io is a scam, so I decided to document my success.

Avoiding Windows 10 in 2019

There's a lot of hype surrounding Windows 10 right now and I'm not sure why. I notice several problems and no real benefits. Everyone is saying to just upgrade, but I don't feel like upgrading to something that is hugely inefficient and spies on me. If anyone has a good reason to upgrade, let me know in the comments.

If Windows versions were cars:

Why do Windows and Office get bigger and slower with each release? Windows XP came on a CD and a typical installation could fit in a couple of gigabytes, let's say just under 2 GiB. Windows Vista and up come on DVD's and a typical installation takes about 15 GiB. Why is that? I use Windows 7 and it works very well for me but I don't know what it could possibly have that would take up an extra 13 GiB compared to Windows XP.

Upgrading past Windows 8.1 just doesn't make sense in my opinion. More importantly, we all know Windows 10 sends usage data to Microsoft. How much is it sending? We probably won't know until there's a data breach and it ends up in the news.

Another thing to ask is, why was Windows 10 free? And why did the free offer last so long past the deadline? I was able to activate Windows 10 using my Windows 7 COA in April 2017, long after that was supposed to stop working. An article on howtogeek.com says this still works as of January 5, 2018. They write, "it’s also possible Microsoft will look the other way and keep this trick around to encourage more Windows 10 upgrades for a long time to come."

At first I thought it was just a way to get free beta testers, but I suspect it also has to do with the data collection. If something is free then you're (usually) the product. I hear that people are quitting Facebook because of privacy and data collection issues, but what if using Windows 10 is just as bad as using Facebook, or even worse? Windows 10 could provide far more data than any social network since it can potentially watch everything you do.

The biggest problem with Windows 10 may not even be the data collection, but instead the forced updates. It almost feels like Microsoft owns any computer running Windows 10 and they're just letting you use it. I heard that they're also moving to a new driver signing system where you have to pay to get kernel-mode drivers signed before Windows 10 can use them. This reminds me of how Blackberry phones won't run native C++ apps unless they're signed by a Blackberry-issued certificate, even for development. Thankfully Blackberry's signing process is free, but it's still not cool to have to get permission to run things on your own devices. Windows should not be as locked-down as iPhone and Blackberry.

Recently a family member bought a laptop from Wal-Mart but after I explained how bad Windows 10 is, they took my suggestion to migrate their old laptop's Windows 8.1 license. After I installed Windows 8.1, I found out that many of the new laptop's devices only had drivers for Windows 10. It seemed odd since Windows 8.1 isn't that old. I've never seen a hardware manufacturer support only one version of Windows, but that was how it was for this laptop.

I tried everything, including unzipping the installers and attempting to manually install the drivers, but nothing worked. It turns out that Intel, the company that made the laptop's CPU, graphics, and Wifi, doesn't want to support anything below Windows 10. I can almost understand with the GPU and Wifi, but since when is an x86 CPU intended for only one version of Windows? In the end, I was asked to just install Linux. I set up Ubuntu and now everything works like it should, without the data collection.

I've decided to keep using older Windows versions as long as possible. I use Windows 7 on my desktop and XP on my laptop. And yes, my laptop is online. I also dual-boot my desktop with Windows XP so I can use an old analog video capture card.

I really like Windows XP because of its simple interface and low requirements. Everyone seems to freak out when they hear of people keeping an XP system on the Internet. I know about viruses like WannaCry, but as I understand it, they depend on someone to run them before than can infect a LAN. I'm behind a firewall and I'm always careful about what I download so I don't see a problem yet.

I'm not saying Windows XP is for everyone. In fact, I only recommend it if you're a power user who can secure it properly. Even then, it's probably better to use Windows 7 since it has better software support.

I can see that Microsoft, Intel, and probably others are planning to push everyone they can onto Windows 10. To prepare, I've archived my Windows ISO's on an external hard drive and downloaded a large collection of offline update installers. In fact, as I write this, I'm downloading all the updates for Windows Server 2003, XP, 7, 8, and 8.1, and Office 2003 and 2007.

Here are some OS options if, like me, you want to avoid Windows 10.

Windows 7

If possible, I'd say to stay with Windows 7 as long as possible unless you're using a tablet, in which case I think Windows 8.1 is best.

You should acquire and archive Windows 7 installation media and updates now, just in case Microsoft tries to kill this version later. After I installed Windows 7 and activated it, I used DriveImage XML to make a drive image so I can revert to a fresh installation whenever I need to.

I can see Windows 7 being supported by software and most hardware for a long time so this should be a good option for quite a while.

Ubuntu or Lubuntu

There are many apps that run on Linux such as Chrome, Opera, LibreOffice, 4K Video Downloader, and Veracrypt, so switching to Linux might work for you. It might be your only other option if you get a new computer.

Windows 8.1 (best for tablets, not PC's)

I know many people disagree, but I think Windows 8.1 is great for tablets. Unfortunately, the Windows 8 app store seems to be losing apps rapidly. If you're interested, you should acquire and archive Windows 8 installation media.

Windows XP

I use this every day on my laptop and it works well for me. It has decent software and hardware support but be prepared for some recent software to not work. I'll probably publish and maintain a list of XP-compatible software and hardware in the future.

Windows XP x64 Edition (not really recommended)

Recently I heard about Windows XP x64 Edition. It sounded interesting so I exercised my Windows 7 Pro downgrade rights (pictured below) to install a copy in a VM and on a laptop. (Don't worry, I had enough spare Windows 7 licenses to cover this)

Above: an excerpt from the Windows 7 Pro license agreement.

It looked and acted just like regular 32-bit Windows XP, and I was surprised at how many current 64-bit apps run on it. The downside is that it requires special drivers. I've observed some drivers to work in 32-bit Windows 7 and XP, but 64-bit Windows 7 and up cannot share drivers with 64-bit Windows XP. It turns out that 64-bit Windows XP uses a different driver model, one that it shares with Windows Server 2003 x64. Because 64-bit Windows XP is so rare, it's hard to find drivers for it but I think searching for drivers for Windows Server 2003 x64 might help. I was able to find a 64-bit XP driver for my GTX 750 Ti, but some Sandisk flash drives don't work.

As for compatibility, I can confirm that the following items work in 64-bit Windows XP:

• Seagate's 5 TB GPT-formatted external hard drive
• K-Lite Codec Pack, including Media Player Classic 64-bit
• UltraSearch x64
• Chromium 44 x64