User Tools

Site Tools


wiki:blog

Leipzig

We have just returned from a short break in Leipzig, formerly the home of Johann Sebastian Bach. It was something of a music pilgrimmage for me, though we found time to do less music 'sights', such as the immense (and immensely impressive) Völkerschlachtdenkmal, a memorial to the dead of the Battle of Leipzig in 1813. I hadn't even realised there was a Battle of Leipzig then, but it turns out to have been the largest battle ever fought by mankind up to that point and sealed the fate of Napoleon, who abdicated shortly thereafter.

Anyway: I have to say I was a bit disappointed in the Bach side of things. There is certainly the Tomaskirche (pictured), where Bach was Director of Music for the last 27 years of his life. We went to a service there, which was moving enough. The church has two organs, JSB's grave, a memorial window to him and an excellent continuing choral tradition. Except the organs are relatively modern and definitely wouldn't have been played by Bach; the church interior was renovated extensively throughout the 20th Century, so that it looks nothing like what Bach would have known; and we're not even quite sure if the contents of the Bach grave are actually of Bach and not some random other gentleman. Bach was actually buried at St. John's, which was destroyed by bombing in WW2, so what were thought to be his remains were moved to the Tomaskirche in 1950 -but the identification of the St. John's grave as Bach's relied on 'folk memories' of him having been buried in a certain location.

Meanwhile, Bach also ran the Nicolaikirche, which was just a hundred yards or so from our hotel. It had its interior completely re-done during the late 18th Century in fantastic neo-classical style -meaning that it too bears no relation to anything Bach would have seen, no matter how splendid it looks today.

So we drove to Eisenach, Bach's birthplace. There, the Bachhaus is the place where Bach grew up until he was about 10. It is full of furniture from the mid-18th Century -except none of it was actually owned by the Bachs. The house, though, is as he would have known it -except for the minor detail revealed by one display photograph, of the morning after an American bombing raid in 1944 -namely, that the building was extensively damaged and thus what you see today is, mostly, a fairly modern repair job.

Back to Leipzig: the Bach Museum opposite the Tomaskirche is well worth an extended visit. I think we spent the best part of three hours in there, which is a bit of a record for us and museums of any sort! At one point, their guided audio tour invites you to look out of a window to a place where the Thomas School that Bach worked in (and composed in) would have been. Why isn't it there now? Because the Mayor of Leipzig decided to demolish it.

Yup. Just like that, in 1902, the place where Bach composed some of the most glorious works of music the world possesses was demolished. They managed to save a door (which they transplanted to the Eisenach Bachhaus museum).

So, sadly, I came to the conclusion that Bach isn't really in Leipzig anymore (unlike, say, Britten in Aldeburgh: go to that part of the Suffolk coast and Britten's presence is still real, tangible and immediate). There are some places he would have wandered, some buildings he would have known -but they mostly don't bear any resemblance to what he would have seen. Some of the most important places he lived and worked in don't exist. Even his grave might not be his.

I came to the conclusion that the only substantial, real monument to Bach these days is his music -and after learning about the Battle of Leipzig (some 60 years after his death), with thousands of Napoleon's troops camping in the town in a cold October, with their tendency to burn anything made of wood or paper to keep warm, I'm rather more surprised than I was that so much of Bach's music has survived the predations of time!

It was a flying visit, of course (three days), so no time to visit Weimar, Arnstadt, Mühlhausen, Köthen or the many other places in that corner of Germany that Bach would have known and visited or lived in. Perhaps the presence of Bach is better felt in those other places? Future visits may tell me!

I nevertheless enjoyed Leipzig a lot. It just wasn't the 'Bach lives Here' memorial I'd hoped to discover there -which probably says more about the level of my expectations than anything else.

2019/02/24 11:27 · dizwell

Determining a CD Drive's Read Offset in Linux

CDs are becoming a bit of an historical artefact these days, what with the proliferation of music streaming services! Even my favourite music purveyor now offers much of its catalogue in the form of downloadable digital files, rather than physical CDs.

Nevertheless, you will still come across music which is only available in physical form -and, of course, you may have hundreds or even thousands of physical CDs from your past which need to be converted to digital files at some point (a process called 'cd ripping')

When you come to do that ripping, though, it's important that the computer CD (or DVD) drive you use to do the deed is reading the physical CD accurately… and that's harder to do than it perhaps ought to be!

For starters, an audio CD doesn't allow random access: you have to start somewhere and read sequentially on from that point. Which means it's rather important to know precisely what that starting point is… and most CD players can't reliably tell you that! Nearly every CD player mis-places its reading head by a number of audio samples: so your audio program says “read audio sample 1000” (for example), and the drive will actually return data from sample 1000 + or - something. The “something” tends to vary by manufacturer and/or by model. In other words, it doesn't vary depending on what CD you're playing but by what device is doing the playing. Once you have determined that your player adds 6 samples when playing CD x, you can be fairly certain it will add 6 samples when playing CDs y, z and a, too. Most NEC CD drives will, for example, add 48 to the sample number it's supposed to be reading. This difference between where a drive is meant to be reading and where it's actually reading is called the 'Drive Offset' -and it's important to know it if you genuinely want perfect duplication of CDs.

There are software tools which can work out what the drive offset is for a particular CD (or DVD) drive; unfortunately, they tend to be Windows-based and require tricks to get them working on Linux (Exact Audio Copy, for example, can tell you your drive's offset, but only runs on Linux via Wine, which I dislike using on my desktops unless I really have to). Whipper is a native Linux program that can also determine your drive's offset -but it's difficult to install on OpenSuse 15 (my distro of choice).

So, instead, I've consulted AccurateRip, which is a publicly-accessible list of known CD and DVD drive models and the known drive offsets for each. If you know the model of your drive, you can consult that list and work out what your offset is.

Unfortunately, that's all a bit manual for my tastes! So, I've automated it somewhat :-)

If you create a blank document (say, testcd.sh) and paste this set of commands in to it:

#!/bin/bash
clear
read -p "Which CD/DVD drive should I check? : " DEVICE
if [ $(lsscsi | egrep -i cd/dvd | egrep -i $DEVICE | wc -l) -eq "0" ]; then
  echo "That's not known as a CD or DVD device!"
  exit 1
else 
  wget --quiet https://www.dizwell.com/lib/exe/fetch.php?media=docs:cddriveoffsets.csv -O /tmp/cddriveoffsets.csv
  DEVICENAME=$(lsscsi | egrep -i cd/dvd | egrep -i $DEVICE | awk '{print $5}')
  DRIVE_OFFSET=$(grep $DEVICENAME /tmp/cddriveoffsets.csv | awk -F "|" '{print $2}')
  rm /tmp/cddriveoffsets.csv
  if [ ! -z "$DRIVE_OFFSET" ]; then
    echo "Drive Offset for $DEVICENAME ($DEVICE) is: " $DRIVE_OFFSET
  else 
    echo "Your drive's model name/number is not in the AccurateRip database."
  fi
  exit 0
fi

…you can then run the script with the command:

sh testcd.sh

You'll see output such as this:

Which CD/DVD drive should I check? : /dev/sr1
Drive Offset for DH-16AES (/dev/sr1) is:  6

When prompted, you type in the device name of the CD or DVD drive you want checked (if you type in a device identifier for a normal hard disk or anything else which isn't explicitly known to be a CD- or DVD-ROM drive, you'll be told as the script aborts).

Usually, that would be “/dev/sr0”, but because my PC has two CD drives and I only use the second of them for ripping, I've specified “/dev/sr1” (without the quotes, though). Armed with that information, the script then checks the model name/number for that device using the lsscsi command and then uses plain old grep to check if that model number is found within the AccurateRip database. If it is, it returns the known offset for that model; otherwise, it tells you your drive is not known by AccurateRip.

With only a limited number of CD drives in the house to test it on, I can't be sure whether lsscsi will always return useful and usable model numbers, but it works for me with my other CD drive, for example:

Which CD/DVD drive should I check? :/dev/sr0 
Drive Offset for DS-8D9SH (/dev/sr0) is:  6

…which is a different model number but happens to have the same offset as my main drive. (Note, a manual check of the AccurateRip database confirms the script isn't just outputting the number 6 no matter what input is thrown at it!!)

Once you know your drive's offset, you can use it when ripping a CD to compensate for your drive's inability to read precisely where it's told to read! For example, a basic command to rip track 1 from an audio CD using cdparanoia would be:

cdparanoia -B -z=40 -d /dev/sr1 "1"

Which means “please rip track 1 from device /dev/sr1, and if you encounter a scratch or other defect that makes reading the audio data tricky, take a maximum of 40 goes to do so before giving up”.

Except that command will rip (in my case) 6 fewer audio samples than it should, because my drive is known to mis-read starting points by 6 samples -so my rip won't be accurate.

Let's say I run it anyway, however:

[email protected]:~> cd Music
[email protected]:~/Music> cdparanoia -B -z=40 -d /dev/sr1 "1"
cdparanoia III release 10.2 (September 11, 2008)

Ripping from sector       0 (track  1 [0:00.00])
          to sector   22874 (track  1 [5:04.74])

outputting to track01.cdda.wav

 (== PROGRESS == [                              | 022874 00 ] == :^D * ==)   

Done.

Cdparanoia itself is happy: the “:^D” smiley face at the end there indicates it believes it's just performed a perfect rip. Now let's save that output file with a new name, and then re-rip the same track, but this time telling cdparanoia about our drive's known offset:

[email protected]:~/Music> mv track01.cdda.wav run1.wav
[email protected]:~/Music> cdparanoia -B -z=40 -d /dev/sr1 "1" -O 6
cdparanoia III release 10.2 (September 11, 2008)

Ripping from sector       0 (track  1 [0:00.00])
          to sector   22874 (track  1 [5:04.74])

outputting to track01.cdda.wav

 (== PROGRESS == [                              | 022874 00 ] == :^D * ==)   

Done.

I therefore now have two different audio files in my Music directory:

I can assure you, though obviously you'll have to take my word for it, that both files play just fine and sound identical. But are they, in fact digitally identical?

[email protected]:~/Music> ffmpeg -i run1.wav -map 0:a -f md5 - 2>/dev/null
MD5=5a6c91d7dd0b129da033fc8fc61d5ec7

[email protected]:~/Music> ffmpeg -i track01.cdda.wav -map 0:a -f md5 - 2>/dev/null
MD5=7e6e92ffff034609b5d691c82f7e5ce7

That ffmpeg command calculates an MD5 hash of the audio data from the named file. If the audio data were exactly the same in both files, the MD5 hashes for them would be identical. But you can see that the hashes are different. So in this case, sounds can be deceptive: the files sound the same, but aren't actually the same, and it's the file ripped with the offset parameter that is technically the more accurate replica of the original.

2019/02/05 17:32 · dizwell

ZFS Troubles

I have long been convinced that if you care about your 'digital assets' (such as 1TB+ of flac music files or multi-terabytes of family photographs), you need to store them on a file system which has in-built 'bit-rot detection' capabilities. Basically, this means a file system which checksums your data on read and compares the result to checksums computed on write: any difference means some data has changed, and the file system should have data redundancy capabilities that allows it, at that point, to go and retrieve known-good versions of the data from its redundancy store and replace the changed data.

Windows does not have such a file system, unless you count **ReFS**, which was introduced in about 2012… and which was removed from Home and Pro versions of Windows 10 in 2017. (That is to say, you can't create new ReFS volumes in today's Windows 10, though it can still read old ReFS volumes created in earlier versions). It is primarily for this reason that I believe Windows is inappropriate to use as an operating system for the bulk storage of important digital assets.

Linux, however, has at least 2 bit-rot resistant file systems: Btrfs and ZFS. They are not without their issues, however! For starters, Btrfs was originally developed by Oracle (which rings alarm bells for many!) and has had a long gestation (since 2007) before being generally regarded as stable. An unstable file system is definitely not something you want to trust your digital assets to! If it's now regarded as mostly stable, it nevertheless has to live with the fact that whilst Suse Enterprise Server made it their default file system in 2015, Red Hat have declared in 2017 that they will not be supporting it in future. I think the 'general mood' in the Linux world, if you can ever say such a thing exists, is that betting on Btrfs in the long-term is not a great idea.

Which leaves ZFS -and indeed, I've used ZFS exclusively for my bulk storage for many years, with a variety of Linux distros, ranging from Fedora to CentOS, Manjaro, Arch and OpenSuse. It's generally not a great idea to use it with a bleeding edge distro like Fedora, since kernel changes between versions can render the ZFS kernel modules unloadable. You're then dependent on the good folk at ZFS on Linux to catch up and release a newer version of their modules before it works once more… and until they do, your data is inaccessible, which isn't much use to anyone! But, stick to a slower-release Linux, such as CentOS or Ubuntu LTS version, and ZFS works reliably and well. /wiki/OpenIndiana|OpenIndiana]]. That's basically a version of open source Solaris, so is genuinely Unix-y and, as a clone of Solaris, supports the Solaris version of ZFS natively. Except… except… Politics!

ZFS was originally developed by Sun (which was, in due course, bought out by Oracle… are your bells ringing yet?!). Sun did open-source it, making it available as part of OpenSolaris in about 2008. However, Sun licensed it with the “CDDL” open source license, which is incompatible with the GPL license used by Linux kernel developers. They apparently did this deliberately to stop ZFS being made part of Linux, with which they were competing at the time. ZFS cannot therefore be part of the mainstream Linux kernel development process… and that fact has now come to bite ZFS hard.

The short version of the unfolding ZFS on Linux bad news story is that ZFS on Linux relies on some functionality exposed by the kernel via two long-deprecated functions. Preparing for the imminent release of the new version 5.0 Linux kernel, the kernel developers finally (after more than 10 years) removed those two functions. That breaks ZFS at a stroke and means that the ZFS on Linux developers now need to code workarounds -which, quite possibly, won't perform as well as the original code that uses the now-removed functions.

Performance isn't exactly an issue when you're using ZFS as a bulk storage file system only: it's not as if you are writing to it hundreds of thousands of times a day, after all. But the fact that the issue arises at all illustrates the danger of being outside the kernel main line: the kernel developers can do things which break your code without having any responsibility for the breakage (and thus no responsibility to fix it). If your code is non-main-line, the breakage is something you have to fix, after it's occurred.

For the moment, none of this really concerns me: I'm running ZFS under CentOS 7, which is still using a version 3 kernel (and will do so until its end of life in 2024); And in five years' time, I think we'll know the state of play of ZFS on linux 5.x much better than we do now.

In the meantime, knowing that my CentOS 7 boxes would one day be end-of-life, I had anyway been considering a switch to FreeBSD, which has had integrated ZFS support out-of-the-box for years. Funnily enough, the FreeBSD developers recently decided to switch their ZFS code base to the ZFS-on-Linux one. This won't break ZFS, though, as the FreeBSD kernel is not the Linux kernel: whilst the Linux 5.x kernel changes break ZFS-on-Linux, it will continue working fine on the FreeBSD kernel.

The other thing I already do, incidentally, is run a server on OpenIndiana. That's basically a version of open source Solaris, so is genuinely Unix-y and, as a clone of Solaris, supports the Solaris version of ZFS natively. Again, as a bulk data server, I don't actually have to interact with that server's operating system very much (basically to check the Zpools are healthy once in a while!), so the fact that it behaves and operates rather differently to Linux isn't a major problem for me.

Anyway: the moral of this story is perhaps to be a little wary of ZFS on Linux. Maybe it's also to be prepared know how to use more than one operating system. And, possible, to shop around for the very best protection for those precious digital assets you need to keep safe: don't complacently settle on one O/S, file system or storage solution!

2019/02/03 11:27 · dizwell

Pro-tip...

Here's a pro-tip: If you write an article about how to set up a mail server and do so with a test-bed server specifically intended for testing and mucking about, remember not to wipe your actual, real, 'production' mail server by accident when you think it's time to wipe the test-bed server.

Admittedly, one IP address can look much like another at 7AM, but it's still no excuse!

On the up-side, at least I can attest that my article is pretty accurate and functional: I had a rebuilt server available in less than 20 minutes by following my own advice there!

Unfortunately, I lost all my mail archives, which is a bit of a bummer. Would pay to remember to set up a backup routine before relying on a server, instead of adding it to the list of 'things to get around to'. :-(

Of course, you only stuff up like that once in a lifetime, yes?!

2019/02/01 15:57 · dizwell

Free At Last!

I remember those ancient days past when one's email account was tied to whoever your ISP happened to be. Hence, I think my first ever personal email account was something like [email protected] Which unfortunately meant that if you changed your ISP, you had to change your email address: that wasn't going to work well in the long run!

So, like everyone else, I swiftly progressed to “immortal” email addresses with the likes of Yahoo!, Hotmail, Gmail and so on. Of course, we all knew that if the product was free you are the product, but the convenience of a web-based email account that never varied no matter who connected you to the Internet was not to be sniffed at.

Except, of course, these days we know rather more about the privacy implications of handing all your data over to mega-corporations like Google and Microsoft, especially thanks to the likes of Edward Snowden.

For the past five years or so, I've been wanting to ditch the likes of Gmail and Outlook (the Yahoo! account having stopped being relevant a decade or more ago!). But setting up your own email infrastructure is not something for the faint-hearted. There are a lot of moving parts; much administrative maintenance to do; complex software that is a real bummer to configure correctly …and on and on.

However, I have now finally kicked the Megacorporation habit, thanks to a cheap virtual server hosted in Amsterdam and the free software available from iRedMail -which, astonishingly enough- practically configures itself!!

It still isn't something I'd recommend you do on a whim, but I am now reasonably comfortable with it… and have accordingly thrown an article together describing the entire process. It's a long read, but worth it for the sense of freedom from snooping it gives me. Naturally, if the NSA want to know anything about me, they'll find it out one way or another. But at least a mere corporation can't have it for free anymore!

2019/01/31 18:07 · dizwell

Older entries >>

wiki/blog.txt · Last modified: 2018/12/12 14:06 by dizwell