User Tools

Site Tools


wiki:blog

Determining a CD Drive's Read Offset in Linux

CDs are becoming a bit of an historical artefact these days, what with the proliferation of music streaming services! Even my favourite music purveyor now offers much of its catalogue in the form of downloadable digital files, rather than physical CDs.

Nevertheless, you will still come across music which is only available in physical form -and, of course, you may have hundreds or even thousands of physical CDs from your past which need to be converted to digital files at some point (a process called 'cd ripping')

When you come to do that ripping, though, it's important that the computer CD (or DVD) drive you use to do the deed is reading the physical CD accurately… and that's harder to do than it perhaps ought to be!

For starters, an audio CD doesn't allow random access: you have to start somewhere and read sequentially on from that point. Which means it's rather important to know precisely what that starting point is… and most CD players can't reliably tell you that! Nearly every CD player mis-places its reading head by a number of audio samples: so your audio program says “read audio sample 1000” (for example), and the drive will actually return data from sample 1000 + or - something. The “something” tends to vary by manufacturer and/or by model. In other words, it doesn't vary depending on what CD you're playing but by what device is doing the playing. Once you have determined that your player adds 6 samples when playing CD x, you can be fairly certain it will add 6 samples when playing CDs y, z and a, too. Most NEC CD drives will, for example, add 48 to the sample number it's supposed to be reading. This difference between where a drive is meant to be reading and where it's actually reading is called the 'Drive Offset' -and it's important to know it if you genuinely want perfect duplication of CDs.

There are software tools which can work out what the drive offset is for a particular CD (or DVD) drive; unfortunately, they tend to be Windows-based and require tricks to get them working on Linux (Exact Audio Copy, for example, can tell you your drive's offset, but only runs on Linux via Wine, which I dislike using on my desktops unless I really have to). Whipper is a native Linux program that can also determine your drive's offset -but it's difficult to install on OpenSuse 15 (my distro of choice).

So, instead, I've consulted AccurateRip, which is a publicly-accessible list of known CD and DVD drive models and the known drive offsets for each. If you know the model of your drive, you can consult that list and work out what your offset is.

Unfortunately, that's all a bit manual for my tastes! So, I've automated it a bit :-)

If you create a blank document (say, testcd.sh) and paste this set of commands in to it:

#!/bin/bash
clear
read -p "Which CD/DVD drive should I check? : " DEVICE
if [ $(lsscsi | egrep -i cd/dvd | egrep -i $DEVICE | wc -l) -eq "0" ]; then
  echo "That's not known as a CD or DVD device!"
  exit 1
else 
  wget --quiet https://www.dizwell.com/lib/exe/fetch.php?media=docs:cddriveoffsets.csv -O /tmp/cddriveoffsets.csv
  DEVICENAME=$(lsscsi | egrep -i cd/dvd | egrep -i $DEVICE | awk '{print $5}')
  DRIVE_OFFSET=$(grep $DEVICENAME /tmp/cddriveoffsets.csv | awk -F "|" '{print $2}')
  rm /tmp/cddriveoffsets.csv
  if [ ! -z "$DRIVE_OFFSET" ]; then
    echo "Drive Offset for $DEVICENAME ($DEVICE) is: " $DRIVE_OFFSET
  else 
    echo "Your drive's model name/number is not in the AccurateRip database."
  fi
  exit 0
fi

…you can then run the script with the command:

sh testcd.sh

You'll see output such as this:

Which CD/DVD drive should I check? : /dev/sr1
Drive Offset for DH-16AES (/dev/sr1) is:  6

When prompted, you type in the device name of the CD or DVD drive you want checked (if you type in a device identifier for a normal hard disk or anything else which isn't explicitly known to be a CD- or DVD-ROM drive, you'll be told as the script aborts).

Usually, that would be “/dev/sr0”, but because my PC has two CD drives and I only use the second of them for ripping, I've specified “/dev/sr1” (without the quotes, though). Armed with that information, the script then checks the model name/number for that device using the lsscsi command and then uses plain old grep to check if that model number is found within the AccurateRip database. If it is, it returns the known offset for that model; otherwise, it tells you your drive is not known by AccurateRip.

With only a limited number of CD drives in the house to test it on, I can't be sure whether lsscsi will always return useful and usable model numbers, but it works for me with my other CD drive, for example:

Which CD/DVD drive should I check? :/dev/sr0 
Drive Offset for DS-8D9SH (/dev/sr0) is:  6

…which is a different model number but happens to have the same offset as my main drive. (Note, a manual check of the AccurateRip database confirms the script isn't just outputting the number 6 no matter what input is thrown at it!!)

Once you know your drive's offset, you can use it when ripping a CD to compensate for your drive's inability to read precisely where it's told to read! For example, a basic command to rip track 1 from an audio CD using cdparanoia would be:

cdparanoia -B -z=40 -d /dev/sr1 "1"

Which means “please rip track 1 from device /dev/sr1, and if you encounter a scratch or other defect that makes reading the audio data tricky, take a maximum of 40 goes to do so before giving up”.

Except that command will rip (in my case) 6 fewer audio samples than it should, because my drive is known to mis-read starting points by 6 samples -so my rip won't be accurate.

Let's say I run it anyway, however:

[email protected]:~> cd Music
[email protected]:~/Music> cdparanoia -B -z=40 -d /dev/sr1 "1"
cdparanoia III release 10.2 (September 11, 2008)

Ripping from sector       0 (track  1 [0:00.00])
          to sector   22874 (track  1 [5:04.74])

outputting to track01.cdda.wav

 (== PROGRESS == [                              | 022874 00 ] == :^D * ==)   

Done.

Cdparanoia itself is happy: the “:^D” smiley face at the end there indicates it believes it's just performed a perfect rip. Now let's save that output file with a new name, and then re-rip the same track, but this time telling cdparanoia about our drive's known offset:

[email protected]:~/Music> mv track01.cdda.wav run1.wav
[email protected]:~/Music> cdparanoia -B -z=40 -d /dev/sr1 "1" -O 6
cdparanoia III release 10.2 (September 11, 2008)

Ripping from sector       0 (track  1 [0:00.00])
          to sector   22874 (track  1 [5:04.74])

outputting to track01.cdda.wav

 (== PROGRESS == [                              | 022874 00 ] == :^D * ==)   

Done.

I therefore now have two different audio files in my Music directory:

I can assure you, though obviously you'll have to take my word for it, that both files play just fine and sound identical. But are they, in fact digitally identical?

[email protected]:~/Music> ffmpeg -i run1.wav -map 0:a -f md5 - 2>/dev/null
MD5=5a6c91d7dd0b129da033fc8fc61d5ec7

[email protected]:~/Music> ffmpeg -i track01.cdda.wav -map 0:a -f md5 - 2>/dev/null
MD5=7e6e92ffff034609b5d691c82f7e5ce7

That ffmpeg command calculates an MD5 hash of the audio data from the named file. If the audio data were exactly the same in both files, the MD5 hashes for them would be identical. But you can see that the hashes are different. So in this case, sounds can be deceptive: the files sound the same, but aren't actually the same, and it's the file ripped with the offset parameter that is technically the more accurate replica of the original.

2019/02/05 17:32 · dizwell

ZFS Troubles

I have long been convinced that if you care about your 'digital assets' (such as 1TB+ of flac music files or multi-terabytes of family photographs), you need to store them on a file system which has in-built 'bit-rot detection' capabilities. Basically, this means a file system which checksums your data on read and compares the result to checksums computed on write: any difference means some data has changed, and the file system should have data redundancy capabilities that allows it, at that point, to go and retrieve known-good versions of the data from its redundancy store and replace the changed data.

Windows does not have such a file system, unless you count **ReFS**, which was introduced in about 2012… and which was removed from Home and Pro versions of Windows 10 in 2017. (That is to say, you can't create new ReFS volumes in today's Windows 10, though it can still read old ReFS volumes created in earlier versions). It is primarily for this reason that I believe Windows is inappropriate to use as an operating system for the bulk storage of important digital assets.

Linux, however, has at least 2 bit-rot resistant file systems: Btrfs and ZFS. They are not without their issues, however! For starters, Btrfs was originally developed by Oracle (which rings alarm bells for many!) and has had a long gestation (since 2007) before being generally regarded as stable. An unstable file system is definitely not something you want to trust your digital assets to! If it's now regarded as mostly stable, it nevertheless has to live with the fact that whilst Suse Enterprise Server made it their default file system in 2015, Red Hat have declared in 2017 that they will not be supporting it in future. I think the 'general mood' in the Linux world, if you can ever say such a thing exists, is that betting on Btrfs in the long-term is not a great idea.

Which leaves ZFS -and indeed, I've used ZFS exclusively for my bulk storage for many years, with a variety of Linux distros, ranging from Fedora to CentOS, Manjaro, Arch and OpenSuse. It's generally not a great idea to use it with a bleeding edge distro like Fedora, since kernel changes between versions can render the ZFS kernel modules unloadable. You're then dependent on the good folk at ZFS on Linux to catch up and release a newer version of their modules before it works once more… and until they do, your data is inaccessible, which isn't much use to anyone! But, stick to a slower-release Linux, such as CentOS or Ubuntu LTS version, and ZFS works reliably and well. /wiki/OpenIndiana|OpenIndiana]]. That's basically a version of open source Solaris, so is genuinely Unix-y and, as a clone of Solaris, supports the Solaris version of ZFS natively. Except… except… Politics!

ZFS was originally developed by Sun (which was, in due course, bought out by Oracle… are your bells ringing yet?!). Sun did open-source it, making it available as part of OpenSolaris in about 2008. However, Sun licensed it with the “CDDL” open source license, which is incompatible with the GPL license used by Linux kernel developers. They apparently did this deliberately to stop ZFS being made part of Linux, with which they were competing at the time. ZFS cannot therefore be part of the mainstream Linux kernel development process… and that fact has now come to bite ZFS hard.

The short version of the unfolding ZFS on Linux bad news story is that ZFS on Linux relies on some functionality exposed by the kernel via two long-deprecated functions. Preparing for the imminent release of the new version 5.0 Linux kernel, the kernel developers finally (after more than 10 years) removed those two functions. That breaks ZFS at a stroke and means that the ZFS on Linux developers now need to code workarounds -which, quite possibly, won't perform as well as the original code that uses the now-removed functions.

Performance isn't exactly an issue when you're using ZFS as a bulk storage file system only: it's not as if you are writing to it hundreds of thousands of times a day, after all. But the fact that the issue arises at all illustrates the danger of being outside the kernel main line: the kernel developers can do things which break your code without having any responsibility for the breakage (and thus no responsibility to fix it). If your code is non-main-line, the breakage is something you have to fix, after it's occurred.

For the moment, none of this really concerns me: I'm running ZFS under CentOS 7, which is still using a version 3 kernel (and will do so until its end of life in 2024); And in five years' time, I think we'll know the state of play of ZFS on linux 5.x much better than we do now.

In the meantime, knowing that my CentOS 7 boxes would one day be end-of-life, I had anyway been considering a switch to FreeBSD, which has had integrated ZFS support out-of-the-box for years. Funnily enough, the FreeBSD developers recently decided to switch their ZFS code base to the ZFS-on-Linux one. This won't break ZFS, though, as the FreeBSD kernel is not the Linux kernel: whilst the Linux 5.x kernel changes break ZFS-on-Linux, it will continue working fine on the FreeBSD kernel.

The other thing I already do, incidentally, is run a server on OpenIndiana. That's basically a version of open source Solaris, so is genuinely Unix-y and, as a clone of Solaris, supports the Solaris version of ZFS natively. Again, as a bulk data server, I don't actually have to interact with that server's operating system very much (basically to check the Zpools are healthy once in a while!), so the fact that it behaves and operates rather differently to Linux isn't a major problem for me.

Anyway: the moral of this story is perhaps to be a little wary of ZFS on Linux. Maybe it's also to be prepared know how to use more than one operating system. And, possible, to shop around for the very best protection for those precious digital assets you need to keep safe: don't complacently settle on one O/S, file system or storage solution!

2019/02/03 11:27 · dizwell

Pro-tip...

Here's a pro-tip: If you write an article about how to set up a mail server and do so with a test-bed server specifically intended for testing and mucking about, remember not to wipe your actual, real, 'production' mail server by accident when you think it's time to wipe the test-bed server.

Admittedly, one IP address can look much like another at 7AM, but it's still no excuse!

On the up-side, at least I can attest that my article is pretty accurate and functional: I had a rebuilt server available in less than 20 minutes by following my own advice there!

Unfortunately, I lost all my mail archives, which is a bit of a bummer. Would pay to remember to set up a backup routine before relying on a server, instead of adding it to the list of 'things to get around to'. :-(

Of course, you only stuff up like that once in a lifetime, yes?!

2019/02/01 15:57 · dizwell

Free At Last!

I remember those ancient days past when one's email account was tied to whoever your ISP happened to be. Hence, I think my first ever personal email account was something like [email protected] Which unfortunately meant that if you changed your ISP, you had to change your email address: that wasn't going to work well in the long run!

So, like everyone else, I swiftly progressed to “immortal” email addresses with the likes of Yahoo!, Hotmail, Gmail and so on. Of course, we all knew that if the product was free you are the product, but the convenience of a web-based email account that never varied no matter who connected you to the Internet was not to be sniffed at.

Except, of course, these days we know rather more about the privacy implications of handing all your data over to mega-corporations like Google and Microsoft, especially thanks to the likes of Edward Snowden.

For the past five years or so, I've been wanting to ditch the likes of Gmail and Outlook (the Yahoo! account having stopped being relevant a decade or more ago!). But setting up your own email infrastructure is not something for the faint-hearted. There are a lot of moving parts; much administrative maintenance to do; complex software that is a real bummer to configure correctly …and on and on.

However, I have now finally kicked the Megacorporation habit, thanks to a cheap virtual server hosted in Amsterdam and the free software available from iRedMail -which, astonishingly enough- practically configures itself!!

It still isn't something I'd recommend you do on a whim, but I am now reasonably comfortable with it… and have accordingly thrown an article together describing the entire process. It's a long read, but worth it for the sense of freedom from snooping it gives me. Naturally, if the NSA want to know anything about me, they'll find it out one way or another. But at least a mere corporation can't have it for free anymore!

2019/01/31 18:07 · dizwell

Distro Dinosaur

Having successfully installed Ubuntu MATE at the start of the year, to replace a defunct Manjaro installation that had done good service for many months before succumbing to the problems of being a rolling distro, I can now report what a fortnight of using it feels like.

First thing to say, emphatically: it works! It works fine, in fact. All the applications I need (which often weren't in other distro repositories whenever I tried them a few years back) were swiftly and painlessly installed and work just as you'd expect them to. Despite MATE being a GTK (and hence Gnome-ish) desktop environment, key bits of KDE-based functionality work flawlessly -in particular, Clementine is happily playing its way through my music collection.

It also manages to work whilst wobbling nicely. Which is to say, I am afraid I am a fan of desktop bling and thus require my windows to wobble (and my virtual desktops to switch by appearing as faces on a rotating cube). Happily, Compiz integrates with MATE well, so all my required desktop effects are in place and mostly work without drama (the occasional tearing of a wobbling window as it's dragged can be a little annoying, but it's rare enough that I don't mind).

But… but… Whilst all the required functionality is definitely there, it looks dated. I didn't think this would bother me: as an old Gnome 1.4 user, I thought I liked things to be a tad old-fashioned! But there's old-fashioned and then there's Jurassic: and MATE is definitely in the latter park, as far as I'm concerned.

So: this week, I decided to pull the plug on Ubuntu MATE for its cosmetics, not its functionality. Casting about for a suitable replacement, I realised I've acquired the following internal rules about Desktop Environments:

  • Gnome is a no-no. It's just extremely irritating. It doesn't work out of the box the way I like, requiring instead a ton of post-install configuration with a bunch of extensions of sometimes dubious provenance to get there. And its native applications increasingly fail to provide the options and tweaks I feel I need. No Gnome-based distro for me, therefore.
  • Rolling distros are a no-no. It's me, not them, but I keep adding little bits and pieces of software as I go and the mix of me constantly altering things and the distro constantly updating itself is a recipe for going bang! at some point (as neatly demonstrated by Manjaro on New Years Eve!) So, no Arch, no Manjaro, no rolling distros of any sort. Stability before bleeding edge.
  • MATE is also a no-no. See above: I studied history at Uni and I love my Ancient Romans and Greeks, but MATE is just too “2000s” to pass muster today.

On the same “ye ancient desktops don't do it for me” principle, I can also rule out things like XFCE and LXDE. I'm also not young and adventurous enough to be enticed by the prospect of Budgie -which, in any case, I find very Gnome-like and thus comes lumbered with all Gnome's attendant problems!

Which leaves? Cinnamon and KDE, I think.

I have some experience with Cinnamon: it is again quite Gnome-y, and although it is now a desktop option for many distros, it began and is still heavily-driven by the Linux Mint developers. I have quite liked it in the past, but it's Minty heritage makes it a bit of a non-starter for me these days. I like Linux Mint a lot, but it's essentially Ubuntu with knobs on, and my last fortnight means I'm a bit over the Ubuntu-experience for at least a few months! It's also still Gnome-like, and 'native' applications for it would be GTK-based ones, and thus subject to Gnome-inspired design principles (meaning menus lacking most of the options you'd like them to have!).

So, for me, at the moment, it comes down to this: use a KDE distro. KDE has lots of graphical bling; it's got more configuration options than you can shake a stick at; its native apps are highly functional and the functionality is readily accessible and tweakable in ye olde fashioned menu-option fashion! It's a little too Windows-like, if I'm being completely honest, but if that bugs you, you can at least tweak it away. But you don't have to tweak much to get a good-looking, bling-filled, desktop experience that's modern and a pleasure to use.

Which brings me to the next question, of course: what's the best KDE-based distro? Obviously, you can install KDE on the top of most distros these days, but I'm after one where it's baked in as a primary option, not added in as an afterthought. Linux Mint is definitely not an option here, of course, because they long ago stopped developing a specifically-KDE version of their distro.

Incidentally, if I were contemplating using Red Hat or CentOS as a desktop distro, I would also have concerns about KDE there. But I'm not, and I don't think Red Hat's refusal to back KDE from 2024 and beyond is a show-stopper for using KDE in other distros in the meantime. You might reasonably expect Fedora to deprecate KDE in the same fashion sooner than that, though -but even if it doesn't, Fedora's integration with KDE in the past has never been exactly a triumph!

So if not Mint or Fedora, what?

Well, this is quite a handy list of distros that primarily use KDE -and implement it well. I'll say right up-front that no matter how good the reviews of those distros are, I'm not touching distros like “Chakra”, “Kaos” and “Netrunner”: I know nothing of where they come from, how big their developer community is, how good (and friendly) their user communities are… nor their long-term prospects, putting it bluntly.

That leaves me a choice of: Kubuntu (forget it: it's Ubuntu with KDE slapped on top); KDE Neon (forget it; it's Ubuntu!); Fedora KDE Spin (see above: forget it, because it's Fedora with KDE slapped on top, badly, and the long-term future of anything Red Hat and KDE is suspect)… and OpenSuse.

It's a Hobson's choice, in other words. If you want a mainstream, KDE-first, distro these days, you can choose OpenSuse or… nothing much!

Accordingly, yesterday evening, I wiped my Ubuntu MATE installation and put OpenSuse 15 on instead (note: OpenSuse has a 'Tumbleweed' offering, which is a rolling distro… and is therefore another no-no these days for me!) It is a pleasure to be back in the arms of KDE, with wobbling windows, desktop cubes, good looks and a sense of the modern!

I've put together a little article about how I did my post-installation tweaking, should anyone be interested (or should I need to do it again in the future… which given my propensity to switch OSes isn't beyond the realms of probability!)

2019/01/18 12:09 · dizwell

Older entries >>

wiki/blog.txt · Last modified: 2018/12/12 14:06 by dizwell