Submitted by mmry404 t3_11ww4pt in headphones
Comments
thatsuaveswede t1_jd2c99c wrote
The statistician has entered the chat. Love the suggestions! 👍
mmry404 OP t1_jd0naen wrote
I totally agree. How would I evaluate the third category though?
Mediocre_Flounder_95 t1_jd3rjgw wrote
I have recently signed up for trials of Qobuz, Tidal, and Deezer (already using Amazon Music Unlimited). I stream via a Bluesound Node to a Roksan Caspian amp and PMC speakers - so a reasonably high/medium quality setup. Frankly, I can not discern any material difference in quality between the four and do not hear any improvement from hires.
Having researched the subject further, science suggests that the human ear is incapable of hearing hires. Perhaps it's just me and my older ears.
ultra_prescriptivist t1_jd09n75 wrote
Appreciate the effort that went into this - I do like to see it when people go the trouble of actually testing these things out rather than just blindly (!) following the internet consensus.
That said, there a few possible issues with this test.
Firstly, what exactly were you trying to establish - A) that Hi-Res tracks sound better than CD quality tracks, B) that exclusive mode sounds better than the system mixer, or C) that Qobuz sounds better than Deezer?
There are a lot of potentially confounding variables here, so establishing what your specific goal is is essential in order to control for those variables.
For example:
-
Did you ensure that both services were using the same master recording for each track on the playlist? If so, how?
-
Shouldn't the DAC have been set to 44.1 KHz
instead of 48KHz in system settings, seeing as that is the native sample rate used by the music playing through Deezer? -
Why didn't you use an electronic level meter to precisely match the playback volume levels of each app? Even a very slight disparity here could have affected your decisions.
-
Doing only 1 trial per track doesn't sufficiently rule out the possibility that you were simply guessing. When doing blind ABX tests, for example, it is customary to do at least 5 for each A/B comparison because you already have a baseline 50% chance of getting the answer right if you only do 1. So your result of getting 10/14 correct isn't really much better than the 7/14 you would have gotten simply by flipping a coin.
So, while I most certainly applaud you for setting all this up, I think narrowing your focus to achieve more reliable results would have been a good idea.
mmry404 OP t1_jd0ciie wrote
Thank you for your thoughtful response! I will try to address your valid points:
-
There is really a two-fold problem here, being Deezer not HiRes and Deezer not bit perfect. But HiRes only makes sense with bit perfect playback, so I decided that these problems kinda entail one another. Primarily I was just choosing between two services while my free month on Qobuz is still going. That's concerning A and B, and C just sounds too vague to me without any technical explanation as to why would that be the case
-
On the same master recording I commented under another comment (shortly, I just used modern recordings that probably don't have many masters)
-
I haven't set the DAC to 44.1 because I forgot (hehe) and also I mostly use this dongle with my android phone with no way to fiddle with its sample rate (that I know of) and it should also be 48kHz by default. I used a laptop instead because Qobuz on the laptop has exclusive bit perfect mode, while on my phone it doesn't, as I haven't bought the USB Audio Player
-
I should have definitely used the meter to get the volumes right, I will next time
-
That's a cool idea on trying the same track multiple times, thanks! Yeah, I did have my statistics courses back in the university and I do remember some things (vaguely), and yeah, the goal was to get as far away from 7/14 as possible. Also, the results do sort of align and make sense in hindsight... While the result is not overwhelming, still it's there and at least I didn't skew to Deezer. Further investigations will be carried out, since it was much fun:)
Thank you very much for your insightful comments again!
ultra_prescriptivist t1_jd0i27x wrote
>- There is really a two-fold problem here, being Deezer not HiRes and Deezer not bit perfect. But HiRes only makes sense with bit perfect playback, so I decided that these problems kinda entail one another.
True, but they still are two distinct factors. It's always best to isolate them and tackle them individually in my experience.
>- On the same master recording I commented under another comment (shortly, I just used modern recordings that probably don't have many masters)
I dunno, I see several tracks on there that have had multiple remasters and reissues over the years, and even 2020s music can occasionally be issued as different masters to different streaming services. You can't just rely on faith that they are probably the same.
Again, it's all about narrowing down to what you are trying to establish. Using different streaming apps when you haven't ruled out that the tracks you are listening to are not the same just introduces too many unknown factors, IMO.
For instance, if your goal is to establish whether or not it's better to use exclusive mode rather than the Windows mixer, it would be better to 1) use the same music app for both so you are not introducing additional, unforseen points of failure by using two, and 2) use local files instead of streaming services.
Perhaps you could use an app like Foobar2000 that allows you to switch between WASAPI exclusive mode and Windows Directsound. By using the same app and same track, that would eliminate a couple of your unknowns - the potentially different masters being used on streaming services and the volume disparity issue.
I'd definitely also shorten the list of tracks and do multiple trials (at least 5) per track.
Either way, let us know if you do any more follow-up tests!
mmry404 OP t1_jd0k9t1 wrote
Thank you very much! I generally agree with all your points. That experiment design choice was mainly dictated by my streaming strategy (continue to use deezer or switch to qobuz), also I wanted to give qobuz the chance to sound the best it could. First try is kinda janky, as always, isn't it? I didn't think I would be able to tell them apart and got excited, maybe a little too soon.
Also, looks like I shouldn't bother my girlfriend too much with these tests and maybe wait till I have a friend over or something, lol
ultra_prescriptivist t1_jd0mkq1 wrote
Ah, well if comparing streaming services is the goal then I certainly have done my fair share of that, as my comment and post history will attest 😄
I did a similar thing, although for me it was testing Spotify versus other streaming services.
As an aside, my testing on Spotify/Deezer showed that they generally use the exact same masters, for whatever reason.
Anyway, I used Audacity and WASAPI loopback recording to take samples of the different services and analyze the results though a combination of A-B comparisons using Audacity and blind ABX testing using Foobar.
These methods may prove useful to you if you want to continue your testing without bothering your girlfriend, lol.
mmry404 OP t1_jd0nwtq wrote
Great! Thank you very much! I wil definitely check those out! With these gizmos, I can satisfy my OCD urges alone and any time :)
mmry404 OP t1_jd0l9po wrote
Could you explain what do you mean by multiple trials? Like, the same track is played to me 5 times and each time I don't know where it comes from and I have to guess? That seems like a strange approach, since I can't go back to the previous playback state and compare, and I kinda just roll with whatever I'm hearing at the moment
Or, do I just ABABAB the same track, but each time I make a decision the person in charge flips a coin that decides which source is A and which is B and I start again (repeat 5 times)?
ultra_prescriptivist t1_jd0nqwb wrote
>Or, do I just ABABAB the same track, but each time I make a decision the person in charge flips a coin that decides which source is A and which is B and I start again (repeat 5 times)?
Pretty much this. The idea is that you do the same comparison (e.g. the Depeche Mode track) multiple times and each time choose which one you think is which, with it being randomized after each decision. The more trials you do, the more you reduce out the possibility that you just happened to pick the right one by chance.
mmry404 OP t1_jd0222i wrote
Recently I became concerned with bit-perfect playback: does it make an audible difference?
TL;DR: it does, seems like, but difference is not huge and depends on the genre. Also, statistics are not very reliable and require further tests
For those not in the know: bit perfect playback bypasses OS's system mixer and delivers music to the DAC "as is", without resampling it to a system wide bit depth and sample rate. This should, in theory, sound better(?). Dedicated digital players (DAPs) usually have their system mixers disabled, but on computers and phones this presents a problem, even when using an external DAC: bit perfect playback requires the player software to connect to DAC directly, and not all players can do that. I was very skeptical that upgrading from lossless CD quality playback, albeit not bit perfect, would make a difference.
So, i conducted a blind test. I was comparing a Hi-Res bit perfect streaming service Qobuz and a CD quality non bit perfect service Deezer. Both services were running using their apps on a Xiaomi laptop (RedMi book pro 15 2022), which runs Windows 11. I used dongle iBasso DC05 DAC and Moondrop S8 IEMs. Qobuz was connected to the DAC using WasApi Exclusive mode, which should provide bit perfect playback. Deezer doesn't have that feature and was using system mixer. Windows was using iBasso in default mode, 32bit 48kHz. I matched volumes by ear. I hear frequncies up to 16.5kHz and have mild tinnitus when in a quiet room. I'm 26.
I created two similar playlists of 14 tracks and my GF played me each track on both services. Random number generator was used to determine which service is used to play the track first and which second. My goal was to guess, which service was used for each playback. I could ask to go back and forth between the services and to go to the next track. Most of the tracks on Qobuz were marked Hi-Res (24bit, from 44.1kHz to 196kHz), CD quality on the others (16bit 44.1kHz); and Deezer offers them all in CD quality. I mostly used tracks from my library, as well as some test signals and some jazz I haven't heard (cuz audiophiles love jazz for some reason?).
On busy math rock stuff (Delta Sleep, The Yacht Club) I couldn't grasp the difference and was frantically going back and forth, just to make a guess in the dark. On modern electronic tracks (i.e. Chivalry Is Not Dead by Hiatus Kayote) one playback sounded a bit sharper then the other, sometimes it had more bass. Every time I liked a playback a little more, I attributed that playback to Qobuz (well, it should sound better, right?). The difference was never obvious. Radiohead tracks sounded virtually the same. The American Beauty soundtrack, Arose, has these heavenly little bell plucks, which sounded more clouded during one of the playbacks. The test sounds seemed to be more revealing, as it seemed to me that one playback had somewhat more aliasing, especially on 14kHz tone. The test lasted about 40 minutes.
So, the results? I guessed the service correctly 10 times out of 14. I missed on both of my favorite math rock bands, but was correct on all the others, except for two jazz tracks that I haven't heard before. I guessed both test tones correcly. I think, I proved to myself that it is worth it to pursue bit perfect playback and maximum file resolution, although not for all genres. It seems like rock music doesn't benefit from it as much as, say, acoustic or electronic music.
I will conduct further blind tests to investigate more granular use cases (i.e. test tones only, and different tests for different genres) and to make the results more reliable.
Yup, I used the word "albeit" haha
UPD: Also, next time I will use an EV meter to set the volumes instead of by ear and make more test cases to rule out the probability of a random guess (as proposed by u/ultra_prescriptivist)
ku1185 t1_jd08l4l wrote
After trying Tidal and seeing there are sometimes multiple copies of the same album (apparently, different masters, according to internet strangers), I no longer know if it's something with the service or if some of them are simply using different masters. Sometimes the different versions on Tidal sound different, sometimes they don't (at least, nothing that I could discern).
FWIW, I think Qobuz generally tended to sound better than Amazon or Tidal (though that album version thing throws a wrench into the whole thing).
mmry404 OP t1_jd099lq wrote
Pretty much all the tracks I used come from modern albums that likely don't have many different masters. There was a track from Depeche Mode's Black Celebration in my playlist that had a different intro and was louder on one of the services, so I didn't use it. The usage of different masters cannot be accounted for, so it the tracks don't obviously sound different, I would assume same masters and just roll with it
And I also didn't want to try Tidal because of its crazy pricing and the MQA controversy
ku1185 t1_jd0d5yi wrote
I don't think modern albums are exempt from this. Taylor Swift's Lover album was one I distinctly remember there being 3 different versions, with one of them sounding quite different. FWIW, I tested this on a non-MQA DAC.
mmry404 OP t1_jd0dwnw wrote
That's kinda odd. I mean, being an artist, why would you put out several masters of your new album? Why is Tidal the only one to carry several masters? Is there a way to check that there are several different masters of a particular recording if I don't have Tidal?
Yeah, it never crossed my mind that having a non-MQA DAC eliminates the MQA issue, lol :)
ku1185 t1_jd0fjem wrote
>That's kinda odd. I mean, being an artist, why would you put out several masters of your new album? Why is Tidal the only one to carry several masters? Is there a way to check that there are several different masters of a particular recording if I don't have Tidal?
No idea. Kind of just gave up on trying to figure out what was going on lol.
As for MQA, don't know if an MQA DAC would've made any difference. I never got around to testing it when I had MQA DACs.
mmry404 OP t1_jd0ijvv wrote
There's a video by Golden Sound, proving that MQA is actually lossy, so an MQA dac sould probably just sound worse. Although it seems like more and more dacs are supporting it now... which in itself is a worrying observation that hardware manufacturers may be prioritizing marketing claims and fancy labels over performance. Ideally everything should be checked in a blind test, but these are cumbersome to carry out correctly. I'm just dipping toes into it..
ku1185 t1_jd0lsk1 wrote
Yeah I'm aware of the video. My understanding of the tech (or at least, what is claimed) is that it stores information under the noise floor or something, and MQA hardware can "unfold" that information to generate a more complete reconstruction. Whether there's any audible benefit, I have no clue. Given all of the different masters on Tidal, it makes it difficult to determine whether the codec is actually making any difference or if its the master itself. So I just gave up trying to see if there's any real sound quality differences for MQA itself.
I think the biggest issue with MQA is false marketing claims and the proprietary nature of the tech. If they made more accurate claims that were verifiable and improved the listening experience, I wouldn't mind it.
mmry404 OP t1_jd0mynl wrote
Indeed. The thing with marketing claims is that people need a deep technical understanding of the subject to be able to discern false marketing from real marketing, and it is just not feasible, since said marketing is addressed to the masses. And even with that level of understanding there is still a chance that something could be dodgy under the hood where we can't see. With enough critical thinking/paranoia it gets impossible to consume and be happy about it, so maybe it's the case that "the less I know, the better". Simulacra and simulation, all that..
[deleted] t1_jd09lw3 wrote
[deleted]
Adrian1616 t1_jd156qz wrote
A few months ago I did a (not blind) A/B test between Qobuz and Spotify. I had been using Spotify and switched to Qobuz for my at home HiFi listening. I wanted to know if the price and UI difference were worth it to me. After listening to numerous tracks within various genres I concluded that in only a small portion of the time, if I was listening very critically, could I hear a difference. Maybe 10-20%. Same source, just switching playback from one to the other for 20-30ish seconds of a song at a time. I was using my LCD-2Cs in a silent room. I'm 22 and have no known hearing issues. I decided that it wasn't worth the extra money for me personally. Do with that info what you will. Not scientific but I did spend a good while doing it and was honestly rooting for Qobuz.
mmry404 OP t1_jd20jwc wrote
For me the question was, would there be any difference or not, and I was sure there would be no difference at all. Yeah, it was small, but if we are not mistaken, it was there, and for me that justifies some extra cash and effort. That is, of course, if the results are reliable, and fellow redditors pointed to some caveats In my setting. Guess I will have to do that again? Think I'll go with test sounds only, to only listen to sine sweeps like a true audiophile affectionado
rajmahid t1_jdqt95m wrote
With all due respect, there’s no such thing as too much classical music.
mmry404 OP t1_jdqxfap wrote
Yeah, I mean if you're a kid and they make you listen to it all the time... I wasn't even interested in music back then. My mom is a pianist, and even that didn't bring me to enjoy it much.
brucie_me t1_jdwjzgs wrote
So you were "abused" as a kid by having your folks play classical music and you had to listen to it? Wow, what a bummer! Culturally deprived.
mmry404 OP t1_jdwkf00 wrote
Well, sort of (and I feel your irony), but on the other hand, my dad always listened to things like Depeche Mode, Pink Floyd, Kraftwerk, Laibach etc. at home and in a car. So, when I grew older, it started resonating with me and I used his music taste as a starting point. Still love these to this day!
My mom wanted me to become acquainted with music as early as possible. Thanks to her, I developed a trained ear, but I wasn't sure yet what to do with it. But then, thanks to my dad, I figured out what I like, and listening to music, performing and composing has been a hobby of mine ever since.
NightOwl_6627 t1_jd4ho4g wrote
I don't really care about high-res. But after using many platforms like Apple music, spotify, tidal,... I only stay with Qobuz now. Somehow, Qobuz feels like right to my ears more than the others.
I'm using IE800S with Fiio Q5 stacked with Samsung Note 20 Ultra (dolby atmos turned on).
mmry404 OP t1_jd4m6jn wrote
I also really like its interface, it's just beautiful
elGatoDiablo69 t1_jd1b2ns wrote
A great post! Like many others suggested - volume leveling is a big factor to consider as sometimes slightly higher volume can be mistaken for better quality. I would also suggest using only the tracks (maybe even album versions) that you know very very well. There’s a big emotional difference in how you might react to a new song versus something you know very well and essentially know what to expect. And in the meantime, we’re gonna sit tight here in Canada and wait for qobuz to arrive…. One day… maybe
mmry404 OP t1_jd1xj6a wrote
Thank you! I pretty much did only use tracks I know very well, although I haven't checked them all out since I upgraded to the S8. Sometimes I just didn't want to go to the next track since it just sounded so freaking good!
elGatoDiablo69 t1_jd3900c wrote
Yeah. Makes sense. Just form personal experience and basic research - your perception of music can change dramatically based on a great number of variables such as your mood, time of day, physical condition, etc etc. our sensory perception is so heavily biased! :) nonetheless, as long as you enjoy the music - it doesn’t matter what’s the source or the emitter! Happy listening.
mmry404 OP t1_jd39sol wrote
Sure, the transducer makes up 95% of listening experience, guess I'm just curious how to get that last drop!
rajmahid t1_jdpvjcx wrote
I have nothing to add regarding your experiment that hasn’t been commented on by other posters except there’s a total absence of classical music on your list. I assume it’s not your cuppa tea, but my most critical evaluations of headphones & streaming services are based on my personal taste for classical. Having tried all 4 lossless streamers, Qobuz is the gold standard in sheer sound quality. Interface and other considerations are another matter for other discussions
mmry404 OP t1_jdq1t4a wrote
Oh yeah, you are totally right! Yeah, it's somehow not my cup of tea, maybe I had too much of it in the music school, lol. Will add it next time!
[deleted] t1_jd01n9w wrote
[deleted]
idontliketopick t1_jd0mlc3 wrote
Props for conducting this, I applaud your curiosity. I see a couple problems worth pointing out.
10/14 isn't actually very good. At n=14 you're at a 68% confidence interval which really isn't much better than a coin flip. A sample size of 14, missing just one is only 80%, not great.
You really should have a third category of "i don't know/i can't tell". By only have two categories you may artificially increase your score by guessing correctly.