Have you tried a blind, level-matched listening test?

[whoaru99]

With the recent topics regarding the sound of amps, I thought I'd post a simple poll related to the subject.

Let's be honest now. :T

[jim777]

Didn't have to level match or I would have. The mac sounded better than the "very popular on this forum brand X" at any volume level (even lower). So why bother. You don't need to put an apple right next to an orange to make sure that both aren't the same...

Of course some prefer apples, others prefer oranges - both are good for our health, and music too!!

[whoaru99]

Didn't have to level match or I would have. The mac sounded better than the "very popular on this forum brand X" at any volume level (even lower). So why bother. You don't need to put an apple right next to an orange to make sure that both aren't the same...

Of course some prefer apples, others prefer oranges - both are good for our health, and music too!!

So you did a blind test, but did not level match, so I trust the answer you gave is "No" (or else did not vote)?

While I was composing the poll I originally had three questions.

I wonder how many people can guess what choice #3 was?

[Andrew M Ward]

Ahhh... Yes

Blind A-B-X
with level matching is the only way to test without bias.

A lot of (very confident) folks on this forum would be shocked at the results... the key word is "BLIND" and then throwing in brand -"X" helps as well instead of just knowing what "A" and "B" are and having a 50/50 shot at it.

When this kind of a test is created frequently there is much embarrassment on the part of the (very confident) listener...

... Which is why I try to argue that there is a lot of really good gear out there at all variety of price ranges, in stead of "A" blows away "B"

... "Blows away?" lets be careful out there... because small percentages of improvement normally have a large price increase associated with them (as they should)

For example, Yes - Mark Levinson amps sound great (I have a few, they’re nice) but they’re 800% more expensive than Rotel amplifiers (I have a few of them as well) are they 800% better? :Z

[jim777]

So you did a blind test, but did not level match, so I trust the answer you gave is "No" (or else did not vote)?

While I was composing the poll I originally had three questions.

I wonder how many people can guess what choice #3 was?

I voted "no" as I did not level match. For integrated amps or pre/amp combos, I had a marked preference even when the prefered amp's volume was lower. Listening tests is part of our work (we design speech & audio codecs) so I can make the difference between level and basic SQ. I still love my amp and don't want to change brand for that.

And when I tested speakers, I didn't level match either, but I was comparing only B&W speakers together. I assumed (probably wrongly) that the level would be matched if I didn't touch the volume. Anyway, the important thing is that I'm still happy with my choice of speakers more than a year later After going to "festival son & image" in Montreal, I found that B&W is really a brand that I love.

However, I needed to use level matching to compare CD players with my system, because the difference wasn't as obvious with my system and two great CD players. Things like louder bass are not a good indication to select a preference if the levels aren't matched. I still voted "no" because that test was not done blind. I still was able to find that I prefered the mac for all except most HDCD discs that sounded better on my previous player (the mac is SACD/CD only). But I didn't have a HDCD white noise test disc and I think the output was louder in HDCD mode so I'm not sure about that one..

[whoaru99]

Blind A-B-X with level matching is the only way to test without bias.

A lot of (very confident) folks on this forum would be shocked at the results... the key word is "BLIND" and then throwing in brand -"X" helps as well instead of just knowing what "A" and "B" are and having a 50/50 shot at it.

Andrew,

Just to be sure I have it straight, when doing an ABX test, X is either component A or component B but the relationship is chosen randomly so that X is not always component A and not always component B, but it is always one or the other.

So, thoughout the test X = A half the time, and X = B the other half. It's up to the listener to make the correct association of X = A or X = B on any given test.

Correct understanding?

[jim777]

"Blows away?" lets be careful out there... because small percentages of improvement normally have a large price increase associated with them (as they should)

For example, Yes - Mark Levinson amps sound great (I have a few, they’re nice) but they’re 800% more expensive than Rotel amplifiers (I have a few of them as well) are they 800% better? :Z

I was not comparing same-priced stuff. I only wanted to know if the increase in performance was worth the 400% price increase to me. The difference (in the case of the CD player) was not night and day, but good enough for me.

I will be honest, sound quality wasn't my only factor for all of my purchases. Sometimes having the CD player that matches your amp (visually) is also worth something

But I'm happy with my system and I guess that's what counts. I'm keeping the strict ABX, AB and Mushra tests for the audio lab - our passion shouldn't be that boring :Z

[soundhound]

I hot swapped one channel of my RB-1080 to a channel in my Carver TFM 24, and the sound instantly became thin in comparison. I was just checking the Carver ,rather than an a-b comparo, and it just struck me as "damn", what a difference.

[Andrew M Ward]

Andrew,

Just to be sure I have it straight, when doing an ABX test, X is either component A or component B but the relationship is chosen randomly so that X is not always component A and not always component B, but it is always one or the other.

So, thoughout the test X = A half the time, and X = B the other half. It's up to the listener to make the correct association of X = A or X = B on any given test.

Correct understanding?

usually,
say in a CD player sound comparison:
In a A-B-X style blind test the listener knows what make and model "A" and "B" are but does not know what "X" is.

The listener will be told in the intro (although the gear is hidden) now this is "A" and now this is "B" (only at the intro)

Then the truly blind listening happens, with random sessions in no particular order...

At an un-named facility in Northridge CA. the "magazine reviewers" tended to score lower than you might imagine... :roll: Also; as you might imagine it's quite pressure packed when you make outrageous declarations and then are asked to back them up ....

My point: (and I do have one) is there is a lot of really great gear out there, and we (with all the proof in the world) buy with our eyes as much as our ears... Which is fine by me, I'm just saying! I get really suspicious when I hear people declare "This is way better than that" and both pieces are of high quality.

The most adamant and defiant listeners have been shamed...

Not that it matters, it's just a point, I happen to love a wide variety of gear and I love fancy cables and interconnects, however I would never say "A" $400.oo interconnect is "WAY BETTER" than "X" $400.oo interconnect, because virtually 100% of listeners fail the high-dollar A-B-X interconnect test...

Source players can be difficult too, once you get into high end audio clocking and balanced analog audio boards...

Speakers - Now that's an area where listeners usually select a favorite and can pick it out with some regularity...other than that it’s usually fairly inconsistent (even when they swear they can tell) once you hide the gear and A-B-X it things get a lot more interesting.

:W

[RebelMan]

I am curious Andrew how the results turn out when the subject has spent considerable time with either A or B prior to the evaluation. Not just before the test but long term. I believe a strong familiarity with the equipment prior to the test would produce more predictable outcomes. Has this been considered in the cases you have seen?

I for one agree with you that the differences between comparable high quality electronics can be slight, but there are times when a "night and day" statement is not a complete exaggeration, especially if two "systems" are compared.

[Bob]

At an un-named facility in Northridge CA. the "magazine reviewers" tended to score lower than you might imagine... Also; as you might imagine it's quite pressure packed when you make outrageous declarations and then are asked to back them up ....

Information found at a internet site, or from a more reliable source?

Who were the participants? How did it come about?

[Andrew M Ward]

Information found at a internet site, or from a more reliable source?

Who were the participants? How did it come about?

Bob,
A) These are my own experiences (so you may, or may not consider that a reliable source)

B) There have been hundreds and hundreds of participants, some of them on occasion were audiophile’s sometimes even audio writers and college students and college music students...people working at the facility etc.

[Andrew M Ward]

I am curious Andrew how the results turn out when the subject has spent considerable time with either A or B prior to the evaluation. Not just before the test but long term. I believe a strong familiarity with the equipment prior to the test would produce more predictable outcomes. Has this been considered in the cases you have seen?

I for one agree with you that the differences between comparable high quality electronics can be slight, but there are times when a "night and day" statement is not a complete exaggeration, especially if two "systems" are compared.

Interesting you should ask that, where the writers were concerned a few brought in there "Reference" piece... or pieces.

Familiarity definitely helps a trained listener; I eventually was able to consistently pick out the CD player that was almost always used as the “X” unit, but not always.

Source material has an effect on the results as well -

[wolfgang]

I believe RebelMan asked what if people were given a few days or weeks to compare a few amps not just their own music. I think some have the impression whenever blind tests were carried out people always failed to identify any 2 things even when they do sound different.

Good to see a few have actually tried this 'dark art' before.

[Andrew M Ward]

I believe RebelMan asked what if people were given a few days or weeks to compare a few amps not just their own music. I think some have the impression whenever blind tests were carried out people always failed to identify any 2 things even when they do sound different.

Good to see a few have actually tried this 'dark art' before.

When I say reference pieces I mean their personal gear, like a CD player that they deem as reference level... from their own collection of gear.