Choosing Attributes for a Fair Comparison

The attributes you choose determine the answer. A comparison of two note-taking apps that includes "number of themes" and "animation quality" will favour a different product from one that includes "data export formats" and "offline use". Neither list is wrong, but each steers the reader. If you are writing a comparison, you carry responsibility for the choice. If you are reading one, it is worth asking how the attributes were chosen.

This article explains how to choose attributes for a comparison that is fair, useful and honest.

Start with the reader's decision

The first step is not to list features but to identify the decision the comparison supports. Who is reading, and what are they trying to decide?

• A freelancer choosing invoicing software cares about setup time, tax features and cost.
• A school choosing a learning platform cares about privacy, accessibility and administration.
• A developer choosing a database cares about performance, licence and operational effort.

Write one sentence: "This comparison helps [reader] choose [kind of product] for [purpose]." Everything else should serve it.

Gather candidate attributes widely

List every attribute that might be relevant. Sources include:

• Reader questions. What do people ask in forums, support emails and reviews?
• Vendor documentation. Features, limits, policies.
• Your own experience using the products.
• Expert advice from people who use them professionally.
• Earlier comparisons by others, to see what they covered and what they missed.

At this stage, do not filter. You can cut later.

Apply five tests

Take each candidate and ask five questions.

1. Is it relevant?

Does it affect the reader's decision? A feature that almost nobody uses, or one that does not differ between products, adds noise. Wikipedia's account of feature creep notes how the excessive addition of features leads to bloat and unnecessary complexity; a comparison that lists every minor feature mirrors that problem.

2. Is it measurable or checkable?

Can you determine the value consistently? "Ease of use" is subjective. "Time to complete a defined task in a test" is measurable. "Privacy" is vague. "Publishes a data processing statement; allows deletion on request" is checkable. If you include a subjective attribute, say how you judged it.

3. Is it independent?

Does it overlap with another attribute? Listing "supports CSV export", "supports JSON export" and "supports data export" counts the same capability three times, which inflates the importance of the topic and can favour a product that does well on it.

4. Is it fair?

Does it favour a type of product for reasons unrelated to the reader's need? "Has a mobile app" disadvantages a web-only tool even if the reader never works on a phone. "Open-source licence" favours one family even if the reader does not care. Fair does not mean neutral about everything. It means that the attributes reflect the reader's real needs, not the author's preferences.

5. Is it stable?

Will the value mean the same thing in six months? Some attributes change quickly, such as prices and feature availability, and need dates. Others, such as the licence, change rarely.

Attributes that pass all five tests are strong candidates. Those that fail one or two may still be worth including if you handle them carefully.

Cover the whole experience

A fair comparison covers more than features. Consider categories such as:

• What it does: core capabilities for the stated purpose.
• How it feels to use: setup, learning curve, accessibility, speed.
• What it costs: pricing model, plans, limits, total cost of ownership.
• How it treats your data: privacy, security, export, retention.
• How it fits: integrations, platforms, standards.
• Who is behind it: age, funding model, governance, support.
• How it can end: exit options, migration, licence.

Wikipedia defines total cost of ownership as an estimate of the direct and indirect costs of a product, and notes that these often exceed the acquisition price. Including cost attributes beyond the price list helps readers see the whole picture.

Define each attribute

A name alone is not a definition. For each attribute, write:

• What it means. One sentence.
• How it is assessed. The test, the source or the scale.
• How to read the values. What is better, what is worse, what is neutral.

For example, "Data export: whether a customer can download all of their content in a documented, open format without contacting support. Values: full, partial, none. Assessed from the product's documentation and a test export."

Publish these definitions with the comparison, so that readers can see exactly what was compared.

Decide before you look

A powerful way to avoid bias is to fix the attributes before you assess the products. If you choose attributes after seeing which product wins, you may be tempted to include those that favour your preference. Write the list, share it with a colleague and agree it. Then gather the data.

If you discover during research that an important attribute was missing, add it for all products, and note that you did. If a planned attribute turns out to be meaningless, drop it for all.

Include attributes where each product is weak

A fair comparison does not tilt toward a favourite. It includes attributes where each product shines and where each struggles. If you notice that every attribute favours the same product, ask whether the list has been built around it. If the author sells one of the products, this is especially important, and the relationship should be disclosed.

Handle subjective attributes carefully

Some things that matter cannot be measured precisely: ease of use, design quality, tone of support. You can include them if you are honest.

• Say how you judged: a panel of testers, a standard task, a survey.
• Show the range of views, not a single score.
• Label them as opinion.
• Weight them modestly unless the reader's need depends on them.

Alternatively, convert them into observable proxies: time to complete a task, number of steps, whether a feature is reachable from the main screen.

Beware of Goodhart's trap

Goodhart's observation says that when a measure becomes a target, it ceases to be a good measure. Applied here: if a particular attribute becomes widely used in comparisons, vendors will optimise for it. Counting integrations leads to shallow integrations. Counting languages leads to poor translations. Choose attributes that reflect real value, and where possible measure depth, not only presence: how well, not just whether.

Group and order sensibly

Organise attributes under clear headings. Put the ones that matter most to the reader first, and make it easy to skip the rest. Avoid ordering that quietly favours a product, such as putting its strongest rows at the top and its weakest at the bottom, without reason. If you allow readers to choose and weight attributes, even better.

Test the list on real readers

Before you publish, ask three people who match your audience to look at the attributes and say what is missing, what they do not care about and what is unclear. Their answers will improve the list more than any amount of private deliberation.

A worked example

A writer prepares a comparison of three password managers for small families. The sentence: "This comparison helps families of two to five people choose a password manager for shared and personal logins." She lists thirty candidate attributes, then applies the five tests.

She drops "number of themes" and "animation" as irrelevant. She merges "supports CSV export" and "supports JSON export" into one attribute, "export of all stored items in an open format". She replaces "easy to use" with "time to add a login and share it with another family member, measured on a standard task". She adds "recovery options if a member forgets the main password" and "cost for five users per year, with plan details dated" because families care about them.

She fixes the list before testing, defines each attribute, and gathers the data. When she finds that one product has no recovery option, she records it plainly. She discloses that she has no financial relationship with any vendor. Readers comment that the list matched their worries, and one points out a missing attribute, which she adds for all three with a dated note.

A checklist

• The reader's decision stated
• Wide list of candidates
• Five tests applied: relevant, checkable, independent, fair, stable
• Whole experience covered
• Each attribute defined and its method stated
• List fixed before assessment
• Strengths and weaknesses of every product included
• Subjective attributes labelled
• Depth considered, not only presence
• Tested on readers

The categories and search pages here help you find products to compare, and the leaderboards page shows how ranking signals are separated and labelled.

Revisiting the list

Attributes that mattered last year may matter less now, and new ones appear as products evolve. Review your list each time you update the comparison. Drop attributes where all products have converged, add ones that readers keep asking about and rewrite definitions that have turned out to be ambiguous. Note each change in a short log, so that earlier comparisons remain understandable and so that readers can see how the method has developed. A list that is treated as a living document stays fair for longer than one that is fixed once and forgotten.

Frequently asked questions

Can readers choose their own attributes? Where the interface allows, yes. It makes the comparison fairer for different needs.

How do I handle attributes that change often? Date them and review regularly.

What if I cannot find data for an attribute? Show it as unknown, and consider dropping it if it is unknown for most products.