Weighted Scoring for Software Comparisons
Method · 9 min read ·
How to turn a messy comparison into a weighted score: set criteria, assign weights, score from evidence, test sensitivity and avoid false precision.
Comparing software usually produces two things: a lot of notes and a vague feeling. One product has the better interface, another has the better price, a third has the integration you need. Each strength pulls in a different direction, and there is no obvious way to add them up. Weighted scoring is a way to add them up openly.
It is not magic. It does not produce objective truth. What it does is make your priorities explicit, force you to score from evidence and show how sensitive the result is to your assumptions. This article explains the method, step by step, and the traps to avoid.
What weighted scoring is
Wikipedia describes a decision matrix as a way of assessing many decision factors by weighting each according to its importance, with alternatives in columns and criteria in rows and each cell showing how an alternative performs on a criterion. Weighted scoring is the arithmetic that completes it: multiply each score by its weight and add the results.
The result is a single number per alternative. The number is only as good as the criteria, the weights and the scores. Its main value is not the number but the discipline of reaching it.
Step one: define the decision and the must-haves
Write one sentence on what you are choosing and for whom. Then list the must-haves: conditions that, if unmet, rule an option out regardless of other strengths. Examples: it must run on your operating system, it must allow data export, it must meet a legal requirement. Treat these as pass or fail. Do not weight them; filter on them first.
Step two: choose criteria
For the options that pass, choose the criteria that distinguish them. Aim for six to ten, each:
- Relevant to the decision
- Defined clearly enough to score consistently
- Independent of the others, to avoid double counting
- Checkable from evidence
Typical software criteria: quality of core functions, ease of use, integrations, data portability, privacy and security information, support and documentation, reliability, vendor stability and total cost of ownership.
Our article on choosing attributes covers this step in more depth.
Step three: set the scale
Choose a scale for scores and define what each point means for each criterion. A scale of one to five is common. Without definitions, one person's four is another's two.
Example for "Support and documentation":
- 1: No documentation; no support route
- 2: Basic documentation; email support with no stated response time
- 3: Good documentation; support with a stated response time
- 4: Comprehensive documentation; fast support; community
- 5: Excellent documentation, multiple support channels, proven responsiveness
Anchors like these keep scoring consistent and make the result explainable.
Step four: assign weights
Weights reflect how much each criterion matters to you. A common method is to distribute a fixed budget, such as a hundred points, across the criteria. The act of splitting a limited budget forces trade-offs: giving more to one means less to another.
Tips:
- Decide weights before you look at candidates. Otherwise they drift towards the product you already like.
- Consider the consequences of failure. A criterion whose failure would be costly, such as security for a system holding sensitive data, deserves a high weight.
- Involve the people who will use the tool. Different roles weigh things differently. Averaging their allocations, and discussing large differences, surfaces hidden disagreements.
- Keep weights stable while you score. Change them only through a deliberate, recorded step.
- Do not weight everything the same. If everything is equally important, nothing is.
Step five: score from evidence
For each product and criterion, assign a score and write the reason and the source. Evidence includes your own tests, documentation, policies, reviews and conversations with vendors.
- Test, do not guess. Where you can, run a standard task and record the result.
- Score independently, with two people if possible, and compare.
- Mark unknowns instead of guessing. Treat an unknown as a question to resolve. If you must score provisionally, use a cautious value and flag it.
- Resist halo effects. A product that impresses on one criterion may bias your scores on others. Score criterion by criterion across all products, not product by product across all criteria.
Step six: compute and look
Multiply each score by its weight and sum for each product. Then look at the result in several ways.
- The ranking. Which scores highest?
- The margin. Is the gap large or small?
- The profile. A product might score well overall but poorly on one heavily weighted criterion. Examine the shape, not only the total.
- The pattern of unknowns. Which scores are uncertain?
If two options are close, the total cannot separate them. Use the extra evidence you need.
Step seven: test sensitivity
Because weights are judgements, check how much the result depends on them.
- Change each weight up and down by a few points and see whether the winner changes.
- Try an alternative set of weights, such as one favouring cost and one favouring quality.
- Remove one criterion and see what happens.
- Note which assumptions matter most.
If the same product wins under all reasonable weights, you can be confident. If the winner flips with small changes, the decision is close and you should seek more information, or accept that either choice is defensible.
Include cost properly
Cost is often the most heavily weighted criterion, and the easiest to get wrong. The idea of total cost of ownership, covered in more detail elsewhere on this site, is that what you pay to acquire a product is usually a minor part of what it costs you to own it. For software, include subscription fees at your expected usage, setup, training, integration, administration, review time and migration, over a defined period such as three years. Convert cost to a score using a stated rule, for example the cheapest scores five and others score in proportion.
Beware false precision
A total of 4.37 against 4.29 looks decisive. It is not. Scores are judgements on a five-point scale, weights are estimates, and the data may be uncertain. Round results, and treat differences smaller than the uncertainty as ties. Show ranges where you can. Report the number as a guide, not a verdict.
Beware of Goodhart's trap
Goodhart's observation is that when a measure becomes a target, it stops being a good measure. If your team learns to produce a score that matches a favoured outcome, the exercise loses meaning. Protect against it by fixing criteria and weights in advance, requiring evidence for scores, having someone outside the preference group review and recording any changes with reasons.
If the scoring will be shared with vendors, expect them to optimise for the criteria. Consider keeping weights private while publishing criteria and evidence requirements.
Use the score to start a conversation
The output is a tool for discussion. Ask the team: Does this match our gut feeling? If not, why? Which criterion drives the difference? Is a criterion missing? Did we weigh something wrongly? A mismatch between score and instinct often reveals an unstated priority. Adding it as a criterion, and rerunning, produces a better model.
A worked example
A four-person agency compares three project tools. Must-haves: runs in the browser, exports all data in an open format. All three pass. Criteria and weights (out of one hundred): core project features 25, ease of use 20, client sharing 15, integrations 10, data handling and privacy 10, support 5, three-year cost 15.
Each person allocates points independently; the averages produce the weights above, and discussion reveals that the designer cared much more about client sharing than the developer. They agree to the final weights before testing. Over two weeks, they score each tool from a standard task and documentation, with sources. Tool A scores best on features and client sharing, but its cost is highest. Tool B is cheapest and easiest, but its privacy information is unclear and is marked unknown. Tool C is in between.
The weighted totals: A 4.1, B 3.9, C 3.6. The sensitivity check shows that if cost weight rises by ten points, B overtakes A. They ask B's vendor about privacy and receive clear answers, raising its score. They run a final pilot of A and B, choose B and record the scores, weights, sources and the sensitivity result in a short note.
A checklist
- Decision and must-haves defined
- Six to ten defined criteria
- Anchored scale
- Weights set before scoring and agreed
- Scores from evidence, with sources
- Unknowns marked
- Totals, margins and profiles examined
- Sensitivity tested
- Total cost of ownership included
- Results shared as a guide, not a verdict
The categories and search pages here help you build a shortlist, the leaderboards page shows how ranking signals are labelled and the blog collects more guides on comparison.
Frequently asked questions
Is it fine to use a simple three-level scale? Yes, if it is defined. Simpler scales can be more reliable.
Should I share my weights? Within your team, yes. Publicly, explain the criteria and let readers adjust.
What if the result surprises me? Investigate. Either your instinct has a missing criterion or the scores need rechecking.
Questions and answers
- What is weighted scoring?
- A method in which each criterion gets a weight reflecting its importance, each alternative gets a score per criterion and the weighted scores are added up.
- Does a higher score mean a better choice?
- It means a better fit to the weights you chose. Change the weights and the order may change.
- How do I choose weights?
- Ask what would make you reject a product, give those criteria the most weight and agree weights before looking at the candidates.
- What is a sensitivity check?
- Changing the weights slightly to see whether the winner changes. If it does, the decision is close and needs more evidence.