To evaluate software, shortlist 2 or 3 tools, run each on work your team already does, and judge it against a baseline measured before the trial. The baseline turns the trial into a measured saving in minutes, corrections and handoffs, which no demo shows. The comparison holds only when a mixed-role trial group repeats the baseline’s assignments.
Take an illustrative weekly report: cutting it from 90 to 55 minutes saves 35 minutes a run, about 23 hours over 4 weeks when 10 employees each do it.
The 9 steps below fit small and mid-size teams choosing business tools, not developers testing code; their length depends on how often the task repeats.
- Task: evaluate a business software tool before your team adopts it
- Result: a written adopt, extend or keep decision with its numbers
- Prerequisites: 1 recurring task, 3-6 participants, trial access, a budget owner, a security contact
- Tools: a timing sheet and the scorecard below
- Limitation: as many trial runs as baseline runs
In This Article (8 sections)
Before You Start
Have these 7 things ready before anyone opens a trial account.
Before you start
- 1 recurring task the team complains about, with a clear start and finish
- 3-6 participants in different roles, including at least 1 occasional user
- 1 budget owner who approves or declines the purchase
- 1 security contact who rules on vendor documents and company data
- A timing sheet for 2 or 3 baseline runs
- Authority to create and remove test accounts
- Each trial’s terms, read before company data goes in
The Six Evaluation Criteria and Scorecard
Every participant scores the same 6 criteria from 1 to 5, privately and with a one-line reason, before any group discussion.
| Criterion | What to check in the trial | Score 1 means | Score 5 means |
|---|---|---|---|
| Business value | Is the target task faster or cleaner than the baseline? | Slower, or more corrections | Faster, with fewer corrections |
| Ease of use (usability) | Can people finish assignments without frequent help? | Repeated help requests | Finished unaided |
| Workflow fit | Does it match how the team shares, assigns and approves work? | Manual copying, late notifications | Handoffs stay inside the tool |
| Total cost | What does year one cost for the whole team? | Over the budget ceiling | Under it, every cost counted |
| Security and control | Can you set permissions and remove access? | A removed account still gets in | Roles and offboarding work as required |
| Support and data export | Does support help, and can you get data out? | No useful answer, or a broken export | A useful answer and an intact export |
Copy the table into a spreadsheet, 1 tab per participant, before the trial starts. Then apply 3 rules:
- Average each criterion and note the spread, the highest score minus the lowest. A spread of 2 or more goes to discussion.
- A 1 or 2 on security and control from any participant stops the product until the security contact rules. That is the gate.
- Leave the criteria unweighted unless the group agrees on weights for the other 5 before the trial.
My take
I would keep security and control as a gate, not 1 score among 6. A product can be simple to learn and fairly priced and still fail your security requirements, and a strong average would hide that.
How to Evaluate Software: Step by Step
The 9 steps, in order: name the problem and people, measure a baseline, write requirements, shortlist, request documents, trial on real work, test controls, score privately, and decide. They run in dependency order, and each leaves a record the next one uses.
Step 1: Name the problem and the people
Pick 1 recurring task the team complains about, and name the stakeholders who will judge the tool. A weekly report built from 3 files and checked by hand is a good pain point: its minutes and corrections can be counted.
The scorers are 2 or more daily users and at least 1 occasional user, 3-6 people in all; a daily user and a once-a-week user notice different problems. Name the budget owner and the security contact beside them: they approve and rule rather than score.
Expected result: a one-line problem statement, 3-6 named participants, a budget owner and a security contact.
Step 2: Measure a baseline before the trial
Time the task end to end with your current tools over 2 or 3 normal runs: minutes per run, corrections, handoffs, and who gets pulled in when something does not add up.
Expected result: 1 baseline row per run.
Step 3: Write must-haves, nice-to-haves and a budget ceiling
Write a one-page list of core requirements and mark each must-have pass or fail. It covers the data the tool will hold, integrations, sign-in and roles, the accessibility level, the export format and an annual budget ceiling.
Name the data types first (customer, employee, financial or regulated records) and send regulated data to whoever owns security or compliance. For integrations, name each system the tool must exchange data with; knowing what an API does helps you ask whether a connection is built in.
For sign-in, NIST defines single sign-on (SSO) as an authentication process in which one account and its authenticators are used to access multiple applications. If your company signs in that way, put it on the must-have list.
For digital accessibility, name a standard and a level. WCAG 2.2 was published on October 5, 2023 and updated on December 12, 2024, with success criteria at 3 levels: A, AA and AAA. W3C encourages using the latest version.
Knowledge base software requirements template
A filled requirements template for one software category that you can adapt to the tool you are buying.
Expected result: a one-page requirements list with every must-have marked pass or fail.
Step 4: Shortlist two or three tools
Keep only the software solutions that meet every must-have on paper, and price the plan the whole team will need at the headcount you expect in 12 months, not the entry plan.
If you are choosing social media software, https://compareforme.net/ compares popular Instagram and social media tools side by side on features, pricing, privacy and usability, which helps narrow the list before anyone signs up. For other categories, start from a ranked list:
- team collaboration: my ranking of small business collaboration software
- customer support: help desk software compared
Expected result: 2 or 3 candidates, each with a named plan and a full-team price.
Step 5: Request security, privacy and accessibility documents
Before company data goes in, ask each vendor for 3 things: a SOC 2 report or an ISO/IEC 27001 certificate, answers to the 2 sign-in questions from CISA’s guide below, and an Accessibility Conformance Report (ACR).
A SOC 2 report records an examination of a service organization’s controls for security, availability, processing integrity, confidentiality or privacy. ISO/IEC 27001:2022 defines the requirements an information security management system must meet.
CISA’s Secure by Demand Guide suggests 2 questions for the same email: does the vendor support standards-based single sign-on (SSO) at no additional cost, and does it enable multi-factor authentication (MFA) by default, at no cost?
Section508.gov notes that ACRs are often referred to as VPATs, after the industry standard template for product accessibility claims, and that vendors often post them on their accessibility page.
I would also ask each vendor for 1 customer reference at your team size.
Expected result: a document set per candidate, or a named gap the security contact rules on.
Step 6: Run the trial on real work
Give participants the same assignments you timed in Step 2, with real deadlines and last-minute changes, on a trial or pilot, and do not walk them through every screen.
Log task times and every point where someone got stuck.
Warning
Uploading customer or employee records into a trial can expose them. Read the trial terms, use copies or non-sensitive records, and get the security contact’s approval before real data goes in.
Expected result: 1 trial log per person, comparable row by row with the Step 2 baseline.
Step 7: Test permissions, offboarding, export and support
Run 4 checks on test accounts: give accounts different roles and confirm what each sees, remove 1 and confirm its access ends, export records and open them in another app, and send support 1 real problem the team hit, timing its answer and asking what response time your plan includes.
A project manager should not see payroll records. In the export, check that dates and file names survive. For a CRM, this export is a small preview of what a CRM migration involves.
Warning
Removing an account or changing a role can cut access to live work. Run these 4 checks on test accounts only.
Expected result: the removed account loses access, the export opens intact, and support’s answer resolves or explains the problem.
Step 8: Score privately, then discuss
Each participant fills in the scorecard alone, 1 score and a one-line reason for each of the 6 criteria, before any group discussion. Private scoring keeps a quieter participant’s concern from disappearing behind the loudest opinion.
Then apply the 3 scorecard rules and keep every sheet on file.
Expected result: 1 score sheet per person and a summary row per criterion.
Step 9: Compare with the baseline, price year one and decide
Set minutes saved per run times runs per year against the first-year cost, then write 1 of 3 decisions: adopt, extend the trial, or keep the current tool.
First-year cost is the plan price times users times 12 months, plus setup, migration help, training, extra users, add-ons and paid connections. Staff hours spent fixing failed connections or preparing exports count too. Some plans bill per user and others cap storage, usage or features, so check how SaaS subscriptions are billed before comparing 2 pricing pages.
For a CRM, the CRM cost calculator runs this arithmetic at your team size.
My take
Adopt when the trial time per run is under the baseline time, the gate is passed and the first-year total stays under the ceiling. The averages only explain the call: they name what training or the vendor must fix, and between 2 tools that both meet the rule, I would pick the higher overall average. If the trial saves a few minutes here and there while switching means weeks of training, I would keep the current tool.
Expected result: a dated, written purchasing decision with minutes saved, first-year cost and the gate result.
Warning
Canceling the current tool before its data is exported and checked can lose records. Keep the old subscription until the Step 7 export test passes in the new tool.
Check That It Worked
The evaluation was sound when all 7 checks hold, whatever the decision was.
Verification checklist
- All 6 scores exist for every participant, each with a reason, collected before discussion
- The Step 2 baseline and the trial used the same assignments
- The security gate was passed, or the product was dropped
- 1 removed test account lost access
- 1 export opened in another app with dates and file names intact
- The first-year cost covers 12 months of the full-team plan, add-ons, setup and training
- The decision is written down with a date to measure again
A few weeks after rollout, run the re-assessment: do the tested tasks still take less time than the Step 2 baseline, and do people use the features they tested?
Worked Example: An Illustrative Reporting-Tool Trial
Here is an illustrative case in which every figure is invented: a weekly reporting task falls from 90 minutes to 55 in the trial, and 10 employees do it weekly.
The saving is 35 minutes a run. Over 4 weeks that is 35 x 10 x 4 = 1,400 minutes, about 23 hours, and over a year 35 x 10 x 52 = 18,200 minutes, about 303 hours.
In this example 3 participants scored the tool:
| Criterion | Coordinator | Analyst | Manager | Average | Spread |
|---|---|---|---|---|---|
| Business value | 4 | 4 | 3 | 3.7 | 1 |
| Ease of use | 4 | 3 | 2 | 3.0 | 2 |
| Workflow fit | 3 | 4 | 3 | 3.3 | 1 |
| Total cost | 4 | 4 | 4 | 4.0 | 0 |
| Security and control | 4 | 4 | 4 | 4.0 | 0 |
| Support and data export | 3 | 3 | 3 | 3.0 | 0 |
Sources: illustrative inputs, not a vendor’s price or a real customer’s results.
The 6 averages come to 3.5 overall, and the gate is passed because nobody scored security and control 1 or 2. Only ease of use has a spread of 2, so it is the 1 criterion the group discusses: the reason behind the manager’s 2 decides whether training closes the gap.
To turn the saving into a cost check, divide the first-year cost (the subscription for all 10 users for 12 months plus any one-time setup) by the 303 hours saved a year; from year two, leave the setup out. If an hour of the team’s time costs you more than that cost per hour saved, the saving on this 1 task covers the subscription.
On these numbers the decision is adopt, with ease-of-use training, if the first-year total stays under the ceiling; over it, keep the current tool.
Common Problems and Fixes
Each of these 4 mistakes voids an evaluation:
- Judging from the demo: a lost file or a slow deadline change never shows up in one. Decide on trial logs only.
- Skipping the baseline: the trial yields impressions, not a saving. Time 2 or 3 runs first.
- Discussing before scoring: a quieter participant’s concern can disappear behind the loudest opinion. Collect the sheets first.
- Pricing the entry plan: the cost jumps when the whole team joins. Price the full-team plan and its add-ons.
When a trial is already running, these 4 checks repair it without a restart.
| Problem | Possible cause | What to check |
|---|---|---|
| People return to the old app | They cannot find what they need, or handoffs need manual copying | Stuck points in the trial log, workflow fit scores |
| The trial ends before the task ran enough times | The trial is shorter than the task’s cycle | Whether the vendor extends it, or 1 paid month covers the gap |
| The export loses dates or file names | The export format drops those fields | Another format, and support’s answer on what exports |
| The security contact blocks the product late | Documents were requested after the trial began | The Step 5 document set, before more data goes in |
My take
Of these 4, the late security block costs the most, because by then the team has learned the tool and argues for it. I would hold every trial until the Step 5 documents are in.
Tips to Go Further
Start the rollout with the tasks employees completed successfully in the trial, and add the rest once those hold. Score workflow fit again when the team grows past the 3-6 people who ran the trial.
The same 9 steps narrow to 1 category in my guide on how to choose knowledge base software, which adds weights and category-specific requirements.
FAQ
How can I measure the quality of software?
ISO/IEC 25010:2023 defines a product quality model of 9 characteristics, and it names acquirers among the stakeholders who can use it for requirements specification and evaluation. For a buyer, the practical measure is narrower: whether the target task ran faster and cleaner in the trial than in the baseline.
How long should a software trial run?
Long enough for the target task to repeat as many times as you timed it for the baseline, so a weekly task with 2 or 3 baseline runs needs 2 or 3 weeks. If the vendor’s trial is shorter, ask for an extension or pay for 1 month.
Should evaluation criteria be weighted?
Only when the group agrees on the weights before the trial. Give the 5 criteria other than security and control weights that add up to 100, multiply each average by its weight, and compare products on the total. Security and control stays a gate, applied first and never weighted. I would skip weights on a first evaluation: 6 plain averages and 1 gate are easier to explain to a budget owner.




