Reliability & validity in psychometric testing: The line between science and propaganda

Data-Driven Hiring or Data-Decorated Decision Making?

In most organizations, psychometric assessments are treated as credible by default. Most assessments available in the market are designed based on personality theories & existing personality inventories or questionnaires. If a personality testing tool produces a detailed report with graphs, personality labels, and scoring scales, it is assumed to be effective! That assumption is dangerous.

Have you checked:

• Is the test proved to be reliable?

• Is it proved to be valid?

• Has it been proven to predict job performance (or is it just marketed well)?

Without these answers, “data-driven hiring” quickly becomes data-decorated decision-making.

What Do Reliability & Validity Actually Mean?

At the core of any credible psychometric test lie two non-negotiables:

Reliability refers to consistency. If a candidate takes the same test after a time gap, the results should be stable. If scores fluctuate wildly, the test is unreliable and therefore untrustworthy.

Validity refers to accuracy. Does the test actually measure what it claims to measure? A leadership assessment, for example, should genuinely evaluate leadership traits; not communication skills & confidence (or test-taking ability masquerading as leadership).

A test can be reliable without being valid (consistently wrong), but it cannot be valid without being reliable.

What Real Science Looks Like (And Why It’s Tough)

Psychologists don’t just “design” tests. Before a test earns the right to be used, they test it on target population for reliability & validity.

They check:

• Whether results remain stable over time. (Reliability)

• Whether different parts of the test measure the construct which the test claims to measure. (Validity)

• Whether scores actually correlate with real-world outcomes like job performance.

They run pilots, collect large datasets, do statistical analysis, compare groups, and refine repeatedly. This process takes time, expertise, and rigor; which is exactly why not every tool in the market truly meets that bar.

Real Psychometrics Takes Patience. That’s Why Shortcuts Exist

In the field of psychology, no test is taken seriously without extensive validation using:

• Large sample testing

• Statistical analysis

• Correlation with real-world outcomes

• Continuous refinement

This process is slow, expensive, and methodologically demanding, which explains why many tools in the market quietly bypass it or present partial evidence as complete validation. You can imagine this as something like, launching a medicine into the market without clinical trials, efficacy data, or safety testing; people may trust the label, but there is no scientific evidence that the medicine actually works as intended or produces consistent outcomes.

Now, even if a test is based on an existing credible personality inventory, that does not transfer its scientific credibility automatically. Even small changes like rewording items, changing the number of questions, altering response scales, or moving from offline to online can affect consistency. A test that was reliable in its original form can become unstable in a new version. So, unless reliability is rechecked, we are assuming consistency without evidence. A test validated in one context (say, Western populations or a different industry) may not be valid in a different geography (e.g., India vs US), for a different role (e.g., leadership vs entry-level hiring), for a different purpose (development vs selection).

Validity is not a permanent property of a test; the evidence is tied to how and where it is used. So, the moment we modify items, change language, alter the length, combine multiple tools or change the context, we are no longer using the original validated instrument. We have effectively created a new version, and that version needs fresh evidence of Reliability & Validity. Using an “inspired by” test without retesting it rigorously is like taking a proven medicine, changing its composition slightly using other pharmaceutical ingredients, and assuming it will still work the same, without testing it again.

When Psychometric Testing Becomes Propaganda

Here’s where it gets uncomfortable.

When reliability and validity are missing or simply not verified extensively, psychometric testing stops being a scientific tool. It becomes a justification tool. It gives decisions a data-backed appearance without necessarily being data-driven. Labels like “not a cultural fit” or “low leadership potential” start sounding authoritative, even though the underlying measurement is questionable. And because the output looks structured and scientific, it is rarely challenged.

Safe to say that companies who opt for such tools have clearly fallen prey to propaganda.

Why Are Companies Paying for Unproven Tools?

Every hiring decision carries a cost.

Assessment tools, vendor contracts, recruiter time, and the cost of a bad hire – everything adds up. Organizations are under constant pressure to optimize hiring budgets and improve ROI on every hire. Yet, money continues to flow into psychometric tools that are rarely questioned at a scientific level.

If a test lacks reliability and validity:

• It doesn’t improve quality of hire.

• It doesn’t reduce attrition.

• It doesn’t strengthen decision-making.

So, food for thought: what exactly are companies paying for?

What HR Needs to Start Asking Now

If psychometric tests are influencing hiring decisions, HR leaders need to get far more critical:

• Was the test conducted for a target population of our industry, geography, and role type?

• Where is the published evidence of reliability and validity?

• Does it actually predict performance, or does it just describe personality?

• Are we interpreting results correctly or are we just accepting them at face value?

Do these questions feel uncomfortable? Well, they should.

Because the cost of not asking them is, hiring decisions built on assumptions rather than evidence.

A More Honest Way Forward

Psychometric assessments aren’t inherently flawed. When built and used correctly, they can add real value in bringing structure, consistency, and insight into hiring.

However, without reliability and validity, they lose their scientific foundation. When that happens, they don’t just become ineffective; they become misleading. Hiring is too important to be guided by tools that look scientific but aren’t held to scientific standards.

So, the problem isn’t that psychometric tests are used in hiring; the problem is that they are rarely challenged. We need to ask ourselves, when a tool looks scientific, do we stop questioning it?

Because in HR, that’s all it takes for guesswork to quietly become policy.

DISCLAIMER: The views expressed are solely of the author and does not necessarily subscribe to it. will not be responsible for any damage caused to any person or organisation directly or indirectly.