The limits of what we can measure

Not the whole story

What is the best way to measure performance? What role should metrics play? Should measurements strive to be entirely “objective”? What is the best way to measure “reality”? These are some questions I have grappled with in recent months trying to understand students’ grading concerns in a graduate course I taught in career counseling.

In the case of the students I taught, their major concern was knowing exactly what they had to do to earn an “A” or a “B”. How did written comments on work corresponded to points that would earn them an “A”? In addressing their questions, I explained that counseling was part science, part art. Although there are performance elements that can be objectively graded by assigning points, there is a degree of subjectivity involved. While points can be assigned to parts of the performance, ultimately there is a “finished product” or a “total gestalt” which might be worth more than the “sum of its parts.”

Using an analogy of grading the Mona Lisa, I explained metrics could be assigned to evaluate elements such as the use of light and dark, the likeness, use of color, composition, etc. Judgments on specific elements have some degree of subjectively involved and then there is the question of how to grade the final impact of the work in its entirety. I didn’t even mention differing views of aesthetics. I can see why I drove some of my students crazy. Never mind that over 70% of the students earned A’s and the rest B’s – they wanted to be sure they knew how they could earn, and continue to earn, an A.

I understand students wanting to know their grade point standing, how to get it, and how to continue to achieve a desired score or rank. Isn’t a sense of surety what we all want in life? The use of metrics to help us determine how we are doing in daily life is endless. When we are presented with grades, numbers or metrics, or what we consider to be the “right” numbers, it gives us a sense that we are where we want to be and all is right with the world – or at least our world.

When a baby is born, we ask for the Apgar score – a scale that evaluates the child on key elements deemed essential in infant health. As parents, we want to know what percentage of arsenic the FDA believes is safe in the apple juice our children drink. We monitor our health by looking at our weight, body mass index (BMI), blood pressure, and cholesterol counts. We look at our credit scores to see if we can get a loan and our investments to see if we have enough to retire. We follow daily stock market numbers, unemployment figures, Black Friday sales, and the national deficit to determine the financial “health” of our country. Organizations use metrics to measure employee performance and company results; project managers use figures to compare actual time, cost, resources, scope, quality, and outcomes with projected targets. Metrics are an important part of everyday life.

When I work as an organizational consultant I often ask, “Do you have any performance metrics?” Metrics can be helpful indicators of status and progress in numerous areas. In education, metrics are often combined with rubrics. Rubrics present specific elements to be learned and spell out what a student must do to achieve different levels of performance or grades for each learning objective. Metrics combined with rubrics can encourage students to proactively participate in their educational experience and continually evaluate their progress.

The use of metrics combined with rubrics makes sense to me in many situations. However I also wonder, “When do metrics fall short?” What would Existentialist philosophy say about attempts to measure performance and “reality”? Is it possible to achieve a true measure of anything that will result in our feeling “secure”? I believe Existentialism posits “reality” is a combination of subjective and objective elements and is in a constant state of flux that changes with the world and individual. As for metrics providing “security,” it is only illusorily – the nature of human existence is fraught with insecurity.

I do not think metrics are “bad” per se; rather, I think it is our use of metrics that can get us into trouble. I think the following questions are important to consider:

1) What is being measured?
2) Do metrics accurately describe the phenomenon measured?
3) What is the best way to measure performance?
4) How will the metrics be used?
5) Do metrics tell the whole story? Or, are there times when metrics simply “fall short”?

Some qualities are difficult to define and measure. One example is “intelligence.” When I was a student teacher years ago, I was instructed to write students’ IQ scores next to their names. I was horrified by the suggestion that an IQ score was synonymous with a student’s intelligence and decided to ignore the figures.

Sometimes metrics do not accurately measure “reality”. When I designed employee opinion surveys, I was quick to point out that employee opinions converted to metrics may or not be “reality” – but it was their perception of reality. Ideally, organizations use data from surveys as a baseline to design and measure ongoing improvement initiatives. As for metrics representing “reality”, how many of us were lulled into a false sense of security as we looked at our bank accounts, 401 Ks, and investments before the economic crises of our country became evident and major organizations had to be bailed out

Because metrics are often the driving force behind how we are compensated or rewarded, people may be less than honest to ensure metrics meet expectations. For example, in an effort to produce revenue figures, some mortgage companies wrongly “qualified” applicants for mortgages borrowers were unable to pay. In an effort to produce acceptable test scores to maintain school ranking and funding, some teachers changed test answers to “improve” results. I am not suggesting dishonest mortgage lenders and teachers were the victims of metrics; they were guilty of wrong doing. I do think it is important to question if an over reliance on metrics contributes to a culture of blame and defensiveness when results fall short.

Finally, metrics often do not tell the “whole story.” As a leadership development coach, I have often been asked to work with employees who have received top performance ratings but were failing because they lacked social skills or political savvy. In the arts, there are times when it is impossible to judge a final product by evaluating various elements. Recently, I discussed how to evaluate performance with a member of the Philadelphia orchestra. She mentioned a time the orchestra performed four performances of Beethoven’s Ninth Symphony – orchestra and audience members agreed one performance had been particularly sublime but she could not define exactly why it was so stellar.

We all treasure moments in life when we are lifted from the mundane to the numinous but it may be difficult to describe or quantify those experiences. How would one grade the feeling when you first see your child smile? What would be the rubric for a spectacular sunset? How would one assign metrics to peak experiences? Hopefully, we can learn to appreciate when metrics are useful, when they fall short, and know they may not guarantee the security we seek.

— Christina Robertson

Read more articles by Christina Robertson

Keep up with our community – follow us on Facebook and Twitter