Measuring what matters: A convening

by | Sunday, May 17, 2020

All of us involved in social design (and I include education in among those as well) ask ourselves, or are asked this question:

How do we measure the impact of the work we do?

This begs the question, why measure in the first place? Lord Kelvin, one of the greatest physicists of the 19th century, provided the canonical answer to this question back in 1883, saying:

…When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.— William Thompson Kelvin (Popular Lectures, Vol. I, “Electrical Units of Measurement,” p. 73, 1883).

In other words, according to Lord Kelvin “to measure is to know” (which as it happens is another of his oft-quoted remarks).

This emphasis on measurement can be seen in all aspects of our lives. We measure our children’s heights, and our GDP; we measure learning gains in students, and recidivism rates of inmates released from prison; we measure the number of steps we walk and the calories we eat.

That said, we also know that there are things, important things, that we have not yet managed to measure, or do not know how to. These are often intangible, ineffable—and yet, important.

How do we measure if students are seeing beauty in mathematics? Or the economic value of public art? Or the capacity of music to move us?

These issues are of concern for all of us engaged in social design (i.e. the application of design methods to tackle complex human problems). In a recent blog post, prompted by the COVID19 crisis and global school closures, I asked the question about the value of school (in two parts, One & Two)– and argued that measures that focus on one dimension (that of learning, via standardized tests) miss all the other values that schools bring to our lives.

The issue, of course, is that if we can’t measure the impact of our work how can we know if what we are doing is working or not? The issue is not whether we should measure but rather if we are measuring what we ought to be measuring. Often, we are limited to measuring what we can easily measure rather than what we ought to measure. Alternatively:

Do we measure what we value or do we merely value what we can measure?

These are not new questions, but they are important ones, and worth revisiting.

Recently we organized a virtual convening of scholars and leaders from across ASU on this very topic. Titled: The Importance of the Ineffable: Measuring What Matters, the convening was organized by the Office of Scholarship and Innovation, the Principled Innovation Team and InnovationSpace. Participants came from a range of units within and outside of ASU, bringing their disciplinary expertise as well as their deep expertise into the conversation. (Complete list of colleges and units that participated given at the end.)

This event had initially been planned as being a face-to-face convening, till COVID19 got in the way. What this shift to an online convening meant that we had to get creative to ensure that the different approaches and perspectives could get properly and thoughtfully addressed and some synthesis reached.

To us that meant that we could not (and should not) just drop people into a zoom room and hope for good things would happen.

Jennifer Stein and Henry Borges put in some significant effort to design a genuinely powerful experience – with additional support from Clarin Collins, Ben Scragg, and Lok-Sze Wong. We moved seamlessly from whole group to smaller teams and back to whole group, toggling between Zoom and Google Slides and Google Draw, sharing ideas, collapsing them in to broad categories of questions and then digging deeper into them.

Screenshot from virtual card-sorting exercise. Names have been deleted.

A range of topics were generated and were virtually categorized and then people self-selected to discuss each of these overarching topics further.

Overall it was an exciting and intellectually challenging event, providing us with lots to chew on and we continue to explore these ideas in actual projects in the future.

Note: Participants included people from the following units or organizations: The School for the Future of Innovation in Society, School of Earth and Space Exploration, College of Public Service and Community Solutions, School of Computing, Informatics, and Decision Systems Engineering, Herberger Institute for Design and the Arts, School of Mathematical and Statistical Sciences, Mary Lou Fulton Teachers College, and the Kern Family Foundation.

End note:

Finally, for two slightly different, and idiosyncratic yet relevant, takes on the whole issue of measurement see these two blog posts from a few years ago:  

A few randomly selected blog posts…

The greatness of teachers

I discovered Hulu TV a few weeks ago and have been using it to catch up on previous episodes of The Daily Show. I decided today, as I was working on a presentation to watch Crawford. It is a documentary about "a small town thrust into big politics when George W. Bush...

Creativity is greatly valued: A poem

Creativity is greatly valued: A poem

Creativity is greatly valuedCreativity is greatly valued For his sharpness His ability to look beyond The surface   And willingness to Give it a shotTo break Out of the box The cubicle   And jump   When the towers burnt   Who knew Box cutters Had...

It’s all Greek to me: TPACK commercial

Last summer Matt and I created a couple of TPACK commercials for a video presentation we had been invited to make at ISTE in Denver. You can see the commercials here and here and the entire video here. Recently, Spyros Doukakis, a PhD candidate at the University of...

My favorite(?) failure

My favorite(?) failure

I was recently asked to write a chapter for a book that my colleague Ron Beghetto was editing with Laura McBain, called My Favorite Failure. Failure is never fun - and to pick one that was your favorite, is like deciding what your favorite form or torture is....

YouTube & Research

In a previous post I mentioned a new study on children and the internet recently completed by Warren Buckleitner for Consumer Reports Web Watch. Anyway, towards the end of the post I mentioned how the final report includes links to YouTube videos of the actual data...

TPACK Newsletter #39, February 2019

TPACK Newsletter #39, February 2019

Here is the latest pdf version of the TPACK Newsletter (#39, February 2019), as curated and shared by Judi Harris and her team. (Previous issues are archived here.) This issue includes 31 articles, 2 books, 39 chapters, and 14 dissertations that have not appeared...

Following up on lunar distance

A followup to my previous posting about the Italian kids calculating the distance to the moon using recordings from the Apollo Space program. As I read the story on the technology Review website, I came to the comments made by readers. One stuck out. This is what...

Good-Evil Ambigram in Pub Med!

Good-Evil Ambigram in Pub Med!

My Good-Evil oscillation ambigram design is easily one of my most popular designs - having made it to multiple publications, websites, covers of magazines, on the TV Show Brain Games... and now it has made its way into a medical research journal Frontiers of...

TPACK in the land down under

I recently received an email from Debra Bourne, IT Coordinator at St. Paul's International College in Australia informing me about some work related to TPACK being done in Queensland. Specifically she mentioned a paper to be presented at the upcoming Australian...

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *