0:00:00.780,0:00:04.200
Awesome, all right, thank you very much for having me.

0:00:04.200,0:00:07.500
So I'm Rashina, I am from Monash University in Melbourne,

0:00:08.400,0:00:12.480
and it's my pleasure to talk to you today about this topic which is,

0:00:12.480,0:00:15.300
"You asked for it: making sense of user feedback."

0:00:16.740,0:00:20.040
So as I begin I would like to acknowledge the traditional owners

0:00:20.040,0:00:23.280
of the land from which I present, which is the Wurundjeri and

0:00:23.280,0:00:27.240
Bunurong peoples of the Kulin nations, and I would like to pay my respects to

0:00:27.240,0:00:31.920
their elders, past, present, and emerging, and to everybody who is here and joining us

0:00:31.920,0:00:37.200
online from various lands around the world. Okay so I'm going to start with

0:00:37.200,0:00:39.893
something that might be a little bit of a controversial statement.

0:00:39.893,0:00:44.940
In all my research on agile, and working with agile for a really long time,

0:00:44.940,0:00:50.940
I think user stories are a modern agile myth. Why do I say that?

0:00:50.940,0:00:56.280
Basically if you really think about it, user stories are made-up stories about what

0:00:56.280,0:01:01.440
we as software developers and software stakeholders think the users want.

0:01:02.940,0:01:07.020
They're based on assumptions, so, the best assumptions we can make about the users.

0:01:07.020,0:01:10.860
They are led by convenience because, well, we're here,

0:01:10.860,0:01:16.560
and oftentimes the users are not. And the stories are obviously not

0:01:16.560,0:01:22.740
real - these are situations that we think up of users actually using our system in advance.

0:01:24.420,0:01:26.520
So this leaves the kind of everyday,

0:01:26.520,0:01:31.020
average user thinking, as a user I want to provide my own feedback,

0:01:31.020,0:01:34.080
so that customers don't have to pretend to know what I need.

0:01:35.820,0:01:38.580
Real user stories, on the other hand, are priceless.

0:01:39.480,0:01:42.660
So they're an important source of identifying things like bugs,

0:01:43.200,0:01:50.580
usability issues, as well as new feature ideas. Because what really matters is the fact that,

0:01:50.580,0:01:54.240
even though customers are the ones paying for the development of the product,

0:01:54.240,0:01:57.840
but it is the users that decide the fate of the product.

0:01:59.340,0:02:03.660
So the real user stories will tell us things like

0:02:03.660,0:02:09.180
their needs - so the needs of the users - and this is needs of users that are diverse.

0:02:09.180,0:02:13.440
And when we understand that diversity, it can lead us to building products that

0:02:13.440,0:02:17.100
are more generalizable across the human population around the world.

0:02:17.100,0:02:21.660
It also leads us to understanding human wants in terms of our preferences,

0:02:21.660,0:02:25.680
things we like, we don't like, and that can lead us to software teams

0:02:25.680,0:02:31.320
personalizing and customizing their solutions, and which, kind of, you know, leads to users

0:02:31.320,0:02:35.280
coming back to their favorite products because of that personalization aspect.

0:02:36.540,0:02:40.740
Real user stories are also an important source for generating business value

0:02:40.740,0:02:45.300
because when users see their feedback being actioned upon,

0:02:45.300,0:02:49.740
that builds trust, loyalty, and the reputation of the brand,

0:02:49.740,0:02:53.820
which ultimately leads to things like sales and profits.

0:02:55.020,0:03:00.420
Now how do we get to these real user stories if we don't want to make them up?

0:03:01.860,0:03:07.920
So we can get all sorts of real user stories or real feed - user feedback from things like user

0:03:07.920,0:03:11.280
ratings, reviews on app stores, feedback through websites,

0:03:11.280,0:03:14.700
complaints through call centers, and these are the kinds of stories

0:03:14.700,0:03:18.720
that I call proactive stories because the user has volunteered to give it to you

0:03:18.720,0:03:23.760
even though you may not have asked for it. Then there are the automatic stories.

0:03:23.760,0:03:26.760
So these are things like activity logs being logged in the background,

0:03:26.760,0:03:29.940
user likes, dislikes, and engagement, and demographics,

0:03:29.940,0:03:34.260
all of it being accumulated in the background without the user actually even finding out.

0:03:35.280,0:03:37.680
The other ones are prompted user stories,

0:03:37.680,0:03:42.660
in that the apps might be prompting people for their ratings or reviews.

0:03:43.740,0:03:47.220
And finally there is this idea of elicited stories,

0:03:47.220,0:03:51.420
which is where we purposefully collect user feedback through

0:03:51.420,0:03:56.460
these intentionally designed activities like co-design sessions, user experience sessions,

0:03:56.460,0:04:01.020
focus groups, observations, interviews, think aloud, beta testing, all of these.

0:04:02.580,0:04:06.480
So what are we really dealing with here? What we're dealing with is,

0:04:06.480,0:04:11.340
if you kind of look across all of these kind of real user stories we find something common,

0:04:11.340,0:04:16.620
which is all of these reviews and feedback forms and complaints and characteristics and so on,

0:04:16.620,0:04:23.220
well all of these are really qualitative data. In fact a majority of it is qualitative data.

0:04:23.220,0:04:27.540
So it goes from everything large scale, so to hundreds and thousands of reviews,

0:04:27.540,0:04:29.940
to more small scale, because, you know,

0:04:29.940,0:04:36.180
you'd probably be doing about tens or twenties of focus groups rather than thousands of them.

0:04:36.180,0:04:44.100
But ultimately they're all qualitative data. Now here's the other fact, which is, we usually

0:04:44.100,0:04:48.360
don't want to deal with real user feedback. And the question is why?

0:04:48.960,0:04:51.900
So some of the reasons are, well, there's lots of it.

0:04:51.900,0:04:56.580
The other is, because it's qualitative, it's usually unstructured and it's messy,

0:04:56.580,0:05:01.260
and it's hard work to try and really get your head around what all of this means altogether.

0:05:02.580,0:05:05.100
So current approaches - and I'm not expecting you to read

0:05:05.100,0:05:08.760
and understand the whole of this diagram - but you can do - you know, look at it in that

0:05:08.760,0:05:13.320
paper from Empirical Software Engineering last year that goes into the details of the steps -

0:05:13.320,0:05:18.060
but what I'll highlight here is that currently - the current approaches usually include two steps.

0:05:18.060,0:05:22.440
So one is the automated analysis, which there are automated tools which

0:05:22.440,0:05:26.340
will filter noise like synonyms and then it classifies based on

0:05:26.340,0:05:30.420
keywords that you provided, And the second step is manual

0:05:30.420,0:05:34.260
analysis of qualitative data using the sample reviews that you have

0:05:34.260,0:05:38.700
So what I'm going to propose is going to be helping with the second part.

0:05:39.840,0:05:43.860
Now, current limitations of this approach is, while this approach helps us answer

0:05:43.860,0:05:49.020
things like the "what" and the "how" it really doesn't help us answer the "why".

0:05:49.020,0:05:52.680
So that in-depth, you know, idea is missing. Also,

0:05:52.680,0:05:57.060
because we - it is keyword based, if we focus on bugs we'll find bugs,

0:05:57.060,0:06:00.720
but we'll miss the proverbial butterflies in that there are

0:06:00.720,0:06:05.100
all sorts of interesting findings in there but because we were not looking for them we

0:06:05.100,0:06:06.660
never find them. However,

0:06:06.660,0:06:10.920
because it's all publicly available data, our competitors can find these butterflies

0:06:10.920,0:06:14.640
and they can act upon it to improve their product even if we're not doing it.

0:06:16.560,0:06:20.940
So here comes social-technical grounded theory or STGT as a

0:06:20.940,0:06:26.460
powerful qualitative data analysis method. So given the time limit I'm going to show

0:06:26.460,0:06:30.720
you a very quick example of what open coding in STGT looks like

0:06:30.720,0:06:35.340
and what I'll ask you to think about is really the idea of hashtags,

0:06:35.940,0:06:39.720
especially if you are a social media user, like Twitter, you will be very

0:06:39.720,0:06:43.500
familiar with the idea of hashtags that are used to capture you know very

0:06:43.500,0:06:47.520
profound ideas in a few words. So here's a user review

0:06:47.520,0:06:52.080
that we're looking at right now. Now, if we were looking - using a keyword-based

0:06:52.080,0:06:56.400
approach - if we were looking for the word "scam" sure enough it's right there, we'll find it

0:06:56.400,0:07:01.800
and then we can report it and that's okay. But we are going to miss all of this other,

0:07:01.800,0:07:06.300
as I said, all of these butterflies, these interesting findings that we were not looking for.

0:07:06.300,0:07:12.540
If we use open coding using socio-technical grounded theory we actually have a chance at

0:07:12.540,0:07:16.980
finding out all of these other interesting ideas. So to give you an example,

0:07:16.980,0:07:21.840
the bit in the blue, I would code this as hashtag "unintentional

0:07:21.840,0:07:27.480
upgrade" because here the user is talking about how unintentionally they were led to upgrading

0:07:27.480,0:07:32.040
to a $99 pro version because their fingerprint was set in the App Store.

0:07:33.060,0:07:38.100
The other part which you see in the gray is all their attempts to come and to contact the company

0:07:38.100,0:07:42.780
using the various channels that are available so I would have, you know, kind of code that

0:07:42.780,0:07:45.660
as "hashtag multiple contact channels" which do exist.

0:07:45.660,0:07:48.840
But then the other code there is "hashtag no customer service".

0:07:48.840,0:07:52.920
So even though there are multiple channels there was no customer service provided.

0:07:53.940,0:07:58.770
Now all of this - the fact that it's a review or usually comes with a rating -

0:07:58.770,0:08:03.240
so at the bottom we see that it's a one star out of five star, so hashtag for review.

0:08:03.240,0:08:07.260
In other words all of this come together makes up a cohesive story about the fact that

0:08:07.260,0:08:10.560
this is a frustrated customer who just didn't like the product,

0:08:10.560,0:08:14.280
it was a scam in their opinion and in their experience,

0:08:14.280,0:08:17.280
and there were a couple of other interesting things that we found

0:08:17.280,0:08:20.160
on top of the fact that we just classified it as scam.

0:08:21.840,0:08:26.820
Now how do you scale this? So I'm going to show you, you know, how

0:08:26.820,0:08:32.580
we go from raw data to this idea of code and then group codes together to come up with concepts.

0:08:33.420,0:08:38.760
So here for example a review said, "App requires you to have location services always on,"

0:08:38.760,0:08:42.420
which is coded as "hashtag forced location sharing".

0:08:43.080,0:08:47.160
The other is, "You have no option but to give them consent to sell your data to a third party,"

0:08:47.160,0:08:50.280
and this is forced consent to policy.

0:08:51.540,0:08:55.320
The third one is, "My account got hacked and another phone was registered",

0:08:55.320,0:09:02.220
and that would be, like, obviously hacked account. If we look at all these codes all together and we

0:09:02.220,0:09:06.420
constantly compare them and take it to the next level of abstraction and group them together

0:09:06.420,0:09:10.260
we're talking essentially about this idea of privacy issues.

0:09:10.260,0:09:11.760
So remember, we started off

0:09:11.760,0:09:16.080
looking for scams but we also found privacy issues because we were using an open approach.

0:09:16.860,0:09:22.320
So if you use a socio-technical grounded theory for qualitative data analysis,

0:09:22.320,0:09:27.240
you're going to be able to systematically analyze data for individual classifications,

0:09:27.240,0:09:30.360
but also for data-wide, you know, patterns that you can find.

0:09:30.360,0:09:34.140
It can lead to rich insights and guidelines and recommendations,

0:09:34.140,0:09:40.200
for example on improving privacy in this case. But for those of you who are more interested in

0:09:40.200,0:09:44.700
coming up with research outputs as well, say for example you're a user experience

0:09:44.700,0:09:49.200
research team within company, you can use this in a more rigorous

0:09:49.200,0:09:52.920
application to develop taxonomies, theoretical models, and theories,

0:09:52.920,0:09:55.320
which you can then report in peer-reviewed journals.

0:09:56.100,0:09:58.740
So the new approach that we're talking about is

0:09:58.740,0:10:04.140
making sense of real user stories using STGT where you still have those two steps

0:10:04.140,0:10:08.040
but instead of being linear it's iterative, for one thing,

0:10:08.040,0:10:12.120
and also because of the fact that we're using open coding in the second

0:10:12.120,0:10:17.700
step using socio-technical grounded theory we're able to find new and refined keywords

0:10:17.700,0:10:21.240
that feed back into the automated analysis and goes back and forth.

0:10:21.240,0:10:28.440
And this allows you to achieve both depth and breath in your analysis of user feedback.

0:10:29.160,0:10:33.660
So the benefit for user software teams is that now you can really work

0:10:33.660,0:10:39.480
with smaller classified piles of user feedback that can be uniquely put under different labels.

0:10:39.480,0:10:44.340
It will lead to richer analysis, to draw out interesting findings, so you catch

0:10:44.340,0:10:48.180
all the bugs that you were looking for but you also don't miss the butterflies.

0:10:48.960,0:10:52.680
The benefit for users of course is it increases their confidence in updating

0:10:52.680,0:10:58.080
apps - they give you better ratings - and benefit for companies, of course,

0:10:58.080,0:11:02.100
as we saw before is because you're actioning the user feedback it increases the trust

0:11:02.100,0:11:06.300
and therefore increases sales. And I'm going to end with this which is,

0:11:06.300,0:11:11.580
if you want to find out more you can use this QR code and or type in my website

0:11:11.580,0:11:15.720
and there you'll find the actual key paper that talks about the guidelines.

0:11:15.720,0:11:19.500
There are also a couple of videos on technical briefing but if you're coming

0:11:19.500,0:11:22.500
to ICSE in 2023 in Melbourne, we'll love to welcome you,

0:11:22.500,0:11:26.520
but I'm also giving a technical briefing on socio-technical grounded theory there.

0:11:26.520,0:11:31.380
And there's an upcoming book. And very quickly I'd love to thank Dr Omar Haggag

0:11:31.380,0:11:34.500
who helped me with this and Michael Hoye, so thank you very much.

0:11:35.220,0:11:38.760
All right, thank you very much Rashina, if you'd like to end your screen share,

0:11:38.760,0:11:40.500
we do have one question coming in.

0:11:42.600,0:11:46.380
One of our viewers would like to know are there any special concerns

0:11:46.380,0:11:50.340
or risks here with confidentiality, with loss of privacy and so forth,

0:11:50.340,0:11:55.440
in doing this kind of analysis compared to, for example, conventional web metrics.

0:11:58.020,0:12:04.140
So, what I'll say there is, that we are in this instance dealing

0:12:04.140,0:12:09.600
with data that is already publicly available, so people have chosen to share that online

0:12:10.800,0:12:14.940
and it's already in the public realm. We also use socio-technical grounded

0:12:14.940,0:12:19.260
theory for more confidential research wherein you're doing interviews and so on,

0:12:19.260,0:12:24.960
and in that case what you would do is, you will try to de-identify the data,

0:12:24.960,0:12:29.040
in that you would remove all of the details that would - could be potentially identifying,

0:12:29.040,0:12:31.920
either the company or the individual or any of those things,

0:12:31.920,0:12:36.180
but you can still analyze it to find the key issues and the key problems