For
Alyssa Wise: Thanks so much everyone for for joining us. As you probably just saw an announcement. We are going to record.
Alyssa Wise: This meeting for most of you who are participants. This doesn't mean much since you're not going to be on
Alyssa Wise: audio or video if you do raise your hand and you put your voice to ask the question, please know that you are being recorded. If you prefer not, you can ask the question in the Q AMP. A and have no recording of you at all.
Alyssa Wise: It's very exciting to be hosting our first webinar versions with the students like it will be the future of some of them for sometimes come
Alyssa Wise: What's exciting is we normally have these events here at NYU, and I see a whole bunch of NYU attendee some of our regular folk, but this is also a talk that is tied to the IPS learning sciences conference which would have been happening in Nashville, Tennessee right now.
Alyssa Wise: And so we've got a whole bunch of people I see from
Alyssa Wise: IU North Carolina California and Italy part of the learning sciences community. And I also see a couple of names of people from the learning analytics community and none of that would be possible if we were not able to
Alyssa Wise: Be doing it online. So I guess that's a bit exciting isn't it everything else.
Alyssa Wise: So that further ado, let me jump in. As you can see from the welcome flies, feel free to keep saying hello in the chat. Feel free to
Alyssa Wise: Send a message to people if you have interesting thoughts as things come along. Make sure when you're sending a message to go ahead and
Alyssa Wise: Make sure to send it to all panelists and attendees. So everybody sees it. Otherwise, if you send it to just panelists. Only I and a couple other people will see it.
Alyssa Wise: There's a Q AMP a function of the bottom where you can ask questions will be killing those up to answer. Towards the end of the session portion
Alyssa Wise: And feel free when we do the discussion. If you want to actually talk, raise your hand in, zoom, and we will give people the mic and we're going to get priority people who've given us a sense of their question and you and I just so we can keep things interesting and coherence.
Alyssa Wise: All right.
Alyssa Wise: So enough of that.
Alyssa Wise: Let me make sure that I slip on my camera.
Alyssa Wise: Hopefully you can see me now.
Alyssa Wise: And
Alyssa Wise: What I want to do is just sign up to the talk. So we've got a three part
Alyssa Wise: Thing for you today. And the first part is going to be a little introduction by me about the connections between learning sciences and learning analytics.
Alyssa Wise: And then mean Reza, who is a PhD student here at NYU so present on the core work which is looking at professional identities and mixed methods, data mining
Alyssa Wise: And then David shares with from the University of Wisconsin has graciously agreed to share some comments or thoughts on that and then we'll open it up for discussion.
Alyssa Wise: So I wanted to frame this in the larger question of learning analytics millennial sciences and point out that well some people have said there's lots of exciting potential for learning analytics to show us big
Alyssa Wise: Patterns and large amount of data off of the from tech enhance learning a lot of people are concerned, these, these nothing don't give a window into really deep interesting processes of learning.
Alyssa Wise: And I will see you know professional identity development, the thing we're talking about today is certainly a deacon complex process. So the question is,
Alyssa Wise: Can we do that or, more importantly, how can we do this. So I would say that there's three three key concerns that I've seen people put out, which is that data mining focuses on algorithmic processing over human insight.
Alyssa Wise: That we sort of create these generalized structures versus dealing in depth with the contextualize processes of how people develop identity.
Alyssa Wise: And that it's just empirical findings and there's no theory building. And what I want to argue is that it doesn't have to be either or we can do both.
Alyssa Wise: Specifically, if we want to think about how algorithmic processing and humans that come together. I want to point out, and somebody will show this in the talk. There's so many decisions that are made when you're going to do a learning analytics method.
Alyssa Wise: We then. So that's sort of the human part. Then we go and apply these things. But then in the end we have to interpret what's there.
Alyssa Wise: And we're not just interpreting patterns in the works with this mixed methods data money we're using the analytic results as a way to go in and look deeply at the data.
Alyssa Wise: And these are just sort of examples of how we do that. That Simeon will show in her talk
Alyssa Wise: I also want to point out that in the the Gulf. We often see between quantitative and qualitative methods learning analytics is actually deeply embedded in prophecies
Alyssa Wise: It looks at regularities but also look both at generalized insights and very particular things for individual learners.
Alyssa Wise: It doesn't just confirm things but it explores them and often the analysis attend itself isn't merged. So in some ways it's perfectly designed to fit into this mixed methods kind of space.
Alyssa Wise: And finally, I just want to point out that the importance of theory and learning analytics is receiving more and more attention and that
Alyssa Wise: I would argue these computational methods offer powerful tools to both instantiate and look at theory.
Alyssa Wise: And you'll see today that actually are learning analytics work, give us new ideas about professional identity development that I don't think would have been possible otherwise.
Alyssa Wise: So to make these concrete. We've got a series of principles that we've been using to do learning analytics work that we claim is learning sciences aware. In other words,
Alyssa Wise: If the attendings all the deep things the learning sciences community cares about. And the one we're going to focus on today is this mixed methods, data mining approach, which is how do we use low level data I dive in deep and qualitatively at what students are thinking
Alyssa Wise: analytic results.
Alyssa Wise: So that is the end of my introduction and I'm excited to hand over to means is going to present to you exactly how we did this with some really exciting data.
Alyssa Wise: I mean,
Yes, hello everyone.
Sameen Reza: I'm trying to set up my slides. I hope you can see them and you can see me also.
Sameen Reza: I'm very happy to present the scissors today, which is about professional identity development to mixed methods, data mining of student deflections
Sameen Reza: I'm very thankful to all of you who could join and my collaborators on this project are Dr. Lisa wise and doctoral candidate region hand and I'm also very grateful to Dr. Lisa for giving me this opportunity to work on this project and also for organizing today. Today's event.
Sameen Reza: So professional identity in healthcare professions comes across like a week and an abstract concept a slack so very precise definition.
Sameen Reza: But it basically is described by the way it manifests itself in behaviors of the professional and in that way it was described as stealing part of an identified group by sharing values and beliefs.
Sameen Reza: But from the medical literature we get that the progression of these values and beliefs was not enough the professionals need to internalize the non of the profession to start thinking acting and feeling like a physician.
Sameen Reza: And it seems that this kind of integrated professional identity sort of has them efficiency translate their knowledge into action. And you can imagine in a critical
Sameen Reza: Profession like healthcare and this is very important to this is sort of a critical foundation to address the gas and what students know and how they actually perform and that is one of the reasons
Sameen Reza: For which identity is being focused as much as competency in healthcare professional education.
Sameen Reza: So despite mean very important concept. It is also one which was difficult to identify and to recognize and from literature we get these two models which help us understand a little bit about how professional development actually takes place. So this first model is a more
Sameen Reza: Is an earlier model coming from medical literature and it's sort of defines professional development at a pace and stages.
Sameen Reza: So it starts from students, knowing the value of a profession to showing them under supervision to actually doing them without supervision.
Sameen Reza: And finally, going on to internalizing those concepts as part of themselves to reach the stage of becoming a professional
Sameen Reza: The other model is more like a closing of a gap that sees development as a difference between how one sees himself in comparison to members of that profession. So for instance, well integrated
Sameen Reza: professional identity could mean a person's core values to be exactly aligned with those defined by the profession.
Sameen Reza: So although these two models who are different views on the development, but they both agree that it is a professional identity development is a very slow process.
Sameen Reza: And it is triggered by professional experiences and it is facilitated by reflection on those experiences.
Sameen Reza: So we find that the use of these models in previous literature has focused on a quantitative evaluation of her professional identity.
Sameen Reza: And the source typically done by using either self report measures or by evaluating students responses to hypothetical situations.
Sameen Reza: So such research did not have to explain why and how the development of taking place and why differing results were found. And it also reported mixed results about when development happens
Sameen Reza: So most of these studies used cross sectional data, which provides analysis of how professional identity development happens
Sameen Reza: So in contrast reflective writing is students respond to situations which they have written real situations which they have faith.
Sameen Reza: And it was more authentic and that way also. It was written in an ongoing manner which provides the basis for it fuels and longitudinal studies.
Sameen Reza: And finally, or it writing evolve is the communication of thoughts so it can identify important elements of how students think and learn over time.
Sameen Reza: So I believe that the four most important contributions of our current study are that firstly instead of measurement investigate how professional identity development takes place.
Sameen Reza: And it twice the students own perspective on professional identity, rather than a textbook definition.
Sameen Reza: Finally accusers reflection texts as a tool to study into how students think and of the also sort of thing that our quantitative approach as a way into the large Corpus of Texts provide a new methodological contribution to this field.
Sameen Reza: So our research purpose was defined in these two research questions and what are the key concepts related to how students think about becoming a dentist. And how do these concepts change over time.
Sameen Reza: So our research on Texas Tech students enrolled in a four year dental education program submitted their reflections using an online portfolio system.
Sameen Reza: As part of their professional development program which took place over the course of the four years and they were asked to submit their reflections in these four categories related to their courses competency advanced knowledge and professional development statements.
Sameen Reza: So we extracted all this data for a complete batch of students, which consisted of 378 students over a period of four years.
Sameen Reza: And I thought, we got a total of 284 20 reflections of all types, but the more the main focus of our analysis was 12,564 professional development statements.
Sameen Reza: And to give you an example. This is how it looks like the professional development statements are typically 10 to 15 sentences long and they describe the professional experiences that students go through
Sameen Reza: So with that, this was our Analytic process which sort of divides our Analytic process is basically
Sameen Reza: consisting of two main steps. So the first one is that of concept identification.
Sameen Reza: And then concept identification, we use all the sections statements submitted in all categories. The core competency knowledge and professional development statements.
Sameen Reza: And then the second main step was that of concept characterization in which we focus on the professional development statements submitted at the start.
Sameen Reza: Which is the first year of the program. The middle which is the start of the third year and the end with is the end of their fourth year in that program.
Sameen Reza: And these were used to build concept networks and finally we expected value 15 sentences per concept to sort of look deeply into the labeling and the interpretation part. So I just explain each of these steps in more detail.
Sameen Reza: So coming to the first step, the concept identification. The first part of the step of building a word embedding model.
Sameen Reza: So a word embedding body is basically a way to convert word into vectors into multiple in multi dimensional space.
Sameen Reza: And why there are a lot of ways that one can do that. But what a word embedding what a word embedding modeling does is that it allows words which have similar meanings to
Sameen Reza: Have a similar representation and this sort of come to a place close together and that multi dimensional space. So, such
Sameen Reza: Word embedding Marty can be created from scratch or one can use pre trained vectors so new and what is need very large amounts of data of the order of billions of words to train, whereas preteen models requires some kind of tuning.
Sameen Reza: So since we have around 6 million instances of 11,604 unique words we use a pre trained data made available by fastest, which is an open source library.
Sameen Reza: And we use all these 6 million words and the reflection data to fine tune that model and we represented each word in a fifth dimensional space.
Sameen Reza: So with that, we come to our second step, which was of clustering. So, since these two words were placed
Sameen Reza: In proximity to each other based on their youth in text. So it made sense to cluster them.
Sameen Reza: And for clustering. We did this P processing steps. The first removing the stock words, these punctuation marks and other meaningless character.
Sameen Reza: And then since the whole the complete quarters of tests which have been too much for interpretation and comparison. So we selected the top 127 words after examining major drop and usage frequency to cluster.
Sameen Reza: We use the agglomerated clustering using what's method and we got around 13 to 18 plausible cluster solutions.
Sameen Reza: So we looked at these classes solutions both quantitatively and qualitatively and we selected the 18 cluster solution after examining the constituent words.
Sameen Reza: And the words in each cluster range from two to 18 plus per cluster and this now our concept of our terminology of concept refers to each cluster of words. So as we had identified the set, we got 18 concepts.
Sameen Reza: So they were interesting insight and the data about how we got interesting insights of how the words will use one would imagine professional and professional
Sameen Reza: To be kind of similar words but we saw that are clustering process. Place them and actually different clusters.
Sameen Reza: And then when we looked at it in detail. We saw that the singular form of the word of us.
Sameen Reza: Occurring more with becoming believe important means and ethical words which shows or if a person I mentioned to becoming a professional
Sameen Reza: While the purest form of the word profession professional occurred with dentists and healthcare professionals responsibilities as it sort of shows a dimension of they're referring to a community of professionals by that.
Sameen Reza: So with that, our second step of concept characterization here our unit of analysis was a sentence because we believe that a sentence is one is the smallest unit of complete and coherent thought in text.
Sameen Reza: And we use these sentences submitted at the start and the middle and the end of the program to build these concept networks for each of the concept for each point in time.
Sameen Reza: So, in that the words of the concept sort of represents the nodes and the nodes represent the words and the node size is represented by the sentence based frequency of the word.
Sameen Reza: And as as defined by the CO occurrence of two words from the concept occurring in the same sentence and against again the edge rate is based on the sentence base frequency
Sameen Reza: So after we built these concept networks. We use them for the process of labeling and interpretation and that
Sameen Reza: We sampled 150 sentences for each concept at the start and the middle and the end of the program. So this thing was done in a way that the sentences contained two or more words from the concept
Sameen Reza: And we sort of looked at the more important or the most the strongest connections between the word to examine each concept. So these concepts were examined first within that a certain data point in time just to make sense of how
Sameen Reza: How they come across in youth. This was done by two researchers done independently and then in conjunction and then
Sameen Reza: We sort of looked at the V sort of grew a basic description of what that concept look like in a sense, at a certain point in time. But then we also
Sameen Reza: sort of looked at that concept across the different data points in time to see if any other emerging three teens came up as students use the same words.
Sameen Reza: And the same words at later points in time. Finally, we looked across concepts and saw how these concepts with similar or different to each other.
Sameen Reza: So this is how basically we use those networks and the examples from year one examples from year three examples from year for to define the emerging themes and then we finally do up a label and an interpretation for each of these concepts.
Sameen Reza: So coming to our findings with out of the 18 concepts that we had identified, we found that 12 of them actually related to professional identity. The other six or more about this, the experiences of the students in the dental school
Sameen Reza: So out of these 12 the, the major themes that emerged with competence and professionalism containing five concept three concepts along into what it means to be a dentist to about self awareness and to about community and culture.
Sameen Reza: So the five concepts in competence and professionalism. They described values associated with the dental profession reflected in these two
Sameen Reza: Concepts, then the reflection on the life long goal. And finally, a strategy to attain competence and knowledge. So to show you, certain examples.
Sameen Reza: So this has been a professional means upholding the standards of ethics and doing good. While become a competent dentist is more about their patient interaction and their work in the field.
Sameen Reza: So what it means to be a dentist defines the boat. They're brought his responsibilities, a dental health professional and also their specific role as care providers.
Sameen Reza: Development self awareness, besides the kind of awareness of the kind of dentists they wish to become and also displays. They are increasing confidence in their abilities.
Sameen Reza: And finally in community and culture they describe both belonging to a broader community of professionals as well as a smaller and more close knit community of their faculty peers and colleagues.
Sameen Reza: So after doing this. We tried to see at the end of their program which were out of the self concept. How were they used in terms of prevalence
Sameen Reza: And this analysis because the number of words, different in each concept we use the top two words in each concept to make this comparison. And we found that there were five low prevalence concept.
Sameen Reza: Five moderate prevalence concepts and by far, the more and the highly prevalent concepts were to namely providing the best patient care and becoming a professional
Sameen Reza: So then we look at these concepts across time and we saw that among the highly prevalent concepts we saw that providing the best patient care shows and increase
Sameen Reza: In us around the middle of the program while becoming a professional is sort of a consistently highly useful concept throughout those three points in time.
Sameen Reza: From the majority prevalent concepts we saw that knowing oneself as a dentist is consistently us, whereas confidence through progress against shows a marked increase around the middle of the program, followed by another small increase towards the end.
Sameen Reza: qualitatively we saw that becoming a professional this the statements of the students that chain in a way that
Sameen Reza: From providing broad and abstract definition, the fans condition to providing concrete views on what becoming a professional means to them. And this is the qualified by inclusion of ethical perspective as well as a personal dimension to the concept
Sameen Reza: And providing the best patient care. We saw we saw them shift in focus from self as provider to an incorporation of patient needs and concerns.
Sameen Reza: And finally, as in responsibility as as dentists. We saw them transition from using the third persons to a first person of us, envy, which shows sort of an ownership and a sense of belonging to the community of professionals.
Sameen Reza: So with that, we think that although we then it was not a purpose or intention and the study was also not set up in a way to sort of compare the two models or to validate or reject either one of them.
Sameen Reza: But we did see that the students have perspective of professional identity alliance with
Sameen Reza: The closing of the gap model of professional identity development and we can see this in the way that the setup the field of dentistry.
Sameen Reza: And the expectations from that field. And then the self evaluate themselves as becoming a competent dentist and then the strategize towards achieving that goal by gaining skills and knowledge.
Sameen Reza: We also find that they define new personal ways of becoming a professional and this has been in these two concepts becoming a professional which shows the personal dimension and also understanding themselves better as a dentist.
Sameen Reza: These found new indicators of professional identity development specially providing the best patient care which changes and focus towards and inclusion of patients. The role in the treatment process and also in their confidence which increases as their patient interaction increases.
Sameen Reza: We finally found that the responsibility as dentists is defined by broader definition, including their responsibility to gain trust and respect of their patients and also as their role as educators of
Sameen Reza: Sort of providing preventive oral care educating patients about preventive oral care, rather than just treating, treating them.
Sameen Reza: So important conclusions from our studies are the students see themselves as closing the gap to become a professional in multiple and varied ways
Sameen Reza: But also they are sort of redefine and reconstruct this they are based on evolving personal view of becoming a professional, we find that patient centered care is an important component of professional identity and increase confidence is an important indicator of this development.
Sameen Reza: So this research world from implication for practice. We think that professional identity needs to be observed and recognized in more nuanced ways
Sameen Reza: Rather than a single quantitative skill and a multi dimensional model can help identify small developments as they take place in different aspects.
Sameen Reza: So teachers and mentors can use this information to provide leverage to every support and focused areas and they can support professional identity development in more personal ways
Sameen Reza: So our future work around this would
Sameen Reza: Be an examination of trajectories of identity development by individual students so that we can provide some sort of individual feedback and support to each student and this will be done in consultation with the faculty at the school.
Sameen Reza: And it would also be interesting to know how our approach can be patterned using other reflection texts and
Sameen Reza: healthcare, education and to see if we can find some similarities and differences in terms of what we have found as the indicators of professional identity development across disciplines.
Sameen Reza: So with that, I would like to thank you all. And if you have any questions, I would encourage you to type them in the question and answer panel and we will be answering them shortly after the discussion. Thank you.
David Shaffer: Everyone. So let's see. You want me to jump in, or do you want, do you want to leave the moment for some questions about the paper before I start
Alyssa Wise: I want to have you do the discussion now and then we'll have a conversation at the end that can kind of bring everything together. Super.
David Shaffer: Look forward to it.
David Shaffer: Alright well. Hi, everyone. I'm David Williamson Shaffer from the University of Wisconsin.
David Shaffer: I'm delighted to be here. To be fair, it's cold right now. So I'm actually delighted to be anywhere meaningful.
David Shaffer: But really more important. I'm delighted to talk about this paper and the issues that it raises up personally I don't love it when discussing use a paper as an excuse to just talk about their own issues and ideas.
David Shaffer: But I do think that it's important for me to declare my own position ality relative to this paper so you can make sense of the comments that I'm making.
David Shaffer: A broadly speaking, I'm interested in quantitative ethnography, which is a unified method of integrating quality qualitative and quantitative research.
David Shaffer: And actually, more specifically, I'm in the process of completing a paper for the upcoming International Conference on quantitative ethnography about coding and the issues that it raises. So this is a really timely timely paper for me and my thinking about these issues.
David Shaffer: You know coding is clearly becoming more and more problematic or at least challenging in different ways, as people move researchers moved to working with larger and larger data sets.
David Shaffer: Just as a quick example. We had a quantitative ethnography coven data challenge last month where 100 people from around the world came together to gather and then examine
David Shaffer: Data Sets related to code and the culture that is creating as you might imagine, number of people were using Twitter and other publicly available data sets.
David Shaffer: And you know this was a data challenge. So they had like one week to gather the data and then make sense of it and come up with analyses.
David Shaffer: And pretty much routinely. What happened was the tier teams would start out with great momentum and they would kind of get stuck when they got to the point of actually
David Shaffer: Coding the data. This was data. They weren't necessarily familiar with. There was a lot of it. And so the problem is, you know, how do, how do you identify what the meaningful codes in the data might be, what are the meaningful issues.
David Shaffer: Just by way of broader context to situate this this paper in that problem. There are a number of approaches to code finding
David Shaffer: These days in the broader analytics communities, particularly learning analytics, all of which are kind of variants of natural language processing. Probably the most common is Layton.
David Shaffer: Allocation which you probably have heard of under is more common moniker topic modeling is a generative statistical model that essentially looks at words and documents and tries and
David Shaffer: Tries to come up with collections of words that could regenerate a similar set of documents. They're also number of vector based approaches.
David Shaffer: The idea and some he did a really nice job of summarizing this is you turn words into vectors in a document space and then do a dimensional reduction.
David Shaffer: So, for example, word to Vic has the words embedded in their local context.
David Shaffer: Latent Semantic analysis or LSA has words embedded in some very large corpus of documents as some mean just said, often a very general one like the
David Shaffer: TOUCHSTONE applied sciences associates documents or tosic corpus.
David Shaffer: My understanding is that what this paper is done is use a sort of a variant of LSA where the pre existing word vectors are condition based on words as a cluster in their original documents.
David Shaffer: But the point about all these approaches, is it they discover clusters of statistically similar words based on some measure of their co occurrence in documents. This is what Simeon is means when she says modeling the words in you use
David Shaffer: The problem with all these approaches with these data mining approaches is that they find patterns of words, often very quickly.
David Shaffer: But as we all know, especially from the learning sciences perspective data is not synonymous with meaning.
David Shaffer: And its meaning that we that we care about and and that's where I personally see the biggest contribution to this work.
David Shaffer: And to be clear, it's not that I don't care about dentistry, or about professionalism or generally I care a little about the former and quite a lot about the latter actually
David Shaffer: But in NLP methods generally statistical where machine learning techniques identify set us to identify sets of words which may or may not be associated with some probability is being part of a group or a topic.
David Shaffer: And then researchers look over the words and try and identify some Gestalt they try and understand what's going on with this collection of words.
David Shaffer: That is the machine identifies the topics and we as researchers, more or less, guess what they mean.
David Shaffer: And then this work that work, we're looking at today is timely because it takes it and importantly, different approach.
David Shaffer: And that is uses the quantification of the words quantification of the word patterns only as a first step in a systematic approach to connecting the quantitative approach, which is the word vector ization to a more qualitative reading of the actual data.
David Shaffer: That is the machine generate a set of related words based on co occurrences and now Simeon her co authors are using this quantitative analysis to selectively sample the qualitative data.
David Shaffer: And then use that use that selective sample of the qualitative data to generate the actual codes. And then, of course, due to subsequent analysis from there.
David Shaffer: And from my point of view this is in the best tradition of quantitative ethnography.
David Shaffer: That is if quantitative ethnography. We're old enough as a field to have its own traditions, but in particular, it highlights the weaknesses that all
David Shaffer: Data mining techniques have for this process of code, finding and that is making sure that things mean what we think they mean
David Shaffer: It illustrates a way of using data mining responsibly as a place to start looking for the meaning of the patterns.
David Shaffer: That the machine is identified, recognizing, of course, that some of those patterns may not be meaningful or may not mean what we thought they mean when we first looked at them.
David Shaffer: And I guess.
David Shaffer: In the, in the spirit of keeping my comments brief, um, I just like to tag on
David Shaffer: A sense in which I see this work as a provocation to future work in a couple of different ways. And to be clear, I want to be clear that I know that the authors have thought of the issues that I'm about to raise this is not going to be news to all this, or some you know RJ
David Shaffer: But I'm going to be so bold as to suggest that connecting this work more closely to other work and quantitative ethnography could actually provide some possible future directions. In addition to some of the things that Semyon laid out.
David Shaffer: On the substantive side of pursuing our understanding of the identity. Identity development in dentistry.
David Shaffer: I guess the first suggestion would be to think about using more in the way of network analyses. So this work, as you saw us as network analysis to visualize words within the machine language topics.
David Shaffer: But then once the topics have been identified qualitatively and understood qualitatively the analysis, more or less focuses on a coating and counting approach.
David Shaffer: And I guess I'm suggesting that a network analysis might be more effective than that for two reasons, or at least shed different light on it for two reasons.
David Shaffer: One is that coding and counting has been pretty consistently shown to underperform relative to network models in understanding the nature of how people make meaning. So Zach's wiki at
David Shaffer: Monash University and in Melbourne in Australia, Andreas is naughty and Frank Fisher and others in Munich there folks in my own epistemic analytics lab.
David Shaffer: Have looked at comparisons of the two methods and seeing that they essentially get more out of that network analysis and this sort of makes sense because coding and counting is actually
David Shaffer: Kind of a poor theory of discourse. Right. It's based on the idea that meaning is made by or we understand how meetings made by counting up the number of times that things occur.
David Shaffer: And the reason that's particularly problematic is is that the number of times it's something occurs in Disney and discourse in written text or unspoken text.
David Shaffer: Is completely insensitive to the order in which those things appear so literally we could take the text that these students have written and just shuffle them up by sentence.
David Shaffer: And the coding and counting model would make the same would give us the same information about
David Shaffer: What was going on in terms of their meaning making, and of course we know that's not really how humans make meaning we make meaning and by connecting things to one another, which is what the network model gets that
David Shaffer: Which brings me to sort of the second reason I think in network analysis might be useful, and that is that professional identity is clearly not just a collection of skills and values and dispositions and
David Shaffer: Habits of Mind and so on.
David Shaffer: But it's about the way these things are systematically related to one another, my culture and therefore also one's identity within a culture is a web of meanings.
David Shaffer: Simeon refers to this is a multi dimensional model. And I think that's spot on.
David Shaffer: But what the network, network model allows you to capture more effectively is not just how much of one or another thing. People are saying, but the way in which they see those things as being related to one another.
David Shaffer: In other words, I think it's possible to use network models, not just to look at the structure of connections between the codes and not just within the codes themselves as they're doing already.
David Shaffer: Ah, I guess the second
David Shaffer: Point that I think this paper speaks to is the question of validation, although I prefer the term closure because validity is is a kind of
David Shaffer: Latent term and often use not correctly. Um, but basically I think that there's an opportunity here that what this what this paper shows is a way of connecting qualitative quantitative and as I've said, I think that's really
David Shaffer: The hallmark of this contribution.
David Shaffer: But I think there's also a way to tie that qualitative data back to the quantitative data. So the quality, the quality, the validation that's going on here is good of qualitative closure of the codes is good.
David Shaffer: But one of the principles of quantitative ethnography is to actually integrate the qualitative and quantitative, not just do them sequentially.
David Shaffer: So that a key idea there is the notion of theoretical saturation. That is kind of making sure that we've seen enough data to draw the conclusion that we want to draw
David Shaffer: How do we know when we've looked at enough data. And that's what inter rater reliability actually does is it provides us a warrant that the
David Shaffer: pattern that we're describing is in fact something that is a property that data. More generally, and not just the piece of the pattern that we're showing
David Shaffer: So I think it would be useful to look at ways to provide statistical warrants that would support the qualitative conclusions both about coding and then about the
David Shaffer: The model. More generally,
David Shaffer: And I guess the third thing third provocation here is that this process of code, finding that Alyssa and and Simeon and RJ have developed
David Shaffer: Is is a really useful way of thinking about as Alyssa said that link, you know, learning sciences and learning analytics.
David Shaffer: But I think it's also a call for us to consider finding ways to integrate this code finding and other processes of coding
David Shaffer: Into a more general coding PATHWAY RIGHT NOW coding sort of the Wild West. There's like lots of different pieces that people do, and there's
David Shaffer: Not really any way to integrate them and bring them together, that is put together a sequence of best qualitative and quantitative practices encoding. It's actually something we're working on. On my, on my lab.
David Shaffer: But I think that that thinking about the way these techniques relate to one another, more generally, it's actually a really important question.
David Shaffer: I'm just quickly here in closing, because I was told to keep it to 10 minutes, um, you know, there are some other
David Shaffer: methodological limitations and issues that the authors already discussed explicitly. And I guess what I would come back to us. This is work in progress. And it's actually excellent work in progress, and it's making an important contribution.
David Shaffer: In particular, it asks us as learning sciences, not just to critique forms of automated code generation good to engage with them, it proposes and explores the solution.
David Shaffer: Doesn't just make the critique it says here's something we can do about it. And I see that is consistent with the best practices in the learning sciences.
David Shaffer: What what the authors here have done is leverage technology without ever losing sight of the situated meaning of the claims that we're trying to understand
David Shaffer: So let me stop there and just thank the authors for this great paper and obviously thank Allison so mean. For a chance to talk a little bit about it and to thank you all for being here and listening.
Alyssa Wise: Thank you so much. And there's a lot
Alyssa Wise: Of things we can do or should do and it. Yes, I have to think more about the network stuff, especially since in our mind, we were doing that were things that different than the times you described.
Alyssa Wise: But let's start with some of the questions that I've seen coming up. They've actually been coming up in the chat.
Alyssa Wise: Thank you both mean for presenting the work and David for for discussing and I think that there's a lot to deal with here as I mean, as we see the world is moving more and more online. We're going to have more and more large corpus as text.
Alyssa Wise: Where we're doing really deep and important conversations about meaningful.
Alyssa Wise: Things, whether its identity. The moment or some of the things that are going on with politics and those the situation in the world. There's a lot of deep interaction happening online. And if we're going to look at using these kinds of methods we we need to think
Alyssa Wise: carefully about how to do them. So I know that she had a question. So I'm gonna give you the, make sure you want to ask your question and then we'll get some responses.
Alyssa Wise: To the tech summit yourself. Do you
Alyssa Wise: Can you hear me.
Shiyan Jiang: Okay, good.
Shiyan Jiang: So thank you so much for this amazing
Shiyan Jiang: Presentation. I'm really excited about this work when I see the
Alyssa Wise: The panelists. I mean, the
Shiyan Jiang: The workshop is coming and I will become so but it hasn't always so great. So I
Shiyan Jiang: I studied I
Alyssa Wise: Studied
Shiyan Jiang: Other in settings.
Shiyan Jiang: And this is professional
Shiyan Jiang: Development, which
Alyssa Wise: I'm in
Shiyan Jiang: So I have my question is that I want to know more about like the different kind of language you use to modo identity, for example, here's a reflections. I have read the paper about the modeling.
Shiyan Jiang: Identity methodology was sending messages to ages and something like a. The other kind of things like a
Shiyan Jiang: They talk to others. These discussions or they send a message to not agent, but a real person. So what are the nuanced differences when
Shiyan Jiang: When you trace identity in this different kind of languages or it has to be a mean. I mean, I assume that some writing will not be a
Shiyan Jiang: Good fit because writing is more natural does not have an audience when you write to. And sometimes it depends on work on join it is. So I'm curious about this question.
Shiyan Jiang: The other question I have is that suppose you have the tracing identity. When that changes happen not likely you have a middle
Shiyan Jiang: Pre and post survey, when you have this country analogy. And what would you do to scaffold. The development of maybe positive or stronger professional identities. I'm so sorry about that. My questions I have one more question. The last question.
Shiyan Jiang: I have is about any conflicts. So I assume that students when they come to our profession they maybe they are. Do some
Alyssa Wise: Other jobs.
Shiyan Jiang: And they will
Alyssa Wise: Come to a
Professional service.
Alyssa Wise: Or they
Shiyan Jiang: May be that journalists.
Alyssa Wise: At the beginning and
Shiyan Jiang: There's some conflicts between the
Shiyan Jiang: Identity problem resolved to
Shiyan Jiang: Resolve the
Shiyan Jiang: Conflicts and how
Shiyan Jiang: Identify those are
Alyssa Wise: Resolving processes and hopefully
Shiyan Jiang: scaffold that part.
Shiyan Jiang: So I know this question, they're
Shiyan Jiang: All very slowly.
Shiyan Jiang: But there are some really interesting in general. Thank you so much.
Alyssa Wise: Thank you so much, and
Alyssa Wise: I'm going to summarize that there's a lot of questions in the comments on the first question she asked, and I'll address which one of them have a second to over to me than any day that has something to say, first question was,
Alyssa Wise: The first question was about clothes, a reflection would be written to around the identity. And I think that's the question that was a lot deeper than perhaps it sounded like at first, in terms of thinking about
Alyssa Wise: His identity. This thing that exists sort of
Alyssa Wise: inside ourselves and we're just expressing it in different ways. And we might express it differently, right, to an agent or to a human, we write it to ourselves extensively
Alyssa Wise: Or is it something that's actually constructed in our interactions with others, and this goes very, very deep into theories of what we mean by
Alyssa Wise: Identity in this case needs to these data will be written for oneself. And I think we treated identity is something
Alyssa Wise: That we we hold within ourselves, but one could argue that in fact you know we construct temporary identities in the moment. And so I think that that's a really interesting question. She on but I think it's
Alyssa Wise: It's one that we're not necessarily equipped to answer here today, empirically that this was the work with them so far.
Alyssa Wise: So I have now who's to the two questions that I think we can say something about it happens to me.
Alyssa Wise: And those are just questions I think about the five seeing what the contracts that were important for identity were to to ask, you know, how can we look at cases where someone had a stronger versus weaker identity and other cases where we might think about looking Friday any conflict.
Sameen Reza: Yes, thank you. Shan for a really interesting question.
Sameen Reza: I think they really define what our research work is going to look like. Because as I said, this is
Sameen Reza: Sort of a first step in an exploratory process of how professional I wat professional identity is and what it means and
Sameen Reza: What we're trying to do here, as you said that eventually we will be closing the loop of how this is going to be fed back into the instructional process and how students are going to be
Sameen Reza: Going to benefit from this information. So I think right now we are going
Sameen Reza: We are a bit early in that stage. So while I cannot exactly pinpoint how this will be reflected back in the instruction process. But we have some ideas. And the first one, as I said is, first of all, we need a consultation with the teachers and have
Sameen Reza: help understand each other's point of view on the same and how do they think of these properties of professional editing.
Sameen Reza: Some other thing that we saw was like community and culture and the collaborative experiences they sound the sound of logged important to us. But in our research. We thought that they were one of the low prevalence concepts in the final year and we think that this
Sameen Reza: Observations like these can be suggested to the teachers like perhaps more collaborative experiences could help them.
Sameen Reza: benefit more from the community of professionals as well as their own closer community and we think that this is going to help develop a more stronger professional identity, but we have these kinds of ideas in our mind, but we need to
Sameen Reza: Make them more precise and more accurate. Once we have done more work on it and also discussed it with the faculty of that school
Sameen Reza: Regarding your question about the conflict and people coming from different backgrounds, having different kind of identities and then
Sameen Reza: Falling in one place and going through the academics experiences. I think it's a very important question of sort of a starting place of where do you
Sameen Reza: See professional identities, starting from and I've seen I've come across literature, which is sort of identifying this issue and also seeing, especially
Sameen Reza: The one of the papers quoted in my paper is elf Adams at all. He talks about these things, like the previous experiences and background knowledge do sort of provide a very important starting place of how professional identity develops.
Sameen Reza: I'm not sure how those conflicts.
Sameen Reza: Will be resolved, but the acknowledgement that yes they are there and different students actually developed differently because of those different starting places that they have. So we hope to look at this further. When we sort of go into the individual ways in which students are developing
Sameen Reza: And hopefully with our research will keep you updated with our results to
Sameen Reza: Thank you much,
Alyssa Wise: David, I don't know if you I know you've done the related work, not necessarily on identity, but certainly on on deep looking at concepts, whether you
Alyssa Wise: Whether you think the question that was sort of strong grads entity or issues of identity conflicts with something that you put up at all.
David Shaffer: Well, I mean, I think it's really important question. I also think the question of which data that you're looking at when you're making claims about identity is is a really important one as well.
David Shaffer: I guess so. The, the nature of these questions reminds me a little bit actually of bobsleds work on
David Shaffer: Evidence centered design. Right. And so one of the general questions we always have to ask when we're making I don't call them assessments specifically because at some point when we're modeling.
David Shaffer: Is what what are the activities that have provoked whatever the data is that we're that we're looking at, and how those. How does that provocation, change the story that we're telling
David Shaffer: I interesting much of the work that I've seen when people are looking at identity development doesn't actually pay that much attention to the different modes and contexts of data that may just be my impoverished.
David Shaffer: View of the field of identity development. Um, but I think it's, I think that's a really important and useful question here. And I suspect that the conflicting identities question.
David Shaffer: That some of the answer to that might come out of looking at how identity gets represented in different contexts.
David Shaffer: People talk about like code switching is an example of that. Right. Um, as a way of seeing whether in different contexts, different facets of identity or are surfaced and whether or not that makes any any difference overall
David Shaffer: So I think those are really good questions for future exploration. I think so mean is is rightly seeing that as an important part of this work going forward.
David Shaffer: Thanks so much, David.
Alyssa Wise: And I'll just add that I think it isn't conflict so much, but I think that we saw people instead of sort of seeing professional identity as a thing to close the gap to this pre pre established. This is what a dentist is
Alyssa Wise: The notion that they were personalizing that I don't know if I would call that conflict, but they're in some sort of strong identity work going on there in terms of that they are
Alyssa Wise: Constantly redefining what it means to the dentist so they're not just taking this is what a dentist, isn't it.
Alyssa Wise: To me in a general sense as given. They're saying, Okay, this is what a dentist is. But what does this kind of has to, I want to be. And so, well, I'm not sure that always involves conflict, it can and it sort of shows a
Alyssa Wise: Deeper wrestling with issues of what does it mean to be in this case a dentist. But I think it's powerful and thinking about how people are conceptualizing their identity.
Alyssa Wise: I want to move on to another question. This one comes from Sarah back here at NYU, and she asks, this was for us to mean
Alyssa Wise: Was there any background literature that he relied on or theory regarding the practice of dentistry to inform your interpretation of the concept that the analysis surface.
Sameen Reza: Yes, thank you.
Sameen Reza: We have been looking at a lot of literature. But as I said that most of the literature was based on quantitative ways of measurement. So we came across very few actually qualitative studies which sort of could help us in the interpretation of the concept
Sameen Reza: And there were some studies, particularly, and since we are talking about, particularly the field of dentistry. They were very few regarding dentistry itself, but generally from the health care professionals, there was some data charges to try to
Sameen Reza: Show you one of the things that I put it in the chat. The one of the studies that we just see, which was a qualitative study and it was also
Sameen Reza: A mix of student debt doctors and dentists in that way. So that has somewhat, but as far as the impact interpretation of the concepts is concerned, it was more a bottom up kind of an approach and
Sameen Reza: The fight aligning to the model presented in the paper that I just put there. But as far as the independent concepts are concerned, I think we have not come across too many studies which define exact reason which students think about the professional identity.
Alyssa Wise: Thanks to me. I just wanted to add one thing on top of that to Sarah's question which is one thing I think is really powerful about
Alyssa Wise: What we're doing here is that there isn't a lot out there these models professional identities sort of go from not being able to do it to kind of be able to do it to being able to do it a more, a little bit more of them to really feeling like you are
Alyssa Wise: Dentists lawyers, etc. Which is kind of generic and what I think was the power of what we did here.
Alyssa Wise: And David talked a little bit is that we use this large corpus of what people were saying about what it meant to be a dentist.
Alyssa Wise: And and as john mentioned it's, it might be different in different contexts of data, but it was something that came from the ground and gave us a much more detailed view of what
Alyssa Wise: People were thinking it meant to be a dentist. That was really specific to that domain. And you could imagine doing this for whatever domains, you're interested in whether it's
Alyssa Wise: Business or engineering and saying that, yeah, we cannot be general models of identity development that I you know I start to become more of something. But what is it would constitute what are the key things
Alyssa Wise: That make one feel like an engineer or that make her feel like a teacher or make one feel like in this case this is, you know, some of the things
Alyssa Wise: We're not surprising. I'm glad patient centered care showed up.
Alyssa Wise: That's something that we like. But, you know, the notion that helping people achieve better oral health as opposed to just treating them. I'm not sure we would have expected that
Alyssa Wise: And some of the things that really related to serving the community were also important and and they may not be true for everybody, but the fact that these are our core components.
Alyssa Wise: starts to become interesting and then we can start to see maybe what what distinguishes one's identity as part of a different profession. So I think that even though there isn't a lot to to compare to. Yet it's hopefully the start of quite a bit more of looking at that.
Alyssa Wise: I think we've got time maybe for one more question if anyone else has anything they'd like to ask or people are just sitting there bit shell shocked.
David Shaffer: Would it be okay if I asked a question, because I'm there's some towns that I was curious. Yeah, please.
Alyssa Wise: Please do. David, yeah.
David Shaffer: So I would love to hear a little bit more about
David Shaffer: What you were what you were doing with those network analyses, especially when you were, it looks like you were showing comparisons between the networks.
David Shaffer: And of course, one of the things that's difficult about network analysis using the
David Shaffer: Sort of spring mass models that you were, you were making our way. As soon as spring I suppose that you were making
David Shaffer: Is the nodes move around and look like that maybe weren't the same nodes in the different networks. So I'm just wondering like how are you actually compare what was the comparison, like, what, what, what were you doing with them. That was was getting you that insight.
Alyssa Wise: You access to me, I'll say something
Sameen Reza: Okay, so yes, a network analysis proved to be a good starting point to look into how the students are using those different words which you make up a concept and
Sameen Reza: Basically a visual representation of those networks in terms of size of the node, which showed the obviously the frequency of the words being used and the strength of the
Sameen Reza: Interconnections which showed how they will use within a sentence were sort of important to us in the study. And those are the main things that we compared as we looked at each concept across time.
Sameen Reza: So comparing concepts across time was an easier task, but obviously comparing across concepts was more difficult and tricky and
Sameen Reza: In that sense, we sort of chose only the top two words to sort of look at the prevalence
Sameen Reza: And also the qualitative examination in which, like in the first instance, we could have thought like becoming a professional is very sort of
Sameen Reza: Similar to providing patient care. But when we looked into the exact sentences which came up, then we were able to distinguish them like
Sameen Reza: Professor becoming a professional is sort of a more global idea and providing the best for patient care is more related to how they would be performing and that role. So yes, I think the network analysis sort of guided us towards these kind of conclusions, but
Sameen Reza: It wasn't
Sameen Reza: And they were i mean i i think we saw more changes arising out of the qualitative
Sameen Reza: Things that we did, and the network connections did not take actually changed too much over time, either for for most of the concept
Alyssa Wise: Yes, I think so. I mean, laid it out very well.
Alyssa Wise: It was interesting to us that after figuring out what concepts to look at most of the interesting insight came from the qualitative, to be honest, on the on the quantitative can show
Alyssa Wise: Oh, a connection between two words and getting bigger. But what does that mean you can't tell. Unless you look at the text and so
Alyssa Wise: I think one of the challenges we face is you have to make a lot of decisions about how to process the text and that is going to drive what you look at later.
Alyssa Wise: I think with that, we're going to close the session. Big thanks to mean for presenting this work and it's unconventional format.
Alyssa Wise: A big thanks to David Schaefer for beaming in from the matrix to share his comments. And a big thank you to everyone who came in and listen and asked question or just
Alyssa Wise: Hopefully took something away from this. We're experimenting with a lot of new formats and for not being able to hold our traditional events.
Alyssa Wise: And we'd love to hear your feedback.
Alyssa Wise: This was a hybrid event where this is a research talk, but we invited our whole community. We often sort of have more research and practice oriented talk so
Alyssa Wise: If you really like this or you prefer the ones we used to do feel free to send us a note because we're going to keep planning on how to
Alyssa Wise: Keep bringing the community together to continue the important conversation about how to do this kind of work to support learning and as the world keeps going, despite it being in a slightly different way than it was before server so much, everyone. Have a great day.
David Shaffer: Thanks, everyone.
