>> THE COLOR COMMITTEE OF THE
TRANS-NIH WORKING GROUP ON WOMEN
IN BIOMEDICAL CAREERS, AND WE'RE
VERY HAPPY TO BE ABLE TO HOST
DR. MELODY GOODMAN TODAY.
DR. GOODMAN RECEIVED HER
BACHELOR'S IN SCIENCE IN APPLIED
MATHEMATICS, STATISTICS AND
ECONOMICS FROM STONY BROOK
UNIVERSITY.
SHE, FOR A PERIOD OF TIME, WAS
ON WALL STREET AND THEN SAW THE
LIGHT AND DECIDED TO GET HER
MASTERS IN BIOSTATISTICS FROM
THE HARVARD SCHOOL OF PUBLIC
HEALTH AND HER PH.D. IN
BIOSTATISTICS FROM HARVARD IN
THEORETICAL STATISTICS AND
SOCIAL DETERMINANTS OF HEALTH
DISPARITIES, AN EMPHASIS THAT
SHE DEVELOPED HERSELF FOR
HARVARD.
SHE'S AN ASSOCIATE PROFESSOR OF
BIOSTATISTICS IN THE SCHOOL OF
PUBLIC HEALTH AT NEW YORK
UNIVERSITY, SHE'S SUPPORTED BY
LONG ISLAND COMMUNITY
FOUNDATION, PCORI, THE SUSAN G.
COMAN FOUNDATION, SHE'S WELL
PUBLISHED WITH MORE THAN 90
REFEREED JOURNAL ARTICLES, TWO
BOOKS, AND SHE'S KNOWN AS A
RESEARCH METHODOLOGIST WITH A
LARGE STATISTICAL TOOLBOX.
HER RESEARCH INTEREST IS ON
IDENTIFYING ORIGINS OF HEALTH
DISPARITIES AND DEVELOPING AS
NECESSARY EVIDENCE-BASED PRIMARY
PREVENTION STRATEGIES TO REDUCE
HEALTH DISPARITIES.
SO I KNOW DR. GOODMAN HAS A LOT
OF REALLY GOOD INFORMATION TO
SHARE WITH YOU.
I'M GOING TO ASK HER TO COME TO
THE PODIUM AND TALK WITH US.
[APPLAUSE]
>> GOOD AFTERNOON, EVERYONE.
THANK YOU FOR HAVING ME.
I'M EXCITED TO TALK ABOUT THE
SCIENCE AND METRICS OF PARTNER
ENGAGEMENT IN RESEARCH.
SO I WANT TO START WITH JUST A
WORKING DEFINITION OF COMMUNITY
ENGAGEMENT THAT I'LL BE USING
THROUGHOUT MY TALK, AND THIS IS
THE PROCESS OF WORKING
COLLABORATIVELY WITH AND THROUGH
GROUPS OF PEOPLE AFFILIATED BY
GEOGRAPHIC PROXIMITY, SPECIAL
INTEREST OR SIMILAR SITUATIONS
TO ADDRESS ISSUES OF AFFECTING
THE WELL-BEING OF THOSE PEOPLE.
IT IS A POWERFUL VEHICLE FOR
BRINGING ABOUT ENVIRONMENTAL AND
BEHAVIORAL CHANGES THAT WILL
IMPROVE THE HEALTH OF THE
COMMUNITY AND ITS MEMBERS, AND
IT OFTEN INVOLVES PARTNERSHIPS
AND COALITIONS THAT HELP
MOBILIZE RESOURCES AND INFLUENCE
SYSTEMS, CHANGE RELATIONSHIPS
AMONG PARTNERS, AND SERVE AS A
CATALYST FOR CHANGING POLICY,
PROGRAMS AND PRACTICES.
SO WE DO STAKEHOLDER ENGAGED
RESEARCH FOR A VARIETY OF
REASONS, TO ENSURE RESEARCH IS
PATIENT OR COMMUNITY CENTERED,
NON-ACADEMIC STAKEHOLDERS
PROVIDE UNIQUE PERSPECTIVE ON
THEIR NEEDS, INCREASED RELEVANCE
TO NON-ACADEMIC STAKEHOLDERS
INCLUDING PATIENTS AND
COMMUNITIES AND POLICY MAKERS,
SUPPORT SUSTAINABILITY OF
INTERVENTIONS POST FUNDING,
BUILD CAPACITY AND TRUST AMONG
ALL THE STAKEHOLDERS INVOLVED,
LEVERAGES EXISTING RESOURCES
WITHIN THE COMMUNITY, IT
INCLUDES RECIPROCAL
RELATIONSHIPS BETWEEN
RESEARCHERS AND NON-ACADEMIC
STAKEHOLDERS, AND IT IS AN
EVIDENCE-BASED APPROACH FOR
ADDRESSING HEALTH DISPARITIES.
PART OF WHAT BRINGS ME TO THIS
WORK, I THINK, IS MOTIVATED BY
AN AFRICAN PROVERB THAT SAYS IF
YOU WANT TO GO FAST, GO ALONE,
BUT IF YOU WANT TO GO FAR, GO
TOGETHER.
AND I HOPE THAT WE ALL WANT TO
GO FAR IN OUR SCIENCE.
SO WHY MEASURE PARTNER
ENGAGEMENTS?
THE EXTENT TO WHICH STAKEHOLDERS
AND RESEARCH PARTNERSHIPS FEEL
ENGAGED HAS NOT RECEIVED
SCIENTIFIC ATTENTION.
IT IS IMPORTANT TO UNDERSTAND
HOW ENGAGEMENT LEVEL IN A
PARTNERSHIP IS DEVELOPING AND TO
WHAT EXTENT ENGAGEMENT LEVEL IS
A PREDICTER OF OUTCOMES IN THE
LARGER STUDY.
SO I'M GOING TO TALK ABOUT OUR
WORK IN THIS SPACE AND WALK YOU
THROUGH SOME OF OUR APPROACH.
THESE ARE THE ELEMENTS WE TOOK.
WE STARTED WITH A SYSTEMATIC
REVIEW, WE DID SOME INITIAL
DEVELOPMENT OF A NEW MEASURE, WE
ARE REFINING AND VALIDATING THE
NEW MEASURE, AND I'M GOING TO
TALK ABOUT SOME UPCOMING WORK
INCLUDING DEVELOPING A COMPUTER
ADAPTIVE ALGORITHM AND OUR
IMPLEMENTATION STUDY WHICH WE'RE
GETTING READY TO LAUNCH IN THE
NEW YEAR.
SO FIRST WE DID A SYSTEMATIC
REVIEW OF THE LITERATURE.
THIS PAPER IS PUBLISHED IN
CLINICAL AND TRANSLATIONAL
SCIENCES, AND IT WAS A TEAM OF
US, WE LOOKED THROUGH ALL PAPERS
THAT HAD EXISTING MEASURES OF
STAKE HOLD EVERY ENGAGEMENT AND
BASICALLY CLASSIFIED THOSE
MEASURES INTO SOME GROUPS.
WE STARTED THIS WORK BY THINKING
THAT SUCH MEASURES EXISTED AND
THAT THEY HAD PROPERTIES THAT
WERE UNDERSTOOD.
WE FOUND THAT LOTS OF PEOPLE HAD
MEASURED SOMETHING BUT REALLY
DIDN'T KNOW WHAT THEY HAD
MEASURED.
AND THE FIELD WAS NOT VERY
STRONG METHODOLOGICALLY.
THE EXISTING MEASURE WE FOUND
CAME IN TWO CAMPS SO WE GROUPED
THEM IN THIS WAY, ONE IN WHICH
INVESTIGATORS SIMPLY COUNTED THE
ATTENDANCE IN VARIOUS EVENTS AND
ACTIVITIES AND ASSUMED THAT THAT
WAS ENGAGEMENT.
ONE OF THE THINGS WE KNOW IS
THAT SHOWING UP DOESN'T
NECESSARILY MEAN YOU'RE ENGAGED.
YOU MAY SHOW UP FOR THE PIZZA
AND THE COOKIES BUT NOT REALLY
BE ENGAGED IN THE CONVERSATION
THAT IS HAPPENING.
THEN THE OTHER SET OF MEASURES
FELL IN THE CAMP IN WHICH
INVESTIGATORS MEASURED SOME
CONSTRUCT THAT WAS POSSIBLY
RELATED TO ENGAGEMENT BUT
NEITHER WAY HAD BEEN VALIDATED
OR CORROBORATED, MOST WERE NOT
IN OUTCOMES IN THE RESEARCH
PROGRESS AND NONE HAD BEEN
TRACKED OVER TIME.
SO I JUST WANTED TO GIVE YOU A
COUPLE OF EXAMPLES OF EACH OF
THE METHODS THAT WE FOUND.
SO IN THE COUNTING METHOD, WE
FOUND THINGS LIKE COUNTING THE
NUMBER OF PEOPLE WHO ATTENDED A
BOARD MEETING OR COUNTS OF
ATTENDEES AT A COMMUNITY
MEETING, FREQUENCY OF ATTENDANCE
AT PROCESS REPORTS.
AND IN THE CONSTRUCT MEASUREMENT
METHOD, WE FOUND PEOPLE MEASURED
THE DEGREE TO WHICH PARTICIPANTS
FELT THEY WERE A PART OF A
POSITIVE COMMUNITY, THE DEGREE
TO WHICH THEY FELT COMFORTABLE
SHARING THEIR THOUGHTS AND
OPINIONS AND THE LEVEL OF
CONFIDENCE IN PARTICIPATE IN THE
PROBLEM SOLVING PROCESS.
ALL OF THESE THINGS ARE RELATED
TO ENGAGEMENT BUT ARE NOT
NECESSARILY MEASURING
ENGAGEMENTS DIRECTLY.
SO WE DEVELOPED AN INITIAL
MEASURE, A QUANTITATIVE MEASURE
OF DEVELOPMENT, AND I'LL TALK A
LITTLE BIT ABOUT THAT.
IT WAS BASED ON 11 ENGAGEMENT
PRINCIPLES WHICH CAME FROM THE
LITERATURE, BUT WE ALSO WORKED
WITH A COMMUNITY ADVISORY BOARD
TO MAKE SURE THAT THESE WERE THE
THINGS THAT WE SHOULD MEASURE,
INCLUDING THINGS LIKE FOCUS ON
LOCAL RELEVANCE AND DETERMINANTS
OF HEALTH, ACKNOWLEDGING THE
COMMUNITY, DISSEMINATING
FINDINGS AND KNOWLEDGE GAINED TO
ALL PARTNERS, SEEK AND USE INPUT
OF COMMUNITY PARTNERS, INVOLVING
A CYCLICAL AND ITERATIVE PROCESS
IN THE PURSUIT OF OBJECTIVE, TO
FOSTER CO-LEARNING, CAPACITY
BUILDING AND CO-BENEFIT FOR ALL
PARTNERS, TO BUILD ON THE
STRENGTHS AND RESOURCES WITHIN
THE COMMUNITY, TO FACILITATE
COLLABORATIVE AND EQUITABLE
PARTNERSHIPS, TO INTEGRATE AND
ACHIEVE A BALANCE OF ALL
PARTNERS, TO INVOLVE ALL
PARTNERS IN THE DISSEMINATION
PROCESS, AND LASTLY, TO PLAN FOR
A LONG TERM PROCESS AND
COMMITMENT.
OUR ITEM IS MEASURED ON TWO
SCALES, SO THERE'S THREE TO FIVE
ITEMS THAT ASSESSED EACH OF
THOSE ENGAGEMENT PRINCIPLES.
THEY USE LIKERT RESPONSE
OPTIONS.
WE HAVE ONE SCALE THAT MEASURES
QUANTITY, HOW MUCH, AND ONE
SCALE THAT MEASURES QUALITY, HOW
WELL.
THIS WAS ALSO DRIVEN BY OUR
COMMUNITY STAKEHOLDERS THAT SAID
YOU CAN DO A LOT OF SOMETHING
POORLY OR YOU CAN DO A LITTLE
BIT OF SOMETHING WELL AND SO
THAT IT WAS IMPORTANT TO MEASURE
BOTH QUANTITY AND QUALITY.
JUST A HISTORICAL ASIDE, THE
LIKERT SCALE WAS DEVELOPED AT
NYU IN 1932, I WAS ABLE TO PULL
THE ORIGINAL PAPER FROM THE
ARCHIVE, I'M A NERD AND I
THOUGHT IT WAS AMAZINGS TO 
AMAZING TO FIN
D
THAT.
A SINGLE STREAMLINED QUESTION
WITH A SCALED SET OF ANSWERS WAS
IDEAL FOR STANDARDIZED
QUESTIONS.
WE FEL THAT EVEN THOUGH WE HAD
DONE THIS SYSTEMATIC REVIEW THAT
OTHER PEOPLE PROBABLY HAD BEEN
MEASURING ENGAGEMENT BUT JUST
NOT PUBLISHING IT IN THEIR WORK.
SO WE WROTE THIS POSITION PAPER
IN TRANSLATIONAL BEHAVIORAL
MEDICINE AND IT WAS REALLY A
CALL TO ACTION TO TRY TO GET
PEOPLE TO START DISSEMINATING
THE WORK THAT THEY WERE DOING,
AND REALLY A SOAP BOX IRK EU FOR
ISSUE FOR
ME IS TREATING PARTNER
ENGAGEMENT ATION  AS A SCIENCE,
HOW DO
WE IMPLEMENT IT AND EVALUATE IT.
SO IN THE PAPER WE PUT THIS
FIGURE THAT WAS SOMEWHAT
CONTROVERSIAL TO PEOPLE AT THE
TIME BECAUSE WE GROUPED
PARTICIPATION IN THREE
CATEGORIES THAT WE CALLED
NON-PARTICIPATION, SYMBOLIC
PARTICIPATION AND ENGAGED
PARTICIPATION.
PEOPLE WHO WERE DOING OUTREACH
IN EDUCATION DIDN'T LIKE THAT
THEY WERE IN THE
NON-PARTICIPATION WORK, BUT
WE'RE NOT SAYING THAT PEOPLE
SHOULDN'T BE DOING THAT WORK,
AND NOT EVERY PROJECT REQUIRES
ENGAGED PARTNERSHIP.
SOME PROJECTS REALLY ONLY
REQUIRE OUTREACH ENGAGEMENT SO
IT REALLY DEPENDS ON YOUR STUDY
AND THE PROJECT AND THE
POPULATIONS THAT YOU'RE WORKING
WITH.
THE NICE THING ABOUT GETTING
ACADEMICS FIRED UP IS THEN THEY
START PUBLISHING SOME STUFF AND
WORKING IN A SPACE THAT YOU WANT
TO SEE OTHER PEOPLE MOVE SCIENCE
FORWARD.
WE SUBSEQUENTLY HAD A
PATIENT-CENTERED OUTCOMES
RESEARCH INSTITUTE STUDY THAT
WAS FUNDED WITH FOUR SORT OF
MAIN SPECIFIC AIMS TO EXAMINE
THE CONSTRUCT VALIDITY
ENGAGEMENT PRINCIPLE ON BOTH
SCALES AND USE A STANDARDIZED
DELPHI PROCESS FOR EXPERT
VALIDATION BASED ON ADVOCACY
GROUPS, PATIENT, FAMILY FEEDBACK
AND PRIORITIZATION OF DOMAINS WE
SHOULD BE MEASURING.
WE ALSO WANT TO EXAMINE THE
PSYCHOMETRIC PROPERTIES, WE'RE
WORKING ON A SHORTER, MORE
CONDENSED VERSION OF THE MEASURE
BECAUSE IT'S REALLY TOO LONG TO
BE USED WIDELY, AND THEN AT THE
BEGINNING OF THE YEAR, WE'RE
GOING TO EVALUATE ITS
IMPLEMENTATION INTO
PATIENT-CENTERED OUTCOMES
RESEARCH STUDIES THAT ARE&
ONGOING.
SO OUR PHASE 1 WAS A DELPHI
PROCESS.
THERE'S A METHOD FOR COLLECTING
AND ORGANIZING INFORMED OPINIONS
FROM A GROUP OF EXPERTS USING AN
ITERATIVE PROCESS, OFTEN USED IN
SURVEY DEVELOPMENT.
THIS APPROACH IS MOST
APPROPRIATE TO ENSURE THAT
FEEDBACK IS OBTAINED FROM ALL
STAKEHOLDERS WITH ALL EXPERTS
BEING TREATED EQUALLY.
THIS WAS IMPORTANT TO US BECAUSE
WE HAD BOTH ACADEMIC
STAKEHOLDERS BUT ALSO COMMUNTY
AND OTHER SORT OF TYPES OF
PARTNERS AND WE WANTED
EVERYONE'S VOICE TO BE HEARD
EQUALLY, NOT JUST THE PEOPLE
WITH THE LOUDEST VOICE OR THE
PEOPLE WHO SPEAK THE MOST.
THIS DELPHI TECHNIQUE AFFORDED A
STAKEHOLDER ENGAGED MEASUREMENT
DEVELOPMENT PROCESS WHICH WE
ACTUALLY THINK IS REALLY
IMPORTANT AND ONE OF THE BIGGEST
OUTCOMES OF OUR STUDY, IS THAT
IF YOU'RE DEVELOPING A MEASURE,
YOU SHOULD REALLY ENGAGE THE
POPULATION THAT WILL BE TAKING
THAT MEASURE IN THE MEASURE
DEVELOPMENT.
WE USE WEB-BASED SURVEYS FOR
ROUNDS 1 THROUGH 3 AND ALSO IN
OUR FIFTH AND FINAL ROUND, BUT
THE ROUND 4 WAS A 2-DAY
IN-PERSON MEETING WHICH WAS
REALLY CRITICAL TO HELP US REACH
CONSENSUS ON THINGS THAT WERE
HARD TO REACH CONSENSUS JUST
BASED ON SURVEYS.
THE RESPONSES WERE ANALYZED BY
THE INVESTIGATOR TEAM AND EACH
MEMBER OF THE DELPHI PANEL
RECEIVED A SUMMARY REPORT THAT
SHOWED THEIR RESPONSES ALONG
WITH RESPONSINGS FROM OTHER
MEMBERS OF THE DELPHI PANEL
STRATIFIED BY WHETHER THOSE
MEMBERS WERE ACADEMIC MEMBERS OR
COMMUNITY OR PATIENT
STAKEHOLDERS SO POEM  PEOPLE
COULD SEE
THE DIFFERENT VOICES AND WHO WAS
SAYING WHAT, BUT THAT WAS
ANONYMOUS SO THEY ONLY SAW THEIR
RESPONSE AND GROUPED RESPONSES
FROM EVERYONE ELSE.
SO WE USED THIS FIVE ROUND
DELPHI PROCESS AND WE STARTED
WITH THINGS LIKE DEMOGRAPHICS
JUST TO SORT OF KNOW WHO WAS IN
OUR DELPHI PROCESS AND WE
ACTUALLY STARTED WITH THE
ENGAGEMENT PRINCIPLES
THEMSELVES, DID WE AGREE THAT
THESE WERE THE THINGS THAT
SHOULD BE MEASURED, AND SOME
ITEMS AND WE ALSO LOOKED AT SOME
OTHER EXISTING MEASURES.
ONE OF THE THINGS THAT BECAME
REALLY CLEAR TO US IN THIS
PROCESS WAS THAT DEFINITIONS
WERE IMPORTANT SO THAT WE COULD
BE ON THE SAME PAGE SO WE WORKED
THROUGH DEFINITIONS FOR EACH OF
THE ENGAGEMENT PRINCIPE TITLES,
AND WE ALSO DEVELOPED SOME
SCENARIOS SO WE HAVE LEVELS OF
ENGAGEMENT LIKE I SHOWED YOU IN
THAT FIRST FIGURE, AND WE HAVE
SCENARIOS THAT ALIGN WITH EACH
OF THOSE ENGAGEMENT LEVELS SO
THAT PEOPLE COULD BE ON THE SAME
PAGE ABOUT WHAT DOES IT MEAN TO
BE DOING OUTREACH, WHAT DOES IT
MEAN TO BE CONSULTATION, WHAT
DOES IT MEAN TO BE DOING
COOPERATION LEVEL RESEARCH, AND
THEN ALL OF THESE THINGS WERE
ITERATIVE, SO ANYTHING THAT
DIDN'T REACH CONSENSUS IS THEN
BROUGHT TO THE NEXT ROUND.
AND LIKE I SAID, THE TWO-DAY
IN-PERSON MEETING WAS AMAZING,
THESE WERE HIGHLY ENGAGED GROUP
OF PEOPLE BUT THEY GAVE US
REALLY GREAT FEEDBACK.
AND WE LEARNED THAT A LOT OF IT
WAS ABOUT LANGUAGE.
AND WE HAD THE GREAT FORTUNE OF
HAVING AN EDITOR PARTICIPATE IN
THAT TWO-DAY MEETING SO SHE
REALLY HELPED US CRAFT LANGUAGE
THAT THEN ENABLED US TO REACH
CONSENSUS FOR THE PARTICIPANTS.
AND BEFORE EACH ROUND, THEY WERE
GIVEN A SUMMARY OF SORT OF WHAT
HAPPENED IN THE LAST ROUND AND
THEY WERE PREPARED TO TAKE THEIR
WEB-BASED SURVEY OR WE DID SOME
POLLING AT THE IN-PERSON
MEETING.
SO PANELISTS IN EACH ROUND ARE
ENCOURAGED TO RECONSIDER THEIR
PREVIOUS RESPONSES AND ONLY IF
APPROPRIATE TO CHANGE THEIR
PREVIOUS RESPONSES IN LIGHT OF
THE REPLIES AND COMMENTS FROM
OTHER PANELISTS.
WE WERE NOT TRYING TO FORCE
CONSENSUS, WE THOUGHT IT WAS
IMPORTANT FOR PEOPLE TO STAND
THEIR GROUND WHEN THEY THOUGHT
THAT THAT WAS IMPORTANT BUT WE
WANTED THEM TO GIVE US STRONG
JUSTIFICATIONS OF WHY THEY
DISAGREED.
SO WE WERE ALSO REALLY ABLE TO
REACH CONSENSUS, LIKE I SAID, A
LOT OF IT WAS ABOUT LANGUAGE
WHICH WE WERE ABLE TO FIX.
OUR VERSION OF CONSENSUS WAS 80%
AGREEMENT AMONG OUR DELPHI
PANELISTS SO WE WERE JUST TRYING
TO REACH 80% AGREEMENT.
SO THESE ARE THE PARTICIPANTS
THAT PARTICIPATED IN THAT
TWO-DAY MEETING.
IT WAS APRIL IN NEW YORK CITY,
IT WAS A GREAT TIME, AND THEY
WERE LOCKED IN A CONFERENCE ROOM
FOR TWO DAYS TALKING ABOUT
ENGAGEMENT, SO WE WERE RIGHT
ACROSS THE STREET FROM THE PARK,
SO THIS WAS A GREAT GROUP OF
DEDICATED PEOPLE WHO WERE REALLY
SORT OF INTERESTED AND INVESTED
IN THIS WORK.
OUR DELPHI PANELISTS WERE
PRIMARILY FEMALE, PRIMARILY
AFRICAN-AMERICAN, MOST HAD
GRADUATE LEVEL EDUCATION.
WE PURPOSELY HAD MORE COMMUNITY
STAKEHOLDERS THAN ACADEMIC
STAKEHOLDERS JUST BECAUSE WE
WANTED THAT VOICE TO BE
AMPLIFIED A LITTLE BIT MORE.
WE WANTED REPRESENTATION FROM
ALL THE REGIONS.
MOST HAD LOTS OF EXPERIENCE
DOING RESEARCH, PARTICULARLY
COMMUNITY ENGAGED RESEARCH, BUT
WE DID HAVE ONE PARTICIPANT WHO
WAS NEW TO THIS TYPE OF WORK,
BECAUSE WE THOUGHT IT WAS
IMPORTANT TO HAVE THE
PERSPECTIVE OF SOMEONE WHO HAD
NOT BEEN ENGAGED IN THIS TYPE OF
WORK BEFORE TO SEE HOW THEY
WOULD RESPOND TO SOME OF THE
ITEMS.
SO WE RECENTLY RELEASED SOME OF
THE WORK FROM THIS DELPHI PANEL,
WHICH IS REALLY TO DO A CONTENT
VALIDATION, ARE WE MEASURING THE
RIGHT THINGS, ARE WE THINKING
ABOUT THESE THINGS IN THE RIGHT
WAY.
SO I'M NOT GOING TO GO THROUGH
ALL THIS INFORMATION BUT I DO
WANT TO SHOW YOU THAT WE KEPT
METICULOUS TRACK OF ALL THE
ITEMS WE STARTED WITH, WHICH WAS
48, AND THEN HOW MANY ITEMS WERE
NOT CHANGED, DROPPED, WE WERE
TRYING NOT TO ADD ITEMS BUT
ITEMS WERE ADDED IN ROUND ONE
AND I'LL TALK ABOUT WHY THAT
WAS, AND HOW MANY ITEMS WERE
MODIFIED, INCLUDING SOME ITEMS
REMOVED FROM ONE ENGAGEMENT
PRINCIPLE TO ANOTHER.
WE DID THIS FOR EACH ROUND TO
SORT OF KEEP TRACK OF HOW THINGS
WERE SORT OF PROGRESSING AS WE
WENT THROUGH THE FIVE ROUND
DELPHI PROCESS, SO WE WENT FROM
48 ITEMS AT THE BEGINNING OF THE
PROCESS AND WE ULTIMATELY ENDED
ON 32 ITEMS, STILL TOO LONG.
WE'RE WORKING ON CUTTING IT
DOWN, BUT THIS WAS A WAY FOR US
TO SORT OF START TO HONE IN ON
WHAT THE KEY POINTS WERE.
THE MAIN THING THAT CAME OUT OF
THE PROCESS IS WE WENT FROM
MEASURING 11 ENGAGEMENT
PRINCIPLES DOWN TO 8.
THE FIRST SEVEN ARE FROM THE 11
ALTHOUGH SOME OF THEM HAVE BEEN
RE-WORDED.
THE LAST ONE WAS ADDED WHICH WAS
AN IMPORTANT ONE OUR COMMUNITY
PARTNERS BROUGHT OUT WHICH WAS
ABOUT TRUST SO THE LAST
PRINCIPLE IS BUILD TRUST,
SOMETHING THEY FELT WAS HINTED
AT IN THE OTHER PRINCIPLES BUT
WASN'T SPOKEN TO DIRECTLY AND
FELT THEY NEEDED TO DO IT.
WE ALSO NAMED OUR MEASURE SO
IT'S CALLED THE RESEARCH
ENGAGEMENT SURVEY TOOL.
WE CALL IT REST, WE THINK EVERY
P.I. SHOULD STOP AND TAKE A REST
AT SOME POINT IF THEY ARE DOING
ENGAGEMENT-TYPE WORK.
WE ALSO LOOKED AT CONSENSUS OF
ALL OF THE TITLES AND WE WERE
ABLE TO REACH HIGH LEVELS OF
CONSENSUS FOR ALL OF THE TITLES,
MEANING WHAT WE WERE TRYING TO
MEASURE, AND THEN WE WENT ITEM
BY ITEM TO MAKE SURE THAT THERE
WAS CONSENSUS ON EACH OF THE
ITEMS.
WE REALLY WENT THROUGH ALL EIGHT
ENGAGEMENT PRINCIPLES, LIKE I
SAID, THEY ALL ARE MEASURED BY
THREE TO FIVE ITEMS, TO MAKE
SURE THAT THERE WAS CONSENSUS.
WE HAD REALLY STRONG CONSENSUS
AMONG STAKEHOLDERS BUT IF ANY OF
YOU HAVE WORKED WITH RESEARCHERS
OR ACADEMICS, ILTS REALLY HARD
TO REACH CONSENSUS AMONG THAT
GROUP SO WE THINK THESE ARE
REALLY GOOD NUMBERS EVEN THOUGH
THERE WERE OFTEN ONE OR TWO
PEOPLE WHO DIDN'T NECESSARILY
AGREE, OVERALL WE HAD OVER 80%
FOR ALL OF THE ITEMS.
SO ONCE WE HAD OUR MEASURE, WE
THOUGHT IT WAS IMPORTANT TO TEST
THIS MEASURE IN PRACTICE.
WE HAD A LONGITUDINAL STUDY
WHICH WAS FOUR WAVES.
OUR GOAL WAS TO ENROLL 500
PEOPLE THAT HAD BEEN ENGAGED IN
SOME SORT OF RESEARCH STUDY.
WE WERE ABLE TO CONSENT 487.
WE HAD A VARIETY OF NUMBER OF
PEOPLE WHO COMPLETED EACH OF THE
SURVEYS WHICH WE STARTED TO
RELEASE IN JULY OF 2017, AND WE
JUST CLOSED OUR SURVEYS AT THE
END OF SEPTEMBER OF 2019.
WE HAD 391 PEOPLE COMPLETE AT
LEAST ONE SURVEY AND 324 PEOPLE
COMPLETE ALL SURVEYS, AND AS YOU
WOULD EXPECT, THERE'S SOME
ATTRITION ACROSS THE FOUR WAVES
BUT THEY'RE ALL HIGHER THAN 300
SO WE THINK WE DID A GOOD JOB.
YOU'LL SEE THAT THERE'S A BIG
GAP BETWEEN WHEN SURVEY THREE
AND WHEN SURVEY FOUR WAS
RELEASED, SO SURVEY THREE WAS
RELEASED IN MARCH OF 2018 AND IT
TOOK US A YEAR ALMOST BEFORE WE
RELEASED SURVEY FOUR, AND THAT'S
BECAUSE WE DID SOME COGNITIVE
RESPONSE TESTING IN BETWEEN
SURVEY THREE AND SURVEY FOUR,
AND THIS IS REALLY TO ENSURE --
SO ONE OF THE THINGS I MENTIONED
WAS THAT OUR DELPHI PANELS WERE
HIGHLY EDUCATED BUT WE WANTED TO
MAKE SURE THAT OUR ITEM REALLY
WORKED FOR ALL POPULATIONS AND
THAT WAS AT A LEVEL THAT WAS
REALLY READABLE AND
UNDERSTANDABLE TO EVERYUP WITH,
EVERYONE.
AND EVEN THOUGH I'M A SURVEY
RESEARCH METHODOLOGIST, OUR
COGNITIVE TESTING PICKED UP
THINGS THAT WERE COMPLEX, BUT WE
ALSO HAD SOME TERMS THAT PEOPLE
JUST THOUGHT WERE REALLY
COMPLEX.
WE USED THE TERM DISSEMINATION
AND GOVERNANCE AND INTELLECTUAL
PROPERTY AND CAPACITY AND WHOSE
CAPACITY AND ALL THOSE TYPES OF
THINGS, SO WE WORKED ON THOSE
TYPES OF COMPLEX QUESTIONS.
THERE WAS SOME CONFUSION IN THE
QUESTION STEM.
I SHOULDN'T SAY CONFUSION.
LOTS OF PEOPLE DRN THE ORIGINAL
QUESTION STEM WAS WHAT ACADEMICS
ARE DOING, SO IT WAS
UNIDIRECTIONAL, SO NOW THE
QUESTION IS ALL PARKT NERS SO 
PARTNERS SO
EVERYONE WHO'S PARTICIPATING IN
THE STUDY DOES, AND I'M A
STATISTICIAN SO I DON'T LOVE
THIS BUT ONE OF THE THINGS WE
DID ADD WAS A "NOT APPLICABLE"
OPTION FOR ALL OF THE ITEMS, YOU
MAY NOT BE THINKING OF
DISSEMINATION YET OR KNOW A LOT
ABOUT HOW DISSEMINATION WILL
WORK.
SO WE MODIFIED THE MEASURE BASED
ON COGNITIVE RESPONSE TESTING
AND THAT'S WHY THIS SURVEY FOUR
WAS RELEASED A LITTLE BIT LATER.
WE ORIGINALLY PROPOSED THAT WE
WOULD JUST RECRUIT PEOPLE FROM
WASHINGTON UNIVERSITY AND
ST. LOUIS, AND THE UNIVERSITY OF
WASHINGTON, BUT ONE OF THE
THINGS THAT CAME OUT IN OUR
DELPHI PROCESS IS THAT ENGAGED
IS DIFFERENT BY POPULATION BUT
ALSO BY REGION, AND SO WE
DECIDED TO ACTUALLY OPEN THIS UP
AND DO A BROADER NATIONAL STUDY
SO THIS IS A MAP OF WHERE
PARTICIPANTS ARE FROM, YOU SEE
MOST ARE IN MISSOURI BECAUSE
THAT'S WHERE WE STARTED BEFORE
WE THE EXPANSION.
WE ALSO HAVE ONE PARTICIPANT
FROM THE VIRGIN ISLANDS, PUERTO
RICO AND ALASKA.
THOSE WERE NOT DISPLAYED ON THE
MAP BUT THEY DID PARTICIPATE,
AND THEN THERE WERE 12 STATES
THAT HAD NO PARTICIPANTS.
BUT WE THINK WE GOT A GOOD
REPRESENTATION, PARTICULARLY
FROM PLACES WHERE THERE'S LOTS
OF ENGAGEMENT WORK HAPPENING.
SO WE HAD PEOPLE WHO ENGAGED
WITH 177 DIFFERENT UNIVERSITIES
ACROSS THE COUNTRY.
WE'VE LISTED HERE UNIVERSITIES
THAT HAD MORE THAN 15 PEOPLE.
WHEN THIS PROJECT WAS STARTED, I
WAS AT WASHINGTON UNIVERSITY IN
ST. LOUIS, I'M NOW AT NEW YORK
UNIVERSITY SO THOSE ARE THE TOP
TWO BECAUSE A LOT OF THESE ARE
MY PARTNERS, BUT WE ALSO HAD
GREAT RESPONSE FROM THE
UNIVERSITY OF ALABAMA,
NORTHWESTERN, THE MAYO CLINIC,
ST. LOUIS UNIVERSITY, AND THE
UNIVERSITY OF WASHINGTON, WHERE
ONE OF OUR COLLABORATORS IS
LOCATED.
OUR PARTICIPANTS HAD A GOOD MIX
BETWEEN WHITE AND BLACK BUT WE
DIDN'T DO A JOB WITH HISPANICS,
ASIANS AND OTHER MULTIETHNIC
GROUPS.
AGAIN FEMALE, ONE OF THE THINGS
WE FIND WAS WOMEN WERE MORE
LIKELY TO PARTICIPATE IN STUDIES
BUT THEY'RE REALLY A LOT MORE
LIKELY TO BE ENGAGED AS RESEARCH
PARTNERS, SO EVEN THOUGH THIS IS
SKEWED HEAVILY FEMALE, WE DO
THINK THIS IS REPRESENTATIVE OF
THE POPULATION IN WHICH WE WERE
TRYING TO REACH, AND IN THIS
CASE, LESS THAN HALF HAD A
GRADUATE DEGREE, SO MORE
VARIABILITY IN THE EDUCATION
LEVEL.
SO ONE OF THE FIRST THINGS WE
WANTED TO DO IN TERMS OF
EXAMINING THE PSYCHOMETRIC
PROPERTIES WAS TO MAKE SURE THAT
THE ITEMS, 3 TO 5 ITEMS THAT
ASSESS ANY ONE ENGAGEMENT
PRINCIPLE WERE ENTIRELY
CONSISTENT WITH ONE ANOTHER, SO
THEY'RE MEASURING THE SAME
THING.
SO WE USED A STATISTIC
CALCULATOR FOR PAIR WISE
CORRELATION BETWEEN ITEMS AND IT
MEASURES WHETHER SEVERAL ITEMS
PRODUCE CONSISTENT RESPONSES.
IN MOST FIELDS, AN ALPHA OF .7
AND HIGHER IS CONSIDERED
ACCEPTABLE.
SO WE LOOKED AT THE INTERNAL
CONSISTENCY ON BOTH SCALES, BOTH
QUALITY AND QUANTITY, AND WE HAD
REALLY GOOD INTERNAL
CONSISTENCY, MEANING THAT THE
ITEMS WE'RE MEASURING, EACH
ENGAGEMENT PRINCIPLE LOOKED TO
BE CONSISTENT WITH ONE ANOTHER.
I HAVE EP7 STARRED BECAUSE
THERE'S ONE ITEM WE PLAN TO
DROP, AND THIS ITEM IS -- ALL
PARTNERS HAVE OPPORTUNITY TO BE
CO-AUTHORS WHEN THE WORK IS
PUBLISHED.
ONE OF THE THINGS WE HEARD FROM
OUR DELPHI PANELISTS IS THEY MAY
OR MAY NOT BE INTERESTED IN
BEING AUTHORS ON PUBLISHED WORK,
IT REALLY DEPENDS ON WHERE THEY
SIT AND WHETHER PUBLICATION IS
REALLY IMPORTANT TO THEM FOR
THEIR OWN WORK.
THE OTHER THING WE WANTED TO
LOOK AT IS CORRELATIVE VALIDITY
WITH OTHER EXISTING MEASURES.
BECAUSE OUR DATA, IN SOME
INSTANCES, IT'S A LICK ERT SCALE
SO IT'S HIGHLY SKEWED, I'M GOING
TO SHOW YOU THE LOWER VALUES
FIRST.
SO WE LOOKED AT IT IN
CORRELATION WITH THE MEDICAL
MISTRUST SCALE, A SCALE OF TRUST
IN MEDICAL RESEARCHERS.
THERE'S A COMMUNITY ENGAGEMENT
IN RESEARCH INDEX, THERE'S A
MEASURE CALLED THE PARTNERSHIP
ASSESSMENT AND COMMUNITY-BASED
RESEARCH, AND THEN THE COALITION
SELF ASSESSMENT SURVEY, WE
SOME OTHER PARTNER RS I'LL SHOW
YOU SOON.
SO YOU'LL SEE, WE WERE ACTUALLY
REALLY EXCITED TO SEE THEM, MOST
OF THEM WERE NEGLIGIBLE OR LOW
BECAUSE WE DO BELIEVE THAT THESE
THINGS ARE TO BE CORRELATED BUT
WE'RE TRYING TO CREATE A NEW
MEASURE THAT WE THINK IS
MEASURING SOMETHING DIFFERENT.
SO WE DON'T WANT TO BE TOO
CORRELATED BECAUSE IT MEANS IT'S
MEASURING THE SAME THING.
SO ON THE MODERATE SCALE,
THERE'S A MEASURE BY KAGAN, A
PARTNERSHIP SELF ASSESSMENT
TOOL, SYNERGY SCALE AND ALSO A
SATISFACTION SCALE AND THEN ALSO
A COLLABORATION SCALE.
AGAIN, WE WERE EXCITED THAT WE
DIDN'T HAVE REALLY HIGH LEVELS
OF CORRELATION BECAUSE THAT
MEANS WE'RE MEASURING PRETTY
MUCH THE SAME THING.
THIS SHOWS YOU WE'RE MEASURING
SOMETHING THAT IS ASSOCIATED
WITH THESE THINGS BUT IS NOT
NECESSARILY THE SAME SO WE THINK
THIS IS A GOOD THING FOR OUR
ENGAGEMENT MEASURE.
WE'RE DOING SOME EXPLORATORY
FACTOR ANALYSIS, REALLY LOOKING
TO SEE HOW OUR MEASURE IS SORT
OF LAYING OUT IN PRACTICE, SO WE
HAVE SOME OBSERVED VARIABLES
WHICH ARE THE ONES THAT WE
MEASURED, AND THEY'RE ASSOCIATED
WITH LATENT VARIABLES THAT ARE
NOT DIRECTLY MEASURED.
IN OUR CASE, WE HAVE SEVERAL
MEASURED VARIABLES, THOSE ARE
THE ITEMS THAT WE'RE ASKING
PEOPLE, AND WE HYPOTHESIZE THAT
THEY MAKE UP THESE LATENT
VARIABLES WHICH WE'RE CALLING
THE ENGAGEMENT PRINCIPLES.
AND ALL OF THIS FALLS UNDER THE
LARGER UMBRELLA OF COMMUNITY
ENGAGEMENT IN RESEARCH.
WE USE HOW MUCH VARIANCE IS
CAPTURED BY A COMPONENT OR
FACTOR.
YOU TYPICALLY RETAIN FACTORS
WITH VALUES GREATER THAN 1.
BECAUSE OUR SCALE IS NOT
CONTINUOUS, IT'S REALLY ORDINAL,
WE USE POLYCORE COMMONLY USED
WITH VARIABLES.
SO ONE OF THE THINGS WE NOTICED
IS THE SCALES IN TERMS OF FACTOR
ANALYSIS DON'T LINE UP THE SAME
WAY AND I THINK THIS BODES WELL
FOR MEASURING BOTH QUANTITY AND
QUALITY.
WE'RE STILL DIGGING THROUGH THIS
TO SEE IF WE CAN GET A SENSE OF
WHAT THIS MEANS BUT IN TERMS OF
FACTOR ANALYSIS, THE QUALITY
SCALE REALLY LOADS ON TWO
FACTORS AND WE'RE WORKING
THROUGH TO SEE IF WE CAN GET AN
UNDERSTANDING OF WHAT IS ON WHAT
FACTOR.
ONE OF THE THINGS WE WERE REALLY
SORT OF INTERESTED IN IS ARE ALL
THE ITEMS FROM ONE ENGAGEMENT
PRINCIPLE ALL ALIGNED WITH ONE
FACTOR OR THE OTHER.
THAT'S NOT THE CASE, WE'RE
SEEING ITEMS FROM MANY OF THE
PRINCIPLES ALIGN ON ONE FACTOR
OR BOTH FACTORS ACTUALLY.
THERE'S SOME ITEMS THAT ACTUALLY
LOADED ON BOTH ITEMS SO WE'RE
LOOKING AT WHICH ONE THEY LOADED
HEAVIER ON, ONE ABOUT PARTNERS
HAVING THE OPPORTUNITY TO SHARE
IDEAS, INPUT AND LEADERSHIP
RESPONSIBILITIES AND TO SHARE IN
THE DETERMINATION OF THE PROJECT
STRUCTURE AND THE LIKE.
SO REALLY TRYING TO UNDERSTAND
DO THESE ITEMS BELONG ON ONE
FACTOR OR THE OTHER.
SIMILARLY FOR QUANTITY, INSTEAD
OF FALLING ON TWO FACTORS, IT
ACTUALLY FALLS ON FOUR, AND WE
HAVE SOME AT LEAST INCLINATION
HERE WHERE THESE FALL.
THERE'S A SET OF ITEMS THAT
REALLY -- THE LENGTH OF ITEMS,
SET OF ITEMS LOOKING AT TRUST,
AT VALUE, AND EVERYTHING ELSE IS
GROUPED INTO THAT FOURTH FACTOR.
BUT AGAIN, WE WERE TRYING TO SEE
WHICH ENGAGEMENT PRINCIPLES WERE
ALIGNED WITH WHICH FACTORS AND
IF THEY WERE ON MULTIPLE FACTORS
LIKE YOU CAN SEE ENGAGEMENT
PRINCIPLE 1 IS SOME ITEMS ARE ON
FACTOR 1 BUT SOME ITEMS ARE ON
FACTOR 4, SO WE'RE TRYING TO
WORK THROUGH THIS IN TERMS OF
THE WAY OUR SCALE IS PERFORMING
IN PRACTICE.
AND HERE WE'RE LOOKING AT ITEMS
THAT CROSS LOADED ON BOTH
SCALES.
ONE IS JUST AN OPPORTUNITY TO
DROP ITEMS, ARE THEY NOT REALLY
PARTICULARLY THERE WERE TWO
ITEMS THAT DIDN'T LOAD ON ANY OF
THE SCALES SO WE THINK THIS MAY
BE A PLACE THAT WE CAN MAYBE CUT
SOME ADDITIONAL ITEMS.
SO THE FIRST FIGURE I SHOWED YOU
SORT OF HAD A STACKED WAY OF
THINKING ABOUT ENGAGEMENT
LEVELS, IN TERMS OF BINS OF
NON-ENGAGEMENTS, SYMBOLIC
ENGAGEMENTS, AND SORT OF FULL
ENGAGEMENT, AND WE WERE REALLY
ORIGINALLY THINKING ABOUT
ENGAGEMENT AS A CONTINUUM, SO
YOU COULD BE ANYWHERE ON THIS
CONTINUUM, AND NOT THAT BEING ON
ONE END OR THE OTHER IS MORE
VALUABLE, BUT JUST RILEY
THINKING ABOUT WHERE YOUR
PROJECT FALLS, THAT'S LIKELY TO
DRIVE THE SCORES THAT YOU'RE
GETTING FOR ENGAGEMENT.
BUT IN PRACTICE, SO ONE OF THE
THINGS WE ASKED THE PEOPLE IN
THE SURVEY TO TALK ABOUT A
SPECIFIC PROJECT AND WE REMINDED
THEM EACH WAVE WHICH PROJECT WE
WERE TALKING ABOUT, THEN WE GAVE
THEM THE DEAF ANYTHINGS WE HAVE
BEEN USING FOR THE
CLASSIFICATION LEVELS AND ASKED
THEM TO CLASSIFY WHERE THEY FELT
THEIR PROJECT FELL ON THIS
ENGAGEMENT SCALE.
OUTREACH AND EDUCATION, WHICH WE
HAD BEEN CONSIDERING AT ONE END
OF THE SPECTRUM ACTUALLY HAD
HIGHER MEAN SCORES THAN
CONSULTATION AND IN SOME CASES
COOPERATION.
SO IT WASN'T PERFORMING EXACTLY
THE WAY WE THOUGHT ABOUT IT.
THEN WE STARTED TO THINK HOW WE
REALLY WERE LAYING OUT THE
CONTINUUM, THIS IS WHERE YOU'LL
SEE THAT I'M A BIOSTATISTICS
BECAUSE NOW WE HAVE TWO AXES
INSTEAD OF ONE BECAUSE I THINK
IN TERMS OF ENGAGEMENT, IT'S
IMPORTANT TO THINK ABOUT IT IN
TWO ASPECTS.
ONE IS THE ACADEMIC RESEARCHER'S
OUTPUT TO NON-ACADEMIC
STAKEHOLDERS AND THE OTHER IS
NON-ACADEMIC INPUT INTO ACADEMIC
RESEARCH.
WE THINK OUTREACH AND ENGAGEMENT
IT'S NOT HIGH ON NON-ACADEMIC-
INPUT INTO RESEARCH, IT IS
REALLY HIGH IN WHAT COMMUNITIES
CAN BENEFIT FROM ACADEMICS GOING
OUT AND PUTTING THEIR WORK OUT
INTO THE COMMUNITIES THEY'RE
TRYING TO SERVE.
SO NOW WE'RE THINKING ABOUT THIS
ON SORT OF A TWO AXIS LEVEL AND
YOU'LL SEE THAT THEY'RE NOT
EXACTLY IN LINE BUT THEY'RE SORT
OF LINED UP IN A GREAT WAY, SO
OUTREACH AND EDUCATION IS
SLIGHTLY HIGHER ON WHAT WOULD
THE Y AXIS AND LOWER ON THE X
AXIS, WHEREAS CONSULTATION IS
HIGHER ON THE X AXIS BUT LOWER
ON THE Y AXIS AND THEN SORT OF
BOX THAT COLLABORATION,
COOPERATION AND PARTNERSHIP.
COLLABORATION IS EXACTLY WHAT
YOU THINK IT IS, IS FULL, STRONG
COLLABORATION AND PARTNERSHIP IS
SUSTAINED COLLABORATION, SO FOR
MORE THAN ONE PROJECT, SO
BASICALLY COLLABORATION
HAPPENING OVER TIME.
SO WE ARE NOW HEADED INTO
PHASE III OF OUR STUDY, AND THIS
IS SOME OF THE STUFF THAT I AM
AS A STATISTICIAN REALLY EXCITED
ABOUT.
SO WE ORIGINALLY THOUGHT THAT WE
WOULD BE ABLE TO REDUCE THIS
MEASURE IN THE DELPHI PROCESS,
AND WE DID A BIT BUT NOT AS
MUCH, AND WE THOUGHT THROUGH
SOME OF THE PSYCHOMETRIC
ANALYSIS, BUT IT LOOKS LIKE AN
ITEM HERE AND THERE.
SO ONE OF THE THINGS THAT WE ARE
WORKING ON NOW IS TO DEVELOP A
COMPUTER ADAPTIVE ALGORITHM SO
THIS WILL ALLOW US TO REDUCE
PARTICIPANT BURDEN ON THE FLY.
AND THINK ABOUT IT LIKE TAKING
THE S.A.T. OR THE G.R.E., YOUR
NEXT QUESTION IS BASED ON
WHETHER YOU GET THE FIRST
QUESTION RIGHT OR WRONG.
IN THIS INSTANCE, WE DON'T HAVE
RIGHT OR WRONG, BUT WE WANT TO
KNOW, DOES YOUR RESPONSES TO
SOME QUESTIONS PREDICT HOW YOU
WILL RESPOND TO OTHERS, MEANING
THAT I DON'T NEED TO NECESSARILY
ASK YOU THOSE QUESTIONS BASED ON
WHAT -- ON THE RESPONSES THAT
YOU'VE GIVEN ME ALREADY.
SO COMPUTER ADAPTIVE ALGORITHM
IS A METHOD THAT MERGES COMPUTER
TECHNOLOGY WITH MODERN
MEASUREMENT THEORY TO INCREASE
EFFICIENCY AND REDUCE
PARTICIPANT BURDEN.
OUR MEASURES SO FAR HAVE BEEN
GIVEN USING A WEB BASED SURVEY
TOOL SO THIS ALLOWS US TO REALLY
BUILD THIS IN TO THE SURVEY
INSTRUMENT, AND WE'RE WORKING ON
CREATING AN ITERATIVE ALGORITHM
USING OUR FULL MEASURE OF ITEMS
AS AN ITEM BANK AND THEN
TAILORING IT BASED ON PEOPLE'S
INDIVIDUAL RESPONSES.
SO ALL ITEMS THAT HAVE NOT YET
BEEN ADMINISTERED OR EVALUATED
TO DETERMINE WHICH ONE WOULD BE
THE NEXT BEST ONE TO GIVE AND
THAT ITEM IS SORT OF GIVEN IN
THE PARTICIPANT RESPONSE, A NEW
ESTIMATE FOR THE LEVEL OF
ENGAGEMENT IS MEASURED, THEN WE
COMPUTE BASED ON THAT AND
DETERMINE WHAT THE NEXT ITEM IS
TO GIVE AND WE SORT OF REPEAT
THIS PROCESS UNTIL WE DON'T HAVE
TO GIVE THE PARTICIPANT ANY
ADDITIONAL ITEMS.
USUALLY ITEM RESPONSE THEORY IS
USED IN THIS TYPE OF WORK BUT
THAT'S USUALLY WHEN YOU HAVE
CASES WHERE THERE'S A RIGHT
ANSWER.
IN TERMS OF ENGAGEMENT, THERE IS
NO RIGHT ANSWER.
SO AS A STATISTICIAN, WE'RE
THINKING ABOUT FRAMING THIS AS
YOU WOULD FRAME A MISSING DATA
PROBLEM.
THE GREAT THING ABOUT THIS
MISSING DATA PROBLEM IS I HAVE
AURAL THE DATA.
SO 
ALL THE DATA SO I CAN RANDOMLY
DROP THE OBSERVATIONS AND
COMPUTE BACK TO GET THE RIGHT
ANSWER SO I'M SORT OF CHEATING A
LITTLE BIT, I KNOW WHAT THE REAL
ANSWER IS, I CAN DROP THEM OUT
AND ESTIMATE WHETHER I
CANPREDICT THEM AND THAT WILL
ALLOW US TO HAVE AN ALGORITHM,
PEOPLE ARE HIGHLY ENGAGED BUT
OFTEN ASKED TO DO THINGS OVER
AND OVER SO WE DON'T WANT TO
BURDEN THEM BY TAKING ON LOTS OF
NEW ITEMS, AND WE ALSO WANT
INVESTIGATORS TO PUT THIS ON
THEIR EXISTING SURVEYS AND
THEY'RE NOT GOING TO DO THAT IF
IT'S TOO LONG.
SO WE'RE REALLY SORT OF EXCITED
ABOUT THAT.
AND THE KEY QUESTION THAT WE'RE
TRYING TO GET AT IS, IS THERE A
SUBSET OF ITEMS THAT PRODUCE THE
SAME MEAN SCORE AS ALL OF THE
ITEMS WOULD, AND WE ARE WORKING
TO DEVELOP THIS ALGORITHM AND
WE'RE PILOT TESTING IT ON OUR
COMMUNITY ADVISORY BOARD FOR
THIS PROJECT.
SO WE KNOW HOW ENGAGED THEY HAVE
BEEN, AND SO WE'RE ALSO ABLE TO
LOOK AT IT IN OUR PROJECT AND
THEN WE'LL BE IMPLEMENTING IF IN
OTHER PC OSM R AND COR AND
EFFECTIVENESS
STUDIES.
SO WE'RE CURRENTLY RECRUITING
AND ROLL OUT IMPLEMENTATION
PHASE AT THE BEGINNING OF NEXT
YEAR.
THE IDEA IS TO EVALUATE THE
STAKEHOLDER ENGAGEMENT MEASURE
IN PRACTICE, AND WE'RE WORKING
WITH A LOT OF PATIENT-CENTERED
OUTCOMES RESEARCH AND
COMPARATIVE EFFECTIVENESS
STUDIES, PARTICULARLY OTHER
STUDIES THAT HAVE BEEN FUNDED BY
PCORI.
WE'RE GOING TO DO WEB BASED
SURVEYS OF PROJECT TEAM MEMBERS
FIRST TO REALLY GET THEIR SENSE
OF WHAT ARE THE BARRIERS AND
FACILITATOR FACTORS FOR YOU
MEASURING PARTNER ENGAGEMENTS IN
YOUR STUDIES.
SO FAR WE'VE HAD 34 PROJECT
TEAMS COMPLETE THE BASELINE
SURVEYS AND 14 HAVE AGREED TO
IMPLEMENT OUR MEASURE IN THEIR
STUDIES.
SO REALLY FROM THEM, WE'RE
TRYING TO FIGURE OUT HOW DOES
THIS WORK IN PRACTICE, DOES IT
REALLY ADD A LOT OF TIME TO YOUR
EXISTING MEASURES, CAN YOU TACK
IT ON TO A SURVEY THAT YOU WERE
ALREADY GIVEN, WHAT ARE THOSE
IMPLEMENTATION CHALLENGES TO
ADDING AN ASSESSMENT TO YOUR
EXISTING SURVEY TOOLS, DOES IT
MAKE YOUR SURVEY WAY TOO LONG.
AND IN BETWEEN OUR BASELINE AND
FOLLOW UP, WE'LL BE DOING SOME
PHONE INTERVIEWS WITH PROJECT
TEAMS THAT HAVE DECIDED TO
COLLABORATE WITH US.
WE'RE USING THE CONSOLIDATED
FRAMEWORK FOR IMPLEMENTATION
RESEARCH AS A FRAME FOR OUR
IMPLEMENTATION STUDY WHICH
ALLOWS US TO LOOK AT
INTERVENTION CHARACTERISTICS,
THE OUTER SETS, WHAT ARE THE
EXTERNAL FACTOR, THE INNER
SETTINGS, WHAT ARE THE INTERNAL
PROJECT TEAM FACTORS, COLORADO
SPRINGS 
CHARACTERISTICS OF INDIVIDUALS
INVOLVED IN THE PROCESS OF
IMPLEMENTING OUR MEASURE IN
THEIR STUDY, AND WE THINK THAT
THIS WILL HELP US REFINE THE
TOOL, PARTICULARLY IN TERMS OF
HOW IT SHOULD BE ADMINISTERED
FOR GUIDELINES, PROCEDURES FOR
IMPLEMENTING THE TOOL IN
PRACTICE, AND REALLY ALLOW US TO
PACKAGE AND DISSEMINATE THE TOOL
IN A WAY THAT IS USEFUL FOR
RESEARCH TEAMS.
SO PLEASE DON'T BLAME PCORI FOR
ANYTHING THAT I SAID, BECAUSE
THEY'RE NOT RESPONSIBLE FOR IT,
BUT I HOPE YOU DID ENJOY.
I DO HAVE TO REALLY THANK MY
COLLEAGUES WHO HAVE BEEN
INVOLVED IN THIS STUDY BOTH FROM
THE SYSTEMATIC REVIEW PROCESS
AND THOSE WHO ARE CURRENTLY
INVOLVED IN OUR PCORI-FUNDED
STUDY, AND ARE IMMENSELY
GRATEFUL TO THE MEMBERS OF OUR
DELPHI PANEL WHO WERE HIGHLY
ENGAGED AND PARTICIPATED AT
REALLY HIGH LEVELS THROUGHOUT
THE DELPHI PROCESS.
I SHOULD MENTION THAT WE STARTED
WITH 19 DELPHI MEMBERS AND WE
HAD 18 COMPLETE ALL FIVE ROUNDS
OF THE PROCESS WHICH IS REALLY
GOOD CONSIDERING ACADEMICS AND
RESEARCHERS AND COMMUNITY PEOPLE
AND HEALTH CENTERS AND DIRECTORS
WHO WERE ALL SUPER, SUPER BUSY
PEOPLE.
AND I WILL TAKE QUESTIONS.
[APPLAUSE]
>> WE'LL ASK PEOPLE TO COME TO
THE MICROPHONES IN THE AISLE TO
ASK QUESTIONS AND I'LL START OFF
WITH A QUESTION.
THIS IS REALLY FASCINATING, AND
CLEARLY YOU'RE STILL REFINING
IT.
BUT WHAT IS YOUR THOUGHT, YOUR
VISION AS TO HOW THIS TOOL MIGHT
BE USEFUL FOR YOUR STANDARD NIH
CLINICAL TRIAL WHERE YOU'RE
TRYING TO ASSESS THE LEVEL OF
INCLUSION AND RETENTION
TECHNIQUES THAT ARE BEING
DEVELOPED?
>> SO WE REALLY THINK THIS IS AN
IMPORTANT STEP TO TAKE BACK, BUT
WE REALLY HOPE THAT PEOPLE -- TO
DEVELOP THE MEASURE, BUT WE
REALLY HOPE PEOPLE USE IT FOR
THINGS LIKE DID ENGAGING
STAKEHOLDERS HELP ME RECRUIT
FASTER, DID IT HELP ME RECRUIT A
MORE DIVERSE SAMPLE OF
PARTICIPANTS, DID IT MAKE ME
REFINE MY RESEARCH QUESTION IN
MEANINGFUL WAYS, DID IT HELP ME
IMPLEMENT MY WORK IN PRACTICE IN
A PRACTICE SETTING IN A BETTER
CASE?
SO WE REALLY HOPE THAT PEOPLE
WILL BE ABLE TO USE IT AND
REALLY LOOK AT IT IN TERMS OF
HOW IT REALLY IMPACTS SOME OF
THE KEY COMPONENTS OF RESEARCH
THAT ARE NOT NECESSARILY THE
EFFECTIVENESS OF MY
INTERVENTION, WHICH PEOPLE
ALREADY SORT OF MEASURE, BUT
DOES IT REALLY HELP ME IN
IMPLEMENTING THE RESEARCH THAT
I'M WORKING ON.
>> I REALLY ENJOYED YOUR
PRESENTATION.
THANK YOU VERY MUCH,
DR. GOODMAN.
YOU TALKED A LITTLE BIT ABOUT
THE ENHANCING ENGAGEMENT OF
WOMEN EARLY IN YOUR
PRESENTATION.
CAN YOU TALK A LITTLE BIT MORE
ABOUT WHAT YOU SEE BETWEEN
DIFFERENCES BETWEEN MEN AND
WOMEN GENERALLY AND WHAT THE
CONSEQUENCES OF THOSE
DIFFERENCES MIGHT BE?
>> SO THERE'S ALREADY SOME
RESEARCH THAT SHOWS THAT WOMEN
ARE MORE LIKELY TO PARTICIPATE
IN RESEARCH STUDIES IN GENERAL.
I DON'T THINK ANYONE HAD REALLY
LOOKED AT IT IN TERMS OF
ENGAGEMENT.
BUT WHEN WE STARTED TO SEE HIGH
NUMBER OF WOMEN IN OUR SAMPLE,
WE WERE A LITTLE BIT CONCERNED
THAT OUR DATA WOULD BE SKEWED,
SO WE ACTUALLY -- SO THE WAY WE
RECRUITED PEOPLE WAS WE WENT
THROUGH PRINCIPLE INVESTIGATORS
AND WE SAID HI, WE REALLY LOVE
YOUR STUDY BECAUSE COMPLIMENTS
ALWAYS HELP, AND WE'RE TRYING TO
REACH OUT TO PEOPLE WHO MAY HAVE
BEEN ENGAGED IN YOUR PROCESS,
CAN YOU FORWARD THIS EMAIL TO
YOUR PARTICIPANTS.
SO THEN WE WENT BACK TO THOSE
P.I.s AND WE SAID, CAN YOU
GIVE US A SENSE OF WHAT THE
MAKEUP WAS OF YOUR PARTNERS?
AND THAT'S WHEN WE REALLY
STARTED TO FIGURE OUT THAT THERE
WERE A LOT MORE WOMEN IN THIS
ENGAGED PROCESS THAN THERE WERE
MEN.
SO IN ONE WAY, I THINK THAT
HELPED VALIDATE WHAT WE WERE
SEEING, THAT THE POPULATION THAT
WE WERE TRYING TO REACH, OUR
SAMPLE WAS REPRESENTATIVE OF
THAT POPULATION.
IN ANOTHER WAY, IT'S REALLY
IMPORTANT TO ENGAGE BOTH
SECTIONS IN  SEXES
IN THE RESEARCH PROCESS,
PARTICULARLY AS RESEARCH
PARTNERS, AND SO IN SOME WAYS,
IT'S A LITTLE BIT DISHEARTENING.
YES, IT'S NICE THAT WE'RE
GETTING A GOOD  GOOD SAMPLE OF
THE
POPULATION, BUT IS THIS EVEN THE
RIGHT POPULATION TO START WITH
IS ONE OF OUR QUESTIONS, AND SO
ONE OF THE THINGS WE THINK IS
REALLY IMPORTANT FOR THOSE WHO
DO ENGAGEMENT WORK IS TO REALLY
ENGAGE MEN MORE AS PARTNERS IN
THE RESEARCH PROCESS.
AND THE INTERESTING THING WAS,
WE CONTACTED A P.I. WHO DID A
LOT OF WORK IN PROSTATE CANCER
AD HE STILL HAD MORE WOMEN
PARTNERS AND THEY WERE LIKE
REPRESENTING THEIR HUSBANDS OR
THEIR -- YOU KNOW, SO IT'S ALL
REALLY INTERESTING BUT I THINK
IT'S IMPORTANT, YOU KNOW,
ENGAGEMENT IS SORT OF MY SOAP
BOX, IT'S REALLY IMPORTANT TO
ENGAGE THE PEOPLE THAT ARE MOST
IMPACTED BY OUR WORK.
AND SO IT IS A LITTLE BIT
DISHEARTENING THAT IT'S MAINLY
WOMEN, AND I REALLY HOPE THAT AS
WE RELEASE SOME OF THIS DATA,
THAT PEOPLE REALLY START TO
CONSIDER WHO THEY'RE ENGAGING.
YES, SOMETIMES IT'S NICE TO HAVE
A SPOUSE IF ONE SPOUSE IS
ACTUALLY HAVING AN ILLNESS, IT'S
SOMETIMES NICE TO HAVE THE OTHER
SPOUSE BEING ENGAGED, BUT IT'S
REALLY A LOT MORE MEANINGFUL TO
HAVE THE PERSON IMPACTED TO BE
THE PERSON THAT IS REALLY
ENGAGED IN DEVELOPING OUR STUDY.
SO I'M GIVING THIS TALK AT NIH.
THOSE OF YOU THAT ENGAGE
PARTICIPANTS IN YOUR WORK, I
THINK YOU SHOULD BE REALLY
THOUGHTFUL AND CONSCIOUS OF WHO
YOUR ACADEMIC PARTNERS ARE, OR
YOUR NON-ACADEMIC PARTNERS ARE,
AND PART OF WHAT PEOPLE WERE
REALLY HONEST WITH US ABOUT IS
THAT WHOEVER COMES IS WHO THEY
TAKE.
SO IF THE WOMEN SHOW UP, THAT'S
WHO THEY'RE GOING TO USE AS
PARTNERS, BUT I DO THINK WE HAVE
TO BE MORE MEANINGFUL, SO IN MY
OWN STUDIES NOW, I'M REALLY MORE
MEANINGFUL ABOUT LOOKING AT THE
DEMOGRAPHICS OF MY PARTNERS, AND
REALLY SAYING THERE'S SOME GAPS,
LET'S TRY TO GO -- SO I'M
WORKING ON A STUDY WITH PEOPLE
STUDYING THE -- GENE, WE FORMED
A BOARD AND IT WAS A LOT WOMEN
AND I SAID THEY ALL CHECK OFF A
BUNCH OF BOXES, NOW LET'S FIND
MEN WHO CHECK OFF THESE SAME
BOXES.
BECAUSE THINK IT'S IMPORTANT TO
HAVE BOTH VOICES HEARD AND WE'RE
NOT DOING A GOOD JOB OF THAT
RIGHT NOW.
>> HI.
SO THAT BRINGS UP ANOTHER
QUESTION.
DO YOU THINK THAT YOUR WORK CAN
HELP IN TARGETING MESSAGES TO
PROMOTE ENGAGEMENT FROM
DISENGAGED POPULATIONS?
>> I DON'T KNOW IF IT COULD HELP
IN DEVELOPING MESSAGES, BUT I DO
THINK IT CAN -- THE REASON WE'RE
LOOKING AT THESE EIGHT
ENGAGEMENT PRINCIPLES, HAVING A
CERTAIN SCORE TELLS YOU WHERE
YOU ARE, CONSULTATION,
COOPERATION, BUT KNOWING, LIKE,
I'M DOING REALLY WELL ON THIS
ENGAGEMENT PRINCIPLE BUT NOT SO
GOOD OVER HEERYLY TELLS YOU 
HERE REALLY T
ELLS YOU
HOW TO STARGT YOUR  TARGET YOUR
ENGAGEMENT
STRATEGIES.
I THINK THAT REALLY TAKES MORE
THOUGHTFULNESS AND MORE
MINDFULNESS ABOUT WHO WE'RE
ENGAGING AND NOT USING PEOPLE
JUST BECAUSE THEY'RE THERE AND
THEY'RE PRESENT AND WILLING TO
COME BUT REALLY TAKING MORE
EFFORT TO ENGAGE THOSE WHO NEED
TO BE ENGAGED, AND YES, THAT MAY
REQUIRE A LITTLE BIT MORE WORK
AND A LITTLE BIT MORE TIME, BUT
I THINK THE SCIENCE WILL GREATLY
BENEFIT FROM THAT.
>> THANK YOU.
>> I HAVE ANOTHER QUESTION.
AS YOU KNOW, WE HAVE THIS NEW
POLICY AT NIH ON INCLUSION
ACROSS LIFESPAN, AND I NOTICE
THAT IT APPEARED FOR YOUR DELPHI
PROCESS THAT THE AVERAGE AGE OF
PEOPLE IS ABOUT 40-SOMETHING.
HOW DO YOU THINK YOUR DATA MIGHT
BE INFLUENCED BY HAVING A LARGE
NUMBER OF PEOPLE OVER THE AGE OF
65?
>> SO WE ACTUALLY HAD A LARGE --
OUR AVERAGE AGE -- WE HAD A
LARGE RANGE IN TERMS OF AGE,
WHAT WE MISSED WAS YOUNGER
FOLKS.
THAT'S PROBABLY BECAUSE THEY
HAVEN'T REALLY BEEN ENGAGED IN
RESEARCH PROJECTS THAT MUCH.
USUALLY CHILDREN ASSENT BUT THEY
DON'T CONSENT SO THEY'RE OFTEN
NOT ON -- SO WHEN I TALK ABOUT
RESEARCH PARTNERS, I'M NOT
TALKING ABOUT RESEARCH
PARTICIPANTS, I'M REALLY TALKING
ABOUT PARTNERS THAT ARE IN THE
DRIVING FORCE OF DEVELOPING THE
RESEARCH STUDY, DESIGNING IT AND
THE LIKE, SO WHERE I REALLY
THINK WE NEED TO DO MORE WORK IS
AT THE YOUNGER END OF THE LIFE
COURSE.
WE'RE TRYING TO REFINE THIS
MEASURE AS IT IS NOW AND WE'RE
REALLY TRYING TO DO A GOOD JOB
OF DEVELOPING A GENERIC MEASURE
SO IT HAS NO DISEASE FOCUS, IT
HAS NO POPULATION FOCUS, BUT I
DO THINK THE NEXT STEP, IF WE
GET THIS RIGHT, IS TO REALLY
START TO THINK ABOUT HOW TO
TAILOR THIS TO CERTAIN GROUPS,
AND THAT INCLUDES BOTH AGE AND
RACE, ETHNICITY AND GEOGRAPHIC
REGION AND THE LIKE.
>> HI.
THANK YOU FOR YOUR TALK,
DR. GOODMAN.
I WAS THINKING ABOUT THE
EXPERIENCE OF ACTUALLY
PARTICIPATING IN A SURVEY, AND
WHETHER OR NOT YOU ALWAYS
ADMINISTER IT ONE WAY OR IS
THERE ANY VARIETY?
SO IS IT ELECTRONIC, IS IT EVER
GIVEN AS SOMEONE SPEAKING TO
ANOTHER PERSON AND COLLECTING
THE RESULTS, OR JUST ON PAPER?
>> SO WE HAVEN'T DONE PAPER.
WE'VE DONE ELECTRONIC, BOTH SELF
ADMINISTERED AND WHERE
RESEARCHERS ARE USING AN iPAD
AND ASKING THE QUESTIONS, SO I'M
REALLY COGNIZANT OF LITERACY AND
HEALTH LITERACY OVERALL.
ONE OF THE THINGS THAT I DIDN'T
MENTION, THERE IS THIS IDEA THAT
BY ASKING YOU A FEW QUESTIONS
ABOUT ENGAGEMENT AND WILL
INFLUENCE MAYBE HOW YOU RESPOND
TO OTHER QUESTIONS, AND SO WE
HAVE CHANGED THE ORDER OF THE
WAY WE ASK QUESTIONS ON OUR
SURVEY TO MAKE SURE THAT'S NOT
SORT OF -- LIKE BY ASKING THESE
INITIAL QUESTIONS UP FRONT, IT
CHANGES THE WAY PEOPLE RESPOND
TO QUESTIONS AT THE END, SO WHEN
WE DID OUR COGNITIVE RESPONSE
TESTING, WE HAD FOUR DIFFERENT
QUESTIONS OF THE SURVEY SO
QUESTIONS WERE ASKED IN A
DIFFERENT ORDER, SO IN OUR
IMPLEMENTATION STUDY, PEOPLE
WILL RANDOMLY GET QUESTIONS IN
DIFFERENT ORDER, SO WE THINK THE
ORDERING WILL MATTER.
RIGHT NOW IT'S BEING
WEB-ADMINISTERED JUST BECAUSE
I'M TRYING TO CONVINCE OTHER
PEOPLE TO USE IT, OTHER
RESEARCHERS AND I DON'T WANT TO
BURDEN OTHER RESEARCHERS MORE
THAN WHAT THEY HAVE TO, AND ONE
OF THE THINGS WE ASK PEOPLE IS,
HOW ARE YOU CURRENTLY GIVING
SURVEYS, AND MANY OF THEM ARE
ALREADY DOING WEB SURVEYS, SO
THIS ALLOWS US TO TACK IT ON TO
AN EXISTING SURVEY.
PEOPLE HAVE TAKEN IT BY PAPER
BEFORE, WE'RE NOT AGAINST IT,
BUT ONCE IT'S ENTERED BY PAPER,
THAT MAKES IT HARDER FOR ME TO
CONVINCE A RESEARCH TEAM TO DO
THIS ON TOP OF ALL THE STUFF
THEY'RE ALREADY DOING.
BUT IF I SAY IT'S A WEB SURVEY,
YOU DON'T HAVE TO DO ANYTHING,
YOU JUST ADD IT ON TO YOUR
EXISTING SCALE, IT'S JUST BEEN A
BETTER SELL TO GET PEOPLE TO
DECIDE TO USE IT.
>> I HAD A BRIEF QUESTION.
HAS YOUR INSTRUMENT EVER BEEN
ADMINISTERED NOT IN ENGLISH?
>> NO.
SO THERE'S A POSTDOC WHO IS
CURRENTLY TRANSLATING IT INTO
SPANISH, BECAUSE SHE WANTS TO
USE IT AND WE'RE EXCITED.
ONE OF THE THINGS WE REALLY FEEL
IS WE NEED TO GET IT RIGHT IN
ENGLISH FIRST AND WE'RE NOT
THERE YET, BUT WE DO THINK THE
NEXT STEP IS TRANSLATING INTO
OTHER LANGUAGES, AND WE EVEN
HAVE SOMEONE WHO WORKS WITH THE
DEAF POPULATION THAT'S REALLY
INTERESTED IN USING OUR MEASURE.
SO WE HOPE THAT IT GOES BEYOND
ENGLISH BUT WE THINK WE NEED TO
SORT OF REFINE IT IN ENGLISH AND
THEN WORK ON TRANSLATING INTO
OTHER LANGUAGES.
>> SO FOLLOW-UP QUESTION.
DO YOU FEEL LIKE THERE WOULD BE
ANY DIFFERENCES, CULTURALLY OR
OTHERWISE THAT WOULD COME UP IN
KIND OF GOING THROUGH THIS
PROCESS IN ANOTHER LANGUAGE AND
NOT JUST DOING AS WE ALWAYS DO,
CREATE EVERYTHING IN ENGLISH,
TRANSLATE TO ANOTHER LANGUAGE
BECAUSE OF COURSE IT WILL WORK.
>> SO WE THOUGHT ABOUT THIS A
LOT.
THIS IS THE NICE THING ABOUT
HAVING THE EDITOR IN THE ROOM
AND REALLY FOCUSING ON PLAIN
LANGUAGE.
I DO HAVE SOME CONCERNS THAT
SOMETIMES SOMETHING IS AT A
READABLE SCALE IN ENGLISH AND
THEN WE DIRECTLY TRANSLATE INTO
ANOTHER LANGUAGE AND IT MOVES
FROM, LIKE, SIXTH GRADE LEVEL TO
NINTH GRADE LEVEL JUST THROUGH
THE TRANSLATION PROCESS.
I DON'T KNOW THE ANSWER TO THAT
YET AND I WON'T KNOW UNTIL WE
START TESTING IN OTHER
LANGUAGES, BUT WE'RE REALLY
COGNIZANT OF MAKING SURE THAT
OUR MEASURE IS AT A REALLY
READABLE LEVEL.
THEY'RE ALL SORT SHORT ITEMS
THAT REALLY NOW USE PLAIN
LANGUAGE, SO THE HOPE IS THAT IT
WILL TRANSLATE WELL, BUT I CAN'T
ANSWER YOUR QUESTION UNTIL WE
ACTUALLY TRANSLATE IT.
BUT WE ARE CONSCIOUS OF THAT,
PARTICULARLY IN ENGLISH, WE'RE
CONSCIOUS OF THE FACT THAT WE
HAVE BEEN USING AN EDUCATED
POPULATION THAT WE REALLY NEED
TO MAKE SURE THAT IT WORKS FOR A
VARIETY OF POPULATION.
PARTICULARLY TRYING TO GET
PEOPLE WITH VARIED LEVELS OF
EDUCATION TO LOOK THROUGH THE
MEASURE TO SEE HOW WELL IT
PERFORMED IN ALL GROUPS.
>> BEFORE I ASK YOU TO AGAIN
THANK DR. GOODMAN FOR A REALLY
MARVELOUS PRESENTATION, I WANT
TO REMIND YOU THAT FAES HAS VERY
KINDLY PROVIDED A RECEPTION
RIGHT OUTSIDE OF THESE DOORS,
AND PLEASE DO JOIN ME IN
THANKING DR. GOODMAN.
00:46:47.071,00:00:00.000
>> THANK YOU VERY MUCH.
