Activity
Hunting for Buried Treasure: Well, buried evidence at least…
Research is a Process
Scoping the domain or literature1
Forming the questions2
Conducting the search3
Evaluating the sources4
Summarizing the literature5
What is “the literature”? What domains are pertinent?
Feedback loop, questions reframed as literature is learned
Look for existing summaries (narrative reviews, systematic reviews, meta-analyses, etc.)
Research is a Process
Forming the questions2
§ Define the problem in a clear, and concise statement § Little is known about the factors associated with Y, and Y
is expensive, important, relevant, etc. to business. § People (consultants, leaders, employees, etc.) think that X
is related to Y, but this seems to be based on a hunch
§ Start broad, but don’t finish until you have specifics § Does team-building work? § Does working virtually improve performance? § Do technology acquisitions speed innovation?
Research is a Process
Forming the questions2
§ Define the problem in a clear, and concise statement § Little is known about the factors associated with Y, and Y
is expensive, important, relevant, etc. to business. § People (consultants, leaders, employees, etc.) think that X
is related to Y, but this seems to be based on a hunch
§ Start broad, but don’t finish until you have specifics § Does team-building work? § Does working virtually improve performance? § Do technology acquisitions speed innovation?
§ Does team-building work? § What is a ‘team’? § In what contexts/settings? § What counts as ‘team-building’? § What does ‘work’ mean (outcomes, time period, etc.)?
Research is a Process
Forming the questions2
Some tips for asking good questions
§ Write answerable questions § PICOC
Population Who? Type of employee, group, etc.
Intervention What or How? Technique, factor, treatment
Comparison Compared to what? Other techniques, factor, treatment
Outcome Desired consequence? Purpose or criteria
Context Under what circumstances? Organization, sector, industry, etc.
Forming an Answerable Question Imagine you are a consultant. Your client is the board of directors of a large health-care organization. The board has plans for a merger with a smaller healthcare organization in a nearby town. However, it has been said that the cultures differ widely between the two organizations. The board asks you if this organizational culture difference will impede the success of the merger. Most of them sense that cultural differences matter, but they want evidence-based advice…
What else would you like to know?
What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff?
What kind of Outcome are we aiming for? Employee productivity, return on investment, profit margin, competitive position, innovation power, market share, customer satisfaction?
And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument?
P:
O:
C:
More information… According to the board, the primary objective of the merger is to integrate the back-office of the two organizations (IT, finance, purchasing, facilities, personnel administration, etc.) in order to create an economy of scale. The front offices and primary processes of the two organizations will remain separate. The cultural difference is not objectively assessed (it is the perception of the senior managers of both organizations).
PICOC
P:
O:
I:
C:
Back-office employees in a healthcare organization
C:
Merger, integration of back office
Status quo (current state)
Economy of scale (efficiency, cost reduction, etc.)
Different organizational cultures
Forming an Answerable Question “I lead the primary development team for a medium- sized software firm. For years, we’ve used conventional project management (PM) techniques with great success (e.g., waterfall, critical path, rational unified process). However, I’ve been reading more and more articles about the PM approach referred to as Agile or Scrum. Everything I read extols the value of Agile, especially its ability to allow nimbler development. Before I implement this new approach with my team, which would be a rather dramatic shift in process, I’d like to know the chances it will work.”
PICOC
P:
O:
I:
C:
Programmers on a software development team
C:
New PM approach (Agile/Scrum)
Current PM approach (traditional methods)
On-time delivery, bug rate, iterations, individual attitudes, team cohesion/conflict/coordination, etc.
Mature team with history of success
Forming an Answerable Question “I work for national retail organization and the SVP of sales and service recently asked for my opinion on a potentially new management program for the company. She knows that I just finished my graduate degree and during our last meeting, she remarked, “The leadership team has been discussing the fact that we don’t really use non-financial rewards for our salespeople and they wondered if the company should be doing so. What I’d like to know is whether or not these kinds of rewards engage employees.”
PICOC
P:
O:
I:
C:
Salespeople at a national retail organization
C:
Non-monetary rewards
Current approach (baseline)
Employee engagement
No prior use of non-financial rewards
Research is a Process
Conducting the search3
§ Popular press/journalistic sources § Wall Street Journal, Financial Times, Businessweek,
Harvard Business Review, MIT Sloan Review
§ Government/industry/company sources § FRED, Census, BLS, MayflowerGroup, McKinsey Reports
§ Academic sources § ABI/INFORM, Business Source Complete, PsychInfo, ERIC,
Web of Science, Google Scholar
Research is a Process
Conducting the search3
Some tips for searching
§ Two general types of searching
§ “Building blocks” method (searching with terms) § Use PICOC terms; synonyms; related subjects;
narrower or broader terms
§ “Snowball” method (searching a known source) § Ancestry – cites used by the found source itself § Descendent – sources that cite the found source
Research is a Process
Conducting the search3
Some tips for searching
§ Follow a concentric search pattern
1. Subject/Thesaurus (SU) 2. Title (TI) 3. Abstract (AB) 4. Anywhere
Synonyms, spellings,
etc.
Research is a Process
Evaluating the sources4
Some tips for evaluating located sources
§ Look for “seminal” articles
§ A publication that summarizes and/or extends scientific thought on a given topic
§ E.g., systematic reviews, meta-analyses, theoretical papers, even some single empirical studies
§ Citation count is a rough-cut indicator of seminal articles
Research is a Process
Evaluating the sources4
Some tips for evaluating located sources
§ Consider journal quality
§ General indicators of journal quality (there’s some debate here)
§ Double-blind, peer-reviewed § Acceptance rates (5-10% “best”; <20% “very good”) § Impact Factors (JCR, SJR, WoS, SNIP) § Reputational lists (ABS list, FT top 50, UT-Dallas)
Research is a Process
Summarizing the literature5
§ Reading academic journal articles
§ Common components Abstract Literature Review
Hypotheses
Methods
Results
Discussion
References
Research is a Process
Summarizing the literature5
§ Critically appraised topics § Rapid evidence assessments § Systematic reviews
§ Narrative reviews § Meta-analysis
Underlying Logic of Meta-analysis • Meta-analysis cumulates effects across studies
whereas a single study cumulates scores across participants
ID Variable 1
Person 1 1.25
Person 2 1.75
Person 3 3.25
Person 4 4.00
Person 5 … 4.15
Person i 4.58
Study Effect (r)
Study 1 .23
Study 2 .55
Study 3 .34
Study 4 .27
Study 5 … .61
Study i .42
Underlying Logic of Meta-analysis • Meta-analysis focuses on the
direction and magnitude of the effects across studies, not statistical significance
*Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.
Example:
21 studies that examined the relationship between an given employment test and job performance
§ 38% statistically significant § avg. effect size (r) = .33
Underlying Logic of Meta-analysis • Meta-analysis focuses on the
direction and magnitude of the effects across studies, not statistical significance
*Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.
Example:
21 studies that examined the relationship between an given employment test and job performance
§ All CIs overlap, even for the smallest and largest r
§ avg. (corrected) r = .22
• Does downsizing promote a firm’s competitiveness and/or performance?
Narrative Review Example
– Less job involvement, org. commitment, & job satisfaction
– More workplace conflict, less autonomy & support
– More voluntary turnover & absenteeism
– Declines in job performance, creativity, & quality
- Announcements have negative effects on stock price
- Most research shows negative effect on firm profitability
- Negative effects on reputation - No consistent evidence of
positive impacts on sales growth, labor productivity, or R&D investments
Individual Outcomes (k=29) Organizational Outcomes (k=36)
Datta et al. (2010): reviewed 91 studies (1984-2008) Examine the antecedents and consequences of employee downsizing, defined as “a planned set of organizational policies & practices aimed at workforce reduction with the goal of improving firm performance”
• How substantially different are women compared to men?
Gender “Differences Hypothesis”
Since 1992, sold over 50 million copies, with multiple reprints…
And, now it’s an Off-Broadway play!
Gender “Similarities Hypothesis”
Examined sex differences on a host of factors § E.g., cognitive abilities, communication, personality traits,
psychological well-being, work behavior, motor skills, moral reasoning
Findings: § 78% of studies showed no meaningful differences § In a handful of cases, major differences were found (e.g.,
motor skills, sexuality, physical & verbal aggression)
The data simply do not support the widely held belief that men and women are polar opposites
Hyde (2005): Reviewed 45 meta-analyses
• Are 10,000 hours of deliberate practice necessary for becoming an expert?
The “10,000 Hour Rule”
The “10,000 Hour Rule”
• Is “grit” the most important predictor of success in today’s world?
“While intelligence matters, a high IQ or talent or any other factor, there is no greater predictor of success. The number one predictor of a person’s success is their unflagging commitment to a long-term goal.”
“Grit matters more than any other talent or trait. The key to success is grit.”
• Is EQ more important to job success than IQ?
Personal Attribute (criterion: job performance) b
Relative Weights (% of R2)
Cognitive ability .66** 69.0%
Emotional intelligence .33** 13.2%
Neuroticism .18 1.5%
Extraversion .01 1.0%
Openness -.27 4.3%
Agreeableness -.01 0.8%
Conscientiousness .28* 10.2%
R2 = .49**