Two months into data collection, something feels off. You can’t quite name it – until you sit down to analyse your first few responses and realise: this instrument can’t actually answer the question you’re asking.
This happens more often than you’d think, and it’s rarely carelessness. It usually happens because the research question was refined and sharpened after the methodology was already locked in – a natural, common sequence in dissertation writing – and nobody went back to check the two still matched.
Why This Mismatch Is So Easy to Miss
A proposal’s research question and its methodology are usually written at the same time, in the same document, so they feel aligned by default. But research questions almost always sharpen as students read more literature and think harder about their topic – a broad question about “the QS role in claims management” narrows into a specific question about how QSs navigate competing loyalties during a dispute. The methodology, meanwhile, often stays as originally written, because revisiting it feels like starting over.
The result: a question that’s become genuinely exploratory and descriptive – “how do QSs experience X” – paired with an instrument still built for the original, broader, more confirmatory version – a structured multiple-choice survey designed to measure agreement with predetermined statements.
The Check: Read Your Instrument as If You’d Never Seen the Question
Here’s a direct test. Take your interview schedule or survey instrument and your research question, and ask honestly: if a stranger only had the instrument, and not the question, would they be able to guess what you’re trying to find out? And separately: if all your data came back exactly as designed, would it actually let you answer your stated question – or just something adjacent to it?
A descriptive, exploratory question (“how do participants experience X,” “what factors do participants describe as influencing Y”) needs open, flexible data collection – semi-structured interviews, open-ended responses, room for participants to raise things you didn’t anticipate. A confirmatory, comparative question (“does X predict Y,” “is there a significant difference between groups”) needs a measurable instrument built around defined variables, usually validated scales.
Mixing the two doesn’t just create messy analysis. It means your findings, however carefully collected, can’t actually speak to the question you said you were answering – because a multiple-choice survey can’t surface the “how” and “why” an exploratory question requires, no matter how many respondents complete it.
Catching It Early vs Catching It Late
Caught before data collection, the fix is usually straightforward: revise the instrument, or in some cases revise the question to match the instrument you’ve already validated and piloted. Caught after data collection – which is when it’s usually noticed, because that’s when someone finally sits down to analyse – the options narrow considerably, and the conversation with a supervisor becomes much harder.
This is why the check is worth running deliberately, rather than trusting that alignment happened automatically because both documents were written in the same proposal.
FAQ
What if I’ve already collected data with a mismatched instrument? It’s sometimes recoverable – occasionally the question can be reframed to fit what the data can actually support, though this needs care to avoid simply working backwards from your results. This is worth a direct conversation rather than guessing alone.
Can qualitative and quantitative elements be mixed at all? Yes, in a genuinely mixed-methods design – but that requires the methodology to explicitly justify why both are needed and how they’ll be integrated, not an accidental blend of question and instrument types.
How do I know if my question is exploratory or confirmatory? A rough test: if your question starts with “how” or “what factors,” it’s likely exploratory. If it asks whether something predicts, causes, or differs significantly from something else, it’s confirmatory.
Next Step
If something about your method feels slightly off but you can’t quite articulate why, that instinct is worth listening to before data collection goes further. I offer a free 15-minute introductory call to talk it through.