Research Methods

From WebScience

Jump to: navigation, search

Fact Box
Module Web and Society
Christiane Gruenloh
Credits 3
Term Term 1, Term 2
Course is required
Current course page Summer 2018
Active Yes

The course Research Methods deals with research methods relevant to Web Science and the design of interactive systems.

The Big Picture

Whether it is the evaluation of existing system, or the conceptualisation, design, or development of a new technology, conducting research is an essential part in this endeavour. The design of such a system "requires finding or generating data, analysing it and drawing conclusions" [1]:9, i.e. doing research. An interdisciplinary field such as Web Science with multiple theoretical approaches opens a world of different research methods originated within the various disciplines like psychology, sociology, art and design, computer science, and so on. Since there is no such thing as the perfect method, methodologies and methods have to be chosen wisely based on the purpose of the research. Some methods might be combined, in order to bring together the strengths and compensate weaknesses of the particular methods. The crucial duty remains to choose between the research methods and combine and arrange them appropriately according to the problems and phenomena under study.

In order to be able to justify the choice and use of methodology and methods, it is crucial to understand the basic elements of any research process: methods, methodology, theoretical perspective, and epistemology [2]:2ff.

The research process has to be planned and designed and starts with a carefully phrased research question, which guides both the research design and its conduct. Research questions are closely related to methodology, theory, and epistemology. While in quantitative research, the question is clear and fixed, qualitative research starts with a research question which can also evolve and be refined during the course of study [3]:44.

Knowledge generation within social research is based on data, which can be distinguished between qualitative and quantitative data. Depending on the research design, this data can be archival (secondary sources, such as websites, blogs, newspapers, etc.) or elicited (collected by the researcher, such as interviews, surveys, observational field notes, etc). The collection and analysis of empirical data requires ethical considerations. Ethical implications in research are e.g. rights of research participants and the researcher's responsibilities towards directly and indirectly involved people [1]:54ff.

Depending on the type of data generated and/or collected and the research question, different types of data analysis methods are appropriate (e.g. statistics, thematic analysis), and consequently different quality criteria apply for evaluation of the research. Finally, the results have to be communicated, i.e. the answer to the research question has to be presented (e.g. in a written conference paper, research report, thesis, and so forth).

Intended Learning Outcomes

The participants of this course

  • understand the basic elements of the research process
  • know how to design a research process in relation to a specific research question
  • know important research methods and appropriate applications within Web Science
  • know ethical implications in relation to social research methods
  • are able to critically reflect on and discuss strengths and weaknesses of research methods
  • know different analytical tools and understand different quality criteria in quantitative and qualitative research

Structure of the Course

Introduction To Research and Research Methods

Oates defines research as: "the creation of new knowledge, using an appropriate process, to the satisfaction of the users of the research."[1]:4 Depending on the academic discipline, criteria for what is considered good research might vary. Research can be carried out for many different reasons, e.g. [1]:16

  • Adding to a body of knowledge
  • Solving a particular problem
  • Investigating what happens (e.g. after a new technology is implemented in the real world)
  • Finding evidence to inform practice (e.g. how are systems developed and how could these practices be improved)
  • Developing a greater understanding of people and their world
  • To contribute to the wellbeing of others
  • To test or disprove a theory
  • To develop, improve or extend a theory, research tool, model
  • Exploration of a topic, area, field or a particular situation

In many approaches for designing interactive systems (e.g. participatory design, value sensitive design, user-centred design, contextual design), some form of research is an essential part in order to understand e.g. the problem that the system is supposed to solve, or the prospective users and their needs which should be addressed in the design.

The research process consists of four elements as outlined by Crotty (1998)[2]:3

Basic Elements of the Research Process
  • Methods: the technique or procedures used to gather and analyse data related to some research question or hypothesis.
  • Methodology: the strategy, plan of action, process or design lying behind the choice and use of particular methods and linking the choice and use of methods to the desired outcomes.
  • Theoretical perspective: the philosophical stance informing the methodology and thus providing a context for the process and grounding its logic and criteria.
  • Epistemology: the theory of knowledge embedded in the theoretical perspective and thereby in the methodology.

Research Questions

Research questions are central in that they help to [4]:65

  • organise the project
  • give the project direction and coherence
  • provide project limitations and boundaries
  • keep the focus
  • provide a framework for writing
  • identify the kind of data needed

A simple model of research involves to phrase research questions, determine what kind of data is necessary in order to answer these questions, design the research to collect and analyse the data, use the data to answer the questions, and finally to communicate the results[4]:67.

According to Punch (2014), the criteria for good research questions are:[4]:76

  • clear: easily understandable and unambiguous
  • specific: specific enough to connect to data indicators
  • empirical: answerable with data and indicate what data will be necessary
  • interconnected: relevant to each other in some meaningful way
  • substantively relevant: interesting and worthwhile

Sampling and Recruitment

Both in qualitative and in quantitative research sampling is equally important, because no study can include everything[4]:160f, 243f. Many decisions have to be made, e.g. which people to involve (e.g. whom to interview or to invite to a focus group), which situations to observe, the settings and processes. Even when it comes to analysis of documents, the researcher will be confronted with sampling issues[4]:161f.

A range of sampling methods exists[5]:93ff[3]:56ff

  • probability sampling: (sample will be representative of its population)
    • random sampling
    • systematic sampling
    • stratified random sampling
  • non-probability samples:
    • convenience sample
    • purposive sample
    • quota samples
    • snowball samples or friendship pyramiding

There is a difference between qualitative and quantitative research, when it comes to sampling. Quantitive research tends to focus on people sampling and often uses probability sampling directed at representativeness, while qualitative research use rather deliberate sampling ('purposive sampling'), with some purpose or focus in mind[4]:161. In general, the sampling strategy should fit with the purposes, questions and overall strategy of the research [4]:164.

The decision what kind of sample will be included is directly related to the recruitment, because potential participants have to be informed about the research through some form of advertisement[3]:59ff. Different approaches may be applied in how the advertisement takes place, however, recruitment materials and strategies may have to be approved under the ethics protocol [3]:59.

Ethical Considerations

Due to scandals in research and horrific experiments on humans in the Second World War, ethical codes and regulations have been developed to define ethically acceptable and unacceptable practice [3]:61ff[4]:41ff. The British Psychological Society developed in their Code of Ethics and Conduct [6] four ethical principles, which have been elaborated by Braun and Clarke (2013)[3]:62ff:

  • Respect
    • the need to maintain privacy and confidentiality
    • the need to obtain informed consent from participants and avoid deception
    • the need for self-determination (i.e. participants know about their right to withdraw from research during or after it has been conducted)
  • Competence
    • awareness of professional ethics
    • standards of ethical decision-making
    • limits of competence
  • Responsibility
    • cover the general concept of "doing no harm"
    • protection of participants
    • minimise risk and inform participants about risks, of their right of withdrawal
    • standard of debriefing after participants have taken part
  • Integrity
    • standards of honesty and accuracy

Obtaining informed consent from participants necessitates that participants have been thoroughly informed and understand why the study is being conducted and what procedures are involved [7]:381ff. Typically participants are provided with[3]:67:

  • some form of advertisement / information
  • a much more detailed participant information sheet (PIS)
  • a consent form

According to the American Psychological Association, participants have to be informed about: [8]

Informed Consent to Research
  1. the purpose of the research, expected duration and procedures;
  2. their right to decline to participate and to withdraw from the research once participation has begun;
  3. the foreseeable consequences of declining or withdrawing;
  4. reasonably foreseeable factors that may be expected to influence their willingness to participate such as potential risks, discomfort or adverse effects;
  5. any prospective research benefits;
  6. limits of confidentiality;
  7. incentives for participation; and
  8. whom to contact for questions about the research and research participants' rights. They provide opportunity for the prospective participants to ask questions and receive answers.

Several associations provide templates for informed consent forms, e.g. the World Health Organization (WHO) [9].

Although most of the common guidelines apply also to online research, some of the requirements relating to "offline" research are difficult to be met in Internet research [1]:64ff, [7]:391. Internet research poses difficulties in terms of how people are regarded: are they subjects of research or authors of works (e.g. websites, blog posts) [1]:65. In this relation it is also questionable, whether the Internet can be seen as a text (which would lead to the adoption of certain procedures, e.g. credit their work) or a space (thus, took the communication place in public or private space) [10]:140f. While some might argue, that information published online is already public knowledge and people should be aware of that, they might in fact "not expect that their remarks are going to be read by others outside of the community"[10]:141. The requirement for informed consent may be difficult to achieve when it comes to online studies [1]:64. As mentioned before, one essential part of the informed consent form is the right to decline to participate and to withdraw from the research, which is not possible for the subjects if they are not informed that their remarks are being used in research.

In addition, ensuring anonymity for participants especially in qualitative research can be difficult, because direct quotes can easily be accessed through a full-text search in a search engine and hereby the original author might be identified[1]:65[10]:145.

In 2014 a study on 'emotional contagion' of Facebook users lead to a debate of the ethics involved in this research, especially in relation to informed consent, the right to withdraw participation and personal privacy [11]:217ff. The researchers in that study conducted an experiment in which the news feeds of randomly selected users were adjusted to reduce the amount of emotional content presented in their feed; which led to the findings that "reducing positive emotional expressions let people to produce fewer positive and more negative posts"[7]:218. In reaction to this, APA issued a press release[12], in which they outlined the content of informed concent (outlined above), while adding:

APA reiterated its policy with respect to informed consent in light of a study involving approximately 700,000 Facebook users without their knowledge.

Qualitative and Quantitative Research Methods

Qualitative and quantitative research methods are presented and discussed with regard to their applicability and strengths and weaknesses. To give a brief overview, the connection between descriptive and relational investigations (for example ethnographic research or case studies) and experimental research will be outlined. According to Lazar et al. 2010 empirical investigations can be categorised into three groups with different goals and results. Descriptive investigations enable for an accurate description of what is observed, but don't allow for an explanation why this is happening. In order to reveal relations between different factors, relational investigations could be carried out. Using the correlation analysis helps the researcher to determine, if the result is significant but this also doesn't explain the cause of the relationship. Experimental research consists of controlled experiments which investigate causal relationship of different factors. [7]:20ff

Different methods for data collection will be introduced and discussed: interactive (interviews, focus group), textual data collection (surveys, diaries, secondary source)[3], and observation. Digital methods will also be discussed, which are not just another set of methods, since social research methods have had a computational dimension for a long time [13]:ix


The terms survey and questionnaire are sometimes used interchangeably, however a survey can refer to a complete methodological approach (including sampling, reminders, incentives), in which a questionnaire (i.e. the list of questions) is one element [7]:100.

Lazar et al. outline the process of the development of a survey [7]:111f. It is important to determine early on, how the survey will be distributed. While online surveys may reach a large amount of potential participants, it may also exclude those who do not have access to computers or internet. After the goal and strategy for the survey has been decided upon, the survey questions have to be developed. The wording of the questions and the overall structure is crucial; the respondents have to be able to understand and use the questionnaire. Depending on the research purpose, there may already exist surveys that have already been tested. It is possible to make use of them in the current study; possibly with some modifications.

Survey questions can be structured as:[7]:112f

  • open-ended questions
  • closed-ended questions
    • with ordered response categories (e.g. using scales like Likert scale)
    • with unordered response categories (provide choices with no logical order)

Common problems when it comes to the survey questions are: [7]:113[5]:61ff

  • questions that ask two separate and possibly related questions
  • Use of negative / ambiguous words can cause confusion.
  • Biased wording in questions („Don’t you agree, that...“) may lead to biased responses.
  • responses that are biased due to scale and social norms (e.g. amount of cigarettes, use of drugs)
  • scale points (even or non-even number of points?)

It is therefore crucial that survey questions are piloted in advance. Although qualitative surveys are not as widely used as quantitative, it can generate great data while not being as time-consuming as for example interviews and focus groups [3]:134ff. Since this data is more focused on the topic than e.g. focus groups, the data can be useful for pattern-based analysis, while participants still provide their own answers in their own words (which is important for qualitative research)[3]:137.

Interview and Focus Groups

In qualitative research, one of the most important methods is the interview. The use of interviews are appropriate in order to get access to people's experiences, perceptions in their own language (their meanings, definitions, concepts), exploring their understanding about things they have a personal stake in. Interviews can be[3]:78ff

  • structured
    • questions and response categories are predetermined by researcher
    • commonest type of interview in quantitative research
    • interview does not attempt to get to any great depth [4]:145.
  • semi-structured
    • dominant form for qualitative interviews; called the "interview guide" approach
    • list of questions prepared, but still scope for (unanticipated) issues raised by the participants
  • unstructured
    • no prepared questions, but at most a list of themes or topics to discuss
    • interview is strongly participant-led

Similar to the questionnaire design, it is very important to thoroughly prepare and pilot the interview guide. Important aspects to consider when conducting interviews are [3]:81ff

  • Interview Guide
    • Opening and closing questions
    • Sequencing of questions
    • Constructing and wording questions
    • Prompts and probes
    • Research questions are not interview questions!
    • Social desirability
  • Participants
    • interviewing people who are familiar
    • interviewing strangers
    • interviewing across differences
    • the power aspect in interviews
    • participants in distress
    • interviewing vulnerable people
  • The Interview
    • Location
    • Data collection (note-taking vs. audio-recording)
    • Personal disclosure
    • Opening the Interview
    • Using silence, managing nerves, showing interest vs. empathy
    • Closing the interview

Today interviews can also be conducted virtually (e.g. via telephone, email or video calls), which has certain advantages and disadvantages. Here the distinction between survey and interview methods can become blurred [14]:36. While it might be convenient and empowering for participants who can participate from home, it might also be more time consuming for participants (e.g. writing takes longer than speaking), the researcher has less control over the interview and some forms of information are lost [3]:97ff. Braun and Clarke (2013) outline important differences between virtual and face-to-face interviews in relation to the process:

  • Length of interviews
  • Participants type rather than speak
  • Negotiating Informed consent
  • Establishing trust and rapport
  • Questions: lots of room for misinterpretation, miscommunication
  • Sending questions (all at once or in batches)
  • Maintaining momentum

Whereas interviews are usually conducted one-to-one, focus groups allow to collect data from multiple participants at the same time. Instead of having an interviewer asking questions, a moderator raises discussion points for the participants to discuss among themselves [3]:107ff. Focus groups are a good way to elicit a wide range of views, perceptions, and understandings of an issue and have been considered suitable to conduct research with people from marginalised communities [3]:110f. While the focus group discussions can be a reasonably effective and inexpensive tool for easily gathering a broad range of opinions[7]:192 the transciption and anaylsis is very time-consuming while not allowing for in depth follow-up of individuals' views or experiences [3]:113.


A diary is generated by participants and require participants to record their thoughts, feelings, experiences and/or practices over a specific period of time[3]:147. They can take many forms (handwritten, hard-copy, digital, audio- or video recorded) and require regular entries over a certain period of time. Diaries can be useful where an observation or experimental design would not be appropriate; where the research question leads to data than cannot easily be observed or measured (e.g. feelings of frustration), and are useful in combination with other research methods (triangulation)[7]:138.

Archival Data

Another way to collect data is to make use of documents that already existed and is publicly available, thus the researcher had no role in the production of the data. Those secondary sources of data include different types of material in printed copy, electronic, and broadcast formats [3]:152. Sources of documents can be [1]:234

  • Organisational documents (e.g. formal records, public records, informal records)
  • Documents produced by individuals (e.g. personal papers and communications)
  • publications (e.g. academic and popular literature, guides)
  • previous research (e.g. research data and field notes from archived studies)
  • multimedia documents (visual, aural, electronic sources, e.g. pictures, sounds and music, websites)

While the Internet is ideally suited for document-based research, certain problems should be considered[1]:239

  • scope for misleading documents might be larger than in the offline world
  • as with offline documents: are a partial, selective representation of something (i.e. what people publish online as their opinion might not be what they really think)
  • like all documents: have been produced with a particular audience and purpose in mind
  • consideration of ethics important: although certain communications might be publicly accessible, authors and intended readers often perceive them as private


Observation has its tradition in social sciences and as a data collection method can be structured or unstructured (similar to interviews) [4]:153ff. In research, observation is used as a data generation method "to find out what people actually do, rather than what they report they do when questioned"[1]:202. Quantitative approaches are highly structured and require predeveloped observation schedules, while qualitative approaches are much more unstructured and no predetermined categories and classifications are used [4]:153. Two types of observation are relevant with regard to the role of the researcher:

  • Non-Participant observation
    Role of researcher is a detached observer of the situation
  • Participant observation
    Role of researcher is participant-in and observer-of the situation

In participant observation researchers not only observe and write notes, the aim is also to take part in the situation under study in order to experience it from the point of view of the others in that setting [1]:208. Oates (2006) outlines four types of participation:

  • complete observer
  • complete participant
  • participant-observer
  • practitioner-researcher

The roles as such form a continuum and roles might evolve throughout the course of a project [7]:228. On both ends of the continuum is the researcher as being complete participant vs. being complete observer, inhibiting the risk of losing perspective (going native) vs. risking misinterpretation if one only observes [7]:229.

According to Hewson et al (2016), the distinction between observational approaches and document analysis methods can sometimes be blurred in Internet-Mediated Research due to the new ways of interacting and publishing (e.g. blogs may be seen as interactively produced document with multiple authors, as individual diaries or something in between) [14]:36. Kozinets (2010) emphasises, that a participative approach to the study of online culture and communities is at the core of netnography, thus netnography is an adaptation of participant-observational ethnographic procedures. [10]:74

Analysis and Quality Criteria

Depending on the research question and the type of data collected, different analytical method have to be applied to analyse the data.

Quantitative Data Analysis

Quantitative data is based on numbers, typically generated by experiments or surveys and mainly used in the positivist paradigm [1]:245ff. Quantitative data analysis looks for pattern in the data and to draw conclusion; a simple analysis would use tables, charts or graphs, while more complex analysis would make use of statistical techniques[1]:245ff, e.g. descriptive statistics (central tendency; distribution); analysis of variance (ANOVA); identifying relationships (correlation coefficients).

Qualitative Data Analysis

Qualitative data includes all non-numerica data collected through interviews, diaries, etc. (e.g. words, images, sounds) [1]:266ff. While some researchers make use of quantitative analysis on qualitative data (e.g. counting the occurrence of a particular word in a text), most qualitative data analysis involves abstracting themes and patterns from the data collected [1]:266f. According to Braun and Clarke (2013) qualitative analyse can be [3]:173ff

  • descriptive: aims to give voice to a topic or group of people
  • interpretative: tries to gain a deeper understanding of the gathered data; looks beneath the surface and tries to understand how and why the accounts were generated; to provide a conceptual account of the data; and/or some theorising around it

Different qualitative analysis approaches are for example Thematic Analysis, Grounded Theory, Discourse Analysis. Qualitative data analysis involves coding performed by the researcher, which can follow a [3]:175[1]:269

  • deductive approach: guided by an existing theory and theoretical context
  • inductive approach: categories are observed in the data; aims to generate analysis from the bottom up (i.e. the data); not supposed to be shaped by existing theory (although analysis is shaped by researcher's knowledge and standpoint)

Quality Criteria

The quality criteria needed to evaluate the quality of research differ between quantitative and qualitative research. Quantitative research is evaluated in regards to [3]:278ff

  • Reliability
  • Validity
  • Generalisability

Because within qualitative research, the theoretical underpinnings of the particular approaches can be quite diverse, generic criteria have been developed by the British health psychologist Lucy Yardley that are supposed to be theoretically neutral and thus can be appropriated to the methods used [3]:289f:

  • Sensitivity to context
  • Commitment and rigour
  • Transparency and coherence
  • Impact and importance

Didactic Concept, Schedule and Assignments

The course concept comprises lectures, online seminars with discussions, online and offline group work and an introductory and final on-site presence. After a first introductory lecture on site, the subject is treated in three online workshops, that are supplemented by a session on-site. Online workshops are held on three evenings with a duration of three hours each.

Introductory lecture on-site

During the introductory meeting the structure of the course and organisational issues will be outlined.

The lecturer will present an introduction to research and research methods, outlining the motivation for research and the basic elements of the research process. The development of research questions, as well as ethical considerations and issues relating to sampling and recruitment will be discussed.

1st Online Session

The first online meeting will deal with qualitative and quantitative research methods. Students will be separated in groups and work out a study plan for a given research question. The results will be presented afterwards in a plenary session. At the end, organisational issues will be discussed.

2nd Online Session

The second online meeting will deal with the analysis of qualitative and quantitative data, as well as quality criteria for research evaluation. During the session, the students will be separated in groups and apply certain methods. The results will be presented afterwards in a plenary session. At the end, organisational issues will be discussed. The session will end with instructions for the preparation of the next online session and the discussion of unresolved questions.

3rd Online Session

Prior to the third online session, students will submit an exposé for a possible research design for the problem they have identified in their Web and Society Project. During the third online session, the students will present their research design, which should comprise

  • Research purpose and questions
  • Outline of research design, strategy and process

Students and lecturer will discuss the research design and feedback will be provided.

Wrap-up Session on-site

Students will present their research design as part of their presentation of the Web and Society Project. Part of the presentation should be dedicated to the discussion of the research design (e.g. research question, design of the process, application, data collection, analysis).


The participants will work on an exposé comprising a specific research design for a specified research question. This exposé will be submitted in writing, is presented during the third online session and will contribute to the final grade. The main part of the final grade will be the appraisal of the method described and used in the Web and Society Project, as well as the presentation and discussion of the used methods at the on-site meeting.


  1. 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 1.10 1.11 1.12 1.13 1.14 1.15 1.16 Oates, Briony J (2006). Researching Information Systems and Computing. Sage Publications. 
  2. 2.0 2.1 Crotty, Michael (1998). The Foundations of Social Research. Sage Publications. 
  3. 3.00 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 Braun, Virginia; Clarke, Victoria (2013). Successful Qualitative Research - a practical guide for beginners. Sage Publications. 
  4. 4.00 4.01 4.02 4.03 4.04 4.05 4.06 4.07 4.08 4.09 4.10 Punch, Keith F. (2014). Introduction to Social Research: Quantitative and Qualitative Approaches (3 ed.). Sage. 
  5. 5.0 5.1 Michael Sheppard (2004). Appraising And Using Social Research In The Human Services : An Introduction For Social Work And Health Professionals. Jessica Kingsley Publishers. 
  6. The British Psychological Society (2009). "Code of Ethics and Conduct" (PDF). The British Psychological Society. 
  7. 7.00 7.01 7.02 7.03 7.04 7.05 7.06 7.07 7.08 7.09 7.10 7.11 Lazar, Jonathan; Feng, Jinjuan Heid; Hochheiser, Harry (2010). Research Methods in Human-Computer Interaction. John Wiley & Sons Ltd. 
  8. "Ethical Principles of Psychologists and Code of Conduct". American Psychological Association (APA). Retrieved 20 January 2017. 
  9. "Informed Consent Form Templates". World Health Organization (WHO). Retrieved 20 January 2017. 
  10. 10.0 10.1 10.2 10.3 Kozinets, Robert V. (2010). Netnography: Doing Ethnographic Research Online. Sage. 
  11. Hewson, Claire (2016). "Ethics Issues in Digital Methods Research". In Snee, Helene; Hine, Christine; Morey, Yvette et al.. Digital Methods for Social Science: An Interdisciplinary Guide to Research Innovation. Palgrave Macmillan UK. ISBN 978-1-137-45366-2. 
  12. "APA Ethics Code Addresses When Obtaining Informed Consent From Research Participants Is Necessary". American Psychological Association (APA). June 30, 2014. Retrieved 20 January 2017. 
  13. Snee, Helene; Hine, Christine; Morey, Yvette et al., eds (2016). Digital Methods for Social Science: An Interdisciplinary Guide to Research Innovation. Palgrave Macmillan UK. ISBN 978-1-137-45366-2. 
  14. 14.0 14.1 Hewson, Claire; Vogel, Carl; Laurent, Dianna, eds (2016). Internet Research Methods (2 ed.). Sage. 

Past Course Pages