Proposal for platform to run javascipts experiments

run by the University of Oslo IT services

Author

Dr. Athanasia M. Mowinckel

Summary

Show the code 
dt |> 
  transmute(dept = work_institute) |> 
  separate_rows(dept, sep = ",|/|and") |> 
  mutate(
    dept = gsub("psychology", "Psychology", dept),
    dept = str_trim(dept),
    dept = gsub("\\([^()]*\\)", "", dept)
  ) |> 
  group_by(dept) |> 
  tally() |> 
  ggplot(aes(label = dept, size = n, color = n)) +
  geom_text_wordcloud(show.legend = FALSE) +
  scale_size_area(max_size = 10)

Freetext affiliations of responders

On November 14, 2022 the survey “Online behavioural data collection” was launched to investigate the interest in an online tools that would help researchers in Norway run experimental behavioural data collection online in a Norwegian legal framework. As of December 28, 2022 the survey has collected 51 responses from 7 institutions. Collectively, the respondents (conservatively) estimate receiving 24 970 responses using such tools each year. All researchers at any Norwegian Institution were welcome to answer the questionnaire, as long as they collected (or were interested in collecting) data using Qualtrics, Pavlovia, Gorilla etc, or who generally collect computerised timeseries data locally.

Background

Running online behavioural data collection that collects time series data is important in several social and medical science fields. Response and response times collected in relation to stimulus shown in a specified way is a crucial instrument to such fields.

Currently, such data is collected through external services that are run on either US or European servers, outside a Norwegian legal context. While many collect data on these anonymously, the providers claim these are truly anonymous, but the end-users (scientists) have no insight into whether these claims are true.

Furthermore, many also need to collect this sort of data in a non-anonymous form, to connect the data to previous or existing data from the same participant. No external service can provide this alternative in a secure way that is compliant with current data security legal framework.  

USIT is in a great position to be able to provide such a tool, either through integration with their existing Nettskjema survey tool or as another stand-alone service connected to other USIT services. In an ideal world, having an instance where a scientist can set up experiments (for simplicity for web-application, we will focus on javascript experiments) that could send data to EduCloud or TSD, would provide a valuable and secure way of collecting such data.

This type of service would provide a legal framework for not only green to red data, but even black data through TSD integration. 

Survey results

Of the 51 responses to the survey, the majority of responders (80%) were from UiO, and are mainly within the social sciences (56%). A total of 27 (52%) responses are from Psychology (PSI, Norment, LCBC, Ritmo, Promenta, other Psychology Institutes). Full presentation of main work place and field of study can be found in Figure 1. The survey relied on snow-ball sampling to reach researchers that would be interested in the tool suggested, and as such has likely not reached all potential interested.

Show the code 
ggbar(dt, work_place) + 
  plot_spacer() +
  ggbar(dt, field) +
  plot_layout(widths = c(1, .3, 1))

Figure 1: Summary of respondents places of work, and their general field of research

Expected yearly responses

Responders estimate receiving 24 970 responses using such tools each year. However, the question for this estimate was by several responders pointed out to be somewhat ambiguous1. As such, several have responded with significantly higher numbers of responses in the free-text answers (some with a scale up 20 fold of their numerical answer).

  • 1 Could you provide an estimate on how much data you would expect to get over the course of a year? Think of an entire time series data set as a single data point, so if you expected 100 time-series datasets for a year, you would respond 100 here.

    • Show the code 
    dt |> 
  mutate(work_place = fct_infreq(work_place)) |> 
  ggplot(aes(1, estimate, fill = work_place, colour = work_place)) + 
  geom_bar(stat =  "identity") +
  labs( y = "Estmiated number of yearly responses") +
  scale_y_continuous(labels = thousand) +
  theme(axis.text.x = element_blank(),
        axis.title.x = element_blank())

    Figure 2: The number of yearly expected responses from each institution.

    While the initial idea was to create a tool for time-series behavioural experiments, the tool could also be used to run more specialised surveys that are not possible to run in the current Nettskjema framework. This could be including features like more advanced conditional logic (forwarding from one question to another), randomisation of question order (often necessary with certain standardised questionnaires for validity), ability to capture time spent on individual questions etc. In a sense, this tool could alleviate some specific requests that are hard to solve within the Nettskjema framework. An application as suggested here could provide an alternative survey tool for those who are comfortable coding their own solutions, for instance javascript (which would be the easiest for web-implementation).

    As the survey was sent out with the expressed intent to reach those interested in time-series data, it would not necessarily reach those interested using this tool to send out highly customised surveys. The possible number of interested parties of such a service could therefore be much larger than here reached.

    Programming language

    There are quite some of the survey respondents that already use javascript for their experiments (40%, see Figure 3 for summary of languages used to create experiments). This is unsurprising, given the high amount of responders that use existing online tools which require javascript experiments (82%). Indeed, using such a tool as an alternative to Nettskjema was pointed out by several respondents with the above mentioned features as key to their needs even when “just” using it as a survey-tool and not for time series data collection.

    Show the code 
    ggbar(dt, tools) + 
  plot_spacer() +
  (
  ggbar(dt, plang) +
    labs(title = "Language")
) +
  plot_layout(widths = c(1, .3, 1))

    Figure 3: Summary of tools and languages reponders already use for theri behavioural experiments.

    During the time this survey was live, I was made aware that there is a service (Testable) that UiO has some license to that a team from Educational sciences has secured. Testable is a possible solution, though during exploration of this I still see little possibility to easily connect survey responses to existing data or get control over ID/anonymity in a secure and easily verifiable way. Also, based on communications with current users, Testable can only be used up to yellow data in LSIS terminology, thus not fulfilling the needs in terms of data security for quite a lot of researchers. There might be options to connect Testable to other University services that to handle more secure data, though this requires disengaging Testable’s participant platform. However, Testable would still only solve part of the requested functionality.

    There are several readily available javascript frameworks to create such surveys/experiments in javascript:

    It should also be noted that a couple of responses did indicate they did not use any programming language to create their experiments. Qualtrics, for instance, also provides the ability to create experiments using a GUI, and several other services provide easily adaptable experiments through csv specifications of stimuli, onset time, stimulus duration, allowed responses etc. The initial proposed service here does not include the build of a GUI for users to create their experiments, but rather a way to deploy already created experiments. Creating such an addition might be something to consider in the long-term, after assessing initial success of the service.

    Connected services

    The last survey question probed what services respondents would like this suggested new service to connect to. Depending on the scope of one’s research, various tools could be of particular interest to connect to to create data flow to be easier. Several mention wanting to connect to Nettskjema (diktafon, video/image), and the possibility to send data to TSD and EduCloud as important. Integration with already existing tools developed by USIT for data collection seems to be a vital point for many.

    Several respondents mention (6) the want to connect to Prolific Academic, which is a service to connect researchers with reliable participants with the ability to pay for their time. Aside from Prolific, several more respondents mention the ability to pay their research subjects for their time without needing to retain their personal information as important.

    Additionally, several mention wanting to connect data to registry data (SSB, student grading, pasientjournal, etc,), and as such some possibility to connect data to other existing data seems to be valuable. This service might likely be highly valuable with possibility to use MinID login to verify participant identity for certain types of projects.

    Scalability & Stability

    Given that quite a large number of the respondents seems to have proficiency in creating their experiments in javascript, one could argue they could set up their own NREC server and launch their experiments from there. However, setting up NREC servers to such a task is not trivial and requires technical expertise that we cannot expect all researchers to have (or have the time and interest to obtain among all their other activities). Furthermore, for red and black data it would require researchers to know how to programatically connect responses to services allowed to store such data. NREC servers might also provide some issues of scalability if the experiments launched require many answers in short succession (look at Nettskjema’s scalability issue when the Corona survey launched). And lastly, any NREC launched service will require maintenance of used software/system libraries to patch security issues. These are not trivial issues for researchers to deal with, and should ideally be dealt of by professionals.

    Proposal

    The survey revealed several applications and needs that were not previously thought of, and therefore the complexity of the overall proposal has also increased. Still, there are ways to split up the process into several pieces, that may be developed over time.

    1. Platform for launching JS experiments

    As a first instance, a platform that can launch already programmed JS experiments could enable advanced users to start exploring and providing feedback on the service.

    Platform functionality / UI:

    • Initiate a project/experiments (i.e. an instance where a JS experiment would run).
    • Choose storage location (on platform server (green, yellow), EduCloud (red), TSD (black)).
    • Generate links to send out to participants (should ideally also take URL arguments like nettskjema to pre-populate certain experiment fields).
    • Something akin to Nettskjemas Response overview, but no need for report generation (just something to show that responses are coming in and how many).

    2. Database of existing experiments for import

    Many experiments are likely to be re-used over time, and some are adaptations of staple experiments that exist in various research fields already. A solution akin to Pavlovia, where one can explore existing experiments and import these to your own projects, could greatly alleviate the pain of programming custom experiments from scratch. Users can make their own experiments (not data) available for reuse, and other users can import/download and adapt the experiment to their needs. These experiments should all be published here with cc-by licences.

    Platform functionality / UI:

    • Gallery of experiments with title
    • Possibility to import to project, or launch a test to see it in action
    • More functionality like tags etc could come over time

    3. Connecting to more services & user-friendliness

    Given the popularity of the platform, the next important aspect would be exploring connections to other USIT services and external services like prolific. At this point, integration with GitHub/GitLab/Bitbucket etc for easy update of JS code would be amazing. Further development for the user friendliness of the platform would also naturally be part of this work.

    4. Platform for creating JS experiments

    The last piece of the puzzle would be something that would aid researchers without programming skills to create their JS experiments. This is non-trivial and the need for this was only expressed by a minority of responders. Still, for these researchers the entire platform would be unusable without this aspect. While the jsPsych-Redux-GUI project seems to be inactive, there might be other similar projects available that would be used for this purpose and minimize USITs need to reinvent the wheel.

    Conclusion

    There is a great deal of interest in a tool that will allow scientists to run more specialised behavioural data collection than may be provided through nettskjema. It is highly likely that such a tool could have wider adoption than indicated here, as this survey has been snowballed among primarily psychologists at UiO, which is the closest colleagues of the survey developer. We expect the general interest is indeed higher, and implementations would likely vary greatly depending on field of research.