Welcome to PsychCloud, a template for designing, programming, and running Psychology experiments in the cloud!
What is PsychCloud?
PsychCloud is a framework - a stack of technologies - for running experiments online using Google's App Engine.
PsychCloud is a set of code available on GitHub that gives examples of many different functions necessary for websites in general, and online psychology experiments in particular. None of this code is required – it is just provided as a way for you to get started to build your own experiment as an online app.
PsychCloud is the website you are looking at, hosted at psychcloud.org. This is the same code that is posted on github. This website is designed for you to download from github, run on your own computer, and modify as you need. The best way to use this website is to simultaneously use the website and also look at the code for how the website works.
What PsychCloud is not:
PsychCloud is not a stand-alone application designed specifically for psychology studies (e.g., E-Prime, Inquisit, Presentation). It is a framework that can be used to make websites in general.
PsychCloud is not a package (e.g., Psychtoolbox for Matlab, or PsychoPy for Python) of functions and classes for you to learn. All you have to learn to use PsychCloud are the standard technologies of the web.
PsychCloud is not intneded as a replacement for online survey software (e.g., Qualtrics, SurveyMonkey) – it is intended for situation in which you need to build dynamic websites that are much more customizable.
Want to get involved?
We welcome you to get in contact by email (rottman@pitt.edu), or better yet, to request features and post solutions on GitHub!
Read the paper! To be released soon...
PsychCloud's STACK
PsychCloud is built with a stack taken (for the most part) from Using Google App Engine by Charles Severance. This book is now out of date, but it is an excellent introduction to programming for the web that is accessable to anyone, even people who have never programmed before. If you want to use PsychCloud, we highly suggest reading this book a few times (though don't run the code in the book because it is out of date) before diving into our code.
We use Google's Cloud Datastore (abbreviated as ndb) four our database. Google has many other options. YouTube has many videos on this topic. We like the cloud datastore because it is really cheap and there is no configuration when used in google app engine. We view no configuration as a huge plus for psychologists.
The webapp2 web application framework (which processes things like 'get' and 'post' requests).
The Django templating language for passing code from the python back end to the html/javascript front end.
gae-sessions, a sessions library to keep cookies for local storage of logged-in users.
Download the googlecloud template from github. You can download the template from github without using github for your own software development. However, if you plan on developing your own app, we high encourage you to use github to keep versions of your app as you make changes. If you are new, take a look at the GitHub Guides especially the Hello World introduction. Also, check out the all new GitHub Desktop application, so you can use github without using any code, and also check out the GitHub Desktop user guides!
Installing the Google App Engine Launcher GUI application. One of the reasons that we initially chose to use GAE is because there was a handy GUI interface that made it easy to use for beginners. However, as google has launched many other cloud services, it is phasing out the GUI interface. As of the publication of this website, you can still access the "Original App Engine SDK for Python", which includes the GoogleAppEngineLauncher, by clicking here and scrolling down and clicking on the "Download and install the original App Engine SDK for Python" link. This application makes it easy to run your application on your local computer and to launch the app onto appspot.com. However, we have noticed that some of the features now seem somewhat buggy at times, and we would encourage new users to just use the command line interface to avoid possible bugs and because it appears that the GUI will not be offered for much longer.
Installing the Command Line GCloud interface. The gcloud command line package provides many features. (For users new to programming, this is the Terminal application on Mac, and the Command Prompt application for Windows.) Follow the instructions to install the Google Cloud SDK. In particular, you must:
Install Python 2.7
Install the Google Cloud SDK
Run the following three lines of code, one by one, in the command line
Running the app. Once you have downloaded the psychcloud template, put it into a folder somewhere you will want to keep it.
Using the GoogleAppEngineLauncher GUI. Drag and drop the folder of the app into the Launcher, and click 'Run'. Wait for the circle to turn green, and then click the browser button, which will take you to the app in your browser. The other buttons will bring you to the local console (to browse the datastore) and will show you errors in your python code.
Using the Command Line Interface.
Get the full pathname of the PsychCloud folder, which I am calling PATH_TO_YOUR_APP. It should be something like /Users/.../Documents/psychcloud.
To start the server: paste the following code into your command line. Note, keep the quotations around your path name in case you have spaces in the path.
dev_appserver.py "PATH_TO_YOUR_APP"
If this is succesful, you will get a response like Starting admin server at: http://localhost:8000.
Type http://localhost:8000 into your browser. You will get a console screen that looks like this. If you click on the "default" link, you will be taken to the web app, which should look just like this page. The main other function of the console is that by clicking on "Datastore Viewer" you can see any data saved in the datastore.
To stop the local server:
Mac:
Control-C
PC:
Control-Break
Editing the App
Editing Text. Now that you know how to get PsychCloud up and running, you are ready to start to edit it! For example, open the file psychcloud/templates/overview.htm in your text editor, and search for "Editing the App". Change some of the text in the text editor, save it, and reload the page in the browser - your edit should appear!
Editing Code. Much of the code is stored in psychcloud/index.py. You can simply edit that file, save, and refresh the page in your broswer - you do not need to stop and restart the server. You can even edit your data models! One recommendation - make small changes at a time, so if you get an error you know what you changed!
Setting Up Cookies / Sessions Cookies / sessions are discussed below. But to get it set up securely, you need to set a secure cookie key, essentially a password. To choose a secure one, follow these steps:
Open a python interpreter on your computer or online
Run these two lines of code to get a string of 64 random bytes, suitable for criptographic use
import os
os.urandom(64)
Copy the string that gets produced
Open the file psychcloud/appengine_config.py, and find the "COOKIE_KEY". Replace the text "a long random string - you should change this!" with your new string of random bytes.
Debugging
Errors in Python.
If you used the command line interface, then errors in python and django (the interface between python and the front end) will show up in the terminal app (Mac) or the command prompt (Windows). If you are using the GoogleAppEngineLauncher GUI, click the 'Logs' button.
You can write to the log using the following code:
Errors in Javascript. Front-end errors show up in the browser. For example, in Chrome, go View->Developer->Javascript Console, or in Firefox, go Tools->Web Developer->Console.
Example Logging Page (/logging, LoggingHandler)
When you load this page, it will send an error message to the python console as well as the javascript console.
Browsing Data in the Datastore
When you are programming on your computer
If using the GoogleAppEngineLauncher application, click on the 'SDK Console' icon, which opens a browser window. Click on 'Datastore Viewer'
If using the command line interface, when you first start the app, it will return something like Starting admin server at: http://localhost:8000. Go to http://localhost:8000, and click on 'Datastore Viewer'.
If you have just launched the app there will be no data in the datastore, but there will be once you create a user. If there is data, choose the Entity Kind that you want to view, and click 'List Entities.'
After launching the website to the internet, you can browse the datastore on the Google Cloud Platform Console. Select your project, then click on Datastore.
Steps to launching the site to the internet.
The Console is google's site where you go to set up your app, monitor usage, check for errors, browse the datastore, and many other functions. The main page of the console looks like this; it has graphs of usage and information about billing. You can also browse the datastore, which looks like this (here the rows are users with fake MTurk Ids). The console is like a more sophisticated version of the local development server web app, but for the delopyed app rather than the local app under development.
Deploying your app.
If you are using the GUI GoogleAppEngineLauncher, click the 'Deploy' button and follow the prompts.
If you are using the command line:
Adding the project. Go to the console. In the top banner, click 'Select a project', then 'NEW PROJECT'. Note the name of your project as NAME_OF_PROJECT.
Uploading the app. In the command line, run the following code, and follow the prompts.
Uploading the index.yaml. The index.yaml file stores indexes of your datastore queries. If you are using the command line, it needs to be uploaded separately from the rest of the app. The reason is that indexes take some time (maybe 10 minutes for a simple app) to start working. See the console -> datastore -> indexes to see if they are serving. To deploy the index file, in the command line, run the following code. Your index.yaml file is stored in the main psychcloud folder, so the path will likely be something like /Users/.../Documents/psychcloud/index.yaml
Setting up Billing. In the Google Cloud Platform console, click 'Google Cloud Platform', select billing, and set up a credit card, or link to another project you already have set up. You can set a limit of perhaps $10 per day, just as a fall-back. So far in our lab we never go above $10/year (except when enabling other services by mistake, so don't go around turning on random services).
Monitoring for Errors. If you go to the console, slect stackdriver -> error reporting from the navigator menu, you will be able to get email notifications if your app has an error. You can also see errors in the console -> app engine -> Application Errors.
Using your own domain. By default, when you launch your site, it will have a url like www.psychcloud-190817@appspotcom; appspot is a google site that hosts webapps. If you want to buy your own domain (e.g., www.psychcloud.org) and host your app on that site, you first need to buy the domain. You can do this via multiple sources such as Google and GoDaddy, but using google makes some of the following steps easier. Some domains are as cheap as $12/year. After buying the domain, follow these instructions. Your app will always remain available at the appspot url as well as your custom url.
FEATURES
Here is a list of the features provided in this template. The text in red identifies the url for the page, and the name of the handler. You can search for these in index.py to find where the relevant code for each feature.
Logging In./loginLoginHandler
The session information needs to be stored in a cookie. The user key can then be used to identify the user.
Logging Out./logoutLogoutHandler
The session / cookie information needs to be deleted.
Condition Randomization and Stimuli Presentation
Condition Randomization.
In this template, subjects are randomized, between-subjects, to one of two conditions when the user is created; see the UserCreateHandler in the index.py file.
Stimuli Presentation./stimuli_presentationStimuliPresentationHandler
This page shows four ways to generate the stimuli that subjects see.
Saving Data to the Datastore
With a Post Request./saving_data_postSavingDataPostHandler
Post requests load a new page.
With a AJAX./saving_data_ajaxSavingDataAjaxHandler
AJAX can save data without loading a new page.
Seeing and Downloading Data
Browsing the datastore. Google app engine provides an easy way to brows the datastore, either when you are running your app on your own computer, or after it is deployed to the cloud. See the section above on "Browsing Data in the Datastore".
Tracking progress within a page.
The fundamental problem is that if a user is doing a task on a page and refreshes the page, you don't want them to start over. There are three ways to solve this.
Tracking progress within a page with HTML5 Web Storage/tracking_progress_within_a_page_with_HTML5_Web_StorageTrackingProgressWithinAPageWithHTML5WebStorageHandler
The first option is a relatively new technology called HTML5 Web Storage, which stores data in the user's browser. Each time the user performs a task on the page, the progress can be stored in the browser, and then if the page is reloaded, or even if the user logs out and logs back in, the browser knows where the uder left off and the page can fast forward to that point. However, the downside is that the progress is not stored in the datastore, so if a user stops the task, you have no record of how far they got.
Tracking progress within a page with AJAX and sessions./tracking_progress_within_a_page_with_AJAX_and_sessionsTrackingProgressWithinAPageWithAJAXAndSessionsHandler
The second option is to use AJAX to update the cookie / session each time progress is made on the page. With this method, if the page is reloaded, the session knows where the user left off and the page can fast forward to that point. However, similar to HTML5 Web Storage, the progress is not stored in the datastore, so if a user stops the task, you have no record of how far they got. Furthermore, if a user logs out, the cookie is lost, though in many situations this is not an issue because in most studies the user would not be given the option to log out until the end of the study.
Tracking progress within a page with AJAX and the datastore./tracking_progress_within_a_page_with_AJAX_and_the_datastoreTrackingProgressWithinAPageWithAJAXAndTheDatastoreHandler
The third option is to use AJAX to write to the datastore. With this method, if the page is reloaded, the datastore knows where the user left off and the page can fast forward to that point. The downside to this method is that it involves repeated writes to the datastore, which do cost money. However, in our experience the cost ends up being very small; we like keeping the datastore up to date.
Tracking progress across pages.
The fundamental problem is that users can always press the back button, and there is no totally reliable way to disable this feature across browsers. We suggest giving clear instructions at the beginning that pressing the back button could invalidate the study, which usually works fairly well. Still, sometimes users press the back button.
Recording each page load in the datastore./recording_each_page_load_page_1RecordingEachPageLoadPage1Handler
One option is to allow them to go back, but record each time the page is loaded. To do this you need to disable caching (see cache-control in the OrderAcrossPagesRecord1Handler) so that the page will load from the server again even when the back and forward buttons are pressed, rather than loading it from the cache. Disabling the cache may work with different levels of success for different browsers. You can see the page load records in the Page Load Data Table on the Downloading Data page. Though we have not implemented this, if a subject submits a response on each page, you could also set it up such that you will record each response rather than overwriting old responses. This way you can at least know if multiple responses are submitted.
Using cookies to try to keep the user from going back./using_cookies_to_keep_the_user_from_going_backUsingCookiesToKeepTheUserFromGoingBackHandler
A second option involves using cookies / sessions to record progress. Cookies get sent to the server on each get and post request allowing the session to keep track of the correct page number, and loading the correct page number even if the back button is pressed or the wrong url is entered. This is stored in the 'page' variable in the session.
This option also requires caching to be disabled, works with different degrees of success across different browsers. At least it sometimes alerts the user that they cannot go back without reloading the page, which will likely discourage them from pressing the back button in the future. (Note, this system and the one before it could also be merged to record page loads into the datastore and to also use the datastore instead of cookies to keep track of which pages have been loaded.)
Single Page App.
If you really care about preventing a user from going back to prior pages, the only true option is to make a single page app. In effect, this is like the 'Tracking progress within a page' examples, but it also involves more javascript to make more dramatic changes in the page as the user progresses through the study, which can be accomplished by hiding and showing different elements as the user progresses through the study.
Piping Text/piping_textPipingTextHandler
A common task in experiments is for you to have use different instructions or cover stories for different conditions, and sometimes you need to repeatedly refer back to the same word or phrase for different conditions. This page shows a couple of different methods for doing this.
SSL / TSL
By default the PsychCloud template is set up to use SSL or TSL for security. You will know it is working because the url should appear as 'https' instead of 'http'. Note that this only shows up once the app is deployed, not when it is running on the local development server. This feature is implemented with the 'secure' tag and importing SSL in app.yaml
Linking with Qualtrics, SurveyMonkey, etc.
Passing data to third party sites.
If you want to have your participants first use your own site, and then pass them off to a third-party survey software to do some additional tasks, it is important to be able to track users across sites by keeping the same username. Many sites including Qualtrics and SurveyMonkey allow you to pass variables through url links. So all you need to do is to is create a custom url for each user that appends their username using the methods for described in the above links.
Retrieving data from third party sites./pass_info_into_psychcloudPassInfoIntoPsychCloudHandler
It is also possible to pass information from some third party sites like Qualtrics into your PsychCloud site. For example, suppose that you want users to first do a survey on qualtrics, and then do a task on your site. You can keep track of their username by first having them enter their username in the other site, and have the other site generate a url with their username added on to the end of the url (see the link above). We have currently made PassInfoIntoPsychCloudHandler with a get request that is able to receive a field for username. For example, if copy and paste the following link into your browser as a url, and substitute in a new username at the ned, it will make a new user with that name. Note that there is actually no html page to go along with this link, and also, note that uses a query string to send data through a get request.
This page provides demonstrations a way to programmatically interact with Amazon Mturk through Boto. You could use this, for example, to automatically bonus a participant at the end of a study based on their performance in a task, or to automatically make a participant eligible or ineligible for a follow-up study, or to automatically email a subject through their MTurk account. In fact, anything that you can do on the MTurk website you can do through Boto, and there are things that you can do through Boto that you cannot do through the website (e.g., such as emailing participants). The documentation for the MTurk Operations and their implementations in Boto are available from the following sites:
Before you can access your MTurk account, you must obtain your Amazon Web Service (AWS) access_key aka AWS_ACCESS_KEY_ID and your secret_key aka AWS_SECRET_ACCESS_KEY. Please note, your AWS Access Key ID and your AWS Secret Access key are confidential information - it is similar to your username and password to your account. Thus, you should be skeptical of giving anyone else access to it, including typing it into this website. This website does not store this information, but if you feel uncomfortable inputting them here, then don't use this page! If you choose to go ahead and create access keys, follow these instructions from TurkPrime, another third-party site that interfaces with MTurk.
Amazon has built a 'sandbox' that allows developers to test code in a test environment. Any code executed in this environment does not actually withdraw fees from the MTurk account and MTurk workers will not accept the hits. You can switch back and forth between the sandbox and the production version of Mturk by
switching between lines 950 and 951 in psychcloud/lib/boto/connection.py.
Connecting with Mturk through boto only works when this site is deployed on the cloud; it does not work when just running on your computer.
In order to get Boto to work, I had to make some modifications. Notes for these modifications as well as thoughts for the future are in psychcloud/lib/boto/__notes.txt.
Boto Get Account Balance/boto_get_account_balanceBotoGetAccountBalanceHandler
This page provides a demonstration of how to get your account balance.
Boto Querying a HIT/boto_querying_a_hitBotoQueryingAHitHandler
This page provides a demonstration of how to get information about a HIT.
USEFUL RESOURCES
Demographics Survey/demographicsDemographicsHandler
This page asks the standard demographics requested by the NIH.
Completion Code/completion_codeCompletionCodeHandler
You can use a page like this at the end of a study to give users a completion code that they can enter into MTurk.
Fixed Width./fixed_widthFixedWidthHandler
Many modern websites are designed so that they scale well to various screen sizes including mobile phones, which is called 'responsiveness'. For example, on this page, see how the page changes as you change the width of the page. However, for psychology experiments, we often want the page to look exactly the same and we would want participants to use a browser width that is at least as large as the page we build. We often set the width of the main div to 800 or 1000 pixels, which should fit on almost all screens participants are likely to use at home.
Displaying a header on all pages. Note how all the pages on this site have the same header. This can be useful for a consistent site look - or you can even have links to instructions in the header. Django's templating language makes this easy using the _base.htm file in the templates folder. Each other html page then must have a line of code at the beginning that extends the _base.htm file.
FUTURE FEATURES
Provide link to paper on psychcloud.
Keeping users from taking the same study multiple times or similar studies.
Get logging to work when handlers put on a separate pages
Explain how to publish to the web directly from github.
Funding:
This material is based upon work supported by the National Science Foundation under Grant Numbers 1651330 and 1430439. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.