Common AppEngine bulk upload issues

Google’s documentation for AppEngine provides a very nice walkthrough for uploading and downloading data. But it doesn’t necessarily help with troubleshooting. Here are a few problems that multiple members of the class have run into and their solutions.

Often you’ll store your database models in a common file (models.py, say) so that your bulk upload and your main code can access the same model. A common error you’ll have in this case is that importing models.py will fail as follows:

  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 3467, in LoadConfig
    ('', 'r', imp.PY_SOURCE))
  File "loader.py", line 10, in 
    import models
ImportError: No module named models

This has to do with the PYTHONPATH environment variable. I’m guessing that because appcfg.py is located in another directory, Python won’t search the current directory for .py files to import modules from. So you want to set (or ideally, append onto) PYTHONPATH to have “.” or the current directory.

How that works on your system depends on your shell. For me, using bash, this worked:

export PYTHONPATH=$PYTHONPATH:.

I expect it’s significantly different on Windows, but a search for how to set PYTHONPATH in Windows should give you some hints.

Next, you may run into authentication errors. Uploading to appspot.com, you might see an error like the following:

[INFO    ] Connecting to test-env.appspot.com/remote_api
[ERROR   ] Exception during authentication
Traceback (most recent call last):
 File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 3063, in Run
   self.request_manager.Authenticate()
 File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1148, in Authenticate
   remote_api_stub.MaybeInvokeAuthentication()
 File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 545, in MaybeInvokeAuthentication
   datastore_stub._server.Send(datastore_stub._path, payload=None)
 File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 344, in Send
   f = self.opener.open(req)
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 389, in open
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 502, in http_response
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 427, in error
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 361, in _call_chain
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 510, in http_error_default
HTTPError: HTTP Error 404: Not Found
[INFO    ] Authentication Failed

When you see “Authentication Failed” after a 404 error, that’s likely a sign that you have the wrong handler in place to accept the remote_api calls. In app.yaml, make sure that you have a URL handler for /remote_api in your app.yaml and make sure it’s listed above any catch-all handler for main.py. It should look like the following:

handlers:
- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin
- url: .*
  script: main.py

If you’re running the bulk uploader against your development server running on your own machine, then (as per this question and this walkthrough) you should leave out the login: admin line in order to avoid having to authenticate. If you are asked to authenticate, you should use your Gmail username and password. And if you’ve entered the wrong username and password at some point, you may need to delete your cookies file.

Uploading or downloading data often exposes problems with non-ASCII characters in either your CSV file or your AppEngine datastore. If you’re uploading non-ASCII data, make sure to encode data in UTF-8 rather than using the plain str function (as described in this bug thread).

class MyLoader(Loader):
  def __init__(self):
    Loader.__init__(self, 'MyModel', [('field1', lambda x: unicode(x, 'utf-8'))])

And our own Karen Nomorosa and Hyunwoo Park discovered that for downloading non-ASCII data from AppEngine, you should modify this slightly:

lambda x: x.encode('utf-8')

Other problems (or answers) for AppEngine bulk uploading or downloading? Let us know, in the comments or via email.