Remote procedure calls to Python using XML-RPC and JSON-RPC with performance comparision

Estimated reading time of this article: 5 minutes

How to call Python function from Java?

The new article Fast remote procedure calls to Python introducing a new JSON-RPC stream protocol has been written, see there for a better solution.

I develop a solar energy simulation software (Polysun). Controllers can be written as plugins. The SimpleRpcPluginController allows to implement controller logic in other programming languages.

Functions from controlFunctions.py starting with "control" are provided as XML-RPC or JSON-RPC.

Measured average time per function call for control_flowrate() in Python are about:

JSON-RPC XML-RPC
PyPy 5.4.1 and Java code compiled: 0.3ms 0.6ms
PyPy 5.4.1 and Eclipse debug: 0.5ms 0.9ms
Python 2.7 and Eclipse debug: 0.7ms 1.3ms
Python 3.4 and Eclipse debug: 0.9ms 1.4ms
Jython 2.7.0 in Eclipse debug: (20ms) (20ms)

This function is called more than 100'000 times from Java code.

On the Java side:

The scripts work with Python 2.7 and Python 3.4.

Call: python controlXmlRpcServer.py or python controlJsonRpcServer.py or: pypy controlXmlRpcServer.py or pypy controlJsonRpcServer.py

Supports pypy (http://pypy.org/) with JIT compliation

SimpleXMLRPCServer is part of Python 2.7 and 3.4. Thus, this server works out of the box.

PyPy is an Python interpreter using Just in Time (JIT) compilation. PyPy is the best way to start the RPC Server.

Jython (http://www.jython.org) is package running Python in a JVM. It works, see below, but it is quite slow. Functions calls are about 20ms. The Jython.jar is about 27MB.

JSON-RPC Server

#!/usr/bin/env python
"""
JSON-RPC Server for Polysun plugin controller functions of SimpleRpcPluginController (RPC-Type: JSON-RPC).
"""

from __future__ import division, unicode_literals, print_function, absolute_import, with_statement  # Ensure compatibility with Python 3
import argparse
import inspect
import sys
import importlib
from utils import indent

PY3 = sys.version_info[0] == 3
PY2 = sys.version_info[0] == 2

from bottle import route, run, request, response, template, server_names
import bottle_jsonrpc

__author__ = 'Roland Kurmann'
__email__ = 'roland dot kurmann at velasolaris dot com'
__url__ = 'http://velasolaris.com'
__license__ = 'MIT'
__version__ = '9.2'

parser = argparse.ArgumentParser(description='Polysun control function (JSON-RPC HTTP Server).')
parser.add_argument('-p', '--port', default='2102', type=int, help='Port of the REST-Server. Default 2102')
parser.add_argument('-s', '--host', default='127.0.0.1', help='Host address of the REST-Server. Default 127.0.0.1')
parser.add_argument('--path', default='/control', help='Path of the JSON-RPC-Server (namespace). Default /control')
parser.add_argument('-f', '--functions', default='controlFunctions', help='Python module with control functions. Default controlFunctions')
parser.add_argument('--noreload', action='store_true', help='Do not automatically reload server after script changes')
parser.add_argument('-d', '--debug', action='store_true', help='Enable debug mode with debug output')

args = parser.parse_args()

# instead of 'import controlFunctions', load dynamically using arguments
importlib.import_module(args.functions)

jsonrpc = bottle_jsonrpc.register(args.path)

# curl -X POST -i -H "Content-type: application/json" -X POST http://localhost:2102/control -d '{ "jsonrpc": "2.0", "method": "ping", "params": [], "id": 1}'
@jsonrpc
def ping():
    """Returns "pong"."""
    return "pong"

# curl -X POST -i -H "Content-type: application/json" -X POST http://localhost:2102/control -d '{ "jsonrpc": "2.0", "method": "print", "params": ["Wow"], "id": 1}'
def print_msg(msg):
    """Prints a message on the server."""
    print(msg)

jsonrpc.methods['print'] = print_msg

jsonrpc.methods['stop'] = quit

# http://stackoverflow.com/questions/4040620/is-it-possible-to-list-all-functions-in-a-module
functions = inspect.getmembers(sys.modules[args.functions], inspect.isfunction)
for function in functions:
    if function[0].startswith("control_"):
        jsonrpc.methods[function[0]] = function[1]

print("\nStart JSON-RPC server...")
print("http://" + args.host + ":" + str(args.port) + args.path)
li = "\n    - "
print("Functions:" + li + li.join(str(x) for x in sorted(jsonrpc.methods)))
print("Ctrl-C to stop")

try:
    # Run the server's main loop
    run(server='wsgiref', host=args.host, port=args.port, quiet=not args.debug, reloader=not args.noreload, debug=args.debug)
except KeyboardInterrupt:
    print("\nKeyboard interrupt received, exiting.")
    sys.exit(0)

XML-RPC Server

#!/usr/bin/env python
"""
XML-RPC Server for Polysun plugin controller functions of SimpleRpcPluginController (RPC-Type: XML-RPC).
"""

from __future__ import division, unicode_literals, print_function, absolute_import, with_statement  # Ensure compatibility with Python 3
import argparse
import inspect
import sys
import importlib
from utils import indent

PY3 = sys.version_info[0] == 3
PY2 = sys.version_info[0] == 2

if PY3:
    from xmlrpc.server import SimpleXMLRPCServer
    from xmlrpc.server import SimpleXMLRPCRequestHandler
elif PY2:
    from SimpleXMLRPCServer import SimpleXMLRPCServer
    from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler

__author__ = 'Roland Kurmann'
__email__ = 'roland dot kurmann at velasolaris dot com'
__url__ = 'http://velasolaris.com'
__license__ = 'MIT'
__version__ = '9.2'

parser = argparse.ArgumentParser(description='Python Polysun control function (XML-RPC HTTP Server).')
parser.add_argument('-p', '--port', default='2102', type=int, help='Port of the XML-RPC-Server. Default 2102')
parser.add_argument('-s', '--host', default='127.0.0.1', help='Host address of the REST-Server. Default 127.0.0.1')
parser.add_argument('--path', default='/control', help='Path of the XML-RPC-Server (namespace). Default /control')
parser.add_argument('-f', '--functions', default='controlFunctions', help='Python module with control functions. Default controlFunctions')
parser.add_argument('-d', '--debug', action='store_true', help='Enable debug mode with debug output')

args = parser.parse_args()

# instead of 'import controlFunctions', load dynamically using arguments
importlib.import_module(args.functions)

# Restrict to a particular path.
class RequestHandler(SimpleXMLRPCRequestHandler):
    rpc_paths = (args.path,)

# Create server
server = SimpleXMLRPCServer((args.host, args.port),
                            requestHandler=RequestHandler,
                            logRequests=args.debug,
                            allow_none=None)
server.register_introspection_functions()
# server.register_multicall_functions()

def ping():
    """Returns "pong"."""
    return "pong"

server.register_function(ping)

def print_msg(msg):
    """Prints a message on the server."""
    print(msg)
    return []

server.register_function(print_msg, "print")

# http://stackoverflow.com/questions/4040620/is-it-possible-to-list-all-functions-in-a-module
functions = inspect.getmembers(sys.modules[args.functions], inspect.isfunction)
for function in functions:
    if function[0].startswith("control_"):
        server.register_function(function[1])

server.register_function(quit, 'stop')
# server.register_instance(MetaService())

print("Start XML-RPC server...")
print("http://" + args.host + ":" + str(args.port) + args.path)
li = "\n    - "
print("Functions:" + li + li.join(str(x) for x in sorted(server.funcs)))
print("Ctrl-C to stop")

try:
    # Run the server's main loop
    server.serve_forever()
except KeyboardInterrupt:
    print("\nKeyboard interrupt received, exiting.")
    server.server_close()
    sys.exit(0)

Jython

Properties props = new Properties();
props.put("python.console.encoding", "UTF-8"); // Used to prevent: console: Failed to install '': java.nio.charset.UnsupportedCharsetException: cp0.
props.put("python.security.respectJavaAccessibility", "false"); //don't respect java accessibility, so that we can access protected members on subclasses
props.put("python.import.site","false");

Properties preprops = System.getProperties();

PythonInterpreter.initialize(preprops, props, new String[0]);
PythonInterpreter pi = new PythonInterpreter();
pi.execfile("~/code/src/main/python/controlFunctions.py");
pi.exec("print(ping())");
pi.exec("result = ping()");
PyString result = (PyString)pi.get("result");
System.out.println(result);
long start = System.nanoTime() / 1000000;
pi.set("simulationTime", 1);
pi.set("status", true);
pi.set("sensors", new Double[] {1d, 2d, 3d});
pi.exec("result = controlTest(simulationTime, status, sensors)");
PyList resultList = (PyList)pi.get("result");
System.out.println(resultList);
long stop = System.nanoTime() / 1000000;
System.out.println("controlTest: " + (stop - start) + "ms");

Conclusion

JSON-RPC is faster than XML-RPC. Python 2 is faster then Python 3. PyPy 5.4.1 (Python 2.7) is faster than CPython 2.7. Jython is slower.

The new article Fast remote procedure calls to Python introducing a new JSON-RPC stream protocol has been written, see there for a better solution.

Files: