How to call a Python function from Java?
I develop a solar energy simulation software (Polysun). Controllers can be written as plugins. The SimpleRpcPluginController allows to implement controller logic in other programming languages.
Functions from controlFunctions.py
starting with control_
should be callable from Java.
As shown in article Remote procedure calls to Python using XML-RPC and JSON-RPC with performance comparision standard XML-RPC and standard JSON-RPC show a poor performance for our simulation tool, where Python functions are called more than 100'000 times from Java. The problem is the HTTP protocol where for each request a new TCP connection is created. On Windows, JSON-RPC cannot even be used as the client runs into "java.net.BindException: Address already in use: connect" due to ephemeral TCP ports exhaustion.
I created a new TCP protocol basing on JSON-RPC 2.0, the JSON-RPC stream protocol.
JSON-RPC stream protocol
One TCP connection per simulation is used.
The simple custom RPC streaming protocol sends JSON-RPC 2.0 string lines encoded as UTF-8:
- Open TCP socket communication connection
- Send JSON-RPC 2.0 request terminated by EOL.
- Read JSON-RPC 2.0 response terminated by EOL.
- Repeat to step 2 until simulation has ended.
- (Optional) Send empty line terminated by EOL to terminate the connection.
EOL = End of line, i.e. \n
in Python/Java or Unicode \u000a
It is not anymore JSON-RPC 2.0 compatible as the communication is not over HTTP.
Measurements
Measured average time per function call for contro_flowrate() in Python and FlowratePluginController in Java 8 (Eclipse debug) on a Intel Core i7-4500U CPU 1.80GHz (dual core) for more than 100'000 function calls on localhost are about:
JSON-RPC stream | JSON-RPC | XML-RPC | Java | Matlab | |
---|---|---|---|---|---|
RPC type: | custom | standard | standard | native | library |
Communication protocol: | TCP socket | HTTP | HTTP | native | RMI |
TCP connection per | simulation | timestep | timestep | N/A | simulation |
PyPy 5.4.1, Java non-debug | 0.06ms | 0.3ms | 0.6ms | 0ms | 0.6ms |
PyPy 5.4.1 | 0.06ms | 0.5ms | 0.9ms | 0ms | 0.6ms |
Python 2.7 | 0.09ms | 0.7ms | 1.3ms | 0ms | 0.6ms |
Python 3.4 | 0.08ms | 0.9ms | 1.4ms | 0ms | 0.6ms |
Jython 2.7.0 | (20ms) |
Performance factors comparing for the same Python interpreter (e.g JSON-RPC using pypy needs five times (500%) more time than JSON-RPC stream):
JSON-RPC stream | JSON-RPC | XML-RPC | Java | Matlab | |
---|---|---|---|---|---|
PyPy 5.4.1, Java non-debug | 1 | 5 | 10 | 0 | 10 |
PyPy 5.4.1 | 1 | 8 | 15 | 0 | 10 |
Python 2.7 | 1 | 8 | 14 | 0 | 7 |
Python 3.4 | 1 | 11 | 18 | 0 | 8 |
Performance factors comparing with JSON-RPC stream using pypy (e.g XML-RPC on python 3.3 needs 23 times (2300%) more time):
JSON-RPC stream | JSON-RPC | XML-RPC | Java | Matlab | |
---|---|---|---|---|---|
PyPy 5.4.1, Java non-debug | 1.0 | 5 | 10 | 0 | 10 |
PyPy 5.4.1 | 1.0 | 8 | 15 | 0 | 10 |
Python 2.7 | 1.5 | 12 | 22 | 0 | 10 |
Python 3.4 | 1.3 | 15 | 23 | 0 | 10 |
The JSON-RPC stream protocol on pypy is five times faster (500%!) than the standard JSON-RPC HTTP-based RPC protocol on pypy. The JSON-RPC stream on pypy is 23 times (2300%!) faster than the XML-RPC on python 3.4.
This function is called more than 100'000 times from Java code.
On the Java side a TCP socket and parts of JSON-RPC for JSON-RPC function wrapping are used.
The scripts work with Python 2.7 and Python 3.4.
Call: python controlJsonRpcStreamServer.py or: pypy controlJsonRpcStreamServer.py
Supports pypy (http://pypy.org/) with JIT compliation
JSON-RPC stream server
#!/usr/bin/env python
"""
JSON-RPC Server for Polysun plugin controller functions of SimpleRpcPluginController (RPC-Type: JSON-RPC stream).
"""
from __future__ import division, unicode_literals, print_function, absolute_import, with_statement # Ensure compatibility with Python 3
import argparse
import inspect
import sys
import json
import threading
import request_jsonrpc
import importlib
from utils import indent
PY3 = sys.version_info[0] == 3
PY2 = sys.version_info[0] == 2
if PY3:
from socketserver import StreamRequestHandler
from socketserver import TCPServer
elif PY2:
from SocketServer import StreamRequestHandler
from SocketServer import TCPServer
__author__ = 'Roland Kurmann'
__email__ = 'roland dot kurmann at velasolaris dot com'
__url__ = 'http://velasolaris.com'
__license__ = 'MIT'
__version__ = '9.2'
parser = argparse.ArgumentParser(description='Polysun control function (JSON-RPC stream server using TCP).')
parser.add_argument('-p', '--port', default='2102', type=int, help='Port of the JSON-RPC stream server (TCP socket). Default 2102')
parser.add_argument('-s', '--host', default='127.0.0.1', help='Host address of the stream server. Default 127.0.0.1')
parser.add_argument('-f', '--functions', default='controlFunctions', help='Python module with control functions. Default controlFunctions')
parser.add_argument('-d', '--debug', action='store_true', help='Enable debug mode with debug output')
args = parser.parse_args()
# instead of 'import controlFunctions', load dynamically using arguments
importlib.import_module(args.functions)
class JsonRpcStreamServerHandler(StreamRequestHandler):
"""
The request handler class for our TCP server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
See https://docs.python.org/2/library/socketserver.html
"""
def handle(self):
try:
print("Connection opened from {} and listening...".format(self.client_address[0]))
ok = True
while ok:
# self.rfile is a file-like object created by the handler;
# we can now use e.g. readline() instead of raw recv() calls
self.data = self.rfile.readline().strip()
# print "{} wrote:".format(self.client_address[0])
if args.debug:
print("'" + self.data + "'")
# Likewise, self.wfile is a file-like object used to write back
# to the client
# self.wfile.write(self.data.upper())
jsonResponse = self.data.decode('UTF-8')
# An empty line means stopping the connection
if jsonResponse != None and jsonResponse.strip() != "":
request = json.loads(jsonResponse)
response = jsonrpc.handle_rpc(request)
jsonResponse = json.dumps(response) + "\n"
self.wfile.write(jsonResponse.encode('UTF-8'))
self.wfile.flush()
if args.debug:
print("handled: " + jsonResponse)
else:
ok = False
print("Connection stopped")
except KeyboardInterrupt:
print("\nKeyboard interrupt received in request, exiting.")
server_shutdown()
jsonrpc = request_jsonrpc.register(args.debug)
# echo '{ "jsonrpc": "2.0", "method": "ping", "params": [], "id": 1}' | nc 127.0.0.1 2102
@jsonrpc
def ping():
"""Returns "pong"."""
return "pong"
# echo '{ "jsonrpc": "2.0", "method": "print", "params": ["Wow"], "id": 1}' | nc 127.0.0.1 2102
def print_msg(msg):
"""Prints a message on the server."""
print(msg)
jsonrpc.methods['print'] = print_msg
def server_shutdown():
"""Shutdown TCP server.
server.shutdown() must not be called in the main thread and sys.exit(0) does not work,
and os._exit(0) does not clean up server which leads to 'error: [Errno 98] Address already in use'
"""
print("Initiate server shutdown...")
threading.Thread(target=server.shutdown).start()
jsonrpc.methods['stop'] = server_shutdown
# http://stackoverflow.com/questions/4040620/is-it-possible-to-list-all-functions-in-a-module
functions = inspect.getmembers(sys.modules[args.functions], inspect.isfunction)
for function in functions:
if function[0].startswith("control_"):
jsonrpc.methods[function[0]] = function[1]
print("\nStart JSON-RPC stream server (TCP)...")
print("jsonrpc2://" + args.host + ":" + str(args.port))
li = "\n - "
print("Functions:" + li + li.join(str(x) for x in sorted(jsonrpc.methods)))
print("Ctrl-C to stop")
server = None
try:
# Create the server
server = TCPServer((args.host, args.port), JsonRpcStreamServerHandler)
server.allow_reuse_address = True
# Activate the server; this will keep running until you interrupt the program with Ctrl-C
server.serve_forever()
except KeyboardInterrupt:
print("\nKeyboard interrupt received, exiting.")
server.shutdown()
finally:
print("Terminate server")
if server != None: # Check if server exists, since in case of server creation errors, server variable may not be set
server.server_close()
sys.exit(0)
"""
Very minimal implementation of JSON-RPC requests.
Adapted from bottle_jsonrpc.py (https://github.com/olemb/bottle_jsonrpc).
Based on bottle_jsonrpc version 0.2.1 of Ole Martin Bjorndalen.
"""
from __future__ import division, unicode_literals, print_function, absolute_import, with_statement # Ensure compatibility with Python 3
import sys, traceback
def get_public_methods(obj):
"""Return a dictionary of all public callables in a namespace.
This can be used for objects, classes and modules.
"""
methods = {}
for name in dir(obj):
method = getattr(obj, name)
if not name.startswith('_') and callable(method):
methods[name] = method
return methods
class NameSpace:
def __init__(self, debug=False, obj=None, catchall=True):
self.debug = debug
self.methods = {}
self.catchall = catchall
if obj is not None:
self.add_object(obj)
def add_object(self, obj):
"""Adds all public methods of the object."""
self.methods.update(get_public_methods(obj))
def handle_rpc(self, request):
try:
name = request['method']
func = self.methods[name]
params = request.get('params', {})
if params != None:
result = func(*params)
else:
result = func()
return {
'jsonrpc': '2.0', # Added by rkurmann for JSON-RPC 2.0 compliancy
'id': request['id'],
'result': result,
# 'error': None, # Removed by rkurmann for JSON-RPC 2.0 compliancy
}
except:
if not self.catchall:
raise
traceback.print_exc(file=sys.stderr)
response = {
'id': request['id'],
# 'result': None, # Removed by rkurmann for JSON-RPC 2.0 compliancy
'error': 'Internal server error',
}
if self.debug:
response['traceback'] = traceback.format_exc()
return response
def __call__(self, func):
"""This is called when the mapper is used as a decorator."""
self.methods[func.__name__] = func
return func
register = NameSpace
"""
Utils package
"""
from __future__ import division, unicode_literals, print_function, absolute_import, with_statement # Ensure compatibility with Python 3
# Example:
# @static_vars(counter=0)
# def foo():
# foo.counter += 1
# print "Counter is %d" % foo.counter
# http://stackoverflow.com/questions/279561/what-is-the-python-equivalent-of-static-variables-inside-a-function
def static_vars(**kwargs):
'''Annotation for static variables in functions. Similar to static in Java or persistent in Matlab.
Works only for a single thread.
'''
def decorate(func):
for k in kwargs:
setattr(func, k, kwargs[k])
return func
return decorate
def indent(lines, padding="\t"):
"Indent each line with padding"
padding = padding
return padding + ('\n' + padding).join(lines.split('\n'))
Conclusion
The JSON-RPC stream protocol is much better suited than the standard JSON-RPC using HTTP for many repeated function calls on a localhost.