XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1
Karsten Lehmann 27 April 2012 18:59:38
This article presents the first demo of my session about NoSQL databases at the German Entwicklercamp conference in March 2012. It demonstrates how the document-oriented NoSQL database MongoDB can be used in an XPages web application for the IBM Lotus Domino server.The rise of location based services
Location based services have become quite popular in the past years: Most of the smartphones carry a GPS sensor, and there are a lot of popular apps out there (e.g. Foursquare, Yelp) that use cloud services to find people, restaurants and places nearby.
For a number of reasons (mostly scalability/performance related), the NSF database of Lotus Notes/Domino is not the best choice to implement these kind of applications/services. MongoDB is much more suited for this use case and the article shows how you can integrate MongoDB into Notes/Domino to combine both worlds and get the best from both of them.
The Dojo based web application that I am going to discuss in detail uses MongoDB's geospatial indexing feature to easily find the nearest points of interest for a mobile web user.
Here is how our application looks like:
The "Administration" tab provides a user interface to add/remove places to/from the MongoDB database. A place is stored as a document in a MongoDB document collection and consists of a name, a type (e.g. shop/gas station) and the position information as [longitude, latitude] value pair.
The database can be queried on the "Search" tab by entering an address, a distance in kilometers and an optional type. The address will automatically get converted to [longitude, latitude] coordinates by using the Google Geocoding API, which is then used to find places within the specified distance.
Our MongoDB document collection is indexed in a way so that we can quickly query the database for places that surround the current user's position and sort them by ascending distance between both points.
Since geospatial indexing is a built-in feature of MongoDB, the solution is very easy to implement.
Limitations of NSF
While it is not impossible to build this application in pure Lotus Notes/Domino technology with an NSF database, the missing geo index and limited scalability of NSF would make it difficult to keep response times low with large data sets, e.g. millions or even billions of places worldwide. MongoDB instead provides a sharding feature to distribute the data evenly to several servers and reduce server load.
Additional indexer like Apache Lucene in combination with the OSGi tasklet service plugin's extension hooks might help to improve Domino's limited indexing support (we already used a Lucene index with Domino data in a big customer project) , but using MongoDB for this use case is just so much easier.
Architecture
The application consists of client-side and server-side code:
Client-side code is written in JavaScript language as a Dojo class using the class declaration system of Dojo 1.6. This Dojo class handles all user interface operations.
Server-side code is written in Java as a servlet and is exposed to the client-side via REST API's (here: GET/POST requests sent via
dojo.xhrGet / dojo.xhrPost). We developed two servlets for this application. The first servlet handles init/add/remove/query operations in MongoDB. The second servlet simply acts as a proxy to access the Google Geocoding API from client-side Javascript in order to convert between addresses and geo coordinates (longitude/latitude).
Since the REST protocol is completely stateless, the server-side code is very easy to test. With a bit of refactoring, it's even possible to run the whole application in alternative servlet containers like Jetty, Tomcat or directly within the Eclipse IDE instead of the Domino server.
Personally, I really like this REST API based approach because it's clean, transparent and keeps technologies separated.
It enables you to replace the server environment, the database system and the client-side UI toolkit at any time. You can even build multiple UIs (e.g. based on Dojo 1.6, 1.7 and Sencha's ExtJS) or mobile clients that work with the same REST API.
Diving into the code: server side
Our sample comes as an Eclipse plugin "
com.mindoo.mongo.test" to be run on Domino's OSGi framework (tested with Domino 8.5.3 GA without any fixpacks or XPages extension library): 
As you can see above, I added the Mongo API classes to the plugin's classpath.
Adding the Mongo API classes to an NSF and using them directly from SSJS code might also be possible, but I prefer the plugin solution, because the MongoDB access is supposed to be a central service on the Domino server. In addition, by using a plugin we don't have any issues with Domino's restricted security manager that prevents operations to run properly in an XPages context (e.g. a lot of drivers are using Log4J for logging, which does not run well within an XPages application).
Our two servlets are defined in the plugin.xml file of the plugin:
<
?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
<extension point="org.eclipse.equinox.http.registry.servlets">
<servlet alias="/mongotest" class="com.mindoo.mongo.test.servlet.MongoTestServlet"
load-on-startup="true">
</servlet>
</extension>
<extension point="org.eclipse.equinox.http.registry.servlets">
<servlet alias="/mongogeo" class="com.mindoo.mongo.test.servlet.GeoQueryServlet"
load-on-startup="true">
</servlet>
</extension>
</plugin> That's all you need to declare your own servlet on Domino.
After that, the servlet can either be accessed on the URL base level (
http://server/mongogeo) or in the context of a Domino database (http://server/path/to/db.nsf/mongotest). If called in the context of a database, the Domino HTTP server first makes sure that the current user is allowed to access the database and if not, it displays a login prompt. A utility class (
com.ibm.domino.osgi.core.context.ContextInfo) can be used to get session/database objects for the current HTTP request. In our sample, we use the first format for our Geo API query, because it's an open service of the server without access restriction. Anyone can use it and the URL is simple to remember.
The second URL format (/path/to/db.nsf/mongotest) is used to access MongoDB data. This enables us to check the user against the Notes database ACL to see if he is allowed to read or write MongoDB data.
Here is the method of our MongoDB access servlet that handles POST requests:
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
Session session=ContextInfo.getUserSession();
Database dbContext=ContextInfo.getUserDatabase();
if (dbContext==null) {
ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
"Servlet must be run in the context of a Domino database", null);
return;
}
try {
String pathInfo=req.getPathInfo();
if ("/deleteplaces".equals(pathInfo)) {
if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
"Current user is not allowed to write data", null);
return;
}
doDeleteGridData(session, dbContext, req, resp);
}
else if ("/addplaces".equals(pathInfo)) {
if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
"Current user is not allowed to write data", null);
return;
}
doAddGridData(session, dbContext, req, resp);
}
else if ("/queryplaces".equals(pathInfo)) {
if (!isUserAuthenticated(MongoAccessMode.Read, session, dbContext)) {
ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
"Current user is not allowed to read data", null);
return;
}
doQueryPlaces(session, dbContext, req, resp);
}
else if ("/initdb".equals(pathInfo)) {
if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
"Current user is not allowed to write data", null);
return;
}
doInitDbWithValues(session, dbContext, req, resp);
}
else if ("/cleardb".equals(pathInfo)) {
if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
"Current user is not allowed to write data", null);
return;
}
doClearDb(session, dbContext, req, resp);
}
else {
ServletUtils.sendHttpError(resp, 500, "Unsupported command "+pathInfo, null);
}
}
catch (Throwable e1) {
e1.printStackTrace();
//avoid posting the full stacktrace to the user (for security reasons)
ServletUtils.sendHttpError(resp, 500, "An error occurred processing the request. The error has been logged. Please refer to the server log files/console for details", null);
return;
}
} The code is pretty simple: We check if the user is allowed to call the specific service (read operations require the role
[MongoDBRead], write operations the role [MongoDBWrite] ) and then forward the request to helper methods. Note that I am not going into full detail in this blog article how we actually access data in MongoDB. You can find that in the provided download archive and there is a great tutorial at the MongoDB website about using the Java API.
Believe me, this stuff is really easy to use!
Click here for part 2 of this article!
- Comments [8]