Gospel Library for JavaME (old)
Note: This article refers to an old version of Gospel Library for JavaME. Concepts may apply to the current version, but implementation details will not
The Gospel Library for JavaME is intended to allow individuals to have access to the official text of the Gospel Library on devices which support Java ME. The JavaME platform covers quite a broad range of devices. The most common of these devices is mobile phones. In past years the number of Java enabled mobile phones has been in the billions and vastly outnumbers the number of desktop computers in the world. It is true to say that in more recent years there has been something of a trend away from JavaME phones given the success of iPhone and Android based phones. JavaME is however still available for a large number of phones. Where the smart phone market is more heavily dominated by the iPhone, Android, Windows Mobile, Blackberry and WebOS amongst other devices, JavaME remains used for more modest "Feature Phones". These devices tend to be more affordable but also often more limited.
One of the aspects of JavaME which makes it attractive to support is that many JavaME enabled phones exist in the developing world. It is unlikely that individuals in these places could afford the high prices of the smart phones. There are also many members of more modest means in the developed world who own a phone which supports JavaME.
In comparison to the other development projects I would suggest that developing for JavaME could be the most challenging of the projects. As JavaME operates on more modest phones, the implementation will need to be very memory and resource constrained. In addition the variation in the different designs of phones which support JavaME is also considerable. The design of the application including the UI will need to be very flexible.
Overview of JavaME
It is worth briefly covering the fundamentals of JavaME. As mentioned before JavaME is not a single platform but covers a range of mobile and embedded devices. It comprises of the core platform and a collection of optional APIs which may or may not be supported. JavaME runs on many mobile phones but can also be found embedded in printers, within set top boxes and more recently in Blu-Ray players. The fact that Blu-Ray players are also within the same JavaME family as JavaME on the mobile phone is relevant to this project. The primary goal is to provide access to the gospel library on mobile phones but it is also potentially possible to reuse JavaME components created for that to create a scripture reader which can be embedded on Blu-Ray discs. This could for example make the scriptures available on Church media such as lesson presentations. The Church has not moved to Blu-Ray at this time which is understandable given the current cost and availability of supported players. However it is likely that some day it will be supported and its is hoped by that time a scripture reader for that platform may be available.
JavaME was initially invented around 1997 which is over ten years ago. The limitations which had to be imposed to get Java to run on mobile phones from that era was quite considerable. The technology has evolved since that time to make it more capable.
At the core of JavaME are the profiles CLDC and CDC.
CLDC stands for Connected Limited Device Configuration. This is the more constrained member of the JavaME family which has been around for Mobile Phones for a number of years. This is the current target platform for creation of a Gospel Library reader for mobile phones. The platform is able to support devices with as little as 160K of memory but typically modern phones will provide much more memory than that. It should be noted however that the amount of memory available to JavaME on a mobile phone is likely to be quite a lot smaller than the amount of memory the phone has overall.
CLDC has two core versions CLDC 1.0 and CLDC 1.1. The only real difference between these two is that CLDC 1.1 supports floating point primitives. Many phones from the early days of JavaME had very simple processors which often did not have a floating point unit. The CLDC 1.0 specification allowed such phones not to have to support floating point number processing. Many applications including the Gospel Library application for JavaME do not need the use of floating point arithmetic.
Only a very simple and limited number of classes and API's exist in CLDC and this profile does not support reflection or the Java Native Interface. A JavaSE implementation does verification of bytecode for problems dynamically as needed. Due to the limitations of CLDC compliant devices the bytecode pre-verification step happens ahead of time. This is an extra step needed when compiling and deploying to CLDC devices.
CDC stands for Connected Device Configuration and is the core of JavaME intended for more capable devices such as set top boxes and more powerful embedded devices and PDAs. This platform looks a lot more like JavaSE than CLDC though it still has a somewhat more cut down collection of libraries and APIs compared with JavaSE. This platform is designed for devices with 2 megabytes or more of memory. This platform is used by Blu-Ray players which are typically much more powerful than mobile phones. This platform may be relevant to the project if Blu-Ray support is desired. In addition to this, due to the advances in mobile phone technology in recent years some phone manufacturers are looking at producing phones based on CDC instead of CLDC. This is not mainstream as yet and so CLDC is still the more logical target.
MIDP stands for Mobile Information Device Profile. It is one of a number of profiles which build on top of CLDC/CDC to provide APIs specific for a type of device. Where CLDC could both be used in a phone and a printer, MIDP is aimed specifically at mobile phone like devices. As such MIDP includes the classes and APIs related to a mobile phones UI, access to Audio and other media capabilities as well as storage and communication most commonly supported by phones. The initial version of MIDP was 1.0 however for a number of years most phones have been able to support MIDP 2.0 which is the current target version for the Gospel Library application. MIDP 3.0 was finalised as a standard more recently but it remains to be seen when the first phones supporting this version of MIDP will be available. Applications running on MIDP devices are called Midlets The equivalent to these on other JavaME platforms such as BD-J and set top boxes is Xlets.
When dealing with JavaME development it is likely that the capability of a mobile phone in terms of Java will be defined in terms of a list of JSRs. JSRs are standards which are agreed on by the Java Community Process (JCP) about Java APIs and technology. This applies to all Java platforms, not just JavaME but a large portion of the JSR standards have come about due to the need for the different mobile handset manufactures to agree on API's. Unlike JavaSE which consists of one complete standard platform, JavaME is more like a collection of building blocks which can be put together to define a platform. Different JSRs provide APIs for such things as access to Bluetooth, mobile messaging, 3D graphics, global positioning, sensor data and file system access. The variation in which technologies are supported by given phones can be quite considerable and JavaME has been criticised for this fragmentation. In recent years efforts have been made to try and deal with the fragmentation and specify a core set of APIs which all phones should aim to support. The use of quoting features in terms of JSRs can be unhelpful as a JSR number is not so easy to remember and does not say anything about what the standard is for.
Luckily for the Java ME Gospel Library application, the number of JSR based APIs which an implementation would really need is quite small. One of the most applicable JSRs is JSR 75 which defines an API for access to the phones file system. One of the biggest theoretical challenges at the moment to supporting a scripture reader on JavaME is having sufficient persistent storage to store the scriptures on the devices. Having direct access to the file system makes life easier as phones in recent years often have gigabytes of storage available such that having enough is not an issue.
Compared to the other mobile device projects, the JavaME project is still in relatively early stages. It is not yet reached a mile stone where it can be demoed but work is proactively being done to get it to that point.
So far the work done has been to implement a SAX based parser to load the content of the scriptures into a Java Object Hierarchy. It took some time to come up with a hierarchy structure to hold the scripture data and support parsing in the data from the scripture xml format. At the time of writing support for virtually all the xml tags in the scripture is supported by the SAX handler and the object hierarchy is able to hold the data and style information as well as preserving the tree structure. Using any single xml file in the hierarchy the xml parser is able to automatically parse that file and all connected files bellow it in the hierarchy. One challenge which took some time to sort out was making sure the while space was normalised after parsing. This is important as the resulting structure uses character index positions extensively to define regions for links and styles etc.
The next phase which is currently being actively worked upon is to serialise the document hierarchy into a compact binary format. This is the format which it is intended that the phone will parse. Some JavaME phones have the capability to parse XML but this feature is not common place. It has therefore been the decision to standardise a compact binary format for transferring the scriptures to the phone. An implementation of writing the binary format has been put in place but it will only be clear whether this is working correctly once the data can be deserialized back from the binary format.
At the time of writing this entry the current work being done is implementing the phone CLDC/MIDP based binary parser to parse the data from the binary format. This is ongoing and an important step in demonstrating the complete working process from original XML data right through to scriptures being displayed on a mobile device screen.
There are several possibilities for loading the data into the phone including sending the data over bluetooth or a HTTP or network connection as well as direct file access where supported. Catering for several different possibilities could be a good idea to give multiple back up way of transferring the data if one should not be available on a given phone. As long as the data source can be abstracted to an input stream it should not matter too much how the data is being received.
Once the data is able to be successfully transferred to the device then the issues of displaying and interacting with the data must be handled as well as persistent storage on the device.
As mentioned, an implementation on the desktop side has been implemented for the process of serialization. Initial test show that with GZip compression the binary writer is able to get the entire scriptures with study helps down to a size of about 2.5mb. This will hopefully be small enough to support a reasonable range of phones. Experimentation has been done with using Zip based compression with different compression modes but 2.5mb still seems to be the smallest size which as been able to be achieved. It will only be clear if the serialization process is working once the data has been successfully deserialized.
Once deserialization has been successfully completed the details of the compact binary format being used by the process will be posted on the wiki. As this stage it makes sense to hold off posting the details of the format until after any tweaks and changes have been made to the format as issues are noticed during deserialization. Hopefully this phase will not take too long and the details will be posted soon.
One particular challenge has been implement JavaME based support for handling GZip compression as this ability is not built in to JavaME. Initially the source code from the OpenJDK project was examined to see if it could be adapted. This was not appropriate as the OpenJDK compression handling makes native calls into the zlib library to make the compression work. As no native access is available for JavaME this was not an option. Instead at this time the version of compression handling from the GNU Classpath has been integrated. It will become clear how well this works as the deserialization implementation advances. The GNU Class path implementation is a pure Java implementation of compression handling.
At this time support for using Zip based compression does not seem to have any advantage as no improvements in compressed size have been noted an there is no need for the ability to store multiple entries in the compressed data. Without tangible benefit it is currently going to be left out of the mobile implementation in order to try and keep the final mobile jar file as small as possible.
Source Code Repository Structure
The source code for the repository has been split into four different projects. At this time Eclipse projects for the four groups have been created but projects for Netbeans or another IDE could be created if requested.
The idea has been to have a project called common which is mostly a collection of interfaces representing the document hierarchy. This provides a common base to the standard and compact projects. The common project has been designed to only use Java source code that is supported by both JavaSE and JavaME. For example, JavaME cannot support generics or enumerations. Also some classes have been defined where equivalents already exist on the JavaSE platform but not on the JavaME platform in order to be able to keep common APIs for both. Only a few such objects are defined as the project tries to avoid reinventing the wheel.
The standard project represents an implementation of the hierarchy defined in common for the JavaSE platform. The implementing classes therefore have the freedom internally to use all of the features of JavaSE to implement the functionality. The implementation of these classes favours functionality and flexibility over raw aggressive efficiency as the desktop platform is not so heavily resource constrained.
The compact project is designed to be compiled for JavaME CLDC. The implementations for these classes are still in the early stages but the implementations are to be optimised for aggressive memory efficiency and simplicity over functionality and flexibility. Due to the desire to keep the compact implementation as simple as possible the common API interfaces are designed to allow objects in the document hierarchy for JavaME to be immutable. The binary format to be sent to the mobile device is being designed such that the structure of the hierarchy of the objects such as the number of child items should be known at the time of object creation. This avoids the need to add children to a document for example and means simple arrays can be used rather than more complex data structures which should simplify implementation and improve efficiency. More recently in the same repository some of the core classes were moved across to a CDC implementation for use with that JavaME platform. The source files in the CDC namespace are set to be ignored by the CLDC project. The CDC classes made use of item types which were classes designed to work like enumerations. These were originally part of the CLDC implementation but the current CLDC versions were simplified from these to be able to represent item types without needing to use any extra objects. This is desirable to reduce the application size and memory footprint on the CLDC platform. The previous implementation moved into the CDC project is a somewhat more elegant solution but requires more memory. This should not be so much of an issue on CDC based platforms such as Blu-Ray players. In the future actual Xlet applications can be put within these CDC packages.
The immutability constraint does not apply on the standard implementation for JavaSE as using generic collections is more flexible on that platform. Also, due the the way in which the document hierarchy is built by using a SAX parser on the XML source where the numbers of children is not known ahead of time, it is more natural to make the document hierarchy items mutable. The standard platforms includes a number of interfaces which extend those from the common project to add the ability to change objects or add items to groups for the standard platform.
The fourth project is the client project. It is intended that this project become a client application which can be used to transfer the gospel library to a mobile device. At the time of writing, only some utility classes exist within it to run and debug / test the xml parsing process and test any classes developed.
Potentially a fifth project is likely to be added to the repository. This will be a "Server" project. This project will hold Java EE based classes such as any Servlets which can produce compact binary data for a mobile device to retrieve. it is likely that such a Servlet will be implemented though at this stage it is not an immediate priority.
At the time of writing the source code only uses core JavaSE or JavaME libraries and has no dependencies on any external libraries. The code is being developed for JavaSE 6 on the desktop.
To develop for mobile the Java Wireless ToolKit Version 3 is being used and should be downloaded and installed for use with mobile development.
The testing of loading the entire scriptures into a document hierarchy can consume quite a lot of memory. Typically on a 64bit development platform the memory taken up by the compete scriptures and study helps in memory is about 100 megabytes or ram. This could cause an out of memory error if the maximum heap size is not increased using the -Xmx Java VM argument. On 32 bit platforms the memory usage will be more like 50-70 megabytes. The performance of xml parsing will slow down considerably if the JavaVM on which it is running is reaching the limits of the available stack memory.
As it is not intended in practice for the entire scriptures to be held in ram at one time, the document hierarchy has been designed to optionally allow for the use of lazy loading proxy objects. These object merely hold a handle to get hold of an item in the document hierarchy rather than the item itself. This allows the items in the scriptures to be loaded only as they are used. If memory usage becomes high, the parent items in the document hierarchy have a method which can be called on them to clear all proxy caches and dispose of the cached proxy items to free up memory. Using proxies can keep memory usage to more manageable levels. The main document hierarchy xml parser has a boolean property which can be set to tell it whether or not to use lazy loading proxies when loading the document hierarchy.
Eclipse is currently being used for development. The Mobile Tools for Java expansion for Eclipse has been installed for the benefit of mobile development. This expansion is able to run code within the Sun/Oracle Wireless Toolkit application as well having support for manufacturer specific toolkits such as Motorola, Nokia and Samsung. Subversion plugins are also available which can be useful for synchronising changes to the repository.
The core objectives of the Gospel Library for JavaME project are:
- Provide the ability to read and navigate the Standard Works of the Church of Jesus Christ of Latter Day Saints.
- Allow for searching within the scriptures
- Provide functionality for users to add annotations such as notes, journal entries and highlighting.
The primary objective of the project is to enable access to the scriptures. In addition until a working prototype is demonstrated and an initial application has been launched the project still needs to prove that a JavaME based Gospel Library is technically viable. In spite of this there is potential for items which can be accomplished beyond the JavaME scriptures. The project has been designed so far to consider to try and make it as easy as possible to adapt to display other materials besides just the standard works. It is also a possibility as has been previously mentioned to adapt the application for use with the Blu-Ray BD-J platform. This could allow for bundling of the scriptures as well as other Church materials on Blu-Ray media in the future. For some time also there has been some interest in having a cross platform desktop application for providing access to the scriptures and it is possible that some of the work from this project could help with that.
Gospel Library Compact Binary Format
Due to a need to try and get the Gospel Library to be stored in as small a spaces as possible the original Gospel Library is converted into a compact binary format. This format aims to represent all the important data of the Gospel Library as concisely as possible.
The details of the binary format can be found under:
If this format for Gospel Library data proves useful to be used in any project is can be reused for such purposes.
The JavaME Development Team
A major reason for the progress of the JavaME project being behind that of the other projects is that the JavaME team consist currently of one individual. To make matters worse this individual is being kept busy with University research. However the desire to see this project succeed is just as strong as ever. There is a desire from the Church to see if more individuals can help with this project to speed up progress.
JavaME Team: Carl Jokl (Carl@Jokl.co.uk)