Open Source: data management and transformation library

Sumbled upon the following post http://flowingdata.com/2012/04/23/miso-an-open-source-toolkit-for-data-visualisation/

Relational data (or data can be stored in a table or matrix) is "old style" but still an an common use-case in todays web applications.

A new JavaScript library called "Miso Project" starts to implement components that simplifies the management and transformation of this kind of data (and will be extended with visualization use-case). This means that you are easy manage relational data on client side which can be very handy in certain use-cases. So its like a client side database with corresponding query syntax.

One of the most common patterns we've found while building JavaScript-based interactive content is the need to handle a variety of data sources such as JSON files, CSVs, remote APIs and Google Spreadsheets. Dataset simplifies this part of the process by providing a set of powerful tools to import those sources and work with the data. Once data is in a Dataset, it becomes simple to select, group, and calculate properties of, the data. Additionally, Dataset makes it easy to work with real-time and changing data, which pose one of the more complex challenges to data visualization work.

In case you have to develop a e.g. simple, standalone HTML application without a permanent server backend this library will help you without adding too much complexity in your (implementation) infrastructure.

XQuery - rare used but ramp-up gets easier

A very nice overview of what XQuery is, what are the relation to other XML based standards and why it is still very rare used is summarized here http://grtjn.blogspot.com/2011/10/xquery-novelties-revisited.html.

Once you want to dive into XQuery a bit deeper, get a feeling if the approach is sufficient for your use case you should try out BaseX. A XML database, open source. The main advantage for tryouts which are not available on alternative solution is a lightweight but very useful UI fronted for the content stored in the database.

Adding XML documents, try out queries and see how / what they match and last but not least a noticeable view of how information in a XML store looks like (from a conceptional point of view) comes aligned with easy install and ramp-up costs.

Alternatives (commercial and open source) are collected here: http://trent-intovalue.blogspot.com/2010/08/xquery-design-patterns.html

analyse and process DTDs

Working with DTD is still a common task for XML (/SGML) driven use-cases. Knowing this it is very amazing that there is no well know DTD visualization tool available supporting this task.

The good old "Near&Far Designer" is gone many years ago and the source is probably lost in the space of Open Text (the company bought Microstar Software Ltd in 1999). This tool is still in use by many organization having to deal with SGML DTDs (e.g. in the military or aircraft industry).

DTD documentation


There are a few open source scripts out there which converting a DTD into HTML pages for documentation purpose which are available free of charge:

• DTDDoc
  http://dtddoc.sourceforge.net/
• LiveDTD
  http://www.sagehill.net/livedtd/

There is one tool out there supporting graphical visualization, documentation and a few function to report key function within the given DTD:

TreeVision (http://www.ovidius.com/meta/download/treevision.html) from German company Ovidius. The tool is available free of charge and provides a very sufficient way to analyze XML / SGML DTDs.


Convert to XML Schema alternatives


If you have to process the content of a DTD for specific use-cases like analyzing the model based on custom specific rules the easiest way is to convert the DTD to RELAX NG (XML syntax) or W3C Schema language. Both are based on XML and therefore can be processed using regular XML based tools.

The best tool to do support this is trang (http://www.thaiopensource.com/relaxng/trang.html source is hosted on http://code.google.com/p/jing-trang/) initially created by James Clark. Compared to commercial alternatives the result is very predictable and for many use cases as good as possible.

DTDs will still exists for many years just because of the many legacy applications created around them. The amount of support is limited but still exists....

Open Source TMS: optentm2 / Open Source Localization framework: okapitools

The open standards in the world of localization becoming more and more mainstream. XLIFF as the main data model to carry localized information between the different automated and manual steps within the localization process, TMX as a format to exchange translation memories, ... the full list of standards can be find here http://www.opentag.com/okapi/wiki/index.php?title=Open_Standards.
This enables more and more open source or free available tools to support certain steps in the localization process without dealing with proprietary and complex formats and conventions.
Few weeks ago the new project "opentm2" released first stable release. Based on IBM TranslationManager this infrastructure claims:

OpenTM2 provides an open platform for managing translation related activities with enterprise level scalability and quality. It serves as an open yet comprehensive localization tool that provides that integration platform. Ultimately, the goal is to create a cost-efficient and high-quality localization deliverable.

Promising and  goes into the same direction as GlobalSight mentioned in a previous post: http://trent-intovalue.blogspot.com/2009/05/open-source-tms-globalsight.html.
In addition infrastructure helps to automate common tasks in localization and dealing with the mentioned standards are available as open source. The project "Okapi framework"  provides a pipeline and several useful steps in localization processes, like

• read several input formats
  http://www.opentag.com/okapi/wiki/index.php?title=Filters
• common steps in localization process
  http://www.opentag.com/okapi/wiki/index.php?title=Steps
• connection to common translation resources (like TMS)
  http://www.opentag.com/okapi/wiki/index.php?title=Connectors

 In addition useful tools to e.g. test text segmentation rules based on SRX and provide a Java based infrastructure which can be embedded in your own application makes this framework something you should look at if you work for / in localization process.

Hopefully the story continues.....

DITA - Beyond OT

DITA-OT (Open Toolkit) is the reference implementation to transform DITA source into various output formats. The reference implementation is open source and maintained by increasing community:

The OT using Apache ANT as pipeline infrastructure. Thats OK in general, but there are several shortcomings once you have to integrate or extend the OT implementation for enterprise use cases (see e.g. discussion "Pipeline refactoring", "Diving Into Performance Improvements").
Because DITA-OT is "only" a reference implementation there might be other implementation out there already using a better approach because they started development once the limitation of the OT implementation already known?
Yes there are two available I'm aware of with different key aspects:

•  XMLmind DITA Converter (see http://www.xmlmind.com/ditac/what_is_ditac.html)
  
  Key Aspects: Easier use, integration and improved output.
  Reality: Bad and monolithic design with hard coded java based pipeline and a small user community. No real advantage compared to existing DITA-OT.
  
• DITA XProc Pipelines (see https://community.emc.com/docs/DOC-8740)
  
  Key Aspects: greater flexibility, extensibility, portability, performance.
  Reality:
  Implementation based on XProc (XML Pipeline Language). The design is one step in the right direction and shows the much higher scalability (functional and non functional) of this design, e.g. in case you want to use a extended semantics for validation you can add ISO-Schematron based rules for validation using existing "ISO Schematron schema for DITA" stylesheet and build-in "p:validate-with-schematron" step in XProc and add it into the existing pipeline.
  
  This implementation isn't perfect, it mainly use XProc markup for the implementation which makes the code long and hard to read / maintain. The usage of the right language for each step which is one advantage of XML pipelining isn't consequent implemented in this implementation. Based on currently available XProc Engines (http://tests.xproc.org/results/) this implementation is not yet ready for enterprise but this will change in the near future the more real world examples are available and used in production.

The XProc based approach is promising and maybe in the future the DITA-OT development also switch over to real XML pipelining. This makes any kind of integration and extension much easier than it is today.....

Open Source CCMS

stumbled upon "Calenco XML CMS" a open source (AGPL) CCMS (for further infos about the different kind of available CMS domains, see my post: http://trent-intovalue.blogspot.com/2010/03/stumbled-upon-microsoft-sharepoint-cms.html).
From the list of features published on their website it looks promising. It seems to me the first CCMS application available as open source. I not verified the application so far but I definitely will. Basic features of course but in case such a application get a strong user and developer community the business case for the small CCMS vendors might be tricky. Will see...
The now implemented support for DITA 1.1 and Docbook 4 and 5. The amount of covered feature is also a matter for the verification.

usage of open source

Working in  software development domain focused on XML processing you have many major and stable open source infrastructure to use.

But how do you decide to use a certain project for your specific needs? you have to consider the following basic questions:

• what is the goal of the project and does it match to the goal of your usage?
  If you are not able to answer the question or if both goals doesn't match the roadmap is likely to mismatch => you cannot participate on improvements and in same cases are no more able to upgrade to newer version.
• how big is the user community?
  the more people using the project the more use-cases are implemented and considered to be stable
• Is your use case similar to a significant group of the user community?
  same rational as in point one and two
• How many active developers working on the project?
  The activity multiplied with the amount of developers divided by the size of the project gives you an impression how mature the current code is and how sufficient development will be
  
  note: big is no value for itself. a small but focused developer community sometimes is a better choice as long as their interest for the project stays stable
• How active is the development?
  Same rational as above
• Does the license fits to my use-case or company policy?
• And of course the most important one: does the project fits to my system requirements?

How to get those information? You can investigate the projects website, taking with the project community, the developers. And you might use Ohloh (http://www.ohloh.net/) which is a community for open source developers provides information around open source projects, the code and the community. Great source for open source....

XQuery Design Patterns

Nice summary of XQuery Design Patterns (see http://patterns.28msec.com/). This side also contains a link to a online XQuery engine based on Zorba (see http://try.zorba-xquery.com/).

The real power of XQuery comes with a database hosting the data for dynamic retrieval and processing. Today there are several databases out there supporting XQuery processing, even as open source.

Open Source

• eXist (http://exist.sourceforge.net/)
  XQuery sandbox here http://demo.exist-db.org/exist/sandbox/sandbox.xql
• sedna (http://modis.ispras.ru/sedna/)
  XQuery sandbox here http://wikixmldb.org/xq/
• BaseX (http://www.inf.uni-konstanz.de/dbis/basex/)
  XQuery sandbox here http://phobos101.inf.uni-konstanz.de/basex/demo
• brandnew TNTBase (http://tntbase.org/)
  XQuery sandbox here http://alpha.tntbase.mathweb.org:8080/tntbase/ores/Try.html
  uses Subversion and the Berkley DB XML. This implementation adds more XML processing infrastructure which intends to be a XML Pipelining infrastructure including a XML database.
• Oracle Berkeley DB XML (http://www.oracle.com/us/products/database/berkeley-db/index-066571.html)
  There is a commercial version available as well.

Commercial

• The big 3 relational db vendors (MS, Oracle, IBM) provide XQuery support. This approach combines relational storage of xml fragments and is therefore only suitable in certain use-cases
• MarkLogic (http://www.marklogic.com/)
  the most powerful XML DB currently available in my point of view.
  no online sandbox available, but a free developer edition can be downloaded from http://developer.marklogic.com/products
• ....

Software Globalization

Software Globalization is always a big deal and to be honest i do not know many software application with more or less shortcomings in this area. Globalization consists of Internationalization (preparing the application that they can be adapted to different locales) and Localization (translation of content for a specific locale).

you have to deal with locale specific

• messages
• dates
• numbers and currencies
  
• replacements
• conversions
  
• encodings
• sorting / collation
• test bounderies
• layouts
• .....
  

most of them seems to be easy for 95% of the use-cases but getting 100% done is at least tricky once you do not have the corresponding and suitable software infrastructure.

the best open source library in this domain I'm aware of is ICU. As always a software tool does not solve the problem itself it just gives you a solid base once you understand what needs to be done ;-)

related links:

• ICU Homepage
  
• CLDR Project
• opentag
• a bit outdated but still a good overview of software internationalization and localization

Open Source TMS: GlobalSight

few days ago i stumbled over GlobalSight a open source TMS solution formally known as Ambassador.

the marketplace for TMS products are dominated by a few commercial vendors most of them doesn't make big money with those products.

why?

the translation / localization domain requires strong domain knowledge which isn't widespread especially the combination of IT knowledge and linguistic, translation and localization knowledge is hard to achieve.

in addition translation / localization today still means to process input files from pretty heterogeneous and in many cases proprietary file formats (e.g. HTML, Word, Framemaker, Quark, Excel, XML, software resources, .....), translate them and create output files using localized content and layout. that is one of the most appalling jobs you can think of as a developer.

in general XML can reduce the requirements for TMS but the majority of users are still not using XML based content creation and therefore are not able to use XML as data format for localization.

what does this mean for success of the solution GobalSight?

to be honest i do not expect huge developer community dive into further development or even contribution in this solution because of the reason mentioned above. only if additional companies (like those part of the mentioned Steering Committee) provide development resource to this project this can be more than "just another way to herald end of life of a not successful product"

different views?

open source search engine: apache lucense & solr

Apache Lucene is one of the most interesting search api's in open source marketplace. it provides a powerful and pluggable interface to provide full test feature set to many java based application.
but Lucene is "only" an api not an application.

Apache Solr is on the other hand is an enterprise search server based on Apache Lucene. the feature set behind that solution is worth to look at, means Solr introducing features some commercial systems still missing. for example using standard enterprise search provided by MS within their Sharepoint product you wonder how less features are available there.....(maybe adding FAST into next generation of Sharepoint change that but thats another story),

since few month professional services for the mentioned open source products can be used from Lucid Imagination. providing support and integration work for open source products makes the usage for the products more useful for the enterprise.

if you have to provide advance "full text" feature double check the open source community before you invest in commercial variations....