XQuery - rare used but ramp-up gets easier

A very nice overview of what XQuery is, what are the relation to other XML based standards and why it is still very rare used is summarized here http://grtjn.blogspot.com/2011/10/xquery-novelties-revisited.html.

Once you want to dive into XQuery a bit deeper, get a feeling if the approach is sufficient for your use case you should try out BaseX. A XML database, open source. The main advantage for tryouts which are not available on alternative solution is a lightweight but very useful UI fronted for the content stored in the database.

Adding XML documents, try out queries and see how / what they match and last but not least a noticeable view of how information in a XML store looks like (from a conceptional point of view) comes aligned with easy install and ramp-up costs.

Alternatives (commercial and open source) are collected here: http://trent-intovalue.blogspot.com/2010/08/xquery-design-patterns.html

HTML5 and XML

HTML5 will be the main syntax for the Internet in the next few years and will replace the today most frequently used HTML 4.01. Main driver for this shift was Google and is now adapted by all major browser / OS vendors and organizations.

The main advantage of HTML5 are the new amount of build in features which reflect most of the todays common requirements for web based applications.


Why XHTML 1.0/1.1 failed so far? It mainly was much to strict for the web community - the web and also the world is a non perfect place and therefore HTML5 is much more suitable to fit into this world than the XML based approach of XHTML can provide.

Does this mean XML for the web has loose and does not make sense at all. No there is still space for the XML based standards beside HTML5:

• XHTML5
  XML serialization of HTML 5 with stricter parsing rules
  mime-type: application/xhtml+xml
• Polyglot XHTML (see http://dev.w3.org/html5/html-author/#polyglot-documents)

Major advantage of using XML to express the content on the web is a much more easier way to integrate the resulting content into XML processing chains using regular XML transformation tool chains.

• Easier Reuse content for different channels using XSLT / XQuery
• Retrieve content as XHTML and extract only dedicated parts (views) required for different use-case
• Store and request the content using XQuery based infrastructure

The drawback is that today only the newest browser support the mime-type "application/xhtml" therefore for a while Polyglot XHTML might be a good opportunity to deliver the mass and keep processing use-cases doable.

A good summary of Polyglot XHTML and related XML based alternatives for HTML5 can be found here: http://www.xmlplease.com/xhtml/xhtml5polyglot/

analyse and process DTDs

Working with DTD is still a common task for XML (/SGML) driven use-cases. Knowing this it is very amazing that there is no well know DTD visualization tool available supporting this task.

The good old "Near&Far Designer" is gone many years ago and the source is probably lost in the space of Open Text (the company bought Microstar Software Ltd in 1999). This tool is still in use by many organization having to deal with SGML DTDs (e.g. in the military or aircraft industry).

DTD documentation


There are a few open source scripts out there which converting a DTD into HTML pages for documentation purpose which are available free of charge:

• DTDDoc
  http://dtddoc.sourceforge.net/
• LiveDTD
  http://www.sagehill.net/livedtd/

There is one tool out there supporting graphical visualization, documentation and a few function to report key function within the given DTD:

TreeVision (http://www.ovidius.com/meta/download/treevision.html) from German company Ovidius. The tool is available free of charge and provides a very sufficient way to analyze XML / SGML DTDs.


Convert to XML Schema alternatives


If you have to process the content of a DTD for specific use-cases like analyzing the model based on custom specific rules the easiest way is to convert the DTD to RELAX NG (XML syntax) or W3C Schema language. Both are based on XML and therefore can be processed using regular XML based tools.

The best tool to do support this is trang (http://www.thaiopensource.com/relaxng/trang.html source is hosted on http://code.google.com/p/jing-trang/) initially created by James Clark. Compared to commercial alternatives the result is very predictable and for many use cases as good as possible.

DTDs will still exists for many years just because of the many legacy applications created around them. The amount of support is limited but still exists....

DITA - Beyond OT

DITA-OT (Open Toolkit) is the reference implementation to transform DITA source into various output formats. The reference implementation is open source and maintained by increasing community:

The OT using Apache ANT as pipeline infrastructure. Thats OK in general, but there are several shortcomings once you have to integrate or extend the OT implementation for enterprise use cases (see e.g. discussion "Pipeline refactoring", "Diving Into Performance Improvements").
Because DITA-OT is "only" a reference implementation there might be other implementation out there already using a better approach because they started development once the limitation of the OT implementation already known?
Yes there are two available I'm aware of with different key aspects:

•  XMLmind DITA Converter (see http://www.xmlmind.com/ditac/what_is_ditac.html)
  
  Key Aspects: Easier use, integration and improved output.
  Reality: Bad and monolithic design with hard coded java based pipeline and a small user community. No real advantage compared to existing DITA-OT.
  
• DITA XProc Pipelines (see https://community.emc.com/docs/DOC-8740)
  
  Key Aspects: greater flexibility, extensibility, portability, performance.
  Reality:
  Implementation based on XProc (XML Pipeline Language). The design is one step in the right direction and shows the much higher scalability (functional and non functional) of this design, e.g. in case you want to use a extended semantics for validation you can add ISO-Schematron based rules for validation using existing "ISO Schematron schema for DITA" stylesheet and build-in "p:validate-with-schematron" step in XProc and add it into the existing pipeline.
  
  This implementation isn't perfect, it mainly use XProc markup for the implementation which makes the code long and hard to read / maintain. The usage of the right language for each step which is one advantage of XML pipelining isn't consequent implemented in this implementation. Based on currently available XProc Engines (http://tests.xproc.org/results/) this implementation is not yet ready for enterprise but this will change in the near future the more real world examples are available and used in production.

The XProc based approach is promising and maybe in the future the DITA-OT development also switch over to real XML pipelining. This makes any kind of integration and extension much easier than it is today.....

Open Source CCMS

stumbled upon "Calenco XML CMS" a open source (AGPL) CCMS (for further infos about the different kind of available CMS domains, see my post: http://trent-intovalue.blogspot.com/2010/03/stumbled-upon-microsoft-sharepoint-cms.html).
From the list of features published on their website it looks promising. It seems to me the first CCMS application available as open source. I not verified the application so far but I definitely will. Basic features of course but in case such a application get a strong user and developer community the business case for the small CCMS vendors might be tricky. Will see...
The now implemented support for DITA 1.1 and Docbook 4 and 5. The amount of covered feature is also a matter for the verification.

XSLT Version 2.1

XSLT Version 2.1 Draft published. As mentioned in the published draft the main goal is making streaming of transformation easier for implementors.

Streaming makes life easier whenever you have to process huge input streams of information to reduce memory footprint and deliver already processed chunks to the next consumer before the complete stream is successful processed.

On the other hand XSLT 2.0 which is a recommendation since 3 years is not widely adopted from implementors of XSLT engines -- Saxon, AltovaXML and few commercial ones-- so far.

Why? The main reason is complexity compared to the corresponding benefit. The implementation of a XSLT engine passes the 2.0 conformance tests is expensive and complex compared to the benefit for most users / use-cases. As in any commercial product each new feature / requirement should be qualified against the value and cost / effort it brings into the standard.

The interesting question would be -- how to measure benefits of a feature? How to interview the "users" of a standard? There is no trivial answer and that might be the main reason why standards tend to get over complex.

CMS of what kind?

stumbled upon "Microsoft SharePoint: The CMS Killer" blog post. A personal view how MS SharePoint fits for ECMS use cases.
The results might be wrong or write depending on what your understanding of CMS is.

Problem

There is no common understanding of the the term CMS and even not for the derived term ECMS.

Cause

CMS means Content Management System. Based on this definition it is a application (system) to manage content. Thats trivial but what does content really means? Content is all and everything. Most of the content can be managed within IT systems as well.

Illustration

General categories of content maintained by IT applications:

• Structured content
  Content maintained and structured as a collection of data (order or offer data). Content in this context means a collection of records with given structure
  Those type of data are typical maintained with applications called ERP or other kind of systems of this type (e.g. ALM systems).
• Unstructured content
  Content maintained within documents. The content within the document is not addressable outside the application the document was created with (the semantic makes only sense in one specific usage scenario).
  Those type of data are typical maintained with applications called DMS or Web-CMS (maintaining HTML) systems.

The propagation of xml introduce a third categorie of content:

• Semi-structured content
  Content used to create documents (information products) of some kind. The content within the document is addressable outside the application the document was created with.
  OR
  Content maintained as system independent instance of data (e.g. order or offer data).
  In both cases XML is the common format today.

Each CMS has a history in one of the mentioned areas and has its specific strength and weakness for usage in different domains.

The corresponding consultants knowing the mentioned issue and trying to create domain specific names for specific usage of systems

• DBMS
  Main goal is to maintain relational data and used by dedicated applications on top.
• DMS
  Main goal is to maintain documents
• Web-CMS
  Main goal is to maintain intranet / extranet / internet sites
• ECMS
  Enterprise-CMS
  Main goal is to provide enterprise ready workflow and records management on top of DMS feature set
• CCMS
  Component-CMS
  Main goal is to maintain content stored in XML for single-source publishing
• ?
  anything i missed, of course there are plenty of buzzwords / domains out there describing mixed-scenario usage.

Because non of the mentioned definitions are "formal approved" the specific vendors use the term with most customer attention in their target domain and each vendor tries to fulfill use-cases from other domains as well. Because each CMS has a particular implementation history the usage in most cases is limited even if the vendor claims to cover most of them.

Examples

• a C-CMS vendor might support xml usage and publishing very well but does not scale if enterprise workflow or records management is required.
• a ECMS vendor has enterprise BPM support build in but lacks of sophisticated xml semantic and functions.
• a Web-CMS makes creating your Internet presence easy but lacks the usage of the same content for printed documents
• .....

Summary

This reflects the current state of content management. I expect in the next few years that new and maybe existing systems will move into the semi-structured content area and some of them might succeed. They might reach the final goal that content can be create, maintain, re-purpose and publish based on different user communities from one single source.

Until that stage is reached....coming back to initial "Microsoft SharePoint: The CMS Killer" statement. Ask the author what kind of content use-case he has in mind and you can validate the statement.

DITA DTD: tools to support specialization

if you are new to DITA and want to create a custom DTD configuration with or without custom specialization this tool is a good starting point.

you may ask why, at least if you do not want to introduce custom semantic to your data model you should at least define which domain belongs to you. a summary why to go this way, read http://drmacros-xml-rants.blogspot.com/2007/04/dita-standard-practice-always-make.html.

Online version of "DITA DTD Generator" is available as online version. source code can be found here: http://code.google.com/p/dita-generator/.

If you have to support custom semantic (which is common in enterprise usage of DITA) you might use "DITA Visual Specialization Manager" or read "DITA Specialization Tutorial".

As always using the right tool is only the baseline. The more advanced task is to identify what you need based on your business case.

Open Source TMS: GlobalSight

few days ago i stumbled over GlobalSight a open source TMS solution formally known as Ambassador.

the marketplace for TMS products are dominated by a few commercial vendors most of them doesn't make big money with those products.

why?

the translation / localization domain requires strong domain knowledge which isn't widespread especially the combination of IT knowledge and linguistic, translation and localization knowledge is hard to achieve.

in addition translation / localization today still means to process input files from pretty heterogeneous and in many cases proprietary file formats (e.g. HTML, Word, Framemaker, Quark, Excel, XML, software resources, .....), translate them and create output files using localized content and layout. that is one of the most appalling jobs you can think of as a developer.

in general XML can reduce the requirements for TMS but the majority of users are still not using XML based content creation and therefore are not able to use XML as data format for localization.

what does this mean for success of the solution GobalSight?

to be honest i do not expect huge developer community dive into further development or even contribution in this solution because of the reason mentioned above. only if additional companies (like those part of the mentioned Steering Committee) provide development resource to this project this can be more than "just another way to herald end of life of a not successful product"

different views?

EXtensions...xml processing

you always faced with "missing features" if you working with existing standards.

two reasons for that:

1. the feature is specific to your domain or solution
2. the feature is common but for some reasons (and there might be good ones) didn't get it in the existing standard

for the second case common EXtensions might be the area you looking for. in the xml domain the following projects trying to define "standard extensions". this means that extending an existing standard out of the scope of the standard process itself but without loosing the portability of applicaton using the extensions.

projects

• XPath
  => EXPath
• XSLT 1.0
  => EXSLT
• XSLT 2.0
  => see discussion about EXSLT 2.0
  
• XQuery
  => EXQuery
• XProc
  => EXProc

beside EXSLT which is well-known and already produced a sufficient amount of output all other EX communities are brand new and therefore doesn't provide any output so far.

watch and contribute......

identify content: powerful but tricky

regular expression and xpath are two approaches to identify a matching subset of content within a given amount of content for further usage. the first one based on plain text the second one based on xml.

both are powerful but if you not using it on a day by day basis are tricky and error prone. error prone means that you have to avoid all edge cases where a given expression shouldn't match but it does or the other way around where a given expression should match but it doesn't.

if you work in XML related world you often need both approaches e.g. using XSLT or XQuery.

there are few commercial IDE's which helping out to develop the right expression for the required use case but there are two which i'm often use dedicated to help you dealing either with regular expressions or xpath

SketchPath

powerful standalone tool to create and verify xpath expression. support for xpath 2.0 and most features you ask for in this context. this tool is based on well-known Saxon for xpath 2.0 and .net subsystem for xpath 1.0

home: http://pgfearo.googlepages.com/

Regulator

powerful standalone tool to create and verify regular expression. with buildin regex analyser and on-the-fly validation, code generation (for .net and vb only), ....

download: http://downloads.sourceforge.net/regulator/Regulator20Bin.zip?modtime=1189740236&big_mirror=0

model a DITA compliant model

DITA today is more and more adopted at least in the techdoc domain. the success and adoption rate is based on two major advantages over other existing standards:

• adaptability
  the DITA data model can be adapted based on defined rules to specific needs and domains. this concept is called specialization
• modularization
  content creation is not based on document paradigm but on module paradigm. authors no more create documents they create topics which represent some artifact of the system they describe.
  

problem for an information architect

since i started working with DITA i reviewed several data models from coworkers, from customers and last but not least created by myself. All of them claimed to be DITA compliant.
Using them with also DITA complaint tools often fails for certain features or in some cases completely.

reason

1. the created models wasn't valid according to the released specification
   creating a complaint DITA model (DTD or W3C Schema) requires to know all rules and requirements the DITA specification provides without having any tool support goes beyond the support of creating regular DTD's and W3C Schema's. this means the information architect is leaving alone with the underlying complexity.
   
   i do not know exactly but based on data models i'm faced with i expect that at least 50% of customized DITA models out there are not complaint to the DITA spec.
   
   
   
   few years ago i already posted this here: http://dita.xml.org/how-verify-certain-datamodel-dita-complaint
   
2. the used tools requires additional, tools specific configuration or semantic or simple doesn't implement a feature according to the specification

the second reason is getting better over time and today you find a bunch of tools which are production ready even for enterprise usage.

the first reason still not resolved.

you might argue, why not use the DITA model out of the box?

• if you have specific requirements in your business process and you require additional semantic to support those
• if you have to simplify the usage of content creation for the authors and users to get better and more consistent content outcome.
  note: i consider DITA subsetting / configuration as just another way for the generic concept specialization
  
• if you introduce new business domain / taxonomy into your content maintenance strategy.

issues if your model doesn't really confirms to the specification?

• interoperability is no more guaranteed
  this is mainly a problem for the enterprise and in most cases not at the beginning of the usage of DITA
  but if you once look into this problem you forced to fix all content created against your model or adopt the processing chain your content is delivered to
• process chain does not work
  you add new tools to your environment and certain features doesn't work. you again can fix all your existing content and the data model and tools or adopt the tool with certain workarounds

this week i stumbled over DITAworks modeling module which is the first tool I'm aware of addresses the modeling use-case in the DITA domain.

I'm not yet verified it to see to until which extend this tool supports the modeling / validation process and how much manual work is still involved and how complete the implementation currently is. because this tool is still beta and is brand new i expect much outstanding work but i hope this tool opens the door for speed up DITA related development work and the more important part of the story improve quality for DITA related information models.

i know that the amount of money / amount of customer can be found in this domain is pretty small and the complexity of the problem isn't small enough therefore i do not expect too much competitors in near future, but i might be wrong.

time will show....

content & service composition: small and simple showcase

if you want to see and learn how easy a information aggregation use case incl. corresponding presentation can be solved take a look at "Make dashboards with XQuery".

this sample is all about composition, from content and service (functional) point of view.

most of the concepts required in the file of information processing are involved. even if the implementation has drawbacks and limitation in several points you see how information centric requirements can be solved.

ODF and OOXML: interoperability issues

as already mentioned here using a standard does not guarantee to be vendor / implementation independent.

the following paper "Lost in Translation: Interoperability Issues for Open Standards -- ODF and OOXML as Examples" shows that complexity is one major show stopper in this area.

why?

OOXML and ODF try to define office documents this means that content, content structure and layout (and application semantic) must be standardized.

because the complexity of both standards is high only limited amount of vendors are able to implement 100% coverage of the standard and even if they try they are not able to prevent errors in implementation.

lessons learned?

standards should carefully consider good old "Everything should be made as simple as possible, but no simpler." (see http://en.wikiquote.org/wiki/Albert_Einstein) principle. in our context this means that those standards require certain, atomic level of conformance which makes it possible for each vendor to implement a certain and complete subset if the complete set is not possible and useful for a particular application.

maps in svg

if you search for maps in svg you can have a look at blank maps from wikipedia: http://en.wikipedia.org/wiki/Wikipedia:Blank_maps

svg is pretty smart if you require automatic creation of dynamic content like charts and visual reports based on dynamic data.

by the way svg standard has still many limitation in terms of interoperability and tool support. but in case svg cannot be published as-is Apache Batik is there to transform the svg source into format you require (pdf, eps, png, tif).

Generate DITA Java API reference documentation using DITADoclet and DITA API specialization

one example to show a step to close the gap between developers who creating the artifacts and outgoing and enriched documentation created from their source without the need of additional redundancy.
it is not too far away from approach introduced by javadoc / doxygen beside the datamodel used behind this approach has much more value for additional information integration as provided by the classical approach.

not all yet done by the "out-of-the box" solution provided here: http://www.ibm.com/developerworks/xml/library/x-DITAdoclet/
but you can imagine the possibilities using this kind of approach:

• integration of additional information provided by marketing or tech-doc groups or service groups without any media break in information usage
• usage of other deployment processes for api documentation available within companies infrastructure
• .....

Pipelines and SOA -> pipelines are more than that

new book "Software Pipelines and SOA" showing the advantages using "software pipeline" approach from very technical and partial point of view.

the pattern itself has much more benefit and performance scalability is only the one of them (and in many cases not the most important one). the principle to process a flow of information in atomic, semantically closed steps is a pattern which unix itself was build on.
if you ever seen how easy even complex processes can be created based on pretty simple and atomic executables you can imagine what pipelining mean in terms of software architecture (see also http://en.wikipedia.org/wiki/Pipeline_(Unix))

unix (and nowadays many other similar infrastructures) use a abstract stream of data passed through the pipeline and therefore the flow of information is less semantic, is is only a abstract stream of data. the usage of pipelines can be improved if more common semantic is part of the overall pipeline, means each step / operation has more information of what is expected to flow through than just data.

XML pipelining is intent to use XML as information flow. this ensures that data processing / transformation can be done using less basic byte stream operation. processing can be done on declarative languages like xpath, xquery, xslt, .... which again reduce the complexity of information access and transformation.

xml itself can express endless amount of user data using different data models. a pipeline defined for a subset of data models can again reduce the complexity and therefore can improve the benefit of a particular pipeline infrastructure.
you can imagine pipelines dedicated to transform content created against a DITA data model into endless of distribution formats. the semantic of a particular pipeline step can be used in several pipes are much higher than on "general purpose XML" level.

if we're looking into the second term "SOA" within the mentioned book title we have to divide:

1. pipelining to orchestrate dedicated services (macro pipelining)
   because each business process can be expressed using the pipeline paradigm the implementation of SOA orchestration is suggested to do using the pipeline pattern.
   
   therefore you have to define the sequence / control flow of services and corresponding message transformation.
   languages like BPEL providing a model to express such kind of pipelines.
   
   this layer often require persistence of pipeline state because execution of such processes can take between hours and years.
   this layer often requires human steps, means not the complete pipeline can be executed by a machine without human interaction. languages like BPEL4People are extension to cover this standard requirement.
   
   there are many frameworks out there trying to provide an easy to start infrastructure. by the way the usage and complexity of current implementation must be still not underestimated.
2. pipelining to solve one or more dedicated business steps (micro pipelining)
   within each business process a given amount of data confirms to specification A must be transformed into data confirms to specification B.
   e.g. extracting order data from ERP system and add sum for particular product groups before the result must be render to HTML for later display to the person in charge.
   
   those operation can of course as well defined as sequence of steps in which the input data is transformed into output data using multiple steps. those definition mainly derived from business rules.
   
   
   
   this layer does not require human interaction and persistence and therefore can be implemented on fully automated frameworks. using XML as data backbone results in "xml pipelining which combines most advantages required for "micro pipelining"
   
   languages like xproc, xpl , .... and corresponding implementation can be used in this area.
   
   in general a micro pipeline transform one input of a macro pipeline step into corresponding output(s)
   

summary

pipelining is one of the most powerful paradigm we faced with for todays common IT problems. but this pattern is either new not magic its more a "back to the roots....".

just using xml provides Interoperability?

"The Anatomy of Interoperability" is one of the best articles summarize the issue of well know and often promised term "interoperability".

one domain often faced with this term is the world of xml and related "standards". lot's of them out there, some of them really stable and useful and even are interoperable (e.g. xml 1.0, xslt 1.0, xpath 1.0) itself.

by the way just using xml does not gurantee interoperablility for your data. this is only available if application behavior is addressed by a related standard. xml related standards try to achive this (e.g. svg) often fail or they are difficult to use because they missing essential features the specific user domain requires and corresponding tool vendors / application provides add them in a tool specific way. or the standard is too complex to implement a 100% complaint application (e.g. xlink).

DITA for example a new OASIS standard / information architecture to maintain mainly techdoc related features more and more faces with those issues. this standard has customization in mind, means specialization to specific needs is part of the design but there are of course still limitation and there a good reasons for those limitations in general.

the initial standard was not feature complete (means essential requirements were missed in user point of view) and therefore vendors /consultants / end user adding specific non complaint features for their specific needs which often results in missing the goal of interoperability.

why is DITA still successful?

to understand this you have two things to consider:

• keep in mind that just using xml does not solve your interoperability goals without any additional effort
• keep in mind that fully inoperable data is not always what you need. regular business cases often working well with a inoperable subset or predefined transformation on demand.

what makes xml worse to look at is the ability to access / transform the tagged data without requiring a vendor specific api (but often you need vendor specific know how to understand the application semantic) and a foreseeable effort to do this.

and that is the key feature if you think about organization specific information models.

additional semantic for existing information sources

how information enrichment works in real world showing two public use-cases based on different technologies:

• freebase
  "Freebase is an open, shared database that contains structured information on millions of topics in hundreds of categories. This information is compiled from open datasets like Wikipedia, MusicBrainz, the Securities and Exchange Commission, and the CIA World Fact Book, as well as contributions from our user community."
  
  this means that applications like freebase using already existing information and trying to add additional semantic to them based on combining and extracting information and context or in this case let user add additional semantics without modification to the source of the information.
  
  the tool thinkbase using freebase to provide a visual graph of information and corresponding link dependencies.
  
• MailMark using a xml database (Mark Logic) as backbone for building the application just on XQuery
  
  the semantic comes from information aggregation and combination. in this application no additional user interaction is possible
  

both examples are very useful and showing real world application which you might transform into your own information landscape.....

XBRL: a language for the electronic communication of business and financial data

good overview can be found here: http://www.ibm.com/developerworks/library/x-think43/