openArchitectureWare XSD Tutorial (tweak)

Recently, I have needed to interogate a set of XML artifacts. I've found that my tooling (Stylus Studio, etc.) has let me down when it comes to transforming and understanding the various XML documents and schemas. For me it's not about having the tool do it all, it's about developing skill in handling the code/language/text etc.

I started with the openArchitectureWare XSD tutorial. In order to do this, I had to "modify" my oAW installation; in fact, I redid it since I found that the XSD Adapter is new and not included in the standard distribution and, in order to use it, newer modules were needed. This entry is good therapy for me in that, by writing it down, I am finalizing what I've learned and perhaps helping others in the process.

Here is what I installed to get things working. That last one was very important in that neither the workflow nor the XSDMetaModel reader worked until it was installed (hence it's last in the list :).

  00-eclipse-oaw-43-ganymede-win32.win32.x86.zip (317,467,119)
  01-dtp-sdk_1.6.0.zip (26,059,296)
  02-eclipse-test-framework-3.4.zip (715,044)
  03-wtp-R-3.0.1-20080818032401.zip (61,813,465)
  04-wtp-wst-R-3.0.1-20080818032401.zip (37,307,900)
  05-mdt-uml2tools-SDK-incubation-0.8.0.zip (12,166,067)
  06-xsd-runtime-2.4.1.zip (1,185,283)
  07-xsd-sourcedoc-2.4.1.zip (5,807,873)
  08-mdt-ocl-SDK-1.2.1.zip (11,984,668)
  09-oaw-xsd-adapter-rev20080702-1635.zip (1,335,497)
  10-gmf-sdk-2.1.1.zip (35,010,940)
  11-orbit-R20080611105805.zip (75,846,607)
  12-emf-sdo-xsd-SDK-2.4.1.zip (28,890,765)

Since I was interested in delving into a fairly complicated schema-set that defined the payload for some fairly sensitive information, another thing that I needed was the mother of all schemas. You can get that at http://www.w3.org/2001/XMLSchema. I saved it as "xmlschema.xsd" and used it as my metamodel and each of the component elements of the schema-set as my input model. Check out the workflow below to see what I mean. I am hoping in the end to be able to generate a decent UML2 model or, if that didn't seem feasable, an EMF model.

Here are the steps that I took:

1. Installed the above software.
At the time, the files above represented the latest releases of all of the software. Perhaps you will find that you can just download and install the latest releases also. But for the record, the files above together comprised a configuration that worked.

2. Created a new openArchitectureWare project and adjust a few settings

- File -> New -> openArchitectureWare Project
- Project -> Properties ... openArchitectureWare ... 
    [X] Enable project specific settings ... 
    [ ] EMF Metamodels, 
    [X] XSD Metamodels
- Open <project>/META-INF/MANIFEST.MF ... 
  - Overview tab ... Execution Environments ... Add ... 
      JavaSE-1.6 ... OK
    - Configure JRE associations ... Execution Environments ... 
        JavaSE-1.6 ... [X] jre 1.6.0_03 [perfect match]
      - Update the classpath settings (warnings should 
          clear if they are/were present)
    - Dependencies tab ... Add ... 
        org.openarchitectureware.xsd (4.3.0) ... 
        OK ... Ctrl-S

now, after several hours, you're ready to do the XSD tutorial if you want to: http://softeys.de/oaw-xsd-adapter/doc/contents/xsd_tutorial.html


3. Prepare the files

- Import the xmlschema.xsd file into src
    (R-Click src -> Import ... File System ... 
    Next ... Browse ... <containing directory> ... 
    [X] xmlschema.xsd ... Finish).
- Import the target XML schema files or, as was my 
    case, the directory containing the schemas  
    (R-Click src -> Import ... File System ... 
    Next ... Browse ... <containing directory> ... 
    [X] <containing folder> ... Finish).

4. Created a Worflow file (e.g. schema-walker.oaw).

<workflow>
 <property name="metamodel" value="xmlschema.xsd"/>
 <property name="model" value="niem/fbi/2.0/fbi.xsd"/>
 <property name="output" value="src-gen/xml" />
 <property name="template" value="schema2xml" />
 
 <component class="org.openarchitectureware.xsd.XMLReader">
  <modelSlot value="model" />
  <uri value="${model}" />
  <metaModel id="mm"
   class="org.openarchitectureware.xsd.XSDMetaModel">
   <schemaFile value="${metamodel}" />
  </metaModel>
 </component>
 
 <component class="org.openarchitectureware.xpand2.Generator">
  <metaModel idRef="mm" />
  <expand value="${template}::Root FOR model" />
  <outlet path="${output}" />
 </component>
</workflow>

6. Created an Xpand template (e.g. schema2xml.xpt). One thing to keep in mind is that the "xmlschema" metamodel needs to be on the classpath somewhere. So the fact that I have "xmlschema.xsd" in the "src" folder which is on the classpath (R-Click <project> ... Properties ... Java Build Path ... Source tab ... <project>/src) and that I've likewise "activated" the "XSD Metamodels" (R-Click <project> ... Properties ... openArchitectureWare), then that file will be found and interpreted into a .ecore metamodel and is therefore referencable by name in various oAW code types including Xpand files.

«IMPORT xmlschema»

«EXTENSION support»

«DEFINE writeFile FOR xmlschema::ComplexType»
«FILE name + ".xml" -»
<complex name="«name»"«complexContent.extension.extends()»>
 «FOREACH annotation.documentation AS doc-»
 «IF doc.mixed != null && doc.mixed.size > 0-»
 <description>«doc.docString()»</description>
 «ENDIF-»
 «ENDFOREACH-»
 «FOREACH complexContent.extension.sequence.element AS elem -»
 <element name="«elem.getElementName().toFirstLower()»" type="«elem.getElementName()»"/>
 «ENDFOREACH-»
 «FOREACH sequence.element AS elem -»
 <element name="«elem.getElementName().toFirstLower()»" type="«elem.getElementName()»"«elem.nullableAttribute()»«elem.ceilingAttribute()»/>
 «ENDFOREACH-»
</complex>
«ENDFILE-»
«ENDDEFINE»

«DEFINE writeFile FOR xmlschema::SimpleType»
«FILE "codes/" + name + ".xml" -»
<enumeration name="«name»">
 «FOREACH annotation.documentation AS doc-»
 «IF doc.mixed != null && doc.mixed.size > 0-»
 <description>«doc.docString()»</description>
 «ENDIF-»
 «ENDFOREACH-»
 «FOREACH restriction.maxExclusive AS me-»
 <max-exclusive>«me.value»</max-exclusive>
 «ENDFOREACH-»
 «FOREACH restriction.maxInclusive AS mi-»
 <max-inclusive>«mi.value»</max-inclusive>
 «ENDFOREACH-»
 «FOREACH restriction.minExclusive AS me-»
 <min-exclusive>«me.value»</min-exclusive>
 «ENDFOREACH-»
 «FOREACH restriction.minInclusive AS mi-»
 <min-inclusive>«mi.value»</min-inclusive>
 «ENDFOREACH-»
 «IF restriction.enumeration.size > 0-»
 «FOREACH restriction.enumeration AS enum-»
 <code key="«enum.value»" value="«enum.annotation.annotationDocString()»"/>
 «ENDFOREACH-»
 «ENDIF-»
</enumeration>
«ENDFILE-»
«ENDDEFINE»

«DEFINE writeFile FOR xmlschema::AnyType»
«ENDDEFINE»

«DEFINE Root FOR xmlschema::SchemaType»
 «FOREACH complexType.typeSelect(xmlschema::ComplexType) AS ct»
  «EXPAND writeFile FOR ct»
 «ENDFOREACH»
 «FOREACH simpleType.typeSelect(xmlschema::SimpleType) AS st»
  «EXPAND writeFile FOR st»
 «ENDFOREACH»
«ENDDEFINE»

The supporting calculations (support.ext) look like this:

import xmlschema;

extends(ExtensionType e) :
 (e.base.localPart.length > 0 ? " extends=\"" + e.base.localPart + "\"" : "");

annotationDocString(AnnotationType at) :
 (at != null && at.documentation != null && at.documentation.size > 0 ? at.documentation.get(0).docString() : "");

docString(DocumentationType dt) :
 (dt != null && dt.mixed != null && dt.mixed.size > 0 ? dt.mixed.get(0).value.toString().escape() : "");
 
getElementName(LocalElement elem) :
 (elem != null ? elem.ref.localPart : "unknown");
 
nullableAttribute(LocalElement elem) :
 (elem != null && elem.minOccurs != null ? " nullable=\"" + (elem.minOccurs != 0 ? "false" : "true" ) + "\"" : "");
 
ceilingAttribute(LocalElement elem) :
 (elem != null && elem.maxOccurs != null ? " ceiling=\"" + elem.maxOccurs + "\"" : "" );

escapeLT (String s) :
 (s.contains("<") ? s.replaceAll("<", "<") : s);
 
escapeAmp (String s) :
 (s.contains("&") ? s.replaceAll("&", "&") : s);
 
escapeQuote (String s) :
 (s.contains("\"") ? s.replaceAll("\"", """) : s);
 
escape (String s) :
 s.escapeAmp().escapeLT().escapeQuote();

7. Ran the output to be sure that everything is working (R-Click the workflow -> Run As -> oAW Workflow). Here is my output.

0    INFO  WorkflowRunner     - --------------------------------------------------------------------------------------
16   INFO  WorkflowRunner     - openArchitectureWare 4.3.0, Build 20080508-1430PRD
16   INFO  WorkflowRunner     - (c) 2005-2008 openarchitectureware.org and contributors
16   INFO  WorkflowRunner     - --------------------------------------------------------------------------------------
16   INFO  WorkflowRunner     - running workflow: schema-walker.oaw
16   INFO  WorkflowRunner     - 
1344 INFO  XSDMetaModel       - Loading XSDSchema from 'xmlschema.xsd'
3828 INFO  XSDResolver        - Importing 'http://www.w3.org/2001/xml.xsd' (http://www.w3.org/XML/1998/namespace) from XSDSchema 'file:/.../bin/xmlschema.xsd' (http://www.w3.org/2001/XMLSchema)
4672 INFO  OawXSDEcoreBuilder - Creating EPackage 'xml' from XSDSchema 'http://www.w3.org/2001/xml.xsd' (http://www.w3.org/XML/1998/namespace)
4688 INFO  OawXSDEcoreBuilder - Creating EPackage 'xmlschema' from XSDSchema 'file:/.../bin/xmlschema.xsd' (http://www.w3.org/2001/XMLSchema)
4938 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'anyAttribute1', EAttribute 'anyAttribute' is in the way. Container1:/EPackage'xmlschema'/EClass'OpenAttrs', Container2:/EPackage'xmlschema'/EClass'AttributeGroup'
4938 WARN  OawXSDEcoreBuilder - Name Conflict: Created EAttribute 'group1', EReference 'group' is in the way. Container:/EPackage'xmlschema'/EClass'RestrictionType'
4969 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'anyAttribute1', EAttribute 'anyAttribute' is in the way. Container1:/EPackage'xmlschema'/EClass'OpenAttrs', Container2:/EPackage'xmlschema'/EClass'RestrictionType'
4969 WARN  OawXSDEcoreBuilder - Name Conflict: Created EAttribute 'group1', EReference 'group' is in the way. Container:/EPackage'xmlschema'/EClass'ComplexType'
4969 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'anyAttribute1', EAttribute 'anyAttribute' is in the way. Container1:/EPackage'xmlschema'/EClass'OpenAttrs', Container2:/EPackage'xmlschema'/EClass'ComplexType'
4969 WARN  OawXSDEcoreBuilder - Name Conflict: Created EAttribute 'group1', EReference 'group' is in the way. Container:/EPackage'xmlschema'/EClass'ExtensionType'
4985 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'anyAttribute1', EAttribute 'anyAttribute' is in the way. Container1:/EPackage'xmlschema'/EClass'OpenAttrs', Container2:/EPackage'xmlschema'/EClass'ExtensionType'
4985 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'all1', EReference 'all' is in the way. Container1:/EPackage'xmlschema'/EClass'Group', Container2:/EPackage'xmlschema'/EClass'RealGroup'
4985 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'choice1', EReference 'choice' is in the way. Container1:/EPackage'xmlschema'/EClass'Group', Container2:/EPackage'xmlschema'/EClass'RealGroup'
4985 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'sequence1', EReference 'sequence' is in the way. Container1:/EPackage'xmlschema'/EClass'Group', Container2:/EPackage'xmlschema'/EClass'RealGroup'
5000 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'group1', EAttribute 'group' is in the way. Container:/EPackage'xmlschema'/EClass'RedefineType'
5000 WARN  OawXSDEcoreBuilder - Name Conflict: Created EClass 'RestrictionType1', EClass 'RestrictionType' is in the way. Container:/EPackage'xmlschema'
5016 WARN  OawXSDEcoreBuilder - Name Conflict: Created EAttribute 'group1', EAttribute 'group' is in the way. Container:/EPackage'xmlschema'/EClass'SchemaType'
5016 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'group2', EAttribute 'group' is in the way. Container:/EPackage'xmlschema'/EClass'SchemaType'
5047 WARN  OawXSDEcoreBuilder - Name Conflict: Created EReference 'annotation1', EReference 'annotation' is in the way. Container:/EPackage'xmlschema'/EClass'SchemaType'
5063 WARN  OawXSDEcoreBuilder - Name Conflict: Created EDataType 'XpathType1', EDataType 'XpathType' is in the way. Container:/EPackage'xmlschema'
5063 INFO  XSDMetaModel       - Registering EPackage 'xmlschema' (http://www.w3.org/2001/XMLSchema) globally.
5266 INFO  CompositeComponent - XMLReader: Loading XML file niem/fbi/2.0/fbi.xsd
7188 INFO  CompositeComponent - Generator: generating 'schema2xml::Root FOR model' => directory 'null'
35938 INFO  Generator          - Written 53 files to outlet [default](src-gen/niem/fbi)
35938 INFO  WorkflowRunner     - workflow completed in 30672ms!

Now I'm off to get to know the schema through altering my template file and reviewing the output. Once I think that I have a good understanding of what's there, then I'll move on to UML2 model creation. "There is joy in the journey" :)

Here is a link to the source.

8. Create an Xtend file (.ext).
This may take some time so I'll fill it in later. Email me if you're getting impatient.

MDSD using patterns

Is it possible to use patterns as both a meta language and an intermediate model in MDSD? My engineering brain tells me that there is enough information there to generate the code, after a few transformations, for, say, a JMS exchange by simply naming a component, stereotyping it with <<Observer>>, and filling in a few tagged values. The first transform would take this component and create a composite model using the name and the tagged values from the abstract model and the pattern model from the "patterns.objectInteration.observer" package. This composite model, in turn, might be transformed a few more times but eventually be used to drive out the implementation code and configuration for the target platform along with the documentation and tests. The "JMS nature" however might be added after the second or third transformation.

Now, I've learned that it's a good idea to scour the web first before launching off on a grand design such as this. I mean, I don't think that I'm the first one to think of this, but, due to it's scope, I might be the first to be willing to share what I'm doing along the way. Perhaps this is a sort of Holy Grail of programming that developers keep close and locked in their tool box. But, from what I've seen in the development world, there are only a few that would be so inclined to have such a tool. So I'm going to do a few prototypes and assess the feasibility of this approach. Keep posted. I'm all for raising the level of abstraction where possible, and this approach, if successful, definitely will. Plus, it might give Gegor Hohpe's neon icons a wider range of use :)

For now, what I think I need is a good intermediate model that would contain the patterns that I'm using in my design and also a UML profile or DSL that could be used in my component model. So my plan is to create a simple component model and see what is needed to generate some of the platform specific artifacts that I'd like to have. My prototype will use the Observer pattern to create a JMS message exchange between the subject and several observers.

A few years back, one company that I was working for was creating a design for collecting NOAA's weather data. At the time, I thought that a Jini solution was in order then but the guy who was actually developing it was keen on using a database as the communication mechanism. We didn't go on to get the development work :( If only he had used the Observer pattern and hadn't committed to an implementing technology, eh :) Here's a thumbnail of a problem statement: the weather station is comprised of a set of measurement instruments that are remotely deployed. Periodically, it makes it's measurements available to all interested parties. This data needs to be collected and managed so that it can eventually be used by various agencies and businesses. Seems like a good candidate for the observer pattern to me; at least the data collection piece anyway. Truth be known, the final solution would be a bit more complicated and would probably include data collection for each weather station using a polling mechanism, this collection system would periodically post it's data, observers could then do whatever is useful to them which might include conditioning, filtering, storing, etc. Here's the initial component diagram.

One of the benefits of keeping it at this level is that the pattern isn't technology bound but is "independent" of the platform. From MDA, this would be labeled the PIM (platform independent model). In MDSD this would be an architecture-centric way of going about designing things which, in it's purest implementation, would be one step further along in the design than the domain-centric design. So perhaps the use of a patterns profile or DSL would be a good language for architecture centric abstract design. This would be helpful since there always is the guy who wants to put the strategy into the vision. You've heard it I'm sure, "Why don't we just stereotype it as an EJB; it's what we're going to use isn't it?" An abstract lingua franca would be very useful for putting such people in their place by being able to simply say, "There isn't an EJB stereotype available but there are several patterns that might do. How about 'Client Proxy', 'Business Delegate', or 'Session Facade'? What exactly are you trying to communicate when you say 'EJB'?"

Onward!

For everything a place, a place for everything

What's in here?

• Intro to MDSD
• Where I'm heading
  

I'm actively trying to get a grip on all of the implications that MDSD brings along with it. I've been listening to a few podcasts and viewing a few videos on InfoQ just to help the conversation that I have going on in my head over this topic. Sven Efftinge, Markus Voelter, Martin Fowler, et al have been close companions of late :) There is the heady world of DSL generation. Likewise, there are the model-to-model (M2M) and model-to-text (M2T) paradigms that, at the same time, make perfect sense but give the distinct impression that there is more to them than meets the eye. The technologies in this space are not few so ... odds are you know what I mean.

I've been a heavy user of AndroMDA over the past few years and have found it to be a great time saver along with producing high quality, feature rich code. For the first time I felt like I could spend most of my time on the business logic and relatively little on the wiring. I can remember the first time that I successfully ran it against one of my models; I was so thankful that I didn't have to handcraft the WSDL file or EJBs or the persistence layer. Don't get me wrong, the learning curve wasn't a walk in the park, but it was ultimately rewarding. The hook had been set and there was no turning back. I've always had a predisposition toward code generation probably more from a productivity boosting motivation rather than one of laziness or boredom. This predisposition is sated with the current release of openArchitectureWare.

One of the main things that I really like about model driven anything is that it marries the design models and the code. I believe that it's a relatively large problem to have design documents that are not only poor images of the deployed programs but, in most cases, bearing very little resemblance to the component parts of the system at all. By adopting MD*, the software development life cycle is in tact because it has to be or the project is out of control. This is good news especially for those working in the government and regulated sectors.

So the two immediate benefits are time savings and a complete life cycle. Yet another benefit is that it reintroduces the creativity back into programming. You see, the generators are very good at creating the mundane stuff, leaving it to the programmers to create (note the use of the word create) the code that realizes the business logic. Generally, this will require two tiers or peers of programming staff, those who attend to the business things and those who attend to the architecture and the plumbing generators. So you see, this is as significant an advance as subroutines, C, 4GLs, and OOP were. So now we are at the point where the target architectures are being codified; we've moved from Assembler all the way up through components and are now focused on the services layer. Next up would be the dream of composable systems and what I like to call the percolating enterprise. Beyond that would be a sort of "universal consciousness" (go to a quiet place, sit on the floor, close your eyes, and say Ummmm) as we create a trusted layer between networked entities; a piece of which is envisioned as the semantic web.

It's my opinion that the leading edge at the moment is the work around creating secure, consistent, high performance, networked systems, i.e. just a few steps behind those bleeding edge pioneers who are working on the "Ummmm" generation of systems. There are a host of tools available in this space and, as for my investment, openArchitectureWare is a clear choice. They have very functional tools for M2M and M2T work as well as a solid platform that integrates it all. They can work with almost everything that could constitute a model from a simple text file via their Xtext DSL modeling framework to UML2 compliant models. It's all there and ready to use. One of the interesting things that the oAW team has chosen to do is to leave out cartridge development; leaving this task to third parties while they concentrate on the enabling platform. This is a very good move in my estimation. Just for reference, one of the cartridge factories that has already cropped up is the Fornax Platform. I really don't see a downside to adopting this platform other than the cartridge issue but given it's flexibility, there is no reason why AndroMDA compliant models can't be generated that could then leverage their cartridges. The sky is the limit.

The waters that I'm wading into seem to be fairly deep. But coming from the AndroMDA world, the terrain is fairly familiar and the good news is that I don't have to loose touch with where I came from; it still can play a significant part in my efforts. At the same time, I am very sure that it will be worth the risk and the effort in the crossing. Along with my experience using MDSD and creating DSLs, odds are good that I'll have a healthy diet of software architecture patterns to talk about and more than a few diagrams to boot. This blog will function as my bulletin board, diary, and forum. A good friend of mine has this maxim when it comes to organizing her world, "for everything a place, a place for everything". Though I'm still coming to grips with the implications of this, I've decided that this blog will be my place to put things about software development and architecture.

'nuf said for now, eh? :)