ServingXML Examples

Daniel Parker

Shaun Woodrow

Contributor of "trades" example 

Dimitrios Varsos

Contributor of "tasks" and "timesheets" examples 

Pedro Mendoza

Contributor of "hot" example 

Pierre-Alexandre Losson

Contributor of "invoices" example 

Aswin Majjiga

Contributor of "books-csv" example 

Dudley

Contributor of Java properties "nested tags" examples 

alexbena

Contributor of "CONV" example 

Zach

Contributor of "ars" example 

Sapna

Contributor of "BNF like flat file" example 

Bill Donovan

Contributor of edifact example 

Ahmadin

Contributor of abtest example 

Scott

Contributor of swath example 

Peter Scholz

Contributor of "segments" example 

Wonne Keysers

Contributor of "persons" example 

Ken Brewer

Contributor of "students" example 

Bala Murali

Contributor of "book orders" and "checked books" examples 

etdwh

Contributor of "opdef" example 

David Leal Valma

Contributor of "field has max width" example 

Anton Melumad

Contributor of "optional elements" example 

Lance Zant

Contributor of "XML-flat header-detail" example 

dealogic

Contributor of XML to flat "sequence" example 

Jim Williams

Contributor of XML to flat "multiple summaries" example 

Alvin

Contributor of XML to flat "book-listing" example 

Jim Nurii

Contributor of flat to XML and XML to flat "special char" examples 

Jose Ignacio Santa Cruz

Contributor of flat to XML "invoice" example 

Valentine

Contributor of XML to flat "ungrouping" example 

Ravikumar Tadysetty

Contributor of XML to flat "all" example 

José Santos

Contributor of XML to flat "formating distinct record lines" example 

Ramesh Ramasamy

Contributor of XML to flat "book hierarchy" example 

Gordon Zhang

Contributed additions to "exotics" example 

Roopashree Hirasave

Contributed "X12 XML to Flat File" example 

Ralph Kutzner

Contributed Flat-to-XML example "Computing an Aggregate Value over Multiple Records" 

Randal Cobb

Contributed Flat-to-XML example "Mapping Two Records at the Top of a File to a Header Element in the XML Output"  

Nicolas Denis

Contributed Flat-to-XML example "A UTF-8 positional flat file with double byte characters"  

Johan Hammar

Contributed Flat-to-XML example "Converting a positional flat file with variable field widths to XML"  

Asanga Porage

Contributed the "eobclaims" examples, the flat-to-XML "Flat File to Summary and Detail XML Documents", and the XML-to-flat "Summary and Detail XML Documents to a Flat File"  

Luca Buraggi

Contributed the XML-to-Database "Load Master Detail" example.  

Graham Hannington

Contributed the Flat-File-to-XML "SMF" example.  

Sajeev Nair

Contributed the Flat-File-to-XML "Record Concatenation" example.  

Gabriele Siviero

Contributed the Flat-File-to-XML "Invoic96A" example.  

Flat File to XML
Delimited Fields, Single Record Type
Converting a CSV File to XML Using Default Record Mapping (countries-default)
Converting a CSV File to XML Using Custom Record Mapping (countries-custom)
Converting a CSV file to XML with Record Validation (countries-validated)
Converting flat files with fields that end either with a delimiter or a maximum width (field has max width)
Converting a flat file to XML with one level of grouping (tasks)
Converting a flat file to XML with many levels of grouping (timesheets)
Converting a flat file with multi-valued fields to XML (family_data)
Search and Replace with Regular Expressions in Field Mappings (ars)
Converting a Tab Delimited Flat File to XML (tab-delimited)
Delimited Fields, Multiple Record Types
Converting a flat file with multiple record formats to XML with grouping (exotics)
Converting flat files with record mappings that have optional elements (optional elements)
Converting a Java properties file to XML (messages)
Transforming a Java properties file with extra name field delimiters to produce nested tags (nested tags 1)
Transforming a Java properties file with name subfield delimiters to produce nested tags (nested tags 2)
Reordering a group of records before mapping to XML (multiple summaries)
Delimited Fields, Multiple Record Types, Repeating Groups
Mapping multiple repeating groups to XML (students-delim)
Mapping a sequence of nested repeating groups to XML (invoice)
EDI Examples
EDI to XML (comptest)
EDI to XML with Escaped Delimiters (Invoic96A)
FpML Examples
FRA to FpML
Swap to FpML
Fixed Width Fields, Single Record Type, Line Delimited
A flat file to XML mapping with field substitutions (abtest)
Mapping a record to separate top level sections in the XML (persons)
Converting a directory of books positional files to XML files (all books)
Converting a flat file to multiple XML book files (multiple books)
Converting a flat file to multiple XML documents with default namespace prefixes (multiple default ns)
Flat Files with Multibyte Characters
A UTF-8 positional flat file with double byte characters
Fixed Width Fields, Multiple Record Types, Line Delimited
Converting a flat file with multiple record formats to XML (trades)
Converting a flat file to XML with header, detail, and summary records (CONV)
Converting a positional file to XML using a record filter (hot 1)
Converting a positional file to XML with outer and inner grouping (hot 2)
Converting a positional flat file with variable field widths to XML
Repeat Header Information in the XML
Fixed Width Fields, Multiple Record Types, Repeating Groups, Line Delimited
Converting fixed position records with repeating groups to XML (segments)
Mapping multiple repeating groups to XML (students-fix)
Fixed Width Fields, Single Record Type, No End-of-Line Delimiters
Converting a flat file with a single record type, but no end-of-line delimiters, to XML (non-delimited book orders)
Fixed Width Fields, Multiple Record Types, No End-of-Line Delimiters
Converting a flat file with multiple record formats, but no end-of-line delimiters, to XML (non-delimited trades)
Converting a flat file with special characters to XML (special char)
Converting a flat file containing Cobol packed decimal data to XML (packed decimal to XML)
Logical Record Consists of Multiple Physical Records
Converting a System Management Facilities (SMF) file to XML (Segment Concatenation)
Aggregating Multiple Physical Records into a Composite Record (Record Composition)
Computing an Aggregate Value over Multiple Records
Mapping Two Records at the Top of a File to a Header Element in the XML Output
Start/End Delimited Records.
Converting flat files with start/end record delimiters (opdef)
Converting flat files with nested start/end record and segment delimiters and name/value pairs (transaction)
Custom Record Reader
Converting the output of a custom record reader to XML (custom record reader)
Batching XML Output
Serializing XML in Batches
Writing Records to Batch Files, and Converting the Batches to Separate XML Files
Many XML Documents
Flat File to Summary and Detail XML Documents (eobclaims)
Many Flat Input Files
Multiple XML Readers
XML to Flat File
Converting an XML document to a positional flat file with record count in trailer (books-positional)
Converting an XML document with multiple occurances of an element to a CSV file (books-csv)
Converting an XML document with multiple occurances of an element to a flat file with subfield delimiters (books-pipe)
Converting each book entry in books.xml to a list of three records (book-listing)
Converting an XML document into flat file records with multi-valued fields (swath)
Converting an XML document into a flat file with header and multiple detail records (XML-flat header-detail)
Breaking up a sequence of elements into flat file records ("sequence")
Converting an XML file with hexidecimal encoded values to a flat file with special characters
Converting an XML document to a positional flat file and ungrouping an XML element (ungrouping)
XML to Flat Repeating Group (all)
Formating Distinct Record Lines (blocks)
Block Subsequences in XML to One Line
Ungrouping Elements with Multiple Levels (book hierarchy)
X12 XML to Flat File using XSLT
A Directory of XML Documents to a Flat File
Summary and Detail XML Documents to a Flat File (eobclaims)
Flat File to Flat File
Converting a positional file to a pipe delimited file with header (positional to delimited)
Converting a positional file to a new positional file with default field values (positional defaults)
Converting an ASCII Positional File to an EBCDIC Delimited File, and Vice Versa (ASCII-EBCDIC)
Converting an ASCII Positional File to an EBCDIC Positional File with a Cobol Packed Decimal Field, and Vice Versa (ASCII-Packed)
Converting a positional file to a pipe delimited file with record filtering (book orders)
Checking the Integrity of a Flat File ("checked books")
Converting a directory of book positional files to pipe delimited files (new format)
Generated records to multiple files (multiple_output_files)
Reading remote directories
Converting a sequential stream of records into multiple batch files
Batched Flat Files
Batched sequences of records
XML to XML
Extracting subtrees and wrapping them in containing tags
Unwrapping subtrees in an XML document
Streaming a file through a pipeline
Streaming dynamically generated SAX events through a pipeline
Processing document subtrees and serializing them to separate files (invoices.)
Processing document subtrees and serializing them as mail attachments ("mail invoices")
Reading dynamic content
XML to Mail
Writing an XML file to an FTP sink
Comparing Two XML Documents with fn:deep-equal
SQL Examples
SQL Query to Flat File
Writing the results of a SQL query to a CSV file (employees-csv)
SQL Query to XML
Mapping the results of a parameterized SQL query into XML (analysts)
Mapping the results of an ad hoc SQL query into XML (employees)
Using multiple readers to join data across different data sources (chained readers)
XML to Database
Load Master Detail
SQL Query to Database
SQL inserts
Batched SQL inserts
Prepared SQL inserts
Batched prepared SQL inserts
Updating an employee record using command-line parameters (employee update)
Insert if record does not exist, update otherwise (insert/update).
Writing rows to a database table as they pass through a pipe, to an XML file (employees-tee)
Index

The source for console app examples may be found in the distribution under the samples directory.

Before running the sample console applications, be sure to build the servingxml project. Note that the examples will be deployed under the target/servingxml/samples directory. See Getting Started for details.

Flat File to XML

Delimited Fields, Single Record Type

Converting a CSV File to XML Using Default Record Mapping (countries-default)

This example shows a simple resources script that will map a CSV file to XML using default record mapping.

The input file is a CSV file.

Figure 1. Input flat file countries.csv

          
code,name
#ABW,ARUBA
ATF,"FRENCH SOUTHERN TERRITORIES, D.R. OF"
VUT,VANUATU
WLF,WALLIS & FUTUNA ISLANDS

        

This file has a number of properties.

  • The first row is a header with comma-separated column labels.

  • The second row is commented out.

  • The third row has a field value, enclosed in quotes, that contains an embedded comma.

  • The fifth row has a field value that contains a '&' symbol, which is a special character in XML, and must therefore be escaped in XML output as the entity '&'.

Shown below is the simplest possible resources script that will read this data and output it as XML.

        
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="countries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="countries-doc"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="countries-doc" name="countries">
    <sx:flatFileReader>
      <sx:commentStarter value="#"/>
      <sx:fieldDelimiter value=","/>
      <sx:urlSource url="data/countries.csv"/>
    </sx:flatFileReader>
  </sx:recordContent>

</sx:resources>

      

This resources script produces the following output.

        
<?xml version="1.0" encoding="utf-8"?>
<countries>
  <record>
    <code>ATF</code>
    <name>FRENCH SOUTHERN TERRITORIES, D.R. OF</name>
  </record>
  <record>
    <code>VUT</code>
    <name>VANUATU</name>
  </record>
  <record>
    <code>WLF</code>
    <name>WALLIS &amp; FUTUNA ISLANDS</name>
  </record>
</countries>

      

Note the following properties of the XML output.

  • The root element name comes from the sx:recordContent element's name attribute, if it has one, and if not it defaults to document .

  • The record element name defaults to record .

  • Each field is mapped to an element, and the name of the element defaults to the column label.

Converting a CSV File to XML Using Custom Record Mapping (countries-custom)

An sx:recordMapping element positioned inside an sx:recordContent element allows for custom record mapping to XML. This is optional, since there is a default mapping that emits the canonical XML representation of records. But typically the XML you want is different from the canonical representation, you may want a field mapped to an attribute rather than an element, for example, or perhaps some additional literal elements around the field mappings.

Continuing with the previous example, suppose you want the following output.

        
<?xml version="1.0" encoding="utf-8"?>
<countries>
  <country countryCode="ATF">
    <countryName>FRENCH SOUTHERN TERRITORIES, D.R. OF</countryName>
  </country>
  <country countryCode="VUT">
    <countryName>VANUATU</countryName>
  </country>
  <country countryCode="WLF">
    <countryName>WALLIS &amp; FUTUNA ISLANDS</countryName>
  </country>
</countries>

      

The following resources script does the job.

        
<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="countries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="countries"/> 
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="countries">
    <sx:flatFileReader>
      <sx:urlSource url="data/countries.csv"/>
      <sx:flatFile ref="countriesFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="countriesToXmlMapping"/>
  </sx:recordContent>
  
  <sx:flatFile id="countriesFile">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="country">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="code"/>
        <sx:delimitedField name="name"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>      
  
  <sx:recordMapping id="countriesToXmlMapping">
    <countries>
      <sx:onRecord>
        <country>
          <sx:fieldElementMap field="name" element="countryName"/>  
          <sx:fieldAttributeMap field="code" attribute="countryCode"/>
        </country>  
      </sx:onRecord>
    </countries>
  </sx:recordMapping>  
  
</sx:resources>

      

You can also take the default mappings for some fields and map others explicitly. For example,

        
<sx:recordMapping id="countries2xml">
  <countries>
    <sx:onRecord>
      <country>
        <sx:defaultFieldElementMap fields="*" except="code"/>
        <sx:fieldAttributeMap field="code" attribute="countryCode"/>
      </country>
    </sx:onRecord>
  </countries>
</sx:recordMapping>

      

Here, the sx:defaultFieldElementMap element maps all the fields in the record to elements with the same names, except the field name "code". This field is handled separately, with the sx:fieldAttributeMap element.

Converting a CSV file to XML with Record Validation (countries-validated)

The example below shows how to convert a flat file to XML. Each row in the flat file is validated with Sun's multi schema validator (MSV), and if an error is encountered, the row is discarded, but processing continues. In addition, the output XML is also validated with Sun's MSV, and if that should fail, processing is stopped.

This time the input file has an invalid country code in the first record.

        
code,name
ATFX,"FRENCH SOUTHERN TERRITORIES, D.R. OF"
VUT,VANUATU
WLF,WALLIS & FUTUNA ISLANDS

      

The desired output is

        
<?xml version="1.0" encoding="utf-8"?>
<countries>
  <country countryCode="ATF">
    <countryName>FRENCH SOUTHERN TERRITORIES, D.R. OF</countryName>
  </country>
  <country countryCode="VUT">
    <countryName>VANUATU</countryName>
  </country>
  <country countryCode="WLF">
    <countryName>WALLIS &amp; FUTUNA ISLANDS</countryName>
  </country>
</countries>

      

The following resources script does the transformation.

Figure 2. Resources script resources-countries.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="countries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="countries"/>
        <msv:schemaValidator>
          <sx:urlSource url="data/countries.xsd"/>
        </msv:schemaValidator>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:flatFile id="countriesFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType ref="country"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType id="country" name="country">
    <sx:fieldDelimiter value=","/>
    <sx:delimitedField name="code"/>
    <sx:delimitedField name="name"/>
  </sx:flatRecordType>

  <sx:flatFile id="discardFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="countryDiscard">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="message"/>
        <sx:flatRecordType ref="country"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="countriesToXmlMapping">
    <countries>
      <sx:onRecord>
        <country>
          <sx:fieldElementMap field="name" element="countryName"/>
          <sx:fieldAttributeMap field="code" attribute="countryCode"/>
        </country>
      </sx:onRecord>
    </countries>
  </sx:recordMapping>

  <sx:recordContent id="countries">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:urlSource url="data/countriesWithDiscards.csv"/>
        <sx:flatFile ref="countriesFile"/>
      </sx:flatFileReader>
      <msv:recordValidator>
        <sx:urlSource url="data/country-record.xsd"/>
      </msv:recordValidator>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <sx:modifyRecord>
          <sx:newField name="message" value="{$sx:message}"/>
        </sx:modifyRecord>
        <sx:flatFileWriter>
          <sx:fileSink file="output/countryDiscards.csv"/>
          <sx:flatFile ref="discardFile"/>
        </sx:flatFileWriter>
      </sx:discardHandler>
    </sx:recordStream>
    <sx:recordMapping ref="countriesToXmlMapping"/>
  </sx:recordContent>

</sx:resources>

        

The schema to validate an individual country record is shown below.

Figure 3. XML Schema file country-record.xsd

          
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

 <!-- This element's name matches the value of the name attribute in the sx:flatRecordType element. -->
 <xs:element name="country" type="CountryType"/>

 <xs:complexType name="CountryType">
  <xs:sequence>
   <xs:element name="code" type="CountryCode"/>
   <xs:element name="name" type="xs:string"/>
  </xs:sequence>
 </xs:complexType>

  <xs:simpleType name='CountryCode'>
    <xs:restriction base='xs:string'>
      <xs:length value='3' fixed='true'/>
    </xs:restriction>
  </xs:simpleType>
</xs:schema>

        

You run this example on the command line by entering

        
servingxml -r resources-countries.xml -o output/countries.xml countries

      

The following error message will be written to the log:

        
Error in record "country" on line 1.  the length of the value is 4, but the required length is 3.

      

The remaining two rows will be written to the output file.

Converting flat files with fields that end either with a delimiter or a maximum width (field has max width)

The example below shows how to convert a flat file with start/end record delimiters to XML.

The input file is shown below.

        
1231234
1 123
12 12

      

This file has the following structure.

  • The first record consists of a field of length 3 ("123") followed by a field of length 4 ("1234")

  • The second line consists of a field of length 1 ("1") followed by a field of length 3 ("123")

Each field has a maximum width, but the field may be terminated by a delimiter (here whitespace) before it attains that width.

The desired output is shown below.

        
<?xml version="1.0" encoding="utf-8"?>
<document>
  <record>
    <field1>123</field1>
    <field2>1234</field2>
  </record>
  <record>
    <field1>1</field1>
    <field2>123</field2>
  </record>
  <record>
    <field1>12</field1>
    <field2>12</field2>
  </record>
</document>

      

The following resources script does the transformation.

        
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="fieldHasMaxWidth">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="fieldHasMaxWidthDoc"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="fieldHasMaxWidthDoc" name="document">
    <sx:flatFileReader>
      <sx:urlSource url="data/fieldHasMaxWidth.txt"/>
      <sx:flatFile ref="fieldHasMaxWidthFile"/>
    </sx:flatFileReader>
  </sx:recordContent>

  <sx:flatFile id="fieldHasMaxWidthFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="record">
        <sx:fieldDelimiter>
          <sx:whitespaceSeparator/>
        </sx:fieldDelimiter>
        <sx:delimitedField name="field1" maxWidth="3"/>
        <sx:delimitedField name="field2" maxWidth="4"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

      

You can run this example on the command line by entering

          
servingxml -r resources-fieldHasMaxWidth.xml  
  -o output/fieldHasMaxWidth.xml fieldHasMaxWidth

        

Converting a flat file to XML with one level of grouping (tasks)

The example below shows how to convert a flat file to XML with grouping levels.

The input file is a CSV file.

Figure 4. Input flat file tasks.csv

          
project_id, task_name, task_start, task_finish
4001,task1,01/01/2003,01/31/2003
4001,task2,02/01/2003,02/28/2003
4001,task3,03/01/2003,03/31/2003
4002,task1,04/01/2003,04/30/2003
4002,task2,05/01/2003,05/31/2003

        

The desired output is shown below.

Figure 5. Output XML file tasks.xml

          
 <?xml version="1.0" encoding="utf-8" ?> 
<Projects>
  <Project projectID="4001">
    <Tasks>
      <Task name="task1" start="01/01/2003" finish="01/31/2003" /> 
      <Task name="task2" start="02/01/2003" finish="02/28/2003" /> 
      <Task name="task3" start="03/01/2003" finish="03/31/2003" /> 
    </Tasks>
  </Project>
  <Project projectID="4002">
    <Tasks>
      <Task name="task1" start="04/01/2003" finish="04/30/2003" /> 
      <Task name="task2" start="05/01/2003" finish="05/31/2003" /> 
    </Tasks>
  </Project>
</Projects>

The following resources script does the transformation.

Figure 6. Resources script resources-tasks.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="tasks">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="tasks"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="tasks">
    <sx:flatFileReader>
      <sx:urlSource url="data/tasks.csv"/>
      <sx:flatFile ref="tasksFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="tasksToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="tasksFlatFile">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="task">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="project_id"/>
        <sx:delimitedField name="task_name"/>
        <sx:delimitedField name="task_start"/>
        <sx:delimitedField name="task_finish"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="tasksToXmlMapping">
    <Projects>
      <sx:groupBy fields="project_id">
        <Project>
          <sx:fieldAttributeMap field="project_id" attribute="projectID"/>
          <Tasks>
            <sx:onRecord>
              <Task>
                <sx:fieldAttributeMap field="task_name" attribute="name"/>
                <sx:fieldAttributeMap field="task_start" attribute="start"/>
                <sx:fieldAttributeMap field="task_finish" attribute="finish"/>
              </Task>
            </sx:onRecord>
          </Tasks>
        </Project>
      </sx:groupBy>
    </Projects>
  </sx:recordMapping>
  
</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -r resources-tasks.xml -o output/tasks.xml tasks

        

Converting a flat file to XML with many levels of grouping (timesheets)

The example below shows how to convert a flat file to XML with many grouping levels.

The input file is a CSV file.

Figure 7. Input flat file timesheets.csv

          
TimePeriod,Start,Finish,Resource,Task,ActualDate,Amount
1,2/9/2004,2/15/2004,Joe_100,1001,2/9/2004,8
1,2/9/2004,2/15/2004,Joe_100,1001,2/10/2004,4
1,2/9/2004,2/15/2004,Joe_100,1002,2/11/2004,8
1,2/9/2004,2/15/2004,Joe_100,1003,2/10/2004,4
1,2/9/2004,2/15/2004,Joe_100,1003,2/12/2004,8
1,2/9/2004,2/15/2004,Mark_101,1001,2/10/2004,8
1,2/9/2004,2/15/2004,Mark_101,1001,2/14/2004,8
1,2/9/2004,2/15/2004,Mark_101,1006,2/9/2004,8
1,2/9/2004,2/15/2004,Mark_101,1008,2/12/2004,4
1,2/9/2004,2/15/2004,Mark_101,1008,2/13/2004,4
2,2/16/2004,2/22/2004,Joe_100,1001,2/16/2004,8
2,2/16/2004,2/22/2004,Joe_100,1001,2/17/2004,4
2,2/16/2004,2/22/2004,Joe_100,1002,2/17/2004,4

        

The desired output is shown below.

Figure 8. Output XML file timesheets.xml

          
<?xml version="1.0" encoding="utf-8"?>
<TimePeriods>
  <TimePeriod start="2/9/2004" finish="2/15/2004">
    <Timesheets>
      <Timesheet resource="Joe_100">
        <TimesheetEntries>
          <TimesheetEntry task="1001">
            <DailyActuals>
              <Actual actualDate="2/9/2004" amount="8"/>
              <Actual actualDate="2/10/2004" amount="4"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1002">
            <DailyActuals>
              <Actual actualDate="2/11/2004" amount="8"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1003">
            <DailyActuals>
              <Actual actualDate="2/10/2004" amount="4"/>
              <Actual actualDate="2/12/2004" amount="8"/>
            </DailyActuals>
          </TimesheetEntry>
        </TimesheetEntries>
      </Timesheet>
      <Timesheet resource="Mark_101">
        <TimesheetEntries>
          <TimesheetEntry task="1001">
            <DailyActuals>
              <Actual actualDate="2/10/2004" amount="8"/>
              <Actual actualDate="2/14/2004" amount="8"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1006">
            <DailyActuals>
              <Actual actualDate="2/9/2004" amount="8"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1008">
            <DailyActuals>
              <Actual actualDate="2/12/2004" amount="4"/>
              <Actual actualDate="2/13/2004" amount="4"/>
            </DailyActuals>
          </TimesheetEntry>
        </TimesheetEntries>
      </Timesheet>
    </Timesheets>
  </TimePeriod>
  <TimePeriod start="2/16/2004" finish="2/22/2004">
    <Timesheets>
      <Timesheet resource="Joe_100">
        <TimesheetEntries>
          <TimesheetEntry task="1001">
            <DailyActuals>
              <Actual actualDate="2/16/2004" amount="8"/>
              <Actual actualDate="2/17/2004" amount="4"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1002">
            <DailyActuals>
              <Actual actualDate="2/17/2004" amount="4"/>
            </DailyActuals>
          </TimesheetEntry>
        </TimesheetEntries>
      </Timesheet>
    </Timesheets>
  </TimePeriod>
</TimePeriods>

        

The following resources script does the transformation.

Figure 9. Resources script resources-timesheets.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="timesheets"> 
    <sx:serialize>
      <sx:transform>
        <sx:content ref="timesheets"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>
  
  <sx:recordContent id="timesheets">
    <sx:flatFileReader>
      <sx:urlSource url="data/timesheets.csv"/>
      <sx:flatFile ref="timesheetsFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="timesheetsToXmlMapping"/>
  </sx:recordContent>
  
  <sx:flatFile id="timesheetsFlatFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="task">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="TimePeriod"/>
        <sx:delimitedField name="Start"/>
        <sx:delimitedField name="Finish"/>
        <sx:delimitedField name="Resource"/>
        <sx:delimitedField name="Task"/>
        <sx:delimitedField name="ActualDate"/>
        <sx:delimitedField name="Amount"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>      
  
  <sx:recordMapping id="timesheetsToXmlMapping">
    <TimePeriods>
      <sx:groupBy fields="Start">
        <TimePeriod>
          <sx:fieldAttributeMap field="Start" attribute="start"/>
          <sx:fieldAttributeMap field="Finish" attribute="finish"/>
          <Timesheets>
            <sx:groupBy fields="Start Resource">
              <Timesheet>
                <sx:fieldAttributeMap field="Resource" attribute="resource"/>
                <TimesheetEntries>
                  <sx:groupBy fields="Start Resource Task">
                    <TimesheetEntry>
                      <sx:fieldAttributeMap field="Task" attribute="task"/>
                      <DailyActuals>
                        <sx:onRecord>
                          <Actual>
                            <sx:fieldAttributeMap field="ActualDate" attribute="actualDate"/>
                            <sx:fieldAttributeMap field="Amount" attribute="amount"/>
                          </Actual>
                        </sx:onRecord>
                      </DailyActuals>
                    </TimesheetEntry>
                  </sx:groupBy>
                </TimesheetEntries>
              </Timesheet>
            </sx:groupBy>
          </Timesheets>
        </TimePeriod>
      </sx:groupBy>
    </TimePeriods>
  </sx:recordMapping>  
  
</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -r resources-timesheets.xml -o output/timesheets.xml timesheets

        

Converting a flat file with multi-valued fields to XML (family_data)

The example below shows how to convert a flat file with multi-valued fields to XML.

The input file is a pipe delimited flat file with semicolon sub-delimiters.

Figure 10. Input flat file employees.txt

          
father_name|mother_name|children
Matthew|Sarah|
Scott||Damian;Janet;Paul

        

The desired output is shown below.

Figure 11. Output XML file multivalued.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<eg:families xmlns:eg="http://examples.com/">
  <eg:family>
    <eg:father-name>Matthew</eg:father-name>
    <eg:mother-name>Sarah</eg:mother-name>
    <eg:children/>
  </eg:family>
  <eg:family>
    <eg:father-name>Scott</eg:father-name>
    <eg:mother-name/>
    <eg:children>
      <eg:child>Damian</eg:child>
      <eg:child>Janet</eg:child>
      <eg:child>Paul</eg:child>
    </eg:children>
  </eg:family>
</eg:families>

        

The following resources script does the transformation.

Figure 12. Resources script resources-multivalued.xml

          
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
                    xmlns:eg="http://examples.com/">

  <sx:service id="families">                               
    <sx:serialize>
      <sx:transform>
        <sx:content ref="families"/>
        <!-- sx:removeEmptyElements elements="eg:children"/ -->
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="families">
    <sx:flatFileReader>
      <sx:flatFile ref="family-records"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="families-to-xml-mapping"/>
  </sx:recordContent>
  
  <sx:flatFile id="family-records">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="family">
        <sx:fieldDelimiter value="|"/>
        <sx:delimitedField name="father_name"/>
        <sx:delimitedField name="mother_name"/>
        <sx:delimitedField name="children">
          <sx:subfieldDelimiter value=";"/>
        </sx:delimitedField>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>
  
  <sx:recordMapping id="families-to-xml-mapping">
    <eg:families xmlns:eg="http://examples.com/">
      <sx:onRecord>
        <eg:family>
          <sx:fieldElementMap field="father_name" element="eg:father-name"/>
          <sx:fieldElementMap field="mother_name" element="eg:mother-name"/>
          <eg:children>
            <sx:fieldElementSequenceMap field="children" element="eg:child"/>
          </eg:children>
        </eg:family>
      </sx:onRecord>
    </eg:families>
  </sx:recordMapping>
  
</sx:resources>

        

Remarks
  • Note that Matthew and Sarah have no children. If you wanted to remove the empty myns:children element, uncomment the sx:removeEmptyElements element in the sx:transform block of the resources script,

You can run this example on the command line by entering

          
servingxml -r resources-family_data.xml 
    -i data/family_data.txt -o output/family_data.xml families

        

Search and Replace with Regular Expressions in Field Mappings (ars)

The example below shows how to convert a flat file to XML using search and replace with regular expressions in field mappings.

The input file is a carot (^) delimited flat file with a header.

Figure 13. Input flat file ars.dat

          
ProjectID^Project_Name^Description^PLAN_IT_RESC_DAYS^PLAN_IT_COST^ANNUAL_EBIT^ACTUAL_NPV^ACTUAL_IRR 
1234-00-0005^XOG new project insert^This is to test and verify the XOG'ing of new Project data from the ARS database.^430^59,627^100,351^3^9.2 

        

Note the following features of this data.

  • The first row is a header row.

  • The fourth and fifth fields contain embedded commas in numbers, which we want to strip out.

The desired output is shown below.

Figure 14. Output XML file ars.xml

          
<?xml version="1.0" encoding="utf-8"?>
<NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Header action="write" externalSource="NIKU" objectType="project" version="6.0.11"/>
  <Projects>
    <Project name="XOG new project insert" projectID="1234-00-0005" description="This is to test and verify the XOG'ing of new Project data from the ARS database.">
      <CustomInformation>
        <PLAN_IT_RESOURCE_DAYS>430</PLAN_IT_RESOURCE_DAYS>
        <PLAN_IT_COST>59627</PLAN_IT_COST>
        <ANNUAL_EBIT>100351</ANNUAL_EBIT>
        <ACTUAL_NPV>3</ACTUAL_NPV>
        <ACTUAL_IRR>9.2</ACTUAL_IRR>
      </CustomInformation>
      <General addedBy="XOG" addedDate="03/26/2005 8:10:09 PM"/>
    </Project>
  </Projects>
</NikuDataBus>

The following resources script does the transformation.

Figure 15. Resources script resources-ars.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="ars">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="ars"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="ars">
    <sx:flatFileReader>
      <sx:flatFile ref="arsFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="arsToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="arsFlatFile">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="detail">
        <sx:fieldDelimiter value="^"/>
        <sx:delimitedField name="ProjectID"/>
        <sx:delimitedField name="Project_Name"/>
        <sx:delimitedField name="Description"/>
        <sx:delimitedField name="PLAN_IT_RESC_DAYS"/>
        <sx:delimitedField name="PLAN_IT_COST"/>
        <sx:delimitedField name="ANNUAL_EBIT"/>
        <sx:delimitedField name="ACTUAL_NPV"/>
        <sx:delimitedField name="ACTUAL_IRR"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="arsToXmlMapping">
    <NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

      <Header version="6.0.11" action="write" objectType="project" externalSource="NIKU"/>
      <Projects>
        <sx:groupBy fields="ProjectID">
          <Project>
            <sx:fieldAttributeMap field="Project_Name" attribute="name"/>
            <sx:fieldAttributeMap field="ProjectID" attribute="projectID"/>
            <sx:fieldAttributeMap field="Description" attribute="description"/>
            <sx:onRecord>
              <CustomInformation>
                <sx:fieldElementMap field="PLAN_IT_RESC_DAYS" element="PLAN_IT_RESOURCE_DAYS"/>
                <sx:elementMap element="PLAN_IT_COST">
                  <sx:findAndReplace searchFor ="," replaceWith ="">
                    <sx:toString value="{PLAN_IT_COST}"/>
                  </sx:findAndReplace>
                </sx:elementMap>
                <sx:elementMap element="ANNUAL_EBIT">
                  <sx:findAndReplace searchFor ="," replaceWith ="">
                    <sx:toString value="{ANNUAL_EBIT}"/>
                  </sx:findAndReplace>
                </sx:elementMap>
                <sx:fieldElementMap field="ACTUAL_NPV" element="ACTUAL_NPV"/>
                <sx:fieldElementMap field="ACTUAL_IRR" element="ACTUAL_IRR"/>
              </CustomInformation>
              <General addedBy="XOG">
                <sx:fieldAttributeMap attribute="addedDate">
                  <sx:formatDateTime format="MM/dd/yyyy h:mm:ss a">
                    <sx:currentDateTime/>
                  </sx:formatDateTime>
                </sx:fieldAttributeMap>
              </General>
            </sx:onRecord>
          </Project>
        </sx:groupBy>
      </Projects>
    </NikuDataBus>
  </sx:recordMapping>

</sx:resources>

        

Note the following points about this script.

You can run this example on the command line by entering

          
servingxml -r resources-ars.xml -i data/ars.txt -o output/ars.xml ars 

        

Converting a Tab Delimited Flat File to XML (tab-delimited)

The example below shows how to convert a flat file with tab delimiters to XML.

The input file is a tab (0x09) delimited flat file with a header.

Figure 16. Input flat file tab_delimited_employees.txt

          
employee-no	employee-name	dept	salary
00000001	Smith, Matthew	sales	150,000.00
00000002	Brown, Sarah	sales	89,000.00
00000003	Oberc, Scott	finance	110,000.00
00000004	Scott, Colette	sales	75,000.00

        

The desired output is shown below.

Figure 17. Output XML file employees.xml

          
<?xml version="1.0" encoding="utf-8"?>
<employees>
  <employee>
    <employee-no>00000001</employee-no>
    <employee-name>Smith, Matthew</employee-name>
    <department>sales</department>
    <salary>150,000.00</salary>
  </employee>
  <employee>
    <employee-no>00000002</employee-no>
    <employee-name>Brown, Sarah</employee-name>
    <department>sales</department>
    <salary>89,000.00</salary>
  </employee>
  <employee>
    <employee-no>00000003</employee-no>
    <employee-name>Oberc, Scott</employee-name>
    <department>finance</department>
    <salary>110,000.00</salary>
  </employee>
  <employee>
    <employee-no>00000004</employee-no>
    <employee-name>Scott, Colette</employee-name>
    <department>sales</department>
    <salary>75,000.00</salary>
  </employee>
</employees>

The following resources script does the transformation.

Figure 18. Resources script resources-tab_delimited_employees.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="employees">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="employee-data"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="employee-data" name="employees">
    <sx:flatFileReader>
      <sx:urlSource url="data/tab_delimited_employees.txt"/>
      <sx:flatFile ref="employee-file"/>
    </sx:flatFileReader>
  </sx:recordContent>

  <sx:flatFile id="employee-file">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:fieldDelimiter value="\t"/>
      <!-- Another way of specifying a horizontal tab character -->
      <!-- <sx:fieldDelimiter value="&#x09;"/> -->
      <sx:flatRecordType name="employee">
        <sx:delimitedField name="employee-no"/>
        <sx:delimitedField name="employee-name"/>
        <sx:delimitedField name="department"/>
        <sx:delimitedField name="salary"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

        

Note the following points about this script.

  • The tab delimiter may be specified either with the \t symbol, or, alternatively, with the numerical entity reference &#x09; numerical entity

You can run this example on the command line by entering

          
servingxml -o output/employees.xml 
    -r resources-tab_delimited_employees.xml employees 

        

Delimited Fields, Multiple Record Types

Converting a flat file with multiple record formats to XML with grouping (exotics)

The example below shows how to convert a pipe delimited flat file with multiple record formats to XM with grouping.

The input file is a pipe delimited file.

Figure 19. Input flat file exotics.txt

          
SWAP|1234567|FLOAT|PAY|CAD|BA|2010/04/28|10000000.00
Cap|1234567||CAD|SELL|2010/04/28|0.03
SWAP|1234567|FIX|REC|CAD
Cap|1234568||CAD|SELL|2010/04/28|0.05

        

The desired output is shown below.

Figure 20. Output XML file exotics.xml

          
<exotics>
  <trade tradeId="1234567">
    <option>SWAP</option>
    <swapLeg style="FLOAT">
      <side>PAY</side>
      <maturityDate original="2010/04/28">2001/04/29</maturityDate>
      <currency>CAD</currency>
      <index>BA</index>
      <notional>10000000.00</notional>
    </swapLeg>
    <capSell>
      <buySell>SELL</buySell>
      <maturityDate original="2010/04/28">2001/04/28</maturityDate>
      <strike>0.03</strike>
    </capSell>
    <swapLeg style="FIX">
      <side>REC</side>
    </swapLeg>
  </trade>
  <trade tradeId="1234568">
    <option>Cap</option>
    <capBuy>
      <buySell>BUY</buySell>
      <maturityDate original="2010/04/28">2001/04/28</maturityDate>
      <strike>0.05</strike>
    </capBuy>
  </trade>
</exotics>

        

The following resources script does the transformation.

Figure 21. Resources script resources-exotics.xml

          
<?xml version="1.0"?>

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="exotics">
	<sx:serialize>
	  <sx:transform>
		<sx:content ref="exotics"/>
	  </sx:transform>
	</sx:serialize>
  </sx:service>

  <sx:recordContent id="exotics">
	<sx:flatFileReader>
	  <sx:fileSource file="data/exotics.txt"/>
	  <sx:flatFile ref="exoticsFlatFile"/>
	</sx:flatFileReader>
	<sx:recordMapping ref="exoticsToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="exoticsFlatFile">
	<sx:flatFileBody>
	  <sx:flatRecordTypeChoice>
		<sx:fieldDelimiter value="|"/>
		<sx:delimitedField name="recordType"/>
		<sx:delimitedField name="tradeId"/>
		<sx:delimitedField name="style"/>
		<sx:when test="recordType='Cap'">
		  <sx:flatRecordType name='Cap'>
			<sx:fieldDelimiter value="|"/>
			<sx:delimitedField name="recordType"/>
			<sx:delimitedField name="tradeId"/>
			<sx:delimitedField name="placeholder1"/>
			<sx:delimitedField name="currency"/>
			<sx:delimitedField name="buySell"/>
			<sx:delimitedField name="maturityDate"/>
			<sx:delimitedField name="strike"/>
		  </sx:flatRecordType>
		</sx:when>
		<sx:when test="recordType='SWAP' and style='FLOAT'">
		  <sx:flatRecordType name='SwapFloatingLeg'>
			<sx:fieldDelimiter value="|"/>
			<sx:delimitedField name="recordType"/>
			<sx:delimitedField name="tradeId"/>
			<sx:delimitedField name="style"/>
			<sx:delimitedField name="side"/>
			<sx:delimitedField name="currency"/>
			<sx:delimitedField name="index"/>
			<sx:delimitedField name="maturityDate"/>
			<sx:delimitedField name="notional"/>
		  </sx:flatRecordType>
		</sx:when>
		<sx:when test="recordType='SWAP' and style='FIX'">
		  <sx:flatRecordType name='SwapFixedLeg'>
			<sx:fieldDelimiter value="|"/>
			<sx:delimitedField name="recordType"/>
			<sx:delimitedField name="tradeId"/>
			<sx:delimitedField name="style"/>
			<sx:delimitedField name="side"/>
			<sx:delimitedField name="placeholder1"/>
			<sx:delimitedField name="currency"/>
		  </sx:flatRecordType>
		</sx:when>
	  </sx:flatRecordTypeChoice>
	</sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="exoticsToXmlMapping">
	<exotics>
	  <sx:groupBy fields="tradeId">
		<trade><option>
		  <sx:choose>
			<sx:when test="recordType='Cap' or (recordType='SWAP' and style='FLOAT')">
			  <sx:toString value="{recordType}"/>
			</sx:when>
		  </sx:choose></option>
		  <sx:fieldAttributeMap field="tradeId" attribute="tradeId"/>
		  <sx:onRecord recordType="Cap">
			  <sx:choose>
				<sx:when test="buySell='BUY'">
		 	<capBuy>
			  <sx:fieldElementMap field="buySell" element="buySell"/>
			  <sx:elementMap element="maturityDate">
				<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
			  	<sx:choose>
			  		<sx:when test="maturityDate='2010/04/28'">
			  			<sx:toString value="2001/04/28"/>
			  		</sx:when>
			  	</sx:choose>
			  </sx:elementMap>
			  <sx:fieldElementMap field="strike" element="strike"/>
			</capBuy>
				</sx:when>
				<sx:otherwise>
		 	<capSell>
			  <sx:fieldElementMap field="buySell" element="buySell"/>
			  <sx:elementMap element="maturityDate">
				<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
			  	<sx:choose>
			  		<sx:when test="maturityDate='2010/04/28'">
			  			<sx:toString value="2001/04/28"/>
			  		</sx:when>
			  	</sx:choose>
			  </sx:elementMap>
			  <sx:fieldElementMap field="strike" element="strike"/>
			</capSell>
				</sx:otherwise>
			  </sx:choose>
		  </sx:onRecord>
		  <sx:onRecord recordType="SwapFloatingLeg">
			<swapLeg>
			  <sx:fieldAttributeMap field="style" attribute="style"/>
			  <sx:fieldElementMap field="side" element="side"/>
			  <maturityDate>
				<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
			  	<sx:choose>
			  		<sx:when test="maturityDate='2010/04/28'">
			  			<sx:toString value="2001/04/29"/>
			  		</sx:when>
			  	</sx:choose>
			  </maturityDate>
			  <sx:fieldElementMap field="currency" element="currency"/>
			  <sx:fieldElementMap field="index" element="index"/>
			  <sx:fieldElementMap field="notional" element="notional"/>
			</swapLeg>
		  </sx:onRecord>
		  <sx:onRecord recordType="SwapFixedLeg">
			<swapLeg>
			  <sx:fieldAttributeMap field="style" attribute="style"/>
			  <sx:fieldElementMap field="side" element="side"/>
			</swapLeg>
		  </sx:onRecord>
		</trade>
	  </sx:groupBy>
	</exotics>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -r resources-exotics.xml -o output/exotics.xml exotics

        

Converting flat files with record mappings that have optional elements (optional elements)

The example below shows how to convert a flat file to XML by applying an XSLT transform to the canonical XML representation of each record.

The input file is shown below.

        
262025218|John|Smith|657827168||1|Main Street|London|45879|56|||||
262091012|Jane|Bell||000000000000|105|Penny Lane|Oxford|12345|56|1|Main Street|London|45879|56

      

Each line has an id, followed by a first and last name, followed by a customer address, and optionally a billing address. If the house number and street of the billing address are empty, the rule is that the entire billing address be omitted from the XML output.

The desired output is shown below.

        
<?xml version="1.0" encoding="utf-8"?>
<mns:Message xmlns:mns="www.abc.com/abc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="www.abc.com/abc abc_v_1.1.xsd ">
   <wf_id>262025218</wf_id>
   <timeStamp>2006-02-11T22:18:21</timeStamp>
   <MessageData>
      <addCustomerRequest>
         <foreName>John</foreName>
         <surName>Smith</surName>
         <custAddress>
            <houseNumber>1</houseNumber>
            <street>Main Street</street>
            <town>London</town>
            <postCode>45879</postCode>
            <country>56</country>
         </custAddress>
      </addCustomerRequest>
   </MessageData>
   <wf_id>262091012</wf_id>
   <timeStamp>2006-02-11T22:18:21</timeStamp>
   <MessageData>
      <addCustomerRequest>
         <foreName>Jane</foreName>
         <surName>Bell</surName>
         <custAddress>
            <houseNumber>105</houseNumber>
            <street>Penny Lane</street>
            <town>Oxford</town>
            <postCode>12345</postCode>
            <country>56</country>
         </custAddress>
         <billAddress>
            <houseNumber>1</houseNumber>
            <street>Main Street</street>
            <town>London</town>
            <postCode>45879</postCode>
            <country>56</country>
         </billAddress>
      </addCustomerRequest>
   </MessageData>
</mns:Message>

      

The following resources script does the transformation.

        
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="addCustomer">
    <sx:serialize>
      <sx:xsltSerializer>
        <sx:outputProperty name="indent" value="yes"/>
      </sx:xsltSerializer>
      <sx:transform>
        <sx:content ref="addCustomer"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:flatFile id="customerFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="customer">
        <sx:fieldDelimiter value="|"/>
        <sx:delimitedField name="rowid"/>
        <sx:delimitedField name="forename"/>
        <sx:delimitedField name="surname"/>
        <sx:delimitedField name="vatnumber"/>
        <sx:delimitedField name="ninumber"/>
        <sx:delimitedField name="custaddr_house_nr"/>
        <sx:delimitedField name="custaddr_street"/>
        <sx:delimitedField name="custaddr_town"/>
        <sx:delimitedField name="custaddr_postcode"/>
        <sx:delimitedField name="custaddr_country"/>
        <sx:delimitedField name="billaddr_house_nr"/>
        <sx:delimitedField name="billaddr_street"/>
        <sx:delimitedField name="billaddr_town"/>
        <sx:delimitedField name="billaddr_postcode"/>
        <sx:delimitedField name="billaddr_country"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordContent id="addCustomer">
    <sx:flatFileReader>
      <sx:fileSource file="data/optionalElements.txt"/>
      <sx:flatFile ref="customerFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="addCustomerToXmlMapping"/>
  </sx:recordContent>

  <sx:recordMapping id="addCustomerToXmlMapping">
    <mns:Message xmlns:mns="www.abc.com/abc"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xsi:schemaLocation="www.abc.com/abc abc_v_1.1.xsd">
      <sx:onRecord>
        <sx:fieldElementMap field="rowid" element="wf_id"/>
        <timeStamp>
          <sx:formatDateTime format = "yyyy-MM-dd'T'HH:mm:ss">
            <sx:currentDateTime/>
          </sx:formatDateTime>
        </timeStamp>
        <sx:transformRecord>
          <sx:xslt>
            <xsl:transform version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
              <xsl:template match="customer">
                <MessageData>
                  <addCustomerRequest>
                    <xsl:element name="foreName">
                      <xsl:value-of select="forename"/>
                    </xsl:element>
                    <xsl:element name="surName">
                      <xsl:value-of select="surname"/>
                    </xsl:element>
                    <custAddress>
                      <xsl:element name="houseNumber">
                        <xsl:value-of select="custaddr_house_nr"/>
                      </xsl:element>
                      <xsl:element name="street">
                        <xsl:value-of select="custaddr_street"/>
                      </xsl:element>
                      <xsl:element name="town">
                        <xsl:value-of select="custaddr_town"/>
                      </xsl:element>
                      <xsl:element name="postCode">
                        <xsl:value-of select="custaddr_postcode"/>
                      </xsl:element>
                      <xsl:element name="country">
                        <xsl:value-of select="custaddr_country"/>
                      </xsl:element>
                    </custAddress>
                    <xsl:if test="string(billaddr_house_nr) and string(billaddr_street)">
                      <billAddress>
                        <xsl:element name="houseNumber">
                          <xsl:value-of select="billaddr_house_nr"/>
                        </xsl:element>
                        <xsl:element name="street">
                          <xsl:value-of select="billaddr_street"/>
                        </xsl:element>
                        <xsl:element name="town">
                          <xsl:value-of select="billaddr_town"/>
                        </xsl:element>
                        <xsl:element name="postCode">
                          <xsl:value-of select="billaddr_postcode"/>
                        </xsl:element>
                        <xsl:element name="country">
                          <xsl:value-of select="billaddr_country"/>
                        </xsl:element>
                      </billAddress>
                    </xsl:if>
                  </addCustomerRequest>
                </MessageData>
              </xsl:template>
            </xsl:transform>
          </sx:xslt>
        </sx:transformRecord>
      </sx:onRecord>
    </mns:Message>
  </sx:recordMapping>

</sx:resources>

      

You can run this example on the command line by entering

          
servingxml -r resources-optionalElements.xml  
  -o output/optionalElements.xml optionalElements

        

Converting a Java properties file to XML (messages)

The example below shows how to convert a Java properties file to XML.

The input file is a Java properties file.

Figure 22. Input property file messages.properties

          
parser.ope.2=')' or '-[' or '+[' or '&[' is expected.
parser.factor.5=A back reference or an anchor or a lookahead or a lookbehind \
is expected in a conditional pattern.
parser.next.2='?' is not expected.  '(?:' or '(?=' or '(?!' or '(?<' or '(?#' or '(?>'?
#parser.next.3='(?<=' or '(?<!' is expected.

        

This file has a number of features.

  • The entries are key-value pairs separated by the '=' symbol.

  • The value in the first line contains an '&' symbol, which is a special character in XML, and will be escaped in the output file.

  • The second line ends with a continuation character.

  • The value in the third line contains an '=' symbol.

  • The last line starts with a comment symbol.

The desired output is shown below.

Figure 23. Output XML file messages.xml

          
<?xml version="1.0" encoding="utf-8"?>
<messages>
  <parser.ope.2>')' or '-[' or '+[' or '&amp;[' is expected.</parser.ope.2>
  <parser.factor.5>A back reference or an anchor or a lookahead or a lookbehind is expected in a conditional pattern.</parser.factor.5>
  <parser.next.2>'?' is not expected.  '(?:' or '(?=' or '(?!' or '(?&lt;' or '(?#' or '(?&gt;'?</parser.next.2>
</messages>

        

The following resources script does the transformation.

Figure 24. Resources script resources-messages.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="messages">                         
    <sx:serialize>
      <sx:transform>
        <sx:content ref="messages"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>
  
  <sx:recordContent id="messages">
    <sx:flatFileReader>
      <sx:urlSource url="data/messages.properties"/>
      <sx:flatFile>
        <sx:recordDelimiter continuation="\" value="\r\n"/> 
        <sx:recordDelimiter continuation="\" value="\n"/> 
        <sx:commentStarter value="#"/>
        <sx:flatFileBody>
          <sx:flatRecordType name="property">
            <sx:delimitedField name="name">
              <sx:fieldDelimiter value="="/>
            </sx:delimitedField>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:flatFileBody>
      </sx:flatFile>
    </sx:flatFileReader>

    <sx:recordMapping>
      <messages>
        <sx:onRecord>
          <sx:fieldElementMap field="value" element="{name}"/>
        </sx:onRecord>
      </messages>
    </sx:recordMapping>
  </sx:recordContent>
  
</sx:resources>

        

Note the following points about this script.

  • The '=' delimiter is associated only with the name field, not the value field.

  • There is no delimiter specified for the value field, nor for the record as a whole. This means that the value field will contain all the text between the first '=' symbol and the record delimiter, including any more '=' symbols.

You can run this example on the command line by entering

          
servingxml -r resources-messages.xml -o output/messages.xml 
    messages

        

Transforming a Java properties file with extra name field delimiters to produce nested tags (nested tags 1)

This example shows one way to write a resources script that will convert a Java properties file to XML with nested tags.

The input file is a Java properties file.

Figure 25. Input property file multipart-keys.properties

          
image.text.info1=hello world 1 
image.text.info2=hello \
world 2 
image.text.info3=hello world 3

        

This file has a number of features.

  • The entries are key-value pairs separated by an '=' symbol.

  • Each name consists of three parts separated by a '.' symbol

  • The second line ends with a continuation character.

The desired output is shown below.

Figure 26. Output XML file nested-tags1.xml

          
<?xml version="1.0" encoding="utf-8"?>
<messages>
  <image>
    <text>
      <info1>hello world 1</info1>
      <info2>hello world 2</info2>
      <info3>hello world 3</info3>
    </text>
  </image>
</messages>

        

Note that each level of nesting corresponds to separate part of the key.

The following resources script does the transformation.

Figure 27. Resources script resources-props2nested1.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="messages">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="messages"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="messages">

    <sx:flatFileReader>
      <sx:flatFile>
        <sx:commentStarter value="#"/>
        <sx:commentStarter value="//"/>
        <sx:recordDelimiter continuation="\" value="\r\n"/>
        <sx:recordDelimiter continuation="\" value="\n"/>
        <sx:flatFileBody>
          <sx:flatRecordType name="property">
            <sx:delimitedField name="key1">
              <sx:fieldDelimiter value="."/>
            </sx:delimitedField>
            <sx:delimitedField name="key2">
              <sx:fieldDelimiter value="."/>
            </sx:delimitedField>
            <sx:delimitedField name="key3">
              <sx:fieldDelimiter value="="/>
            </sx:delimitedField>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:flatFileBody>
      </sx:flatFile>
    </sx:flatFileReader>

    <sx:recordMapping>
      <messages>
        <sx:groupBy fields="key1">
          <sx:elementMap element="{key1}">
            <sx:groupBy fields="key2 key2">
              <sx:elementMap element="{key2}">
                <sx:onRecord>
                  <sx:fieldElementMap field="value" element="{key3}"/>
                </sx:onRecord>
              </sx:elementMap>
            </sx:groupBy>
          </sx:elementMap>
        </sx:groupBy>
      </messages>
    </sx:recordMapping>

  </sx:recordContent>

</sx:resources>

        

This script relies on reading each record as four fields.

  • key1 is terminated with a '.' delimiter
  • key2 is terminated with a '.' delimiter
  • key3 is terminated with an '=' delimiter
  • value is terminated with a record delimiter

You can run this example on the command line by entering

          
servingxml -i data/multipart-keys.properties 
    -o output/nested-tags1.xml -r resources-props2nested1.xml messages

        

Transforming a Java properties file with name subfield delimiters to produce nested tags (nested tags 2)

This example shows another way to write a resources script that will convert a Java properties file to XML with nested tags.

The properties input file and the desired XML output file are the same as the previous example.

This time the resources script is written as follows.

Figure 28. Resources script resources-props2nested2.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="messages">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="messages"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="messages">

    <sx:flatFileReader>
      <sx:flatFile>
        <sx:commentStarter value="#"/>
        <sx:commentStarter value="//"/>
        <sx:recordDelimiter continuation="\" value="\r\n"/>
        <sx:recordDelimiter continuation="\" value="\n"/>
        <sx:flatFileBody>
          <sx:flatRecordType name="property">
            <sx:delimitedField name="name">
              <sx:fieldDelimiter value="="/>
              <sx:subfieldDelimiter value="."/>
            </sx:delimitedField>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:flatFileBody>
      </sx:flatFile>
    </sx:flatFileReader>

    <sx:recordMapping>
      <messages>
        <sx:groupBy fields="name[1]">
          <sx:elementMap element="{name[1]}">
            <sx:groupBy fields="name[1] name[2]">
              <sx:elementMap element="{name[2]}">
                <sx:onRecord>
                  <sx:fieldElementMap field="value" element="{name[3]}"/>
                </sx:onRecord>
              </sx:elementMap>
            </sx:groupBy>
          </sx:elementMap>
        </sx:groupBy>
      </messages>
    </sx:recordMapping>

  </sx:recordContent>

</sx:resources>

        

This script relies on reading each record as two fields, a multi-valued name field and a value field.

  • name has three values that may be indexed as 1...3
  • value has one value

You can run this example on the command line by entering

          
servingxml -i data/multipart-keys.properties 
    -o output/nested-tags2.xml -r resources-props2nested2.xml messages

        

Reordering a group of records before mapping to XML (multiple summaries)

The example below shows how to reorder a group of records before mapping them to XML.

The input file is shown below.

        
HEADER  
S 001 
S 002 
B 001
S 003 
B 002
TRAILER 

      

Each 'B' record represents a summary of the 'S' records since the last summary..

Suppose you want to turn it into the following:.

        
<?xml version="1.0" encoding="utf-8"?>
<DOCUMENT>
  <HEADER/>
  <B value="001">
    <S value="001"/>
    <S value="002"/>
  </B>
  <B value="003">
    <S value="003"/>
  </B>
  <TRAILER/>
</DOCUMENT>

      

If the summary records came before the details, rather than after, the mapping would be straightforward: see the examples with sx:innerGroup grouping elements. This suggests reordering the records as they come in.

The following resources script does the transformation.

        
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="multipleSummaries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="multipleSummariesContent"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="multipleSummariesContent">
    <sx:flatFileReader>
      <sx:urlSource url="data/multipleSummaries.txt"/>
      <sx:flatFile ref="multipleSummariesFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="multipleSummariesMapping"/>
  </sx:recordContent>

  <sx:recordMapping id="multipleSummariesMapping">
    <DOCUMENT>
      <sx:groupChoice>

        <sx:innerGroup startTest="sx:current/Header" 
                       endTest="sx:previous/Header">
          <sx:onRecord>
            <HEADER/>
          </sx:onRecord>
        </sx:innerGroup>

        <sx:innerGroup startTest="sx:current/Trailer" 
                       endTest="sx:previous/Trailer">
          <sx:onRecord>
            <TRAILER/>
          </sx:onRecord>                                        
        </sx:innerGroup>

        <sx:innerGroup startTest="sx:current/S" 
                       endTest="sx:previous/B">
          <!-- reorder the records in this group - move the B records to the front -->
          <sx:reorderRecords recordTypes="B S"/>
            <B>
              <sx:fieldAttributeMap field="value" attribute="value"/>
              <sx:onRecord recordType='S'>
                <S>
                  <sx:fieldAttributeMap field="value" attribute="value"/>
                </S>
              </sx:onRecord>
            </B>
        </sx:innerGroup>

      </sx:groupChoice>
    </DOCUMENT>
  </sx:recordMapping>

  <sx:flatFile id="multipleSummariesFile">
    <sx:flatFileBody>
      <sx:fieldDelimiter>                            
        <sx:whitespaceSeparator/>
      </sx:fieldDelimiter>
      <sx:flatRecordTypeChoice>
        <sx:delimitedField name="tag"/>
        <sx:when test="tag='HEADER'">
          <sx:flatRecordType name="Header">
            <sx:delimitedField name="tag"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='TRAILER'">
          <sx:flatRecordType name="Trailer">
            <sx:delimitedField name="tag"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='S'">
          <sx:flatRecordType name="S">
            <sx:delimitedField name="tag"/>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='B'">
          <sx:flatRecordType name="B">
            <sx:delimitedField name="tag"/>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

      

You can run this example on the command line by entering

          
servingxml -r resources-multipleSummaries.xml  
  -o output/multipleSummaries.xml multipleSummaries

        

Delimited Fields, Multiple Record Types, Repeating Groups

Mapping multiple repeating groups to XML (students-delim)

Suppose you have the following delimited file of students, their course grades, and their addresses.

Figure 29. Input flat file students-delim.txt

          
JANE^ENGL^C-~MATH^A+|1972^BLUE^CHICAGO^IL~ATLANTA^GA

        

The file has the following layout.

name the field delimiter ^ defines the end of the field
subject-grade repeating group, repeating segments are separated by the repeat delimiter ~, the segment delimiter | defines the end of the group
year-born the field delimiter ^ defines the end of the field
favorite-color the field delimiter ^ defines the end of the field
address repeating group, repeating segments are separated by the repeat delimiter ~, the segment delimiter | defines the end of the group

You want the XML output to look as follows.

Figure 30. Output XML file students-fix.xml

          
<?xml version="1.0" encoding="utf-8"?>
<StudentGrades>
  <StudentGrade>
    <Name>JANE</Name>
    <SubjectGrade>
      <Subject>ENGL</Subject>
      <Grade>C-</Grade>
    </SubjectGrade>
    <SubjectGrade>
      <Subject>MATH</Subject>
      <Grade>A+</Grade>
    </SubjectGrade>
    <YearBorn>1972</YearBorn>
    <FavoriteColor>BLUE</FavoriteColor>
    <Address>
      <City>CHICAGO</City>
      <State>IL</State>
    </Address>
    <Address>
      <City>ATLANTA</City>
      <State>GA</State>
    </Address>
  </StudentGrade>
</StudentGrades>

        

The required resources script is

Figure 31. Resources script resources-students-delim.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="students">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="students-content"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="students-content">
    <sx:flatFileReader>
      <sx:flatFile ref="students-file"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="students-mapping"/>
  </sx:recordContent>

  <sx:flatFile id="students-file">
    <sx:flatFileBody>
      <sx:flatRecordType name="student">
        <sx:fieldDelimiter value="^"/>                                              
        <sx:repeatDelimiter value="~"/>
        <sx:segmentDelimiter value="|"/>
        <sx:delimitedField name="name"/>
        <sx:repeatingGroup name="grades">
          <sx:flatRecordType name="subject-grade">
            <sx:fieldDelimiter value="^"/>                                              
            <sx:delimitedField name="subject"/>
            <sx:delimitedField name="grade"/>
          </sx:flatRecordType>
        </sx:repeatingGroup>
        <sx:delimitedField name="year-born"/>
        <sx:delimitedField name="favorite-color"/>
        <sx:repeatingGroup name="addresses">
          <sx:flatRecordType name="address">
            <sx:fieldDelimiter value="^"/>                                              
            <sx:delimitedField name="city"/>
            <sx:delimitedField name="state"/>
          </sx:flatRecordType>
        </sx:repeatingGroup>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="students-mapping">
    <StudentGrades>
      <sx:onRecord>
        <StudentGrade>
          <sx:fieldElementMap field="name" element="Name"/>
          <sx:subrecordMapping repeatingGroup="grades">
            <sx:onRecord>
              <SubjectGrade>
                <sx:fieldElementMap field="subject" element="Subject"/>
                <sx:fieldElementMap field="grade" element="Grade"/>
              </SubjectGrade>
            </sx:onRecord>
          </sx:subrecordMapping>
          <sx:fieldElementMap field="year-born" element="YearBorn"/>
          <sx:fieldElementMap field="favorite-color" element="FavoriteColor"/>
          <sx:subrecordMapping repeatingGroup="addresses">
            <sx:onRecord>
              <Address>
                <sx:fieldElementMap field="city" element="City"/>
                <sx:fieldElementMap field="state" element="State"/>
              </Address>
            </sx:onRecord>
          </sx:subrecordMapping>
        </StudentGrade>
      </sx:onRecord>
    </StudentGrades>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml  -r resources-students-delim.xml -i data/students-delim.txt -o output/students-delim.xml 
    students

        

Mapping a sequence of nested repeating groups to XML (invoice)

Suppose you have the following data file.

Figure 32. Input flat file invoice.txt

          
12|2007-02-02|DocType|} 
SndId|SndName|SndAddress|SndZip|} 
RecId|RecName|RecAddress|RecZip|} 
~ 
1|Item 1|2|Kg|150|300|} 
2|Item 2|2|Kg|350|700|} 
3|Item 3|1|$|50|50|} 
4|Item 4|10|Unt|30|100|} 
~ 
1|Ref 1|Doc 1|Text|} 
2|Ref 2|Doc 2|Text|} 
~ 
1|Disc 1|Item 1|} 
2|Disc 2|Item 2|} 
3|Disc 3|Item 3|} 
~ 
Code a|Value 1|} 
Code z|Value 3|} 
Code x|Value 2|} 
Code w|Value 2|} 
Code y|Value 1|}
~ 
!

        

The file has n sections separated by the "~" character. The first section has several non repeating records separated by a "}". But after that, there are 4 repeating sections separated by the same "~" as above. Each repeating section starts where the previous one ends.

The character "!" as the first character in the last line indicates the end of the data.

You want the XML output to look as follows.

Figure 33. Output XML file invoice.xml

          
<?xml version="1.0" encoding="utf-8"?>
<InvoiceDocument>
   <Header>
      <DocId>
         <Id>12</Id>
         <DocDate>2007-02-02</DocDate>
         <DocumentType>DocType</DocumentType>
      </DocId>
      <Sender>
         <SenderPassport>SndId</SenderPassport>
         <SenderFullName>SndName</SenderFullName>
         <SenderAddress>SndAddress</SenderAddress>
         <SenderZip>SndZip</SenderZip>
      </Sender>
      <Recipient>
         <RecipientPassport>RecId</RecipientPassport>
         <RecipientFullName>RecName</RecipientFullName>
         <RecipientAddress>RecAddress</RecipientAddress>
         <RecipientZip>RecZip</RecipientZip>
      </Recipient>
   </Header>
   <Detail>
      <Item>1</Item>
      <Description>Item 1</Description>
      <Quantity>2</Quantity>
      <Unit>Kg</Unit>
      <ItemPrice>150</ItemPrice>
      <TotalPrice>300</TotalPrice>
   </Detail>
   <Detail>
      <Item>2</Item>
      <Description>Item 2</Description>
      <Quantity>2</Quantity>
      <Unit>Kg</Unit>
      <ItemPrice>350</ItemPrice>
      <TotalPrice>700</TotalPrice>
   </Detail>
   <Detail>
      <Item>3</Item>
      <Description>Item 3</Description>
      <Quantity>1</Quantity>
      <Unit>$</Unit>
      <ItemPrice>50</ItemPrice>
      <TotalPrice>50</TotalPrice>
   </Detail>
   <Detail>
      <Item>4</Item>
      <Description>Item 4</Description>
      <Quantity>10</Quantity>
      <Unit>Unt</Unit>
      <ItemPrice>30</ItemPrice>
      <TotalPrice>100</TotalPrice>
   </Detail>
   <Reference>
      <RefId>1</RefId>
      <RefType>Ref 1</RefType>
      <RefDoc>Doc 1</RefDoc>
      <RefText>Text</RefText>
   </Reference>
   <Reference>
      <RefId>2</RefId>
      <RefType>Ref 2</RefType>
      <RefDoc>Doc 2</RefDoc>
      <RefText>Text</RefText>
   </Reference>
   <Discount>
      <DiscId>1</DiscId>
      <DiscDescription>Disc 1</DiscDescription>
      <DiscItem>Item 1</DiscItem>
   </Discount>
   <Discount>
      <DiscId>2</DiscId>
      <DiscDescription>Disc 2</DiscDescription>
      <DiscItem>Item 2</DiscItem>
   </Discount>
   <Discount>
      <DiscId>3</DiscId>
      <DiscDescription>Disc 3</DiscDescription>
      <DiscItem>Item 3</DiscItem>
   </Discount>
   <Code>
      <Code>Code a</Code>
      <Value>Value 1</Value>
   </Code>
   <Code>
      <Code>Code z</Code>
      <Value>Value 3</Value>
   </Code>
   <Code>
      <Code>Code x</Code>
      <Value>Value 2</Value>
   </Code>
   <Code>
      <Code>Code w</Code>
      <Value>Value 2</Value>
   </Code>
   <Code>
      <Code>Code y</Code>
      <Value>Value 1</Value>
   </Code>
</InvoiceDocument>

        

The required resources script is

Figure 34. Resources script resources-invoice.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="invoice">
    <sx:serialize>
      <sx:xsltSerializer>
        <sx:outputProperty name="indent" value="yes"/>
      </sx:xsltSerializer>
      <sx:transform>
        <sx:content ref="invoice-content"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="invoice-content">
    <sx:flatFileReader>
      <sx:flatFile ref="invoice-file"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="invoice-mapping"/>
  </sx:recordContent>

  <sx:flatFile id="invoice-file">
    <sx:recordDelimiter value="\r\n!"/>
    <sx:recordDelimiter value="\n!"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="invoice" omitFinalRepeatDelimiter="false">
        <sx:repeatDelimiter value="~"/>
        <sx:repeatingGroup name="Header" count="1">
          <sx:flatRecordType name="Header">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <!--sx:segmentDelimiter value="~"/-->
            <sx:repeatingGroup name="Document" count="1">
              <sx:flatRecordType name="Doc">
                <sx:delimitedField name="Id"/>
                <sx:delimitedField name="DocDate"/>
                <sx:delimitedField name="DocumentType"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
            <sx:repeatingGroup name="Sender" count="1">
              <sx:flatRecordType name="Sender">
                <sx:delimitedField name="SenderPassport"/>
                <sx:delimitedField name="SenderFullName"/>
                <sx:delimitedField name="SenderAddress"/>
                <sx:delimitedField name="SenderZip"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
            <sx:repeatingGroup name="Recipient" count="1">
              <sx:flatRecordType name="Recipient">
                <sx:delimitedField name="RecipientPassport"/>
                <sx:delimitedField name="RecipientFullName"/>
                <sx:delimitedField name="RecipientAddress"/>
                <sx:delimitedField name="RecipientZip"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>

        <sx:repeatingGroup name="Detail" count="1">
          <sx:flatRecordType name="Detail">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <sx:repeatingGroup name="Item">
              <sx:flatRecordType name="Item">
                <sx:delimitedField name="Item"/>
                <sx:delimitedField name="Description"/>
                <sx:delimitedField name="Quantity"/>
                <sx:delimitedField name="Unit"/>
                <sx:delimitedField name="ItemPrice"/>
                <sx:delimitedField name="TotalPrice"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>

        <sx:repeatingGroup name="Reference" count="1">
          <sx:flatRecordType name="Reference">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <sx:segmentDelimiter value="~"/>
            <sx:repeatingGroup name="Ref">
              <sx:flatRecordType name="Ref">
                <sx:delimitedField name="RefId"/>
                <sx:delimitedField name="RefType"/>
                <sx:delimitedField name="RefDoc"/>
                <sx:delimitedField name="RefText"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>

        <sx:repeatingGroup name="Discount" count="1">
          <sx:flatRecordType name="Discount">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <sx:segmentDelimiter value="~"/>
            <sx:repeatingGroup name="Disc">
              <sx:flatRecordType name="Disc">
                <sx:delimitedField name="DiscId"/>
                <sx:delimitedField name="DiscDescription"/>
                <sx:delimitedField name="DiscItem"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>

        <sx:repeatingGroup name="Codes" count="1">
          <sx:flatRecordType name="Codes">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <sx:segmentDelimiter value="~"/>
            <sx:repeatingGroup name="Code">
              <sx:flatRecordType name="Code">
                <sx:delimitedField name="Code"/>
                <sx:delimitedField name="Value"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="invoice-mapping">
    <InvoiceDocument>
      <sx:onRecord>
        <sx:subrecordMapping repeatingGroup="Header">
          <sx:onRecord>
            <Header>
              <sx:subrecordMapping repeatingGroup="Document">
                <sx:onRecord>
                  <DocId>
                    <sx:fieldElementMap field="Id" element="Id"/>
                    <sx:fieldElementMap field="DocDate" element="DocDate"/>
                    <sx:fieldElementMap field="DocumentType" element="DocumentType"/>
                  </DocId>
                </sx:onRecord>
              </sx:subrecordMapping>
              <sx:subrecordMapping repeatingGroup="Sender">
                <sx:onRecord>
                  <Sender>
                    <sx:fieldElementMap field="SenderPassport" element="SenderPassport"/>
                    <sx:fieldElementMap field="SenderFullName" element="SenderFullName"/>
                    <sx:fieldElementMap field="SenderAddress" element="SenderAddress"/>
                    <sx:fieldElementMap field="SenderZip" element="SenderZip"/>
                  </Sender>
                </sx:onRecord>
              </sx:subrecordMapping>
              <sx:subrecordMapping repeatingGroup="Recipient">
                <sx:onRecord>
                  <Recipient>
                    <sx:fieldElementMap field="RecipientPassport" element="RecipientPassport"/>
                    <sx:fieldElementMap field="RecipientFullName" element="RecipientFullName"/>
                    <sx:fieldElementMap field="RecipientAddress" element="RecipientAddress"/>
                    <sx:fieldElementMap field="RecipientZip" element="RecipientZip"/>
                  </Recipient>
                </sx:onRecord>
              </sx:subrecordMapping>
            </Header>
          </sx:onRecord>
        </sx:subrecordMapping>

        <sx:subrecordMapping repeatingGroup="Detail">
          <sx:onRecord>
            <sx:subrecordMapping repeatingGroup="Item">
              <sx:onRecord>
                <Detail>
                  <sx:fieldElementMap field="Item" element="Item"/>
                  <sx:fieldElementMap field="Description" element="Description"/>
                  <sx:fieldElementMap field="Quantity" element="Quantity"/>
                  <sx:fieldElementMap field="Unit" element="Unit"/>
                  <sx:fieldElementMap field="ItemPrice" element="ItemPrice"/>
                  <sx:fieldElementMap field="TotalPrice" element="TotalPrice"/>
                </Detail>
              </sx:onRecord>
            </sx:subrecordMapping>
          </sx:onRecord>
        </sx:subrecordMapping>

        <sx:subrecordMapping repeatingGroup="Reference">
          <sx:onRecord>
            <sx:subrecordMapping repeatingGroup="Ref">
              <sx:onRecord>
                <Reference>
                  <sx:fieldElementMap field="RefId" element="RefId"/>
                  <sx:fieldElementMap field="RefType" element="RefType"/>
                  <sx:fieldElementMap field="RefDoc" element="RefDoc"/>
                  <sx:fieldElementMap field="RefText" element="RefText"/>
                </Reference>
              </sx:onRecord>
            </sx:subrecordMapping>
          </sx:onRecord>
        </sx:subrecordMapping>

        <sx:subrecordMapping repeatingGroup="Discount">
          <sx:onRecord>
            <sx:subrecordMapping repeatingGroup="Disc">
              <sx:onRecord>
                <Discount>
                  <sx:fieldElementMap field="DiscId" element="DiscId"/>
                  <sx:fieldElementMap field="DiscDescription" element="DiscDescription"/>
                  <sx:fieldElementMap field="DiscItem" element="DiscItem"/>
                </Discount>
              </sx:onRecord>
            </sx:subrecordMapping>
          </sx:onRecord>
        </sx:subrecordMapping>

        <sx:subrecordMapping repeatingGroup="Codes">
          <sx:onRecord>
            <sx:subrecordMapping repeatingGroup="Code">
              <sx:onRecord>
                <Code>
                  <sx:fieldElementMap field="Code" element="Code"/>
                  <sx:fieldElementMap field="Value" element="Value"/>
                </Code>
              </sx:onRecord>
            </sx:subrecordMapping>
          </sx:onRecord>
        </sx:subrecordMapping>
      </sx:onRecord>
    </InvoiceDocument>
  </sx:recordMapping>

</sx:resources>

        

The idea is to map all the data up to the terminating "!" as one record, and to structure the record as a sequence of nested repeating groups.

You can run this example on the command line by entering

          
servingxml  -r resources-invoice.xml -i data/invoice.txt -o output/invoice.xml 
    invoice

        

EDI Examples

EDI to XML (comptest)

The example below shows how to convert a composite field in an edi file to XML.

The input file is an edi file.

Figure 35. Input edi file comptest.txt

          
BKG+1++4+Y:1:B:0:M:0:L:1'

        

This file has a number of features.

  • The segment delimiter "'" terminates the segment.

  • The field delimiter "+" terminates a field.

  • Composite fields are delimited with ":".

The desired output is shown below.

Figure 36. Output XML file messages.xml

          
<?xml version="1.0" encoding="utf-8"?>
<edifact>
  <segment segmentType="BKG">
    <flightSegmentNumber>1</flightSegmentNumber>
    <accessID/>
    <bookingCodeCount>4</bookingCodeCount>
    <C704>
      <bookingCode>Y</bookingCode>
      <seatAvailabilityCount>1</seatAvailabilityCount>
    </C704>
    <C704>
      <bookingCode>B</bookingCode>
      <seatAvailabilityCount>0</seatAvailabilityCount>
    </C704>
    <C704>
      <bookingCode>M</bookingCode>
      <seatAvailabilityCount>0</seatAvailabilityCount>
    </C704>
    <C704>
      <bookingCode>L</bookingCode>
      <seatAvailabilityCount>1</seatAvailabilityCount>
    </C704>
  </segment>
</edifact>

        

The following resources script does the transformation.

Figure 37. Resources script resources-comptest.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="edifact">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="edifact"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="edifact">
    <sx:flatFileReader>
      <sx:flatFile ref="edifactFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="edifactToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="edifactFlatFile">
    <sx:recordDelimiter value="'"/>
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:fieldDelimiter value="+"/>
        <sx:delimitedField name="segmentType"/>
        <!-- edifact3 - Body Segments.-->
        <!-- BKG BOOKING CODE DATA .-->
        <sx:when test="segmentType='BKG'">
          <sx:flatRecordType name='BKG'>
            <sx:fieldDelimiter value="+"/>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="flightSegmentNumber"/>
            <sx:delimitedField name="accessID"/>
            <sx:delimitedField name="bookingCodeCount"/>
            <sx:repeatingGroup name="booking" count="{bookingCodeCount}">
              <sx:flatRecordType name="C704">
                <sx:fieldDelimiter value=":"/>
                <sx:delimitedField name="bookingCode">
                </sx:delimitedField>
                <sx:delimitedField name="seatAvailabilityCount">
                </sx:delimitedField>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="edifactToXmlMapping">
    <edifact>
      <!-- edifact3 - Body Segments.-->
      <!-- BKG BOOKING CODE DATA.-->
      <sx:onRecord recordType="BKG">
        <sx:elementMap element="segment">
          <sx:fieldAttributeMap field="segmentType" attribute="segmentType"/>
          <sx:fieldElementMap field="flightSegmentNumber" element="flightSegmentNumber"/>
          <sx:fieldElementMap field="accessID" element="accessID"/>
          <sx:fieldElementMap field="bookingCodeCount" element="bookingCodeCount"/>
          <sx:subrecordMapping repeatingGroup="booking">
            <sx:onRecord recordType="C704">
              <C704>
                <sx:fieldElementMap field="bookingCode" element="bookingCode"/>
                <sx:fieldElementMap field="seatAvailabilityCount" element="seatAvailabilityCount"/>
              </C704>
            </sx:onRecord>
          </sx:subrecordMapping>
        </sx:elementMap>
      </sx:onRecord>
    </edifact>
  </sx:recordMapping>

</sx:resources>

        

Note the following points about this script.

  • All the values of the composite field are put in the multiple valued field "C704".

  • The sx:repeatingGroup element allows us to map these values, repeated two at a time, named as bookingCode and seatAvailabilityCount .

You can run this example on the command line by entering

          
servingxml -i data/comptest.txt -r resources-comptest.xml 
  -o output/comptest.xml edifact

        

EDI to XML with Escaped Delimiters (Invoic96A)

The example below shows how to convert a composite field in an edi file to XML.

The input file is an edi file.

Figure 38. Input edi file invoic96A.txt

          
UNB+UNOA:3+0000008033253:14+01534730278:ZZ+080130:0000+04000020++++++1'
UNH+1+INVOIC:D:96A:UN:EAN008'
BGM+381+1000322+9'
DTM+137:20080130:102'
NAD+SU+++COMPANY XXXX ?+ YYYY+VIA ROMA, 1+CITTA?' DI?: GENOVA+GE+16163+IT'
RFF+VA:05577790207'
CUX+2:EUR:4+3'
PAT+10E+ZZZ:::BONIFICO BANCARIO 90 GG D.F.+5:1:D:090'
DTM+13:20080430:102'
LIN+1+L02+8026495011408:EN'
IMD+B++TU:::BAGUETTE PREC. GR.280 40PZ'
QTY+47:5.000:PCE'
QTY+59:40.000:PCE'
MOA+203:54.130:EUR:4'
PRI+AAA:10.830::::PCE'
UNS+S'
MOA+125:54.130'
UNT+46+1'
UNZ+1+04000020'

        

This file has a number of features.

  • Records are terminated with '

  • Fields are terminated with +

  • Subfields are terminated with :

  • The character ? is used as an escape character. For example, COMPANY?: XXXX?+YYYY+ will be interpreted as: COMPANY: XXXX+YYYY

  • Sometimes the input file could be formatted with a fixed 80 column width. Linefeed characters should be stripped.

The desired output is shown below.

Figure 39. Output XML file invoic96A.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<Invoic96A>
   <UNB>
      <segmentType>UNB</segmentType>
   </UNB>
   <UNH>
      <segmentType>UNH</segmentType>
      <DE0062>1</DE0062>
      <S009>
         <S009>
            <DE0065>INVOIC</DE0065>
            <DE0052>D</DE0052>
            <DE0054>96A</DE0054>
            <DE0051>UN</DE0051>
            <DE0057>EAN008</DE0057>
         </S009>
      </S009>
      <DE0068/>
   </UNH>
   <BGM>
      <segmentType>BGM</segmentType>
      <C002>
         <C002>
            <DE1001>381</DE1001>
            <DE1131/>
            <DE3055/>
            <DE1000/>
         </C002>
      </C002>
      <DE1004>1000322</DE1004>
      <DE1225>9</DE1225>
      <DE4343/>
   </BGM>
   <DTM>
      <segmentType>DTM</segmentType>
      <C507>
         <C507>
            <DE2005>137</DE2005>
            <DE2380>20080130</DE2380>
            <DE2379>102</DE2379>
         </C507>
      </C507>
   </DTM>
   <NAD>
      <segmentType>NAD</segmentType>
      <DE3035>SU</DE3035>
      <C080>
         <C080>
            <DE3036-1>COMPANY XXXX + YYYY</DE3036-1>
            <DE3036-2/>
            <DE3036-3/>
            <DE3036-4/>
            <DE3036-5/>
            <DE3045/>
         </C080>
      </C080>
      <C059>
         <C059>
            <DE3042-1>VIA ROMA, 1</DE3042-1>
            <DE3042-2/>
            <DE3042-3/>
            <DE3042-4/>
         </C059>
      </C059>
      <DE3164>CITTA' DI: GENOVA</DE3164>
      <DE3229>GE</DE3229>
      <DE3251>16163</DE3251>
      <DE3207>IT</DE3207>
   </NAD>
   <RFF>
      <segmentType>RFF</segmentType>
      <C506>
         <C506>
            <DE1153>VA</DE1153>
            <DE1154>05577790207</DE1154>
            <DE1156/>
            <DE4000/>
         </C506>
      </C506>
   </RFF>
   <CUX>
      <segmentType>CUX</segmentType>
      <C504>
         <C504>
            <DE6347>2</DE6347>
            <DE6345>EUR</DE6345>
            <DE6343>4</DE6343>
            <DE6348/>
         </C504>
         <C504>
            <DE6347>3</DE6347>
            <DE6345/>
            <DE6343/>
            <DE6348/>
         </C504>
      </C504>
      <DE5402/>
      <DE6341/>
   </CUX>
   <PAT>
      <segmentType>PAT</segmentType>
      <DE4279>10E</DE4279>
      <C110>
         <C110>
            <DE4277>ZZZ</DE4277>
            <DE1131/>
            <DE3055/>
            <DE4276-1>BONIFICO BANCARIO 90 GG D.F.</DE4276-1>
            <DE4276-2/>
         </C110>
      </C110>
      <C112>
         <C112>
            <DE2475>5</DE2475>
            <DE2009>1</DE2009>
            <DE2151>D</DE2151>
            <DE2152>090</DE2152>
         </C112>
      </C112>
   </PAT>
   <DTM>
      <segmentType>DTM</segmentType>
      <C507>
         <C507>
            <DE2005>13</DE2005>
            <DE2380>20080430</DE2380>
            <DE2379>102</DE2379>
         </C507>
      </C507>
   </DTM>
   <LIN>
      <segmentType>LIN</segmentType>
      <DE1082>1</DE1082>
      <DE1229>L02</DE1229>
      <C212>
         <C212>
            <DE7140>8026495011408</DE7140>
            <DE7143>EN</DE7143>
            <DE1131/>
            <DE3055/>
         </C212>
      </C212>
      <DE1222/>
      <DE7083/>
   </LIN>
   <IMD>
      <segmentType>IMD</segmentType>
      <DE7077>B</DE7077>
      <DE7081/>
      <C273>
         <C273>
            <DE7009>TU</DE7009>
            <DE1131/>
            <DE3055/>
            <DE7008-1>BAGUETTE PREC. GR.280 40PZ</DE7008-1>
            <DE7008-2/>
            <DE3435/>
         </C273>
      </C273>
      <DE7383/>
   </IMD>
   <QTY>
      <segmentType>QTY</segmentType>
      <C186>
         <C186>
            <DE6063>47</DE6063>
            <DE6060>5.000</DE6060>
            <DE6411>PCE</DE6411>
         </C186>
      </C186>
   </QTY>
   <QTY>
      <segmentType>QTY</segmentType>
      <C186>
         <C186>
            <DE6063>59</DE6063>
            <DE6060>40.000</DE6060>
            <DE6411>PCE</DE6411>
         </C186>
      </C186>
   </QTY>
   <MOA>
      <segmentType>MOA</segmentType>
      <C516>
         <C516>
            <DE5025>203</DE5025>
            <DE5004>54.130</DE5004>
            <DE6345>EUR</DE6345>
            <DE6343>4</DE6343>
            <DE4405/>
         </C516>
      </C516>
   </MOA>
   <PRI>
      <segmentType>PRI</segmentType>
      <C509>
         <C509>
            <DE5125>AAA</DE5125>
            <DE5118>10.830</DE5118>
            <DE5375/>
            <DE5387/>
            <DE5284/>
            <DE6411>PCE</DE6411>
         </C509>
      </C509>
      <DE5213/>
   </PRI>
   <UNS>
      <segmentType>UNS</segmentType>
      <DE0081>S</DE0081>
   </UNS>
   <MOA>
      <segmentType>MOA</segmentType>
      <C516>
         <C516>
            <DE5025>125</DE5025>
            <DE5004>54.130</DE5004>
            <DE6345/>
            <DE6343/>
            <DE4405/>
         </C516>
      </C516>
   </MOA>
   <UNT>
      <segmentType>UNT</segmentType>
      <DE0074>46</DE0074>
      <DE0062>1</DE0062>
   </UNT>
   <UNZ>
      <segmentType>UNZ</segmentType>
      <F0036>1</F0036>
      <F0020>04000020</F0020>
   </UNZ>
</Invoic96A>

        

The following resources script does the transformation.

Figure 40. Resources script resources-invoic96A.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv"
              xmlns:fn="http://www.w3.org/2005/xpath-functions">

  <sx:service id="invoic96A-to-xml">
    <sx:serialize>
      <sx:xsltSerializer>
        <sx:outputProperty name="indent" value="yes"/>
      </sx:xsltSerializer>
      <sx:transform>
        <sx:content ref="Invoic_96A_1"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="Invoic_96A_1" name="Invoic96A">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="edifactFlatFile"/>
      </sx:flatFileReader>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
      </sx:discardHandler>
    </sx:recordStream>
  </sx:recordContent>
	
  <sx:flatFile id="edifactFlatFile">
    <sx:recordDelimiter value="'" escapeCharacter="?"/>
    <!-- If a line is broken with a new line character, strip the new line character -->
    <sx:recordDelimiter value="\r\n" continuationSequence="\r\n"/>
    <sx:recordDelimiter value="\n" continuationSequence="\n"/>
    <!-- if a ' character is found after a ? (?')-->
    <!-- it is an escape sequence and should remain the ' character-->
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:fieldDelimiter value="+" escapeCharacter="?"/>
        <sx:delimitedField name="segmentType"/>

        <sx:when test="segmentType='UNA' or segmentType='UNA:'">
          <sx:flatRecordType name='UNA'>
            <sx:fieldDelimiter value="+" escapeCharacter="?"/>
            <sx:fieldDelimiter value=":" escapeCharacter="?"/>
            <sx:delimitedField name="segmentType"/>
          </sx:flatRecordType>
        </sx:when>

        <sx:when test="segmentType='UNB'">
          <sx:flatRecordType name='UNB'>
            <sx:fieldDelimiter value="+" escapeCharacter="?"/>
            <sx:fieldDelimiter value=":" escapeCharacter="?"/>
            <sx:delimitedField name="segmentType"/>
          </sx:flatRecordType>
        </sx:when>

        <sx:when test="segmentType='UNH'">
          <sx:flatRecordType name='UNH'>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE0062"/>
            <sx:nonrepeatingGroup name="S009">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="S009">
                <sx:delimitedField name="DE0065"/>
                <sx:delimitedField name="DE0052"/>
                <sx:delimitedField name="DE0054"/>
                <sx:delimitedField name="DE0051"/>
                <sx:delimitedField name="DE0057"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE0068"/>
            <sx:nonrepeatingGroup name="S010">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="S010">
                <sx:delimitedField name="DE0070"/>
                <sx:delimitedField name="DE0073"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='BGM'">
          <sx:flatRecordType name='BGM'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C002">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C002">
                <sx:delimitedField name="DE1001"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
                <sx:delimitedField name="DE1000"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE1004"/>
            <sx:delimitedField name="DE1225"/>
            <sx:delimitedField name="DE4343"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='DTM'">
          <sx:flatRecordType name='DTM'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C507">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C507">
                <sx:delimitedField name="DE2005"/>
                <sx:delimitedField name="DE2380"/>
                <sx:delimitedField name="DE2379"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='PAI'">
          <sx:flatRecordType name='PAI'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C534">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C534">
                <sx:delimitedField name="DE4439"/>
                <sx:delimitedField name="DE4431"/>
                <sx:delimitedField name="DE4461"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
                <sx:delimitedField name="DE4435"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='FTX'">
          <sx:flatRecordType name='FTX'>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE4451"/>
            <sx:delimitedField name="DE4453"/>
            <sx:nonrepeatingGroup name="C107">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C107">
                <sx:delimitedField name="DE4441"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:nonrepeatingGroup name="C108">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C108">
                <sx:delimitedField name="DE4440-1"/>
                <sx:delimitedField name="DE4440-2"/>
                <sx:delimitedField name="DE4440-3"/>
                <sx:delimitedField name="DE4440-4"/>
                <sx:delimitedField name="DE4440-5"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE3453"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='NAD'">
          <sx:flatRecordType name='NAD'>
            <sx:fieldDelimiter value="+" escapeCharacter="?"/>
            <sx:fieldDelimiter value=":" escapeCharacter="?"/>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE3035"/>
            <sx:nonrepeatingGroup name="C082">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C082">
                <sx:delimitedField name="DE3039"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:nonrepeatingGroup name="C058">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C058">
                <sx:delimitedField name="DE3124-1"/>
                <sx:delimitedField name="DE3124-2"/>
                <sx:delimitedField name="DE3124-3"/>
                <sx:delimitedField name="DE3124-4"/>
                <sx:delimitedField name="DE3124-5"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:nonrepeatingGroup name="C080">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C080">
                <sx:delimitedField name="DE3036-1"/>
                <sx:delimitedField name="DE3036-2"/>
                <sx:delimitedField name="DE3036-3"/>
                <sx:delimitedField name="DE3036-4"/>
                <sx:delimitedField name="DE3036-5"/>
                <sx:delimitedField name="DE3045"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:nonrepeatingGroup name="C059">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C059">
                <sx:delimitedField name="DE3042-1"/>
                <sx:delimitedField name="DE3042-2"/>
                <sx:delimitedField name="DE3042-3"/>
                <sx:delimitedField name="DE3042-4"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE3164"/>
            <sx:delimitedField name="DE3229"/>
            <sx:delimitedField name="DE3251"/>
            <sx:delimitedField name="DE3207"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='FII'">
          <sx:flatRecordType name='FII'>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE3035"/>
            <sx:nonrepeatingGroup name="C078">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C078">
                <sx:delimitedField name="DE3194"/>
                <sx:delimitedField name="DE3192-1"/>
                <sx:delimitedField name="DE3192-2"/>
                <sx:delimitedField name="DE6345"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:nonrepeatingGroup name="C088">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C088">
                <sx:delimitedField name="DE3433"/>
                <sx:delimitedField name="DE1131-1"/>
                <sx:delimitedField name="DE3055-1"/>
                <sx:delimitedField name="DE3434"/>
                <sx:delimitedField name="DE1131-2"/>
                <sx:delimitedField name="DE3055-2"/>
                <sx:delimitedField name="DE3432"/>
                <sx:delimitedField name="DE3436"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE3207"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='RFF'">
          <sx:flatRecordType name='RFF'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C506">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C506">
                <sx:delimitedField name="DE1153"/>
                <sx:delimitedField name="DE1154"/>
                <sx:delimitedField name="DE1156"/>
                <sx:delimitedField name="DE4000"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='CTA'">
          <sx:flatRecordType name='CTA'>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE3139"/>
            <sx:nonrepeatingGroup name="C056">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C056">
                <sx:delimitedField name="DE3413"/>
                <sx:delimitedField name="DE3412"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='COM'">
          <sx:flatRecordType name='COM'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C076">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C076">
                <sx:delimitedField name="DE3148"/>
                <sx:delimitedField name="DE3155"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType> 
        </sx:when>
        <sx:when test="segmentType='CUX'">
          <sx:flatRecordType name='CUX'>
            <sx:delimitedField name="segmentType"/>
            <sx:repeatingGroup name="C504" count="2">
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:repeatDelimiter value="+" escapeCharacter="?"/>
              <!--<sx:segmentDelimiter value="+" escapeCharacter="?"/>-->
              <sx:flatRecordType name="C504">
                <sx:delimitedField name="DE6347"/>
                <sx:delimitedField name="DE6345"/>
                <sx:delimitedField name="DE6343"/>
                <sx:delimitedField name="DE6348"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
            <sx:delimitedField name="DE5402"/>
            <sx:delimitedField name="DE6341"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='PAT'">
          <sx:flatRecordType name='PAT'>
            <sx:fieldDelimiter value="+" escapeCharacter="?"/>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE4279"/>
            <sx:nonrepeatingGroup name="C110">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C110">
                <sx:delimitedField name="DE4277"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
                <sx:delimitedField name="DE4276-1"/>
                <sx:delimitedField name="DE4276-2"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:nonrepeatingGroup name="C112">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C112">
                <sx:delimitedField name="DE2475"/>
                <sx:delimitedField name="DE2009"/>
                <sx:delimitedField name="DE2151"/>
                <sx:delimitedField name="DE2152"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='PCD'">
          <sx:flatRecordType name='PCD'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C501">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C501">
                <sx:delimitedField name="DE5245"/>
                <sx:delimitedField name="DE5482"/>
                <sx:delimitedField name="DE5249"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='MOA'">
          <sx:flatRecordType name='MOA'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C516">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C516">
                <sx:delimitedField name="DE5025"/>
                <sx:delimitedField name="DE5004"/>
                <sx:delimitedField name="DE6345"/>
                <sx:delimitedField name="DE6343"/>
                <sx:delimitedField name="DE4405"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='ALC'">
          <sx:flatRecordType name='ALC'>
            <sx:fieldDelimiter value="+" escapeCharacter="?"/>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE5463"/>
            <sx:nonrepeatingGroup name="C552">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C552">
                <sx:delimitedField name="DE1230"/>
                <sx:delimitedField name="DE5189"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE4471"/>
            <sx:delimitedField name="DE1227"/>
            <sx:nonrepeatingGroup name="C214">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C214">
                <sx:delimitedField name="DE7161"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
                <sx:delimitedField name="DE7160-1"/>
                <sx:delimitedField name="DE7160-2"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <!-- Detail segments -->
        <sx:when test="segmentType='LIN'">
          <sx:flatRecordType name='LIN'>
            <sx:fieldDelimiter value="+" escapeCharacter="?"/>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE1082"/>
            <sx:delimitedField name="DE1229"/>
            <sx:nonrepeatingGroup name="C212">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C212">
                <sx:delimitedField name="DE7140"/>
                <sx:delimitedField name="DE7143"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:nonrepeatingGroup name="C829">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C829">
                <sx:delimitedField name="DE5495"/>
                <sx:delimitedField name="DE1082"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE1222"/>
            <sx:delimitedField name="DE7083"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='PIA'">
          <sx:flatRecordType name='PIA'>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE4347"/>
            <sx:nonrepeatingGroup name="C212">
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <!--<sx:segmentDelimiter value="+" escapeCharacter="?"/>-->
              <sx:flatRecordType name="C212">
                <sx:delimitedField name="DE7140"/>
                <sx:delimitedField name="DE7143"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='IMD'">
          <sx:flatRecordType name='IMD'>
            <sx:fieldDelimiter value="+" escapeCharacter="?"/>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE7077"/>
            <sx:delimitedField name="DE7081"/>
            <sx:nonrepeatingGroup name="C273">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C273">
                <sx:delimitedField name="DE7009"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
                <sx:delimitedField name="DE7008-1"/>
                <sx:delimitedField name="DE7008-2"/>
                <sx:delimitedField name="DE3435"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE7383"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='QTY'">
          <sx:flatRecordType name='QTY'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C186">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C186">
                <sx:delimitedField name="DE6063"/>
                <sx:delimitedField name="DE6060"/>
                <sx:delimitedField name="DE6411"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='PRI'">
          <sx:flatRecordType name='PRI'>
            <sx:delimitedField name="segmentType"/>
            <sx:nonrepeatingGroup name="C509">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C509">
                <sx:delimitedField name="DE5125"/>
                <sx:delimitedField name="DE5118"/>
                <sx:delimitedField name="DE5375"/>
                <sx:delimitedField name="DE5387"/>
                <sx:delimitedField name="DE5284"/>
                <sx:delimitedField name="DE6411"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE5213"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='TAX'">
          <sx:flatRecordType name='TAX'>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE5283"/>
            <sx:nonrepeatingGroup name="C241">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C241">
                <sx:delimitedField name="DE5153"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
                <sx:delimitedField name="DE5152"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:nonrepeatingGroup name="C533">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C533">
                <sx:delimitedField name="DE5289"/>
                <sx:delimitedField name="DE1131"/>
                <sx:delimitedField name="DE3055"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE5286"/>
            <sx:nonrepeatingGroup name="C243">
              <sx:segmentDelimiter value="+" escapeCharacter="?"/>
              <sx:fieldDelimiter value=":" escapeCharacter="?"/>
              <sx:flatRecordType name="C243">
                <sx:delimitedField name="DE5279"/>
                <sx:delimitedField name="DE1131-1"/>
                <sx:delimitedField name="DE3055-1"/>
                <sx:delimitedField name="DE5278"/>
                <sx:delimitedField name="DE5273"/>
                <sx:delimitedField name="DE1131-2"/>
                <sx:delimitedField name="DE3055-2"/>
              </sx:flatRecordType>
            </sx:nonrepeatingGroup>
            <sx:delimitedField name="DE5305"/>
            <sx:delimitedField name="DE3446"/>
          </sx:flatRecordType>
        </sx:when>
		
        <!-- Trailer segments -->
        <sx:when test="segmentType='UNS'">
          <sx:flatRecordType name='UNS'>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE0081"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='UNT'">
          <sx:flatRecordType name='UNT'>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="DE0074"/>
            <sx:delimitedField name="DE0062"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="segmentType='UNZ'">
          <sx:flatRecordType name='UNZ'>
            <sx:fieldDelimiter value="+" escapeCharacter="?"/>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="F0036"/>
            <sx:delimitedField name="F0020"/>
          </sx:flatRecordType>
        </sx:when>

        <sx:otherwise>
          <sx:flatRecordType name='ERROR'>
            <sx:delimitedField name="segmentType"/>
          </sx:flatRecordType>
        </sx:otherwise>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

Note the following points about this script.

  • Record delimiters delimit the boundaries of a record, segment delimiters delimit areas within a record, and repeat delimiters separate subrecords from each other within segments. An sx:repeatingGroup reads the subrecords separated by repeat delimiters within a segment. A special case is when there are a number of segments, but at most one subrecord per segment, a nonrepeating group. An sx:nonrepeatingGroup reads the single subrecord, if any.

  • The continuation sequence for a linefeed is defined be the linefeed itself, e.g.

        <sx:recordDelimiter value="\r\n" continuationSequence="\r\n"/>
        <sx:recordDelimiter value="\n" continuationSequence="\n"/>
    

    This has the effect of stripping out the linefeed characters.

You can run this example on the command line by entering

          
servingxml -r resources-invoic96A.xml -i data/invoic96A.txt 
    -o output/invoic96A.xml invoic96A-to-xml
    
        

FpML Examples

FRA to FpML

Suppose "Bank A" has the following CSV file of FRA trades. Each FRA trade is represented by one record in the file.

Figure 41. Input flat file fra.csv

          
product,trade_id,client_id,client_name,trade_date,buy_sell_code,currency,index,tenor,day_count_basis,fixed_rate,start_date,maturity_date,payment_date,notional
FRA,1001,party1,Party 1,2006/12/07,BUY,CAD,BA,3M,ACT/365,0.04,   2008/09/08,2008/12/08,2008/09/08,250000000
FRA,1002,party1,Party 1,2007/03/19,B,CAD,BA,3M,ACT/365,0.04,   2008/09/15,2008/12/15,2008/09/15,110000000
FRA,1003,party1,Party 1,2007/03/19,SELL,CAD,BA,3M,ACT/365,0.04,  2008/12/15,2009/03/16,2008/12/15,70000000 
FRA,1004,party2,Party 2,2007/05/02,BUY,USD,LIBOR,3M,ACT/360,0.04,2009/02/02,2009/05/04,2009/02/02,100000000
FRA,1005,party2,Party 2,2007/05/02,BUY,USD,LIBOR,3M,ACT/360,0.04,2009/02/02,2009/05/04,2009/02/02,100000000
FRA,1006,party3,Party 3,2007/08/01,B,USD,LIBOR,3M,ACT/360,0.05,09/05/01,2009/08/04,2009/05/01,50000000 

        

Note that two of the FRA trades have errors. The trade with trade_id 1002 has an invalid buy_sell_code, "B" instead of "BUY". The trade with trade id 1006 has the same problem with the buy_sell_code, and also has an invalid start_date, with a two digit year instead of a four digit year. Bank A wants to reject the trades with errors and write them along with an error message to a discard file.

Bank A wants the XML output to look as follows.

Figure 42. Output XML file fra.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<dataDocument xmlns="http://www.fpml.org/FpML-5-0/reporting" 
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" fpmlVersion="5-0" 
              xsi:schemaLocation="http://www.fpml.org/FpML-5-0/reporting ../fpml-main-5-0.xsd">
  <trade>
    <tradeHeader>
      <partyTradeIdentifier>
        <tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">1001</tradeId>
      </partyTradeIdentifier>
      <tradeDate>2006-12-07</tradeDate>
    </tradeHeader>
    <fra>
      <sellerPartyReference>party1</sellerPartyReference>
      <adjustedEffectiveDate>2008-09-08</adjustedEffectiveDate>
      <adjustedTerminationDate>2008-12-08</adjustedTerminationDate>
      <paymentDate>
        <adjustedDate>2008-09-08</adjustedDate>
      </paymentDate>
      <dayCountFraction>ACT/365</dayCountFraction>
      <notional>
        <currency>CAD</currency>
        <amount>250000000</amount>
      </notional>
      <fixedRate>0.04</fixedRate>
      <floatingRateIndex>BA</floatingRateIndex>
      <indexTenor>
        <periodMultiplier>3</periodMultiplier>
        <period>M</period>
      </indexTenor>
    </fra>
  </trade>
  <trade>
    <tradeHeader>
      <partyTradeIdentifier>
        <tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">1003</tradeId>
      </partyTradeIdentifier>
      <tradeDate>2007-03-19</tradeDate>
    </tradeHeader>
    <fra>
      <buyerPartyReference>party1</buyerPartyReference>
      <adjustedEffectiveDate>2008-12-15</adjustedEffectiveDate>
      <adjustedTerminationDate>2009-03-16</adjustedTerminationDate>
      <paymentDate>
        <adjustedDate>2008-12-15</adjustedDate>
      </paymentDate>
      <dayCountFraction>ACT/365</dayCountFraction>
      <notional>
        <currency>CAD</currency>
        <amount>70000000</amount>
      </notional>
      <fixedRate>0.04</fixedRate>
      <floatingRateIndex>BA</floatingRateIndex>
      <indexTenor>
        <periodMultiplier>3</periodMultiplier>
        <period>M</period>
      </indexTenor>
    </fra>
  </trade>
  <trade>
    <tradeHeader>
      <partyTradeIdentifier>
        <tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">1004</tradeId>
      </partyTradeIdentifier>
      <tradeDate>2007-05-02</tradeDate>
    </tradeHeader>
    <fra>
      <sellerPartyReference>party2</sellerPartyReference>
      <adjustedEffectiveDate>2009-02-02</adjustedEffectiveDate>
      <adjustedTerminationDate>2009-05-04</adjustedTerminationDate>
      <paymentDate>
        <adjustedDate>2009-02-02</adjustedDate>
      </paymentDate>
      <dayCountFraction>ACT/360</dayCountFraction>
      <notional>
        <currency>USD</currency>
        <amount>100000000</amount>
      </notional>
      <fixedRate>0.04</fixedRate>
      <floatingRateIndex>LIBOR</floatingRateIndex>
      <indexTenor>
        <periodMultiplier>3</periodMultiplier>
        <period>M</period>
      </indexTenor>
    </fra>
  </trade>
  <trade>
    <tradeHeader>
      <partyTradeIdentifier>
        <tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">1005</tradeId>
      </partyTradeIdentifier>
      <tradeDate>2007-05-02</tradeDate>
    </tradeHeader>
    <fra>
      <sellerPartyReference>party2</sellerPartyReference>
      <adjustedEffectiveDate>2009-02-02</adjustedEffectiveDate>
      <adjustedTerminationDate>2009-05-04</adjustedTerminationDate>
      <paymentDate>
        <adjustedDate>2009-02-02</adjustedDate>
      </paymentDate>
      <dayCountFraction>ACT/360</dayCountFraction>
      <notional>
        <currency>USD</currency>
        <amount>100000000</amount>
      </notional>
      <fixedRate>0.04</fixedRate>
      <floatingRateIndex>LIBOR</floatingRateIndex>
      <indexTenor>
        <periodMultiplier>3</periodMultiplier>
        <period>M</period>
      </indexTenor>
    </fra>
  </trade>
  <party id="bankA">
    <partyId>Bank A</partyId>
  </party>
  <party id="party1">
    <partyId>Party 1</partyId>
  </party>
  <party id="party3">
    <partyId>Party 3</partyId>
  </party>
  <party id="party2">
    <partyId>Party 2</partyId>
  </party>
</dataDocument>

        

Note that the document concludes with a unique list of parties that are referenced by id in the buyerPartyReference and sellerPartyReference elements.

The required resources script is

Figure 43. Resources script resources-fra.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:fn="http://www.w3.org/2005/xpath-functions">

  <sx:service id="fra-to-xml">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="fra-xml"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="fra-xml">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="fra-flat-file"/>
      </sx:flatFileReader>
      <sx:recordValidator message="Error in trade {trade_id}.">
        <sx:fieldValidator field="buy_sell_code"
                          message="Invalid buy_sell_code {buy_sell_code}.">
          <sx:or>
            <sx:valueRestriction pattern="BUY"/>
            <sx:valueRestriction pattern="SELL"/>
          </sx:or>
        </sx:fieldValidator>
        <sx:fieldValidator field="start_date"
                           message="Invalid start_date {start_date}.">
          <sx:valueRestriction pattern="(19|20)\d\d[/](0[1-9]|1[012])[/](0[1-9]|[12][0-9]|3[01])"/>
        </sx:fieldValidator>
        <sx:fieldValidator field="maturity_date"
                           message="Invalid maturity_date {maturity_date}.">
          <sx:valueRestriction pattern="(20|21)\d\d[/](0[1-9]|1[012])[/](0[1-9]|[12][0-9]|3[01])"/>
        </sx:fieldValidator>
      </sx:recordValidator>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <sx:modifyRecord>
          <sx:newField name="message" value="{$sx:message}"/>
        </sx:modifyRecord>
        <sx:flatFileWriter>
          <sx:fileSink file="output/fra-error.csv"/>
        </sx:flatFileWriter>
      </sx:discardHandler>
    </sx:recordStream>
    <sx:recordMapping ref="fra-to-xml-mapping"/>
  </sx:recordContent>

  <sx:flatFile id="fra-flat-file">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="record_type"/>
        <sx:when test="record_type='FRA'">
          <sx:flatRecordType name='fra'>
            <sx:fieldDelimiter value=","/>
            <sx:delimitedField name="record_type"/>
            <sx:delimitedField name="trade_id"/>
            <sx:delimitedField name="client_id"/>
            <sx:delimitedField name="client_name"/>
            <sx:delimitedField name="trade_date"/>
            <sx:delimitedField name="buy_sell_code"/>
            <sx:delimitedField name="currency"/>
            <sx:delimitedField name="index"/>
            <sx:delimitedField name="tenor"/>
            <sx:delimitedField name="day_count_basis"/>
            <sx:delimitedField name="fixed_rate"/>
            <sx:delimitedField name="start_date"/>
            <sx:delimitedField name="maturity_date"/>
            <sx:delimitedField name="payment_date"/>
            <sx:delimitedField name="notional"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="fra-to-xml-mapping">
    <dataDocument xmlns="http://www.fpml.org/FpML-5-0/reporting"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  fpmlVersion="5-0"
                  xsi:schemaLocation="http://www.fpml.org/FpML-5-0/reporting ../fpml-main-5-0.xsd">
      <sx:groupBy fields="trade_id">
        <trade>
          <tradeHeader>
            <partyTradeIdentifier>
              <sx:fieldElementMap field="trade_id"
                                  element="tradeId">
                <sx:fieldAttributeMap value="http://www.banka.com/ird/trade-id"
                                      attribute="tradeIdScheme"/>
              </sx:fieldElementMap>
            </partyTradeIdentifier>
            <tradeDate>
              <sx:toXmlDate fromFormat="yyyy/MM/dd">
                <sx:toString value="{trade_date}"/>
              </sx:toXmlDate>
            </tradeDate>
          </tradeHeader>
          <sx:onRecord recordType="fra">
            <fra>
              <sx:choose>
                <sx:when test="buy_sell_code='BUY'">
                  <sx:fieldElementMap field="client_id"
                                      element="sellerPartyReference"/>
                </sx:when>
                <sx:otherwise>
                  <sx:fieldElementMap field="client_id"
                                      element="buyerPartyReference"/>
                </sx:otherwise>
              </sx:choose>
              <adjustedEffectiveDate>
                <sx:toXmlDate fromFormat="yyyy/MM/dd">
                  <sx:toString value="{start_date}"/>
                </sx:toXmlDate>
              </adjustedEffectiveDate>
              <adjustedTerminationDate>
                <sx:toXmlDate fromFormat="yyyy/MM/dd">
                  <sx:toString value="{maturity_date}"/>
                </sx:toXmlDate>
              </adjustedTerminationDate>
              <paymentDate>
                <adjustedDate>
                  <sx:toXmlDate fromFormat="yyyy/MM/dd">
                    <sx:toString value="{payment_date}"/>
                  </sx:toXmlDate>
                </adjustedDate>
              </paymentDate>
              <sx:fieldElementMap field="day_count_basis"
                                  element="dayCountFraction"/>
              <notional>
                <sx:fieldElementMap field="currency"
                                    element="currency"/>
                <sx:fieldElementMap field="notional"
                                    element="amount"/>
              </notional>
              <sx:fieldElementMap field="fixed_rate"
                                  element="fixedRate"/>
              <sx:fieldElementMap field="index"
                                  element="floatingRateIndex"/>
              <indexTenor>
                <sx:choose>
                  <sx:when test="fn:ends-with(tenor,'D')">
                    <sx:fieldElementMap element="periodMultiplier">
                      <sx:toString select="fn:substring-before(tenor,'D')"/>
                    </sx:fieldElementMap>
                    <period>D</period>
                  </sx:when>
                  <sx:when test="fn:ends-with(tenor,'M')">
                    <sx:fieldElementMap element="periodMultiplier">
                      <sx:toString select="fn:substring-before(tenor,'M')"/>
                    </sx:fieldElementMap>
                    <period>M</period>
                  </sx:when>
                  <sx:when test="fn:ends-with(tenor,'Y')">
                    <sx:fieldElementMap element="periodMultiplier">
                      <sx:toString select="fn:substring-before(tenor,'Y')"/>
                    </sx:fieldElementMap>
                    <period>Y</period>
                  </sx:when>
                </sx:choose>
              </indexTenor>
            </fra>
          </sx:onRecord>
        </trade>
      </sx:groupBy>

      <sx:nestedContent>
        <sx:recordContent ref="clients"/>
      </sx:nestedContent>
    </dataDocument>
  </sx:recordMapping>

  <sx:recordContent id="clients">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="fra-flat-file"/>
      </sx:flatFileReader>
      <sx:recordProjection fields="client_id client_name"/>
    </sx:recordStream>
    <sx:recordMapping>
      <party id="bankA">
        <partyId>Bank A</partyId>
      </party>
      <sx:onRecord>
        <party>
          <sx:fieldAttributeMap field="client_id" attribute="id"/>
          <sx:fieldElementMap field="client_name" element="partyId"/>
        </party>
      </sx:onRecord>
    </sx:recordMapping>
  </sx:recordContent>

</sx:resources>

        

We use the sx:nestedContent element to insert a list of parties from the swap.csv file into the output XML, and we use the sx:recordProjection element to eliminate duplicate pairs of client_id and client_name.

You can run this example on the command line by entering

        
servingxml -i data/fra.csv -o output/fra.xml -r resources-fra.xml fra-to-xml

      

The following error messages will be written to the console (or log file.)

25-Sep-2008 10:10:47 PM com.servingxml.util.system.DefaultLogger error
SEVERE: Error in record "fra" on line 3.  Error in trade 1002. Invalid buy_sell_code B.
25-Sep-2008 10:10:47 PM com.servingxml.util.system.DefaultLogger error
SEVERE: Error in record "fra" on line 7.  Error in trade 1006. Invalid buy_sell_code B. Invalid start_date 09/05/01.
        

Swap to FpML

Suppose "Bank A" has the following CSV file of SWAP trades. Each swap trade is represented by two records in the file, one for the pay (PAY) side, and one for the receive (REC) side.

Figure 44. Input flat file swap.csv

          
product,trade_id,client_id,client_name,trade_date,pay_receive_code,currency,index,tenor,day_count_basis,fixed_rate,start_date,maturity_date,reset_date,payment_date,notional,calculation_period
SWAP,2001,party1,Party 1,2008/05/14,PAY,FIXED,CAD,BA,6M,30/360,0.04,2008/07/17,2009/01/17,2008/07/17,2009/01/17,10000000.00,6M
SWAP,2001,party1,Party 1,2008/05/14,REC,FLOAT,CAD,BA,6M,30/360,0.04,2008/07/17,2009/01/17,2008/07/17,2009/01/17,10000000.00,6M
SWAP,2002,party2,Party 2,2008/05/14,PAY,FIXED,CAD,BA,6M,30/360,0.04,2008/07/17,2009/01/17,2008/07/17,2009/01/17,10000000.00,6M
SWAP,2002,party2,Party 2,2008/05/14,R,FLOAT,CAD,BA,6M,30/360,0.04,2008/07/17,2009/01/17,2008/07/17,2009/01/17,10000000.00,6M

        

Note that one of the records, the receive side for the trade with trade_id 2002, has an invalid value for pay_receive_code, the value has been mistakenly entered as "R" instead of "REC". Bank A wants to validate this field, and if it is wrong for either side of the swap, Bank A wants to reject both sides, and write them both with an error message to a discard file.

Bank A wants the XML output to look as follows.

Figure 45. Output XML file swap.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<dataDocument xmlns="http://www.fpml.org/FpML-5-0/reporting" 
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
              fpmlVersion="5-0" 
              xsi:schemaLocation="http://www.fpml.org/FpML-5-0/reporting ../fpml-main-5-0.xsd">
  <trade>
    <tradeHeader>
      <partyTradeIdentifier>
        <tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">2001</tradeId>
      </partyTradeIdentifier>
      <tradeDate>2008-05-14</tradeDate>
    </tradeHeader>
    <swap>
      <swapStream>
        <payerPartyReference href="bankA"/>
        <calculationPeriodDates id="floatingCalcPeriodDates">
          <effectiveDate>
            <adjustedDate>2008-07-17</adjustedDate>
          </effectiveDate>
          <terminationDate>
            <adjustedDate>2009-01-17</adjustedDate>
          </terminationDate>
          <calculationPeriodFrequency>
            <periodMultiplier>6</periodMultiplier>
            <period>M</period>
          </calculationPeriodFrequency>
        </calculationPeriodDates>
        <calculationPeriodAmount>
          <calculation>
            <notionalSchedule>
              <notionalStepSchedule>
                <initialValue>10000000.00</initialValue>
                <currency currencyScheme="http://www.fpml.org/ext/iso4217">CAD</currency>
              </notionalStepSchedule>
            </notionalSchedule>
            <fixedRateSchedule>
              <initialValue>0.04</initialValue>
            </fixedRateSchedule>
            <dayCountFraction>30/360</dayCountFraction>
          </calculation>
        </calculationPeriodAmount>
      </swapStream>
      <swapStream>
        <payerPartyReference href="party1"/>
        <calculationPeriodDates id="floatingCalcPeriodDates">
          <effectiveDate>
            <adjustedDate>2008-07-17</adjustedDate>
          </effectiveDate>
          <terminationDate>
            <adjustedDate>2009-01-17</adjustedDate>
          </terminationDate>
          <calculationPeriodFrequency>
            <periodMultiplier>6</periodMultiplier>
            <period>M</period>
          </calculationPeriodFrequency>
        </calculationPeriodDates>
        <calculationPeriodAmount>
          <calculation>
            <notionalSchedule>
              <notionalStepSchedule>
                <initialValue>10000000.00</initialValue>
                <currency currencyScheme="http://www.fpml.org/ext/iso4217">CAD</currency>
              </notionalStepSchedule>
            </notionalSchedule>
            <floatingRateCalculation>
              <floatingRateIndex>BA</floatingRateIndex>
              <indexTenor>
                <periodMultiplier>6</periodMultiplier>
                <period>M</period>
              </indexTenor>
            </floatingRateCalculation>
            <dayCountFraction>30/360</dayCountFraction>
          </calculation>
        </calculationPeriodAmount>
      </swapStream>
    </swap>
  </trade>
  <party id="bankA">
    <partyId>Bank A</partyId>
  </party>
  <party id="party1">
    <partyId>Party 1</partyId>
  </party>
  <party id="party2">
    <partyId>Party 2</partyId>
  </party>
</dataDocument>

        

Note that the document concludes with a unique list of parties that are referenced by id in the payerPartyReference elements.

The required resources script is

Figure 46. Resources script resources-swap.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:fn="http://www.w3.org/2005/xpath-functions">

  <sx:service id="swap-to-xml">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="swap-xml"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="swap-xml">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="swap-flat-file"/>
      </sx:flatFileReader>
      <sx:modifyRecord>
        <sx:newField name="trade_id" select="legs/child::node()[1]/trade_id"/>
        <sx:newField name="trade_date" select="legs/child::node()[1]/trade_date"/>
      </sx:modifyRecord>
      <sx:recordValidator ref="swap-validator"/>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <sx:splitRecord recordType="swap" repeatingGroup="legs"/>
        <sx:flatFileWriter>
          <sx:fileSink file="output/swap-error.csv"/>
        </sx:flatFileWriter>
      </sx:discardHandler>
    </sx:recordStream>
    <sx:recordMapping ref="swap-to-xml-mapping"/>
  </sx:recordContent>

  <sx:flatFile id="swap-flat-file">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:compositeFlatRecordType>
        <sx:delimitedField name="record_type"/>
        <sx:delimitedField name="trade_id"/>
        <sx:combinePhysicalRecords recordType="swap"
                              repeatingGroup="legs"
                              endTest="sx:current//trade_id != sx:previous//trade_id">
          <sx:flatRecordTypeChoice>
            <sx:fieldDelimiter value=","/>
            <sx:delimitedField name="record_type"/>
            <sx:delimitedField name="trade_id"/>
            <sx:delimitedField name="client_id"/>
            <sx:delimitedField name="client_name"/>
            <sx:delimitedField name="trade_date"/>
            <sx:delimitedField name="pay_receive_code"/>
            <sx:delimitedField name="fixed_float_code"/>
            <sx:when test="record_type='SWAP' and fixed_float_code='FLOAT'">
              <sx:flatRecordType name='swap_floating_leg'>
                <sx:fieldDelimiter value=","/>
                <sx:delimitedField name="record_type"/>
                <sx:delimitedField name="trade_id"/>
                <sx:delimitedField name="client_id"/>
                <sx:delimitedField name="client_name"/>
                <sx:delimitedField name="trade_date"/>
                <sx:delimitedField name="pay_receive_code"/>
                <sx:delimitedField name="fixed_float_code"/>
                <sx:delimitedField name="currency"/>
                <sx:delimitedField name="index"/>
                <sx:delimitedField name="tenor"/>
                <sx:delimitedField name="day_count_basis"/>
                <sx:delimitedField name="fixed_rate"/>
                <sx:delimitedField name="start_date"/>
                <sx:delimitedField name="maturity_date"/>
                <sx:delimitedField name="reset_date"/>
                <sx:delimitedField name="payment_date"/>
                <sx:delimitedField name="notional"/>
                <sx:delimitedField name="calculation_period"/>
              </sx:flatRecordType>
            </sx:when>
            <sx:when test="record_type='SWAP' and fixed_float_code='FIXED'">
              <sx:flatRecordType name='swap_fixed_leg'>
                <sx:fieldDelimiter value=","/>
                <sx:delimitedField name="record_type"/>
                <sx:delimitedField name="trade_id"/>
                <sx:delimitedField name="client_id"/>
                <sx:delimitedField name="client_name"/>
                <sx:delimitedField name="trade_date"/>
                <sx:delimitedField name="pay_receive_code"/>
                <sx:delimitedField name="fixed_float_code"/>
                <sx:delimitedField name="currency"/>
                <sx:delimitedField name="index"/>
                <sx:delimitedField name="tenor"/>
                <sx:delimitedField name="day_count_basis"/>
                <sx:delimitedField name="fixed_rate"/>
                <sx:delimitedField name="start_date"/>
                <sx:delimitedField name="maturity_date"/>
                <sx:delimitedField name="reset_date"/>
                <sx:delimitedField name="payment_date"/>
                <sx:delimitedField name="notional"/>
                <sx:delimitedField name="calculation_period"/>
              </sx:flatRecordType>
            </sx:when>
          </sx:flatRecordTypeChoice>
        </sx:combinePhysicalRecords>
      </sx:compositeFlatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>
  <sx:recordValidator id="swap-validator"
                      recordType="swap"
                      message="Error in trade {trade_id}.">
    <sx:fieldValidator field="legs"
                       message="">
      <sx:recordValidator>
        <sx:fieldValidator field="pay_receive_code"
                           message="Invalid pay_receive_code {pay_receive_code}.">
          <sx:or>
            <sx:valueRestriction pattern="PAY"/>
            <sx:valueRestriction pattern="REC"/>
          </sx:or>
        </sx:fieldValidator>
        <sx:fieldValidator field="fixed_float_code"
                           message="Invalid fixed_float_code {fixed_float_code}.">
          <sx:or>
            <sx:valueRestriction pattern="FIXED"/>
            <sx:valueRestriction pattern="FLOAT"/>
          </sx:or>
        </sx:fieldValidator>
        <sx:fieldValidator field="start_date"
                           message="Invalid start_date {start_date}.">
          <sx:valueRestriction pattern="(19|20)\d\d[/](0[1-9]|1[012])[/](0[1-9]|[12][0-9]|3[01])"/>
        </sx:fieldValidator>
        <sx:fieldValidator field="maturity_date"
                           message="Invalid maturity_date {maturity_date}.">
          <sx:valueRestriction pattern="(20|21)\d\d[/](0[1-9]|1[012])[/](0[1-9]|[12][0-9]|3[01])"/>
        </sx:fieldValidator>
      </sx:recordValidator>
    </sx:fieldValidator>
  </sx:recordValidator>

  <sx:recordMapping id="swap-to-xml-mapping">
    <dataDocument xmlns="http://www.fpml.org/FpML-5-0/reporting"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  fpmlVersion="5-0"
                  xsi:schemaLocation="http://www.fpml.org/FpML-5-0/reporting ../fpml-main-5-0.xsd">
      <sx:groupBy fields="trade_id">
        <trade>
          <tradeHeader>
            <partyTradeIdentifier>
              <sx:fieldElementMap field="trade_id" element="tradeId">
                <sx:fieldAttributeMap value="http://www.banka.com/ird/trade-id"
                                      attribute="tradeIdScheme"/>
              </sx:fieldElementMap>
            </partyTradeIdentifier>
            <tradeDate>
              <sx:toXmlDate fromFormat="yyyy/MM/dd">
                <sx:toString value="{trade_date}"/>
              </sx:toXmlDate>
            </tradeDate>
          </tradeHeader>
          <swap>
            <sx:subrecordMapping repeatingGroup="legs">
              <sx:groupBy fields="trade_id pay_receive_code">
                <swapStream>
                  <sx:choose>
                    <sx:when test="pay_receive_code='PAY'">
                      <payerPartyReference>
                        <sx:fieldAttributeMap value="bankA"
                                              attribute="href"/>
                      </payerPartyReference>
                    </sx:when>
                    <sx:when test="pay_receive_code='REC'">
                      <payerPartyReference>
                        <sx:fieldAttributeMap field="client_id"
                                              attribute="href"/>
                      </payerPartyReference>
                    </sx:when>
                  </sx:choose>
                  <calculationPeriodDates id="floatingCalcPeriodDates">
                    <effectiveDate>
                      <adjustedDate>
                        <sx:toXmlDate fromFormat="yyyy/MM/dd">
                          <sx:toString value="{start_date}"/>
                        </sx:toXmlDate>
                      </adjustedDate>
                    </effectiveDate>
                    <terminationDate>
                      <adjustedDate>
                        <sx:toXmlDate fromFormat="yyyy/MM/dd">
                          <sx:toString value="{maturity_date}"/>
                        </sx:toXmlDate>
                      </adjustedDate>
                    </terminationDate>
                    <calculationPeriodFrequency>
                      <sx:parameter name="term" value="{calculation_period}"/>
                      <sx:choose ref="expand-term"/>
                    </calculationPeriodFrequency>
                  </calculationPeriodDates>
                  <calculationPeriodAmount>
                    <calculation>
                      <notionalSchedule>
                        <notionalStepSchedule>
                          <sx:fieldElementMap field="notional" element="initialValue"/>
                          <sx:fieldElementMap field="currency" element="currency">
                            <sx:fieldAttributeMap value="http://www.fpml.org/ext/iso4217"
                                                  attribute="currencyScheme"/>
                          </sx:fieldElementMap>
                        </notionalStepSchedule>
                      </notionalSchedule>
                      <sx:choose>
                        <sx:when test="fixed_float_code='FLOAT'">
                          <floatingRateCalculation>
                            <sx:fieldElementMap field="index"
                                                element="floatingRateIndex"/>
                            <indexTenor>
                              <sx:parameter name="term" value="{tenor}"/>
                              <sx:choose ref="expand-term"/>
                            </indexTenor>
                          </floatingRateCalculation>
                        </sx:when>
                        <sx:when test="fixed_float_code='FIXED'">
                          <fixedRateSchedule>
                            <sx:fieldElementMap field="fixed_rate"
                                                element="initialValue"/>
                          </fixedRateSchedule>
                        </sx:when>
                      </sx:choose>
                      <sx:fieldElementMap field="day_count_basis"
                                          element="dayCountFraction"/>
                    </calculation>
                  </calculationPeriodAmount>
                </swapStream>
              </sx:groupBy>
            </sx:subrecordMapping>
          </swap>
        </trade>
      </sx:groupBy>

      <sx:nestedContent>
        <sx:recordContent ref="clients"/>
      </sx:nestedContent>
    </dataDocument>
  </sx:recordMapping>

  <!-- Expands a parameter named "term" with a value <number><D|M|Y> as
    <periodMultiplier>number</periodMultiplier>
    <period>D|M|Y</period>
  -->
  <sx:choose id="expand-term">
    <sx:when test="fn:ends-with($term,'D')">
      <sx:fieldElementMap element="periodMultiplier">
        <sx:toString select="fn:substring-before($term,'D')"/>
      </sx:fieldElementMap>
      <period>D</period>
    </sx:when>
    <sx:when test="fn:ends-with($term,'M')">
      <sx:fieldElementMap element="periodMultiplier">
        <sx:toString select="fn:substring-before($term,'M')"/>
      </sx:fieldElementMap>
      <period>M</period>
    </sx:when>
    <sx:when test="fn:ends-with($term,'Y')">
      <sx:fieldElementMap element="periodMultiplier">
        <sx:toString select="fn:substring-before($term,'Y')"/>
      </sx:fieldElementMap>
      <period>Y</period>
    </sx:when>
  </sx:choose>

  <sx:recordContent id="clients">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile>
          <sx:flatFileHeader lineCount="1"/>
          <sx:flatFileBody>
            <sx:flatRecordType>
              <sx:fieldDelimiter value=","/>
              <sx:delimitedField name="record_type"/>
              <sx:delimitedField name="trade_id"/>
              <sx:delimitedField name="client_id"/>
              <sx:delimitedField name="client_name"/>
            </sx:flatRecordType>
          </sx:flatFileBody>
        </sx:flatFile>
      </sx:flatFileReader>
      <sx:recordProjection fields="client_id client_name"/>
    </sx:recordStream>
    <sx:recordMapping>
      <party id="bankA">
        <partyId>Bank A</partyId>
      </party>
      <sx:onRecord>
        <party>
          <sx:fieldAttributeMap field="client_id" attribute="id"/>
          <sx:fieldElementMap field="client_name" element="partyId"/>
        </party>
      </sx:onRecord>
    </sx:recordMapping>
  </sx:recordContent>

</sx:resources>

        

Note the following points about this script.

  • We use the sx:combineRecords element to compose a single record from the two legs of the swap. This way, when we validate the composite record, a failure in either of the legs will result in the entire swap being discarded, not just one of the legs.

  • We use the sx:nestedContent element to insert a list of parties from the swap.csv file into the output XML, and we use the sx:recordProjection element to eliminate duplicate pairs of client_id and client_name.

  • Inside the sx:discardHandler element, we use the sx:splitRecord element to decompose the composite swap record back into separate leg records. This allows us to easily use the default record writer to write the legs to a discard file.

  • We have two fields, calculation_period and tenor , that have values like 3M and 30D , where the numeric value and the trailing letter need to be mapped separately to the elements periodMultiplier and period . We use a parameterized sx:choose instruction to handle both cases.

You can run this example on the command line by entering

        
servingxml -i data/swap.csv -o output/swap.xml -r resources-swap.xml swap-to-xml

      

The following error message will be written to the console (or log file.)

25-Sep-2008 10:10:54 PM com.servingxml.util.system.DefaultLogger error
SEVERE: Error in record "swap" on line 5.  Error in trade 2002.  Invalid pay_receive_code R.

Fixed Width Fields, Single Record Type, Line Delimited

A flat file to XML mapping with field substitutions (abtest)

The example below shows how to convert a flat file to XML.

The input file is a positional file.

Figure 47. Input flat file abtest.txt

          
ABTESTCD10 
ACTESTCD12 
ABTESTCC80

        

The requirements are as follows:

  • First 2 chars: if AB the output will be SOUTH, if AC then output is WEST.
  • For the 7th and 9th chars: if CD, then output change to 01, otherwise 02

The desired output is shown below.

Figure 48. Output XML file Recs.xml

          
<?xml version="1.0" encoding="utf-8"?>
<Recs>
  <Rec>
    <First>SOUTH</First>
    <Second>TEST</Second>
    <Third>01</Third>
    <Last>10</Last>
  </Rec>
  <Rec>
    <First>WEST</First>
    <Second>TEST</Second>
    <Third>01</Third>
    <Last>12</Last>
  </Rec>
  <Rec>
    <First>SOUTH</First>
    <Second>TEST</Second>
    <Third>02</Third>
    <Last>80</Last>
  </Rec>
</Recs>
        

The following resources script does the transformation.

Figure 49. Resources script resources-abtest.xml

          
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="abtest">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="data"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="data">
    <sx:flatFileReader>
      <sx:urlSource url="data/abtest.txt"/>
      <sx:flatFile ref="flatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="abtestToXml"/>
  </sx:recordContent>

  <sx:flatFile id="flatFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="match">
        <sx:positionalField name="first" width="2"/>
        <sx:positionalField name="second" width="4"/>
        <sx:positionalField name="third" width="2"/>
        <sx:positionalField name="fourth" width="2"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="abtestToXml">
    <Recs>
      <sx:onRecord>
        <Rec>
          <sx:fieldElementMap match="first[text()='AB']" select="'SOUTH'" element="First"/>
          <sx:fieldElementMap match="first[text()='AC']" select="'WEST'" element="First"/>
          <sx:fieldElementMap field="second" element="Second"/>
          <sx:fieldElementMap match="third[text()='CD']" select="'01'" element="Third"/>
          <sx:fieldElementMap match="third[text()!='CD']" select="'02'" element="Third"/>
          <sx:fieldElementMap field="fourth" element="Last"/>
        </Rec>
      </sx:onRecord>
    </Recs>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -r resources-abtest.xml -o output/abtest.xml abtest

        

Mapping a record to separate top level sections in the XML (persons)

Suppose you have the following positional file of persons and their addresses.

Figure 50. Input flat file persons.txt

          
PersonId Name       FirstName       Street              Postcode    CityTown
1        Jones      Joe             215 Walmer Rd       M5R3P7      Toronto
1        Jones      Joe             180 Redwood ST      94102-3280  San Francisco
2        Davis      Ken             212 Harbord St      M5S1H6      Toronto
3        Morris     Jane            1100 Danforth Ave   M4J1N2      Toronto
3        Morris     Jane            6 Green St          94111-1402  San Francisco

        

Each line containes information about a person and an address. A person may have multiple lines indicating multiple addresses.

You want the XML output to look as follows.

Figure 51. Output XML file persons.xml

          
<?xml version="1.0" encoding="utf-8"?>
<Persons-Addresses>
  <Persons>
    <Person>
      <PersonId>1</PersonId>
      <Name>Jones</Name>
      <FirstName>Joe</FirstName>
    </Person>
    <Person>
      <PersonId>2</PersonId>
      <Name>Davis</Name>
      <FirstName>Ken</FirstName>
    </Person>
    <Person>
      <PersonId>3</PersonId>
      <Name>Morris</Name>
      <FirstName>Jane</FirstName>
    </Person>
  </Persons>
  <Addresses>
    <Address>
      <PersonId>1</PersonId>
      <Street>215 Walmer Rd</Street>
      <PostCode>M5R3P7</PostCode>
      <CityTown>Toronto</CityTown>
    </Address>
    <Address>
      <PersonId>1</PersonId>
      <Street>180 Redwood ST</Street>
      <PostCode>94102-3280</PostCode>
      <CityTown>San Francisco</CityTown>
    </Address>
    <Address>
      <PersonId>2</PersonId>
      <Street>212 Harbord St</Street>
      <PostCode>M5S1H6</PostCode>
      <CityTown>Toronto</CityTown>
    </Address>
    <Address>
      <PersonId>3</PersonId>
      <Street>1100 Danforth Ave</Street>
      <PostCode>M4J1N2</PostCode>
      <CityTown>Toronto</CityTown>
    </Address>
    <Address>
      <PersonId>3</PersonId>
      <Street>6 Green St</Street>
      <PostCode>94111-1402</PostCode>
      <CityTown>San Francisco</CityTown>
    </Address>
  </Addresses>
</Persons-Addresses>

        

There are two ways to do this:

  • Use a record mapping section with two sx:groupBy sections. In this case the implementation will buffer the records in memory until the first group is finished, and only then do the second.

  • Within a record mapping section, read the flat file twice, first inserting the person data content, and then inserting the address content.

Taking the first approach, the resources script looks like this:

Figure 52. Resources script resources-persons1.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="personsAddresses">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="persons"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:flatFile id="personsData">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType ref="persons"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordContent id="persons">
    <sx:flatFileReader>
      <sx:flatFile ref="personsData"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="personsAddressesMapping"/>
  </sx:recordContent>

  <sx:flatRecordType id="persons" name="persons">
    <sx:positionalField name="PersonId" width="9"/>
    <sx:positionalField name="Name" width="11"/>
    <sx:positionalField name="FirstName" width="16"/>
    <sx:positionalField name="Street" width="20"/>
    <sx:positionalField name="PostCode" width="12"/>
    <sx:positionalField name="CityTown" width="20" />
  </sx:flatRecordType>

  <sx:recordMapping id="personsAddressesMapping">
    <Persons-Addresses>
      <Persons>
        <sx:groupBy fields="PersonId">
          <Person>
            <sx:fieldElementMap field="PersonId" element="PersonId"/>
            <sx:fieldElementMap field="Name" element="Name"/>
            <sx:fieldElementMap field="FirstName" element="FirstName"/>
            <sx:onRecord/>
          </Person>
        </sx:groupBy>
      </Persons>
      <Addresses>
        <sx:onRecord>
          <Address>
            <sx:fieldElementMap field="PersonId" element="PersonId"/>
            <sx:fieldElementMap field="Street" element="Street"/>
            <sx:fieldElementMap field="PostCode" element="PostCode"/>
            <sx:fieldElementMap field="CityTown" element="CityTown"/>
          </Address>
        </sx:onRecord>
      </Addresses>
    </Persons-Addresses>
  </sx:recordMapping>

</sx:resources>

        

Taking the second approach, the resources script looks like this:

Figure 53. Resources script resources-persons2.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="personsAddresses">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="persons"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:flatFile id="personsData">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType ref="persons"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordContent id="persons">
    <sx:recordMapping ref="personsAddressesMapping"/>
  </sx:recordContent>

  <sx:flatRecordType id="persons" name="persons">
    <sx:positionalField name="PersonId" width="9"/>
    <sx:positionalField name="Name" width="11"/>
    <sx:positionalField name="FirstName" width="16"/>
    <sx:positionalField name="Street" width="20"/>
    <sx:positionalField name="PostCode" width="12"/>
    <sx:positionalField name="CityTown" width="20" />
  </sx:flatRecordType>

  <sx:recordMapping id="personsAddressesMapping">
    <Persons-Addresses>
      <sx:nestedContent>
        <sx:recordContent>
          <sx:flatFileReader>
            <sx:flatFile ref="personsData"/>
          </sx:flatFileReader>
          <sx:recordMapping ref="personsMapping"/>
        </sx:recordContent>
      </sx:nestedContent>
      <sx:nestedContent>
        <sx:recordContent>
          <sx:flatFileReader>
            <sx:flatFile ref="personsData"/>
          </sx:flatFileReader>
          <sx:recordMapping ref="addressesMapping"/>
        </sx:recordContent>
      </sx:nestedContent>
    </Persons-Addresses>
  </sx:recordMapping>

  <sx:recordMapping id="personsMapping">
    <Persons>
      <sx:groupBy fields="PersonId">
        <Person>
          <sx:fieldElementMap field="PersonId" element="PersonId"/>
          <sx:fieldElementMap field="Name" element="Name"/>
          <sx:fieldElementMap field="FirstName" element="FirstName"/>
          <sx:onRecord/>
        </Person>
      </sx:groupBy>
    </Persons>
  </sx:recordMapping>

  <sx:recordMapping id="addressesMapping">
    <Addresses>
      <sx:onRecord>
        <Address>
          <sx:fieldElementMap field="PersonId" element="PersonId"/>
          <sx:fieldElementMap field="Street" element="Street"/>
          <sx:fieldElementMap field="PostCode" element="PostCode"/>
          <sx:fieldElementMap field="CityTown" element="CityTown"/>
        </Address>
      </sx:onRecord>
    </Addresses>
  </sx:recordMapping>

</sx:resources>

        

Note the following:

  • Since you only want to show distinct persons in the persons section, group by PersonId.

You can run the two examples on the command line by entering

          
servingxml -i data/persons.txt -o output/persons1.xml -r resources-persons1.xml 
    personsAddresses 

        

          
servingxml -i data/persons.txt -o output/persons2.xml -r resources-persons2.xml 
    personsAddresses 

        

Converting a directory of books positional files to XML files (all books)

The example below shows how to process all the files in the data directory matching the regular expression "books.*[.]txt", convert them from positional flat file formats to XML files, and write them out to the output directory.

As before, the input is a directory of files.

Figure 54. Directory containing input files

          
[.]                     countries.csv           multivalued-field.csv
[..]                    countries.xsd           plans.txt
3545_JH4DA3_4_H_.xml    country-record.xsd      tasks.csv
bad-countries.csv       exotics.txt             timesheets.csv
books.20040613.txt      hot-record.xsd          trades.txt
books.20040802.txt      hot-record.xsx
books.txt               hot.txt
books.xml               messages.properties
        

We want to take the positional files whose names match the regular expression "(books.*)[.]txt" and convert them all to XML files.

Figure 55. Books XML output files.

          
books.xml            books.20040802.xml
books.20040613.xml

        

The following resources script does the transformation.

Figure 56. Resources script resources-allbooks.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="all-books">
    <sx:recordStream>
      <sx:directoryReader directory="data">
        <sx:fileFilter pattern="books.*[.]txt"/>
      </sx:directoryReader>
      <sx:processRecord>
        <sx:parameter name="output-file">
          <sx:findAndReplace searchFor ="(books.*)[.]txt" replaceWith ="$1.xml">
            <sx:toString value="{name}"/>
          </sx:findAndReplace>
        </sx:parameter>
        <sx:serialize>
          <sx:xsltSerializer>
            <sx:fileSink directory="output" file="{$output-file}"/>
          </sx:xsltSerializer>
          <sx:transform>
            <sx:content ref="books"/>
          </sx:transform>
        </sx:serialize>
      </sx:processRecord>
    </sx:recordStream>
  </sx:service>

  <sx:recordContent id="books">
    <sx:flatFileReader>
      <sx:fileSource directory="{parentDirectory}" file="{name}"/>
      <sx:flatFile ref="booksFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="booksToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="booksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="bookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>      

  <sx:flatRecordType id="bookType" name="bookType">
    <sx:positionalField name="category" label="Category" width="1"/>
    <sx:positionalField name="author" label="Author" width="30"/>
    <sx:positionalField name="title" label="Title" width="30"/>
    <sx:positionalField name="price" label="Price" width="10" justify="right"/>
  </sx:flatRecordType>

  <sx:recordMapping id="booksToXmlMapping">
    <myns:books xmlns:myns="http://mycompany.com/mynames/" 
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     xsi:schemaLocation="url2">
      <sx:onRecord>
        <myns:book>
          <sx:fieldAttributeMap field="category" attribute="categoryCode"/>
          <sx:fieldElementMap field="title" element="myns:title"/>  
          <sx:fieldElementMap field="author" element="myns:author"/>
          <sx:fieldElementMap field="price" element="myns:price"/>
        </myns:book>  
      </sx:onRecord>
    </myns:books>
  </sx:recordMapping>  
  
</sx:resources>

        

You can run this example on the command line by entering

          
servingxml  -r resources-allbooks.xml all-books

        

Converting a flat file to multiple XML book files (multiple books)

The example below shows how to process one flat file and produce multiple XML output files.

The input file is shown below.

Figure 57. Books input file

          
CAuthor                        Title                              Price ISBN

FCharles Bukowski              Factotum                           22.95 0876852630
FSergei Lukyanenko             The Night Watch                    17.99 0434014125
FAndrew Crumey                 Mr Mee                             22.00 0312282354
CSteven John Metsker           Building Parsers with Java         39.95 0201719622

This is a trailer record

        

We want to produce multiple files like the one shown below, with filenames differentiated by the isbn number.

Figure 58. Book XML output file book-0876852630.xml.

          
<?xml version="1.0" encoding="UTF-8"?>
<myns:book xmlns:myns="http://mycompany.com/mynames/" 
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
           categoryCode="F">
  <myns:title>Factotum</myns:title>
  <myns:author>Charles Bukowski</myns:author>
  <myns:price>22.95</myns:price>
  <myns:isbn>0876852630</myns:isbn>
</myns:book>

        

The following resources script does the transformation.

Figure 59. Resources script resources-books_multiple.xml

          
<?xml version="1.0"?>

<sx:resources xmlns:sx="http://www.servingxml.com/core"
                     xmlns:myns="http://mycompany.com/mynames/" >

  <sx:service id="books">
    <sx:transform>
      <sx:content ref="books"/>
      <sx:processSubtree path="/myns:books/myns:book">
        <sx:parameter name="isbn" select="myns:isbn"/>
        <sx:serialize>
          <sx:xsltSerializer> 
            <sx:fileSink directory="output" file="book-{$isbn}.xml"/> 
          </sx:xsltSerializer> 
        </sx:serialize>
      </sx:processSubtree>
    </sx:transform>
  </sx:service>
  
  <!--
  This sx:flatFileReader element does not specify a stream source, so 
  the source will default to the file specified with the -i option on the command line.
  -->
  <sx:recordContent id="books">
    <sx:flatFileReader>
      <sx:flatFile ref="booksFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="booksToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="booksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="bookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>

  <sx:flatRecordType id="bookType" name="bookType">
    <sx:positionalField name="category" label="Category" width="1"/>
    <sx:positionalField name="author" label="Author" width="30"/>
    <sx:positionalField name="title" label="Title" width="30"/>
    <sx:positionalField name="price" label="Price" width="10" justify="right"/>
    <sx:positionalField name="isbn" label="ISBN" start="73" width="10"/>
  </sx:flatRecordType>

  <sx:recordMapping id="booksToXmlMapping">
    <myns:books xmlns:myns="http://mycompany.com/mynames/"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     xsi:schemaLocation="url2">
      <sx:onRecord>
        <myns:book>
          <sx:fieldAttributeMap field="category" attribute="categoryCode"/>
          <sx:fieldElementMap field="title" element="myns:title"/>
          <sx:fieldElementMap field="author" element="myns:author"/>
          <sx:fieldElementMap field="price" element="myns:price"/>
          <sx:fieldElementMap field="isbn" element="myns:isbn"/>
        </myns:book>
      </sx:onRecord>
    </myns:books>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -r resources-books_multiple.xml -i data/books.txt books 

        

Converting a flat file to multiple XML documents with default namespace prefixes (multiple default ns)

The example below shows how to process one flat file and produce multiple XML documents with default namespace prefixes.

The input file is the same as in the previous example.

Figure 60. Books input file

          
CAuthor                        Title                              Price ISBN

FCharles Bukowski              Factotum                           22.95 0876852630
FSergei Lukyanenko             The Night Watch                    17.99 0434014125
FAndrew Crumey                 Mr Mee                             22.00 0312282354
CSteven John Metsker           Building Parsers with Java         39.95 0201719622

This is a trailer record

        

This time we want to produce multiple files like the one shown below, with a default namespace prefix.

Figure 61. Book XML output file book-with-default-ns-0156028972.xml.

          
<?xml version="1.0" encoding="UTF-8"?>
<book xmlns="http://mycompany.com/mynames/"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      categoryCode="F">
   <title>Gun, with Occasional Music</title>
   <author>Jonathan Lethem</author>
   <price>17.99</price>
   <isbn>0156028972</isbn>
</book>

        

The following resources script does the transformation.

Figure 62. Resources script resources-multiple_default_ns.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="books">
    <sx:transform>
      <sx:content ref="books"/>
      <sx:processSubtree path="/myns:books/myns:book">
        <sx:parameter name="isbn" select="myns:isbn"/>
        <sx:serialize>
          <sx:xsltSerializer>
            <sx:fileSink directory="output" file="book-with-default-ns-{$isbn}.xml"/>
            <sx:outputProperty name="indent" value="yes"/>
          </sx:xsltSerializer>
        </sx:serialize>
      </sx:processSubtree>
    </sx:transform>
  </sx:service>
  
  <!--
  This sx:flatFileReader element does not specify a stream source, so 
  the source will default to the file specified with the -i option on the command line.
  -->
  <sx:recordContent id="books">
    <sx:flatFileReader>
      <sx:flatFile ref="booksFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="booksToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="booksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="bookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>

  <sx:flatRecordType id="bookType" name="bookType">
    <sx:positionalField name="category" label="Category" width="1"/>
    <sx:positionalField name="author" label="Author" width="30"/>
    <sx:positionalField name="title" label="Title" width="30"/>
    <sx:positionalField name="price" label="Price" width="10" justify="right"/>
    <sx:positionalField name="isbn" label="ISBN" start="73" width="10"/>
  </sx:flatRecordType>

  <sx:recordMapping id="booksToXmlMapping">
    <books xmlns="http://mycompany.com/mynames/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="url2">
      <sx:onRecord>
        <book>
          <sx:fieldAttributeMap field="category" attribute="myns:categoryCode"/>
          <sx:fieldElementMap field="title" element="myns:title"/>
          <sx:fieldElementMap field="author" element="myns:author"/>
          <sx:fieldElementMap field="price" element="myns:price"/>
          <sx:fieldElementMap field="isbn" element="myns:isbn"/>
        </book>
      </sx:onRecord>
    </books>
  </sx:recordMapping>

</sx:resources>

        

Remarks
  • Note that the root element in the sx:recordMapping section contains a default namespace declaration, which will appear in the output XML, and which will have the effect that XML names belonging to the namespace "http://mycompany.com/mynames/" will be output without an explicit prefix.

  • Note that the sx:fieldAttributeMap and sx:fieldElementMap elements use the myns prefix to associate attribute and element names with the namespace "http://mycompany.com/mynames/". The prefix is resolved against the nearest in-scope prefix declaration for myns , which happens to be the one in the sx:resources element. Since this declaration does not appear inside the record mapping section, it is used only to associate a name with a namespace, not for serialization.

  • If a prefix were omitted in an attribute or element mapping, say on title , that element would belong to no namespace, all others would belong to the default namespace, and the output would become

                    
    <book xmlns="http://mycompany.com/mynames/"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          categoryCode="F">
       <title xmlns="">Gun, with Occasional Music</title>
       <author>Jonathan Lethem</author>
       <price>17.99</price>
       <isbn>0156028972</isbn>
    </book>
    
    
                  

You can run this example on the command line by entering

          
servingxml -r resources-multiple_default_ns.xml -i data/books.txt books 

        

Flat Files with Multibyte Characters

A UTF-8 positional flat file with double byte characters

The example below shows how to convert a positional flat file with multi-byte encoding to XML.

The input file is a positional flat file containing the following hex UTF-8 byte input.

Figure 63. Input flat file doubleByte.txt

          
41414242 43430A41 C3964242 43430A

        

The two byte sequence C396 represents the single character Ö.

The desired output is shown below.

Figure 64. Output XML file doubleByte.xml

          
<?xml version="1.0" encoding="UTF-8" ?> 
<document>
  <documentData>
    <param1>AA</param1> 
    <param2>BB</param2> 
    <param3>CC</param3> 
  </documentData>
  <documentData>
    <param1>AÖ</param1> 
    <param2>BB</param2> 
    <param3>CC</param3> 
  </documentData>
</document>

        

The following resources script does the transformation.

Figure 65. Resources script resources-doubleByte.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="convert">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="data"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="data">
    <sx:flatFileReader>
      <sx:flatFile ref="flatfile"/>
      <sx:defaultStreamSource encoding="UTF-8"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="recordmap"/>
  </sx:recordContent>

  <sx:flatFile id="flatfile">
    <sx:flatFileBody>
      <sx:flatRecordType ref="recordtype"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType id="recordtype">
    <sx:positionalField name="param1" width="2"/>
    <sx:positionalField name="param2" width="2"/>
    <sx:positionalField name="param3" width="2"/>
  </sx:flatRecordType>

  <sx:recordMapping id="recordmap">
    <document>
      <sx:onRecord>
        <documentData>
          <sx:fieldElementMap element="param1" field="param1"/>
          <sx:fieldElementMap element="param2" field="param2"/>
          <sx:fieldElementMap element="param3" field="param3"/>
        </documentData>
      </sx:onRecord>
    </document>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -r resources-doubleByte.xml -i data/doubleByte.dat 
    -o output/doubleByte.xml convert

        

Fixed Width Fields, Multiple Record Types, Line Delimited

Converting a flat file with multiple record formats to XML (trades)

The example below shows how to convert a flat file with multiple record formats to XML.

The input file is a positional flat file.

Figure 66. Input flat file trades.txt

          
TR000103/25/2005 1:50:00This is a trade record
TN0002X1234A child transaction
TN0003X1235Another child transaction

        

The desired output is shown below.

Figure 67. Output XML file trades.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<trades feed="LONDON">
  <trade id="0001">
    <trade-date>2005-03-25T01:50:00.000-05:00</trade-date>
    <description>This is a trade record</description>
    <transaction id="0002">
      <reference>X1234</reference>
      <description>A child transaction</description>
    </transaction>
    <transaction id="0003">
      <reference>X1235</reference>
      <description>Another child transaction</description>
    </transaction>
  </trade>
</trades>

        

The following resources script does the transformation.

Figure 68. Resources script resources-trades.xml

          
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="trades">
    <sx:parameter name="feed">
      <sx:defaultValue>NY</sx:defaultValue>
    </sx:parameter>
    <sx:serialize>
      <sx:transform>
        <sx:content ref="trades"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="trades">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:urlSource url="data/trades.txt"/>
        <sx:flatFile ref="tradesFlatFile"/>
      </sx:flatFileReader>
      <sx:combineRecords recordType="composite" repeatingGroup="detail"
                                             startTest="sx:current//record_type='TR'"
                                             endTest="sx:current//record_type='TR'">
        <sx:newField name="id" select="detail/trade/id"/>
      </sx:combineRecords>
    </sx:recordStream>
    <sx:recordMapping ref="tradesToXmlMapping"/>
  </sx:recordContent>

  <sx:recordMapping id="tradesToXmlMapping">
    <trades>
      <sx:fieldAttributeMap value="{$feed}" attribute="feed"/>
      <sx:onRecord recordType="composite">
        <trade>
          <sx:fieldAttributeMap field="id" attribute="id"/>
          <sx:subrecordMapping repeatingGroup="detail">
            <sx:onRecord recordType="trade">
              <sx:elementMap element="trade-date">
                <sx:convertToDateTime format="MM/dd/yyyy H:mm:ss">
                  <sx:concat separator=" "> 
                    <sx:toString value="{trade_date}"/>
                    <sx:toString value="{trade_time}"/>
                  </sx:concat>
                </sx:convertToDateTime>
              </sx:elementMap>
              <sx:fieldElementMap field="description" element="description"/>
            </sx:onRecord>
            <sx:onRecord recordType="transaction">
              <transaction>
                <sx:fieldAttributeMap field="id" attribute="id"/>
                <sx:fieldElementMap field="reference" element="reference"/>
                <sx:fieldElementMap field="description" element="description"/>
              </transaction>
            </sx:onRecord>
          </sx:subrecordMapping>
        </trade>
      </sx:onRecord>
    </trades>
  </sx:recordMapping>

  <sx:flatFile id="tradesFlatFile">
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="record_type" width="2"/>
        <sx:when test="record_type='TR'">
          <sx:flatRecordType name="trade">
            <sx:positionalField name="record_type" width="2"/>
            <sx:positionalField name="id" width="4"/>
            <sx:positionalField name="trade_date" width="10"/>
            <sx:positionalField name="trade_time" width="8"/>
            <sx:positionalField name="description" width="30"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="record_type='TN'">
          <sx:flatRecordType name="transaction">
            <sx:positionalField name="record_type" width="2"/>
            <sx:positionalField name="id" width="4"/>
            <sx:positionalField name="reference" width="5"/>
            <sx:positionalField name="description" width="30"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

        

Note the following points about the content of the<sx:elementMap element="trade-date"> element.

  • The inner sx:concat element concatenates the date and time fields from the record, separating them with a space.
  • The sx:convertDate element converts the datetime value into something close to an ISO 8601 date, where the input and output formats follow the syntax specified for the JDK SimpleDateFormat class.
  • The outer sx:findAndReplace element inserts a colon into the time format.

You can run this example on the command line by entering

          
servingxml -r resources-trades.xml -o output/trades.xml trades feed=LONDON

        

Converting a flat file to XML with header, detail, and summary records (CONV)

The example below shows how to convert a flat file with header, detail, and summary records to XML.

The input file is a positional flat file with variable record layouts.

Figure 69. Input flat file CONV_in.dat

          
90112345678abcdefg1000001 
100SURNAME1 GIVEN1 A 
100SURNAME2 GIVEN2 B 
3011111111AB111111 

        

The desired output is shown below.

Figure 70. Output XML file CONV.xml

          
<?xml version="1.0" encoding="utf-8"?>
<Submission>
  <Header>
    <record_type>901</record_type>
    <sbmt_ref_id>abcdefg</sbmt_ref_id>
    <trnmtr_tcd>1</trnmtr_tcd>
    <summ_cnt>000001</summ_cnt>
  </Header>
  <All>
    <Level1>
      <Detail>
        <snm>SURNAME1</snm>
        <gvn_nm>GIVEN1</gvn_nm>
        <init>A</init>
      </Detail>
      <Detail>
        <snm>SURNAME2</snm>
        <gvn_nm>GIVEN2</gvn_nm>
        <init>B</init>
      </Detail>
      <Summary>
        <bn>1111111AB111111</bn>
      </Summary>
    </Level1>
  </All>
</Submission>

The following resources script does the transformation.

Figure 71. Resources script resources-CONV.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="CONV">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="CONV"/>
        <sx:removeEmptyElements elements="*"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="CONV">
    <sx:flatFileReader>
      <sx:urlSource url="data/CONV_in.dat"/>
      <sx:flatFile ref="CONVFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="CONVToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="CONVFlatFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="record_type" width="3"/>
        <sx:when test="record_type='901'">
          <sx:flatRecordType name="header">
            <sx:positionalField name="record_type" width="3"/>
            <sx:positionalField name="trnmtr_nbr" width="8"/>
            <sx:positionalField name="sbmt_ref_id" width="7"/>
            <sx:positionalField name="trnmtr_tcd" width="1"/>
            <sx:positionalField name="summ_cnt" width="6"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="record_type='100'">
          <sx:flatRecordType name="detail">
            <sx:positionalField name="record_type" width="3"/>
            <sx:positionalField name="snm" width="9"/>
            <sx:positionalField name="gvn_nm" width="7"/>
            <sx:positionalField name="init" width="1"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="record_type='301'">
          <sx:flatRecordType name="summary">
            <sx:positionalField name="record_type" width="3"/>
            <sx:positionalField name="bn" width="15"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="CONVToXmlMapping">
    <Submission>
      <sx:onRecord recordType="header">
        <Header>
          <sx:fieldElementMap field="record_type" element="record_type"/>
          <sx:fieldElementMap field="sbmt_ref_id" element="sbmt_ref_id"/>
          <sx:fieldElementMap field="trnmtr_tcd" element="trnmtr_tcd"/>
          <sx:fieldElementMap field="summ_cnt" element="summ_cnt"/>
        </Header>
      </sx:onRecord>
      <sx:innerGroup startTest="sx:current/detail or sx:current/summary"
                     endTest="not(sx:current/detail or sx:current/summary)">
        <All>
          <Level1>
            <sx:onRecord recordType='detail'>
              <Detail>
                <sx:fieldElementMap field="snm" element="snm"/>
                <sx:fieldElementMap field="gvn_nm" element="gvn_nm"/>
                <sx:fieldElementMap field="init" element="init"/>
              </Detail>
            </sx:onRecord>
            <sx:onRecord recordType="summary">
              <Summary>
                <sx:fieldElementMap field="bn" element="bn"/>
              </Summary>
            </sx:onRecord>
          </Level1>
        </All>
      </sx:innerGroup>
    </Submission>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -o output/CONV.xml -r resources-CONV.xml CONV

        

Converting a positional file to XML using a record filter (hot 1)

This is the first of two examples that map the hot.txt flat file to XML.

The input file is a variable record positional file.

Figure 72. Input flat file hot.txt

          
BFH01value01
BCH02value02
BOH03value03
BKT06value04issue
BKP84value05A6USD2
BOT93value06
BOT94value07
BOH03value08
BKT06value09issue
BKP84value10A6CAD2
BOT93value11
BKT06value12refund
BKP84value13A6USD2
BOT93value14
BOT94value15
BCT95value16
BFT99value17

        

This file has variant record types. The first five characters identify the record type, and the next seven characters represent a hypothetical attribute. The BKT06 and BKP84 records have additional fields.

The desired output is shown below.

Figure 73. Output XML file hot1.xml

          
<?xml version="1.0" encoding="utf-8" ?>
<hot>
   <BFH01 attr1="value01"/>
   <BCH02 attr1="value02"/>
   <BOH03 attr1="value03"/>
   <BKT06 attr1="value04"/>
   <BKP84 attr1="value05" amount="1.66" currency="USD"/>
   <BOT93 attr1="value06"/>
   <BOT94 attr1="value07"/>
   <BOH03 attr1="value08"/>
   <BKT06 attr1="value09"/>
   <BKP84 attr1="value10" amount="2.46" currency="CAD"/>
   <BOT93 attr1="value11"/>
   <BKT06 attr1="value12"/>
   <BKP84 attr1="value13" amount="2.27" currency="USD"/>
   <BOT93 attr1="value14"/>
   <BOT94 attr1="value15"/>
   <BCT95 attr1="value16"/>
   <BFT99 attr1="value17"/>
</hot>

        

The following resources script does the transformation.

Figure 74. flatfile-hot.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:flatFile id="hotFlatFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="record-type" width="5"/>
        
        <sx:when  test="record-type='BKP84'">
          <sx:flatRecordType name="bkp84">
            <sx:positionalField name="record-type-prefix"  width="3"/>
            <sx:positionalField name="record-type"  start="1" width="5"/>
            <sx:positionalField name="value" width="7"/>
            <sx:positionalField name="amount" width="2"/>
            <sx:positionalField name="currency" width="3"/>
            <sx:positionalField name="precision" width="1"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="record-type='BKT06'">
          <sx:flatRecordType name="bkt06">
            <sx:positionalField name="record-type-prefix"  width="3"/>
            <sx:positionalField name="record-type"  start="1" width="5"/>
            <sx:positionalField name="value" width="7"/>
            <sx:positionalField name="type" width="6"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:otherwise>
          <sx:flatRecordType name="other">
            <sx:positionalField name="record-type-prefix"  width="3"/>
            <sx:positionalField name="record-type" start="1" width="5"/>
            <sx:positionalField name="value" width="7"/>
          </sx:flatRecordType>
        </sx:otherwise>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>
    
</sx:resources>

        

The values in the bkp84 record need to be transformed as follows:

  • The amount field is encoded as a hexadecimal value and needs to be converted to a decimal value.

  • The resulting decimal value needs to be converted to the specified precision.

In the hot 1 example in this section, we transform the values using a sx:customRecordFilter element to define a custom record filter written in Java. In the hot 2 example in the next section, we take a different approach, transforming the values within an XPath select expression against the record fields, making use of extension calls to static Java functions.

Figure 75. XML Schema file hot-record.xsd

          
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <!-- This element's name matches the value of the name attribute in the "bkp84" sx:flatFileRecordType. -->
  <xs:element name="bkp84" type="bkp84Type"/>
  
  <!-- This element's name matches the value of the name attribute in the "bkt06" sx:flatFileRecordType. -->
  <xs:element name="bkt06" type="bkt06Type"/>
  
  <!-- This element's name matches the value of the name attribute in the "other" sx:flatFileRecordType. -->
  <xs:element name="other" type="otherType"/>

  <xs:complexType name="bkp84Type">
    <xs:sequence>
      <xs:element name="record-type-prefix" type="xs:string"/>
      <xs:element name="record-type" type="xs:string"/>
      <xs:element name="value" type="xs:string"/>
      <xs:element name="amount" type="xs:hexBinary"/>
      <xs:element name="currency" type="xs:string"/>
      <xs:element name="precision" type="xs:nonNegativeInteger"/>
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="bkt06Type">
    <xs:sequence>
      <xs:element name="record-type-prefix" type="xs:string"/>
      <xs:element name="record-type" type="xs:string"/>
      <xs:element name="value" type="xs:string"/>
      <xs:element name="type" type="xs:string"/>
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="otherType">
    <xs:sequence>
      <xs:element name="record-type-prefix" type="xs:string"/>
      <xs:element name="record-type" type="xs:string"/>
      <xs:element name="value" type="xs:string"/>
    </xs:sequence>
  </xs:complexType>
</xs:schema>

        

Figure 76. Resources script resources-hot1.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:include href="flatfile-hot.xml"/>

  <sx:service id="hot1">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="hot1"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="hot1">
    <sx:flatFileReader>
      <sx:flatFile ref="hotFlatFile"/>
    </sx:flatFileReader>
    <msv:recordValidator>
      <sx:urlSource url="data/hot-record.xsd"/>
    </msv:recordValidator>
    <sx:customRecordFilter class="flat2xml.HotRecordFilter"/>
    <sx:discardHandler>
      <sx:log message="{$sx:message}"/>
    </sx:discardHandler>
    <sx:recordMapping ref="hot1ToXmlMapping"/>
  </sx:recordContent>

  <sx:recordMapping id="hot1ToXmlMapping">
    <hot>
      <sx:onRecord recordType="bkp84">
        <sx:elementMap element="{record-type}">
          <sx:fieldAttributeMap field="value" attribute="attr1"/>
          <sx:fieldAttributeMap field="calculatedAmount"  attribute="amount"/>
          <sx:fieldAttributeMap field="currency" attribute="currency"/>
        </sx:elementMap>
      </sx:onRecord>
      <sx:onRecord recordType="bkt06">
        <sx:elementMap element="{record-type}">
          <sx:fieldAttributeMap field="value" attribute="attr1"/>
        </sx:elementMap>
      </sx:onRecord>
      <sx:onRecord recordType="other">
        <sx:elementMap element="{record-type}">
          <sx:fieldAttributeMap field="value" attribute="attr1"/>
        </sx:elementMap>
      </sx:onRecord>
    </hot>
  </sx:recordMapping>
    
</sx:resources>

        

Figure 77. HotRecordFilter class

          
package flat2xml;

import com.servingxml.app.ServiceContext;
import com.servingxml.components.recordio.AbstractRecordFilter;
import com.servingxml.app.Flow;
import com.servingxml.util.Name;
import com.servingxml.util.QualifiedName;
import com.servingxml.util.ServingXmlException;
import com.servingxml.util.record.Record;
import com.servingxml.util.record.RecordBuilder;

public class HotRecordFilter extends AbstractRecordFilter {
  private static final Name BKP84_RECORD_TYPE = new QualifiedName("bkp84");
  private static final Name AMOUNT_NAME = new QualifiedName("amount");
  private static final Name PRECISION_NAME = new QualifiedName("precision");
  private static final Name CALCULATED_AMOUNT_NAME = new QualifiedName("calculatedAmount");
  
  public void writeRecord(ServiceContext context, Flow flow) 
   {

    Record record = flow.getRecord();
    Flow newFlow = flow;
    if (record.getRecordType().getName().equals(BKP84_RECORD_TYPE)) {
      RecordBuilder recordBuilder = new RecordBuilder(record);
      String amountString = record.getString(AMOUNT_NAME);
      if (amountString == null) {
        throw new ServingXmlException("amount is NULL");
      }
      String precisionString = record.getString(PRECISION_NAME);
      if (precisionString == null) {
        throw new ServingXmlException("precision is NULL");
      }
      int amount = Integer.parseInt(amountString,16);
      int precision = Integer.parseInt(precisionString);
      double calculatedAmount = (double)amount/Math.pow(10.0,(double)precision);
      recordBuilder.setDouble(CALCULATED_AMOUNT_NAME,calculatedAmount);
      record = recordBuilder.toRecord();
      newFlow = flow.replaceRecord(context, record);
    }

    super.writeRecord(context, newFlow);
  }
}  

You can run this example on the command line by

  • Compiling the Java class HotRecordFilter and copying the resulting .class file into the dir/classes directory
  • Entering the command
                  
    servingxml -r resources-hot1.xml -i data/hot.txt -o output/hot1.xml hot1
    
                

Converting a positional file to XML with outer and inner grouping (hot 2)

This version of the hot example shows how to convert a flat file to XML document with outer and inner grouping. Each row in the flat file is validated with Sun's multi schema validator, and the row is discarded if an error is encountered.

This examples illustrates the sx:innerGroup and sx:outerGroup elements. Both elements use the startTest and endTest attributes to define the beginning and end of a group through XPath boolean expressions applied to records. They differ in that sx:innerGroup will never allow any descendent elements to be produced unless a group is recognized at its level, whereas a sx:outerGroup will only suppress content between this element and the next succeeding grouping element.

The resources-hot2.xml file defines the record mapping for this version of the hot example. The BFH01 record type marks the beginning of a group of level 1, the BFT99 s mark the end. The BCH02 s mark the beginning of a group of level 2, the BCT95 s mark the end. The BOH03 marks the beginning of a group of level 3, the BOT94s mark the end.

The desired output is shown below.

Figure 78. Output XML file hot2.xml

          
<?xml version="1.0" encoding="utf-8" ?>
<hot>
   <BFH01 attr1="value01"/>
   <periods>
      <period>
         <BCH02s>
            <BCH02 attr1="value02"/>
         </BCH02s>
         <agents>
            <agent>
               <BOH03 attr1="value03"/>
               <issues>
                  <issue>
                     <BKT06 attr1="value04"/>
                     <BKP84 attr1="value05" amount="1.66" currency="USD"/>
                  </issue>
                  <BOT93s>
                     <BOT93 attr1="value06"/>
                  </BOT93s>
               </issues>
               <BOT94s>
                  <BOT94 attr1="value07"/>
               </BOT94s>
            </agent>
            <agent>
               <BOH03 attr1="value08"/>
               <issues>
                  <issue>
                     <BKT06 attr1="value09"/>
                     <BKP84 attr1="value10" amount="2.46" currency="CAD"/>
                  </issue>
                  <BOT93s>
                     <BOT93 attr1="value11"/>
                  </BOT93s>
               </issues>
               <refunds>
                  <refund>
                     <BKT06 attr1="value12"/>
                     <BKP84 attr1="value13" amount="2.27" currency="USD"/>
                  </refund>
                  <BOT93s>
                     <BOT93 attr1="value14"/>
                  </BOT93s>
               </refunds>
               <BOT94s>
                  <BOT94 attr1="value15"/>
               </BOT94s>
            </agent>
         </agents>
         <BCT95s>
            <BCT95 attr1="value16"/>
         </BCT95s>
      </period>
   </periods>
   <BFT99s>
      <BFT99 attr1="value17"/>
   </BFT99s>
</hot>

        

The following resources script does the transformation.

Figure 79. Resources script resources-hot2.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv"
              xmlns:integer="java.lang.Integer" xmlns:math="java.lang.Math">

  <sx:include href="flatfile-hot.xml"/>

  <sx:service id="hot2">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="hot2"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="hot2">
    <sx:flatFileReader>
      <sx:flatFile ref="hotFlatFile"/>
    </sx:flatFileReader>
    <msv:recordValidator>
      <sx:urlSource url="data/hot-record.xsd"/>
    </msv:recordValidator>
    <sx:discardHandler>
      <sx:log message="{$sx:message}"/>
    </sx:discardHandler>
    <sx:recordMapping ref="hot2ToXmlMapping"/>
  </sx:recordContent>

  <sx:recordMapping id="hot2ToXmlMapping">
    <hot>
      <sx:outerGroup startTest="sx:previous//record-type='BFH01'" 
                     endTest="sx:current//record-type='BFT99'">
        <periods>
          <sx:outerGroup startTest="sx:current//record-type='BCH02'"
                         endTest="sx:previous//record-type='BCT95'">
            <period>
              <sx:outerGroup startTest="sx:previous//record-type='BCH02'" 
                             endTest="sx:current//record-type='BCT95'">
                <agents>
                  <sx:outerGroup startTest="sx:current//record-type='BOH03'" 
                                 endTest="sx:previous//record-type='BOT94'">
                    <agent>
                      <sx:groupChoice>
                        <sx:innerGroup startTest="sx:current//record-type-prefix='BFH'" 
                                       endTest="sx:previous//record-type-prefix='BFH'">
                          <sx:onRecord>
                            <sx:elementMap element="{record-type}">
                              <sx:fieldAttributeMap field="value" attribute="attr1"/>
                            </sx:elementMap>
                          </sx:onRecord>
                        </sx:innerGroup>
                        <sx:innerGroup startTest="sx:current//record-type-prefix='BOH'" 
                                       endTest="sx:previous//record-type-prefix='BOH'">
                          <sx:onRecord>
                            <sx:elementMap element="{record-type}">
                              <sx:fieldAttributeMap field="value" attribute="attr1"/>
                            </sx:elementMap>
                          </sx:onRecord>
                        </sx:innerGroup>
                        <sx:innerGroup startTest="sx:current//record-type-prefix='BFT'" 
                                       endTest="sx:previous//record-type-prefix='BFT'">
                          <sx:elementMap element="{record-type}s">
                            <sx:onRecord>
                              <sx:elementMap element="{record-type}">
                                <sx:fieldAttributeMap field="value" attribute="attr1"/>
                              </sx:elementMap>
                            </sx:onRecord>
                          </sx:elementMap>
                        </sx:innerGroup>
                        <sx:innerGroup startTest="sx:current//record-type-prefix='BCT'" 
                                       endTest="sx:previous//record-type-prefix='BCT'">
                          <sx:elementMap element="{record-type}s">
                            <sx:onRecord>
                              <sx:elementMap element="{record-type}">
                                <sx:fieldAttributeMap field="value" attribute="attr1"/>
                              </sx:elementMap>
                            </sx:onRecord>
                          </sx:elementMap>
                        </sx:innerGroup>
                        <sx:innerGroup startTest="sx:current//record-type='BOT94'" 
                                       endTest="sx:previous//record-type='BOT94'">
                          <sx:elementMap element="{record-type}s">
                            <sx:onRecord>
                              <sx:elementMap element="{record-type}">
                                <sx:fieldAttributeMap field="value" attribute="attr1"/>
                              </sx:elementMap>
                            </sx:onRecord>
                          </sx:elementMap>
                        </sx:innerGroup>
                        <sx:innerGroup startTest="sx:current//record-type='BCH02'" 
                                       endTest="sx:previous//record-type='BOH03'">
                          <sx:elementMap element="{record-type}s">
                            <sx:onRecord>
                              <sx:elementMap element="{record-type}">
                                <sx:fieldAttributeMap field="value" attribute="attr1"/>
                              </sx:elementMap>
                            </sx:onRecord>
                          </sx:elementMap>
                        </sx:innerGroup>
                        <sx:innerGroup startTest="sx:current//record-type='BKT06'" 
                                       endTest="sx:previous//record-type='BOT93'">
                          <sx:elementMap element="{type}s">
                            <sx:groupChoice>
                              <sx:innerGroup startTest="sx:current//record-type='BKT06'"
                                             endTest="sx:previous//record-type='BKP84'">
                                <sx:elementMap element="{type}">
                                  <sx:onRecord recordType="bkt06">
                                    <sx:elementMap element="{record-type}">
                                      <sx:fieldAttributeMap field="value" attribute="attr1"/>
                                    </sx:elementMap>
                                  </sx:onRecord>
                                  <sx:onRecord recordType="bkp84">
                                    <sx:elementMap element="{record-type}">
                                      <sx:fieldAttributeMap field="value" attribute="attr1"/>
                                      <sx:fieldAttributeMap select="integer:valueOf(amount,16) div math:pow(10,precision)"  attribute="amount"/>
                                      <sx:fieldAttributeMap field="currency" attribute="currency"/>
                                    </sx:elementMap>
                                  </sx:onRecord>
                                </sx:elementMap>
                              </sx:innerGroup>
                              <sx:innerGroup startTest="sx:current//record-type-prefix='BOT'" 
                                             endTest="sx:previous//record-type-prefix='BOT'">
                                <sx:elementMap element="{record-type}s">
                                  <sx:onRecord>
                                    <sx:elementMap element="{record-type}">
                                      <sx:fieldAttributeMap field="value" attribute="attr1"/>
                                    </sx:elementMap>
                                  </sx:onRecord>
                                </sx:elementMap>
                              </sx:innerGroup>
                            </sx:groupChoice>
                          </sx:elementMap>
                        </sx:innerGroup>
                      </sx:groupChoice>
                    </agent>
                  </sx:outerGroup>
                </agents>
              </sx:outerGroup>
            </period>
          </sx:outerGroup>
        </periods>
      </sx:outerGroup>
    </hot>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -r resources-hot2.xml -i data/hot.txt -o output/hot2.xml hot2

        

Converting a positional flat file with variable field widths to XML

The example below shows how to convert a flat file with variable field widths to XML.

The input file is a positional flat file.

Figure 80. Input flat file testWidth.txt

          
TX#00000000010016This is a remarkXXXXXXXXXX
TX#0000000002              
TX#0000000003    YYYYYYYYYY
TX#00000000030004TestZZZZZZZZZZ

        

This file contains an optional positional field called "remarksText". The field size varies and when the field exists it's width is specified using the value of its preceding field called "remarksSize". When no "remarksText" exists, 4 space characters are specified as "remarksSize" in the file directly followed by the value of "followingField".

The desired output is shown below.

Figure 81. Output XML file testWidth.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<Doc>
  <Transaction>
    <Reference>0000000001</Reference>
    <Remarks>This is a remark</Remarks>
    <FollowingField>XXXXXXXXXX</FollowingField>
  </Transaction>
  <Transaction>
    <Reference>0000000002</Reference>
    <Remarks/>
    <FollowingField/>
  </Transaction>
  <Transaction>
    <Reference>0000000003</Reference>
    <Remarks/>
    <FollowingField>YYYYYYYYYY</FollowingField>
  </Transaction>
  <Transaction>
    <Reference>0000000003</Reference>
    <Remarks>Test</Remarks>
    <FollowingField>ZZZZZZZZZZ</FollowingField>
  </Transaction>
</Doc>

        

The following resources script does the transformation.

Figure 82. Resources script resources-testWidth.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="testWidth">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="data"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="data">
    <sx:flatFileReader>
      <sx:flatFile ref="flatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="segmentsToXml"/>
  </sx:recordContent>

  <sx:flatFile id="flatFile">
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="tag" width="3"/>
        <sx:when test="tag='TX#'">
          <sx:flatRecordType name="transaction">
            <sx:positionalField name="recordType" width="3"/>
            <sx:positionalField name="reference" width="10"/>
            <sx:positionalField name="remarksSize" width="4">
              <sx:defaultValue value="0"/>
            </sx:positionalField>
            <sx:positionalField name="remarksText" width="{remarksSize}"/>
            <sx:positionalField name="followingField" width="10"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="segmentsToXml">
    <Doc>
      <sx:innerGroup startTest="sx:current/transaction">
        <Transaction>
          <sx:fieldElementMap field="reference"  element="Reference"/>
          <sx:fieldElementMap field="remarksText" element="Remarks"/>
          <sx:fieldElementMap field="followingField" element="FollowingField"/>
        </Transaction>
      </sx:innerGroup>
    </Doc>
  </sx:recordMapping>
</sx:resources>

        

Note the following.

  • When the "remarksSize" is not specified as shown in row 2 and 3 of the flat file, the value for {remarksSize} must default to 0 . The "remarksSize" positional field definition therefore contains a sx:defaultValue element, which supplies the default.

You can run this example on the command line by entering

          
servingxml -r resources-testWidth.xml -i testWidth.txt -o output/testWidth.xml testWidth

        

Repeat Header Information in the XML

The example below shows how to convert a flat file to XML with header information repeated when processing the other records.

The record input is

Figure 83. Record Input

         
1A
2B
3C
2D
3E

        

The desired output is shown below.

Figure 84. Output XML file repeated_header_info.xml

          
<doc>
  <b attr="B">
    <c attr="C">A</c>
  </b>
  <b attr="D">
    <c attr="E">A</c>
  </b>
</doc>

        

The following resources script does the transformation.

Figure 85. Resources script resources-repeatHeader.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="hut">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="HutContent"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="HutContent">
    <sx:flatFileReader ref="HutFlatFileReader"/>
    <sx:recordMapping ref="hut2ToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="HutFlatFile">
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="record-type" start="1" width="1"/>
        <sx:when test='record-type="1"'>
          <sx:flatRecordType id="R1" name="R1">
            <sx:positionalField name="TOTO" start="2" width="1"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test='record-type="2"'>
          <sx:flatRecordType id="R2" name="R2">
            <sx:positionalField name="TATA" start="2" width="1"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test='record-type="3"'>
          <sx:flatRecordType id="R3" name="R3">
            <sx:positionalField name="TITI" start="2" width="1"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatFileReader id="HutFlatFileReader">
    <sx:inlineSource>1A
2B
3C
2D
3E</sx:inlineSource>
    <sx:flatFile ref="HutFlatFile"/>
  </sx:flatFileReader>

  <sx:recordMapping id="hut2ToXmlMapping">
    <doc>
      <sx:parameter name="temp" value="{TOTO}"/>
      <sx:outerGroup startTest="sx:current/R2">
        <b>
          <sx:fieldAttributeMap value="{TATA}" attribute="attr"/>
          <sx:onRecord recordType="R3">
            <sx:fieldElementMap value="{$temp}" element="c">
              <sx:fieldAttributeMap value="{TITI}" attribute="attr"/>
            </sx:fieldElementMap>
          </sx:onRecord>
        </b>
      </sx:outerGroup>
    </doc>
  </sx:recordMapping>

</sx:resources>

        

Note the parameter declaration for "temp" in the record mapping section. This parameter declaration occurs immediately below the document element, which represents a grouping of all records. Within a group, ServingXML ServingXML binds the parameter name to the record that begins the grouping section, in this case the first header record in the input.

You can run this example on the command line by entering

          
servingxml -r resources-repeatheader.xml -o output/repeated_header_info.xml hut 

        

Fixed Width Fields, Multiple Record Types, Repeating Groups, Line Delimited

Converting fixed position records with repeating groups to XML (segments)

This shows how to convert fixed position records that contain repeatable/optional fixed parts identified by tags.

The input file is a positional file.

Figure 86. Input flat file segments.txt

          
Z01xxxyyyZ02aaaabbccZ02aaaabbccZ01xxxyyy

        

Note the following:

  • The whole record is positional, but contains "segments" that do not have a terminator but a starting tag and a fixed length.
  • Z01 and Z02 are "segment"-tags and a,b,x and y are positional fields within the "segment"

The desired output is shown below.

Figure 87. Output XML file segments.xml

          
<?xml version="1.0" encoding="utf-8"?>
<Recs>
  <Rec>
    <Z01>
      <xxx>xxx</xxx>
      <yyy>yyy</yyy>
    </Z01>
    <Z02>
      <aaaa>aaaa</aaaa>
      <bb>bb</bb>
      <cc>cc</cc>
    </Z02>
    <Z02>
      <aaaa>aaaa</aaaa>
      <bb>bb</bb>
      <cc>cc</cc>
    </Z02>
    <Z01>
      <xxx>xxx</xxx>
      <yyy>yyy</yyy>
    </Z01>
  </Rec>
</Recs>

        

The following resources script does the transformation.

Figure 88. Resources script resources-segments.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="segments">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="data"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="data">
    <sx:flatFileReader>
      <sx:urlSource url="data/segments.txt"/>
      <sx:flatFile ref="flatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="segmentsToXml"/>
  </sx:recordContent>

  <sx:flatFile id="flatFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="rec">
        <sx:repeatingGroup name="segments">
          <sx:flatRecordTypeChoice>
            <sx:positionalField name="tag" width="3"/>
            <sx:when test="tag='Z01'">
              <sx:flatRecordType name="Z01">
                <sx:positionalField name="tag" width="3"/>
                <sx:positionalField name="xxx" width="3"/>
                <sx:positionalField name="yyy" width="3"/>
              </sx:flatRecordType>
            </sx:when>
            <sx:otherwise>
              <sx:flatRecordType name="Z02">
                <sx:positionalField name="tag" width="3"/>
                <sx:positionalField name="aaaa" width="4"/>
                <sx:positionalField name="bb" width="2"/>
                <sx:positionalField name="cc" width="2"/>
              </sx:flatRecordType>
            </sx:otherwise>
          </sx:flatRecordTypeChoice>
        </sx:repeatingGroup>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="segmentsToXml">
    <Recs>
      <sx:onRecord>
        <Rec>
          <sx:subrecordMapping repeatingGroup="segments">
            <sx:onRecord recordType="Z01">
              <Z01>
                <sx:fieldElementMap field="xxx" element="xxx"/>
                <sx:fieldElementMap field="yyy"  element="yyy"/>
              </Z01>
            </sx:onRecord>
            <sx:onRecord recordType="Z02">
              <Z02>
                <sx:fieldElementMap field="aaaa" element="aaaa"/>
                <sx:fieldElementMap field="bb" element="bb"/>
                <sx:fieldElementMap field="cc" element="cc"/>
              </Z02>
            </sx:onRecord>
          </sx:subrecordMapping>
        </Rec>
      </sx:onRecord>
    </Recs>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -o output/segments.xml -r resources-segments.xml 
    segments 

        

Mapping multiple repeating groups to XML (students-fix)

Suppose you have the following positional file of students, their course grades, and their addresses.

Figure 89. Input flat file students-fix.txt

          
JANEENGLC-MATHA+1972BLUECHICAGOILATLANTAGA

        

The file has the following layout.

name 4 characters
subject-grade repeating group, repeats twice
year-born 4 characters
favorite-color 4 characters
address repeating group, repeats twice

You want the XML output to look as follows.

Figure 90. Output XML file students-fix.xml

          
<?xml version="1.0" encoding="utf-8"?>
<StudentGrades>
  <StudentGrade>
    <Name>JANE</Name>
    <SubjectGrade>
      <Subject>ENGL</Subject>
      <Grade>C-</Grade>
    </SubjectGrade>
    <SubjectGrade>
      <Subject>MATH</Subject>
      <Grade>A+</Grade>
    </SubjectGrade>
    <YearBorn>1972</YearBorn>
    <FavoriteColor>BLUE</FavoriteColor>
    <Address>
      <City>CHICAGO</City>
      <State>IL</State>
    </Address>
    <Address>
      <City>ATLANTA</City>
      <State>GA</State>
    </Address>
  </StudentGrade>
</StudentGrades>

        

The required resources script is

Figure 91. Resources script resources-students-fix.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="students">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="students-content"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="students-content">
    <sx:flatFileReader>
      <sx:flatFile ref="students-file"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="students-mapping"/>
  </sx:recordContent>

  <sx:flatFile id="students-file">
    <sx:flatFileBody>
      <sx:flatRecordType name="student">
        <sx:positionalField name="name" width="4"/>
        <sx:repeatingGroup name="grades" count="2">
          <sx:flatRecordType name="subject-grade">
            <sx:positionalField name="subject" width="4"/>
            <sx:positionalField name="grade" width="2"/>
          </sx:flatRecordType>
        </sx:repeatingGroup>
        <sx:positionalField name="year-born" width="4"/>
        <sx:positionalField name="favorite-color" width="4"/>
        <sx:repeatingGroup name="addresses" count="2">
          <sx:flatRecordType name="address">
            <sx:positionalField name="city" width="7"/>
            <sx:positionalField name="state" width="2"/>
          </sx:flatRecordType>
        </sx:repeatingGroup>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="students-mapping">
    <StudentGrades>
      <sx:onRecord>
        <StudentGrade>
          <sx:fieldElementMap field="name" element="Name"/>
          <sx:subrecordMapping repeatingGroup="grades">
            <sx:onRecord>
              <SubjectGrade>
                <sx:fieldElementMap field="subject" element="Subject"/>
                <sx:fieldElementMap field="grade" element="Grade"/>
              </SubjectGrade>
            </sx:onRecord>
          </sx:subrecordMapping>
          <sx:fieldElementMap field="year-born" element="YearBorn"/>
          <sx:fieldElementMap field="favorite-color" element="FavoriteColor"/>
          <sx:subrecordMapping repeatingGroup="addresses">
            <sx:onRecord>
              <Address>
                <sx:fieldElementMap field="city" element="City"/>
                <sx:fieldElementMap field="state" element="State"/>
              </Address>
            </sx:onRecord>
          </sx:subrecordMapping>
        </StudentGrade>
      </sx:onRecord>
    </StudentGrades>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml  -r resources-students-fix.xml -i data/students-fix.txt -o output/students-fix.xml 
    students

        

Fixed Width Fields, Single Record Type, No End-of-Line Delimiters

Converting a flat file with a single record type, but no end-of-line delimiters, to XML (non-delimited book orders)

The example below shows how to convert a flat file with a single record format, but no end-of-line delimiters, to XML.

The input file is a fixed field width flat file.

Figure 92. Input flat file non_delimited_book_orders.txt

          
CAuthor                        Title                              Price InvoiceNo InvoiceDate       FCharles Bukowski              Factotum                           22.95 001  12/Mar/2005            FSergei Lukyanenko             The Night Watch                    17.99 002  13/Mar/2005            FAndrew Crumey                 Mr Mee                             22.00 003 14/June/2005            CSteven John Metsker           Building Parsers with Java         39.95 004 15/June/2005            This is a trailer record                                                                            

        

The desired output is shown below.

Figure 93. Output XML file book_orders.xml

          
<?xml version="1.0" encoding="utf-8"?>
<book-orders>
  <book-type>
    <category>F</category>
    <author>Charles Bukowski</author>
    <title>Factotum</title>
    <price>22.95</price>
    <invoiceno>001</invoiceno>
    <invoicedate>12/Mar/2005</invoicedate>
    <filler/>
  </book-type>
  <book-type>
    <category>F</category>
    <author>Sergei Lukyanenko</author>
    <title>The Night Watch</title>
    <price>17.99</price>
    <invoiceno>002</invoiceno>
    <invoicedate>13/Mar/2005</invoicedate>
    <filler/>
  </book-type>
  <book-type>
    <category>F</category>
    <author>Andrew Crumey</author>
    <title>Mr Mee</title>
    <price>22.00</price>
    <invoiceno>003</invoiceno>
    <invoicedate>14/June/2005</invoicedate>
    <filler/>
  </book-type>
  <book-type>
    <category>C</category>
    <author>Steven John Metsker</author>
    <title>Building Parsers with Java</title>
    <price>39.95</price>
    <invoiceno>004</invoiceno>
    <invoicedate>15/June/2005</invoicedate>
    <filler/>
  </book-type>
</book-orders>

        

The following resources script does the transformation.

Figure 94. Resources script resources-non_delimited_book_orders.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="book-orders"> 
    <sx:serialize>
      <sx:transform>
        <sx:content ref="book-orders"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="book-orders" name="book-orders">
    <sx:flatFileReader>
      <sx:flatFile ref="books-flatfile"/>
    </sx:flatFileReader>
  </sx:recordContent>
  
  <sx:flatFile id="books-flatfile" lineDelimited="false">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="book-type"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="book-type"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord width="100">This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>      

  <sx:flatRecordType id="book-type" name="book-type">
    <sx:positionalField name="category" width="1"/>
    <sx:positionalField name="author" width="30"/>
    <sx:positionalField name="title" width="30"/>
    <sx:positionalField name="price" width="10" justify="right"/>
    <sx:positionalField name="invoiceno" width="4" justify="right"/>
    <sx:positionalField name="invoicedate" width="13" justify="right"/>
    <sx:positionalField name="filler" width="12"/>
  </sx:flatRecordType>
  
</sx:resources>

        

Note the following points.

You can run this example on the command line by entering

          
servingxml -r resources-non_delimited_book_orders.xml 
  -i data/non_delimited_book_orders.txt
  -o output/book_orders.xml  book-orders

        

Fixed Width Fields, Multiple Record Types, No End-of-Line Delimiters

Converting a flat file with multiple record formats, but no end-of-line delimiters, to XML (non-delimited trades)

The example below shows how to convert a flat file with multiple record formats, but no end-of-line delimiters, to XML.

The input file is a fixed field width flat file.

Figure 95. Input flat file non_delimited_variant_trades.txt

          
TR000103/25/2005 1:50:00This is a trade record        TN0002X1234A child transaction           TN0003X1235Another child transaction    

        

The desired output is shown below.

Figure 96. Output XML file trades.xml

          
<?xml version="1.0" encoding="utf-8"?>
<trades>
  <trade id="0001">
    <trade-date>2005-03-25T01:50:00.000-05:00</trade-date>
    <description>This is a trade record</description>
  </trade>
  <transaction id="0002">
    <reference>X1234</reference>
    <description>A child transaction</description>
  </transaction>
  <transaction id="0003">
    <reference>X1235</reference>
    <description>Another child transaction</description>
  </transaction>
</trades>

        

The following resources script does the transformation.

Figure 97. Resources script resources-non_delimited_variant_trades.xm

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="trades">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="trades"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="trades">
    <sx:flatFileReader>
      <sx:urlSource url="data/non_delimited_variant_trades.txt"/>
      <sx:flatFile ref="trades-flatfile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="trades-xml-mapping"/>
  </sx:recordContent>

  <sx:flatFile id="trades-flatfile" name="trades" lineDelimited="false">
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="record_type" width="2"/>
        <sx:when test="record_type='TR'">
          <sx:flatRecordType name="trade">
            <sx:positionalField name="record_type" width="2"/>
            <sx:positionalField name="id" width="4"/>
            <sx:positionalField name="trade_date" width="10"/>
            <sx:positionalField name="trade_time" width="8"/>
            <sx:positionalField name="description" width="30"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="record_type='TN'">
          <sx:flatRecordType name="transaction">
            <sx:positionalField name="record_type" width="2"/>
            <sx:positionalField name="id" width="4"/>
            <sx:positionalField name="reference" width="5"/>
            <sx:positionalField name="description" width="30"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="trades-xml-mapping">
    <trades>
      <sx:onRecord recordType="trade">
        <trade>
          <sx:fieldAttributeMap field="id" attribute="id"/>
          <sx:elementMap element="trade-date">
            <sx:convertToDateTime format="MM/dd/yyyy H:mm:ss">
              <sx:concat separator=" ">
                <sx:toString value="{trade_date}"/>
                <sx:toString value="{trade_time}"/>
              </sx:concat>
            </sx:convertToDateTime>
          </sx:elementMap>
          <sx:fieldElementMap field="description" element="description"/>
        </trade>
      </sx:onRecord>
      <sx:onRecord recordType="transaction">
        <transaction>
          <sx:fieldAttributeMap field="id" attribute="id"/>
          <sx:fieldElementMap field="reference" element="reference"/>
          <sx:fieldElementMap field="description" element="description"/>
        </transaction>
      </sx:onRecord>
    </trades>
  </sx:recordMapping>

</sx:resources>

        

Note the following points.

You can run this example on the command line by entering

          
servingxml -r resources-non_delimited_variant_trades.xml 
  -o output/trades.xml trades 

        

Converting a flat file with special characters to XML (special char)

The example below shows how to convert a flat file with special characters to XML.

The input file is a flat file with two adjacent records (no line delimiters):

  • A record beginning with the hexadecimal value 03, followed by the text "1111111"

  • A record beginning with the hexadecimal value 04, followed by the text "John Smith"

The two hexadecimal values are needed to differentiate the record type, but these valued cannot be written as &#x03; and&#x04; in XML, as these are not valid XML characters.

The desired output is shown below.

Figure 98. Output XML file specialChar.xml

          
<?xml version="1.0" encoding="utf-8"?>
<transaction>
  <R03>
    <firstField>03</firstField>
    <CLIENUM>1111111</CLIENUM>
  </R03>
  <R04>
    <firstField>04</firstField>
    <NAME>John Smith</NAME>
  </R04>
</transaction>

        

The following resources script does the transformation.

Figure 99. Resources script resources-specialChar.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="transaction">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="transactionDoc"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="transactionDoc" name="transaction">
    <sx:flatFileReader>
      <sx:flatFile ref="transactionFile"/>
    </sx:flatFileReader>
  </sx:recordContent>

  <sx:flatFile id="transactionFile" lineDelimited="false">
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:binaryField name="firstField" width="1"/>
        <sx:when test="firstField='03'">
          <sx:flatRecordType id="R03" name="R03">
            <sx:binaryField name="firstField" label="firstField" width="1" />
            <sx:positionalField name="CLIENUM" label="CLIENUM" width="007" />
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="firstField='04'">
          <sx:flatRecordType id="R04" name="R04">
            <sx:binaryField name="firstField" label="firstField" width="1" />
            <sx:positionalField name="NAME" label="NAME" width="020" />
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>
</sx:resources>

        

Note that the first field of each record is declared as an sx:binaryField . The string rendering of this field is as an hexidecimal value.

You can run this example on the command line by entering

          
servingxml  -r resources-specialChar.xml -i data/specialChar.txt 
  -o output/specialChar.xml transaction

        

Converting a flat file containing Cobol packed decimal data to XML (packed decimal to XML)

The example below shows how to convert a non-delimited flat file containing Cobol packed decimal data to XML.

The input file, shown below, is encoded in ebcdic, except for the price field, which is expressed as a Cobol packed decimal number 99999999.99 .

Figure 100. Input positional flat file books.txt

          
CAuthor                        Title                              Price

FCharles Bukowski              Factotum                           22.95
FSergei Lukyanenko             The Night Watch                    17.99
FAndrew Crumey                 Mr Mee                             22.00
CSteven John Metsker           Building Parsers with Java         39.95

This is a trailer record

        

The desired output is shown below.

Figure 101. Output XML file books.xml

          
<?xml version="1.0" encoding="utf-8"?>
<myns:books xmlns:myns="http://mycompany.com/mynames/" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="url2">
  <myns:book categoryCode="F">
    <myns:title>Factotum</myns:title>
    <myns:author>Charles Bukowski</myns:author>
    <myns:price>22.95</myns:price>
  </myns:book>
  <myns:book categoryCode="F">
    <myns:title>The Night Watch</myns:title>
    <myns:author>Sergei Lukyanenko</myns:author>
    <myns:price>17.99</myns:price>
  </myns:book>
  <myns:book categoryCode="F">
    <myns:title>Mr Mee</myns:title>
    <myns:author>Andrew Crumey</myns:author>
    <myns:price>22.00</myns:price>
  </myns:book>
  <myns:book categoryCode="C">
    <myns:title>Building Parsers with Java</myns:title>
    <myns:author>Steven John Metsker</myns:author>
    <myns:price>39.95</myns:price>
  </myns:book>
</myns:books>

        

The following resources script does the transformation.

Figure 102. Resources script resources-books_ebcdic_packed.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="books-ebcdic-packed2xml"> 
    <sx:serialize>
      <sx:transform>
        <sx:content ref="books"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>
  
  <!--
  This sx:flatFileReader element does not specify a stream source, so 
  the source will default to the file specified with the -i option on the command line.
  -->
  <sx:recordContent id="books">
    <sx:flatFileReader>
      <sx:flatFile ref="booksFile"/>
      <sx:defaultStreamSource encoding="Cp1047"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="booksToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="booksFile" lineDelimited="false">
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
  </sx:flatFile>      

  <sx:flatRecordType id="bookType" name="bookType">
    <sx:positionalField name="category" label="Category" width="1"/>
    <sx:positionalField name="author" label="Author" width="30"/>
    <sx:positionalField name="title" label="Title" width="30"/>
    <sx:packedDecimalField name="price" label="Price" digitCount="10" decimalPlaces="2"/>
  </sx:flatRecordType>
  
  <sx:recordMapping id="booksToXmlMapping">
    <myns:books xmlns:myns="http://mycompany.com/mynames/" 
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     xsi:schemaLocation="url2">
      <sx:onRecord>
        <myns:book>
          <sx:fieldAttributeMap field="category" attribute="categoryCode"/>
          <sx:fieldElementMap field="title" element="myns:title"/>  
          <sx:fieldElementMap field="author" element="myns:author"/>
          <sx:fieldElementMap field="price" element="myns:price"/>
        </myns:book>  
      </sx:onRecord>
    </myns:books>
  </sx:recordMapping>  
  
</sx:resources>

        

Note that the last field of each record is declared as an sx:packedDecimalField .

You can run this example on the command line by entering

          
servingxml  -r resources-books_ebcdic_packed.xml 
  -i data/books_ebcdic_packed.dat 
  -o  output/books.xml    books-ebcdic-packed2xml

        

Logical Record Consists of Multiple Physical Records

Converting a System Management Facilities (SMF) file to XML (Segment Concatenation)

The example below shows how to convert a System Management Facilities (SMF) file to XML. The example includes support for variable block spanned (VBS) records. The format of an SMF file is described in SMF records.

The desired output is shown below.

Figure 103. Output XML file smf.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<smf>
   <smfRecord>
      <systemIndicator>1E</systemIndicator>
      <type>2</type>
   </smfRecord>
   <smfRecord>
      <systemIndicator>DE</systemIndicator>
      <type>110</type>
   </smfRecord>
   <smfRecord>
      <systemIndicator>1E</systemIndicator>
      <type>64</type>
      <jobName>SMF</jobName>
   </smfRecord>
   <smfRecord>
      <systemIndicator>5E</systemIndicator>
      <type>42</type>
      <subType>6</subType>
   </smfRecord>
   <smfRecord>
      <systemIndicator>1E</systemIndicator>
      <type>64</type>
      <jobName>GPL1</jobName>
   </smfRecord>
   <smfRecord>
      <systemIndicator>1E</systemIndicator>
      <type>64</type>
      <jobName>GPL1</jobName>
   </smfRecord>
   <smfRecord>
      <systemIndicator>1E</systemIndicator>
      <type>64</type>
      <jobName>GPL1</jobName>
   </smfRecord>
   <smfRecord>
      <systemIndicator>1E</systemIndicator>
      <type>3</type>
   </smfRecord>
</smf>

        

The following resources script does the transformation.

Figure 104. Resources script resources-smf.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:xs="http://www.w3.org/2001/XMLSchema"
              xmlns:fn="http://www.w3.org/2005/xpath-functions">

  <sx:service id="smf-xml">
    <sx:serialize>
      <sx:xsltSerializer>
        <sx:outputProperty name="indent" value="yes"/>
      </sx:xsltSerializer>
      <sx:transform>
        <sx:content ref="smf"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="smf">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="smfFile"/>
        <sx:defaultStreamSource encoding="Cp1047"/>
      </sx:flatFileReader>
    </sx:recordStream>
    <sx:recordMapping ref="smfToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="smfFile" lineDelimited="false" countPositionsInBytes="true">
    <sx:flatFileBody>
      <sx:flatRecordType ref="smfType"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:vbsFlatRecordType id="smfType">
    <sx:integerField name="recordLength" width="2"/>
    <sx:integerField name="segmentControlCode" width="1"/>
    <sx:integerField name="reserved" width="1"/>
    <sx:mergePhysicalSegments startTest = "number(sx:current//segmentControlCode) = 0 
                                          or number(sx:current//segmentControlCode) = 1"
                             segmentLength="{recordLength}">
      <sx:flatRecordTypeChoice>
        <sx:integerField name="recordLength" width="2"/>
        <sx:integerField name="segmentControlCode" width="1"/>
        <sx:integerField name="reserved" width="1"/>
        <sx:binaryField name="systemIndicator" width="1"/>
        <sx:integerField name="recordType" width="1"/>
        <sx:when test="recordType=64">
          <sx:flatRecordType name="smfType64">
            <sx:integerField name="recordLength" width="2"/>
            <sx:integerField name="segmentControlCode" width="1"/>
            <sx:integerField name="reserved" width="1"/>
            <sx:binaryField name="systemIndicator" label="systemIndicator" width="1"/>
            <sx:integerField name="recordType" label="recordType" width="1"/>
            <sx:positionalField name="jobName" label="Job Name" start="19" width="8"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="recordType=42">
          <sx:flatRecordType name="smfType42">
            <sx:integerField name="recordLength" width="2"/>
            <sx:integerField name="segmentControlCode" width="1"/>
            <sx:integerField name="reserved" width="1"/>
            <sx:binaryField name="systemIndicator" label="systemIndicator" width="1"/>
            <sx:integerField name="recordType" label="recordType" width="1"/>
            <sx:integerField name="subType" label="Sub Type" start="23" width="2"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="recordType=110">
          <sx:flatRecordType name="smfType110">
            <sx:integerField name="recordLength" width="2"/>
            <sx:integerField name="segmentControlCode" width="1"/>
            <sx:integerField name="reserved" width="1"/>
            <sx:binaryField name="systemIndicator" label="systemIndicator" width="1"/>
            <sx:integerField name="recordType" label="recordType" width="1"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:otherwise>
          <sx:flatRecordType name="other">
            <sx:integerField name="recordLength" width="2"/>
            <sx:integerField name="segmentControlCode" width="1"/>
            <sx:integerField name="reserved" width="1"/>
            <sx:binaryField name="systemIndicator" label="systemIndicator" width="1"/>
            <sx:integerField name="recordType" label="recordType" width="1"/>
          </sx:flatRecordType>
        </sx:otherwise>
      </sx:flatRecordTypeChoice>
    </sx:mergePhysicalSegments>
  </sx:vbsFlatRecordType>

  <sx:recordMapping id="smfToXmlMapping">
    <smf>
      <sx:onRecord recordType="smfType64">
        <smfRecord>
          <sx:fieldElementMap field="systemIndicator" element="systemIndicator"/>
          <sx:fieldElementMap field="recordType" element="type"/>
          <sx:fieldElementMap field="jobName" element="jobName"/>
        </smfRecord>
      </sx:onRecord>
      <sx:onRecord recordType="smfType42">
        <smfRecord>
          <sx:fieldElementMap field="systemIndicator" element="systemIndicator"/>
          <sx:fieldElementMap field="recordType" element="type"/>
          <sx:fieldElementMap field="subType" element="subType"/>
        </smfRecord>
      </sx:onRecord>
      <sx:onRecord recordType="smfType110">
        <smfRecord>
          <sx:fieldElementMap field="systemIndicator" element="systemIndicator"/>
          <sx:fieldElementMap field="recordType" element="type"/>
        </smfRecord>
      </sx:onRecord>
      <sx:onRecord recordType="other">
        <smfRecord>
          <sx:fieldElementMap field="systemIndicator" element="systemIndicator"/>
          <sx:fieldElementMap field="recordType" element="type"/>
        </smfRecord>
      </sx:onRecord>
    </smf>
  </sx:recordMapping>

</sx:resources> 

Remarks
  • The field definitions that appear as children of sx:vbsFlatRecordType comprise the 4-byte segment descriptor word (SDW), which describes the segment. The first 2 bytes contain the length of the segment, including the 4-byte SDW. The third byte of the SDW contains the segment control code, a value of 00 indicates that the segment is a complete logical record, and 01 indicates that the segment is the first segment of a multisegment record. For more details, see segment descriptor word.

  • The sx:mergePhysicalSegments element controls the composition of the mulitple segments into a single logical record. Note that a segmentControlCode of either 00 or 01 marks the beginning of a logical record. By default, the first segment descriptor word goes at the front of the logical record, in the place of a record descriptor word (RDW), subsequent segment descriptor words are discarded. If the optional suppressRDW attribute is set to "true", the first segment descriptor word is discarded as well, the SDW fields must be removed from the content of sx:mergePhysicalSegments, and four bytes need to be subtracted from any positions specified in the record definitions.

  • Note that ServingXML positions are one-based, while the offsets in the IBM documentation are zero-based.

You can run this example on the command line by entering

          
servingxml -r resources-smf.xml -i data/G6744V00.smf -o output/smf.xml smf-xml

        

Aggregating Multiple Physical Records into a Composite Record (Record Composition)

The input file is a fixed field width flat file.

Figure 105. Input flat file composite-records.txt

          
ODANIEL    356996699001 
A233ServingXML132433SAS 
B343443DD2322ARDE021101 
CDANIEL    029299839907    
B344233DD2322PREE021100 
D0000040000002300000001 
A233ServingXML132433AGS 
B3434432323000022021101 
CDANIEL    029299839907 
B3433132323000022021101 
CDANIEL    029299839907 
D0000040000002300000002 
E0030000000100009803600 

        

Note the following properties of this data:

  • The file begins with an "O" record and ends with an "E" record.
  • An "A" record begins a group of records.
  • The last three character field of an A record (e.g. "SAS") determines a variant format for a B record.

The desired output is shown below.

Figure 106. Output XML file composite-records.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<document>
  <header>
    <O>
      <recordType>O</recordType>
      <name>DANIEL</name>
      <id>356</id>
      <code>996699</code>
      <tracking>001</tracking>
    </O>
  </header>
  <body>
    <A>
      <recordType>A</recordType>
      <code>233</code>
      <book>ServingXML</book>
      <isbn>132433</isbn>
      <identifier>SAS</identifier>
    </A>
    <B-SAS>
      <recordType>B</recordType>
      <library>343443</library>
      <shelf>DD2322ARDE</shelf>
      <ebook>021</ebook>
      <code>101</code>
    </B-SAS>
    <C>
      <recordType>C</recordType>
      <author>DANIEL</author>
      <publisher>0292998399</publisher>
      <flag>07</flag>
    </C>
    <B-SAS>
      <recordType>B</recordType>
      <library>344233</library>
      <shelf>DD2322PREE</shelf>
      <ebook>021</ebook>
      <code>100</code>
    </B-SAS>
    <D>
      <recordType>D</recordType>
      <count>000004</count>
      <amount>0000002300</amount>
      <tracking>000001</tracking>
    </D>
    <A>
      <recordType>A</recordType>
      <code>233</code>
      <book>ServingXML</book>
      <isbn>132433</isbn>
      <identifier>AGS</identifier>
    </A>
    <B-AGS>
      <recordType>B</recordType>
      <store>34344323</store>
      <cost>23000022</cost>
      <code>021</code>
      <make>101</make>
    </B-AGS>
    <C>
      <recordType>C</recordType>
      <author>DANIEL</author>
      <publisher>0292998399</publisher>
      <flag>07</flag>
    </C>
    <B-AGS>
      <recordType>B</recordType>
      <store>34331323</store>
      <cost>23000022</cost>
      <code>021</code>
      <make>101</make>
    </B-AGS>
    <C>
      <recordType>C</recordType>
      <author>DANIEL</author>
      <publisher>0292998399</publisher>
      <flag>07</flag>
    </C>
    <D>
      <recordType>D</recordType>
      <count>000004</count>
      <amount>0000002300</amount>
      <tracking>000002</tracking>
    </D>
  </body>
  <trailer>
    <E>
      <recordType>E</recordType>
      <total>00300</total>
      <amount>0000010000</amount>
      <count>9803</count>
      <tracking>600</tracking>
    </E>
  </trailer>
</document>

        

The following resources script does the transformation.

Figure 107. Resources script resources-recordComposition.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:fn="http://www.w3.org/2005/xpath-functions">

  <sx:service id="composite-records-to-xml">
    <sx:task ref="serialized-composite-records"/>
  </sx:service>

  <sx:serialize id="serialized-composite-records">
    <sx:transform>
      <sx:recordContent ref="composite-record-content"/>
    </sx:transform>
  </sx:serialize>

  <sx:recordContent id="composite-record-content">
    <sx:flatFileReader ref="composite-record-reader"/>
    <sx:recordMapping ref="composite-record-mapping"/>
  </sx:recordContent>

  <sx:flatFileReader id="composite-record-reader">
    <sx:flatFile>
      <sx:flatFileBody>
        <sx:flatRecordTypeChoice>
          <sx:positionalField name="recordType" width="1"/>
          <sx:when test="recordType='O'">
            <sx:flatRecordType ref="O"/>
          </sx:when>
          <sx:when test="recordType='E'">
            <sx:flatRecordType ref="E"/>
          </sx:when>
          <sx:otherwise>
            <sx:compositeFlatRecordType>
              <sx:positionalField name="recordType" width="1"/>
              <sx:positionalField name="code" width="3"/>
              <sx:positionalField name="book" width="10"/>
              <sx:positionalField name="isbn" width="6"/>
              <sx:positionalField name="identifier" width="3"/>
              <sx:combinePhysicalRecords recordType="composite" repeatingGroup="group"
                                    startTest = "sx:current//recordType = 'A' 
                                                 and sx:current//identifier = 'SAS'"
                                    endTest = "sx:current//recordType = 'A'">
                <sx:flatRecordTypeChoice>
                  <sx:positionalField name="recordType" width="1"/>
                  <sx:when test="recordType='A'">
                    <sx:flatRecordType ref="A"/>
                  </sx:when>
                  <sx:when test="recordType='B'">
                    <sx:flatRecordType ref="B-SAS"/>
                  </sx:when>
                  <sx:when test="recordType='C'">
                    <sx:flatRecordType ref="C"/>
                  </sx:when>
                  <sx:when test="recordType='D'">
                    <sx:flatRecordType ref="D"/>
                  </sx:when>
                </sx:flatRecordTypeChoice>
              </sx:combinePhysicalRecords>
              <sx:combinePhysicalRecords recordType="composite" repeatingGroup="group"
                                    startTest = "sx:current//recordType = 'A' 
                                                 and sx:current//identifier = 'AGS'"
                                    endTest = "sx:current//recordType = 'A'">
                <sx:flatRecordTypeChoice>
                  <sx:positionalField name="recordType" width="1"/>
                  <sx:when test="recordType='A'">
                    <sx:flatRecordType ref="A"/>
                  </sx:when>
                  <sx:when test="recordType='B'">
                    <sx:flatRecordType ref="B-AGS"/>
                  </sx:when>
                  <sx:when test="recordType='C'">
                    <sx:flatRecordType ref="C"/>
                  </sx:when>
                  <sx:when test="recordType='D'">
                    <sx:flatRecordType ref="D"/>
                  </sx:when>
                </sx:flatRecordTypeChoice>
              </sx:combinePhysicalRecords>
            </sx:compositeFlatRecordType>
          </sx:otherwise>
        </sx:flatRecordTypeChoice>
      </sx:flatFileBody>
    </sx:flatFile>
  </sx:flatFileReader>

  <sx:flatRecordType id="A" name="A">
    <sx:positionalField name="recordType" width="1"/>
    <sx:positionalField name="code" width="3"/>
    <sx:positionalField name="book" width="10"/>
    <sx:positionalField name="isbn" width="6"/>
    <sx:positionalField name="identifier" width="3"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="C" name="C">
    <sx:positionalField name="recordType" width="1"/>
    <sx:positionalField name="author" width="10"/>
    <sx:positionalField name="publisher" width="10"/>
    <sx:positionalField name="flag" width="2"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="D" name="D">
    <sx:positionalField name="recordType" width="1"/>
    <sx:positionalField name="count" width="6"/>
    <sx:positionalField name="amount" width="10"/>
    <sx:positionalField name="tracking" width="6"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="E" name="E">
    <sx:positionalField name="recordType" width="1"/>
    <sx:positionalField name="total" width="5"/>
    <sx:positionalField name="amount" width="10"/>
    <sx:positionalField name="count" width="4"/>
    <sx:positionalField name="tracking" width="3"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="B-SAS" name="B-SAS">
    <sx:positionalField name="recordType" width="1"/>
    <sx:positionalField name="library" width="6"/>
    <sx:positionalField name="shelf" width="10"/>
    <sx:positionalField name="ebook" width="3"/>
    <sx:positionalField name="code" width="3"/>
  </sx:flatRecordType>
  <sx:flatRecordType id="B-AGS" name="B-AGS">
    <sx:positionalField name="recordType" width="1"/>
    <sx:positionalField name="store" width="8"/>
    <sx:positionalField name="cost" width="8"/>
    <sx:positionalField name="code" width="3"/>
    <sx:positionalField name="make" width="3"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="O" name="O">
    <sx:positionalField name="recordType" width="1"/>
    <sx:positionalField name="name" width="10"/>
    <sx:positionalField name="id" width="3"/>
    <sx:positionalField name="code" width="6"/>
    <sx:positionalField name="tracking" width="3"/>
  </sx:flatRecordType>

  <sx:recordMapping id="composite-record-mapping">
    <document>
      <header>
        <sx:onRecord recordType="O">
          <O>
            <sx:defaultFieldElementMap fields="*"/>
          </O>
        </sx:onRecord>
      </header>
      <body>
        <sx:onRecord recordType="composite">
          <sx:subrecordMapping repeatingGroup="group">
            <sx:onRecord recordType="A">
              <A>
                <sx:defaultFieldElementMap fields="*"/>
              </A>
            </sx:onRecord>
            <sx:onRecord recordType="B-SAS">
              <B-SAS>
                <sx:defaultFieldElementMap fields="*"/>
              </B-SAS>
            </sx:onRecord>
            <sx:onRecord recordType="B-AGS">
              <B-AGS>
                <sx:defaultFieldElementMap fields="*"/>
              </B-AGS>
            </sx:onRecord>
            <sx:onRecord recordType="C">
              <C>
                <sx:defaultFieldElementMap fields="*"/>
              </C>
            </sx:onRecord>
            <sx:onRecord recordType="D">
              <D>
                <sx:defaultFieldElementMap fields="*"/>
              </D>
            </sx:onRecord>
            <sx:onRecord recordType="O">
              <O>
                <sx:defaultFieldElementMap fields="*"/>
              </O>
            </sx:onRecord>
          </sx:subrecordMapping>
        </sx:onRecord>
      </body>
      <trailer>
        <sx:onRecord recordType="E">
          <E>
            <sx:defaultFieldElementMap fields="*"/>
          </E>
        </sx:onRecord>
      </trailer>
    </document>
  </sx:recordMapping>

</sx:resources>

Remarks
  • The field definitions that appear as children of sx:compositeFlatRecordType belong to the header of a group of related records.

  • The sx:combinePhysicalRecords element controls the aggregation of a group of physical records into a composite logical record.

You can run this example on the command line by entering

          
servingxml -r resources-recordComposition.xml -i data/composite-records.txt 
    -o output/composite-records.xml composite-records-to-xml

        

Computing an Aggregate Value over Multiple Records

This example shows a resources script that will concatenate the values of several records into a single XML element value. It illustrates the use of the sx:combineRecords element.

The input file is

Figure 108. Input flat file orders.txt

          
HEADER
ORDER/CUS/CON/USD/
PO1/PAR1/MFR FADD1/5/
REM AAAAA
REM BBBBB
REM CCCCC
PO2/PAR2/MFR 30066/1/
PO3/PAR3/MFR PAGT2/2/
REM CCCCC
REM DDDDD
TRAILER

        

The desired output is

Figure 109. Output XML file order.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<Order>
  <OrderHeader>
    <CUS>CUS</CUS>
    <CON>CON</CON>
    <CUR>USD</CUR>
  </OrderHeader>
  <OrderPosition>
    <ORN>PO1</ORN>
    <PAR>PAR1</PAR>
    <QTY>5</QTY>
    <MFR>FADD1</MFR>
    <REM>AAAAA BBBBB CCCCC</REM>
  </OrderPosition>
  <OrderPosition>
    <ORN>PO2</ORN>
    <PAR>PAR2</PAR>
    <QTY>1</QTY>
    <MFR>30066</MFR>
  </OrderPosition>
  <OrderPosition>
    <ORN>PO3</ORN>
    <PAR>PAR3</PAR>
    <QTY>2</QTY>
    <MFR>PAGT2</MFR>
    <REM>CCCCC DDDDD</REM>
  </OrderPosition>
</Order>

        

Note that the REM values are concatenated in a space separated list.

The following resources script does the transformation.

Figure 110. Resources script resources-order.xml

          
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:fn="http://www.w3.org/2005/xpath-functions">

  <sx:service id="Order">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="Order"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="Order" name="Order">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="OrderFile"/>
      </sx:flatFileReader>
      <sx:combineRecords recordType="composite" repeatingGroup="detail"
                        startTest="sx:current/orderRemarks"
                        endTest="not(sx:current/orderRemarks)">
        <sx:newField name="remarks" select="detail/orderRemarks/REM"/>
      </sx:combineRecords>
    </sx:recordStream>
    <sx:recordMapping ref="OrderToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="OrderFile">
    <sx:commentStarter value="#"/>
    <sx:fieldDelimiter value="/"/>
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="tag" width="3"/>
        <sx:when test="tag='ORD'">
          <sx:flatRecordType name="orderHeader">
            <sx:delimitedField name="CMD" label="Command Code"/>
            <sx:delimitedField name="CUS" label="Customer"/>
            <sx:delimitedField name="CON" label="Contractor"/>
            <sx:delimitedField name="CUR" label="Currency Code"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='REM'">
          <sx:flatRecordType name="orderRemarks">
            <sx:positionalField name="REMTEI" label="Remark" width="3"/>
            <sx:delimitedField name="REM" label="Remark"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='TRA'">
          <sx:flatRecordType name="orderTrailer">
          </sx:flatRecordType>
        </sx:when>
        <sx:otherwise>
          <sx:flatRecordType name="orderPosition">
            <sx:delimitedField name="ORN" label="Order Number"/>
            <sx:delimitedField name="PAR" label="Part Number"/>
            <sx:delimitedField name="MFRTEI" maxWidth="3"/>
            <sx:delimitedField name="MFR" label="Manufacturer"/>
            <sx:delimitedField name="QTY" label="Order Quantity"/>
          </sx:flatRecordType>
        </sx:otherwise>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="OrderToXmlMapping">
    <Order>
      <sx:innerGroup startTest="sx:current/orderHeader">
        <OrderHeader>
          <sx:fieldElementMap field="CUS" element="CUS"/>
          <sx:fieldElementMap field="CON" element="CON"/>
          <sx:fieldElementMap field="CUR" element="CUR"/>
        </OrderHeader>

        <sx:innerGroup startTest="sx:current/orderPosition">
          <OrderPosition>
            <sx:fieldElementMap field="ORN" element="ORN" minOccurs="1"/>
            <sx:fieldElementMap field="PAR" element="PAR"/>
            <sx:fieldElementMap field="QTY" element="QTY"/>
            <sx:fieldElementMap field="MFR" element="MFR" minOccurs="0"/>
            <sx:onRecord recordType="composite">
              <sx:choose>
                <sx:when test="remarks != ''">
                  <sx:fieldElementMap select="remarks" element="REM"/>
                </sx:when>
              </sx:choose>
            </sx:onRecord>
          </OrderPosition>
        </sx:innerGroup>
      </sx:innerGroup>
    </Order>
  </sx:recordMapping>
</sx:resources>

        

The effect of sx:combineRecords is to collect all the adjacent orderRemarks records, where they become sub-records belonging to a field called "detail", and create one record of type "composite" that has the repeating group field "detail", plus another field "remarks" that is computed from "detail".

The orderRemarks records flowing into sx:combineRecords look like, in their XML representation,

        
<orderRemarks><REMTEI>REM</REMTEI><REM>AAAA</REM></orderRemarks> 
<orderRemarks><REMTEI>REM</REMTEI><REM>BBBB</REM></orderRemarks> 
<orderRemarks><REMTEI>REM</REMTEI><REM>CCCC</REM></orderRemarks> 

      

The composite record flowing out looks like

        
<composite> 
  <detail> 
    <orderRemarks><REMTEI>REM</REMTEI><REM>AAAA</REM></orderRemarks> 
    <orderRemarks><REMTEI>REM</REMTEI><REM>BBBB</REM></orderRemarks> 
    <orderRemarks><REMTEI>REM</REMTEI><REM>CCCC</REM></orderRemarks> 
  </detail> 
  <remarks>AAAA BBBB CCCC</remarks> 
</composite> 

      

You can run this example on the command line by entering

          
servingxml -i data/order.txt -r resources-order.xml -o output/order.xml Order

        

Mapping Two Records at the Top of a File to a Header Element in the XML Output

This example shows a resources script that maps two records at the top of a file to a header element in the XML output. To map fields from multiple records to the attributes of a single element, you first need to compose an aggregate of the two records with the sx:combineRecords instruction. You can then select against the composite record with an sx:fieldAttributeMap and populate the attributes of the parent element.

The input file is

Figure 111. Input flat file files.txt

          
DP1 458293048kd02 
*****2LV 2008070132 EX9 
E0000015498273 
F0000014938294 
G000001E7839201 
D0000024839305

        

The rules for mapping the third through last records are as follows.

  • If a "D" record is found, start a new file element, but "D" records are optional.

  • If an "E" record is found, start a new file element, unless one has already been started because of a "D" record.

The desired output is

Figure 112. Output XML file files.xml

          
<?xml version="1.0" encoding="UTF-8"?>
<output date="20080701" sequence="32" 
        code="EX9" reference="458293048kd0">
  <file filenumber="000001"/>
  <file filenumber="000002"/>
</output>

        

The following resources script does the transformation.

Figure 113. Resources script resources-files.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="files">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="filesContent"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="filesContent">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="fileFlatFile"/>
      </sx:flatFileReader>
      <sx:combineRecords recordType="header" repeatingGroup="headerRecords"
                        startTest="sx:current/header1" endTest="sx:previous/header2">
      </sx:combineRecords>
    </sx:recordStream>
    <sx:recordMapping ref="fileToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="fileFlatFile">
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="tag" width="1"/>
        <sx:positionalField name="tag2" width="2"/>
        <sx:when test="tag='D' and tag2='P1'">
          <sx:flatRecordType name="header1">
            <sx:positionalField name="reference" start="5" width="13"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='*'">
          <sx:flatRecordType name="header2">
            <sx:positionalField name="date" start="10" width="8"/>
            <sx:positionalField name="sequence" width="2"/>
            <sx:positionalField name="code" start="21" width="3"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='E'">
          <sx:flatRecordType name="E">
            <sx:positionalField name="filenumber" start="2" width="6"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='D'">
          <sx:flatRecordType name="D">
            <sx:positionalField name="filenumber" start="2" width="6"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="fileToXmlMapping">
    <output>
      <sx:fieldAttributeMap select="headerRecords/header2/date" attribute="date"/>
      <sx:fieldAttributeMap select="headerRecords/header2/sequence" attribute="sequence"/>
      <sx:fieldAttributeMap select="headerRecords/header2/code" attribute="code"/>
      <sx:fieldAttributeMap select="headerRecords/header1/reference" attribute="reference"/>
      <sx:groupChoice>
        <sx:innerGroup startTest="sx:current/D" endTest="sx:current/D">
          <file>
            <sx:fieldAttributeMap field="filenumber" attribute="filenumber"/>
          </file>
        </sx:innerGroup>
        <sx:innerGroup startTest="sx:current/E" endTest="sx:current/D or sx:current/E">
          <file>
            <sx:fieldAttributeMap field="filenumber" attribute="filenumber"/>
          </file>
        </sx:innerGroup>
      </sx:groupChoice>
    </output>
  </sx:recordMapping>

</sx:resources>

        

The effect of sx:combineRecords is to collect the top two header records, header1 and header2 , where they become sub-records belonging to a field called "headerRecords", and create one record of type "header" that has the repeating group field "headerRecords".

The header record has the following XML representation,

        
<header>
  <headerRecords>
    <header1>
      <reference>458293048kd0</reference>
    </header1>
    <header2>
      <date>20080701</date>
      <sequence>32</sequence>
      <code>EX9</code>
    </header2>
  </headerRecords>
</header>

      

You can run this example on the command line by entering

          
servingxml -i data/files.txt -r resources-files.xml -o output/files.xml files

        

Start/End Delimited Records.

Converting flat files with start/end record delimiters (opdef)

The example below shows how to convert a flat file with start/end record delimiters.

The input file is shown below.

Figure 114. Input file opdef.txt

          
1 20050505 121205 xtyzzya 
{:HJSED 
:20:886452311980706 
:23:CREDDFED 
:32:030612USD5443,99 
:33:USD5443,99 
:50K:GIAN 
NNPLLS 
:52A:B0 
:53A:B33 
:54A:I3N 
:57A:BRE 
:59:/20041010050500001M02606 
KKL S.A 
GNRBBL 
:70:/FF/INV 559661 
:71:SAH 
-} 
3 20050505 121205 bahthy 

        

This file has a number of features.

  • The first and last lines are header and footers, delimited by new line symbols.

  • The middle part must be taken as a whole chunk, starting from the opening { and ending at the closing } .

The desired output is shown below.

Figure 115. Output XML file opdef.xml

          
<?xml version="1.0" encoding="utf-8"?>
<opdef>
  <start>
    <opdate>20050505</opdate>
    <optime>121205</optime>
    <optarget>xtyzzya</optarget>
  </start>
:HJSED
:20:886452311980706
:23:CREDDFED
:32:030612USD5443,99
:33:USD5443,99
:50K:GIAN
NNPLLS
:52A:B0
:53A:B33
:54A:I3N
:57A:BRE
:59:/20041010050500001M02606
KKL S.A
GNRBBL
:70:/FF/INV 559661
:71:SAH
-
  <end>
    <tpdate>20050505</tpdate>
    <tptime>121205</tptime>
    <tptarget>bahthy</tptarget>
  </end>
</opdef>

        

The following resources script does the transformation.

Figure 116. Resources script resources-opdef.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="opdef">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="opdef"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="opdef">
    <sx:flatFileReader>
      <sx:flatFile ref="opdefFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="opdefRecordMapping"/>
  </sx:recordContent>

  <sx:flatFile id="opdefFlatFile" name="opdef">
    <sx:recordDelimiter value="\r\n"/>
    <sx:recordDelimiter value="\n"/>
    <sx:recordDelimiter start="{" end="}"/>
    <sx:flatFileBody trim="true">
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="tag" width="1"/>
        <sx:when test="tag='1'">
          <sx:flatRecordType name="start">
            <sx:positionalField name="tag" width="2"/>
            <sx:positionalField name="opdate" width="9"/>
            <sx:positionalField name="optime" width="7"/>
            <sx:positionalField name="optarget" width="7"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='3'">
          <sx:flatRecordType name="end">
            <sx:positionalField name="tag" width="2"/>
            <sx:positionalField name="tpdate" width="9"/>
            <sx:positionalField name="tptime" width="7"/>
            <sx:positionalField name="tptarget" width="7"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:otherwise>
          <sx:flatRecordType name="data">
            <sx:delimitedField name="field1"/>
          </sx:flatRecordType>
        </sx:otherwise>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="opdefRecordMapping">
    <opdef>
      <sx:onRecord recordType="start">
        <start>
          <sx:defaultFieldElementMap fields="opdate optime optarget"/>
        </start>  
      </sx:onRecord>
      <sx:onRecord recordType="end">
        <end>
          <sx:defaultFieldElementMap fields="tpdate tptime tptarget"/>
        </end>  
      </sx:onRecord>
      <sx:onRecord recordType="data">
        <sx:cdata>
          <sx:toString value="{field1}"/>
        </sx:cdata>
      </sx:onRecord>
    </opdef>
  </sx:recordMapping>  

</sx:resources>

        

Note the following points about this script.

  • There are three record delimiters, which are checked sequentially. The third record delimiter specifies both start and end values. When the start value is found, the matching end value is looked for, taking account of nesting, and the whole returned as one record. The chunk returned may contain occurances of the other two delimiters.

  • The data is mapped to a single field by using a sx:delimitedField element, taking the default field delimiter, which is the end of the record. The data is enclosed in a CDATA section, using the sx:cdata element.

You can run this example on the command line by entering

          
servingxml -i data/opdef.txt -r resources-opdef.xml 
  -o output/opdef.xml opdef

        

Converting flat files with nested start/end record and segment delimiters and name/value pairs (transaction)

The example below shows how to convert a flat file with nested start/end record and segment delimiters and name/value pairs.

The input file is shown below.

Figure 117. Input file transaction.txt

          
{TRAN:trantype=PRIN#origin=Toronto#
{TRANREF:reftype=TREF#ref=1234567890123456B#}
{TRANREF:reftype=FOTR#ref=1234567890123456#}
{DRIVER:drivertype=OPER#drivercode=SELL#}
{DRIVER:drivertype=STRM#drivercode=FOP#}
{INSTR:instrtype=PINS#instrreftype=INT#instrref=PTEQTY1#qty=1000000#}
{INSTR:instrtype=TCCY#instrreftype=ISO#instrref=USD#}
{INSTR:instrtype=SINS#instrreftype=ISO#instrref=USD#}
{PARTY:partytype=COMP#partyreftype=X3#partyref=CMP1#}
{PARTY:partytype=SECP#partyreftype=X3#partyref=PTSECP1#}
{DATE:datetype=TDAT#date=01-Jan-2007#}
{DATE:datetype=VDAT#date=01-Jan-2007#}
{CHARGE:chargetype=COMM#calctype=NONE#chargeamount=50.00#instrreftype=ISO#instrref=USD}
{PRICE:ratetype=TPRC#price=4.9#multdiv=X1#pricetype=YP#}
}

        

This file has a number of features.

  • The entire record is contained inside "{" - "}" record delimiters.

  • The first field in the record is delimited by a ":" and identifies the type of the record.

  • The record contains a number of name/value pairs, with names and values separated by the character "=", and fields terminated by the character "#".

  • The record contains a number of segments enclosed in "{" - "}" segment delimiters.

  • The first field in each segment is delimited by a ":" and identifies the type of the segment.

  • Each segment contains a number of name/value pairs, with names and values separated by the character "=", and fields terminated by the character "#".

The desired output is shown below.

Figure 118. Output XML file transaction.xml

          
<?xml version="1.0" encoding="utf-8"?>
<transaction>
  <TRAN>
    <record-type>TRAN</record-type>
    <trantype>PRIN</trantype>
    <origin>Toronto</origin>
    <segments>
      <TRANREF>
        <record-type>TRANREF</record-type>
        <reftype>TREF</reftype>
        <ref>1234567890123456B</ref>
      </TRANREF>
      <TRANREF>
        <record-type>TRANREF</record-type>
        <reftype>FOTR</reftype>
        <ref>1234567890123456</ref>
      </TRANREF>
      <DRIVER>
        <record-type>DRIVER</record-type>
        <drivertype>OPER</drivertype>
        <drivercode>SELL</drivercode>
      </DRIVER>
      <DRIVER>
        <record-type>DRIVER</record-type>
        <drivertype>STRM</drivertype>
        <drivercode>FOP</drivercode>
      </DRIVER>
      <INSTR>
        <record-type>INSTR</record-type>
        <instrtype>PINS</instrtype>
        <instrreftype>INT</instrreftype>
        <instrref>PTEQTY1</instrref>
        <qty>1000000</qty>
      </INSTR>
      <INSTR>
        <record-type>INSTR</record-type>
        <instrtype>TCCY</instrtype>
        <instrreftype>ISO</instrreftype>
        <instrref>USD</instrref>
      </INSTR>
      <INSTR>
        <record-type>INSTR</record-type>
        <instrtype>SINS</instrtype>
        <instrreftype>ISO</instrreftype>
        <instrref>USD</instrref>
      </INSTR>
      <PARTY>
        <record-type>PARTY</record-type>
        <partytype>COMP</partytype>
        <partyreftype>X3</partyreftype>
        <partyref>CMP1</partyref>
      </PARTY>
      <PARTY>
        <record-type>PARTY</record-type>
        <partytype>SECP</partytype>
        <partyreftype>X3</partyreftype>
        <partyref>PTSECP1</partyref>
      </PARTY>
      <DATE>
        <record-type>DATE</record-type>
        <datetype>TDAT</datetype>
        <date>01-Jan-2007</date>
      </DATE>
      <DATE>
        <record-type>DATE</record-type>
        <datetype>VDAT</datetype>
        <date>01-Jan-2007</date>
      </DATE>
      <CHARGE>
        <record-type>CHARGE</record-type>
        <chargetype>COMM</chargetype>
        <calctype>NONE</calctype>
        <chargeamount>50.00</chargeamount>
        <instrreftype>ISO</instrreftype>
        <instrref>USD</instrref>
      </CHARGE>
      <PRICE>
        <record-type>PRICE</record-type>
        <ratetype>TPRC</ratetype>
        <price>4.9</price>
        <multdiv>X1</multdiv>
        <pricetype>YP</pricetype>
      </PRICE>
    </segments>
  </TRAN>
</transaction>

        

The following resources script does the transformation.

Figure 119. Resources script resources-transaction.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="transaction">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="transaction-content"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="transaction-content" name="transaction">
    <sx:flatFileReader>
      <sx:flatFile ref="transaction-flat-file"/>
    </sx:flatFileReader>
  </sx:recordContent>

  <sx:flatFile id="transaction-flat-file">
    <sx:recordDelimiter start="{" end="}"/>
    <sx:flatFileBody>
      <sx:repeatDelimiter start="{" end="}"/>
      <sx:nameDelimiter value="="/>
      <sx:flatRecordType name="TRAN">
        <sx:delimitedField name="record-type">
          <sx:fieldDelimiter value=":"/>
        </sx:delimitedField>
        <sx:repeatingField>
          <sx:fieldDelimiter value="#"/>
          <sx:segmentDelimiter value="{"/> 
          <sx:delimitedNamedField/>
        </sx:repeatingField>
        <sx:repeatingGroup name="segments">
          <sx:flatRecordTypeChoice>
            <sx:delimitedField name="record-type">
              <sx:fieldDelimiter value=":"/>
            </sx:delimitedField>
            <sx:when test="1=1">
              <sx:flatRecordType name="{record-type}">
                <sx:delimitedField name="record-type">
                  <sx:fieldDelimiter value=":"/>
                </sx:delimitedField>
                <sx:repeatingField> 
                  <sx:delimitedNamedField/>
                  <sx:fieldDelimiter value="#"/>
                </sx:repeatingField>
              </sx:flatRecordType>
            </sx:when>
          </sx:flatRecordTypeChoice>
        </sx:repeatingGroup>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

        

Note the following points about this script.

  • The value of the first field in the segment, terminated by ":", is assigned as the record type name of the segment.

  • All the name/value pairs within a segment are read with a single sx:repeatingField element.

You can run this example on the command line by entering

          
servingxml -i data/transaction.txt -r resources-transaction.xml 
  -o output/transaction.xml transaction

        

Custom Record Reader

Converting the output of a custom record reader to XML (custom record reader)

The example below shows how to convert the the output of a custom record reader to XML.

The input records are generated by the following Java class.

Figure 120. TradeRecordReader Java class

          
package flat2xml;

import com.servingxml.app.ServiceContext;
import com.servingxml.components.recordio.AbstractRecordReader;
import com.servingxml.app.Flow;
import com.servingxml.util.Name;
import com.servingxml.util.QualifiedName;
import com.servingxml.util.ServingXmlException;
import com.servingxml.util.record.Record;
import com.servingxml.util.record.RecordBuilder;

public class TradeRecordReader extends AbstractRecordReader {
  private static final Name TRADE_RECORD_TYPE = new QualifiedName("trade");
  private static final Name TRANSACTION_RECORD_TYPE = new QualifiedName("transaction");
  private static final Name RECORD_TYPE_NAME = new QualifiedName("record_type");
  private static final Name ID_NAME = new QualifiedName("id");
  private static final Name TRADE_DATE_NAME = new QualifiedName("trade_date");
  private static final Name TRADE_TIME_NAME = new QualifiedName("trade_time");
  private static final Name DESCRIPTION_NAME = new QualifiedName("description");
  private static final Name REFERENCE_NAME = new QualifiedName("reference");
  
  public void readRecords(ServiceContext context, Flow flow) 
   {

    //  Start the record stream
    startRecordStream(context, flow);

    RecordBuilder trRecordBuilder = new RecordBuilder(TRADE_RECORD_TYPE);
    RecordBuilder tnRecordBuilder = new RecordBuilder(TRANSACTION_RECORD_TYPE);

    Record record;

    trRecordBuilder.setString(RECORD_TYPE_NAME,"TR");
    trRecordBuilder.setString(ID_NAME,"0001");
    trRecordBuilder.setString(TRADE_DATE_NAME,"03/25/2005");
    trRecordBuilder.setString(TRADE_TIME_NAME,"01:50:00");
    trRecordBuilder.setString(DESCRIPTION_NAME,"This is a trade record");
    record = trRecordBuilder.toRecord();
    trRecordBuilder.clear();

    //  Write the "trade" record
    writeRecord(context, flow, record);

    tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
    tnRecordBuilder.setString(ID_NAME,"0002");
    tnRecordBuilder.setString(REFERENCE_NAME,"X1234");
    tnRecordBuilder.setString(DESCRIPTION_NAME,"A child transaction");
    record = tnRecordBuilder.toRecord();   
    tnRecordBuilder.clear();

    //  Write the first "transaction" record
    writeRecord(context, flow, record);

    tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
    tnRecordBuilder.setString(ID_NAME,"0003");
    tnRecordBuilder.setString(REFERENCE_NAME,"X1235");
    tnRecordBuilder.setString(DESCRIPTION_NAME,"Another child transaction");
    record = tnRecordBuilder.toRecord();
    tnRecordBuilder.clear();

    //  Write the second "transaction" record
    writeRecord(context, flow, record);

    //  End the record stream
    endRecordStream(context, flow);
  }
}

        

The TradeRecordReader class will generate the following stream of records.

record_typeidtrade_datetrade_timedescription
TR000103/25/2005 1:50:00This is a trade record

record_typeidreferencedescription
TN0002X1234A child transaction
TN0003X1235Another child transaction

The desired output is the same as for the trades example.

Figure 121. Output XML file trades.xml

          
<?xml version="1.0" encoding="utf-8"?>
<trades feed="LONDON">
  <trade id="0001">
    <trade-date>2005-03-25T01:50:00.000-05:00</trade-date>
    <description>This is a trade record</description>
  </trade>
  <transaction id="0002">
    <reference>X1234</reference>
    <description>A child transaction</description>
  </transaction>
  <transaction id="0003">
    <reference>X1235</reference>
    <description>Another child transaction</description>
  </transaction>
</trades>

        

The following resources script does the transformation.

Figure 122. Resources script resources-custom_trade_reader.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="trades">
    <sx:parameter name="feed">
      <sx:defaultValue>NY</sx:defaultValue>
    </sx:parameter>
    <sx:serialize>
      <sx:transform>
        <sx:content ref="trades"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="trades">
    <sx:customRecordReader class="flat2xml.TradeRecordReader"/>
    <sx:recordMapping ref="tradesToXmlMapping"/>
  </sx:recordContent>

  <sx:recordMapping id="tradesToXmlMapping">
    <trades>
      <sx:fieldAttributeMap value="{$feed}" attribute="feed"/>
      <sx:onRecord recordType="trade">
        <trade>
          <sx:fieldAttributeMap field="id" attribute="id"/>
          <sx:elementMap element="trade-date">
            <sx:convertToDateTime format="MM/dd/yyyy H:mm:ss">
              <sx:concat separator=" ">
                <sx:toString value="{trade_date}"/>
                <sx:toString value="{trade_time}"/>
              </sx:concat>
            </sx:convertToDateTime>
          </sx:elementMap>
          <sx:fieldElementMap field="description" element="description"/>
        </trade>
      </sx:onRecord>
      <sx:onRecord recordType="transaction">
        <transaction>
          <sx:fieldAttributeMap field="id" attribute="id"/>
          <sx:fieldElementMap field="reference" element="reference"/>
          <sx:fieldElementMap field="description" element="description"/>
        </transaction>
      </sx:onRecord>
    </trades>
  </sx:recordMapping>

</sx:resources>

        

Note the following points.

  • The record content is defined by a custom record reader, implemented in the Java class TradeRecordReader .

You can run this example on the command line by

  • Compiling the Java class TradeRecordReader and copying the resulting .class file into the dir/classes directory
  • Entering the command
                  
    servingxml -r resources-custom_trade_reader.xml
      -o output/trades.xml trades feed=LONDON
    
                

Batching XML Output

Serializing XML in Batches

This example shows how to serialize an XML document in batches. It illustrates the use of the sx:batchedSerializer element.

Suppose you want to read the CSV file countries.csv and serialize it to XML, but you also want the following:

  • Country subtrees to be split over multiple documents, no more than 50 to any one file.

                
              06/12/2007  07:37 PM    <DIR>          .
              06/12/2007  07:37 PM    <DIR>          ..
              06/12/2007  07:37 PM             3,773 countries-1.xml
              06/12/2007  07:37 PM             3,758 countries-2.xml
              06/12/2007  07:37 PM             3,747 countries-3.xml
              06/12/2007  07:37 PM             3,753 countries-4.xml
              06/12/2007  07:37 PM             3,416 countries-5.xml
              
              
  • Each document to begin with all the XML nodes up to the occurance of the first country subtree.

The following resources script produces the desired output.

        
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="countries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="countries"/>
      </sx:transform>
      <sx:batchedSerializer path="country" batchSize="50">
        <sx:xsltSerializer>
          <sx:fileSink file="output/countries-{$sx:batchSequenceNumber}.xml"/>
        </sx:xsltSerializer>
      </sx:batchedSerializer>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="countries" name="countries">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:urlSource url="data/countries.csv"/>
        <sx:flatFile ref="countriesFile"/>
      </sx:flatFileReader>
      <msv:recordValidator>
        <sx:urlSource url="data/country-record.xsd"/>
      </msv:recordValidator>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
      </sx:discardHandler>
    </sx:recordStream>
    <sx:recordMapping ref="countriesToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="countriesFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="country">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="code"/>
        <sx:delimitedField name="name" trimLeading="true"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="countriesToXmlMapping">
    <countries>
      <sx:onRecord>
        <country>
          <sx:fieldElementMap field="name" element="countryName"/>
          <sx:fieldAttributeMap field="code" attribute="countryCode"/>
        </country>
      </sx:onRecord>
    </countries>
  </sx:recordMapping>

</sx:resources>

      

You can run this example on the command line by entering

          
servingxml  -r resources-batches.xml countries 
  rootName=countries

        

Writing Records to Batch Files, and Converting the Batches to Separate XML Files

The example illustrates an alternative way of producing XML in batches, in two steps:

  • First, split a large flat file into a number of smaller flat files.

  • Second, convert each of the smaller flat files to XML.

Suppose you want to convert the countries.csv file containing 245 country records to XML, each country record mapped to one country subtree, and the output spilt over multiple XML files.

The first step is to produce the batched csv files and output them to an intermediate directory.

Figure 123. Batched intermediate CSV files.

          
06/12/2007  07:24 PM    <DIR>          .
06/12/2007  07:24 PM    <DIR>          ..
06/12/2007  07:24 PM               814 countries-1.csv
06/12/2007  07:24 PM               795 countries-2.csv
06/12/2007  07:24 PM               786 countries-3.csv
06/12/2007  07:24 PM               784 countries-4.csv
06/12/2007  07:24 PM               793 countries-5.csv

        

The second step is to read all the csv files in the intermediate directory and serialize them to XML.

Figure 124. Batched XML files.

          
06/12/2007  07:37 PM    <DIR>          .
06/12/2007  07:37 PM    <DIR>          ..
06/12/2007  07:37 PM             3,773 countries-1.xml
06/12/2007  07:37 PM             3,758 countries-2.xml
06/12/2007  07:37 PM             3,747 countries-3.xml
06/12/2007  07:37 PM             3,753 countries-4.xml
06/12/2007  07:37 PM             3,416 countries-5.xml

        

The following resources script performs both steps.

        
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="countries">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:fileSource file="data/{$rootName}.csv"/>
        <sx:flatFile ref="countriesFile"/>
      </sx:flatFileReader>
      <sx:batchedRecordWriter batchSize="50">
        <sx:flatFileWriter>
          <sx:fileSink file="intermediate/{$rootName}-{$sx:batchSequenceNumber}.csv"/>
          <sx:flatFile ref="countriesFile"/>
        </sx:flatFileWriter>
      </sx:batchedRecordWriter>
    </sx:recordStream>

    <sx:recordStream>
      <sx:directoryReader directory="intermediate">
        <sx:fileFilter pattern="countries.*[.]csv"/>
      </sx:directoryReader>
      <sx:processRecord>
        <sx:parameter name="output-file">
          <sx:findAndReplace searchFor ="(countries.*)[.]csv" replaceWith ="$1.xml">
            <sx:toString value="{name}"/>
          </sx:findAndReplace>
        </sx:parameter>
        <sx:serialize>
          <sx:xsltSerializer>
            <sx:fileSink directory="output" file="{$output-file}"/>
          </sx:xsltSerializer>
          <sx:transform>
            <sx:content ref="countries"/>
          </sx:transform>
        </sx:serialize>
      </sx:processRecord>
    </sx:recordStream>
  </sx:service>

  <sx:recordContent id="countries">
    <sx:flatFileReader>
      <sx:fileSource directory="{parentDirectory}" file="{name}"/>
      <sx:flatFile ref="countriesFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="countriesToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="countriesFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="country">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="code"/>
        <sx:delimitedField name="name" trimLeading="true"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="countriesToXmlMapping">
    <countries>
      <sx:onRecord>
        <country>
          <sx:fieldElementMap field="name" element="countryName"/>
          <sx:fieldAttributeMap field="code" attribute="countryCode"/>
        </country>
      </sx:onRecord>
    </countries>
  </sx:recordMapping>

</sx:resources>

      

You can run this example on the command line by entering

          
servingxml  -r resources-batchRecords.xml countries 
  rootName=countries

        

Many XML Documents

Flat File to Summary and Detail XML Documents (eobclaims)

This example requires reading the flat file

        
  ADM2301EOBS
  HDR06/13/08200815501300101MEMBER01          BISHOP         
  ADR1234 TEST DR                                                                                       ARVADA            
  HDR06/13/08200815501300102MEMBER02          BISHOP         
  ADR5678 TEST DR                                                                                       ARVADA            
  TLRTOTAL EOBS GENERATED: 0000001000000000000010000018                                                                                                                                                                                                                                                       
  
      

and producing this output:

  • One summary XML document capturing the header ("ADM") and trailer ("TLR") records

                eobinputoutput.xml
              
  • Separate XML documents for each pair of "HDR" and "ADR" records

                eobinputoutput-061308200815501300101.xml
                eobinputoutput-061308200815501300102.xml
              

The summary XML document is to look like

        
  <?xml version="1.0" encoding="UTF-8"?>
  <eobs>
     <ADMIN_RECTYPE record_type="ADM">
        <ADM_0001_TEXT>2301EOBS</ADM_0001_TEXT>
     </ADMIN_RECTYPE>
     <TLR-REC-TYPE record_type="TLR">
        <TLR_0001_TEXT>TOTAL EOBS GENERATED: 0</TLR_0001_TEXT>
        <TLR_0001_COUNT>0000010</TLR_0001_COUNT>
        <FILLER>00000000000010000018</FILLER>
     </TLR-REC-TYPE>
  </eobs>
  
      

and the first detail documents as

        
<?xml version="1.0" encoding="UTF-8"?>
<eob>
   <HEADER_RECTYPE record_type="HDR">
      <HDR_STMT_DATE>06/13/08</HDR_STMT_DATE>
      <HDR_CLAIM_NUMBER>200815501300101</HDR_CLAIM_NUMBER>
      <HDR_PATIENT_NAME_FIRST>MEMBER01</HDR_PATIENT_NAME_FIRST>
      <HDR_PATIENT_NAME_MIDDLE/>
      <HDR_PATIENT_NAME_LAST>BISHOP</HDR_PATIENT_NAME_LAST>
   </HEADER_RECTYPE>
   <ADR_REC_TYPE record_type="ADR">
      <ADR_8512_ATTN>1234 TEST DR</ADR_8512_ATTN>
      <ADR_8513_STREET1/>
      <ADR_8513_STREET2/>
      <ADR_8514_CITY>ARVADA</ADR_8514_CITY>
   </ADR_REC_TYPE>
</eob>
  
      

Below is the required resources script. Since the structure of the record input will be shared with another example, we'll collect that in a separate script and include it in the main one.

Figure 125. Record structure resource-eobclaims-flatfile.xml

          
<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:flatFile id="eobclaims-file">
    <sx:flatFileBody>
      <sx:flatRecordType ref="eobclaims-record"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordTypeChoice id="eobclaims-record">
    <sx:positionalField name="record_type" width="3"/>
    <sx:when test="record_type='ADM'">
      <sx:flatRecordType ref="adm"/>
    </sx:when>
    <sx:when test="record_type='HDR'">
      <sx:flatRecordType ref="hdr"/>
    </sx:when>
    <sx:when test="record_type='ADR'">
      <sx:flatRecordType ref="adr"/>
    </sx:when>
    <sx:when test="record_type='TLR'">
      <sx:flatRecordType ref="tlr"/>
    </sx:when>
  </sx:flatRecordTypeChoice>

  <sx:flatRecordType id="adm" name="ADM">
    <sx:positionalField name="record_type" width="3"/>
    <sx:positionalField name="ADM_0001_TEXT" width="20"/>
    <sx:positionalField name="ADM_0001_RUN_DATE" width="249"/>
    <sx:positionalField name="FILLER" width="8"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="tlr" name="TLR">
    <sx:positionalField name="record_type" width="3"/>
    <sx:positionalField name="TLR_0001_TEXT" width="23"/>
    <sx:positionalField name="TLR_0001_COUNT" width="7"/>
    <sx:positionalField name="FILLER" width="268"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="hdr" name="HDR">
    <sx:positionalField name="record_type" width="3"/>
    <sx:positionalField name="HDR_STMT_DATE" width="8"/>
    <sx:positionalField name="HDR_CLAIM_NUMBER" width="15"/>
    <sx:positionalField name="HDR_PATIENT_NAME_FIRST" width="18"/>
    <sx:positionalField name="HDR_PATIENT_NAME_LAST" width="15"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="adr" name="ADR">
    <sx:positionalField name="record_type" width="3"/>
    <sx:positionalField name="ADR_8512_ATTN" width="99"/>
    <sx:positionalField name="ADR_8514_CITY" start="103" width="18"/>
  </sx:flatRecordType>

</sx:resources>

        

Figure 126. Main resources script resource-eobclaims.xml

          
<?xml version="1.0"?>

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:include href="resources-eobclaims-flatfile.xml"/>

  <sx:service id="eobclaims-to-xml">
    <!-- Do the header/trailer first -->
    <sx:serialize>
      <sx:xsltSerializer>
        <sx:fileSink directory="output" file="eobinputoutput.xml"/>
        <sx:outputProperty name="indent" value="yes"/>
      </sx:xsltSerializer>
      <sx:transform>
        <sx:recordContent>
          <sx:flatFileReader>
            <sx:flatFile ref="eobclaims-file"/>
          </sx:flatFileReader>
          <sx:recordMapping ref="eobclaims-header-trailer-mapping"/>
        </sx:recordContent>
      </sx:transform>
    </sx:serialize>

    <!-- Now do the body -->
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="eobclaims-file"/>
      </sx:flatFileReader>
      <!-- Compose a composite record out of each HDR-ADR pair -->
      <sx:combineRecords recordType="composite" repeatingGroup="eob"
                  startTest="sx:current/HDR"
                  endTest="sx:previous/ADR">
        <sx:newField name="HDR_STMT_DATE" select="eob/HDR/HDR_STMT_DATE"/>
        <sx:newField name="HDR_CLAIM_NUMBER" select="eob/HDR/HDR_CLAIM_NUMBER"/>
      </sx:combineRecords>
      <!-- Discard the records that are not HDR-ADR pairs -->
      <sx:restrictRecordFilter>
        <sx:recordRestriction recordType="composite"/>
      </sx:restrictRecordFilter>
      <!-- Process each HDR-ADR composite record -->
      <sx:processRecord>
        <!-- Construct and store the output filename in the parameter "output-file" -->
        <sx:parameter name="output-file">
          <sx:findAndReplace searchFor="/" replaceWith ="">
            <sx:toString value="eobinputoutput-{HDR_STMT_DATE}{HDR_CLAIM_NUMBER}.xml"/>
          </sx:findAndReplace>
        </sx:parameter>

        <sx:serialize>
          <sx:xsltSerializer>
            <sx:fileSink directory="output" file="{$output-file}"/>
            <sx:outputProperty name="indent" value="yes"/>
          </sx:xsltSerializer>
          <sx:transform>
            <sx:recordContent>
              <sx:recordMapping ref="eobclaims-body-mapping"/>
            </sx:recordContent>
          </sx:transform>
        </sx:serialize>
      </sx:processRecord>
    </sx:recordStream>
  </sx:service>

  <sx:recordMapping id="eobclaims-header-trailer-mapping">
    <eobs>
      <sx:onRecord recordType="ADM">
        <ADMIN_RECTYPE record_type="ADM">
          <sx:fieldElementMap field="ADM_0001_TEXT" element="ADM_0001_TEXT"/>
        </ADMIN_RECTYPE>
      </sx:onRecord>
      <sx:onRecord recordType="TLR">
        <TLR-REC-TYPE record_type="TLR">
          <sx:fieldElementMap field="TLR_0001_TEXT" element="TLR_0001_TEXT"/>
          <sx:fieldElementMap field="TLR_0001_COUNT" element="TLR_0001_COUNT"/>
          <sx:fieldElementMap field="FILLER" element="FILLER"/>
        </TLR-REC-TYPE>
      </sx:onRecord>
    </eobs>
  </sx:recordMapping>

  <sx:recordMapping id="eobclaims-body-mapping">
    <eob>
      <!-- Our composite record has a repeating group field, "eob", 
           that has two sub-records, "HDR" and "ADR" -->
      <sx:onRecord recordType="composite">
        <sx:subrecordMapping repeatingGroup="eob">
          <sx:onRecord recordType="HDR">
            <HEADER_RECTYPE>
              <sx:fieldAttributeMap field="record_type" attribute="record_type"/>
              <sx:fieldElementMap field="HDR_STMT_DATE" element="HDR_STMT_DATE"/>
              <sx:fieldElementMap field="HDR_CLAIM_NUMBER" element="HDR_CLAIM_NUMBER"/>
              <sx:fieldElementMap field="HDR_PATIENT_NAME_FIRST" element="HDR_PATIENT_NAME_FIRST"/>
              <sx:fieldElementMap field="HDR_PATIENT_NAME_MIDDLE" element="HDR_PATIENT_NAME_MIDDLE"/>
              <sx:fieldElementMap field="HDR_PATIENT_NAME_LAST" element="HDR_PATIENT_NAME_LAST"/>
            </HEADER_RECTYPE>
          </sx:onRecord>
          <sx:onRecord recordType="ADR">
            <ADR_REC_TYPE>
              <sx:fieldAttributeMap field="record_type" attribute="record_type"/>
              <sx:fieldElementMap field="ADR_8512_ATTN" element="ADR_8512_ATTN"/>
              <sx:fieldElementMap field="ADR_8513_STREET1" element="ADR_8513_STREET1"/>
              <sx:fieldElementMap field="ADR_8513_STREET2" element="ADR_8513_STREET2"/>
              <sx:fieldElementMap field="ADR_8514_CITY" element="ADR_8514_CITY"/>
            </ADR_REC_TYPE>
          </sx:onRecord>
        </sx:subrecordMapping>
      </sx:onRecord>
    </eob>
  </sx:recordMapping>

</sx:resources>

        

You can run this example on the command line by entering

          
servingxml -i data/eobclaims.txt -r resources-eobclaims.xml eobclaims-to-xml

        

Many Flat Input Files

Multiple XML Readers

This example shows a resources script that maps a primary input file to XML, with lookups from a secondary file.

The primary input file is the pipe delimited file, primary-input.txt.

          
HDR|a|b 
DET|a|b 
DET|a|b 
MSG|1|a 
MSG|3|a 
END 

        

The secondary input file is the pipe delimited file, messages.txt,

          
1|test message 1 
2|test message 2 
3|test message 3 

        

You need to read the primary input file and replace the id tag values under message with the message corresponding to the id. The expected output is

          
<?xml version="1.0" encoding="UTF-8"?>
<mytest>
  <field1>a</field1>
  <field2>b</field2>
  <detail>
    <field1>a</field1>
    <field2>b</field2>
  </detail>
  <detail>
    <field1>a</field1>
    <field2>b</field2>
  </detail>
  <message>
    <message>test message 1</message>
    <field2>a</field2>
  </message>
  <message>
    <message>test message 3</message>
    <field2>a</field2>
  </message>
</mytest>

        

The required resources script is

        
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="multiple-reader-test">
    <sx:serialize>
      <sx:xsltSerializer>
        <sx:outputProperty name="indent" value="yes"/>
      </sx:xsltSerializer>
      <sx:transform>
        <sx:content ref="multiple-reader-content"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="multiple-reader-content">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="primary-file"/>
      </sx:flatFileReader>
    </sx:recordStream>
    <sx:recordMapping ref="primary-file-to-xml"/>
  </sx:recordContent>

  <sx:flatFile id="primary-file">
    <sx:commentStarter value="#"/>
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:fieldDelimiter value="|"/>
        <sx:delimitedField name="tag"/>
        <sx:when test="tag='HDR'">
          <sx:flatRecordType name="header">
            <sx:delimitedField name="tag"/>
            <sx:delimitedField name="field1"/>
            <sx:delimitedField name="field2"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='DET'">
          <sx:flatRecordType name="detail">
            <sx:delimitedField name="tag"/>
            <sx:delimitedField name="field1"/>
            <sx:delimitedField name="field2"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='MSG'">
          <sx:flatRecordType name="message">
            <sx:delimitedField name="tag"/>
            <sx:delimitedField name="field1"/>
            <sx:delimitedField name="field2"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="primary-file-to-xml">
    <mytest>
      <sx:onRecord recordType="header">
        <sx:fieldElementMap field="field1" element="field1"/>
        <sx:fieldElementMap field="field2" element="field2"/>
      </sx:onRecord>
      <sx:onRecord recordType="detail">
        <detail>
          <sx:fieldElementMap field="field1" element="field1"/>
          <sx:fieldElementMap field="field2" element="field2"/>
        </detail>
      </sx:onRecord>
      <sx:onRecord recordType="message">
        <message>
          <sx:parameter name="my-id" value="{field1}"/>
          <sx:fieldElementMap select="document('messageContent')/messages/message[id=$my-id]/description" element="message"/>
          <sx:fieldElementMap field="field2" element="field2"/>
        </message>
      </sx:onRecord>
    </mytest>
  </sx:recordMapping>

  <sx:recordContent id="messageContent" name="messages">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:urlSource url="messages.txt"/>
        <sx:flatFile ref="messagesFile"/>
      </sx:flatFileReader>
    </sx:recordStream>
  </sx:recordContent>

  <sx:flatFile id="messagesFile" name="messages">
    <sx:flatFileBody>
      <sx:fieldDelimiter value="|"/>
      <sx:flatRecordType name="message">
        <sx:delimitedField name="id"/>
        <sx:delimitedField name="description"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

      

You can run this example on the command line by entering

          
servingxml -i primary-input.txt -r resources.xml -o output.xml multiple-reader-test

        

XML to Flat File

Converting an XML document to a positional flat file with record count in trailer (books-positional)

The example below illustrates how to prepare a resources script that will convert an XML document into a positional flat file, and include the record count in a trailer section.

The input file is an XML file.

Figure 127. Input XML file books.xml


<myns:books xmlns:myns="http://mycompany.com/mynames/">
  <myns:book categoryCode="F">
    <myns:title>Kafka on the Shore</myns:title>
    <myns:author>Haruki Murakami</myns:author>
    <myns:price>25.17</myns:price>
  </myns:book>
  <myns:book categoryCode="C">
    <myns:title>Concurrent Programming in Java</myns:title>
    <myns:author>Doug Lea</myns:author>
    <myns:price>77.13</myns:price>
  </myns:book>
</myns:books>


The desired output is shown below.

Figure 128. Output positional flat file books.txt


CAuthor                        Title                              Price

FHaruki Murakami               Kafka on the Shore                 25.17
CDoug Lea                      Concurrent Programming in Java     77.13


Number of records: 2


The following resources script does the transformation.

Figure 129. Resources script resources-books.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="books2pos">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="booksToFileMapping"/>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="booksFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="booksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="bookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:flatRecordType ref="trailer"/>
    </sx:flatFileTrailer>
  </sx:flatFile>

  <sx:flatRecordType id="bookType" name="bookType">
    <sx:positionalField name="category" label="Category" width="1"/>
    <sx:positionalField name="author" label="Author" width="30"/>
    <sx:positionalField name="title" label="Title" width="30"/>
    <sx:positionalField name="price" label="Price" width="10" justify="right"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="trailer" name="trailer">
    <sx:positionalField name="recordCountLabel" width="19" label="Number of records: "/>
    <sx:positionalField name="recordCount"  width="10" label="{$sx:recordCount}"/>
  </sx:flatRecordType>

  <sx:inverseRecordMapping id="booksToFileMapping">
    <sx:onSubtree path="/myns:books/myns:book">
      <sx:flattenSubtree recordType="book">
        <sx:subtreeFieldMap select="myns:title" field="title"/>
        <sx:subtreeFieldMap select="@categoryCode" field="category"/>
        <sx:subtreeFieldMap select="myns:author" field="author"/>
        <sx:subtreeFieldMap select="myns:price" field="price"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>
</sx:resources>


Note the following:

  • In header and trailer sections, the label attributes of fields are treated as data values.

  • In a sx:positionalField element, the value of the label attribute can contain references to parameters, in particular to $sx:recordCount, which is a parameter that the file record writer adds after finishing writing records.

You can run this example on the command line by entering


servingxml -r resources-books2pos.xml -i data/books.xml 
    -o output/books.txt books2pos

Converting an XML document with multiple occurances of an element to a CSV file (books-csv)

This example shows one way to prepare a resources script that will convert an XML document having multiple occurances of an element into a CSV file.

The input file is an XML file with multiple occurances of the element myns:review.

Figure 130. Input XML file books.xml


<myns:books xmlns:myns="http://mycompany.com/mynames/">
  <myns:book categoryCode="F">
    <myns:title>Factotum</myns:title>
    <myns:author>Charles Bukowski</myns:author>
    <myns:price/>
    <myns:reviews>
      <myns:review>*****</myns:review>
      <myns:review>*</myns:review>
    </myns:reviews>
  </myns:book>
  <myns:book categoryCode="F">
    <myns:title>Kafka on the Shore</myns:title>
    <myns:author>Haruki Murakami</myns:author>
    <myns:price>25.17</myns:price>
    <myns:reviews>
      <myns:review>*****</myns:review>
      <myns:review>****</myns:review>
    </myns:reviews>
  </myns:book>
  <myns:book categoryCode="F">
    <myns:title>Mr Mee</myns:title>
    <myns:author>Andrew Crumey</myns:author>
    <myns:price>22.00</myns:price>
    <myns:reviews>
      <myns:review>*****</myns:review>
      <myns:review>*</myns:review>
    </myns:reviews>
  </myns:book>
  <myns:book categoryCode="C">
    <myns:title>Concurrent Programming in Java</myns:title>
    <myns:author>Doug Lea</myns:author>
    <myns:price>77.13</myns:price>
    <myns:reviews>
      <myns:review>****</myns:review>
    </myns:reviews>
  </myns:book>
  <myns:book categoryCode="B">
    <myns:title>Sex, Lies, and Headlocks</myns:title>
    <myns:author>Shaun Assael</myns:author>
    <myns:price>7.24 </myns:price>
    <myns:reviews>
      <myns:review>****</myns:review>
    </myns:reviews>
  </myns:book>
</myns:books>


The desired output is shown below.

Figure 131. Output CSV file books.csv


Category,Author,Title,Price,Review 1,Review 2
F,Charles Bukowski,Factotum,,*****,*
F,Haruki Murakami,Kafka on the Shore,25.17,*****,****
F,Andrew Crumey,Mr Mee,22.00,*****,*
C,Doug Lea,Concurrent Programming in Java,77.13,****,
B,Shaun Assael,"Sex, Lies, and Headlocks",7.24 ,****,


Note that, because it contains embedded commas, the title of the last book must be enclosed in quote marks.

The following resources script does the transformation.

Figure 132. Resources script resources-books2csv.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="books2csv"> 
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="booksToFileMapping"/>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="booksFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="booksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
  </sx:flatFile>      

  <sx:flatRecordType id="bookType" name="bookType">
    <sx:fieldDelimiter value=","/>
    <sx:delimitedField name="category" label="Category"/>
    <sx:delimitedField name="author" label="Author"/>
    <sx:delimitedField name="title" label="Title"/>
    <sx:delimitedField name="price" label="Price"/>
    <sx:delimitedField name="review1" label="Review 1"/>
    <sx:delimitedField name="review2" label="Review 2"/>
  </sx:flatRecordType>
  
  <sx:inverseRecordMapping id="booksToFileMapping">
    <sx:onSubtree path="/myns:books/myns:book">
      <sx:flattenSubtree recordType="book">
        <sx:subtreeFieldMap select="myns:title" field="title"/>
        <sx:subtreeFieldMap select="@categoryCode" field="category"/>
        <sx:subtreeFieldMap select="myns:author" field="author"/>
        <sx:subtreeFieldMap select="myns:price" field="price"/>
        <sx:subtreeFieldMap select="myns:reviews/myns:review[1]" field="review1"/>
        <sx:subtreeFieldMap select="myns:reviews/myns:review[2]" field="review2"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>
  
</sx:resources>


You can run this example on the command line by entering


servingxml -o output/persons.xml -r resources-persons.xml 
    personsAddresses 

Converting an XML document with multiple occurances of an element to a flat file with subfield delimiters (books-pipe)

This example shows another way of writing a resources script that will convert an XML document having multiple occurances of an element into a delimited flat file.

The XML input file is the same as the previous example. The desired output, though, is slightly different.

Figure 133. Output pipe delimited flat file books-pipe.txt


Category|Author|Title|Price|review
Fiction|Charles Bukowski|Factotum||*****;*
Fiction|Haruki Murakami|Kafka on the Shore|25.17|*****;****
Fiction|Andrew Crumey|Mr Mee|22.00|*****;*
Computers|Doug Lea|Concurrent Programming in Java|77.13|****
Biography|Shaun Assael|Sex, Lies, and Headlocks|7.24 |****


Note the following features of this output.

  • Fields are separated by the pipe delimiter.

  • Ratings appear in a multivalued field separated by semi-colon subfield delimiters.

  • "Fiction" has been substituted for "F", "Computers" for "C", etc.

This time the resources script looks like this.

Figure 134. Resources script resources-books2pipe.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="books2pipe">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="booksToFileMapping"/>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>

      <sx:choose>
        <sx:when test="category='F'">
          <sx:modifyRecord>
            <sx:newField name="category" value="Fiction"/>
          </sx:modifyRecord>
        </sx:when>
        <sx:when test="category='C'">
          <sx:modifyRecord>
            <sx:newField name="category" value="Computers"/>
          </sx:modifyRecord>
        </sx:when>
        <sx:when test="category='B'">
          <sx:modifyRecord>
            <sx:newField name="category" value="Biography"/>
          </sx:modifyRecord>
        </sx:when>
      </sx:choose>
      
      <sx:flatFileWriter>
        <sx:flatFile ref="booksFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="booksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType id="bookType" name="bookType">
    <sx:fieldDelimiter value="|"/>
    <sx:delimitedField name="category" label="Category"/>
    <sx:delimitedField name="author" label="Author"/>
    <sx:delimitedField name="title" label="Title"/>
    <sx:delimitedField name="price" label="Price"/>
    <sx:delimitedField name="review" label="review">
      <sx:subfieldDelimiter value=";"/>
    </sx:delimitedField>
  </sx:flatRecordType>

  <sx:inverseRecordMapping id="booksToFileMapping">
    <sx:onSubtree path="/myns:books/myns:book">
      <sx:flattenSubtree recordType="book">
        <sx:subtreeFieldMap select="myns:title" field="title"/>
        <sx:subtreeFieldMap select="@categoryCode" field="category"/>
        <sx:subtreeFieldMap select="myns:author" field="author"/>
        <sx:subtreeFieldMap select="myns:price" field="price"/>
        <sx:subtreeFieldMap match="myns:reviews/myns:review" select="text()" field="review"/>
      </sx:flattenSubtree>
    </sx:onSubtree>                              
  </sx:inverseRecordMapping>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-books2pipe.xml -i data/books.xml 
    -o output/books-pipe.txt books2pipe

Converting each book entry in books.xml to a list of three records (book-listing)

This example shows how to write a resources script that will convert each book entry in the books.xml document into a list of three records, the first comma separated, and the last two space separated.

The XML input file is the same as the previous example.

The desired output is follows.

Figure 135. Output flat file book_lisiting.txt


"Factotum", Charles Bukowski
Category Fiction
Price $
"Kafka on the Shore", Haruki Murakami
Category Fiction
Price $25.17
"Mr Mee", Andrew Crumey
Category Fiction
Price $22.00
"Concurrent Programming in Java", Doug Lea
Category Computers
Price $77.13
"Sex, Lies, and Headlocks", Shaun Assael
Category Biography
Price $7.24 


Note the following features of this output.

  • The categoryCode 'F' is mapped to "Fiction", 'C' to "Computers", 'B' to "Biography".

  • A dollar sign '$' is prepended to the price (even if the price is empty.)

  • Each book title is enclosed in quotation marks.

  • Each book subtree is output as three records; the first delimited by ", ", the second delimited by a space character.

This time the resources script looks like this.

Figure 136. Resources script resources-book_listing.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="book-listing">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="booksToFileMapping"/>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>

      <!-- Convert category code to label -->
      <sx:choose>
        <sx:when test="category='F'">
          <sx:modifyRecord>
            <sx:newField name="category" value="Fiction"/>
          </sx:modifyRecord>
        </sx:when>
        <sx:when test="category='C'">
          <sx:modifyRecord>
            <sx:newField name="category" value="Computers"/>
          </sx:modifyRecord>
        </sx:when>
        <sx:when test="category='B'">
          <sx:modifyRecord>
            <sx:newField name="category" value="Biography"/>
          </sx:modifyRecord>
        </sx:when>
      </sx:choose>

      <sx:flatFileWriter>
        <sx:flatFile ref="booksFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="booksFile">
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordTypeChoice id="bookType">
    <sx:delimitedField name="xxx"/>  <!-- unused here, used for input only -->
    <sx:when test="/book">  <!-- matches against record type "book" -->
      <sx:flatRecordType name="book">
        <sx:fieldDelimiter value=", "/>
        <sx:delimitedField name="title" quote="always"/>
        <sx:delimitedField name="author"/>
      </sx:flatRecordType>
    </sx:when>
    <sx:when test="/bookCategory">  <!-- matches against record type "bookCategory" -->
      <sx:flatRecordType name="book">
        <sx:fieldDelimiter value=" "/>
        <sx:delimitedField name="label"/>
        <sx:delimitedField name="category"/>
      </sx:flatRecordType>
    </sx:when>
    <sx:when test="/bookPrice">  <!-- matches against record type "bookPrice" -->
      <sx:flatRecordType name="book">
        <sx:fieldDelimiter value=" "/>
        <sx:delimitedField name="label"/>
        <sx:delimitedField name="price"/>
      </sx:flatRecordType>
    </sx:when>
  </sx:flatRecordTypeChoice>

  <sx:inverseRecordMapping id="booksToFileMapping">
    <sx:onSubtree path="/myns:books/myns:book">
      <sx:flattenSubtree recordType="book">
        <sx:subtreeFieldMap select="myns:title" field="title"/>
        <sx:subtreeFieldMap select="myns:author" field="author"/>
      </sx:flattenSubtree>
      <sx:flattenSubtree recordType="bookCategory">
        <sx:subtreeFieldMap select="'Category'" field="label"/>
        <sx:subtreeFieldMap select="@categoryCode" field="category"/>
      </sx:flattenSubtree>
      <sx:flattenSubtree recordType="bookPrice">
        <sx:subtreeFieldMap select="'Price'" field="label"/>
        <!-- Append dollar sign to price -->
        <sx:subtreeFieldMap select="concat('$',myns:price)" field="price"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-book_listing.xml -i data/books.xml 
    -o output/book_listing.txt book-listing

Converting an XML document into flat file records with multi-valued fields (swath)

This example shows another way of writing a resources script that will convert an XML document into flat file records with multi-valued fields.

The input file is an XML file with multiple occurances of the element GeoPoint inside a Swath element.

Figure 137. Input XML file swath.xml


<Swath>
  <Shape>Polygon</Shape>
  <GeoPoint>
    <lat>1</lat>
    <lon>2</lon>
  </GeoPoint>
  <GeoPoint>
    <lat>3</lat>
    <lon>4</lon>
  </GeoPoint>
  <GeoPoint>
    <lat>5</lat>
    <lon>6</lon>
  </GeoPoint>
</Swath>


The desired output is shown below.

Figure 138. Output pipe delimited flat file swath.txt


Shape|GeoPoint
Polygon|1 2;3 4;5 6


This time the resources script looks like this.

Figure 139. Resources script resources-swath2pipe.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="swath">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping >
          <sx:onSubtree  path="/Swath">
            <sx:flattenSubtree recordType="hdrType">
              <sx:subtreeFieldMap select="Shape" field="Shape"/>
              <sx:subtreeFieldMap match="GeoPoint" select="concat(lat,' ',lon)" field="GeoPoint"/>
            </sx:flattenSubtree>
          </sx:onSubtree>
        </sx:inverseRecordMapping>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="swathFlatFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="swathFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="hdrType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="hdrType"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType id="hdrType" name="hdrType">
    <sx:delimitedField name="Shape">
      <sx:fieldDelimiter value="|"/>
    </sx:delimitedField>
    <sx:delimitedField name="GeoPoint">
      <sx:subfieldDelimiter value=";"/>
    </sx:delimitedField>
  </sx:flatRecordType>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-swath.xml -i data/swath.xml 
    -o output/swath.txt swath

Converting an XML document into a flat file with header and multiple detail records (XML-flat header-detail)

This example shows how to write a resources script that will convert an XML document into a flat file records with header and multiple detail records.

The input file is an XML file with multiple occurances of the element myns:review inside a myns:book/myns:reviews path.

Figure 140. Input XML file bookReviews.xml


<?xml version="1.0" encoding="utf-8"?>
<myns:books xmlns:myns="http://mycompany.com/mynames/">
  <myns:book id="002" categoryCode="F">
    <myns:title>Kafka on the Shore</myns:title>
    <myns:author>Haruki Murakami</myns:author>
    <myns:price>25.17</myns:price>
    <myns:reviews>
      <myns:review>
        <myns:reviewer>Curley</myns:reviewer>
        <myns:rating>*****</myns:rating>
      </myns:review>
      <myns:review>
        <myns:reviewer>Larry</myns:reviewer>
        <myns:rating>***</myns:rating>
      </myns:review>
      <myns:review>
        <myns:reviewer>Moe</myns:reviewer>
        <myns:rating>*</myns:rating>
      </myns:review>
    </myns:reviews>
  </myns:book>
</myns:books>


The desired output is shown below.

Figure 141. Output positional flat file bookReviews.txt


TID CAuthor                        Title                              Price

H002FHaruki Murakami               Kafka on the Shore                 25.17
L002Curley    *****
L002Larry     ***  
L002Moe       *    

This is a trailer record


The following resources script does the job.

Figure 142. Resources script resources-bookReviews.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="bookReviews">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="booksToFileMapping"/>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>
    </sx:recordStream>
    <sx:flatFileWriter>
      <sx:flatFile ref="booksFile"/>
    </sx:flatFileWriter>
  </sx:service>

  <sx:flatFile id="booksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="book"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="tag"  width="1"/>
        <sx:when test="tag='H'">
          <sx:flatRecordType ref="book"/>
        </sx:when>
        <sx:when test="tag='L'">
          <sx:flatRecordType ref="review"/>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>

  <sx:flatRecordType id="book" name="book">
    <sx:positionalField name="tag" label="Tag" width="1"/>
    <sx:positionalField name="id" label="ID" width="3"/>
    <sx:positionalField name="category" label="Category" width="1"/>
    <sx:positionalField name="author" label="Author" width="30"/>
    <sx:positionalField name="title" label="Title" width="30"/>
    <sx:positionalField name="price" label="Price" width="10" justify="right"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="review" name="review">
    <sx:positionalField name="tag" label="Tag" width="1"/>
    <sx:positionalField name="id" label="ID" width="3"/>
    <sx:positionalField name="reviewer" label="Reviewer" width="10"/>
    <sx:positionalField name="rating" label="Rating" width="5"/>
  </sx:flatRecordType>

  <sx:inverseRecordMapping id="booksToFileMapping">
    <sx:onSubtree path="/myns:books/myns:book">
      <sx:parameter name="id" select="@id"/>
      <sx:flattenSubtree recordType="book">
        <sx:subtreeFieldMap select="'H'" field="tag"/>
        <sx:subtreeFieldMap select="$id" field="id"/>
        <sx:subtreeFieldMap select="myns:title" field="title"/>
        <sx:subtreeFieldMap select="@categoryCode" field="category"/>
        <sx:subtreeFieldMap select="myns:author" field="author"/>
        <sx:subtreeFieldMap select="myns:price" field="price"/>
      </sx:flattenSubtree>
      <sx:flattenSubtree match="myns:reviews/myns:review" recordType="review">
        <sx:subtreeFieldMap select="'L'" field="tag"/>
        <sx:subtreeFieldMap select="$id" field="id"/>
        <sx:subtreeFieldMap select="myns:reviewer" field="reviewer"/>
        <sx:subtreeFieldMap select="myns:rating" field="rating"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>
</sx:resources>


You can run this example on the command line by entering


servingxml  -r resources-bookReviews.xml -i data/bookReviews.xml 
    -o output/bookReviews.txt bookReviews

Breaking up a sequence of elements into flat file records ("sequence")

The example below illustrates how to use an XSLT stylesheet in an XML-flat pipeline to transform the XML into a form that is easier to map.

The input file is an XML file containing a sequence of elements.

Figure 143. Input XML file sequence.xml


<root>
  <first>1111</first>
  <second>AD</second>
  <third>1111</third>
  <fourth>1111</fourth>
  <first>2222</first>
  <second>MO</second>
  <third>2222</third>
  <fourth>2222</fourth>
</root>


The desired output file is shown below.

Figure 144. Output CSV file sequence.csv.


1111,AD,1111,1111
2222,MO,2222,2222


The following resources script does the transformation.

Figure 145. Resources script resources-sequence.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="sequence-csv">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:transform>
          <sx:document>
            <sx:fileSource file="data/sequence.xml"/>
          </sx:document>
          <sx:xslt ref="sequence-hierarchy"/>
        </sx:transform>
        <sx:inverseRecordMapping ref="sequenceToFileMapping"/>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="sequenceFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:xslt id="sequence-hierarchy">
    <xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
      <xsl:template match="/root">
        <xsl:copy>
          <xsl:apply-templates select="first"/>
        </xsl:copy>
      </xsl:template>
      <xsl:template match="first">
        <PivotNode>
          <first>
            <xsl:value-of select="."/>
          </first>
          <second>
            <xsl:value-of select="following-sibling::second[1]"/>
          </second>
          <third>
            <xsl:value-of select="following-sibling::third[1]"/>
          </third>
          <fourth>
            <xsl:value-of select="following-sibling::fourth[1]"/>
          </fourth>
        </PivotNode>
      </xsl:template>
    </xsl:stylesheet>
  </sx:xslt>

  <sx:flatFile id="sequenceFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="sequence">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="first"/>
        <sx:delimitedField name="second"/>
        <sx:delimitedField name="third"/>
        <sx:delimitedField name="fourth"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:inverseRecordMapping id="sequenceToFileMapping">
    <sx:onSubtree path="/root/PivotNode">
      <sx:flattenSubtree  recordType="sequence">
        <sx:subtreeFieldMap select="first" field="first"/>
        <sx:subtreeFieldMap select="second" field="second"/>
        <sx:subtreeFieldMap select="third" field="third"/>
        <sx:subtreeFieldMap select="fourth" field="fourth"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>                           

</sx:resources>


You can run this example on the command line by entering


servingxml  -o output/sequence.csv -r resources-sequence.xml sequence-csv

Converting an XML file with hexidecimal encoded values to a flat file with special characters

The example below illustrates how to prepare a resources script that will convert an XML document with hexadecimal encoded values to a flat file with special characters.

The input XML document is shown below.

Figure 146. Input XML file transaction.xml


<?xml version="1.0" encoding="utf-8"?>
<transaction>
  <R03>
    <firstField>03</firstField>
    <CLIENUM>1111111</CLIENUM>
  </R03>
  <R04>
    <firstField>04</firstField>
    <NAME>John Smith</NAME>
  </R04>
</transaction>


The desired output file is a flat file with two adjacent records (no line delimiters):

  • A record beginning with the hexadecimal value 03, followed by the text "1111111"

  • A record beginning with the hexadecimal value 04, followed by the text "John Smith"

The following resources script does the transformation.

Figure 147. Resources script resources-specialChar.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="transaction">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="fieldsMapping"/>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter >
        <sx:flatFile ref="transactionFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="transactionFile" lineDelimited="false">
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:binaryField name="firstField" width="1"/>
        <sx:when test="firstField='03'">
          <sx:flatRecordType id="R03" name="R03">
            <sx:binaryField name="firstField" label="firstField" width="1" />
            <sx:positionalField name="CLIENUM" label="CLIENUM" width="007" />
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="firstField='04'">
          <sx:flatRecordType id="R04" name="R04">
            <sx:binaryField name="firstField" label="firstField" width="1" />
            <sx:positionalField name="NAME" label="NAME" width="020" />
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:inverseRecordMapping id="fieldsMapping">
    <sx:onSubtree path="/transaction/R03">
      <sx:flattenSubtree match="/R03" recordType="R03">
        <sx:subtreeFieldMap select="firstField" field="firstField"/>
        <sx:subtreeFieldMap select="CLIENUM" field="CLIENUM"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
    <sx:onSubtree path="/transaction/R04">
      <sx:flattenSubtree match="/R04" recordType="R04">
        <sx:subtreeFieldMap select="firstField" field="firstField"/>
        <sx:subtreeFieldMap select="NAME" field="NAME"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>
</sx:resources>


Note that the first field of each record is declared as an sx:binaryField. The string rendering of this field is as an hexidecimal value.

You can run this example on the command line by entering


servingxml  -r resources-specialChar.xml -i data/transaction.xml 
  -o output/transaction.txt transaction

Converting an XML document to a positional flat file and ungrouping an XML element (ungrouping)

The example below illustrates how to prepare a resources script that will convert an XML document into a positional flat file and ungroup an XML element.

The input file is an XML file.

Figure 148. Input XML file grouped.xml


<RT_STLMT>
  <OPERATING_DATE>03/04/2003</OPERATING_DATE>
  <SETTLEMENT_CODE>S14</SETTLEMENT_CODE>
  <LINE_ITEMS>
    <CHG_TYP>
      <CHG_TYP_ID>RT_ADMIN1</CHG_TYP_ID>
      <CHG_TYP_NM>Market Administration Amount1</CHG_TYP_NM>
      <TOTAL>
        <AMT>288.81</AMT>
        <INT>
          <INT_NUM>1</INT_NUM>
          <VAL>54.4</VAL>
        </INT>
        <INT>
          <INT_NUM>2</INT_NUM>
          <VAL>31.16</VAL>
        </INT>
        <INT>
          <INT_NUM>3</INT_NUM>
          <VAL>81.48</VAL>
        </INT>
        <INT>
          <INT_NUM>4</INT_NUM>
          <VAL>121.77</VAL>
        </INT>
      </TOTAL>
    </CHG_TYP>
    <CHG_TYP>
      <CHG_TYP_ID>RT_ADMIN2</CHG_TYP_ID>
      <CHG_TYP_NM>Market Administration Amount2</CHG_TYP_NM>
      <TOTAL>
        <AMT>288.81</AMT>
        <INT>
          <INT_NUM>1</INT_NUM>
          <VAL>54.4</VAL>
        </INT>
        <INT>
          <INT_NUM>2</INT_NUM>
          <VAL>31.16</VAL>
        </INT>
        <INT>
          <INT_NUM>3</INT_NUM>
          <VAL>81.48</VAL>
        </INT>
        <INT>
          <INT_NUM>4</INT_NUM>
          <VAL>121.77</VAL>
        </INT>
      </TOTAL>
    </CHG_TYP>
  </LINE_ITEMS>
</RT_STLMT>


The desired output is shown below.

Figure 149. Output positional flat file ungrouped.csv


OPERATING_DATE,SETTLEMENT_CODE,DETERMINANT_ID,DETERMINANT_NAME,TOTAL_AMT,INT01,INT02,INT03,INT04
03/04/2003,S14,RT_ADMIN1,Market Administration Amount1,288.81,54.4,31.16,81.48,121.77
03/04/2003,S14,RT_ADMIN2,Market Administration Amount2,288.81,54.4,31.16,81.48,121.77


The following resources script does the transformation.

Figure 150. Resources script resources-ungrouping.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core" xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="settlements2csv">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:parameter name="foo">bar</sx:parameter>
        <sx:inverseRecordMapping ref="settlementsToFileMapping"/>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="settlementsFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="settlementsFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="settlementType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="settlementType"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType id="settlementType" name="settlementType">
    <sx:fieldDelimiter value=","/>
    <sx:delimitedField name="OPERATING_DATE" label="OPERATING_DATE"/>
    <sx:delimitedField name="SETTLEMENT_CODE" label="SETTLEMENT_CODE"/>
    <sx:delimitedField name="DETERMINANT_ID" label="DETERMINANT_ID"/>
    <sx:delimitedField name="DETERMINANT_NAME" label="DETERMINANT_NAME"/>
    <sx:delimitedField name="TOTAL_AMT" label="TOTAL_AMT"/>
    <sx:delimitedField name="INT01" label="INT01"/>
    <sx:delimitedField name="INT02" label="INT02"/>
    <sx:delimitedField name="INT03" label="INT03"/>
    <sx:delimitedField name="INT04" label="INT04"/>
  </sx:flatRecordType>

  <sx:inverseRecordMapping id="settlementsToFileMapping">
    <sx:onSubtree path="/RT_STLMT">
      <sx:parameter name="OPERATING_DATE" select="OPERATING_DATE"/>
      <sx:parameter name="SETTLEMENT_CODE" select="SETTLEMENT_CODE"/>
      <sx:parameter name="id" select="@id"/>
      <sx:onSubtree path="LINE_ITEMS/CHG_TYP">
        <sx:flattenSubtree recordType="settlement">
          <sx:subtreeFieldMap select="$OPERATING_DATE" field="OPERATING_DATE"/>
          <sx:subtreeFieldMap select="$SETTLEMENT_CODE" field="SETTLEMENT_CODE"/>
          <sx:subtreeFieldMap select="'RT'" field="STLMT_TYP"/>
          <sx:subtreeFieldMap select="CHG_TYP_ID" field="DETERMINANT_ID"/>
          <sx:subtreeFieldMap select="CHG_TYP_NM" field="DETERMINANT_NAME"/>
          <sx:subtreeFieldMap select="TOTAL/AMT" field="TOTAL_AMT"/>
          <sx:subtreeFieldMap select="TOTAL/INT[1]/VAL" field="INT01"/>
          <sx:subtreeFieldMap select="TOTAL/INT[2]/VAL" field="INT02"/>
          <sx:subtreeFieldMap select="TOTAL/INT[3]/VAL" field="INT03"/>
          <sx:subtreeFieldMap select="TOTAL/INT[4]/VAL" field="INT04"/>
        </sx:flattenSubtree>
      </sx:onSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>
</sx:resources>


Note the use of nested sx:onSubtree elements, and the sx:parameter elements appearing in the outermost sx:onSubtree. The select attribute in these parameter declarations can refer to the outer tags in the XML document under "/RT_STLMT", but not the inner tags handled by the inner sx:onSubtree elements.

An alternative inverse record mapping, which will produce the same output, is shown below.


  <sx:inverseRecordMapping id="settlementsToFileMapping">
    <sx:onSubtree path="/RT_STLMT">
      <sx:parameter name="OPERATING_DATE" select="OPERATING_DATE"/>
      <sx:parameter name="SETTLEMENT_CODE" select="SETTLEMENT_CODE"/>
      <sx:parameter name="id" select="@id"/>
      <sx:flattenSubtree match="LINE_ITEMS/CHG_TYP" recordType="settlement">
        <sx:subtreeFieldMap select="$OPERATING_DATE" field="OPERATING_DATE"/> 
        <sx:subtreeFieldMap select="$SETTLEMENT_CODE" field="SETTLEMENT_CODE"/> 
        <sx:subtreeFieldMap select="'RT'" field="STLMT_TYP"/>
        <sx:subtreeFieldMap select="CHG_TYP_ID" field="DETERMINANT_ID"/>
        <sx:subtreeFieldMap select="CHG_TYP_NM" field="DETERMINANT_NAME"/>
        <sx:subtreeFieldMap select="TOTAL/AMT" field="TOTAL_AMT"/>
        <sx:subtreeFieldMap select="TOTAL/INT[1]/VAL" field="INT01"/>
        <sx:subtreeFieldMap select="TOTAL/INT[2]/VAL" field="INT02"/>
        <sx:subtreeFieldMap select="TOTAL/INT[3]/VAL" field="INT03"/>
        <sx:subtreeFieldMap select="TOTAL/INT[4]/VAL" field="INT04"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

This mapping allows for more flexibility in the sx:parameter declarations, as the select attribute can now refer to any content in the XML document under "/RT_STLMT", but at the expense of keeping more of the document in memory.

You can run this example on the command line by entering


servingxml -r resources-ungrouping.xml -i data/grouped.xml 
    -o output/ungrouped.csv  settlements2csv

XML to Flat Repeating Group (all)

The example below illustrates how to prepare a resources script that will convert an XML document into a positional flat file with repeating groups.

The input file is an XML file.

Figure 151. Input XML file all.xml


<all>
    <field1>val1</field1>
    <field2>val2</field2>
    <field3>val3</field3>
    <compositeA attr1="av11" attr2="av21" attr3="av31"/> 
    <compositeA attr1="av21" attr2="av22" attr3="av23"/>
    <compositeA attr1="av31" attr2="av32" attr3="av33"/>
    <compositeB attrx="avx" attry="avy"/> 
</all>


The desired output is shown below.

Figure 152. Output positional flat file all.txt


val1      val2      val3      av11 av21 av31 av21 av22 av23 av31 av32 av33 avx  avy  


The following resources script does the transformation.

Figure 153. Resources script resources-all.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="all2flat">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="allToFileMapping"/>
        <sx:transform>
          <sx:document/>
        </sx:transform>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="allFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="allFile">
    <sx:flatFileBody>
      <sx:flatRecordType ref="allType"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType id="allType" name="allType">
    <sx:positionalField name="f1" label="F1" width="10"/>
    <sx:positionalField name="f2" label="F2" width="10"/>
    <sx:positionalField name="f3" label="F3" width="10"/>
    <sx:repeatingGroup name="compA">
      <sx:flatRecordType name="compARecord">
        <sx:positionalField name="ca1"  width="5"/>
        <sx:positionalField name="ca2"  width="5"/>
        <sx:positionalField name="ca3"  width="5"/>
      </sx:flatRecordType>
    </sx:repeatingGroup>
    <sx:repeatingGroup name="compB">
      <sx:flatRecordType name="compBRecord">
        <sx:positionalField name="cb1"  width="5"/>
        <sx:positionalField name="cb2"  width="5"/>
      </sx:flatRecordType>
    </sx:repeatingGroup>
  </sx:flatRecordType>

  <sx:inverseRecordMapping id="allToFileMapping">
    <sx:onSubtree path="/all">
      <sx:flattenSubtree recordType="allType">
        <sx:subtreeFieldMap select="field1" field="f1"/>
        <sx:subtreeFieldMap select="field2" field="f2"/>
        <sx:subtreeFieldMap select="field3" field="f3"/>
        <sx:subtreeFieldMap match="compositeA" field="compA">
          <sx:flattenSubtree  recordType="compARecord">
            <sx:subtreeFieldMap select="@attr1" field="ca1"/>
            <sx:subtreeFieldMap select="@attr2" field="ca2"/>
            <sx:subtreeFieldMap select="@attr3" field="ca3"/>
          </sx:flattenSubtree>
        </sx:subtreeFieldMap>
        <sx:subtreeFieldMap match="compositeB" field="compB">
          <sx:flattenSubtree  recordType="compBRecord">
            <sx:subtreeFieldMap select="@attrx" field="cb1"/>
            <sx:subtreeFieldMap select="@attry" field="cb2"/>
          </sx:flattenSubtree>
        </sx:subtreeFieldMap>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-all.xml -i data/all.xml 
    -o output/all.txt  all2flat

Formating Distinct Record Lines (blocks)

The example below illustrates how to prepare a resources script that will convert an XML document into a positional flat file with different record types corresponding to different XML subtrees.

The input file is an XML file.

Figure 154. Input XML file blocks.xml


<BLOCKS>
  <BLOCK1>
    <FIELD1>XXXX</FIELD1>
    <FIELD2>YYYY</FIELD2>
    <FIELD3>ZZZZ</FIELD3>
    <FIELD4>WWWW</FIELD4>
  </BLOCK1>
  <BLOCK2>
    <TEXT1>AAAA</TEXT1>
    <TEXT2>BBBB</TEXT2>
    <TEXT3>CCCC</TEXT3>
    <TEXT4>DDDD</TEXT4>
  </BLOCK2>
  <BLOCK3>
    <INTEGER1>1111</INTEGER1>
    <INTEGER2>2222</INTEGER2>
    <INTEGER3>3333</INTEGER3>
  </BLOCK3>
</BLOCKS>


The desired output is shown below.

Figure 155. Output positional flat file blocks.txt


XXXXYYYYZZZZWWWW
AAAABBBBCCCCDDDD
111122223333


The following resources script does the transformation.

Figure 156. Resources script resources-blocks.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="blocks">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:transform>
          <sx:document/>
        </sx:transform>
        <sx:inverseRecordMapping ref="blocksToFileMapping"/>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile>
          <sx:flatFileBody>
            <sx:flatRecordType ref="blockType"/>
          </sx:flatFileBody>
        </sx:flatFile>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatRecordTypeChoice id="blockType" name="blockType">
    <sx:positionalField name="dummy" width="1"/>
    <sx:when test="/Block1Record">
      <sx:flatRecordType name="BLOCK1">
        <sx:positionalField name="Field1" width="4"/>
        <sx:positionalField name="Field2" width="4"/>
        <sx:positionalField name="Field3" width="4"/>
        <sx:positionalField name="Field4" width="4"/>
      </sx:flatRecordType>
    </sx:when>
    <sx:when test="/Block2Record">
      <sx:flatRecordType name="BLOCK2">
        <sx:positionalField name="Text1" width="4"/>
        <sx:positionalField name="Text2" width="4"/>
        <sx:positionalField name="Text3" width="4"/>
        <sx:positionalField name="Text4" width="4"/>
      </sx:flatRecordType>
    </sx:when>
    <sx:when test="/Block3Record">
      <sx:flatRecordType name="BLOCK3">
        <sx:positionalField name="Integer1" width="4"/>
        <sx:positionalField name="Integer2" width="4"/>
        <sx:positionalField name="Integer3" width="4"/>
      </sx:flatRecordType>
    </sx:when>
  </sx:flatRecordTypeChoice>

  <sx:inverseRecordMapping id="blocksToFileMapping">
    <sx:onSubtree path="/BLOCKS">
      <sx:onSubtree path="BLOCK1">
        <sx:flattenSubtree recordType="Block1Record">
          <sx:subtreeFieldMap select="FIELD1" field="Field1"/>
          <sx:subtreeFieldMap select="FIELD2" field="Field2"/>
          <sx:subtreeFieldMap select="FIELD3" field="Field3"/>
          <sx:subtreeFieldMap select="FIELD4" field="Field4"/>
        </sx:flattenSubtree>
      </sx:onSubtree>
      <sx:onSubtree path="BLOCK2">
        <sx:flattenSubtree recordType="Block2Record">
          <sx:subtreeFieldMap select="TEXT1" field="Text1"/>
          <sx:subtreeFieldMap select="TEXT2" field="Text2"/>
          <sx:subtreeFieldMap select="TEXT3" field="Text3"/>
          <sx:subtreeFieldMap select="TEXT4" field="Text4"/>
        </sx:flattenSubtree>
      </sx:onSubtree>
      <sx:onSubtree path="BLOCK3">
        <sx:flattenSubtree recordType="Block3Record">
          <sx:subtreeFieldMap select="INTEGER1" field="Integer1"/>
          <sx:subtreeFieldMap select="INTEGER2" field="Integer2"/>
          <sx:subtreeFieldMap select="INTEGER3" field="Integer3"/>
        </sx:flattenSubtree>
      </sx:onSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-blocks.xml -i data/blocks.xml 
    -o output/blocks.txt  blocks

Block Subsequences in XML to One Line

This example illustrates a resources script that will map subsequences of blocks in the XML to a single line in the flat file.

The input file is an XML file.

Figure 157. Input XML file block_subsequences.xml


<BLOCKS>
  <BLOCK1>
    <FIELD1>XXXX1</FIELD1>
    <FIELD2>YYYY1</FIELD2>
    <FIELD3>ZZZZ1</FIELD3>
    <FIELD4>WWWW1</FIELD4>
  </BLOCK1>
  <BLOCK2>
    <TEXT1>AAAA1</TEXT1>
    <TEXT2>BBBB1</TEXT2>
    <TEXT3>CCCC1</TEXT3>
    <TEXT4>DDDD1</TEXT4>
  </BLOCK2>
  <BLOCK1>
    <FIELD1>XXXX2</FIELD1>
    <FIELD2>YYYY2</FIELD2>
    <FIELD3>ZZZZ2</FIELD3>
    <FIELD4>WWWW2</FIELD4>
  </BLOCK1>
  <BLOCK2>
    <TEXT1>AAAA2</TEXT1>
    <TEXT2>BBBB2</TEXT2>
    <TEXT3>CCCC2</TEXT3>
    <TEXT4>DDDD2</TEXT4>
  </BLOCK2>
</BLOCKS>


Note that BLOCK1 and BLOCK2 repeat.

The desired output is shown below.

Figure 158. Output positional flat file block_subsequences.txt


XXXX1YYYY1ZZZZ1WWWW1AAAA1BBBB1CCCC1DDDD1
XXXX2YYYY2ZZZZ2WWWW2AAAA2BBBB2CCCC2DDDD2


Note that BLOCK1 and BLOCK2 form a single line in the Flat output file, and then BLOCK1 and BLOCK2 repeat.

The following resources script does the transformation.

Figure 159. Resources script resources-subsequences.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="blocks">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:transform>
          <sx:document/>
        </sx:transform>
        <sx:inverseRecordMapping ref="blocksToFileMapping"/>
      </sx:subtreeRecordReader>
      <sx:combineRecords recordType="BLOCKS" repeatingGroup="blocks" 
                        beginTest="sx:current/BLOCK1"
                        endTest="sx:current/BLOCK1">
      </sx:combineRecords>
      <sx:flatFileWriter>
        <sx:flatFile>
          <sx:flatFileBody>
            <sx:flatRecordType ref="blockType"/>
          </sx:flatFileBody>
        </sx:flatFile>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatRecordType id="blockType" name="blockType">
    <sx:repeatingGroup name="blocks">
      <sx:flatRecordTypeChoice>
        <sx:when test="/BLOCK1">
          <sx:flatRecordType name="BLOCK1">
            <sx:positionalField name="Field1" width="5"/>
            <sx:positionalField name="Field2" width="5"/>
            <sx:positionalField name="Field3" width="5"/>
            <sx:positionalField name="Field4" width="5"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="/BLOCK2">
          <sx:flatRecordType name="BLOCK2">
            <sx:positionalField name="Text1" width="5"/>
            <sx:positionalField name="Text2" width="5"/>
            <sx:positionalField name="Text3" width="5"/>
            <sx:positionalField name="Text4" width="5"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:repeatingGroup>
  </sx:flatRecordType>

  <sx:inverseRecordMapping id="blocksToFileMapping">
    <sx:onSubtree path="/BLOCKS">
      <sx:onSubtree path="BLOCK1">
        <sx:flattenSubtree recordType="BLOCK1">
          <sx:subtreeFieldMap select="FIELD1" field="Field1"/>
          <sx:subtreeFieldMap select="FIELD2" field="Field2"/>
          <sx:subtreeFieldMap select="FIELD3" field="Field3"/>
          <sx:subtreeFieldMap select="FIELD4" field="Field4"/>
        </sx:flattenSubtree>
      </sx:onSubtree>
      <sx:onSubtree path="BLOCK2">
        <sx:flattenSubtree recordType="BLOCK2">
          <sx:subtreeFieldMap select="TEXT1" field="Text1"/>
          <sx:subtreeFieldMap select="TEXT2" field="Text2"/>
          <sx:subtreeFieldMap select="TEXT3" field="Text3"/>
          <sx:subtreeFieldMap select="TEXT4" field="Text4"/>
        </sx:flattenSubtree>
      </sx:onSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

</sx:resources>


The strategy is to map the BLOCK1 and BLOCK2 subtrees to BLOCK1 and BLOCK2 records, compose the BLOCK1 and BLOCK2 records into a composite record, and to map the composite record to a single line in the flat file.

You can run this example on the command line by entering


servingxml -r resources-subsequences.xml -i data/block_subsequences.xml
    -o output/block_subsequences.txt blocks 

Ungrouping Elements with Multiple Levels (book hierarchy)

The example below illustrates how to prepare a resources script that will convert an XML document into a delimited flat file with different record types corresponding to different XML subtrees.

The input file is an XML file.

Figure 160. Input XML file book_hierarchy.xml


<myns:rootNode xmlns:myns="http://mycompany.com/mynames/">
  <myns:name>root</myns:name>
  <attribute>
    <myns:name>NONE</myns:name>
    <myns:value>NONE</myns:value>
  </attribute>
  <myns:childNode>
    <myns:name>Entity.LONDON</myns:name>
    <attribute>
      <myns:name>Entity</myns:name>
      <myns:value>LONDON</myns:value>
    </attribute>
    <myns:childNode>
      <myns:name>BookName.EQUITIES_NY</myns:name>
      <attribute>
        <myns:name>BookName</myns:name>
        <myns:value>EQUITIES_NY</myns:value>
      </attribute>
      <myns:childNode>
        <myns:name>BookName.EQUITIES_NY_TR_JAYME</myns:name>
        <attribute>
          <myns:name>BookName</myns:name>
          <myns:value>EQUITIES_NY_TR_JAYME</myns:value>
        </attribute>
        <myns:childNode>
          <myns:name>BookName.PR_EQUITY_NY</myns:name>
          <attribute>
            <myns:name>BookName</myns:name>
            <myns:value>PR_EQUITY_NY</myns:value>
          </attribute>
        </myns:childNode>
        <myns:childNode>
          <myns:name>BookName.PR_EQUITY_NY1</myns:name>
          <attribute>
            <myns:name>BookName</myns:name>
            <myns:value>PR_EQUITY_NY1</myns:value>
          </attribute>
        </myns:childNode>
      </myns:childNode>
    </myns:childNode>
    <myns:childNode>
      <myns:name>BookName.EQUITIES_LONDON</myns:name>
      <attribute>
        <myns:name>BookName</myns:name>
        <myns:value>EQUITIES_LONDON</myns:value>
      </attribute>
      <myns:childNode>
        <myns:name>BookName.EQUITIES_NY_TR_JAYME</myns:name>
        <attribute>
          <myns:name>BookName</myns:name>
          <myns:value>EQUITIES_NY_TR_JAYME</myns:value>
        </attribute>
        <myns:childNode>
          <myns:name>BookName.PR_EQUITY_NY</myns:name>
          <attribute>
            <myns:name>BookName</myns:name>
            <myns:value>PR_EQUITY_NY</myns:value>
          </attribute>
        </myns:childNode>
      </myns:childNode>
    </myns:childNode>
  </myns:childNode>
</myns:rootNode>


The desired output is shown below.

Figure 161. Output delimited flat file book_hierarchy.txt


root,NONE,,Entity.LONDON,Entity,LONDON,,BookName,EQUITIES_NY,BookName,EQUITIES_NY_TR_JAYME,BookName,PR_EQUITY_NY,BookName,PR_EQUITY_NY1
root,NONE,,Entity.LONDON,Entity,LONDON,,BookName,EQUITIES_LONDON,BookName,EQUITIES_NY_TR_JAYME,BookName,PR_EQUITY_NY


The following resources script does the transformation.

Figure 162. Resources script resources-bookHierarchy.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core"
  xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="bookHierarchyToFile">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:transform>
          <sx:document/>
        </sx:transform>
        <sx:inverseRecordMapping ref="bookHierarchyToFileMapping"/>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile>
          <sx:flatFileBody>
            <sx:flatRecordType ref="settlementType"/>
          </sx:flatFileBody>
        </sx:flatFile>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatRecordType id="settlementType" name="settlementType">
    <sx:fieldDelimiter value=","/>
    <sx:repeatDelimiter value=","/>
    <sx:segmentDelimiter value="|"/>
    <sx:delimitedField name="root" label="root"/>
    <sx:delimitedField name="bookCode" label="bookCode"/>
    <sx:delimitedField name="bookValue" label="bookValue"/>
    <sx:delimitedField name="level1" label="level1"/>
    <sx:delimitedField name="level1Code" label="level1Code"/>
    <sx:delimitedField name="level1Value" label="level1Value"/>
    <sx:delimitedField name="level2" label="level2"/>
    <sx:delimitedField name="level2Code" label="level2Code"/>
    <sx:delimitedField name="level2Value" label="level2Value"/>
    <sx:repeatingGroup name="level3" label="level3">
      <sx:flatRecordType name="level3Record">
        <sx:delimitedField name="code" label="code"/>
        <sx:delimitedField name="value" label="value"/>
        <sx:repeatingGroup name="level4" label="level4">
          <sx:flatRecordType name="level4Record">
            <sx:delimitedField name="code" label="code"/>
            <sx:delimitedField name="value" label="value"/>
          </sx:flatRecordType>
        </sx:repeatingGroup>
      </sx:flatRecordType>
    </sx:repeatingGroup>
  </sx:flatRecordType>

  <sx:inverseRecordMapping id="bookHierarchyToFileMapping">
    <sx:onSubtree path="/myns:rootNode">
      <sx:parameter name="root" select="myns:name"/>
      <sx:parameter name="bookCode" select="attribute/myns:name"/>
      <sx:parameter name="bookValue" select="attribute/value"/>

      <sx:onSubtree path="/myns:rootNode/myns:childNode">
        <sx:parameter name="level1" select="myns:name"/>
        <sx:parameter name="level1Code" select="attribute/myns:name"/>
        <sx:parameter name="level1Value" select="attribute/myns:value"/>
        <sx:onSubtree path="/myns:childNode/myns:childNode">
          <sx:flattenSubtree recordType="settlement">
            <sx:subtreeFieldMap select="$root" field="root"/>
            <sx:subtreeFieldMap select="$bookCode" field="bookCode"/>
            <sx:subtreeFieldMap select="$bookValue" field="bookValue"/>
            <sx:subtreeFieldMap select="$level1" field="level1"/>
            <sx:subtreeFieldMap select="$level1Code" field="level1Code"/>
            <sx:subtreeFieldMap select="$level1Value" field="level1Value"/>
            <sx:subtreeFieldMap select="name" field="level2"/>
            <sx:subtreeFieldMap select="attribute/myns:name" field="level2Code"/>
            <sx:subtreeFieldMap select="attribute/myns:value" field="level2Value"/>
            <sx:subtreeFieldMap match="myns:childNode" field="level3">
              <sx:flattenSubtree  recordType="level3Record">
                <sx:subtreeFieldMap select="attribute/myns:name" field="code"/>
                <sx:subtreeFieldMap select="attribute/myns:value" field="value"/>
                <sx:subtreeFieldMap match="myns:childNode" field="level4">
                  <sx:flattenSubtree  recordType="level4Record">
                    <sx:subtreeFieldMap select="attribute/myns:name" field="code"/>
                    <sx:subtreeFieldMap select="attribute/myns:value" field="value"/>
                  </sx:flattenSubtree>
                </sx:subtreeFieldMap>
              </sx:flattenSubtree>
            </sx:subtreeFieldMap>
          </sx:flattenSubtree>
        </sx:onSubtree>
      </sx:onSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-bookHierarchy.xml -i data/book_hierarchy.xml 
    -o output/book_hierarchy.txt bookHierarchyToFile

X12 XML to Flat File using XSLT

The example below illustrates how to prepare a resources script that will convert an XML representation of an X12 file into a positional flat file. Different record types correspond to different subtrees in the XML document.

The input file is an XML file.

Figure 163. Input file Edi517M2xml.xml


<?xml version="1.0" encoding="utf-8"?>
<X12>
  <!-- Reapting group with in the x12 document -->
  <ST>
    <ST01>517</ST01>
    <ST02>0006</ST02>
  </ST>
  <BR>
    <BR01>00</BR01>
    <BR02>AN</BR02>
    <BR03>20071019</BR03>
    <BR04/>
    <BR05/>
    <BR06/>
    <BR07>X7</BR07>
    <BR08>0101</BR08>
    <BR09>150401</BR09>
    <BR10>XM</BR10>
    <BR11>003</BR11>
    <BR12>0101003</BR12>
  </BR>
  <G62>
    <G6201>CA</G6201>
    <G6202>20070720</G6202>
  </G62>
  <G62>
    <G6201>64</G6201>
    <G6202>20070903</G6202>
  </G62>
  <LM>
    <LM01>DF</LM01>
  </LM>
  <LQ>
    <LQ01>0</LQ01>
    <LQ02>AN9</LQ02>
  </LQ>
  <N1>
    <N101>Z4</N101>
    <N102/>
    <N103>M4</N103>
    <N104>GSA</N104>
    <N105/>
    <N106>FR</N106>
  </N1>
  <N1>
    <N101>ZR</N101>
    <N102/>
    <N103>10</N103>
    <N104>N23194</N104>
    <N105/>
    <N106>TO</N106>
  </N1>
  <!-- Repating group with in ST, nmber of times based on value in BR11( current value =3) -->
  <QTY>
    <QTY01>63</QTY01>
    <QTY02>4</QTY02>
    <QTY03>BX</QTY03>
    <QTY04>BX4</QTY04>
  </QTY>
  <N9>
    <N901>TN</N901>
    <N902>N2319471430196</N902>
  </N9>
  <N9>
    <N901>NS</N901>
    <N902>7920006199162</N902>
  </N9>
  <G62>
    <G6201>17</G6201>
    <G6202>20070718</G6202>
  </G62>
  <LM>
    <LM01>DF</LM01>
  </LM>
  <LQ>
    <LQ01>0</LQ01>
    <LQ02>AN1</LQ02>
  </LQ>
  <LQ>
    <LQ01>81</LQ01>
    <LQ02>BV</LQ02>
  </LQ>
  <LQ>
    <LQ01>COG</LQ01>
    <LQ02>9Q</LQ02>
  </LQ>
  <LQ>
    <LQ01>DE</LQ01>
    <LQ02>C</LQ02>
  </LQ>
  <LQ>
    <LQ01>DF</LQ01>
    <LQ02>S</LQ02>
  </LQ>
  <LQ>
    <LQ01>A9</LQ01>
    <LQ02>N23194</LQ02>
  </LQ>
  <LQ>
    <LQ01>78</LQ01>
    <LQ02>ZQ0</LQ02>
  </LQ>
  <LQ>
    <LQ01>79</LQ01>
    <LQ02>03</LQ02>
  </LQ>
  <FA1>
    <FA101>DN</FA101>
    <FA102>D340</FA102>
  </FA1>
  <FA2>
    <FA201>B5</FA201>
    <FA202>RC</FA202>
  </FA2>
  <QTY>
    <QTY01>63</QTY01>
    <QTY02>1</QTY02>
    <QTY03>EA</QTY03>
    <QTY04>EA1</QTY04>
  </QTY>
  <N9>
    <N901>TN</N901>
    <N902>N2319471430158</N902>
  </N9>
  <N9>
    <N901>NS</N901>
    <N902>5140003694927</N902>
  </N9>
  <G62>
    <G6201>17</G6201>
    <G6202>20070725</G6202>
  </G62>
  <LM>
    <LM01>DF</LM01>
  </LM>
  <LQ>
    <LQ01>0</LQ01>
    <LQ02>AN1</LQ02>
  </LQ>
  <LQ>
    <LQ01>81</LQ01>
    <LQ02>BB</LQ02>
  </LQ>
  <LQ>
    <LQ01>COG</LQ01>
    <LQ02>9Q</LQ02>
  </LQ>
  <LQ>
    <LQ01>DE</LQ01>
    <LQ02>C</LQ02>
  </LQ>
  <LQ>
    <LQ01>DF</LQ01>
    <LQ02>S</LQ02>
  </LQ>
  <LQ>
    <LQ01>A9</LQ01>
    <LQ02>N23194</LQ02>
  </LQ>
  <LQ>
    <LQ01>78</LQ01>
    <LQ02>ZQ0</LQ02>
  </LQ>
  <LQ>
    <LQ01>79</LQ01>
    <LQ02>03</LQ02>
  </LQ>
  <FA1>
    <FA101>DN</FA101>
    <FA102>D340</FA102>
  </FA1>
  <FA2>
    <FA201>B5</FA201>
    <FA202>RC</FA202>
  </FA2>
  <QTY>
    <QTY01>63</QTY01>
    <QTY02>6</QTY02>
    <QTY03>BG</QTY03>
    <QTY04>BG6</QTY04>
  </QTY>
  <N9>
    <N901>TN</N901>
    <N902>N2319471430272</N902>
  </N9>
  <N9>
    <N901>NS</N901>
    <N902>5610009851800</N902>
  </N9>
  <G62>
    <G6201>17</G6201>
    <G6202>20070703</G6202>
  </G62>
  <LM>
    <LM01>DF</LM01>
  </LM>
  <LQ>
    <LQ01>0</LQ01>
    <LQ02>AN1</LQ02>
  </LQ>
  <LQ>
    <LQ01>81</LQ01>
    <LQ02>BV</LQ02>
  </LQ>
  <LQ>
    <LQ01>COG</LQ01>
    <LQ02>9Q</LQ02>
  </LQ>
  <LQ>
    <LQ01>DE</LQ01>
    <LQ02>C</LQ02>
  </LQ>
  <LQ>
    <LQ01>DF</LQ01>
    <LQ02>S</LQ02>
  </LQ>
  <LQ>
    <LQ01>A9</LQ01>
    <LQ02>N23194</LQ02>
  </LQ>
  <LQ>
    <LQ01>78</LQ01>
    <LQ02>ZQ0</LQ02>
  </LQ>
  <LQ>
    <LQ01>79</LQ01>
    <LQ02>03</LQ02>
  </LQ>
  <FA1>
    <FA101>DN</FA101>
    <FA102>D340</FA102>
  </FA1>
  <FA2>
    <FA201>B5</FA201>
    <FA202>RC</FA202>
  </FA2>
  <SE>
    <SE01>54</SE01>
    <SE02>0006</SE02>
  </SE>
</X12>


The desired output is shown below.

Figure 164. Output delimited flat file Edi517M.txt


AN9GSA0101003               N23194           
AN1  7920006199162 CBX4N2319471430196N23194SRC9QZQ003  BV       
AN1  5140003694927 CEA1N2319471430158N23194SRC9QZQ003  BB       
AN1  5610009851800 CBG6N2319471430272N23194SRC9QZQ003  BV       


The following resources script does the transformation.

Figure 165. Resources script resources-xml-517M.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="xmltodlss">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:transform>
          <sx:document/>
          <sx:xslt ref="dlss-layout"/>
        </sx:transform>
        <sx:inverseRecordMapping ref="dlssToFileMapping"/>
      </sx:subtreeRecordReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="dlssFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:xslt id="dlss-layout">
    <xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
      <xsl:strip-space elements="*"/>
      <xsl:template match="/X12">
        <xsl:copy>
          <xsl:apply-templates/>
        </xsl:copy>
      </xsl:template>
      
      <xsl:template match="ST">
        <PivotNode1>
          <ST>
            <xsl:value-of select="."/>
          </ST>
          <BR>
            <xsl:value-of select="following-sibling::BR/BR08"/>
          </BR>
          <BR01>
            <xsl:value-of select="following-sibling::BR/BR11"/>
          </BR01>
          <G62Sub1>
            <xsl:value-of select="following-sibling::G62[1]/G6203"/>
          </G62Sub1>
          <G62Sub2>
            <xsl:value-of select="following-sibling::G62[1]/G6204"/>
          </G62Sub2>
          <G62Sub3>
            <xsl:value-of select="following-sibling::G62[2]/G6203"/>
          </G62Sub3>
          <G62Sub4>
            <xsl:value-of select="following-sibling::G62[2]/G6204"/>
          </G62Sub4>
          <LMSub1>
            <xsl:value-of select="following-sibling::LM[1]/LM01"/>
          </LMSub1>
          <LQSub1>
            <xsl:value-of select="following-sibling::LQ[1]/LQ02"/>
          </LQSub1>
          <N1Sub1>
            <xsl:value-of select="following-sibling::N1[1]/N104"/>
          </N1Sub1>
          <N1Sub2>
            <xsl:value-of select="following-sibling::N1[2]/N104"/>
          </N1Sub2>
          <SE>
            <xsl:value-of select="following-sibling::SE/SE01"/>
          </SE>
          <xsl:apply-templates />
        </PivotNode1>
      </xsl:template>

      <xsl:template match="QTY">
        <PivotNode>
          <QTY>
            <xsl:value-of select="."/>
          </QTY>
          <QTY01>
            <xsl:value-of select="QTY04"/>
          </QTY01>
          <N104Sub>
            <xsl:value-of select="QTY05"/>
          </N104Sub>
          <N9Sub1>
            <xsl:value-of select="following-sibling::N9[1]/N902"/>
          </N9Sub1>
          <N9Sub2>
            <xsl:value-of select="following-sibling::N9[2]/N902"/>
          </N9Sub2>
          <G62Sub5>
            <xsl:value-of select="following-sibling::G62/G6203"/>
          </G62Sub5>
          <G62Sub6>
            <xsl:value-of select="following-sibling::G62/G6204"/>
          </G62Sub6>
          <G62Sub7>
            <xsl:value-of select="following-sibling::G62/G6205"/>
          </G62Sub7>
          <G62Sub8>
            <xsl:value-of select="following-sibling::G62/G6206"/>
          </G62Sub8>
          <LMSub2>
            <xsl:value-of select="following-sibling::LM[1]/LM01"/>
          </LMSub2>
          <LQSub2>
            <xsl:value-of select="following-sibling::LQ[1]/LQ02"/>
          </LQSub2>
          <LQSub3>
            <xsl:value-of select="following-sibling::LQ[2]/LQ02"/>
          </LQSub3>
          <LQSub4>
            <xsl:value-of select="following-sibling::LQ[3]/LQ02"/>
          </LQSub4>
          <LQSub5>
            <xsl:value-of select="following-sibling::LQ[4]/LQ02"/>
          </LQSub5>
          <LQSub6>
            <xsl:value-of select="following-sibling::LQ[5]/LQ02"/>
          </LQSub6>
          <LQSub7>
            <xsl:value-of select="following-sibling::LQ[6]/LQ02"/>
          </LQSub7>
          <LQSub8>
            <xsl:value-of select="following-sibling::LQ[7]/LQ02"/>
          </LQSub8>
          <LQSub9>
            <xsl:value-of select="following-sibling::LQ[8]/LQ02"/>
          </LQSub9>
          <FA1>
            <xsl:value-of select="following-sibling::FA1/FA102"/>
          </FA1>
          <FA2>
            <xsl:value-of select="following-sibling::FA2/FA202"/>
          </FA2>
        </PivotNode>
      </xsl:template>

    </xsl:stylesheet>
  </sx:xslt>

  <sx:flatFile id="dlssFile">
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:positionalField name="placeholder" width="1"/>
        <sx:when test="/dlss1">
          <sx:flatRecordType name="dlss1">
            <sx:positionalField name="LQSub1" width="3"/>
            <sx:positionalField name="N1Sub1" width="3"/>
            <sx:positionalField name="BR" width="4"/>
            <sx:positionalField name="BR01" width="3"/>
            <sx:positionalField name="spaces1" width="15"/>
            <sx:positionalField name="N1Sub2" width="6"/>
            <sx:positionalField name="G62Sub1" width="2"/>
            <sx:positionalField name="G62Sub2" width="3"/>
            <sx:positionalField name="G62Sub3" width="4"/>
            <sx:positionalField name="G62Sub4" width="2"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="/dlss2">
          <sx:flatRecordType name="dlss2">
            <sx:positionalField name="LQSub2" width="3"/>
            <sx:positionalField name="N104Sub" width="2"/>
            <sx:positionalField name="N9Sub2" width="14"/>
            <sx:positionalField name="LQSub5" width="1"/>
            <sx:positionalField name="QTY01" width="3"/>
            <sx:positionalField name="N9Sub1" width="14"/>
            <sx:positionalField name="LQSub7" width="6"/>
            <sx:positionalField name="LQSub6" width="1"/>
            <sx:positionalField name="FA2" width="2"/>
            <sx:positionalField name="LQSub4" width="2"/>
            <sx:positionalField name="LQSub8" width="3"/>
            <sx:positionalField name="LQSub9" width="2"/>
            <sx:positionalField name="G62Sub6" width="2"/>
            <sx:positionalField name="LQSub3" width="3"/>
            <sx:positionalField name="G62Sub7" width="3"/>
            <sx:positionalField name="G62Sub8" width="3"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:inverseRecordMapping id="dlssToFileMapping">
    <sx:onSubtree path="/X12/PivotNode1">
      <sx:flattenSubtree recordType="dlss1">
        <sx:subtreeFieldMap select="LQSub1" field="LQSub1"/>
        <sx:subtreeFieldMap select="N1Sub1" field="N1Sub1"/>
        <sx:subtreeFieldMap select="BR" field="BR"/>
        <sx:subtreeFieldMap select="BR01" field="BR01"/>
        <sx:subtreeFieldMap select="N1Sub2" field="N1Sub2"/>
        <sx:subtreeFieldMap select="G62Sub1" field="G62Sub1"/>
        <sx:subtreeFieldMap select="G62Sub2" field="G62Sub2"/>
        <sx:subtreeFieldMap select="G62Sub3" field="G62Sub3"/>
        <sx:subtreeFieldMap select="G62Sub4" field="G62Sub4"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
    <sx:onSubtree path="/X12/PivotNode">
      <sx:flattenSubtree recordType="dlss2">
        <sx:subtreeFieldMap select="LQSub2" field="LQSub2"/>
        <sx:subtreeFieldMap select="N104Sub" field="N104Sub"/>
        <sx:subtreeFieldMap select="N9Sub2" field="N9Sub2"/>
        <sx:subtreeFieldMap select="LQSub5" field="LQSub5"/>
        <sx:subtreeFieldMap select="QTY01" field="QTY01"/>
        <sx:subtreeFieldMap select="N9Sub1" field="N9Sub1"/>
        <sx:subtreeFieldMap select="LQSub7" field="LQSub7"/>
        <sx:subtreeFieldMap select="LQSub6" field="LQSub6"/>
        <sx:subtreeFieldMap select="FA2" field="FA2"/>
        <sx:subtreeFieldMap select="LQSub4" field="LQSub4"/>
        <sx:subtreeFieldMap select="LQSub8" field="LQSub8"/>
        <sx:subtreeFieldMap select="LQSub9" field="LQSub9"/>
        <sx:subtreeFieldMap select="G62Sub6" field="G62Sub6"/>
        <sx:subtreeFieldMap select="LQSub3" field="LQSub3"/>
        <sx:subtreeFieldMap select="G62Sub7" field="G62Sub7"/>
        <sx:subtreeFieldMap select="G62Sub8" field="G62Sub8"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

</sx:resources>


You can run this example on the command line by entering


servingxml -i Edi517M2xml.xml -r resources-xml-517M.xml -o Edi517M.txt xmltodlss 

A Directory of XML Documents to a Flat File

This example shows a resources script that will read a number of XML documents in a directory and transform them into a single CSV file. It illustrates the use of the sx:documentSequence element.

The input is a directory of XML documents.


  countries-1.xml
  countries-2.xml
  countries-3.xml
  countries-4.xml
  countries-5.xml

The desired output is one CSV file containing all of the country entries.


country_code,country_name
ABW,ARUBA
ADH,UNITED ARAB EMIRATES
...
ZWE,ZIMBABWE

The resources script resources-countrySequence.xml does the transformation.


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="countries-to-csv">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="countriesToFileMapping"/>
        <sx:transform>
          <sx:documentSequence wrapWith="result">
            <sx:recordStream>
              <sx:directoryReader directory="data">
                <sx:fileFilter pattern="countries.*[.]xml"/>
              </sx:directoryReader>
            </sx:recordStream>
            <sx:transform>
              <sx:document>
                <sx:fileSource directory="{parentDirectory}" file="{name}"/>
              </sx:document> 
              <msv:schemaValidator>
                <sx:urlSource url="data/countries.xsd"/>
              </msv:schemaValidator>
            </sx:transform>
          </sx:documentSequence>
        </sx:transform>
      </sx:subtreeRecordReader>

      <sx:flatFileWriter>
        <sx:flatFile ref="countriesFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="countriesFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileHeader>
      <sx:flatRecordType ref="countryType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="countryType"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType name="country" id="countryType">
    <sx:fieldDelimiter value=","/>
    <sx:delimitedField name="code" label="country_code"/>
    <sx:delimitedField name="name" label="country_name"/>
  </sx:flatRecordType>

  <sx:inverseRecordMapping id="countriesToFileMapping">
    <sx:onSubtree path="/result/countries/country">
      <sx:flattenSubtree  recordType="country">
        <sx:subtreeFieldMap select="countryName" field="name"/>
        <sx:subtreeFieldMap select="@countryCode" field="code"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

</sx:resources>

The effect of the sx:documentSequence instruction is to read all the documents in the data directory whose file names match the regular expression "countries.*[.]xml". The content of these documents is wrapped in the root element result.

You can run this example on the command line by entering


servingxml -r resources-countrySequence.xml -o output/countries.csv countries-to-csv

Summary and Detail XML Documents to a Flat File (eobclaims)

This example shows a resources script that will read a number of XML documents in a directory and transform them into a single CSV file. It illustrates the use of the sx:documentSequence and saxon:xquery, elements.

The input is a directory of XML documents.


eobclaims1.xml
eobclaims2.xml
eobclaimshdrtlr.xml

The desired output is one CSV file containing all of the country entries.


ADM2301EOBS GENERATED ON 06/1 3/08                                                                                                                                                                                                                                                     
HDR06/13/08200815501300101MEMBER01          BISHOP         
ADR1234 TEST DR                                                                                       ARVADA            
HDR06/13/08200815501300102MEMBER02          BISHOP         
ADR5678 TEST DR                                                                                       ARVADA                                 
TLRTOTAL EOBS GENERATED: 0000001000000000000010000018                                                                                                                                                                                                                                                       

The resources script resources-eobclaims.xml does the transformation. Note that this script includes the shared eobclaims record structure declarations.


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:saxon="http://www.servingxml.com/extensions/saxon">

  <sx:include href="resources-eobclaims-flatfile.xml"/>

  <sx:service id="eobclaims-to-flat">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:inverseRecordMapping ref="eobclaims-to-flat-mapping"/>
        <sx:transform>
          <sx:documentSequence wrapWith="result">
            <sx:content ref="eobclaims-document-header"/>
            <sx:content ref="eobclaims-document-body"/>
            <sx:content ref="eobclaims-document-trailer"/>
          </sx:documentSequence>
        </sx:transform>
      </sx:subtreeRecordReader>

      <sx:flatFileWriter>
        <sx:flatFile ref="eobclaims-file"/>
        <sx:fileSink file="output/eobclaims.txt"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <saxon:xquery id="eobclaims-document-header">
    <sx:preserveMarkup>
      <header>
          {doc('data/eobclaimshdrtlr.xml')/headertrailer/ADMIN_RECTYPE}
      </header>
    </sx:preserveMarkup>
  </saxon:xquery>

  <saxon:xquery id="eobclaims-document-trailer">
    <sx:preserveMarkup>
        <trailer>
            {doc('data/eobclaimshdrtlr.xml')/headertrailer/TLR-REC-TYPE}
        </trailer>
    </sx:preserveMarkup>
  </saxon:xquery>

  <sx:documentSequence id="eobclaims-document-body">
    <sx:directoryReader directory="data">
      <sx:fileFilter pattern=".*[1-9].xml"/>
    </sx:directoryReader>
    <sx:document>
      <sx:fileSource directory="{parentDirectory}" file="{name}"/>
    </sx:document>
  </sx:documentSequence>

  <sx:inverseRecordMapping id="eobclaims-to-flat-mapping">
    <sx:onSubtree path="header/ADMIN_RECTYPE">
      <sx:flattenSubtree recordType="ADM">
        <sx:subtreeFieldMap select="@record_type" field="record_type"/>
        <sx:subtreeFieldMap select="ADM_0001_TEXT" field="ADM_0001_TEXT"/>
        <sx:subtreeFieldMap select="ADM_0001_RUN_DATE" field="ADM_0001_RUN_DATE"/>
        <sx:subtreeFieldMap select="FILLER" field="FILLER"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
    <sx:onSubtree path="eob/HEADER_RECTYPE">
      <sx:flattenSubtree recordType="HDR">
        <sx:subtreeFieldMap select="@record_type" field="record_type"/>
        <sx:subtreeFieldMap select="HDR_STMT_DATE" field="HDR_STMT_DATE"/>
        <sx:subtreeFieldMap select="HDR_CLAIM_NUMBER" field="HDR_CLAIM_NUMBER"/>
        <sx:subtreeFieldMap select="HDR_PATIENT_NAME_FIRST" field="HDR_PATIENT_NAME_FIRST"/>
        <sx:subtreeFieldMap select="HDR_PATIENT_NAME_LAST" field="HDR_PATIENT_NAME_LAST"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
    <sx:onSubtree path="eob/ADR_REC_TYPE">
      <sx:flattenSubtree recordType="HDR">
        <sx:subtreeFieldMap select="@record_type" field="record_type"/>
        <sx:subtreeFieldMap select="ADR_8512_ATTN" field="ADR_8512_ATTN"/>
        <sx:subtreeFieldMap select="HDR_CLAIM_NUMBER" field="HDR_CLAIM_NUMBER"/>
        <sx:subtreeFieldMap select="ADR_8514_CITY" field="ADR_8514_CITY"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
    <sx:onSubtree path="trailer/TLR-REC-TYPE">
      <sx:flattenSubtree recordType="TLR">
        <sx:subtreeFieldMap select="@record_type" field="record_type"/>
        <sx:subtreeFieldMap select="TLR_0001_TEXT" field="TLR_0001_TEXT"/>
        <sx:subtreeFieldMap select="TLR_0001_COUNT" field="TLR_0001_COUNT"/>
        <sx:subtreeFieldMap select="FILLER" field="FILLER"/>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

</sx:resources>

You can run this example on the command line by entering


servingxml -r resources-eobclaims.xml -o output/eobclaims.txt eobclaims-to-flat

Flat File to Flat File

Converting a positional file to a pipe delimited file with header (positional to delimited)

The example below illustrates how to prepare a resources script that will convert the positional flat file books.txt into the delimited flat file new-books.txt.

The input file is a positional file.

Figure 166. Input positional flat file books.txt


CAuthor                        Title                              Price

FCharles Bukowski              Factotum                           22.95
FSergei Lukyanenko             The Night Watch                    17.99
FAndrew Crumey                 Mr Mee                             22.00
CSteven John Metsker           Building Parsers with Java         39.95

This is a trailer record


The desired output file is a pipe delimited flat file.

Figure 167. Output pipe delimited flat file new-books.txt


Author|Category|Title|Price
Charles Bukowski|F|Factotum|22.95
Sergei Lukyanenko|F|The Night Watch|17.99
Andrew Crumey|F|Mr Mee|22.00
Steven John Metsker|C|Building Parsers with Java|39.95


The following resources script does the transformation.

Figure 168. Resources script resources-conversion.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="new-books"> 
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="oldBooksFlatFile"/>
      </sx:flatFileReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="newBooksFlatFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="newBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="newBookType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="newBookType"/>
    </sx:flatFileBody>
  </sx:flatFile>      

  <sx:flatRecordType id="newBookType" name="newBookType">
    <sx:fieldDelimiter value="|"/>
    <sx:delimitedField name="author" label="Author"/>
    <sx:delimitedField name="category" label="Category"/>
    <sx:delimitedField name="title" label= "Title"/>
    <sx:delimitedField name="price" label="Price"/>
  </sx:flatRecordType>

  <sx:flatFile id="oldBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="oldBookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="oldBookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>      

  <sx:flatRecordType id="oldBookType" name="oldBookType">
    <sx:positionalField name="category" width="1"/>
    <sx:positionalField name="author" width="30"/>
    <sx:positionalField name="title" width="30"/>
    <sx:positionalField name="price" width="10" justify="right"/>
  </sx:flatRecordType>
  
</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-conversion.xml  -i data/books.txt 
    -o output/new-books.txt new-books

Converting a positional file to a new positional file with default field values (positional defaults)

The example below illustrates how to prepare a resources script that will convert the positional flat file books.txt into the new positional flat file book-defaults.txt.

The input file is a positional file.

Figure 169. Input positional flat file books.txt


CAuthor                        Title                              Price

FCharles Bukowski              Factotum                           22.95
FSergei Lukyanenko             The Night Watch                    17.99
FAndrew Crumey                 Mr Mee                             22.00
CSteven John Metsker           Building Parsers with Java         39.95

This is a trailer record


The desired output file is a new positional flat file with a different field ordering, a comma instead of a period in "Price", a new "Pub Date" field initialized to spaces, and a new Publisher field initialized to "Acme".

Figure 170. Output positional flat file books-defaults.txt


Author                        Title                         CPub Date  Publisher           Price
Charles Bukowski              Factotum                      F          Acme                22,95
Sergei Lukyanenko             The Night Watch               F          Acme                17,99
Andrew Crumey                 Mr Mee                        F          Acme                22,00
Steven John Metsker           Building Parsers with Java    C          Acme                39,95


This can be accomplished with the following resources script..

Figure 171. Resources script resources-bookDefaults.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="book-defaults">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="oldBooksFlatFile"/>
      </sx:flatFileReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="newBooksFlatFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="newBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="newBookType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="newBookType"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType id="newBookType" name="newBookType">
    <sx:positionalField name="author" width="30" label="Author"/>
    <sx:positionalField name="title" width="30" label="Title"/>
    <sx:positionalField name="category" width="1" label="C"/>
    <sx:positionalField name="pubDate" width="10" label="Pub Date"/>
    <sx:positionalField name="publisher" width="15" label="Publisher">
      <sx:defaultValue>Acme</sx:defaultValue>
    </sx:positionalField>
    <sx:positionalField name="price2" width="10" justify="right" label="Price">
      <sx:defaultValue>
        <sx:findAndReplace searchFor ="." replaceWith="," useRegex="false">
          <sx:toString value="{price}"/>    
        </sx:findAndReplace>
      </sx:defaultValue>
    </sx:positionalField>
  </sx:flatRecordType>

  <sx:flatFile id="oldBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="oldBookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="oldBookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>

  <sx:flatRecordType id="oldBookType" name="oldBookType">
    <sx:positionalField name="category" width="1"/>
    <sx:positionalField name="author" width="30"/>
    <sx:positionalField name="title" width="30"/>
    <sx:positionalField name="price" width="10" justify="right"/>
  </sx:flatRecordType>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-bookDefaults.xml  -i data/books.txt 
    -o output/book-defaults.txt book-defaults

Converting an ASCII Positional File to an EBCDIC Delimited File, and Vice Versa (ASCII-EBCDIC)

The example below illustrates how to prepare a resources script that will convert the ASCII books.txt file of the last example to an EBCDIC file, and vice versa.

The following resources script does the transformation.

Figure 172. Resources script resources-ascii-ebcdic.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="ascii2ebcdic"> 
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="asciiBooksFile"/>
      </sx:flatFileReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="ebcdicBooksFile"/>
        <sx:defaultStreamSink encoding="Cp1047"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:service id="ebcdic2ascii"> 
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="ebcdicBooksFile"/>
        <sx:defaultStreamSource encoding="Cp1047"/>
      </sx:flatFileReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="asciiBooksFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>
  
  <sx:flatFile id="ebcdicBooksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="ebcdicBookType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="ebcdicBookType"/>
    </sx:flatFileBody>
  </sx:flatFile>      

  <sx:flatRecordType id="ebcdicBookType" name="ebcdicBookType">
    <sx:fieldDelimiter value="|"/>
    <sx:delimitedField name="author" label="Author"/>
    <sx:delimitedField name="category" label="Category"/>
    <sx:delimitedField name="title" label= "Title"/>
    <sx:delimitedField name="price" label="Price"/>
  </sx:flatRecordType>
  
  <sx:flatFile id="asciiBooksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="asciiBookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="asciiBookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>      

  <sx:flatRecordType id="asciiBookType" name="asciiBookType">
    <sx:positionalField name="category" width="1"/>
    <sx:positionalField name="author" width="30"/>
    <sx:positionalField name="title" width="30"/>
    <sx:positionalField name="price" width="10" justify="right"/>
  </sx:flatRecordType>
  
</sx:resources>


You can run convert the ASCII file books.txt to the EBCDIC file books-ebcdic.dat with the command


servingxml -r resources-ascii-ebcdic.xml -i data/books.txt 
    -o output/books-ebcdic.dat ascii2ebcdic

You can run convert the EBCDIC file books-ebcdic.dat to the ASCII file books-ascii.txt with the command


servingxml -r resources-ascii-ebcdic.xml -i data/books-ebcdic.dat 
    -o output/books-ascii.txt ebcdic2ascii

Converting an ASCII Positional File to an EBCDIC Positional File with a Cobol Packed Decimal Field, and Vice Versa (ASCII-Packed)

The example below illustrates how to prepare a resources script that will convert the ASCII books.txt file to an EBCDIC positional file with a Cobol packed decimal field, and vice versa.

The following resources script does the transformation.

Figure 173. Resources script resources-ascii-ebcdic.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="books2ebcdic-packed"> 
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="booksFile"/>
      </sx:flatFileReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="booksFile2"/>
        <sx:defaultStreamSink encoding="Cp1047"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:service id="ebcdic-packed2books"> 
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="booksFile2"/>
        <sx:defaultStreamSource encoding="Cp1047"/>
      </sx:flatFileReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="booksFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>
  
  <sx:flatFile id="booksFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="bookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>      

  <sx:flatFile id="booksFile2" lineDelimited="false">
    <sx:flatFileBody>
      <sx:flatRecordType ref="bookType2"/>
    </sx:flatFileBody>
  </sx:flatFile>      

  <sx:flatRecordType id="bookType" name="bookType">
    <sx:positionalField name="category" label="Category" width="1"/>
    <sx:positionalField name="author" label="Author" width="30"/>
    <sx:positionalField name="title" label="Title" width="30"/>
    <sx:positionalField name="price" label="Price" width="10" justify="right"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="bookType2" name="bookType">
    <sx:positionalField name="category" label="Category" width="1"/>
    <sx:positionalField name="author" label="Author" width="30"/>
    <sx:positionalField name="title" label="Title" width="30"/>
    <sx:packedDecimalField name="price" label="Price" digitCount="10" decimalPlaces="2"/>
  </sx:flatRecordType>
  
</sx:resources>


You can run convert the ASCII file, books.txt, to the EBCDIC file with the packed decimal price field, books-ebcdic.dat, with the command


servingxml -r resources-books_ebcdic_packed.xml -i data/books.txt 
    -o output/books_ebcdic_packed.dat   books2ebcdic-packed

You can run convert the EBCDIC file with the packed decimal price field, books_ebcdic_packed.dat, to the ASCII file, books-ascii_unpacked.txt, with the command


servingxml -r resources-books_ebcdic_packed.xml -i data/books_ebcdic_packed.dat 
    -o output/books_ascii_unpacked.txt   ebcdic-packed2books

Converting a positional file to a pipe delimited file with record filtering (book orders)

The example below illustrates how to prepare a resources script that will convert the positional flat file bookorders-pos.txt into the pipe-delimited flat file bookorders-delim.txt.

The input file is a positional file.

Figure 174. Input positional flat file bookorders-pos.txt


CAuthor                        Title                              Price InvoiceNo InvoiceDate

FCharles Bukowski              Factotum                           22.95 001  12/Mar/2005
FSergei Lukyanenko             The Night Watch                    17.99 002  13/Mar/2005
FAndrew Crumey                 Mr Mee                             22.00 003 14/June/2005
CSteven John Metsker           Building Parsers with Java         39.95 004 15/June/2005

This is a trailer record


The desired output file is a pipe delimited flat file.

Figure 175. Output pipe delimited flat file bookorders-delim.txt


Author|Category|Title|Price|InvoiceNo|InvoiceDate
Charles Bukowski|F|Factotum|22.95|INV001|12/03/2005
Sergei Lukyanenko|F|The Night Watch|17.99|INV002|13/03/2005
Andrew Crumey|F|Mr Mee|22.00|INV003|14/06/2005
Steven John Metsker|C|Building Parsers with Java|39.95|INV004|15/06/2005


Note the following:

  • The invoice number has the text "INV" prepended, e.g. "001" becomes "INV001"

  • The month labels Mar, June etc. have been converted to numbers.

The following resources script does the transformation.

Figure 176. Resources script resources-bookorders.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="orders-pos-delim"> 
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="posBooksFlatFile"/>                                                                                          
      </sx:flatFileReader>
      <sx:replaceRecord ref="changeBookOrder"/>
      <sx:flatFileWriter>
        <sx:flatFile ref="delimBooksFlatFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:replaceRecord id="changeBookOrder">
    <sx:newRecord fields="*">
      <sx:newField name="invoiceno">
        <sx:findAndReplace searchFor ="([0-9]{3})" replaceWith ="INV$1"><sx:toString value="{invoiceno}"/></sx:findAndReplace>
      </sx:newField>
      <sx:newField name="invoicedate">
        <sx:convertDate fromFormat="dd/MMM/yyyy"
                        toFormat="dd/MM/yyyy">
          <sx:toString value="{invoicedate}"/>
        </sx:convertDate>
      </sx:newField>
    </sx:newRecord>
  </sx:replaceRecord>

   <sx:flatFile id="delimBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="delimBookType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="delimBookType"/>
    </sx:flatFileBody>
  </sx:flatFile>      

  <sx:flatRecordType id="delimBookType" name="delimBookType">
    <sx:fieldDelimiter value="|"/>
    <sx:delimitedField name="author" label="Author"/>
    <sx:delimitedField name="category" label="Category"/>
    <sx:delimitedField name="title" label= "Title"/>
    <sx:delimitedField name="price" label="Price"/>
    <sx:delimitedField name="invoiceno" label="InvoiceNo"/>
    <sx:delimitedField name="invoicedate" label="InvoiceDate"/>
  </sx:flatRecordType>
  
  <sx:flatFile id="posBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="posBookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="posBookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>      

  <sx:flatRecordType id="posBookType" name="posBookType">
    <sx:positionalField name="category" width="1"/>
    <sx:positionalField name="author" width="30"/>
    <sx:positionalField name="title" width="30"/>
    <sx:positionalField name="price" width="10" justify="right"/>
    <sx:positionalField name="invoiceno" width="4" justify="right"/>
    <sx:positionalField name="invoicedate" width="13" justify="right"/>
  </sx:flatRecordType>
  
</sx:resources>


You can run this example on the command line by entering


servingxml  -i data/bookorders-pos.txt -r resources-bookorders.xml 
    -o output/bookorders-delim.txt orders-pos-delim 

Checking the Integrity of a Flat File ("checked books")

The example below illustrates how to prepare a resources script for checking and signing flat files with CRC values and byte counts. Here, the input file is the positional flat file checkedBooks.txt, to be converted into the pipe-delimited flat file signedBooks.txt.

The input file is a positional file.

Figure 177. Input positional flat file checkedBooks.txt


356      2360012630

FCharles Bukowski              Factotum                           22.95 001  12/Mar/2005
FSergei Lukyanenko             The Night Watch                    17.99 002  13/Mar/2005
FAndrew Crumey                 Mr Mee                             22.00 003  14/Jun/2005
CSteven John Metsker           Building Parsers with Java         39.95 004  15/Jun/2005

This is a trailer record


Note the following.

  • The file has a two line header section, followed by a data section, followed by a two line trailer section.

  • The first line of the header contains two positional fields, the size of the data section in bytes, followed by the CRC for the data section.

The desired output file is a pipe delimited flat file.

Figure 178. Output pipe delimited flat file signedBooks.txt


Author|Category|Title|Price|InvoiceNo|InvoiceDate
237|1661800843
Charles Bukowski|F|Factotum|22.95|001|12/Mar/2005
Sergei Lukyanenko|F|The Night Watch|17.99|002|13/Mar/2005
Andrew Crumey|F|Mr Mee|22.00|003|14/Jun/2005
Steven John Metsker|C|Building Parsers with Java|39.95|004|15/Jun/2005


The output file has a two line header section. The two delimited fields in the second line store the size and CRC of the data section below.

The following resources script does the transformation.

Figure 179. Resources script resources-checkedBooks.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="books-pos-delim">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:flatFile ref="posBooksFlatFile"/>
      </sx:flatFileReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="delimBooksFlatFile"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="delimBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="delimBookType"/>
      <sx:flatRecordType ref="delimBookHeaderType"/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="delimBookType"/>
    </sx:flatFileBody>
    <sx:flatFileSignature recordType="bookHeaderType" field="filecrc" method="crc"/>
    <sx:flatFileSignature recordType="bookHeaderType" field="filesize" method="size"/>
  </sx:flatFile>

  <sx:flatRecordType id="delimBookType" name="delimBookType">
    <sx:fieldDelimiter value="|"/>
    <sx:delimitedField name="author" label="Author"/>
    <sx:delimitedField name="category" label="Category"/>
    <sx:delimitedField name="title" label= "Title"/>
    <sx:delimitedField name="price" label="Price"/>
    <sx:delimitedField name="invoiceno" label="InvoiceNo"/>
    <sx:delimitedField name="invoicedate" label="InvoiceDate"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="delimBookHeaderType" name="bookHeaderType">
    <sx:fieldDelimiter value="|"/>
    <sx:delimitedField name="filesize"/>
    <sx:delimitedField name="filecrc"/>
  </sx:flatRecordType>

  <sx:flatFile id="posBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="posBookHeaderType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>

    <sx:flatFileBody>
      <sx:flatRecordType ref="posBookType"/>
    </sx:flatFileBody>

    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>

    <sx:flatFileSignature recordType="bookHeaderType" field="filesize" method="size"/>
    <sx:flatFileSignature recordType="bookHeaderType" field="filecrc" method="crc"/>
  </sx:flatFile>

  <sx:flatRecordType id="posBookHeaderType" name="bookHeaderType">
    <sx:positionalField name="filesize" width="9"/>
    <sx:positionalField name="filecrc" width="10"/>
  </sx:flatRecordType>

  <sx:flatRecordType id="posBookType" name="posBookType">
    <sx:positionalField name="category" width="1"/>
    <sx:positionalField name="author" width="30"/>
    <sx:positionalField name="title" width="30"/>
    <sx:positionalField name="price" width="10" justify="right"/>
    <sx:positionalField name="invoiceno" width="4" justify="right"/>
    <sx:positionalField name="invoicedate" width="13" justify="right"/>
  </sx:flatRecordType>

</sx:resources>


Note the following.

  • The sx:recordContent element contains two sx:flatFileSignature elements, which define the rules for checking the CRC and size when reading the file, and signing the header or trailer with a CRC and size when writing the file.

  • The sx:flatFileSignature element has attributes recordType and field, which identify where in the header or trailer the signature value belongs.

You can run this example on the command line by entering


servingxml  -i data/checkedBooks.txt -r resources-checkedBooks.xml 
    -o output/checkedBooks-delim.txt books-pos-delim 

If the CRC value in the header does not match the computed CRC value for the data, processing will be stopped and a message will be written to the log:


CRC integrity check failed.  Expected 2360012630, found 252430032.

Converting a directory of book positional files to pipe delimited files (new format)

The example below illustrates how to prepare a resources script that will process all the files in the data directory matching the regular expression "books.*[.]txt", convert them from positional flat file formats to pipe delimited formats, and write them out to the output directory.

This time the input is a directory of files.

Figure 180. Directory containing books fixed record length input files


[.]                     countries.csv           multivalued-field.csv
[..]                    countries.xsd           plans.txt
3545_JH4DA3_4_H_.xml    country-record.xsd      tasks.csv
bad-countries.csv       exotics.txt             timesheets.csv
books.20040613.txt      hot-record.xsd          trades.txt
books.20040802.txt      hot-record.xsx
books.txt               hot.txt
books.xml               messages.properties

We want to take the positional files whose names match the regular expression "(books.*)[.]txt" and convert them all into pipe delimited files.

Figure 181. Books pipe delimited output files.


books-new.txt            books.20040802-new.txt
books.20040613-new.txt


The following resources script does the transformation.

Figure 182. Resources script resources-books2csv.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="all-books"> 
    <sx:recordStream>
      <sx:directoryReader directory="data">
        <sx:fileFilter pattern="books.*[.]txt"/>
      </sx:directoryReader>
      <sx:processRecord>
        <sx:parameter name="output-file">
          <sx:findAndReplace searchFor ="(books.*)[.]txt" replaceWith ="$1-new.txt"><sx:toString value="{name}"/></sx:findAndReplace>
        </sx:parameter>   
        <sx:recordStream>
          <sx:flatFileReader>
            <sx:fileSource directory="{parentDirectory}" file="{name}"/>
            <sx:flatFile ref="oldBooksFlatFile"/>
          </sx:flatFileReader>
          <sx:flatFileWriter>
            <sx:fileSink directory="output" file="{$output-file}"/> 
            <sx:flatFile ref="newBooksFlatFile"/>
          </sx:flatFileWriter>
        </sx:recordStream>
      </sx:processRecord>
    </sx:recordStream>
  </sx:service>

  <sx:flatRecordType name="newBookType">
    <sx:fieldDelimiter value="|"/>
    <sx:delimitedField name="author" label="Author"/>
    <sx:delimitedField name="category" label="Category"/>
    <sx:delimitedField name="title" label= "Title"/>
    <sx:delimitedField name="price" label="Price"/>
  </sx:flatRecordType>

  <sx:flatFile id="oldBooksFlatFile">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="oldBookType"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="oldBookType"/>
    </sx:flatFileBody>
    <sx:flatFileTrailer>
      <sx:annotationRecord></sx:annotationRecord>
      <sx:annotationRecord>This is a trailer record</sx:annotationRecord>
    </sx:flatFileTrailer>
  </sx:flatFile>      

  <sx:flatRecordType id="oldBookType" name="oldBookType">
    <sx:positionalField name="category" width="1"/>
    <sx:positionalField name="author" width="30"/>
    <sx:positionalField name="title" width="30"/>
    <sx:positionalField name="price" width="10" justify="right"/>
  </sx:flatRecordType>

</sx:resources>


You can run this example on the command line by entering


servingxml  -r resources-conversion.xml all-books

Generated records to multiple files (multiple_output_files)

The example below illustrates how to prepare a resources script that will send the the output of a custom record reader to multiple output files, depending on parameter and record field values supplied by the custom record reader.

The input records are generated by the following Java class.

Figure 183. TradeRecordReader Java class


package flat2flat;

import com.servingxml.app.ServiceContext;
import com.servingxml.components.recordio.AbstractRecordReader;
import com.servingxml.app.Flow;
import com.servingxml.util.Name;
import com.servingxml.util.QualifiedName;
import com.servingxml.util.ServingXmlException;
import com.servingxml.util.record.Record;
import com.servingxml.util.record.RecordBuilder;
import com.servingxml.util.record.ParameterBuilder;

public class TradeRecordReader extends AbstractRecordReader {
  private static final Name FEED_NAME = new QualifiedName("feed");
  private static final Name TRADE_RECORD_TYPE = new QualifiedName("trade");
  private static final Name TRANSACTION_RECORD_TYPE = new QualifiedName("transaction");
  private static final Name RECORD_TYPE_NAME = new QualifiedName("record_type");
  private static final Name ID_NAME = new QualifiedName("id");
  private static final Name TRADE_DATE_NAME = new QualifiedName("trade_date");
  private static final Name TRADE_TIME_NAME = new QualifiedName("trade_time");
  private static final Name DESCRIPTION_NAME = new QualifiedName("description");
  private static final Name REFERENCE_NAME = new QualifiedName("reference");
  
  public void readRecords(ServiceContext context, Flow flow) 
   {

    //  Start the record stream
    startRecordStream(context, flow);

    ParameterBuilder parameterBuilder = new ParameterBuilder(flow.getParameters());

    RecordBuilder trRecordBuilder = new RecordBuilder(TRADE_RECORD_TYPE);
    RecordBuilder tnRecordBuilder = new RecordBuilder(TRANSACTION_RECORD_TYPE);

    Record newParameters;
    Record record;

    //  Load London trades
    //  Set the parameter feed=LONDON
    parameterBuilder.setString(FEED_NAME,"LONDON");
    newParameters = parameterBuilder.toRecord();

    trRecordBuilder.setString(RECORD_TYPE_NAME,"TR");
    trRecordBuilder.setString(ID_NAME,"0001");
    trRecordBuilder.setString(TRADE_DATE_NAME,"03/25/2005");
    trRecordBuilder.setString(TRADE_TIME_NAME,"01:50:00");
    trRecordBuilder.setString(DESCRIPTION_NAME,"This is a trade record");
    record = trRecordBuilder.toRecord();
    trRecordBuilder.clear();

    //  Write the London "trade" record
    Flow newFlow = flow.replaceParameters(context, newParameters).replaceRecord(context, record);
    writeRecord(context, newFlow);

    tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
    tnRecordBuilder.setString(ID_NAME,"0002");
    tnRecordBuilder.setString(REFERENCE_NAME,"X1234");
    tnRecordBuilder.setString(DESCRIPTION_NAME,"A child transaction");
    record = tnRecordBuilder.toRecord();   
    tnRecordBuilder.clear();

    //  Write the first London "transaction" record
    newFlow = flow.replaceRecord(context, record);
    writeRecord(context, newFlow);

    tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
    tnRecordBuilder.setString(ID_NAME,"0003");
    tnRecordBuilder.setString(REFERENCE_NAME,"X1235");
    tnRecordBuilder.setString(DESCRIPTION_NAME,"Another child transaction");
    record = tnRecordBuilder.toRecord();
    tnRecordBuilder.clear();

    //  Write the second London "transaction" record
    newFlow = flow.replaceRecord(context, record);
    writeRecord(context, newFlow);

    //  Load Toronto trades
    //  Set the parameter feed=TORONTO
    parameterBuilder.setString(FEED_NAME,"TORONTO");
    newParameters = parameterBuilder.toRecord();

    trRecordBuilder.setString(RECORD_TYPE_NAME,"TR");
    trRecordBuilder.setString(ID_NAME,"0004");
    trRecordBuilder.setString(TRADE_DATE_NAME,"03/25/2005");
    trRecordBuilder.setString(TRADE_TIME_NAME,"04:50:00");
    trRecordBuilder.setString(DESCRIPTION_NAME,"This is a trade record");
    record = trRecordBuilder.toRecord();
    trRecordBuilder.clear();

    //  Write the Toronto "trade" record
    newFlow = flow.replaceParameters(context, newParameters).replaceRecord(context, record);
    writeRecord(context, newFlow);

    tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
    tnRecordBuilder.setString(ID_NAME,"0005");
    tnRecordBuilder.setString(REFERENCE_NAME,"X1236");
    tnRecordBuilder.setString(DESCRIPTION_NAME,"A child transaction");
    record = tnRecordBuilder.toRecord();   
    tnRecordBuilder.clear();

    //  Write the first Tronto "transaction" record
    newFlow = flow.replaceRecord(context, record);
    writeRecord(context, newFlow);

    tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
    tnRecordBuilder.setString(ID_NAME,"0006");
    tnRecordBuilder.setString(REFERENCE_NAME,"X1237");
    tnRecordBuilder.setString(DESCRIPTION_NAME,"Another child transaction");
    record = tnRecordBuilder.toRecord();
    tnRecordBuilder.clear();

    //  Write the second Toronto "transaction" record
    newFlow = flow.replaceRecord(context, record);
    writeRecord(context, newFlow);

    //  End the record stream
    endRecordStream(context, flow);
  }
}


With parameter feed=LONDON, the TradeRecordReader class will generate the following stream of records.

record_typeid   
TR000103/25/2005 1:50:00This is a trade record
TN0002X1234A child transaction 
TN0003X1235Another child transaction 

With parameter feed=TORONTO, the TradeRecordReader class will generate the following stream of records.

record_typeid   
TR000403/25/2005 4:50:00This is a trade record
TN0005X1236A child transaction 
TN0006X1237Another child transaction 

The LONDON master trades need to be written to the file trades-London.txt.

Figure 184. Output flat file trades-London.txt


000103/25/200501:50:00This is a trade record        

The LONDON transaction details need to be written to the file transaction-London.txt.


Figure 185. Output flat file transactions-London.txt


0002X1234A child transaction           
0003X1235Another child transaction     


The TORONTO master trades need to be written to the file trades-Toronto.txt.

Figure 186. Output flat file trades-Toronto.txt


000403/25/200504:50:00This is a trade record        

The TORONTO transaction details need to be written to the file transaction-Toronto.txt.


Figure 187. Output flat file transactions-Toronto.txt


0005X1236A child transaction           
0006X1237Another child transaction     


The following resources script does the transformation.

Figure 188. Resources script resources-multiple_output_files.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="trades">
    <sx:recordStream>
      <sx:customRecordReader class="flat2flat.TradeRecordReader"/>
      <!-- Which output file is determined by the parameter "feed"
            and the record field "record_type", which are both
            supplied in the custom record reader.
      -->
      <sx:choose>
        <sx:when test="$feed='LONDON' and record_type = 'TR'">
          <sx:flatFileWriter>
            <sx:fileSink directory="output" file="trades-London.txt"/>
            <sx:flatFile ref="trades"/>
          </sx:flatFileWriter>
        </sx:when>
        <sx:when test="$feed = 'LONDON' and record_type = 'TN'">
          <sx:flatFileWriter>
            <sx:fileSink directory="output" file="transactions-London.txt"/>
            <sx:flatFile ref="transactions"/>
          </sx:flatFileWriter>
        </sx:when>
        <sx:when test="$feed='TORONTO' and record_type = 'TR'">
          <sx:flatFileWriter>
            <sx:fileSink directory="output" file="trades-Toronto.txt"/>
            <sx:flatFile ref="trades"/>
          </sx:flatFileWriter>
        </sx:when>
        <sx:when test="$feed = 'TORONTO' and record_type = 'TN'">
          <sx:flatFileWriter>
            <sx:fileSink directory="output" file="transactions-Toronto.txt"/>
            <sx:flatFile ref="transactions"/>
          </sx:flatFileWriter>
        </sx:when>
      </sx:choose>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="trades">
    <sx:flatFileBody>
          <sx:flatRecordType name="trade">
            <sx:positionalField name="id" width="4"/>
            <sx:positionalField name="trade_date" width="10"/>
            <sx:positionalField name="trade_time" width="8"/>
            <sx:positionalField name="description" width="30"/>
          </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatFile id="transactions">
    <sx:flatFileBody>
          <sx:flatRecordType name="transaction">
            <sx:positionalField name="id" width="4"/>
            <sx:positionalField name="reference" width="5"/>
            <sx:positionalField name="description" width="30"/>
          </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>


Note the following points.

  • The record content is defined by a custom record reader, implemented in the Java class TradeRecordReader.

You can run this example on the command line by

  • Compiling the Java class TradeRecordReader and copying the resulting .class file into the dir/classes directory
  • Entering the command
    
    servingxml -r resources-multiple_output_files.xml trades
    
    

Reading remote directories

This example shows how to read a remote directory and write a listing of its contents to a flat file.

The desired output is a positional flat file showing a remote directory listing.

Figure 189. Output flat file with directory listing


Parent                        Name                          Size      Date Modified                 

/                             incoming                      28672     Mon Dec 20 03:27:00 PST 2004  
/                             lost+found                    16384     Wed Feb 07 00:00:00 PST 2001  
/                             pub                           4096      Sun Oct 17 23:45:00 PDT 2004  
/pub                          sf-overflow                   17        Sun Oct 17 23:45:00 PDT 2004  
/pub                          sourceforge                   581632    Mon Nov 15 18:34:00 PST 2004  
/pub/sourceforge              a                             28672     Fri Mar 19 00:00:00 PST 2004  
/pub/sourceforge/a            a-                            4096      Wed Oct 06 18:36:00 PDT 2004  
/pub/sourceforge/a/a-         a-4                           4096      Tue Dec 16 00:00:00 PST 2003  
/pub/sourceforge/a/a-/a-4     a4-0.01777.tar.gz             173990    Thu Feb 13 00:00:00 PST 2003  
/pub/sourceforge/a/a-/a-4     a4-0.03109.tar.gz             175327    Tue May 06 00:00:00 PDT 2003  

The following resources script does the transformation.

Figure 190. Resources script


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:edt="http://www.servingxml.com/extensions/edtftp">
  
  <edt:ftpClient id="sourceforge" host="upload.sourceforge.net" 
                 user="anonymous" password="xxx"/>

  <sx:service id="remote-dir-listing">
    <sx:recordStream>
      <edt:ftpDirectoryReader remoteDirectory="." recurse="true" maxItems="10">
        <edt:ftpClient ref="sourceforge"/>
      </edt:ftpDirectoryReader>
      <sx:flatFileWriter>
        <sx:flatFile ref="directory-flat-file"/>
      </sx:flatFileWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatRecordType id="directory-record-type" name="directory-record-type">
    <sx:positionalField name="parentDirectory" label="Parent" width="30"/>
    <sx:positionalField name="name" label="Name" width="30"/>
    <sx:positionalField name="size" label="Size" width="10"/>
    <sx:positionalField name="lastModified" label="Date Modified" width="30"/>
  </sx:flatRecordType>

  <sx:flatFile id="directory-flat-file">
    <sx:flatFileHeader>
      <sx:flatRecordType ref="directory-record-type"/>
      <sx:annotationRecord/>
    </sx:flatFileHeader>
    <sx:flatFileBody>
      <sx:flatRecordType ref="directory-record-type"/>
    </sx:flatFileBody>
  </sx:flatFile>      
  
</sx:resources>

You can run this example on the command line by entering


servingxml -r resources.xml -o output/remote-dir-listing.txt remote-dir-listing

Converting a sequential stream of records into multiple batch files

The example below illustrates how to prepare a resources script that wil break up a stream of records into multiple batches.

We want to read the CSV file countries.csv containing 245 country records and produce five files containing 50, 50, 50, 50 and 45 country records.

Figure 191. Batched countries files.


06/12/2007  07:24 PM    <DIR>          .
06/12/2007  07:24 PM    <DIR>          ..
06/12/2007  07:24 PM               814 countries-1.csv
06/12/2007  07:24 PM               795 countries-2.csv
06/12/2007  07:24 PM               786 countries-3.csv
06/12/2007  07:24 PM               784 countries-4.csv
06/12/2007  07:24 PM               793 countries-5.csv


The following resources script does the transformation.

Figure 192. Resources script resources-batchRecords.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="countries">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:fileSource file="data/{$rootName}.csv"/>
        <sx:flatFile ref="countriesFile"/>
      </sx:flatFileReader>
      <sx:batchedRecordWriter batchSize="50">
        <sx:flatFileWriter>
          <sx:fileSink file="output/{$rootName}-{$sx:batchSequenceNumber}.csv"/>
          <sx:flatFile ref="countriesFile"/>
        </sx:flatFileWriter>
      </sx:batchedRecordWriter>
    </sx:recordStream>
  </sx:service>

  <sx:flatFile id="countriesFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="country">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="code"/>
        <sx:delimitedField name="name" trimLeading="true"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>


You can run this example on the command line by entering


servingxml  -r resources-batchRecords.xml countries 
  rootName=countries

Batched Flat Files

Batched sequences of records

The example below illustrates how to prepare a resources script that wil break up a flat file into smaller batches.

Suppose you want to read the CSV file countries.csv containing 245 country records and write it out as five files containing 50, 50, 50, 50 and 45 records.

Figure 193. Batched CSV files.


06/12/2007  07:24 PM    <DIR>          .
06/12/2007  07:24 PM    <DIR>          ..
06/12/2007  07:24 PM               814 countries-1.csv
06/12/2007  07:24 PM               795 countries-2.csv
06/12/2007  07:24 PM               786 countries-3.csv
06/12/2007  07:24 PM               784 countries-4.csv
06/12/2007  07:24 PM               793 countries-5.csv



<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="countries">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:fileSource file="data/{$rootName}.csv"/>
        <sx:flatFile ref="countriesFile"/>
      </sx:flatFileReader>
      <sx:batchedRecordWriter batchSize="50">
        <sx:flatFileWriter>
          <sx:fileSink file="intermediate/{$rootName}-{$sx:batchSequenceNumber}.csv"/>
          <sx:flatFile ref="countriesFile"/>
        </sx:flatFileWriter>
      </sx:batchedRecordWriter>
    </sx:recordStream>
  </sx:service>

  <sx:recordContent id="countries">
    <sx:flatFileReader>
      <sx:fileSource directory="{parentDirectory}" file="{name}"/>
      <sx:flatFile ref="countriesFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="countriesToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="countriesFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="country">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="code"/>
        <sx:delimitedField name="name" trimLeading="true"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

You can run this example on the command line by entering


servingxml  -r resources-batchRecords.xml countries 
  rootName=countries

XML to XML

Extracting subtrees and wrapping them in containing tags

This example shows how to extract a set of subtrees from an XML document according to some criteria and wrap them in containing tags.

The input file is an XML document.

Figure 194. Input XML document books.xml


<?xml version="1.0" encoding="UTF-8"?>
<myns:books xmlns:myns="http://mycompany.com/mynames/" 
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
            xsi:schemaLocation="url2">
  <myns:book categoryCode="F">
    <myns:title>Factotum</myns:title>
    <myns:author>Charles Bukowski</myns:author>
    <myns:price>22.95</myns:price>
    <myns:isbn>0876852630</myns:isbn>
  </myns:book>
  <myns:book categoryCode="F">
    <myns:title>The Night Watch</myns:title>
    <myns:author>Sergei Lukyanenko</myns:author>
    <myns:price>17.99</myns:price>
    <myns:isbn>0434014125</myns:isbn>
  </myns:book>
  <myns:book categoryCode="F">
    <myns:title>Mr Mee</myns:title>
    <myns:author>Andrew Crumey</myns:author>
    <myns:price>22.00</myns:price>
    <myns:isbn>0312282354</myns:isbn>
  </myns:book>
  <myns:book categoryCode="C">
    <myns:title>Building Parsers with Java</myns:title>
    <myns:author>Steven John Metsker</myns:author>
    <myns:price>39.95</myns:price>
    <myns:isbn>0201719622</myns:isbn>
  </myns:book>
</myns:books>


The desired output is.


<?xml version="1.0" encoding="UTF-8"?>
<envelope>
  <header mode="xml">
    <creationDate>2007-12-03T21:39:48.265-05:00</creationDate>
  </header>
  <body>
    <myns:books xmlns:myns="http://mycompany.com/mynames/" 
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
                xsi:schemaLocation="url2">
      <myns:book categoryCode="F">
        <myns:title>Factotum</myns:title>
        <myns:author>Charles Bukowski</myns:author>
        <myns:price>22.95</myns:price>
        <myns:isbn>0876852630</myns:isbn>
      </myns:book>
      <myns:book categoryCode="F">
        <myns:title>The Night Watch</myns:title>
        <myns:author>Sergei Lukyanenko</myns:author>
        <myns:price>17.99</myns:price>
        <myns:isbn>0434014125</myns:isbn>
      </myns:book>
      <myns:book categoryCode="F">
        <myns:title>Mr Mee</myns:title>
        <myns:author>Andrew Crumey</myns:author>
        <myns:price>22.00</myns:price>
        <myns:isbn>0312282354</myns:isbn>
      </myns:book>
    </myns:books>
  </body>
  <trailer>This is a trailer element</trailer>
</envelope>

The resources script below does the transformation.

Figure 195. Resources script


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:myns="http://mycompany.com/mynames/">

  <sx:service id="wrap">
    <sx:wrap>
      <sx:xsltSerializer/>
      <envelope>
        <header>
          <sx:attribute name="mode" value="xml"/>
          <creationDate>
            <sx:currentDateTime/>
          </creationDate>
        </header>
        <body>
          <myns:books xmlns:myns="http://mycompany.com/mynames/"
                      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                      xsi:schemaLocation="url2">
            <sx:transform>
              <sx:document/>
              <sx:processSubtree path="/myns:books/myns:book">
                <sx:choose>
                  <sx:when test="@categoryCode='F'">
                    <sx:transform/>
                  </sx:when>
                </sx:choose>
              </sx:processSubtree>
            </sx:transform>
          </myns:books>
        </body>
        <trailer>This is a trailer element</trailer>
      </envelope>
    </sx:wrap>
  </sx:service>

</sx:resources>

You can run this example on the command line by entering


servingxml -r resources-wrap.xml -i documents/books.xml -o output/wrap.xml wrap

Unwrapping subtrees in an XML document

Continuing from the previous example, the input file is the same, but you want to produce multiple files like the one shown below, with filenames differentiated by the isbn number.

Figure 196. Book XML output file book-0876852630.xml.


<?xml version="1.0" encoding="UTF-8"?>
<myns:book xmlns:myns="http://mycompany.com/mynames/" 
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
           categoryCode="F">
  <myns:title>Factotum</myns:title>
  <myns:author>Charles Bukowski</myns:author>
  <myns:price>22.95</myns:price>
  <myns:isbn>0876852630</myns:isbn>
</myns:book>


The resources script below does the transformation.

Figure 197. Resources script


<sx:resources xmlns:sx="http://www.servingxml.com/core"
                     xmlns:myns="http://mycompany.com/mynames/" >

  <sx:service id="unwrap">
    <sx:transform>
      <sx:document/>
      <sx:processSubtree path="/myns:books/myns:book">
        <sx:choose>
          <sx:when test="@categoryCode='F'">
            <sx:parameter name="isbn" select="myns:isbn"/>
            <sx:transform>
              <sx:xsltSerializer>
                <sx:fileSink file="output/book-{$isbn}.xml"/>
              </sx:xsltSerializer>
            </sx:transform>
          </sx:when>
        </sx:choose>
      </sx:processSubtree>
    </sx:transform>
  </sx:service>

</sx:resources>

You can run this example on the command line by entering


servingxml -r resources-unwrap.xml -i documents/books.xml unwrap

Streaming a file through a pipeline

This example shows an implementation of the SAX Pipeline example in Michael Kay's XSLT 2nd Edition Programmer's Reference, Appendix F, Example 5.

Figure 198. Resources script


<sx:resources xmlns:sx="http://www.servingxml.com/core">
  
  <sx:service id="pipeline">
    <sx:serialize>
      <sx:transform>
        <sx:saxFilter class="PreFilter"/>
        <sx:xslt><sx:urlSource url="styles/filter.xsl"/></sx:xslt>
        <sx:saxFilter class="PostFilter"/>     
      </sx:transform>
    </sx:serialize>
  </sx:service>

</sx:resources>

PreFilter and PostFilter are the names of Java classes that implement the org.xml.sax.XMLFilter interface. The listings below show the Java code for the PreFilter and PostFilter classes (from Kay's book.)

Figure 199. PreFilter class


public class PreFilter extends XMLFilterImpl {

  public void startElement (String uri, String localName, String qName,
  Attributes atts) throws SAXException {
        
    String newLocalName = localName.toLowerCase();
    String newQName = qName.toUpperCase();
    AttributesImpl newAtts =
      (atts.getLength()>0 ?
       new AttributesImpl(atts) :
       new AttributesImpl());
    newAtts.addAttribute("", "old-local-name", "old-local-name", "CDATA", localName);
    newAtts.addAttribute("", "old-qname", "old-qname", "CDATA", qName);
    super.startElement(uri, newLocalName, newQName, newAtts);
  }

  public void endElement (String uri, String localName, String qName)
  throws SAXException {
    String newLocalName = localName.toLowerCase();
    String newQName = qName.toUpperCase();
    super.endElement(uri, newLocalName, newQName);
  }
}

Figure 200. PostFilter class


public class PostFilter extends XMLFilterImpl {

  public Stack stack;

  public void startDocument() throws SAXException {
    stack = new Stack();
    super.startDocument();
  }

  public void startElement (String uri, String localName, String qName,
                            Attributes atts)
  throws SAXException {

    String originalLocalName = localName;
    String originalQName = qName;
    AttributesImpl newAtts = new AttributesImpl();
    for (int i=0; i<atts.getLength(); i++) {
      String name = atts.getQName(i);
      String val = atts.getValue(i);
      if (name.equals("old-local-name")) {
        originalLocalName = val;
      } else if (name.equals("old-qname")) {
        originalQName = val;
      } else {
        newAtts.addAttribute(atts.getURI(i),
          atts.getLocalName(i),
          name,
          atts.getType(i),
          val);
      }
    }
    super.startElement(uri, originalLocalName, originalQName, newAtts);
    stack.push(originalLocalName);
    stack.push(originalQName);
  }

  public void endElement (String uri, String localName, String qName)
  throws SAXException {
    String originalQName = (String)stack.pop();
    String originalLocalName = (String)stack.pop();
    super.endElement(uri, originalLocalName, originalQName);
  }
}

You can run this example on the command line by

  • Running the sample build script at the command line, to compile the pre and post Java XMLFilter classes and copy them into the dir/classes directory.
  • Entering the following command to produce the XML output file.
    
    servingxml -r resources-file.xml -i documents/mixed-up.xml 
        -o output/original-file.html pipeline
    
    

Streaming dynamically generated SAX events through a pipeline

This example produces the same output as the previous, but this time, instead of reading an XML document from a file, we generate the SAX events dynamically in a custom SAX reader.

Figure 201. Resources script


<sx:resources xmlns:sx="http://www.servingxml.com/core">
  
  <sx:service id="pipeline">
    <sx:serialize>
      <sx:transform>
        <sx:document>
          <sx:saxReader class="MySaxReader"/>
        </sx:document>
        <sx:saxFilter class="PreFilter"/>
        <sx:xslt><sx:urlSource url="styles/filter.xsl"/></sx:xslt>
        <sx:saxFilter class="PostFilter"/>     
      </sx:transform>
    </sx:serialize>
  </sx:service>

</sx:resources>

The Java class MySaxReader looks like this.


public class MySaxReader implements XMLReader ...                                                             

  private static final AttributesImpl NO_ATTRIBUTES = new AttributesImpl();

  public void parse(String systemId)
  throws IOException, SAXException {
    if (contentHandler != null) {
      contentHandler.startDocument();
      contentHandler.startElement("","HTML","HTML",NO_ATTRIBUTES);
      contentHandler.startElement("","Head","Head",NO_ATTRIBUTES);
      contentHandler.startElement("","TITLE","TITLE",NO_ATTRIBUTES);
      char[] title = "This is a mixed-case HTML-like XML document".toCharArray();
      contentHandler.characters(title,0,title.length);
      contentHandler.endElement("","TITLE","TITLE");
      contentHandler.endElement("","Head","Head");
      contentHandler.startElement("","Body","Body");
      
      ...
      
      contentHandler.endElement("","Body","Body");
      contentHandler.endElement("","HTML","HTML");
      contentHandler.endDocument();
    }  
  }



You can run this example on the command line by

  • Run the sample build script at the command line, to compile the pre and post Java XMLFilter classes and copy them into the dir/classes directory.
  • Entering the following command to produce the XML output file.
    
    servingxml -r resources-sax.xml -o output/original-sax.html pipeline
    
    

Processing document subtrees and serializing them to separate files (invoices.)

This example shows how to serialize individual document subtrees to separate html and pdf files.

The input file is an XML document containing a number of invoices.

Figure 202. Input XML file invoices.xml


<?xml version="1.0" encoding="iso-8859-1"?>
<inv:invoices xmlns:inv="http://www.telio.be/ns/2002/invoice">
  <inv:invoice currency="EUR" id="200302-01" paid="false">
  	<inv:client id="test">
      <inv:email>john@abc.com</inv:email>
    </inv:client>
  	<inv:date>2003-02-03</inv:date>
  	<inv:description>Test description</inv:description>
  	<inv:item>
  	<inv:quantity>67.5</inv:quantity>
  	<inv:unit>h</inv:unit>
  	<inv:description>January 2004</inv:description>
  	<inv:unitprice>400</inv:unitprice>
  	<inv:tax>21.00</inv:tax>
  	</inv:item>
  </inv:invoice>
  <inv:invoice currency="EUR" id="200302-02" paid="true">
    <inv:client id="test">
      <inv:email>jane@efg.com</inv:email>
    </inv:client>
    <inv:date>2003-02-03</inv:date>
    <inv:description>Test description</inv:description>
    <inv:item>
    <inv:quantity>67.5</inv:quantity>
    <inv:unit>h</inv:unit>
    <inv:description>January 2004</inv:description>
    <inv:unitprice>400</inv:unitprice>
    <inv:tax>21.00</inv:tax>
    </inv:item>
  </inv:invoice>
</inv:invoices>


The desired output is a set of HTML and PDF invoice documents, one pair for each inv:invoice element in the master XML document.

Figure 203. Directory listing of output files


invoice200302-01.html
invoice200302-01.pdf
invoice200302-02.html
invoice200302-02.pdf


The following resources script does the transformation.

Figure 204. Resources script


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:fop="http://www.servingxml.com/extensions/fop"
              xmlns:inv="http://www.telio.be/ns/2002/invoice">

  <sx:service id="invoices">
    <sx:transform>
      <sx:document/>
      <!-- Here we extract a document subtree from the SAX stream -->
      <sx:processSubtree path="/inv:invoices/inv:invoice">
        <!-- Main pipeline - invoice document subtree to pdf-->
        <sx:transform>
          <!-- We initialize a parameter with an XPATH expression
               applied to the document subtree -->
          <sx:parameter name="invoice-name" select="@id"/>
          <fop:foSerializer>
            <sx:fileSink file="output/invoice{$invoice-name}.pdf"/>
          </fop:foSerializer>
          <sx:transform>
            <sx:transform ref="steps1-4"/>
            <!-- Tee - invoice document subtree to html-->
            <sx:tagTee>
              <sx:xslt documentBase="documents/">
                <sx:urlSource url="styles/invoice2html.xsl"/>
              </sx:xslt>
              <sx:xsltSerializer>
                <sx:fileSink file="output/invoice{$invoice-name}.html"/>
              </sx:xsltSerializer>
            </sx:tagTee>
            <sx:xslt documentBase="documents/">
              <sx:urlSource url="styles/invoice2fo.xsl"/>
            </sx:xslt>
          </sx:transform>
        </sx:transform>
      </sx:processSubtree>
    </sx:transform>
  </sx:service>

  <sx:transform id="steps1-4">
    <sx:xslt><sx:urlSource url="styles/step1.xsl"/></sx:xslt>
    <sx:xslt><sx:urlSource url="styles/step2.xsl"/></sx:xslt>
    <sx:xslt><sx:urlSource url="styles/step3.xsl"/></sx:xslt>
    <sx:xslt documentBase="documents/">
      <sx:urlSource url="styles/step4.xsl"/>
    </sx:xslt>
  </sx:transform>

</sx:resources>

Not the sx:tagTee element, which forks the SAX event stream, and serializes this stream as HTML.

You can run this example on the command line by entering


servingxml -r resources-invoices.xml -i documents/invoices.xml invoices

Processing document subtrees and serializing them as mail attachments ("mail invoices")

Continuing with the previous example, suppose you want to send the invoices as pdf mail attachments. Then you need the following resources script.

Figure 205. Resources script


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:fop="http://www.servingxml.com/extensions/fop"
              xmlns:inv="http://www.telio.be/ns/2002/invoice"
              xmlns:jm="http://www.servingxml.com/extensions/javamail">

  <sx:service id="invoices">
    <sx:transform>
      <sx:document/>
      <!-- Here we extract a document subtree from the SAX stream -->
      <sx:processSubtree path="/inv:invoices/inv:invoice">                        
        <sx:parameter name="email" select="inv:client/inv:email"/>

        <!--  We want to send an email with the PDF invoice as an attachment   -->
        <jm:sendMail subject="Invoice" to="{$email}">
          <sx:parameter name="invoice-name" select="@id"/>

          <jm:mailAccount ref="myMailAccount"/>
          <jm:message>
            <jm:attachment filename="{$invoice-name}.pdf">
              <sx:serialize>
                <!-- We initialize a parameter with an XPATH expression
                     applied to the document subtree -->
                <fop:foSerializer/>
                <sx:transform>
                  <sx:transform ref="steps1-4"/>
                  <sx:xslt documentBase="documents/">
                    <sx:urlSource url="styles/invoice2fo.xsl"/>
                  </sx:xslt>
                </sx:transform>
              </sx:serialize>
            </jm:attachment>
          </jm:message>
          <sx:onError>
            <!-- If send mail fails (e.g. no smtp host), just proceed with the next action -->
            <!-- (the error will still be logged) -->
          </sx:onError>
        </jm:sendMail>
      </sx:processSubtree>
    </sx:transform>
  </sx:service>

  <sx:transform id="steps1-4">
    <sx:xslt>
      <sx:urlSource url="styles/step1.xsl"/>
    </sx:xslt>
    <sx:xslt>
      <sx:urlSource url="styles/step2.xsl"/>
    </sx:xslt>
    <sx:xslt>
      <sx:urlSource url="styles/step3.xsl"/>
    </sx:xslt>
    <sx:xslt documentBase="documents/">
      <sx:urlSource url="styles/step4.xsl"/>
    </sx:xslt>
  </sx:transform>

  <!-- Set up this mail account, substituting your smtp host, display name, and email address -->
  <!-- You will also need to enter the 'to' and 'cc' recipient addresses in the  jm:sendMail element -->
  <jm:mailAccount id="myMailAccount"
                  smtpHost="smtp1.abc.ca">
    <jm:sender displayName="Daniel"
                 emailAddress="danielaparker@abc.ca"/>
  </jm:mailAccount>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-mailInvoices.xml -i documents/invoices.xml invoices

Reading dynamic content

This example shows an implementation of dynamic content handlers.

Running this example requires the following steps:

  • Run the sample build script at the command line, to compile the DynamicContentHandler classes and copy them into the distribution classes directory.

  • Enter the following command to produce a page of books in the fiction (F) category,

    
    servingxml -r resources.xml -o output/books.html books category=F
    
    

Figure 206. Resources script


<sx:resources>

  <sx:service id="swath">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="swath"/>
        <sx:xslt ref="swath"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>
  
  <sx:xslt id="swath">
    <sx:urlSource url="styles/books.xsl"/>
    <sx:parameter name="category"/>
  </sx:xslt>
  
  <!-- BookCatalog is the name of a Java class used to produce dynamic content for the primary document -->
  <sx:dynamicContent name="swath" class="dynamiccontent.BookCatalog"/>
  
  <!-- Categories is the name of a Java class used to produce dynamic content in a 
        document function in the stylesheet -->
  <sx:dynamicContent name="categories" class="dynamiccontent.Categories"/>

</sx:resources>

BookCatalog and Categories are the names of Java classes that implement the com.servingxml.components.content.dynamic.DynamicContentHandler interface. The listing below shows the Java code for the Categories class.

Figure 207. Categories


package dynamiccontent;

public class Categories implements DynamicContentHandler {

  static class Category {
    private final String code;
    private final String description;

    public Category(String code, String description) {
      this.code = code;
      this.description = description;
    }

    public String getCode() {
      return code;
    }

    public String getDescription() {
      return description;
    }
  }

  private final HashMap categoryMap;
  private final String namespace = "";

  public Categories() {
    categoryMap = new HashMap();
    categoryMap.put("S", new Category("S","Science"));
    categoryMap.put("C", new Category("C","Computing"));
    categoryMap.put("CHILD", new Category("CHILD","Children"));
    categoryMap.put("F", new Category("F","Fiction"));
    categoryMap.put("P", new Category("P","Philospohy"));
    categoryMap.put("R", new Category("R","Religion"));
  }
  
  //  The onRequest method must be reentrant
  public void onRequest(ServiceContext context, Book parameters, 
  ContentWriter contentWriter)  {

    contentWriter.startElement(namespace,"categories");             
      
    String id = parameters.getCategory();

    if (id != null && id.length() != 0) {
      if (id.equals("all")) {
        Collection values = categoryMap.values();
        Iterator iter = values.iterator();
        while (iter.hasNext()) {
          Category category = (Category)iter.next();
          writeCategory(category,contentWriter);
        }
      } else {
        Category category = (Category)categoryMap.get(id);
        writeCategory(category,contentWriter);
      }
    }

    contentWriter.endElement(namespace,"categories");
  }

  private void writeCategory(Category category, ContentWriter contentWriter) 
   {
    AttributeSet attributes = contentWriter.newAttributeSet();
    attributes.addAttribute(namespace,"code",category.getCode());
    attributes.addAttribute(namespace,"desc",category.getDescription());
    contentWriter.startEndElement(namespace,"category",attributes); 
  }
}

You can run this example on the command line by

  • Running the sample build script at the command line, to compile the DynamicContentHandler classes and copy them into the distribution classes directory.
  • Entering the following command to produce a page of swath in the fiction (F) category,
    
    servingxml -r resources.xml -o output/books.html swath category=F
    
    

XML to Mail

The example below illustrates how to prepare a resources script that will transform an XML document to XSL-FO, format it to PDF, and send the result as an attachment in a mail message.

This example requires the mail module.

Once you have everything configured, rebuild the ServingXML jar file in the servingxml directory.

Figure 208. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:fop="http://www.servingxml.com/extensions/fop"
              xmlns:jm="http://www.servingxml.com/extensions/javamail">
  
  <sx:service id="pulp">                         
      <!--  Our first action is to send an email with the pulp novel as an attachment   -->
      <jm:sendMail subject="test" to="john@yyy.com" cc="linda@zzz.com">
        <jm:mailAccount ref="myMailAccount"/>
        <jm:message>

          <!-- Put some HTML in the message body --> 
          <jm:part type="alternative">
            <sx:transform>           
              <sx:document><xxx/></sx:document>
              <sx:xslt ref="welcome/html"/>
            </sx:transform>        
          </jm:part>
          
          <jm:attachment filename="pulp.pdf">
            <fop:foSerializer/>
            <sx:transform>
               <sx:content ref="pulp"/>
               <sx:xslt ref="novel/fo"/>
            </sx:transform>
          </jm:attachment>      
        </jm:message>
        <sx:onError>
            <!-- If send mail fails (e.g. no smtp host), just proceed with the next action -->
            <!-- (the error will still be logged) -->
        </sx:onError>
      </jm:sendMail> 
                                        
      <!--  Our second action is to write the pulp novel to a file   -->
      <sx:serialize>
        <fop:foSerializer/>
        <sx:transform>
           <sx:content ref="pulp"/>
           <sx:xslt ref="novel/fo"/>
        </sx:transform>
      </sx:serialize>
    
  </sx:service>                                                                                                             

  <!-- Set up this mail account, substituting your smtp host, display name, and email address -->
  <!-- You will also need to enter the 'to' and 'cc' recipient addresses in the  jm:sendMail element -->
  <jm:mailAccount id="myMailAccount" 
                           smtpHost="smtp1.abc.ca"> 
    <jm:sender displayName="DP" 
                     emailAddress="danielaparker@abc.ca"/>
  </jm:mailAccount>

  <sx:document id="pulp">        
    <sx:urlSource url="data/pulp.xml"/>
  </sx:document>

  <sx:xslt id="novel/fo">
    <sx:urlSource url="data/novel.xsl"/>
  </sx:xslt>

  <sx:xslt id="welcome/html">
    <sx:urlSource url="data/welcome.xsl"/>
  </sx:xslt>
  
</sx:resources>

This example produces two outputs:

  • A mail message with an HTML body part displaying "Welcome to Presenting Books" and a PDF attachment.

  • A PDF file pulp.pdf written to the output directory.

You can run this example on the command line by entering


servingxml -r resources.xml -o output/pulp.pdf pulp

Writing an XML file to an FTP sink

This example shows how to send stream output to an FTP host.

Figure 209. Resources script


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:edt="http://www.servingxml.com/extensions/edtftp">
  
  <edt:ftpClient id="myFtpSite" host="myFtpHost" 
                 user="anonymous" password="xxx"/>
  
  <sx:service id="books">
    <sx:serialize>
      <sx:xsltSerializer>
        <edt:ftpSink remoteDirectory="incoming" remoteFile="books{$version}.xml">
          <edt:ftpClient ref="myFtpSite"/>
        </edt:ftpSink>
      </sx:xsltSerializer/>
        
      <sx:transform>
        <sx:content ref="books"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:document id="books">
    <sx:urlSource url="documents/books.xml"/>
  </sx:document>
  
</sx:resources>

You can run this example on the command line by

  • Updating the resources.xml file with a valid FTP host

  • Entering the following command to send an XML file to the FTP host,

    
    servingxml -r resources.xml books version=1.0
    
    

Comparing Two XML Documents with fn:deep-equal

The example below shows how to compare two XML documents for equality with the XPATH fn:deep-equal function.


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:fn="http://www.w3.org/2005/xpath-functions">

  <sx:service id="compare-documents">
    <sx:task ref="test-documents-are-equal"/>
  </sx:service>

  <sx:document id="odd-element-document">
    <document>
      <title>Title</title>
      <para>First paragraph.</para>
    </document>
  </sx:document>

  <sx:document id="another-odd-element-document">
    <document>
      <title>Title</title>
      <para>First paragraph.</para>
    </document>
  </sx:document>

  <sx:serialize id="test-documents-are-equal">
    <sx:transform>
      <sx:choose>
        <sx:when test="fn:deep-equal(fn:document('odd-element-document'),
                     fn:document('another-odd-element-document'))">
          <sx:document>
            <result>equal</result>
          </sx:document>
        </sx:when>
        <sx:otherwise>
          <sx:document>
            <result>unequal</result>
          </sx:document>
        </sx:otherwise>
      </sx:choose>
    </sx:transform>
  </sx:serialize>

</sx:resources>

        

The documents are equal, so the script produces the output <result>equal</result>

You can run this example on the command line by entering


servingxml -r resources-compare.xml -o output/compare-documents.xml 
  compare-documents

SQL Examples

The java run time needs to be able to find the database driver. The examples in this section are set up for the Oracle driver, oracle.jdbc.driver.OracleDriver. To configure ServingXML to recognize this driver, copy the Oracle classes12.jar file (not distributed) to the lib/local directory, and then rebuild. Of course, you can specify a different driver, making the appropriate changes in the sample resources script and copying the right jar file to the lib/local directory.

SQL Query to Flat File

Writing the results of a SQL query to a CSV file (employees-csv)

This example shows how to prepare a resources script that will write the results of a SQL query to a CSV file, using a default flat file writer.

The desired CSV file is shown below.

Figure 210. Output XML file employees.csv


EMPNO,NAME,JOB
7902,FORD,ANALYST
7788,SCOTT,ANALYST
7876,ADAMS,CLERK
7900,JAMES,CLERK
7934,MILLER,CLERK
7369,SMITH,CLERK

The following resources script does the transformation.

Figure 211. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="employees-to-csv">
    <sx:recordStream>
      <sx:recordReader ref="employees"/>
      <sx:flatFileWriter/>
    </sx:recordStream>
  </sx:service>

  <sx:sqlReader id="employees">
    <sx:sqlConnectionPool ref="jdbcPool"/>

    <sx:parameter name="jobList">
      <sx:toString value="{$job}" separator=",">
        <sx:quoteSymbol character="'" escapeCharacter="'"/>
      </sx:toString>
    </sx:parameter>

    <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
      <sx:choose>
        <sx:when test="$jobList">
            AND JOB IN ({$jobList})
        </sx:when>
      </sx:choose>
        ORDER BY JOB,ENAME
    </sx:sqlQuery>

  </sx:sqlReader>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="{$user}"
                        password="{$password}"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Executing the employees service with two job parameters, like this


    servingxml  -r resources-employees_csv.xml -o output/employees.csv employees-to-csv
         user=scott password=spring job=ANALYST job=CLERK

results in the following SQL statement being sent to the Oracle server:


SELECT EMPNO, ENAME AS NAME,JOB 
  FROM EMP 
  WHERE 1=1 AND JOB IN ('ANALYST','CLERK') 
  ORDER BY JOB,ENAME

The attribute names EMPNO, NAME and JOB become the headings of the CSV file, and the values in each row are written out below.

SQL Query to XML

Mapping the results of a parameterized SQL query into XML (analysts)

The example below illustrates how to prepare a resources script that will convert the results of a SQL query into an XML document.

The desired output is shown below.

Figure 212. Output XML file employees.xml


<?xml version="1.0" encoding="utf-8"?>
<employees>
  <employee employee-no="7788">
    <name>SCOTT</name>
    <job>ANALYST</job>
  </employee>
  <employee employee-no="7902">
    <name>FORD</name>
    <job>ANALYST</job>
  </employee>
</employees>


The following resources script does the transformation.

Figure 213. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core">                   
                                                                   
  <sx:service id="employees">                         
    <sx:serialize>
      <sx:content ref="employees"/>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="employees">
    <sx:sqlReader>
      <sx:sqlConnectionPool ref="jdbcPool"/>
      
      <sx:parameter name="jobList">
        <sx:toString value="{$job}" separator=",">
          <sx:quoteSymbol character="'" escapeCharacter="'"/>
        </sx:toString>
      </sx:parameter>
      
      <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE JOB = '{$job}' 
      </sx:sqlQuery>
        
    </sx:sqlReader>
    
    <sx:recordMapping ref="employeesToXml"/>
    
  </sx:recordContent>
  
  <sx:recordMapping id="employeesToXml">
    <employees>
      <sx:groupBy fields="JOB">
        <sx:elementMap element="{JOB}">
          <sx:onRecord>
            <employee>
              <sx:fieldAttributeMap field="EMPNO" attribute="employee-no"/>
              <sx:fieldElementMap field="NAME" element="name"/>
            </employee>
          </sx:onRecord>
        </sx:elementMap>
      </sx:groupBy>
    </employees>
  </sx:recordMapping>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="{$user}"
                        password="{$password}"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>
                         
  </sx:resources>

You can run this example on the command line by entering


  servingxml -r resources.xml employees -o output/employees.txt employees 
       user=scott password=spring job=ANALYST

Mapping the results of an ad hoc SQL query into XML (employees)

The example below illustrates how to prepare a resources script that will generate and execute a different SQL statement depending on passed parameters.

The desired output is shown below.

Figure 214. Output XML file employees.xml


<?xml version="1.0" encoding="utf-8"?>
<employees>
   <ANALYST>
      <employee employee-no="7902">
         <name>FORD</name>
      </employee>
      <employee employee-no="7788">
         <name>SCOTT</name>
      </employee>
   </ANALYST>
   <CLERK>
      <employee employee-no="7876">
         <name>ADAMS</name>
      </employee>
      <employee employee-no="7900">
         <name>JAMES</name>
      </employee>
      <employee employee-no="7934">
         <name>MILLER</name>
      </employee>
      <employee employee-no="7369">
         <name>SMITH</name>
      </employee>
   </CLERK>
</employees>

The following resources script does the transformation.

Figure 215. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="employees">
    <sx:serialize>
      <sx:content ref="employeesDoc"/>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="employeesDoc">
    <sx:sqlReader>
      <sx:sqlConnectionPool ref="jdbcPool"/>
      
      <sx:parameter name="jobList">
        <sx:toString value="{$job}" separator=",">
          <sx:quoteSymbol character="'" escapeCharacter="'"/>
        </sx:toString>
      </sx:parameter>
      
      <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1 
        <sx:choose>
          <sx:when test="$jobList">
            AND JOB IN ({$jobList})
          </sx:when>
        </sx:choose>
        ORDER BY JOB,ENAME
      </sx:sqlQuery>
        
    </sx:sqlReader>
    
    <sx:recordMapping ref="employeesToXml"/>
    
  </sx:recordContent>
  
  <sx:recordMapping id="employeesToXml">
    <employees>
      <sx:groupBy fields="JOB">
        <sx:elementMap element="{JOB}">
          <sx:onRecord>
            <employee>
              <sx:fieldAttributeMap field="EMPNO" attribute="employee-no"/>
              <sx:fieldElementMap field="NAME" element="name"/>
            </employee>
          </sx:onRecord>
        </sx:elementMap>
      </sx:groupBy>
    </employees>
  </sx:recordMapping>

  <sx:sqlConnectionPool id="jdbcPool"
                         driver="oracle.jdbc.driver.OracleDriver"
                         databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                         user="scott"
                         password="spring"
                         minConnections="2"
                         testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Executing the employees service with two job parameters, like this


    servingxml -r resources.xml employees -o output/employees.xml employees 
        job=ANALYST job=CLERK

results in the following SQL statement being sent to the Oracle server:


SELECT EMPNO, ENAME AS NAME,JOB 
  FROM EMP 
  WHERE 1=1 AND JOB IN ('ANALYST','CLERK') 
  ORDER BY JOB,ENAME

But executing the employees service with no parameters results in a SQL select statement that returns all employees for all job categories.

Using multiple readers to join data across different data sources (chained readers)

The example below illustrates how to prepare a resources script that will execute two queries to two different databases, where the second is joined with the first.

The desired output is shown below.

Figure 216. Output XML file deptEmployees.xml


<?xml version="1.0" encoding="utf-8"?>
<employees>
  <department dept-no="10" dept-name="ACCOUNTING">
    <employee employee-no="7782">
      <name>CLARK</name>
    </employee>
    <employee employee-no="7839">
      <name>KING</name>
    </employee>
  </department>
  <department dept-no="30" dept-name="SALES">
    <employee employee-no="7499">
      <name>ALLEN</name>
    </employee>
    <employee employee-no="7698">
      <name>BLAKE</name>
    </employee>
  </department>
</employees>

The following resources script does the transformation.

Figure 217. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="employees">
    <sx:serialize>
      <sx:content ref="employeesDoc"/>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="employeesDoc">
    <sx:recordStream>
      <sx:sqlReader>
        <sx:sqlConnectionPool ref="devPool"/>
        <sx:sqlQuery recordType = "employee">
        SELECT DEPTNO, DNAME FROM DEPT ORDER BY DNAME
        </sx:sqlQuery>
      </sx:sqlReader>
                                                                                            
      <!-- for each row from the previous reader, run this reader -->
      <sx:sqlReader>
        <sx:sqlConnectionPool ref="dev2Pool"/>
        <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME,JOB, {DEPTNO} AS DEPTNO, '{DNAME}' AS DEPT FROM EMP  WHERE DEPTNO = {DEPTNO}
        ORDER BY ENAME
        </sx:sqlQuery>
      </sx:sqlReader>
    </sx:recordStream>

    <sx:recordMapping ref="employeesToXml"/>

  </sx:recordContent>

  <sx:recordMapping id="employeesToXml">
    <employees>
      <sx:groupBy fields="DEPT">
        <department>
          <sx:fieldAttributeMap field="DEPTNO" attribute="dept-no"/>
          <sx:fieldAttributeMap field="DEPT" attribute="dept-name"/>
          <sx:onRecord>
            <employee>
              <sx:fieldAttributeMap field="EMPNO" attribute="employee-no"/>
              <sx:fieldElementMap field="NAME" element="name"/>
            </employee>
          </sx:onRecord>
        </department>
      </sx:groupBy>
    </employees>
  </sx:recordMapping>

  <sx:sqlConnectionPool id="devPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

  <sx:sqlConnectionPool id="dev2Pool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev2"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Executing the employees service, like this


    servingxml -r resources-multipleReaders.xml employees -o output/deptEmployees.xml employees 

results in the following two SQL statements being sent to the Oracle "dev2" database by the second reader:


SELECT EMPNO, ENAME AS NAME, 10 AS DEPTNO, 'ACCOUNTING' AS DEPT 
  FROM EMP WHERE DEPTNO = 10
  ORDER ENAME

SELECT EMPNO, ENAME AS NAME, 30 AS DEPTNO, 'SALES' AS DEPT 
  FROM EMP WHERE DEPTNO = 30
  ORDER ENAME

XML to Database

Load Master Detail

This example shows how to prepare a resources script that will insert master/detail records into separate master and detail tables as a single transaction.

The input file is an XML file.

Figure 218. Input XML file masterDetail.xml


<?xml version="1.0" encoding="utf-8"?>
<masterDetail>
  <master id="1">
    <name>aName</name>
    <detail id="11">
      <date>2008-09-03</date>
      <type>aType</type>
    </detail>
    <detail id="12">
      <date>2008-09-06</date>
      <type>aType</type>
    </detail>
    <detail id="13">
      <date>2008-09-10</date>
      <type>aType</type>
    </detail>
  </master>
  <master id="2">
    <name>anotherName</name>
    <detail id="21">
      <date>2008-10-03</date>
      <type>aType</type>
    </detail>
    <detail id="22">
      <date>2008-10-06</date>
      <type>aType</type>
    </detail>
    <detail id="23">
      <date>2008-10-07</date>
      <type>anotherType</type>
    </detail>
  </master>
</masterDetail>


The following resources script performs the update.

Figure 219. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="loadMasterDetail">
    <sx:recordStream>
      <sx:subtreeRecordReader>
        <sx:transform>
          <sx:document/>
        </sx:transform>
        <sx:inverseRecordMapping ref="masterDetailMapping"/>
      </sx:subtreeRecordReader>
      <!-- You should validate the record with an sx:recordValidator here -->
      <sx:sqlWriter ref="masterDetailWriter"/>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <!-- You can set up a record pipeline here to write bad records to a file or database table -->
      </sx:discardHandler>
    </sx:recordStream>
  </sx:service>

  <sx:inverseRecordMapping id="masterDetailMapping">
    <sx:onSubtree path="/masterDetail/master">
      <sx:flattenSubtree recordType="master">
        <sx:subtreeFieldMap select="@id" field="masterId"/>
        <sx:subtreeFieldMap select="name" field="name"/>
        <sx:subtreeFieldMap match="detail" field="details">
          <sx:flattenSubtree recordType="detail">
            <sx:subtreeFieldMap select="@id" field="detailId"/>
            <sx:subtreeFieldMap select="date" field="date"/>
            <sx:subtreeFieldMap select="type" field="type"/>
          </sx:flattenSubtree>
        </sx:subtreeFieldMap>
      </sx:flattenSubtree>
    </sx:onSubtree>
  </sx:inverseRecordMapping>

  <sx:sqlWriter id="masterDetailWriter">                          
    <sx:sqlConnectionPool ref="jdbcPool"/>
    <sx:sqlUpdate>
        INSERT INTO master(master_id,name) VALUES({masterId},'{name}')
      <sx:sqlUpdateDetail field="details">
        <sx:sqlUpdate>
          INSERT INTO detail(detail_id,detail_date,detail_type) 
          VALUES({detailId},to_date('{date}','yyyy-mm-dd'),'{type}')
        </sx:sqlUpdate>
      </sx:sqlUpdateDetail>
    </sx:sqlUpdate>
  </sx:sqlWriter>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

The inverse record mapping instructions in the script will produce a stream of composite records of type "master" that contain a repeating group field named "details" having subrecords of type "detail".

Executing the loadMasterDetail service, like this,


servingxml -i data/masterDetail.xml -r resources-masterDetail.xml loadMasterDetail

results in the following update statements being sent to the RDMS and committed to the database:


  INSERT INTO master(master_id,name) VALUES(1,'aName')
  INSERT INTO detail(detail_id,detail_date,detail_type) 
  VALUES(11,to_date('2008-09-03','yyyy-mm-dd'),'aType')
  INSERT INTO detail(detail_id,detail_date,detail_type) 
  VALUES(12,to_date('2008-09-06','yyyy-mm-dd'),'aType')
  INSERT INTO detail(detail_id,detail_date,detail_type) 
  VALUES(13,to_date('2008-09-10','yyyy-mm-dd'),'aType')

The second set of master/detail records fails, however, because the last detail has a value for detail_type, "anotherType", that exceeds the length of the column, 10. The following error message is written to the log:


SEVERE: ORA-12899: value too large for column "SCOTT"."DETAIL"."DETAIL_TYPE" 
(actual: 11, maximum: 10) 

SQL Query to Database

SQL inserts

This example shows how to prepare a resources script that will read rows from one database table and write them to another.

The following resources script does the transformation.

Figure 220. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core"
               xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="loadEmployees">
    <sx:recordStream>
      <sx:sqlReader ref="employeesReader"/>
      <msv:recordValidator>
        <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
          <!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
          <xsd:element name="employee" type="EmployeeType"/>
          <xsd:complexType name="EmployeeType">
            <xsd:sequence>
              <xsd:element name="EMPNO" type="xsd:integer"/>
              <xsd:element name="NAME" type="xsd:string"/>
              <xsd:element name="JOB" type="xsd:string"/>
            </xsd:sequence>
          </xsd:complexType>
        </xsd:schema>
      </msv:recordValidator>

      <sx:sqlWriter ref="employeesWriter"/>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <!-- You can include a record pipeline here to write bad records to a file or database table -->
      </sx:discardHandler>
    </sx:recordStream>
  </sx:service>

  <sx:sqlWriter id="employeesWriter">                          
    <sx:sqlConnectionPool ref="jdbcPool"/>
    <sx:sqlUpdate>
        INSERT INTO EMP_HISTORY(EMPNO, ENAME) VALUES({EMPNO},'{NAME}')
    </sx:sqlUpdate>
  </sx:sqlWriter>

  <sx:sqlReader id="employeesReader">
    <sx:sqlConnectionPool ref="jdbcPool"/>

    <sx:parameter name="jobList">
      <sx:toString value="{$job}" separator=",">
        <sx:quoteSymbol character="'" escapeCharacter="'"/>
      </sx:toString>
    </sx:parameter>

    <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
      <sx:choose>
        <sx:when test="$jobList">
            AND JOB IN ({$jobList})
        </sx:when>
      </sx:choose>
        ORDER BY JOB,ENAME
    </sx:sqlQuery>

  </sx:sqlReader>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Executing the loadEmployees service with two job parameters, like this


    servingxml -r resources-insert_employees.xml loadEmployees 
        job=ANALYST job=CLERK

results in the following SQL statement being sent to the Oracle server:


SELECT EMPNO, ENAME AS NAME,JOB 
  FROM EMP 
  WHERE 1=1 AND JOB IN ('ANALYST','CLERK') 
  ORDER BY JOB,ENAME

These results are then written to the database table EMP_HISTORY.

Batched SQL inserts

This example shows how to prepare a resources script that will read rows from one database table and write them in batches to another.

The following resources script does the transformation.

Figure 221. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core"
               xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="loadEmployees">
    <sx:recordStream>
      <sx:sqlReader ref="employeesReader"/>
      <msv:recordValidator>
        <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
          <!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
          <xsd:element name="employee" type="EmployeeType"/>
          <xsd:complexType name="EmployeeType">
            <xsd:sequence>
              <xsd:element name="EMPNO" type="xsd:integer"/>
              <xsd:element name="NAME" type="xsd:string"/>
              <xsd:element name="JOB" type="xsd:string"/>              
            </xsd:sequence>
          </xsd:complexType>
        </xsd:schema>
      </msv:recordValidator>

      <sx:sqlWriter ref="employeesWriter"/>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <!-- You can include a record pipeline here to write bad records to a file or database table -->
      </sx:discardHandler>
    </sx:recordStream>
  </sx:service>

  <sx:sqlBatchWriter id="employeesWriter" batchSize="4">
    <sx:sqlConnectionPool ref="jdbcPool"/>
    <sx:sqlUpdate>
        INSERT INTO EMP_HISTORY(EMPNO, ENAME) VALUES( {EMPNO}, '{NAME}')
    </sx:sqlUpdate>
  </sx:sqlBatchWriter>

  <sx:sqlReader id="employeesReader">
    <sx:sqlConnectionPool ref="jdbcPool"/>

    <sx:parameter name="jobList">
      <sx:toString value="{$job}" separator=",">
        <sx:quoteSymbol character="'" escapeCharacter="'"/>
      </sx:toString>
    </sx:parameter>

    <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
      <sx:choose>
        <sx:when test="$jobList">
            AND JOB IN ({$jobList})
        </sx:when>
      </sx:choose>
        ORDER BY JOB,ENAME
    </sx:sqlQuery>

  </sx:sqlReader>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Executing the loadEmployees service with two job parameters, like this


    servingxml -r resources-batch_insert_employees.xml loadEmployees 
        job=ANALYST job=CLERK

results in the following SQL statement being sent to the Oracle server:


SELECT EMPNO, ENAME AS NAME,JOB 
  FROM EMP 
  WHERE 1=1 AND JOB IN ('ANALYST','CLERK') 
  ORDER BY JOB,ENAME

These results are then written in batches of 4 to the database table EMP_HISTORY.

Prepared SQL inserts

This example shows how to prepare a resources script that will read rows from one database table and write them using a prepared SQL statement to another.

The following resources script does the transformation.

Figure 222. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core"
               xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="loadEmployees">
    <sx:recordStream>
      <sx:sqlReader ref="employeesReader"/>

      <msv:recordValidator>
        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
          <!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
          <xs:element name="employee" type="EmployeeType"/>
          <xs:complexType name="EmployeeType">
            <xs:sequence>
              <xs:element name="EMPNO" type="xs:integer"/>
              <xs:element name="NAME" type="xs:string"/>
              <xs:element name="JOB" type="xs:string"/>
              <xs:element name="MGR" type="xs:integer"/>
              <xs:element name="HIREDATE" type="xs:date"/>
              <xs:element name="SAL" type="xs:decimal"/>
              <xs:element name="COMM" type="xs:decimal" minOccurs="0"/> 
              <xs:element name="DEPTNO" type="xs:integer"/>
            </xs:sequence>
          </xs:complexType>
        </xs:schema>
      </msv:recordValidator>

      <sx:sqlWriter ref="employeesWriter"/>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <!-- You can include a record pipeline here to write bad records to a database table -->
        <sx:sqlWriter ref="discardWriter"/> 
        <sx:discardHandler>
          <sx:log message="{$sx:message}"/>
        </sx:discardHandler>
      </sx:discardHandler>
    </sx:recordStream>
  </sx:service>

  <sx:sqlWriter id="employeesWriter" xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <sx:sqlConnectionPool ref="jdbcPool"/>
    <sx:sqlUpdate>                                                                                                                        
      <!-- Because the SQL insert executes a prepared statement, values must be
            associated with appropriate types.  In this example, because these values are 
            coming from a SQL query source, they will already be associated with SQL types, 
            and no further action is really necessary.  In the case of other data sources, however,
            values by default will be associated with string types, and may need
            to be recast in parameter assignments, as shown below.
      -->
      <sx:parameter name="employee-no" value="{EMPNO}" type="xs:long"/>
      <sx:parameter name="mgr" value="{MGR}" type="xs:long"/>
      <sx:parameter name="hiredate" value="{HIREDATE}" type="xs:date"/>              
      <sx:parameter name="salary" value="{SAL}" type="xs:decimal"/>              
      <sx:parameter name="commission" value="{COMM}" type="xs:decimal"/>              
      <sx:sqlPrepare>
        INSERT INTO EMP_HISTORY(EMPNO, ENAME, JOB, MGR, HIREDATE, SAL, COMM) 
        VALUES({$employee-no},{NAME},{JOB}, {$mgr}, {$hiredate},  {$salary}, {$commission})
      </sx:sqlPrepare>
    </sx:sqlUpdate>
  </sx:sqlWriter>                                                                              

  <sx:sqlWriter id="discardWriter" xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <sx:sqlConnectionPool ref="jdbcPool"/>
    <sx:sqlUpdate>
      <sx:parameter name="employee-no" value="{EMPNO}" type="xs:long"/>
      <sx:parameter name="mgr" value="{MGR}" type="xs:long"/>
      <sx:parameter name="hiredate" value="{HIREDATE}" type="xs:date"/>              
      <sx:parameter name="salary" value="{SAL}" type="xs:decimal"/>              
      <sx:parameter name="commission" value="{COMM}" type="xs:decimal"/>              
      <sx:sqlPrepare>
        INSERT INTO EMP_DISCARD(MESSAGE, EMPNO, ENAME, JOB, MGR, HIREDATE, SAL, 
                                COMM) 
        VALUES({$message}, {$employee-no}, {NAME}, {JOB}, {$mgr}, {$hiredate},
               {$salary}, {$commission})
      </sx:sqlPrepare>
    </sx:sqlUpdate>
  </sx:sqlWriter>                                                                              
                                                              
  <sx:sqlReader id="employeesReader">                         
    <sx:sqlConnectionPool ref="jdbcPool"/>

    <sx:parameter name="jobList">
      <sx:toString value="{$job}" separator=",">
        <sx:quoteSymbol character="'" escapeCharacter="'"/>
      </sx:toString>
    </sx:parameter>

    <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO FROM EMP WHERE 1=1
      <sx:choose>
        <sx:when test="$jobList">
            AND JOB IN ({$jobList})
        </sx:when>
      </sx:choose>
        ORDER BY JOB,ENAME
    </sx:sqlQuery>

  </sx:sqlReader>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Executing the loadEmployees service with two job parameters, like this


    servingxml -r resources-prepare_insert_employees.xml loadEmployees 
        job=SALESMAN job=MANAGER

results in the following SQL statement being sent to the Oracle server:


SELECT EMPNO, ENAME AS NAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO 
  FROM EMP 
  WHERE 1=1 AND JOB IN ('SALESMAN','MANAGER') 
  ORDER BY JOB,ENAME

These results are then written in batches of 4 to the database table EMP_HISTORY.

Batched prepared SQL inserts

This example shows how to prepare a resources script that will read rows from one database table and write them in batches, using a prepared SQL statement, to another.

The following resources script does the transformation.

Figure 223. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core"
               xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="loadEmployees">
    <sx:recordStream>
      <sx:sqlReader ref="employeesReader"/>
      <msv:recordValidator>
        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
          <!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
          <xs:element name="employee" type="EmployeeType"/>
          <xs:complexType name="EmployeeType">
            <xs:sequence>
              <xs:element name="EMPNO" type="xs:integer"/>
              <xs:element name="NAME" type="xs:string"/>
              <xs:element name="JOB" type="xs:string"/>
              <xs:element name="MGR" type="xs:integer"/>
              <xs:element name="HIREDATE" type="xs:date"/>
              <xs:element name="SAL" type="xs:decimal"/>
              <xs:element name="COMM" type="xs:decimal" minOccurs="0"/> 
              <xs:element name="DEPTNO" type="xs:integer"/>
            </xs:sequence>
          </xs:complexType>
        </xs:schema>
      </msv:recordValidator>

      <sx:sqlWriter ref="employeesWriter"/>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <!-- You can include a record pipeline here to write bad records to a file or database table -->
      </sx:discardHandler>
    </sx:recordStream>
  </sx:service>

  <sx:sqlBatchWriter id="employeesWriter" xmlns:xs="http://www.w3.org/2001/XMLSchema" batchSize="4">
    <sx:sqlConnectionPool ref="jdbcPool"/>
    <sx:sqlUpdate>
      <!-- Because the SQL insert executes a prepared statement, values must be
           associated with appropriate types.  In this example, because these values are 
           coming from a SQL query source, they will already be associated with SQL types, 
           and no further action is really necessary.  In the case of other data sources, however,
           values by default will be associated with string types, and may need
           to be recast in parameter assignments, as shown below.
      -->
      <sx:parameter name="employee-no" value="{EMPNO}" type="xs:long"/>
      <sx:parameter name="mgr" value="{MGR}" type="xs:long"/>
      <sx:parameter name="hiredate" value="{HIREDATE}" type="xs:date"/>              
      <sx:parameter name="salary" value="{SAL}" type="xs:decimal"/>              
      <sx:parameter name="commission" value="{COMM}" type="xs:decimal"/>              
      <sx:sqlPrepare>                                                                                                            
        INSERT INTO EMP_HISTORY(EMPNO, ENAME, JOB, MGR, HIREDATE, SAL, COMM) 
        VALUES({$employee-no},{NAME},{JOB}, {$mgr}, {$hiredate},  {$salary}, {$commission})
      </sx:sqlPrepare>
    </sx:sqlUpdate>
  </sx:sqlBatchWriter>

  <sx:sqlReader id="employeesReader">
    <sx:sqlConnectionPool ref="jdbcPool"/>

    <sx:parameter name="jobList">
      <sx:toString value="{$job}" separator=",">
        <sx:quoteSymbol character="'" escapeCharacter="'"/>
      </sx:toString>
    </sx:parameter>

    <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO FROM EMP WHERE 1=1
      <sx:choose>
        <sx:when test="$jobList">
            AND JOB IN ({$jobList})
        </sx:when>
      </sx:choose>
        ORDER BY JOB,ENAME
    </sx:sqlQuery>

  </sx:sqlReader>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Executing the loadEmployees service with two job parameters, like this


    servingxml -r resources-batch_prepare_insert_employees.xml loadEmployees 
        job=SALESMAN job=MANAGER

results in the following SQL statement being sent to the Oracle server:


SELECT EMPNO, ENAME AS NAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO 
  FROM EMP 
  WHERE 1=1 AND JOB IN ('SALESMAN','MANAGER') 
  ORDER BY JOB,ENAME

These results are then written in batches of 4 to the database table EMP_HISTORY, using prepared SQL statements.

Updating an employee record using command-line parameters (employee update)

The example below illustrates how to prepare a resources script that will change an employee job categorization using command line parameters.

Figure 224. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="updateEmployee">
    <sx:recordStream>
      <sx:parameterReader recordType="employee"/>
      <sx:sqlWriter>
        <sx:sqlConnectionPool ref="jdbcPool"/>
        <sx:sqlUpdate>
          UPDATE EMP SET JOB = '{$job}' WHERE EMPNO='{$empNo}'
        </sx:sqlUpdate>
      </sx:sqlWriter>
    </sx:recordStream>
  </sx:service>
  
  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

You can run this example on the command line by entering


servingxml -r resources.xml -o updateEmployee 
    empNo=7499 job=ANALYST

Insert if record does not exist, update otherwise (insert/update).

This example shows how to prepare a resources script that will read rows from one database table and write them to another, inserting if the record does not already exist, updating otherwise.

The following resources script does the transformation.

Figure 225. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core"
               xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="loadEmployees">
    <sx:recordStream>
      <sx:sqlReader ref="employeesReader"/>
      <msv:recordValidator>
        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
          <!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
          <xs:element name="employee" type="EmployeeType"/>
          <xs:complexType name="EmployeeType">
            <xs:sequence>
              <xs:element name="EMPNO" type="xs:integer"/>
              <xs:element name="NAME" type="xs:string"/>
              <xs:element name="JOB" type="xs:string"/>
            </xs:sequence>
          </xs:complexType>
        </xs:schema>
      </msv:recordValidator>

      <sx:sqlWriter ref="employeesWriter"/>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <!-- You can include a record pipeline here to write bad records to a file or database table -->
      </sx:discardHandler>
    </sx:recordStream>
  </sx:service>

  <sx:sqlWriter id="employeesWriter">
    <sx:sqlConnectionPool ref="jdbcPool"/>
    <sx:sqlUpdateChoice>
      <sx:sqlQuery>
        SELECT EMPNO FROM EMP_HISTORY WHERE EMPNO = {EMPNO}
      </sx:sqlQuery>
      <sx:recordNotFound>
        <sx:sqlUpdate>
          INSERT INTO EMP_HISTORY(EMPNO, ENAME) VALUES({EMPNO},'{NAME}')
        </sx:sqlUpdate>
      </sx:recordNotFound>
      <sx:recordFound>
        <sx:sqlUpdate>
          UPDATE EMP_HISTORY SET ENAME = '{NAME}' WHERE EMPNO = {EMPNO}
        </sx:sqlUpdate>
      </sx:recordFound>
    </sx:sqlUpdateChoice>
  </sx:sqlWriter>

  <sx:sqlReader id="employeesReader">
    <sx:sqlConnectionPool ref="jdbcPool"/>

    <sx:parameter name="jobList">
      <sx:toString value="{$job}" separator=",">
        <sx:quoteSymbol character="'" escapeCharacter="'"/>
      </sx:toString>
    </sx:parameter>

    <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
      <sx:choose>
        <sx:when test="$jobList">
            AND JOB IN ({$jobList})
        </sx:when>
      </sx:choose>
        ORDER BY JOB,ENAME
    </sx:sqlQuery>

  </sx:sqlReader>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Execute the loadEmployees service with two job parameters, like this


    servingxml -r resources-insertUpdate.xml loadEmployees 
        job=ANALYST job=CLERK

Writing rows to a database table as they pass through a pipe, to an XML file (employees-tee)

This example shows how to prepare a resources script that will read rows from an RDBMS, write them to a database table as they pass through the pipe, and finally produce an XML file.

The desired XML file is shown below.

Figure 226. Output XML file employees.xml


<?xml version="1.0" encoding="utf-8"?>
<employees>
   <ANALYST>
      <employee employee-no="7902">
         <name>FORD</name>
      </employee>
      <employee employee-no="7788">
         <name>SCOTT</name>
      </employee>
   </ANALYST>
   <CLERK>
      <employee employee-no="7876">
         <name>ADAMS</name>
      </employee>
      <employee employee-no="7900">
         <name>JAMES</name>
      </employee>
      <employee employee-no="7934">
         <name>MILLER</name>
      </employee>
      <employee employee-no="7369">
         <name>SMITH</name>
      </employee>
   </CLERK>
</employees>

The following resources script does the transformation.

Figure 227. Resources script

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="employees">
    <sx:serialize>
      <sx:content ref="employeesDoc"/>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="employeesDoc">
    <sx:sqlReader>
      <sx:sqlConnectionPool ref="jdbcPool"/>

      <sx:parameter name="jobList">
        <sx:toString value="{$job}" separator=",">
          <sx:quoteSymbol character="'" escapeCharacter="'"/>
        </sx:toString>
      </sx:parameter>

      <sx:sqlQuery recordType = "employee">
        SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
        <sx:choose>
          <sx:when test="$jobList">
            AND JOB IN ({$jobList})
          </sx:when>
        </sx:choose>
        ORDER BY JOB,ENAME
      </sx:sqlQuery>
    </sx:sqlReader>

    <sx:recordTee>
      <sx:sqlWriter>
        <sx:sqlConnectionPool ref="jdbcPool"/>
        <sx:sqlUpdate>
            INSERT INTO EMP_LOG(EMPNO, ENAME) VALUES({EMPNO},'{NAME}')
        </sx:sqlUpdate>
      </sx:sqlWriter>
    </sx:recordTee>

    <sx:recordMapping ref="employeesToXml"/>
  </sx:recordContent>
                           
  <sx:recordMapping id="employeesToXml">
    <employees>
      <sx:groupBy fields="JOB">
        <sx:elementMap element="{JOB}">
          <sx:onRecord>
            <employee>
              <sx:fieldAttributeMap field="EMPNO" attribute="employee-no"/>
              <sx:fieldElementMap field="NAME" element="name"/>
            </employee>
          </sx:onRecord>
        </sx:elementMap>
      </sx:groupBy>
    </employees>
  </sx:recordMapping>

  <sx:sqlConnectionPool id="jdbcPool"
                        driver="oracle.jdbc.driver.OracleDriver"
                        databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
                        user="scott"
                        password="spring"
                        minConnections="2"
                        testStatement="SELECT * FROM DUAL"/>

</sx:resources>

Executing the employees service with two job parameters, like this


    servingxml -r resources-employeesTee.xml employees -o output/employees.xml employees 
        job=ANALYST job=CLERK

results in the following SQL statement being sent to the Oracle server:


SELECT EMPNO, ENAME AS NAME,JOB 
  FROM EMP 
  WHERE 1=1 AND JOB IN ('ANALYST','CLERK') 
  ORDER BY JOB,ENAME

Each EMPNO and ENAME pair is written to the database table EMP_LOG, and all rows are ultimately mapped to the XML file employees.xml.

Index

C

CDATA, opdef

F

field, transaction
field delimiter
comma, countries-default
numerical entity references, tab-delimited to XML
pipe, family_data (multi-valued) to XML
tab, tab-delimited to XML
field substitutions, field substitutions
file integrity
crc, checked books
size, checked books
filter
XSLT, element sequence to flat
flat file
CSV, countries-default, countries-custom, countries-validated, tasks (CSV) to XML, timesheets (CSV) to XML (grouped), books (XML) to flat file (CSV)
delimited, family_data (multi-valued) to XML, ars (delimited) to XML, exotics (delimited) to XML (grouped), books (XML) to flat file (delimited), books (XML) to listing, book hierarchy
Java properties, properties to XML, properties to XML (grouped), properties (muti-valued) to XML (grouped)
multi-valued fields, family_data (multi-valued) to XML, books (XML) to flat file (delimited), books (XML) to listing
multiple readers, multiple-readers
multiple record format, exotics (delimited) to XML (grouped), trades (positional) to XML, non-delimited trades to XML
no end-of-line delimiters, non-delimited book orders to XML, non-delimited trades to XML
positional, trades (positional) to XML, conv (positional) to XML, hot (preliminaries), hot (positional) to XML (grouped), books (XML) to flat file (positional), ungrouping, all, blocks, block subsequences, books (positional) to books (delimited), books (positional) to books (positional), declarative record filter, checked books, book directory to new format, batch records
variable width fields, variable field widths
record composition, record composition
segment concatenation, smf
single record format, conv (positional) to XML, non-delimited book orders to XML
flat file field
binary, special char to XML, XML to special char
integer, smf
packed decimal, packed decimal to XML, smf
flat file trailer
recordCount, books (XML) to flat file (positional)
flat file writer
default, employees csv
flat to XML
repeat header in XML, repeat header in XML
fo, document subtrees to PDF files, document subtrees to mail
FpML
FRA, FRA
Swap, Swap
FTP
sink, FTP sink
source, remote directories

O

optional elements, optional elements

R

record
composite, aggregates over records, two records to one element
record delimiter
continuation, properties to XML, properties to XML (grouped)
start/end, opdef, transaction
record filter
custom, hot (preliminaries)
schema validation, countries-validated, hot (preliminaries)
sx:replaceRecord, declarative record filter
record filters
sx:choose, books (XML) to flat file (delimited), books (XML) to listing
record mapping
default, countries-default
record reader
custom, custom record reader, multiple output files
record writer
multiple output files, multiple output files
records
reordering, multiple summaries
regular expressions
match and replace, books directory to XML, trades (positional) to XML, declarative record filter, book directory to new format
search and replace, ars (delimited) to XML
repeating groups, edi to XML, Invoice96A, segments
delimited, students-delim, invoice
fixed, students, all
nested, invoice
repeating segment, transaction

U

UTF-8
double byte, UTF-8 double byte

X

X12, X12 XML to Flat File
XML files
default namespace, multiple default ns
multiple, multiple XML book files, multiple default ns
XML input
multiple documents, aggregates over records, eobclaims to flat file
XML output
multiple documents, eobclaims to XMLs
XSLT extensions
using, hot (positional) to XML (grouped)