ServingXML Examples

Daniel Parker

Shaun Woodrow

Contributor of "trades" example 

Dimitrios Varsos

Contributor of "tasks" and "timesheets" examples 

Pedro Mendoza

Contributor of "hot" example 

Pierre-Alexandre Losson

Contributor of "invoices" example 

Aswin Majjiga

Contributor of "books-csv" example 

Dudley

Contributor of Java properties "nested tags" examples 

alexbena

Contributor of "CONV" example 

Zach

Contributor of "ars" example 

Sapna

Contributor of "BNF like flat file" example 

Bill Donovan

Contributor of edifact example 

Ahmadin

Contributor of abtest example 

Scott

Contributor of swath example 

Peter Scholz

Contributor of "segments" example 

Wonne Keysers

Contributor of "persons" example 

Ken Brewer

Contributor of "students" example 

Bala Murali

Contributor of "book orders" and "checked books" examples 

etdwh

Contributor of "opdef" example 

David Leal Valma

Contributor of "field has max width" example 

Anton Melumad

Contributor of "optional elements" example 

Lance Zant

Contributor of "XML-flat header-detail" example 

dealogic

Contributor of XML to flat "sequence" example 

Jim Williams

Contributor of XML to flat "multiple summaries" example 

Alvin

Contributor of XML to flat "book-listing" example 

Jim Nurii

Contributor of flat to XML and XML to flat "special char" examples 

Jose Ignacio Santa Cruz

Contributor of flat to XML "invoice" example 

Valentine

Contributor of XML to flat "ungrouping" example 

Ravikumar Tadysetty

Contributor of XML to flat "all" example 

José Santos

Contributor of XML to flat "formating distinct record lines" example 

Ramesh Ramasamy

Contributor of XML to flat "book hierarchy" example 

Gordon Zhang

Contributed additions to "exotics" example 

Roopashree Hirasave

Contributed "X12 XML to Flat File" example 

Ralph Kutzner

Contributed Flat-to-XML example "Computing an Aggregate Value over Multiple Records" 

Randal Cobb

Contributed Flat-to-XML example "Mapping Two Records at the Top of a File to a Header Element in the XML Output"  

Flat File to XML
Delimited Fields, Single Record Type
Converting a CSV File to XML Using Default Record Mapping (countries-default)
Converting a CSV File to XML Using Custom Record Mapping (countries-custom)
Converting a CSV file to XML with Record Validation (countries-validated)
Converting flat files with fields that end either with a delimiter or a maximum width (field has max width)
Converting a flat file to XML with one level of grouping (tasks)
Converting a flat file to XML with many levels of grouping (timesheets)
Converting a flat file with multi-valued fields to XML (family_data)
Search and Replace with Regular Expressions in Field Mappings (ars)
Converting a Tab Delimited Flat File to XML (tab-delimited)
Delimited Fields, Multiple Record Types
Converting a flat file with multiple record formats to XML with grouping (exotics)
Converting flat files with record mappings that have optional elements (optional elements)
Converting a Java properties file to XML (messages)
Transforming a Java properties file with extra name field delimiters to produce nested tags (nested tags 1)
Transforming a Java properties file with name subfield delimiters to produce nested tags (nested tags 2)
Reordering a group of records before mapping to XML (multiple summaries)
Delimited Fields, Multiple Record Types, Repeating Groups
Mapping multiple repeating groups to XML (students-delim)
EDI to XML (comptest)
Mapping a sequence of nested repeating groups to XML (invoice)
Fixed Width Fields, Single Record Type, Line Delimited
A flat file to XML mapping with field substitutions (abtest)
Mapping a record to separate top level sections in the XML (persons)
Converting a directory of books positional files to XML files (all books)
Converting a flat file to multiple XML book files (multiple books)
Converting a flat file to multiple XML documents with default namespace prefixes (multiple default ns)
Fixed Width Fields, Multiple Record Types, Line Delimited
Converting a flat file with multiple record formats to XML (trades)
Converting a flat file to XML with header, detail, and summary records (CONV)
Converting a positional file to XML using a record filter (hot 1)
Converting a positional file to XML with outer and inner grouping (hot 2)
Fixed Width Fields, Multiple Record Types, Repeating Groups, Line Delimited
Converting fixed position records with repeating groups to XML (segments)
Mapping multiple repeating groups to XML (students-fix)
Fixed Width Fields, Single Record Type, No End-of-Line Delimiters
Converting a flat file with a single record type, but no end-of-line delimiters, to XML (non-delimited book orders)
Fixed Width Fields, Multiple Record Types, No End-of-Line Delimiters
Converting a flat file with multiple record formats, but no end-of-line delimiters, to XML (non-delimited trades)
Converting a flat file with special characters to XML (special char)
Converting a flat file containing Cobol packed decimal data to XML (packed decimal to XML)
Start/End Delimited Records.
Converting flat files with start/end record delimiters (opdef)
Converting flat files with nested start/end record and segment delimiters and name/value pairs (transaction)
Custom Record Reader
Converting the output of a custom record reader to XML (custom record reader)
Composing an Aggregate over Adjacent Records
Computing an Aggregate Value over Multiple Records
Mapping Two Records at the Top of a File to a Header Element in the XML Output
Batching XML Output
Serializing XML in Batches
Writing Records to Batch Files, and Converting the Batches to Separate XML Files
SQL Query to XML
Mapping the results of a parameterized SQL query into XML (analysts)
Mapping the results of an ad hoc SQL query into XML (employees)
Using multiple readers to join data across different data sources (chained readers)
SQL Query to Flat File
Writing the results of a SQL query to a CSV file (employees-csv)
SQL Query to Database
SQL inserts
Batched SQL inserts
Prepared SQL inserts
Batched prepared SQL inserts
Updating an employee record using command-line parameters (employee update)
Insert if record does not exist, update otherwise (insert/update).
Writing rows to a database table as they pass through a pipe, to an XML file (employees-tee)
XML to Flat File
Converting an XML document to a positional flat file with record count in trailer (books-positional)
Converting an XML document with multiple occurances of an element to a CSV file (books-csv)
Converting an XML document with multiple occurances of an element to a flat file with subfield delimiters (books-pipe)
Converting each book entry in books.xml to a list of three records (book-listing)
Converting an XML document into flat file records with multi-valued fields (swath)
Converting an XML document into a flat file with header and multiple detail records (XML-flat header-detail)
Breaking up a sequence of elements into flat file records ("sequence")
Converting an XML file with hexidecimal encoded values to a flat file with special characters
Converting an XML document to a positional flat file and ungrouping an XML element (ungrouping)
XML to Flat Repeating Group (all)
Formating Distinct Record Lines (blocks)
Ungrouping Elements with Multiple Levels (book hierarchy)
X12 XML to Flat File using XSLT
Transforming a Sequence of XML Documents to a Flat File
Flat File to Flat File
Converting a positional file to a pipe delimited file with header (positional to delimited)
Converting a positional file to a new positional file with default field values (positional defaults)
Converting an ASCII Positional File to an EBCDIC Delimited File, and Vice Versa (ASCII-EBCDIC)
Converting an ASCII Positional File to an EBCDIC Positional File with a Cobol Packed Decimal Field, and Vice Versa (ASCII-Packed)
Converting a positional file to a pipe delimited file with record filtering (book orders)
Checking the Integrity of a Flat File ("checked books")
Converting a directory of book positional files to pipe delimited files (new format)
Generated records to multiple files (multiple_output_files)
Reading remote directories
Converting a sequential stream of records into multiple batch files
Batched Flat Files
Batched sequences of records
XML to XML
Extracting subtrees and wrapping them in containing tags
Unwrapping subtrees in an XML document
Streaming a file through a pipeline
Streaming dynamically generated SAX events through a pipeline
Processing document subtrees and serializing them to separate files (invoices.)
Processing document subtrees and serializing them as mail attachments ("mail invoices")
Reading dynamic content
XML to Mail
Writing an XML file to an FTP sink
Index

The source for console app examples may be found in the distribution under the samples directory.

Before running the sample console applications, be sure to build the servingxml project. Note that the examples will be deployed under the target/servingxml/samples directory. See Getting Started for details.

Flat File to XML

Delimited Fields, Single Record Type

Converting a CSV File to XML Using Default Record Mapping (countries-default)

This example shows a simple resources script that will map a CSV file to XML using default record mapping.

The input file is a CSV file.

Figure 1. Input flat file countries.csv


code,name
#ABW,ARUBA
ATF,"FRENCH SOUTHERN TERRITORIES, D.R. OF"
VUT,VANUATU
WLF,WALLIS & FUTUNA ISLANDS


This file has a number of properties.

  • The first row is a header with comma-separated column labels.

  • The second row is commented out.

  • The third row has a field value, enclosed in quotes, that contains an embedded comma.

  • The fifth row has a field value that contains a '&' symbol, which is a special character in XML, and must therefore be escaped in XML output as the entity '&'.

Shown below is the simplest possible resources script that will read this data and output it as XML.


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="countries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="countries-doc"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="countries-doc" name="countries">
    <sx:flatFileReader>
      <sx:commentStarter value="#"/>
      <sx:fieldDelimiter value=","/>
      <sx:urlSource url="data/countries.csv"/>
    </sx:flatFileReader>
  </sx:recordContent>

</sx:resources>

This resources script produces the following output.


<?xml version="1.0" encoding="utf-8"?>
<countries>
  <record>
    <code>ATF</code>
    <name>FRENCH SOUTHERN TERRITORIES, D.R. OF</name>
  </record>
  <record>
    <code>VUT</code>
    <name>VANUATU</name>
  </record>
  <record>
    <code>WLF</code>
    <name>WALLIS &amp; FUTUNA ISLANDS</name>
  </record>
</countries>

Note the following properties of the XML output.

  • The root element name comes from the sx:recordContent element's name attribute, if it has one, and if not it defaults to document.

  • The record element name defaults to record.

  • Each field is mapped to an element, and the name of the element defaults to the column label.

Converting a CSV File to XML Using Custom Record Mapping (countries-custom)

An sx:recordMapping element positioned inside an sx:recordContent element allows for custom record mapping to XML. This is optional, since there is a default mapping that emits the canonical XML representation of records. But typically the XML you want is different from the canonical representation, you may want a field mapped to an attribute rather than an element, for example, or perhaps some additional literal elements around the field mappings.

Continuing with the previous example, suppose you want the following output.


<?xml version="1.0" encoding="utf-8"?>
<countries>
  <country countryCode="ATF">
    <countryName>FRENCH SOUTHERN TERRITORIES, D.R. OF</countryName>
  </country>
  <country countryCode="VUT">
    <countryName>VANUATU</countryName>
  </country>
  <country countryCode="WLF">
    <countryName>WALLIS &amp; FUTUNA ISLANDS</countryName>
  </country>
</countries>

The following resources script does the job.


<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="countries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="countries"/> 
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="countries">
    <sx:flatFileReader>
      <sx:urlSource url="data/countries.csv"/>
      <sx:flatFile ref="countriesFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="countriesToXmlMapping"/>
  </sx:recordContent>
  
  <sx:flatFile id="countriesFile">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="country">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="code"/>
        <sx:delimitedField name="name"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>      
  
  <sx:recordMapping id="countriesToXmlMapping">
    <countries>
      <sx:onRecord>
        <country>
          <sx:fieldElementMap field="name" element="countryName"/>  
          <sx:fieldAttributeMap field="code" attribute="countryCode"/>
        </country>  
      </sx:onRecord>
    </countries>
  </sx:recordMapping>  
  
</sx:resources>

You can also take the default mappings for some fields and map others explicitly. For example,


<sx:recordMapping id="countries2xml">
  <countries>
    <sx:onRecord>
      <country>
        <sx:defaultFieldMapping fields="*" except="code"/>
        <sx:fieldAttributeMap field="code" attribute="countryCode"/>
      </country>
    </sx:onRecord>
  </countries>
</sx:recordMapping>

Here, the sx:defaultFieldMapping element maps all the fields in the record to elements with the same names, except the field name "code". This field is handled separately, with the sx:fieldAttributeMap element.

Converting a CSV file to XML with Record Validation (countries-validated)

The example below illustrates how to prepare a resources script that will convert a flat file to XML. Each row in the flat file is validated with Sun's multi schema validator (MSV), and if an error is encountered, the row is discarded, but processing continues. In addition, the output XML is also validated with Sun's MSV, and if that should fail, processing is stopped.

This time the input file has an invalid country code in the first record.


code,name
ATFX,"FRENCH SOUTHERN TERRITORIES, D.R. OF"
VUT,VANUATU
WLF,WALLIS & FUTUNA ISLANDS

The desired output is


<?xml version="1.0" encoding="utf-8"?>
<countries>
  <country countryCode="ATF">
    <countryName>FRENCH SOUTHERN TERRITORIES, D.R. OF</countryName>
  </country>
  <country countryCode="VUT">
    <countryName>VANUATU</countryName>
  </country>
  <country countryCode="WLF">
    <countryName>WALLIS &amp; FUTUNA ISLANDS</countryName>
  </country>
</countries>

The following resources script does the transformation.

Figure 2. Resources script resources-countries.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="countries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="countries"/>
        <msv:schemaValidator>
          <sx:urlSource url="data/countries.xsd"/>
        </msv:schemaValidator>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:flatFile id="countriesFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType ref="country"/>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:flatRecordType id="country" name="country">
    <sx:fieldDelimiter value=","/>
    <sx:delimitedField name="code"/>
    <sx:delimitedField name="name"/>
  </sx:flatRecordType>

  <sx:flatFile id="discardFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="countryDiscard">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="message"/>
        <sx:flatRecordType ref="country"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="countriesToXmlMapping">
    <countries>
      <sx:onRecord>
        <country>
          <sx:fieldElementMap field="name" element="countryName"/>
          <sx:fieldAttributeMap field="code" attribute="countryCode"/>
        </country>
      </sx:onRecord>
    </countries>
  </sx:recordMapping>

  <sx:recordContent id="countries">
    <sx:recordStream>
      <sx:flatFileReader>
        <sx:urlSource url="data/countriesWithDiscards.csv"/>
        <sx:flatFile ref="countriesFile"/>
      </sx:flatFileReader>
      <msv:recordValidator>
        <sx:urlSource url="data/country-record.xsd"/>
      </msv:recordValidator>
      <sx:discardHandler>
        <sx:log message="{$sx:message}"/>
        <sx:modifyRecord>
          <sx:newField name="message" value="{$sx:message}"/>
        </sx:modifyRecord>
        <sx:flatFileWriter>
          <sx:fileSink file="output/countryDiscards.csv"/>
          <sx:flatFile ref="discardFile"/>
        </sx:flatFileWriter>
      </sx:discardHandler>
    </sx:recordStream>
    <sx:recordMapping ref="countriesToXmlMapping"/>
  </sx:recordContent>

</sx:resources>


The schema to validate an individual country record is shown below.

Figure 3. XML Schema file country-record.xsd


<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

 <!-- This element's name matches the value of the name attribute in the sx:flatRecordType element. -->
 <xs:element name="country" type="CountryType"/>

 <xs:complexType name="CountryType">
  <xs:sequence>
   <xs:element name="code" type="CountryCode"/>
   <xs:element name="name" type="xs:string"/>
  </xs:sequence>
 </xs:complexType>

  <xs:simpleType name='CountryCode'>
    <xs:restriction base='xs:string'>
      <xs:length value='3' fixed='true'/>
    </xs:restriction>
  </xs:simpleType>
</xs:schema>


You run this example on the command line by entering


servingxml -r resources-countries.xml -o output/countries.xml countries

The following error message will be written to the log:


Error in record "country" on line 1.  the length of the value is 4, but the required length is 3.

The remaining two rows will be written to the output file.

Converting flat files with fields that end either with a delimiter or a maximum width (field has max width)

The example below illustrates how to prepare a resources script that will convert a flat file with start/end record delimiters.

The input file is shown below.


1231234
1 123
12 12

This file has the following structure.

  • The first record consists of a field of length 3 ("123") followed by a field of length 4 ("1234")

  • The second line consists of a field of length 1 ("1") followed by a field of length 3 ("123")

Each field has a maximum width, but the field may be terminated by a delimiter (here whitespace) before it attains that width.

The desired output is shown below.


<?xml version="1.0" encoding="utf-8"?>
<document>
  <record>
    <field1>123</field1>
    <field2>1234</field2>
  </record>
  <record>
    <field1>1</field1>
    <field2>123</field2>
  </record>
  <record>
    <field1>12</field1>
    <field2>12</field2>
  </record>
</document>

The following resources script does the transformation.


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="fieldHasMaxWidth">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="fieldHasMaxWidthDoc"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="fieldHasMaxWidthDoc" name="document">
    <sx:flatFileReader>
      <sx:urlSource url="data/fieldHasMaxWidth.txt"/>
      <sx:flatFile ref="fieldHasMaxWidthFile"/>
    </sx:flatFileReader>
  </sx:recordContent>

  <sx:flatFile id="fieldHasMaxWidthFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="record">
        <sx:fieldDelimiter>
          <sx:whitespaceSeparator/>
        </sx:fieldDelimiter>
        <sx:delimitedField name="field1" maxWidth="3"/>
        <sx:delimitedField name="field2" maxWidth="4"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

You can run this example on the command line by entering


servingxml -r resources-fieldHasMaxWidth.xml  
  -o output/fieldHasMaxWidth.xml fieldHasMaxWidth

Converting a flat file to XML with one level of grouping (tasks)

The example below illustrates how to prepare a resources script that will convert a flat file to XML with grouping levels.

The input file is a CSV file.

Figure 4. Input flat file tasks.csv


project_id, task_name, task_start, task_finish
4001,task1,01/01/2003,01/31/2003
4001,task2,02/01/2003,02/28/2003
4001,task3,03/01/2003,03/31/2003
4002,task1,04/01/2003,04/30/2003
4002,task2,05/01/2003,05/31/2003


The desired output is shown below.

Figure 5. Output XML file tasks.xml


 <?xml version="1.0" encoding="utf-8" ?> 
<Projects>
  <Project projectID="4001">
    <Tasks>
      <Task name="task1" start="01/01/2003" finish="01/31/2003" /> 
      <Task name="task2" start="02/01/2003" finish="02/28/2003" /> 
      <Task name="task3" start="03/01/2003" finish="03/31/2003" /> 
    </Tasks>
  </Project>
  <Project projectID="4002">
    <Tasks>
      <Task name="task1" start="04/01/2003" finish="04/30/2003" /> 
      <Task name="task2" start="05/01/2003" finish="05/31/2003" /> 
    </Tasks>
  </Project>
</Projects>

The following resources script does the transformation.

Figure 6. Resources script resources-tasks.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="tasks">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="tasks"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="tasks">
    <sx:flatFileReader>
      <sx:urlSource url="data/tasks.csv"/>
      <sx:flatFile ref="tasksFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="tasksToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="tasksFlatFile">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="task">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="project_id"/>
        <sx:delimitedField name="task_name"/>
        <sx:delimitedField name="task_start"/>
        <sx:delimitedField name="task_finish"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="tasksToXmlMapping">
    <Projects>
      <sx:groupBy fields="project_id">
        <Project>
          <sx:fieldAttributeMap field="project_id" attribute="projectID"/>
          <Tasks>
            <sx:onRecord>
              <Task>
                <sx:fieldAttributeMap field="task_name" attribute="name"/>
                <sx:fieldAttributeMap field="task_start" attribute="start"/>
                <sx:fieldAttributeMap field="task_finish" attribute="finish"/>
              </Task>
            </sx:onRecord>
          </Tasks>
        </Project>
      </sx:groupBy>
    </Projects>
  </sx:recordMapping>
  
</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-tasks.xml -o output/tasks.xml tasks

Converting a flat file to XML with many levels of grouping (timesheets)

The example below illustrates how to prepare a resources script that will convert a flat file to XML with many grouping levels.

The input file is a CSV file.

Figure 7. Input flat file timesheets.csv


TimePeriod,Start,Finish,Resource,Task,ActualDate,Amount
1,2/9/2004,2/15/2004,Joe_100,1001,2/9/2004,8
1,2/9/2004,2/15/2004,Joe_100,1001,2/10/2004,4
1,2/9/2004,2/15/2004,Joe_100,1002,2/11/2004,8
1,2/9/2004,2/15/2004,Joe_100,1003,2/10/2004,4
1,2/9/2004,2/15/2004,Joe_100,1003,2/12/2004,8
1,2/9/2004,2/15/2004,Mark_101,1001,2/10/2004,8
1,2/9/2004,2/15/2004,Mark_101,1001,2/14/2004,8
1,2/9/2004,2/15/2004,Mark_101,1006,2/9/2004,8
1,2/9/2004,2/15/2004,Mark_101,1008,2/12/2004,4
1,2/9/2004,2/15/2004,Mark_101,1008,2/13/2004,4
2,2/16/2004,2/22/2004,Joe_100,1001,2/16/2004,8
2,2/16/2004,2/22/2004,Joe_100,1001,2/17/2004,4
2,2/16/2004,2/22/2004,Joe_100,1002,2/17/2004,4


The desired output is shown below.

Figure 8. Output XML file timesheets.xml


<?xml version="1.0" encoding="utf-8"?>
<TimePeriods>
  <TimePeriod start="2/9/2004" finish="2/15/2004">
    <Timesheets>
      <Timesheet resource="Joe_100">
        <TimesheetEntries>
          <TimesheetEntry task="1001">
            <DailyActuals>
              <Actual actualDate="2/9/2004" amount="8"/>
              <Actual actualDate="2/10/2004" amount="4"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1002">
            <DailyActuals>
              <Actual actualDate="2/11/2004" amount="8"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1003">
            <DailyActuals>
              <Actual actualDate="2/10/2004" amount="4"/>
              <Actual actualDate="2/12/2004" amount="8"/>
            </DailyActuals>
          </TimesheetEntry>
        </TimesheetEntries>
      </Timesheet>
      <Timesheet resource="Mark_101">
        <TimesheetEntries>
          <TimesheetEntry task="1001">
            <DailyActuals>
              <Actual actualDate="2/10/2004" amount="8"/>
              <Actual actualDate="2/14/2004" amount="8"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1006">
            <DailyActuals>
              <Actual actualDate="2/9/2004" amount="8"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1008">
            <DailyActuals>
              <Actual actualDate="2/12/2004" amount="4"/>
              <Actual actualDate="2/13/2004" amount="4"/>
            </DailyActuals>
          </TimesheetEntry>
        </TimesheetEntries>
      </Timesheet>
    </Timesheets>
  </TimePeriod>
  <TimePeriod start="2/16/2004" finish="2/22/2004">
    <Timesheets>
      <Timesheet resource="Joe_100">
        <TimesheetEntries>
          <TimesheetEntry task="1001">
            <DailyActuals>
              <Actual actualDate="2/16/2004" amount="8"/>
              <Actual actualDate="2/17/2004" amount="4"/>
            </DailyActuals>
          </TimesheetEntry>
          <TimesheetEntry task="1002">
            <DailyActuals>
              <Actual actualDate="2/17/2004" amount="4"/>
            </DailyActuals>
          </TimesheetEntry>
        </TimesheetEntries>
      </Timesheet>
    </Timesheets>
  </TimePeriod>
</TimePeriods>


The following resources script does the transformation.

Figure 9. Resources script resources-timesheets.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="timesheets"> 
    <sx:serialize>
      <sx:transform>
        <sx:content ref="timesheets"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>
  
  <sx:recordContent id="timesheets">
    <sx:flatFileReader>
      <sx:urlSource url="data/timesheets.csv"/>
      <sx:flatFile ref="timesheetsFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="timesheetsToXmlMapping"/>
  </sx:recordContent>
  
  <sx:flatFile id="timesheetsFlatFile">
    <sx:commentStarter value="#"/>
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="task">
        <sx:fieldDelimiter value=","/>
        <sx:delimitedField name="TimePeriod"/>
        <sx:delimitedField name="Start"/>
        <sx:delimitedField name="Finish"/>
        <sx:delimitedField name="Resource"/>
        <sx:delimitedField name="Task"/>
        <sx:delimitedField name="ActualDate"/>
        <sx:delimitedField name="Amount"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>      
  
  <sx:recordMapping id="timesheetsToXmlMapping">
    <TimePeriods>
      <sx:groupBy fields="Start">
        <TimePeriod>
          <sx:fieldAttributeMap field="Start" attribute="start"/>
          <sx:fieldAttributeMap field="Finish" attribute="finish"/>
          <Timesheets>
            <sx:groupBy fields="Start Resource">
              <Timesheet>
                <sx:fieldAttributeMap field="Resource" attribute="resource"/>
                <TimesheetEntries>
                  <sx:groupBy fields="Start Resource Task">
                    <TimesheetEntry>
                      <sx:fieldAttributeMap field="Task" attribute="task"/>
                      <DailyActuals>
                        <sx:onRecord>
                          <Actual>
                            <sx:fieldAttributeMap field="ActualDate" attribute="actualDate"/>
                            <sx:fieldAttributeMap field="Amount" attribute="amount"/>
                          </Actual>
                        </sx:onRecord>
                      </DailyActuals>
                    </TimesheetEntry>
                  </sx:groupBy>
                </TimesheetEntries>
              </Timesheet>
            </sx:groupBy>
          </Timesheets>
        </TimePeriod>
      </sx:groupBy>
    </TimePeriods>
  </sx:recordMapping>  
  
</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-timesheets.xml -o output/timesheets.xml timesheets

Converting a flat file with multi-valued fields to XML (family_data)

The example below illustrates how to prepare a resources script that will convert a flat file with multi-valued fields to XML.

The input file is a pipe delimited flat file with semicolon sub-delimiters.

Figure 10. Input flat file employees.txt


father_name|mother_name|children
Matthew|Sarah|
Scott||Damian;Janet;Paul


The desired output is shown below.

Figure 11. Output XML file multivalued.xml


<?xml version="1.0" encoding="UTF-8"?>
<eg:families xmlns:eg="http://examples.com/">
  <eg:family>
    <eg:father-name>Matthew</eg:father-name>
    <eg:mother-name>Sarah</eg:mother-name>
    <eg:children/>
  </eg:family>
  <eg:family>
    <eg:father-name>Scott</eg:father-name>
    <eg:mother-name/>
    <eg:children>
      <eg:child>Damian</eg:child>
      <eg:child>Janet</eg:child>
      <eg:child>Paul</eg:child>
    </eg:children>
  </eg:family>
</eg:families>


The following resources script does the transformation.

Figure 12. Resources script resources-multivalued.xml


<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
                    xmlns:eg="http://examples.com/">

  <sx:service id="families">                               
    <sx:serialize>
      <sx:transform>
        <sx:content ref="families"/>
        <!-- sx:removeEmptyElements elements="eg:children"/ -->
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="families">
    <sx:flatFileReader>
      <sx:flatFile ref="family-records"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="families-to-xml-mapping"/>
  </sx:recordContent>
  
  <sx:flatFile id="family-records">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="family">
        <sx:fieldDelimiter value="|"/>
        <sx:delimitedField name="father_name"/>
        <sx:delimitedField name="mother_name"/>
        <sx:delimitedField name="children">
          <sx:subfieldDelimiter value=";"/>
        </sx:delimitedField>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>
  
  <sx:recordMapping id="families-to-xml-mapping">
    <eg:families xmlns:eg="http://examples.com/">
      <sx:onRecord>
        <eg:family>
          <sx:fieldElementMap field="father_name" element="eg:father-name"/>
          <sx:fieldElementMap field="mother_name" element="eg:mother-name"/>
          <eg:children>
            <sx:fieldElementSequenceMap field="children" element="eg:child"/>
          </eg:children>
        </eg:family>
      </sx:onRecord>
    </eg:families>
  </sx:recordMapping>
  
</sx:resources>


Remarks
  • Note that Matthew and Sarah have no children. If you wanted to remove the empty myns:children element, uncomment the sx:removeEmptyElements element in the sx:transform block of the resources script,

You can run this example on the command line by entering


servingxml -r resources-family_data.xml 
    -i data/family_data.txt -o output/family_data.xml families

Search and Replace with Regular Expressions in Field Mappings (ars)

The example below illustrates how to prepare a resources script that will convert a flat file to XML using search and replace with regular expressions in field mappings.

The input file is a carot (^) delimited flat file with a header.

Figure 13. Input flat file ars.dat


ProjectID^Project_Name^Description^PLAN_IT_RESC_DAYS^PLAN_IT_COST^ANNUAL_EBIT^ACTUAL_NPV^ACTUAL_IRR 
1234-00-0005^XOG new project insert^This is to test and verify the XOG'ing of new Project data from the ARS database.^430^59,627^100,351^3^9.2 


Note the following features of this data.

  • The first row is a header row.

  • The fourth and fifth fields contain embedded commas in numbers, which we want to strip out.

The desired output is shown below.

Figure 14. Output XML file ars.xml


<?xml version="1.0" encoding="utf-8"?>
<NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Header action="write" externalSource="NIKU" objectType="project" version="6.0.11"/>
  <Projects>
    <Project name="XOG new project insert" projectID="1234-00-0005" description="This is to test and verify the XOG'ing of new Project data from the ARS database.">
      <CustomInformation>
        <PLAN_IT_RESOURCE_DAYS>430</PLAN_IT_RESOURCE_DAYS>
        <PLAN_IT_COST>59627</PLAN_IT_COST>
        <ANNUAL_EBIT>100351</ANNUAL_EBIT>
        <ACTUAL_NPV>3</ACTUAL_NPV>
        <ACTUAL_IRR>9.2</ACTUAL_IRR>
      </CustomInformation>
      <General addedBy="XOG" addedDate="03/26/2005 8:10:09 PM"/>
    </Project>
  </Projects>
</NikuDataBus>

The following resources script does the transformation.

Figure 15. Resources script resources-ars.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="ars">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="ars"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="ars">
    <sx:flatFileReader>
      <sx:flatFile ref="arsFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="arsToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="arsFlatFile">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="detail">
        <sx:fieldDelimiter value="^"/>
        <sx:delimitedField name="ProjectID"/>
        <sx:delimitedField name="Project_Name"/>
        <sx:delimitedField name="Description"/>
        <sx:delimitedField name="PLAN_IT_RESC_DAYS"/>
        <sx:delimitedField name="PLAN_IT_COST"/>
        <sx:delimitedField name="ANNUAL_EBIT"/>
        <sx:delimitedField name="ACTUAL_NPV"/>
        <sx:delimitedField name="ACTUAL_IRR"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="arsToXmlMapping">
    <NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

      <Header version="6.0.11" action="write" objectType="project" externalSource="NIKU"/>
      <Projects>
        <sx:groupBy fields="ProjectID">
          <Project>
            <sx:fieldAttributeMap field="Project_Name" attribute="name"/>
            <sx:fieldAttributeMap field="ProjectID" attribute="projectID"/>
            <sx:fieldAttributeMap field="Description" attribute="description"/>
            <sx:onRecord>
              <CustomInformation>
                <sx:fieldElementMap field="PLAN_IT_RESC_DAYS" element="PLAN_IT_RESOURCE_DAYS"/>
                <sx:elementMap element="PLAN_IT_COST">
                  <sx:replace searchFor="," replaceWith ="">
                    <sx:toString value="{PLAN_IT_COST}"/>
                  </sx:replace>
                </sx:elementMap>
                <sx:elementMap element="ANNUAL_EBIT">
                  <sx:replace searchFor="," replaceWith ="">
                    <sx:toString value="{ANNUAL_EBIT}"/>
                  </sx:replace>
                </sx:elementMap>
                <sx:fieldElementMap field="ACTUAL_NPV" element="ACTUAL_NPV"/>
                <sx:fieldElementMap field="ACTUAL_IRR" element="ACTUAL_IRR"/>
              </CustomInformation>
              <General addedBy="XOG">
                <sx:fieldAttributeMap attribute="addedDate">
                  <sx:formatDateTime format="MM/dd/yyyy h:mm:ss a">
                    <sx:currentDateTime/>
                  </sx:formatDateTime>
                </sx:fieldAttributeMap>
              </General>
            </sx:onRecord>
          </Project>
        </sx:groupBy>
      </Projects>
    </NikuDataBus>
  </sx:recordMapping>

</sx:resources>


Note the following points about this script.

You can run this example on the command line by entering


servingxml -r resources-ars.xml -i data/ars.txt -o output/ars.xml ars 

Converting a Tab Delimited Flat File to XML (tab-delimited)

The example below illustrates how to prepare a resources script that will convert a flat file with tab delimiters to XML.

The input file is a tab (0x09) delimited flat file with a header.

Figure 16. Input flat file tab_delimited_employees.txt


employee-no	employee-name	dept	salary
00000001	Smith, Matthew	sales	150,000.00
00000002	Brown, Sarah	sales	89,000.00
00000003	Oberc, Scott	finance	110,000.00
00000004	Scott, Colette	sales	75,000.00


The desired output is shown below.

Figure 17. Output XML file employees.xml


<?xml version="1.0" encoding="utf-8"?>
<employees>
  <employee>
    <employee-no>00000001</employee-no>
    <employee-name>Smith, Matthew</employee-name>
    <department>sales</department>
    <salary>150,000.00</salary>
  </employee>
  <employee>
    <employee-no>00000002</employee-no>
    <employee-name>Brown, Sarah</employee-name>
    <department>sales</department>
    <salary>89,000.00</salary>
  </employee>
  <employee>
    <employee-no>00000003</employee-no>
    <employee-name>Oberc, Scott</employee-name>
    <department>finance</department>
    <salary>110,000.00</salary>
  </employee>
  <employee>
    <employee-no>00000004</employee-no>
    <employee-name>Scott, Colette</employee-name>
    <department>sales</department>
    <salary>75,000.00</salary>
  </employee>
</employees>

The following resources script does the transformation.

Figure 18. Resources script resources-tab_delimited_employees.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="employees">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="employee-data"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="employee-data" name="employees">
    <sx:flatFileReader>
      <sx:urlSource url="data/tab_delimited_employees.txt"/>
      <sx:flatFile ref="employee-file"/>
    </sx:flatFileReader>
  </sx:recordContent>

  <sx:flatFile id="employee-file">
    <sx:flatFileHeader lineCount="1"/>
    <sx:flatFileBody>
      <sx:fieldDelimiter value="\t"/>
      <!-- Another way of specifying a horizontal tab character -->
      <!-- <sx:fieldDelimiter value="&#x09;"/> -->
      <sx:flatRecordType name="employee">
        <sx:delimitedField name="employee-no"/>
        <sx:delimitedField name="employee-name"/>
        <sx:delimitedField name="department"/>
        <sx:delimitedField name="salary"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>


Note the following points about this script.

  • The tab delimiter may be specified either with the \t symbol, or, alternatively, with the numerical entity reference &#x09; numerical entity

You can run this example on the command line by entering


servingxml -o output/employees.xml 
    -r resources-tab_delimited_employees.xml employees 

Delimited Fields, Multiple Record Types

Converting a flat file with multiple record formats to XML with grouping (exotics)

The example below illustrates how to prepare a resources script that will convert a pipe delimited flat file with multiple record formats to XM with grouping.

The input file is a pipe delimited file.

Figure 19. Input flat file exotics.txt


SWAP|1234567|FLOAT|PAY|CAD|BA|2010/04/28|10000000.00
Cap|1234567||CAD|SELL|2010/04/28|0.03
SWAP|1234567|FIX|REC|CAD
Cap|1234568||CAD|SELL|2010/04/28|0.05


The desired output is shown below.

Figure 20. Output XML file exotics.xml


<exotics>
  <trade tradeId="1234567">
    <option>SWAP</option>
    <swapLeg style="FLOAT">
      <side>PAY</side>
      <maturityDate original="2010/04/28">2001/04/29</maturityDate>
      <currency>CAD</currency>
      <index>BA</index>
      <notional>10000000.00</notional>
    </swapLeg>
    <capSell>
      <buySell>SELL</buySell>
      <maturityDate original="2010/04/28">2001/04/28</maturityDate>
      <strike>0.03</strike>
    </capSell>
    <swapLeg style="FIX">
      <side>REC</side>
    </swapLeg>
  </trade>
  <trade tradeId="1234568">
    <option>Cap</option>
    <capBuy>
      <buySell>BUY</buySell>
      <maturityDate original="2010/04/28">2001/04/28</maturityDate>
      <strike>0.05</strike>
    </capBuy>
  </trade>
</exotics>


The following resources script does the transformation.

Figure 21. Resources script resources-exotics.xml


<?xml version="1.0"?>

<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="exotics">
	<sx:serialize>
	  <sx:transform>
		<sx:content ref="exotics"/>
	  </sx:transform>
	</sx:serialize>
  </sx:service>

  <sx:recordContent id="exotics">
	<sx:flatFileReader>
	  <sx:fileSource file="data/exotics.txt"/>
	  <sx:flatFile ref="exoticsFlatFile"/>
	</sx:flatFileReader>
	<sx:recordMapping ref="exoticsToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="exoticsFlatFile">
	<sx:flatFileBody>
	  <sx:flatRecordTypeChoice>
		<sx:fieldDelimiter value="|"/>
		<sx:delimitedField name="recordType"/>
		<sx:delimitedField name="tradeId"/>
		<sx:delimitedField name="style"/>
		<sx:when test="recordType='Cap'">
		  <sx:flatRecordType name='Cap'>
			<sx:fieldDelimiter value="|"/>
			<sx:delimitedField name="recordType"/>
			<sx:delimitedField name="tradeId"/>
			<sx:delimitedField name="placeholder1"/>
			<sx:delimitedField name="currency"/>
			<sx:delimitedField name="buySell"/>
			<sx:delimitedField name="maturityDate"/>
			<sx:delimitedField name="strike"/>
		  </sx:flatRecordType>
		</sx:when>
		<sx:when test="recordType='SWAP' and style='FLOAT'">
		  <sx:flatRecordType name='SwapFloatingLeg'>
			<sx:fieldDelimiter value="|"/>
			<sx:delimitedField name="recordType"/>
			<sx:delimitedField name="tradeId"/>
			<sx:delimitedField name="style"/>
			<sx:delimitedField name="side"/>
			<sx:delimitedField name="currency"/>
			<sx:delimitedField name="index"/>
			<sx:delimitedField name="maturityDate"/>
			<sx:delimitedField name="notional"/>
		  </sx:flatRecordType>
		</sx:when>
		<sx:when test="recordType='SWAP' and style='FIX'">
		  <sx:flatRecordType name='SwapFixedLeg'>
			<sx:fieldDelimiter value="|"/>
			<sx:delimitedField name="recordType"/>
			<sx:delimitedField name="tradeId"/>
			<sx:delimitedField name="style"/>
			<sx:delimitedField name="side"/>
			<sx:delimitedField name="placeholder1"/>
			<sx:delimitedField name="currency"/>
		  </sx:flatRecordType>
		</sx:when>
	  </sx:flatRecordTypeChoice>
	</sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="exoticsToXmlMapping">
	<exotics>
	  <sx:groupBy fields="tradeId">
		<trade><option>
		  <sx:choose>
			<sx:when test="recordType='Cap' or (recordType='SWAP' and style='FLOAT')">
			  <sx:toString value="{recordType}"/>
			</sx:when>
		  </sx:choose></option>
		  <sx:fieldAttributeMap field="tradeId" attribute="tradeId"/>
		  <sx:onRecord recordType="Cap">
			  <sx:choose>
				<sx:when test="buySell='BUY'">
		 	<capBuy>
			  <sx:fieldElementMap field="buySell" element="buySell"/>
			  <sx:elementMap element="maturityDate">
				<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
			  	<sx:choose>
			  		<sx:when test="maturityDate='2010/04/28'">
			  			<sx:toString value="2001/04/28"/>
			  		</sx:when>
			  	</sx:choose>
			  </sx:elementMap>
			  <sx:fieldElementMap field="strike" element="strike"/>
			</capBuy>
				</sx:when>
				<sx:otherwise>
		 	<capSell>
			  <sx:fieldElementMap field="buySell" element="buySell"/>
			  <sx:elementMap element="maturityDate">
				<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
			  	<sx:choose>
			  		<sx:when test="maturityDate='2010/04/28'">
			  			<sx:toString value="2001/04/28"/>
			  		</sx:when>
			  	</sx:choose>
			  </sx:elementMap>
			  <sx:fieldElementMap field="strike" element="strike"/>
			</capSell>
				</sx:otherwise>
			  </sx:choose>
		  </sx:onRecord>
		  <sx:onRecord recordType="SwapFloatingLeg">
			<swapLeg>
			  <sx:fieldAttributeMap field="style" attribute="style"/>
			  <sx:fieldElementMap field="side" element="side"/>
			  <maturityDate>
				<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
			  	<sx:choose>
			  		<sx:when test="maturityDate='2010/04/28'">
			  			<sx:toString value="2001/04/29"/>
			  		</sx:when>
			  	</sx:choose>
			  </maturityDate>
			  <sx:fieldElementMap field="currency" element="currency"/>
			  <sx:fieldElementMap field="index" element="index"/>
			  <sx:fieldElementMap field="notional" element="notional"/>
			</swapLeg>
		  </sx:onRecord>
		  <sx:onRecord recordType="SwapFixedLeg">
			<swapLeg>
			  <sx:fieldAttributeMap field="style" attribute="style"/>
			  <sx:fieldElementMap field="side" element="side"/>
			</swapLeg>
		  </sx:onRecord>
		</trade>
	  </sx:groupBy>
	</exotics>
  </sx:recordMapping>

</sx:resources>


You can run this example on the command line by entering


servingxml -r resources-exotics.xml -o output/exotics.xml exotics

Converting flat files with record mappings that have optional elements (optional elements)

The example below shows how to prepare a resources script for converting a flat file to XML by applying an XSLT transform to the canonical XML representation of each record.

The input file is shown below.


262025218|John|Smith|657827168||1|Main Street|London|45879|56|||||
262091012|Jane|Bell||000000000000|105|Penny Lane|Oxford|12345|56|1|Main Street|London|45879|56

Each line has an id, followed by a first and last name, followed by a customer address, and optionally a billing address. If the house number and street of the billing address are empty, the rule is that the entire billing address be omitted from the XML output.

The desired output is shown below.


<?xml version="1.0" encoding="utf-8"?>
<mns:Message xmlns:mns="www.abc.com/abc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="www.abc.com/abc abc_v_1.1.xsd ">
   <wf_id>262025218</wf_id>
   <timeStamp>2006-02-11T22:18:21</timeStamp>
   <MessageData>
      <addCustomerRequest>
         <foreName>John</foreName>
         <surName>Smith</surName>
         <custAddress>
            <houseNumber>1</houseNumber>
            <street>Main Street</street>
            <town>London</town>
            <postCode>45879</postCode>
            <country>56</country>
         </custAddress>
      </addCustomerRequest>
   </MessageData>
   <wf_id>262091012</wf_id>
   <timeStamp>2006-02-11T22:18:21</timeStamp>
   <MessageData>
      <addCustomerRequest>
         <foreName>Jane</foreName>
         <surName>Bell</surName>
         <custAddress>
            <houseNumber>105</houseNumber>
            <street>Penny Lane</street>
            <town>Oxford</town>
            <postCode>12345</postCode>
            <country>56</country>
         </custAddress>
         <billAddress>
            <houseNumber>1</houseNumber>
            <street>Main Street</street>
            <town>London</town>
            <postCode>45879</postCode>
            <country>56</country>
         </billAddress>
      </addCustomerRequest>
   </MessageData>
</mns:Message>

The following resources script does the transformation.


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="addCustomer">
    <sx:serialize>
      <sx:xsltSerializer>
        <sx:outputProperty name="indent" value="yes"/>
      </sx:xsltSerializer>
      <sx:transform>
        <sx:content ref="addCustomer"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:flatFile id="customerFile">
    <sx:flatFileBody>
      <sx:flatRecordType name="customer">
        <sx:fieldDelimiter value="|"/>
        <sx:delimitedField name="rowid"/>
        <sx:delimitedField name="forename"/>
        <sx:delimitedField name="surname"/>
        <sx:delimitedField name="vatnumber"/>
        <sx:delimitedField name="ninumber"/>
        <sx:delimitedField name="custaddr_house_nr"/>
        <sx:delimitedField name="custaddr_street"/>
        <sx:delimitedField name="custaddr_town"/>
        <sx:delimitedField name="custaddr_postcode"/>
        <sx:delimitedField name="custaddr_country"/>
        <sx:delimitedField name="billaddr_house_nr"/>
        <sx:delimitedField name="billaddr_street"/>
        <sx:delimitedField name="billaddr_town"/>
        <sx:delimitedField name="billaddr_postcode"/>
        <sx:delimitedField name="billaddr_country"/>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordContent id="addCustomer">
    <sx:flatFileReader>
      <sx:fileSource file="data/optionalElements.txt"/>
      <sx:flatFile ref="customerFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="addCustomerToXmlMapping"/>
  </sx:recordContent>

  <sx:recordMapping id="addCustomerToXmlMapping">
    <mns:Message xmlns:mns="www.abc.com/abc"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xsi:schemaLocation="www.abc.com/abc abc_v_1.1.xsd">
      <sx:onRecord>
        <sx:fieldElementMap field="rowid" element="wf_id"/>
        <timeStamp>
          <sx:formatDateTime format = "yyyy-MM-dd'T'HH:mm:ss">
            <sx:currentDateTime/>
          </sx:formatDateTime>
        </timeStamp>
        <sx:transformRecord>
          <sx:xslt>
            <xsl:transform version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
              <xsl:template match="customer">
                <MessageData>
                  <addCustomerRequest>
                    <xsl:element name="foreName">
                      <xsl:value-of select="forename"/>
                    </xsl:element>
                    <xsl:element name="surName">
                      <xsl:value-of select="surname"/>
                    </xsl:element>
                    <custAddress>
                      <xsl:element name="houseNumber">
                        <xsl:value-of select="custaddr_house_nr"/>
                      </xsl:element>
                      <xsl:element name="street">
                        <xsl:value-of select="custaddr_street"/>
                      </xsl:element>
                      <xsl:element name="town">
                        <xsl:value-of select="custaddr_town"/>
                      </xsl:element>
                      <xsl:element name="postCode">
                        <xsl:value-of select="custaddr_postcode"/>
                      </xsl:element>
                      <xsl:element name="country">
                        <xsl:value-of select="custaddr_country"/>
                      </xsl:element>
                    </custAddress>
                    <xsl:if test="string(billaddr_house_nr) and string(billaddr_street)">
                      <billAddress>
                        <xsl:element name="houseNumber">
                          <xsl:value-of select="billaddr_house_nr"/>
                        </xsl:element>
                        <xsl:element name="street">
                          <xsl:value-of select="billaddr_street"/>
                        </xsl:element>
                        <xsl:element name="town">
                          <xsl:value-of select="billaddr_town"/>
                        </xsl:element>
                        <xsl:element name="postCode">
                          <xsl:value-of select="billaddr_postcode"/>
                        </xsl:element>
                        <xsl:element name="country">
                          <xsl:value-of select="billaddr_country"/>
                        </xsl:element>
                      </billAddress>
                    </xsl:if>
                  </addCustomerRequest>
                </MessageData>
              </xsl:template>
            </xsl:transform>
          </sx:xslt>
        </sx:transformRecord>
      </sx:onRecord>
    </mns:Message>
  </sx:recordMapping>

</sx:resources>

You can run this example on the command line by entering


servingxml -r resources-optionalElements.xml  
  -o output/optionalElements.xml optionalElements

Converting a Java properties file to XML (messages)

The example below illustrates how to prepare a resources script that will convert a Java properties file to XML.

The input file is a Java properties file.

Figure 22. Input property file messages.properties


parser.ope.2=')' or '-[' or '+[' or '&[' is expected.
parser.factor.5=A back reference or an anchor or a lookahead or a lookbehind \
is expected in a conditional pattern.
parser.next.2='?' is not expected.  '(?:' or '(?=' or '(?!' or '(?<' or '(?#' or '(?>'?
#parser.next.3='(?<=' or '(?<!' is expected.


This file has a number of features.

  • The entries are key-value pairs separated by the '=' symbol.

  • The value in the first line contains an '&' symbol, which is a special character in XML, and will be escaped in the output file.

  • The second line ends with a continuation character.

  • The value in the third line contains an '=' symbol.

  • The last line starts with a comment symbol.

The desired output is shown below.

Figure 23. Output XML file messages.xml


<?xml version="1.0" encoding="utf-8"?>
<messages>
  <parser.ope.2>')' or '-[' or '+[' or '&amp;[' is expected.</parser.ope.2>
  <parser.factor.5>A back reference or an anchor or a lookahead or a lookbehind is expected in a conditional pattern.</parser.factor.5>
  <parser.next.2>'?' is not expected.  '(?:' or '(?=' or '(?!' or '(?&lt;' or '(?#' or '(?&gt;'?</parser.next.2>
</messages>


The following resources script does the transformation.

Figure 24. Resources script resources-messages.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">
   
  <sx:service id="messages">                         
    <sx:serialize>
      <sx:transform>
        <sx:content ref="messages"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>
  
  <sx:recordContent id="messages">
    <sx:flatFileReader>
      <sx:urlSource url="data/messages.properties"/>
      <sx:flatFile>
        <sx:recordDelimiter continuation="\" value="\r\n"/> 
        <sx:recordDelimiter continuation="\" value="\n"/> 
        <sx:commentStarter value="#"/>
        <sx:flatFileBody>
          <sx:flatRecordType name="property">
            <sx:delimitedField name="name">
              <sx:fieldDelimiter value="="/>
            </sx:delimitedField>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:flatFileBody>
      </sx:flatFile>
    </sx:flatFileReader>

    <sx:recordMapping>
      <messages>
        <sx:onRecord>
          <sx:fieldElementMap field="value" element="{name}"/>
        </sx:onRecord>
      </messages>
    </sx:recordMapping>
  </sx:recordContent>
  
</sx:resources>


Note the following points about this script.

  • The '=' delimiter is associated only with the name field, not the value field.

  • There is no delimiter specified for the value field, nor for the record as a whole. This means that the value field will contain all the text between the first '=' symbol and the record delimiter, including any more '=' symbols.

You can run this example on the command line by entering


servingxml -r resources-messages.xml -o output/messages.xml 
    messages

Transforming a Java properties file with extra name field delimiters to produce nested tags (nested tags 1)

This example shows one way to write a resources script that will convert a Java properties file to XML with nested tags.

The input file is a Java properties file.

Figure 25. Input property file multipart-keys.properties


image.text.info1=hello world 1 
image.text.info2=hello \
world 2 
image.text.info3=hello world 3


This file has a number of features.

  • The entries are key-value pairs separated by an '=' symbol.

  • Each name consists of three parts separated by a '.' symbol

  • The second line ends with a continuation character.

The desired output is shown below.

Figure 26. Output XML file nested-tags1.xml


<?xml version="1.0" encoding="utf-8"?>
<messages>
  <image>
    <text>
      <info1>hello world 1</info1>
      <info2>hello world 2</info2>
      <info3>hello world 3</info3>
    </text>
  </image>
</messages>


Note that each level of nesting corresponds to separate part of the key.

The following resources script does the transformation.

Figure 27. Resources script resources-props2nested1.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="messages">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="messages"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="messages">

    <sx:flatFileReader>
      <sx:flatFile>
        <sx:commentStarter value="#"/>
        <sx:commentStarter value="//"/>
        <sx:recordDelimiter continuation="\" value="\r\n"/>
        <sx:recordDelimiter continuation="\" value="\n"/>
        <sx:flatFileBody>
          <sx:flatRecordType name="property">
            <sx:delimitedField name="key1">
              <sx:fieldDelimiter value="."/>
            </sx:delimitedField>
            <sx:delimitedField name="key2">
              <sx:fieldDelimiter value="."/>
            </sx:delimitedField>
            <sx:delimitedField name="key3">
              <sx:fieldDelimiter value="="/>
            </sx:delimitedField>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:flatFileBody>
      </sx:flatFile>
    </sx:flatFileReader>

    <sx:recordMapping>
      <messages>
        <sx:groupBy fields="key1">
          <sx:elementMap element="{key1}">
            <sx:groupBy fields="key2 key2">
              <sx:elementMap element="{key2}">
                <sx:onRecord>
                  <sx:fieldElementMap field="value" element="{key3}"/>
                </sx:onRecord>
              </sx:elementMap>
            </sx:groupBy>
          </sx:elementMap>
        </sx:groupBy>
      </messages>
    </sx:recordMapping>

  </sx:recordContent>

</sx:resources>


This script relies on reading each record as four fields.

  • key1 is terminated with a '.' delimiter
  • key2 is terminated with a '.' delimiter
  • key3 is terminated with an '=' delimiter
  • value is terminated with a record delimiter

You can run this example on the command line by entering


servingxml -i data/multipart-keys.properties 
    -o output/nested-tags1.xml -r resources-props2nested1.xml messages

Transforming a Java properties file with name subfield delimiters to produce nested tags (nested tags 2)

This example shows another way to write a resources script that will convert a Java properties file to XML with nested tags.

The properties input file and the desired XML output file are the same as the previous example.

This time the resources script is written as follows.

Figure 28. Resources script resources-props2nested2.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="messages">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="messages"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="messages">

    <sx:flatFileReader>
      <sx:flatFile>
        <sx:commentStarter value="#"/>
        <sx:commentStarter value="//"/>
        <sx:recordDelimiter continuation="\" value="\r\n"/>
        <sx:recordDelimiter continuation="\" value="\n"/>
        <sx:flatFileBody>
          <sx:flatRecordType name="property">
            <sx:delimitedField name="name">
              <sx:fieldDelimiter value="="/>
              <sx:subfieldDelimiter value="."/>
            </sx:delimitedField>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:flatFileBody>
      </sx:flatFile>
    </sx:flatFileReader>

    <sx:recordMapping>
      <messages>
        <sx:groupBy fields="name[1]">
          <sx:elementMap element="{name[1]}">
            <sx:groupBy fields="name[1] name[2]">
              <sx:elementMap element="{name[2]}">
                <sx:onRecord>
                  <sx:fieldElementMap field="value" element="{name[3]}"/>
                </sx:onRecord>
              </sx:elementMap>
            </sx:groupBy>
          </sx:elementMap>
        </sx:groupBy>
      </messages>
    </sx:recordMapping>

  </sx:recordContent>

</sx:resources>


This script relies on reading each record as two fields, a multi-valued name field and a value field.

  • name has three values that may be indexed as 1...3
  • value has one value

You can run this example on the command line by entering


servingxml -i data/multipart-keys.properties 
    -o output/nested-tags2.xml -r resources-props2nested2.xml messages

Reordering a group of records before mapping to XML (multiple summaries)

The example below shows how to prepare a resources script for reordering a group of records before mapping them to XML.

The input file is shown below.


HEADER  
S 001 
S 002 
B 001
S 003 
B 002
TRAILER 

Each 'B' record represents a summary of the 'S' records since the last summary..

Suppose you want to turn it into the following:.


<?xml version="1.0" encoding="utf-8"?>
<DOCUMENT>
  <HEADER/>
  <B value="001">
    <S value="001"/>
    <S value="002"/>
  </B>
  <B value="003">
    <S value="003"/>
  </B>
  <TRAILER/>
</DOCUMENT>

If the summary records came before the details, rather than after, the mapping would be straightforward: see the examples with sx:innerGroup grouping elements. This suggests reordering the records as they come in.

The following resources script does the transformation.


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="multipleSummaries">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="multipleSummariesContent"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="multipleSummariesContent">
    <sx:flatFileReader>
      <sx:urlSource url="data/multipleSummaries.txt"/>
      <sx:flatFile ref="multipleSummariesFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="multipleSummariesMapping"/>
  </sx:recordContent>

  <sx:recordMapping id="multipleSummariesMapping">
    <DOCUMENT>
      <sx:groupChoice>

        <sx:innerGroup startTest="sx:current/Header" 
                       endTest="sx:previous/Header">
          <sx:onRecord>
            <HEADER/>
          </sx:onRecord>
        </sx:innerGroup>

        <sx:innerGroup startTest="sx:current/Trailer" 
                       endTest="sx:previous/Trailer">
          <sx:onRecord>
            <TRAILER/>
          </sx:onRecord>                                        
        </sx:innerGroup>

        <sx:innerGroup startTest="sx:current/S" 
                       endTest="sx:previous/B">
          <!-- reorder the records in this group - move the B records to the front -->
          <sx:reorderRecords recordTypes="B S"/>
            <B>
              <sx:fieldAttributeMap field="value" attribute="value"/>
              <sx:onRecord recordType='S'>
                <S>
                  <sx:fieldAttributeMap field="value" attribute="value"/>
                </S>
              </sx:onRecord>
            </B>
        </sx:innerGroup>

      </sx:groupChoice>
    </DOCUMENT>
  </sx:recordMapping>

  <sx:flatFile id="multipleSummariesFile">
    <sx:flatFileBody>
      <sx:fieldDelimiter>                            
        <sx:whitespaceSeparator/>
      </sx:fieldDelimiter>
      <sx:flatRecordTypeChoice>
        <sx:delimitedField name="tag"/>
        <sx:when test="tag='HEADER'">
          <sx:flatRecordType name="Header">
            <sx:delimitedField name="tag"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='TRAILER'">
          <sx:flatRecordType name="Trailer">
            <sx:delimitedField name="tag"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='S'">
          <sx:flatRecordType name="S">
            <sx:delimitedField name="tag"/>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:when>
        <sx:when test="tag='B'">
          <sx:flatRecordType name="B">
            <sx:delimitedField name="tag"/>
            <sx:delimitedField name="value"/>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

</sx:resources>

You can run this example on the command line by entering


servingxml -r resources-multipleSummaries.xml  
  -o output/multipleSummaries.xml multipleSummaries

Delimited Fields, Multiple Record Types, Repeating Groups

Mapping multiple repeating groups to XML (students-delim)

Suppose you have the following delimited file of students, their course grades, and their addresses.

Figure 29. Input flat file students-delim.txt


JANE^ENGL^C-~MATH^A+|1972^BLUE^CHICAGO^IL~ATLANTA^GA


The file has the following layout.

namethe field delimiter ^ defines the end of the field
subject-graderepeating group, repeating segments are separated by the repeat delimiter ~, the segment delimiter | defines the end of the group
year-bornthe field delimiter ^ defines the end of the field
favorite-colorthe field delimiter ^ defines the end of the field
addressrepeating group, repeating segments are separated by the repeat delimiter ~, the segment delimiter | defines the end of the group

You want the XML output to look as follows.

Figure 30. Output XML file students-fix.xml


<?xml version="1.0" encoding="utf-8"?>
<StudentGrades>
  <StudentGrade>
    <Name>JANE</Name>
    <SubjectGrade>
      <Subject>ENGL</Subject>
      <Grade>C-</Grade>
    </SubjectGrade>
    <SubjectGrade>
      <Subject>MATH</Subject>
      <Grade>A+</Grade>
    </SubjectGrade>
    <YearBorn>1972</YearBorn>
    <FavoriteColor>BLUE</FavoriteColor>
    <Address>
      <City>CHICAGO</City>
      <State>IL</State>
    </Address>
    <Address>
      <City>ATLANTA</City>
      <State>GA</State>
    </Address>
  </StudentGrade>
</StudentGrades>


The required resources script is

Figure 31. Resources script resources-students-delim.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="students">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="students-content"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="students-content">
    <sx:flatFileReader>
      <sx:flatFile ref="students-file"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="students-mapping"/>
  </sx:recordContent>

  <sx:flatFile id="students-file">
    <sx:flatFileBody>
      <sx:flatRecordType name="student">
        <sx:fieldDelimiter value="^"/>                                              
        <sx:repeatDelimiter value="~"/>
        <sx:segmentDelimiter value="|"/>
        <sx:delimitedField name="name"/>
        <sx:repeatingGroup name="grades">
          <sx:flatRecordType name="subject-grade">
            <sx:fieldDelimiter value="^"/>                                              
            <sx:delimitedField name="subject"/>
            <sx:delimitedField name="grade"/>
          </sx:flatRecordType>
        </sx:repeatingGroup>
        <sx:delimitedField name="year-born"/>
        <sx:delimitedField name="favorite-color"/>
        <sx:repeatingGroup name="addresses">
          <sx:flatRecordType name="address">
            <sx:fieldDelimiter value="^"/>                                              
            <sx:delimitedField name="city"/>
            <sx:delimitedField name="state"/>
          </sx:flatRecordType>
        </sx:repeatingGroup>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="students-mapping">
    <StudentGrades>
      <sx:onRecord>
        <StudentGrade>
          <sx:fieldElementMap field="name" element="Name"/>
          <sx:segmentMapping field="grades">
            <sx:onRecord>
              <SubjectGrade>
                <sx:fieldElementMap field="subject" element="Subject"/>
                <sx:fieldElementMap field="grade" element="Grade"/>
              </SubjectGrade>
            </sx:onRecord>
          </sx:segmentMapping>
          <sx:fieldElementMap field="year-born" element="YearBorn"/>
          <sx:fieldElementMap field="favorite-color" element="FavoriteColor"/>
          <sx:segmentMapping field="addresses">
            <sx:onRecord>
              <Address>
                <sx:fieldElementMap field="city" element="City"/>
                <sx:fieldElementMap field="state" element="State"/>
              </Address>
            </sx:onRecord>
          </sx:segmentMapping>
        </StudentGrade>
      </sx:onRecord>
    </StudentGrades>
  </sx:recordMapping>

</sx:resources>


You can run this example on the command line by entering


servingxml  -r resources-students-delim.xml -i data/students-delim.txt -o output/students-delim.xml 
    students

EDI to XML (comptest)

The example below illustrates how to prepare a resources script that will convert a composite field in an edi file to XML.

The input file is an edi file.

Figure 32. Input edi file comptest.txt


BKG+1++4+Y:1:B:0:M:0:L:1'


This file has a number of features.

  • The segment delimiter "'" terminates the segment.

  • The field delimiter "+" terminates a field.

  • Composite fields are delimited with ":".

The desired output is shown below.

Figure 33. Output XML file messages.xml


<?xml version="1.0" encoding="utf-8"?>
<edifact>
  <segment segmentType="BKG">
    <flightSegmentNumber>1</flightSegmentNumber>
    <accessID/>
    <bookingCodeCount>4</bookingCodeCount>
    <C704>
      <bookingCode>Y</bookingCode>
      <seatAvailabilityCount>1</seatAvailabilityCount>
    </C704>
    <C704>
      <bookingCode>B</bookingCode>
      <seatAvailabilityCount>0</seatAvailabilityCount>
    </C704>
    <C704>
      <bookingCode>M</bookingCode>
      <seatAvailabilityCount>0</seatAvailabilityCount>
    </C704>
    <C704>
      <bookingCode>L</bookingCode>
      <seatAvailabilityCount>1</seatAvailabilityCount>
    </C704>
  </segment>
</edifact>


The following resources script does the transformation.

Figure 34. Resources script resources-comptest.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="edifact">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="edifact"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="edifact">
    <sx:flatFileReader>
      <sx:flatFile ref="edifactFlatFile"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="edifactToXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="edifactFlatFile">
    <sx:recordDelimiter value="'"/>
    <sx:flatFileBody>
      <sx:flatRecordTypeChoice>
        <sx:fieldDelimiter value="+"/>
        <sx:delimitedField name="segmentType"/>
        <!-- edifact3 - Body Segments.-->
        <!-- BKG BOOKING CODE DATA .-->
        <sx:when test="segmentType='BKG'">
          <sx:flatRecordType name='BKG'>
            <sx:fieldDelimiter value="+"/>
            <sx:delimitedField name="segmentType"/>
            <sx:delimitedField name="flightSegmentNumber"/>
            <sx:delimitedField name="accessID"/>
            <sx:delimitedField name="bookingCodeCount"/>
            <sx:repeatingGroup name="booking" count="{bookingCodeCount}">
              <sx:flatRecordType name="C704">
                <sx:fieldDelimiter value=":"/>
                <sx:delimitedField name="bookingCode">
                </sx:delimitedField>
                <sx:delimitedField name="seatAvailabilityCount">
                </sx:delimitedField>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:when>
      </sx:flatRecordTypeChoice>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="edifactToXmlMapping">
    <edifact>
      <!-- edifact3 - Body Segments.-->
      <!-- BKG BOOKING CODE DATA.-->
      <sx:onRecord recordType="BKG">
        <sx:elementMap element="segment">
          <sx:fieldAttributeMap field="segmentType" attribute="segmentType"/>
          <sx:fieldElementMap field="flightSegmentNumber" element="flightSegmentNumber"/>
          <sx:fieldElementMap field="accessID" element="accessID"/>
          <sx:fieldElementMap field="bookingCodeCount" element="bookingCodeCount"/>
          <sx:segmentMapping field="booking">
            <sx:onRecord recordType="C704">
              <C704>
                <sx:fieldElementMap field="bookingCode" element="bookingCode"/>
                <sx:fieldElementMap field="seatAvailabilityCount" element="seatAvailabilityCount"/>
              </C704>
            </sx:onRecord>
          </sx:segmentMapping>
        </sx:elementMap>
      </sx:onRecord>
    </edifact>
  </sx:recordMapping>

</sx:resources>


Note the following points about this script.

  • All the values of the composite field are put in the multiple valued field "C704".

  • The sx:repeatingGroup element allows us to map these values, repeated two at a time, named as bookingCode and seatAvailabilityCount.

You can run this example on the command line by entering


servingxml -i data/comptest.txt -r resources-comptest.xml 
  -o output/comptest.xml edifact

Mapping a sequence of nested repeating groups to XML (invoice)

Suppose you have the following data file.

Figure 35. Input flat file invoice.txt


12|2007-02-02|DocType|} 
SndId|SndName|SndAddress|SndZip|} 
RecId|RecName|RecAddress|RecZip|} 
~ 
1|Item 1|2|Kg|150|300|} 
2|Item 2|2|Kg|350|700|} 
3|Item 3|1|$|50|50|} 
4|Item 4|10|Unt|30|100|} 
~ 
1|Ref 1|Doc 1|Text|} 
2|Ref 2|Doc 2|Text|} 
~ 
1|Disc 1|Item 1|} 
2|Disc 2|Item 2|} 
3|Disc 3|Item 3|} 
~ 
Code a|Value 1|} 
Code z|Value 3|} 
Code x|Value 2|} 
Code w|Value 2|} 
Code y|Value 1|}
~ 
!


The file has n sections separated by the "~" character. The first section has several non repeating records separated by a "}". But after that, there are 4 repeating sections separated by the same "~" as above. Each repeating section starts where the previous one ends.

The character "!" as the first character in the last line indicates the end of the data.

You want the XML output to look as follows.

Figure 36. Output XML file invoice.xml


<?xml version="1.0" encoding="utf-8"?>
<InvoiceDocument>
   <Header>
      <DocId>
         <Id>12</Id>
         <DocDate>2007-02-02</DocDate>
         <DocumentType>DocType</DocumentType>
      </DocId>
      <Sender>
         <SenderPassport>SndId</SenderPassport>
         <SenderFullName>SndName</SenderFullName>
         <SenderAddress>SndAddress</SenderAddress>
         <SenderZip>SndZip</SenderZip>
      </Sender>
      <Recipient>
         <RecipientPassport>RecId</RecipientPassport>
         <RecipientFullName>RecName</RecipientFullName>
         <RecipientAddress>RecAddress</RecipientAddress>
         <RecipientZip>RecZip</RecipientZip>
      </Recipient>
   </Header>
   <Detail>
      <Item>1</Item>
      <Description>Item 1</Description>
      <Quantity>2</Quantity>
      <Unit>Kg</Unit>
      <ItemPrice>150</ItemPrice>
      <TotalPrice>300</TotalPrice>
   </Detail>
   <Detail>
      <Item>2</Item>
      <Description>Item 2</Description>
      <Quantity>2</Quantity>
      <Unit>Kg</Unit>
      <ItemPrice>350</ItemPrice>
      <TotalPrice>700</TotalPrice>
   </Detail>
   <Detail>
      <Item>3</Item>
      <Description>Item 3</Description>
      <Quantity>1</Quantity>
      <Unit>$</Unit>
      <ItemPrice>50</ItemPrice>
      <TotalPrice>50</TotalPrice>
   </Detail>
   <Detail>
      <Item>4</Item>
      <Description>Item 4</Description>
      <Quantity>10</Quantity>
      <Unit>Unt</Unit>
      <ItemPrice>30</ItemPrice>
      <TotalPrice>100</TotalPrice>
   </Detail>
   <Reference>
      <RefId>1</RefId>
      <RefType>Ref 1</RefType>
      <RefDoc>Doc 1</RefDoc>
      <RefText>Text</RefText>
   </Reference>
   <Reference>
      <RefId>2</RefId>
      <RefType>Ref 2</RefType>
      <RefDoc>Doc 2</RefDoc>
      <RefText>Text</RefText>
   </Reference>
   <Discount>
      <DiscId>1</DiscId>
      <DiscDescription>Disc 1</DiscDescription>
      <DiscItem>Item 1</DiscItem>
   </Discount>
   <Discount>
      <DiscId>2</DiscId>
      <DiscDescription>Disc 2</DiscDescription>
      <DiscItem>Item 2</DiscItem>
   </Discount>
   <Discount>
      <DiscId>3</DiscId>
      <DiscDescription>Disc 3</DiscDescription>
      <DiscItem>Item 3</DiscItem>
   </Discount>
   <Code>
      <Code>Code a</Code>
      <Value>Value 1</Value>
   </Code>
   <Code>
      <Code>Code z</Code>
      <Value>Value 3</Value>
   </Code>
   <Code>
      <Code>Code x</Code>
      <Value>Value 2</Value>
   </Code>
   <Code>
      <Code>Code w</Code>
      <Value>Value 2</Value>
   </Code>
   <Code>
      <Code>Code y</Code>
      <Value>Value 1</Value>
   </Code>
</InvoiceDocument>


The required resources script is

Figure 37. Resources script resources-invoice.xml


<sx:resources xmlns:sx="http://www.servingxml.com/core">

  <sx:service id="invoice">
    <sx:serialize>
      <sx:xsltSerializer>
        <sx:outputProperty name="indent" value="yes"/>
      </sx:xsltSerializer>
      <sx:transform>
        <sx:content ref="invoice-content"/>
      </sx:transform>
    </sx:serialize>
  </sx:service>

  <sx:recordContent id="invoice-content">
    <sx:flatFileReader>
      <sx:flatFile ref="invoice-file"/>
    </sx:flatFileReader>
    <sx:recordMapping ref="invoice-mapping"/>
  </sx:recordContent>

  <sx:flatFile id="invoice-file">
    <sx:recordDelimiter value="\r\n!"/>
    <sx:recordDelimiter value="\n!"/>
    <sx:flatFileBody>
      <sx:flatRecordType name="invoice" omitFinalRepeatDelimiter="false">
        <sx:repeatDelimiter value="~"/>
        <sx:repeatingGroup name="Header" count="1">
          <sx:flatRecordType name="Header">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <!--sx:segmentDelimiter value="~"/-->
            <sx:repeatingGroup name="Document" count="1">
              <sx:flatRecordType name="Doc">
                <sx:delimitedField name="Id"/>
                <sx:delimitedField name="DocDate"/>
                <sx:delimitedField name="DocumentType"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
            <sx:repeatingGroup name="Sender" count="1">
              <sx:flatRecordType name="Sender">
                <sx:delimitedField name="SenderPassport"/>
                <sx:delimitedField name="SenderFullName"/>
                <sx:delimitedField name="SenderAddress"/>
                <sx:delimitedField name="SenderZip"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
            <sx:repeatingGroup name="Recipient" count="1">
              <sx:flatRecordType name="Recipient">
                <sx:delimitedField name="RecipientPassport"/>
                <sx:delimitedField name="RecipientFullName"/>
                <sx:delimitedField name="RecipientAddress"/>
                <sx:delimitedField name="RecipientZip"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>

        <sx:repeatingGroup name="Detail" count="1">
          <sx:flatRecordType name="Detail">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <sx:repeatingGroup name="Item">
              <sx:flatRecordType name="Item">
                <sx:delimitedField name="Item"/>
                <sx:delimitedField name="Description"/>
                <sx:delimitedField name="Quantity"/>
                <sx:delimitedField name="Unit"/>
                <sx:delimitedField name="ItemPrice"/>
                <sx:delimitedField name="TotalPrice"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>

        <sx:repeatingGroup name="Reference" count="1">
          <sx:flatRecordType name="Reference">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <sx:segmentDelimiter value="~"/>
            <sx:repeatingGroup name="Ref">
              <sx:flatRecordType name="Ref">
                <sx:delimitedField name="RefId"/>
                <sx:delimitedField name="RefType"/>
                <sx:delimitedField name="RefDoc"/>
                <sx:delimitedField name="RefText"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>

        <sx:repeatingGroup name="Discount" count="1">
          <sx:flatRecordType name="Discount">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <sx:segmentDelimiter value="~"/>
            <sx:repeatingGroup name="Disc">
              <sx:flatRecordType name="Disc">
                <sx:delimitedField name="DiscId"/>
                <sx:delimitedField name="DiscDescription"/>
                <sx:delimitedField name="DiscItem"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>

        <sx:repeatingGroup name="Codes" count="1">
          <sx:flatRecordType name="Codes">
            <sx:fieldDelimiter value="|"/>
            <sx:repeatDelimiter value="}"/>
            <sx:segmentDelimiter value="~"/>
            <sx:repeatingGroup name="Code">
              <sx:flatRecordType name="Code">
                <sx:delimitedField name="Code"/>
                <sx:delimitedField name="Value"/>
              </sx:flatRecordType>
            </sx:repeatingGroup>
          </sx:flatRecordType>
        </sx:repeatingGroup>
      </sx:flatRecordType>
    </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="invoice-mapping">
    <InvoiceDocument>
      <sx:onRecord>
        <sx:segmentMapping field="Header">
          <sx:onRecord>
            <Header>
              <sx:segmentMapping field="Document">
                <sx:onRecord>
                  <DocId>
                    <sx:fieldElementMap field="Id" element="Id"/>
                    <sx:fieldElementMap field="DocDate" element="DocDate"/>
                    <sx:fieldElementMap field="DocumentType" element="DocumentType"/>
                  </DocId>
                </sx:onRecord>
              </sx:segmentMapping>
              <sx:segmentMapping field="Sender">
                <sx:onRecord>
                  <Sender>
                    <sx:fieldElementMap field="SenderPassport" element="SenderPassport"/>
                    <sx:fieldElementMap field="SenderFullName" element="SenderFullName"/>
                    <sx:fieldElementMap field="SenderAddress" element="SenderAddress"/>
                    <sx:fieldElementMap field="SenderZip" element="SenderZip"/>
                  </Sender>
                </sx:onRecord>
              </sx:segmentMapping>
              <sx:segmentMapping field="Recipient">
                <sx:onRecord>
                  <Recipient>
                    <sx:fieldElementMap field="RecipientPassport" element="RecipientPassport"/>
                    <sx:fieldElementMap field="RecipientFullName" element="RecipientFullName"/>
                    <sx:fieldElementMap field="RecipientAddress" element="RecipientAddress"/>
                    <sx:fieldElementMap field="RecipientZip" element="RecipientZip"/>
                  </Recipient>
                </sx:onRecord>
              </sx:segmentMapping>
            </Header>
          </sx:onRecord>
        </sx:segmentMapping>

        <sx:segmentMapping field="Detail">
          <sx:onRecord>
            <sx:segmentMapping field="Item">
              <sx:onRecord>
                <Detail>
                  <sx:fieldElementMap field="Item" element="Item"/>
                  <sx:fieldElementMap field="Description" element="Description"/>
                  <sx:fieldElementMap field="Quantity" element="Quantity"/>
                  <sx:fieldElementMap field="Unit" element="Unit"/>
                  <sx:fieldElementMap field="ItemPrice" element="ItemPrice"/>
                  <sx:fieldElementMap field="TotalPrice" element="TotalPrice"/>
                </Detail>
              </sx:onRecord>
            </sx:segmentMapping>
          </sx:onRecord>
        </sx:segmentMapping>

        <sx:segmentMapping field="Reference">
          <sx:onRecord>
            <sx:segmentMapping field="Ref">
              <sx:onRecord>
                <Reference>
                  <sx:fieldElementMap field="RefId" element="RefId"/>
                  <sx:fieldElementMap field="RefType" element="RefType"/>
                  <sx:fieldElementMap field="RefDoc" element="RefDoc"/>
                  <sx:fieldElementMap field="RefText" element="RefText"/>
                </Reference>
              </sx:onRecord>
            </sx:segmentMapping>
          </sx:onRecord>
        </sx:segmentMapping>

        <sx:segmentMapping field="Discount">
          <sx:onRecord>
            <sx:segmentMapping field="Disc">
              <sx:onRecord>
                <Discount>
                  <sx:fieldElementMap field="DiscId" element="DiscId"/>
                  <sx:fieldElementMap field="DiscDescription" element="DiscDescription"/>
                  <sx:fieldElementMap field="DiscItem" element="DiscItem"/>
                </Discount>
              </sx:onRecord>
            </sx:segmentMapping>
          </sx:onRecord>
        </sx:segmentMapping>

        <sx:segmentMapping field="Codes">
          <sx:onRecord>
            <sx:segmentMapping field="Code">
              <sx:onRecord>
                <Code>
                  <sx:fieldElementMap field="Code" element="Code"/>
                  <sx:fieldElementMap field="Value" element="Value"/>
                </Code>
              </sx:onRecord>
            </sx:segmentMapping>
          </sx:onRecord>
        </sx:segmentMapping>
      </sx:onRecord>
    </InvoiceDocument>
  </sx:recordMapping>

</sx:resources>


The idea is to map all the data up to the terminating "!" as one record, and to structure the record as a sequence of nested repeating groups.

You can run this example on the command line by entering


servingxml  -r resources-invoice.xml -i data/invoice.txt -o output/invoice.xml 
    invoice

Fixed Width Fields, Single Record Type, Line Delimited

A flat file to XML mapping with field substitutions (abtest)

The example below illustrates how to prepare a resources script that will convert a flat file to XML.

The input file is a positional file.

Figure 38. Input flat file abtest.txt


ABTESTCD10 
ACTESTCD12 
ABTESTCC80


The requirements are as follows:

  • First 2 chars: if AB the output will be SOUTH, if AC then output is WEST.
  • For the 7th and 9th chars: if CD, then output change to 01, otherwise 02

The desired output is shown below.

Figure 39. Output XML file countries.xml


<?xml version="1.0" encoding="utf-8"?>
<Recs>
  <Rec>
    <First>SOUTH</First>
    <Second>TEST</Second>
    <Third>01</Third>
    <Last>10</Last>
  </Rec>
  <Rec>
    <First>WEST</First>
    <Second>TEST</Second>
    <Third>01</Third>
    <Last>12</Last>
  </Rec>
  <Rec>
    <First>SOUTH</First>
    <Second>TEST</Second>
    <Third>02</Third>
    <Last>80</Last>
  </Rec>
</Recs>

The following resources script does the transformation.

Figure 40. Resources script resources-abtest.xml


<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
              xmlns:msv="http://www.servingxml.com/extensions/msv">

  <sx:service id="abtest">
    <sx:serialize>
      <sx:transform>
        <sx:content ref="data"/>
      </sx:tran