The source for console app examples may be found in the distribution under the samples
directory.
Before running the sample console applications, be sure to build the servingxml project.
Note that the examples will be deployed under the target/servingxml/samples
directory. See Getting Started for details.
This example shows a simple resources script that will map a CSV file to XML using default record mapping.
The input file is a CSV file.
Figure 1. Input flat file countries.csv
code,name
#ABW,ARUBA
ATF,"FRENCH SOUTHERN TERRITORIES, D.R. OF"
VUT,VANUATU
WLF,WALLIS & FUTUNA ISLANDS
This file has a number of properties.
The first row is a header with comma-separated column labels.
The second row is commented out.
The third row has a field value, enclosed in quotes, that contains an embedded comma.
The fifth row has a field value that contains a '&' symbol, which is a special character in XML, and must therefore be escaped in XML output as the entity '&'.
Shown below is the simplest possible resources script that will read this data and output it as XML.
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="countries">
<sx:serialize>
<sx:transform>
<sx:content ref="countries-doc"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="countries-doc" name="countries">
<sx:flatFileReader>
<sx:commentStarter value="#"/>
<sx:fieldDelimiter value=","/>
<sx:urlSource url="data/countries.csv"/>
</sx:flatFileReader>
</sx:recordContent>
</sx:resources>
This resources script produces the following output.
<?xml version="1.0" encoding="utf-8"?>
<countries>
<record>
<code>ATF</code>
<name>FRENCH SOUTHERN TERRITORIES, D.R. OF</name>
</record>
<record>
<code>VUT</code>
<name>VANUATU</name>
</record>
<record>
<code>WLF</code>
<name>WALLIS & FUTUNA ISLANDS</name>
</record>
</countries>
Note the following properties of the XML output.
The root element name comes from the
sx:recordContent
element's
name
attribute, if it has one, and if not
it defaults to
document
.
The record element name defaults to
record
.
Each field is mapped to an element, and the name of the element defaults to the column label.
An sx:recordMapping element positioned inside an sx:recordContent element allows for custom record mapping to XML. This is optional, since there is a default mapping that emits the canonical XML representation of records. But typically the XML you want is different from the canonical representation, you may want a field mapped to an attribute rather than an element, for example, or perhaps some additional literal elements around the field mappings.
Continuing with the previous example, suppose you want the following output.
<?xml version="1.0" encoding="utf-8"?>
<countries>
<country countryCode="ATF">
<countryName>FRENCH SOUTHERN TERRITORIES, D.R. OF</countryName>
</country>
<country countryCode="VUT">
<countryName>VANUATU</countryName>
</country>
<country countryCode="WLF">
<countryName>WALLIS & FUTUNA ISLANDS</countryName>
</country>
</countries>
The following resources script does the job.
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="countries">
<sx:serialize>
<sx:transform>
<sx:content ref="countries"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="countries">
<sx:flatFileReader>
<sx:urlSource url="data/countries.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="countriesToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="countriesFile">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType name="country">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="code"/>
<sx:delimitedField name="name"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="countriesToXmlMapping">
<countries>
<sx:onRecord>
<country>
<sx:fieldElementMap field="name" element="countryName"/>
<sx:fieldAttributeMap field="code" attribute="countryCode"/>
</country>
</sx:onRecord>
</countries>
</sx:recordMapping>
</sx:resources>
You can also take the default mappings for some fields and map others explicitly. For example,
<sx:recordMapping id="countries2xml">
<countries>
<sx:onRecord>
<country>
<sx:defaultFieldElementMap fields="*" except="code"/>
<sx:fieldAttributeMap field="code" attribute="countryCode"/>
</country>
</sx:onRecord>
</countries>
</sx:recordMapping>
Here, the sx:defaultFieldElementMap element maps all the fields in the record to elements with the same names, except the field name "code". This field is handled separately, with the sx:fieldAttributeMap element.
The example below shows how to convert a flat file to XML. Each row in the flat file is validated with Sun's multi schema validator (MSV), and if an error is encountered, the row is discarded, but processing continues. In addition, the output XML is also validated with Sun's MSV, and if that should fail, processing is stopped.
This time the input file has an invalid country code in the first record.
code,name
ATFX,"FRENCH SOUTHERN TERRITORIES, D.R. OF"
VUT,VANUATU
WLF,WALLIS & FUTUNA ISLANDS
The desired output is
<?xml version="1.0" encoding="utf-8"?>
<countries>
<country countryCode="ATF">
<countryName>FRENCH SOUTHERN TERRITORIES, D.R. OF</countryName>
</country>
<country countryCode="VUT">
<countryName>VANUATU</countryName>
</country>
<country countryCode="WLF">
<countryName>WALLIS & FUTUNA ISLANDS</countryName>
</country>
</countries>
The following resources script does the transformation.
Figure 2. Resources script resources-countries.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="countries">
<sx:serialize>
<sx:transform>
<sx:content ref="countries"/>
<msv:schemaValidator>
<sx:urlSource url="data/countries.xsd"/>
</msv:schemaValidator>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:flatFile id="countriesFile">
<sx:commentStarter value="#"/>
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType ref="country"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="country" name="country">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="code"/>
<sx:delimitedField name="name"/>
</sx:flatRecordType>
<sx:flatFile id="discardFile">
<sx:flatFileBody>
<sx:flatRecordType name="countryDiscard">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="message"/>
<sx:flatRecordType ref="country"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="countriesToXmlMapping">
<countries>
<sx:onRecord>
<country>
<sx:fieldElementMap field="name" element="countryName"/>
<sx:fieldAttributeMap field="code" attribute="countryCode"/>
</country>
</sx:onRecord>
</countries>
</sx:recordMapping>
<sx:recordContent id="countries">
<sx:recordStream>
<sx:flatFileReader>
<sx:urlSource url="data/countriesWithDiscards.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileReader>
<msv:recordValidator>
<sx:urlSource url="data/country-record.xsd"/>
</msv:recordValidator>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<sx:modifyRecord>
<sx:newField name="message" value="{$sx:message}"/>
</sx:modifyRecord>
<sx:flatFileWriter>
<sx:fileSink file="output/countryDiscards.csv"/>
<sx:flatFile ref="discardFile"/>
</sx:flatFileWriter>
</sx:discardHandler>
</sx:recordStream>
<sx:recordMapping ref="countriesToXmlMapping"/>
</sx:recordContent>
</sx:resources>
The schema to validate an individual country record is shown below.
Figure 3. XML Schema file country-record.xsd
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- This element's name matches the value of the name attribute in the sx:flatRecordType element. -->
<xs:element name="country" type="CountryType"/>
<xs:complexType name="CountryType">
<xs:sequence>
<xs:element name="code" type="CountryCode"/>
<xs:element name="name" type="xs:string"/>
</xs:sequence>
</xs:complexType>
<xs:simpleType name='CountryCode'>
<xs:restriction base='xs:string'>
<xs:length value='3' fixed='true'/>
</xs:restriction>
</xs:simpleType>
</xs:schema>
You run this example on the command line by entering
servingxml -r resources-countries.xml -o output/countries.xml countries
The following error message will be written to the log:
Error in record "country" on line 1. the length of the value is 4, but the required length is 3.
The remaining two rows will be written to the output file.
The example below shows how to convert a flat file with start/end record delimiters to XML.
The input file is shown below.
1231234
1 123
12 12
This file has the following structure.
The first record consists of a field of length 3 ("123") followed by a field of length 4 ("1234")
The second line consists of a field of length 1 ("1") followed by a field of length 3 ("123")
Each field has a maximum width, but the field may be terminated by a delimiter (here whitespace) before it attains that width.
The desired output is shown below.
<?xml version="1.0" encoding="utf-8"?>
<document>
<record>
<field1>123</field1>
<field2>1234</field2>
</record>
<record>
<field1>1</field1>
<field2>123</field2>
</record>
<record>
<field1>12</field1>
<field2>12</field2>
</record>
</document>
The following resources script does the transformation.
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="fieldHasMaxWidth">
<sx:serialize>
<sx:transform>
<sx:content ref="fieldHasMaxWidthDoc"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="fieldHasMaxWidthDoc" name="document">
<sx:flatFileReader>
<sx:urlSource url="data/fieldHasMaxWidth.txt"/>
<sx:flatFile ref="fieldHasMaxWidthFile"/>
</sx:flatFileReader>
</sx:recordContent>
<sx:flatFile id="fieldHasMaxWidthFile">
<sx:flatFileBody>
<sx:flatRecordType name="record">
<sx:fieldDelimiter>
<sx:whitespaceSeparator/>
</sx:fieldDelimiter>
<sx:delimitedField name="field1" maxWidth="3"/>
<sx:delimitedField name="field2" maxWidth="4"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-fieldHasMaxWidth.xml
-o output/fieldHasMaxWidth.xml fieldHasMaxWidth
The example below shows how to convert a flat file to XML with grouping levels.
The input file is a CSV file.
Figure 4. Input flat file tasks.csv
project_id, task_name, task_start, task_finish
4001,task1,01/01/2003,01/31/2003
4001,task2,02/01/2003,02/28/2003
4001,task3,03/01/2003,03/31/2003
4002,task1,04/01/2003,04/30/2003
4002,task2,05/01/2003,05/31/2003
The desired output is shown below.
Figure 5. Output XML file tasks.xml
<?xml version="1.0" encoding="utf-8" ?>
<Projects>
<Project projectID="4001">
<Tasks>
<Task name="task1" start="01/01/2003" finish="01/31/2003" />
<Task name="task2" start="02/01/2003" finish="02/28/2003" />
<Task name="task3" start="03/01/2003" finish="03/31/2003" />
</Tasks>
</Project>
<Project projectID="4002">
<Tasks>
<Task name="task1" start="04/01/2003" finish="04/30/2003" />
<Task name="task2" start="05/01/2003" finish="05/31/2003" />
</Tasks>
</Project>
</Projects>
The following resources script does the transformation.
Figure 6. Resources script resources-tasks.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="tasks">
<sx:serialize>
<sx:transform>
<sx:content ref="tasks"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="tasks">
<sx:flatFileReader>
<sx:urlSource url="data/tasks.csv"/>
<sx:flatFile ref="tasksFlatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="tasksToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="tasksFlatFile">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType name="task">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="project_id"/>
<sx:delimitedField name="task_name"/>
<sx:delimitedField name="task_start"/>
<sx:delimitedField name="task_finish"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="tasksToXmlMapping">
<Projects>
<sx:groupBy fields="project_id">
<Project>
<sx:fieldAttributeMap field="project_id" attribute="projectID"/>
<Tasks>
<sx:onRecord>
<Task>
<sx:fieldAttributeMap field="task_name" attribute="name"/>
<sx:fieldAttributeMap field="task_start" attribute="start"/>
<sx:fieldAttributeMap field="task_finish" attribute="finish"/>
</Task>
</sx:onRecord>
</Tasks>
</Project>
</sx:groupBy>
</Projects>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-tasks.xml -o output/tasks.xml tasks
The example below shows how to convert a flat file to XML with many grouping levels.
The input file is a CSV file.
Figure 7. Input flat file timesheets.csv
TimePeriod,Start,Finish,Resource,Task,ActualDate,Amount
1,2/9/2004,2/15/2004,Joe_100,1001,2/9/2004,8
1,2/9/2004,2/15/2004,Joe_100,1001,2/10/2004,4
1,2/9/2004,2/15/2004,Joe_100,1002,2/11/2004,8
1,2/9/2004,2/15/2004,Joe_100,1003,2/10/2004,4
1,2/9/2004,2/15/2004,Joe_100,1003,2/12/2004,8
1,2/9/2004,2/15/2004,Mark_101,1001,2/10/2004,8
1,2/9/2004,2/15/2004,Mark_101,1001,2/14/2004,8
1,2/9/2004,2/15/2004,Mark_101,1006,2/9/2004,8
1,2/9/2004,2/15/2004,Mark_101,1008,2/12/2004,4
1,2/9/2004,2/15/2004,Mark_101,1008,2/13/2004,4
2,2/16/2004,2/22/2004,Joe_100,1001,2/16/2004,8
2,2/16/2004,2/22/2004,Joe_100,1001,2/17/2004,4
2,2/16/2004,2/22/2004,Joe_100,1002,2/17/2004,4
The desired output is shown below.
Figure 8. Output XML file timesheets.xml
<?xml version="1.0" encoding="utf-8"?>
<TimePeriods>
<TimePeriod start="2/9/2004" finish="2/15/2004">
<Timesheets>
<Timesheet resource="Joe_100">
<TimesheetEntries>
<TimesheetEntry task="1001">
<DailyActuals>
<Actual actualDate="2/9/2004" amount="8"/>
<Actual actualDate="2/10/2004" amount="4"/>
</DailyActuals>
</TimesheetEntry>
<TimesheetEntry task="1002">
<DailyActuals>
<Actual actualDate="2/11/2004" amount="8"/>
</DailyActuals>
</TimesheetEntry>
<TimesheetEntry task="1003">
<DailyActuals>
<Actual actualDate="2/10/2004" amount="4"/>
<Actual actualDate="2/12/2004" amount="8"/>
</DailyActuals>
</TimesheetEntry>
</TimesheetEntries>
</Timesheet>
<Timesheet resource="Mark_101">
<TimesheetEntries>
<TimesheetEntry task="1001">
<DailyActuals>
<Actual actualDate="2/10/2004" amount="8"/>
<Actual actualDate="2/14/2004" amount="8"/>
</DailyActuals>
</TimesheetEntry>
<TimesheetEntry task="1006">
<DailyActuals>
<Actual actualDate="2/9/2004" amount="8"/>
</DailyActuals>
</TimesheetEntry>
<TimesheetEntry task="1008">
<DailyActuals>
<Actual actualDate="2/12/2004" amount="4"/>
<Actual actualDate="2/13/2004" amount="4"/>
</DailyActuals>
</TimesheetEntry>
</TimesheetEntries>
</Timesheet>
</Timesheets>
</TimePeriod>
<TimePeriod start="2/16/2004" finish="2/22/2004">
<Timesheets>
<Timesheet resource="Joe_100">
<TimesheetEntries>
<TimesheetEntry task="1001">
<DailyActuals>
<Actual actualDate="2/16/2004" amount="8"/>
<Actual actualDate="2/17/2004" amount="4"/>
</DailyActuals>
</TimesheetEntry>
<TimesheetEntry task="1002">
<DailyActuals>
<Actual actualDate="2/17/2004" amount="4"/>
</DailyActuals>
</TimesheetEntry>
</TimesheetEntries>
</Timesheet>
</Timesheets>
</TimePeriod>
</TimePeriods>
The following resources script does the transformation.
Figure 9. Resources script resources-timesheets.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="timesheets">
<sx:serialize>
<sx:transform>
<sx:content ref="timesheets"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="timesheets">
<sx:flatFileReader>
<sx:urlSource url="data/timesheets.csv"/>
<sx:flatFile ref="timesheetsFlatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="timesheetsToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="timesheetsFlatFile">
<sx:commentStarter value="#"/>
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType name="task">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="TimePeriod"/>
<sx:delimitedField name="Start"/>
<sx:delimitedField name="Finish"/>
<sx:delimitedField name="Resource"/>
<sx:delimitedField name="Task"/>
<sx:delimitedField name="ActualDate"/>
<sx:delimitedField name="Amount"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="timesheetsToXmlMapping">
<TimePeriods>
<sx:groupBy fields="Start">
<TimePeriod>
<sx:fieldAttributeMap field="Start" attribute="start"/>
<sx:fieldAttributeMap field="Finish" attribute="finish"/>
<Timesheets>
<sx:groupBy fields="Start Resource">
<Timesheet>
<sx:fieldAttributeMap field="Resource" attribute="resource"/>
<TimesheetEntries>
<sx:groupBy fields="Start Resource Task">
<TimesheetEntry>
<sx:fieldAttributeMap field="Task" attribute="task"/>
<DailyActuals>
<sx:onRecord>
<Actual>
<sx:fieldAttributeMap field="ActualDate" attribute="actualDate"/>
<sx:fieldAttributeMap field="Amount" attribute="amount"/>
</Actual>
</sx:onRecord>
</DailyActuals>
</TimesheetEntry>
</sx:groupBy>
</TimesheetEntries>
</Timesheet>
</sx:groupBy>
</Timesheets>
</TimePeriod>
</sx:groupBy>
</TimePeriods>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-timesheets.xml -o output/timesheets.xml timesheets
The example below shows how to convert a flat file with multi-valued fields to XML.
The input file is a pipe delimited flat file with semicolon sub-delimiters.
Figure 10. Input flat file employees.txt
father_name|mother_name|children
Matthew|Sarah|
Scott||Damian;Janet;Paul
The desired output is shown below.
Figure 11. Output XML file multivalued.xml
<?xml version="1.0" encoding="UTF-8"?>
<eg:families xmlns:eg="http://examples.com/">
<eg:family>
<eg:father-name>Matthew</eg:father-name>
<eg:mother-name>Sarah</eg:mother-name>
<eg:children/>
</eg:family>
<eg:family>
<eg:father-name>Scott</eg:father-name>
<eg:mother-name/>
<eg:children>
<eg:child>Damian</eg:child>
<eg:child>Janet</eg:child>
<eg:child>Paul</eg:child>
</eg:children>
</eg:family>
</eg:families>
The following resources script does the transformation.
Figure 12. Resources script resources-multivalued.xml
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:eg="http://examples.com/">
<sx:service id="families">
<sx:serialize>
<sx:transform>
<sx:content ref="families"/>
<!-- sx:removeEmptyElements elements="eg:children"/ -->
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="families">
<sx:flatFileReader>
<sx:flatFile ref="family-records"/>
</sx:flatFileReader>
<sx:recordMapping ref="families-to-xml-mapping"/>
</sx:recordContent>
<sx:flatFile id="family-records">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType name="family">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="father_name"/>
<sx:delimitedField name="mother_name"/>
<sx:delimitedField name="children">
<sx:subfieldDelimiter value=";"/>
</sx:delimitedField>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="families-to-xml-mapping">
<eg:families xmlns:eg="http://examples.com/">
<sx:onRecord>
<eg:family>
<sx:fieldElementMap field="father_name" element="eg:father-name"/>
<sx:fieldElementMap field="mother_name" element="eg:mother-name"/>
<eg:children>
<sx:fieldElementSequenceMap field="children" element="eg:child"/>
</eg:children>
</eg:family>
</sx:onRecord>
</eg:families>
</sx:recordMapping>
</sx:resources>
Note that Matthew and Sarah have no children. If you wanted to remove the empty
myns:children
element, uncomment the
sx:removeEmptyElements
element in the
sx:transform
block of the resources script,
You can run this example on the command line by entering
servingxml -r resources-family_data.xml
-i data/family_data.txt -o output/family_data.xml families
The example below shows how to convert a flat file to XML using search and replace with regular expressions in field mappings.
The input file is a carot (^) delimited flat file with a header.
Figure 13. Input flat file ars.dat
ProjectID^Project_Name^Description^PLAN_IT_RESC_DAYS^PLAN_IT_COST^ANNUAL_EBIT^ACTUAL_NPV^ACTUAL_IRR
1234-00-0005^XOG new project insert^This is to test and verify the XOG'ing of new Project data from the ARS database.^430^59,627^100,351^3^9.2
Note the following features of this data.
The first row is a header row.
The fourth and fifth fields contain embedded commas in numbers, which we want to strip out.
The desired output is shown below.
Figure 14. Output XML file ars.xml
<?xml version="1.0" encoding="utf-8"?>
<NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Header action="write" externalSource="NIKU" objectType="project" version="6.0.11"/>
<Projects>
<Project name="XOG new project insert" projectID="1234-00-0005" description="This is to test and verify the XOG'ing of new Project data from the ARS database.">
<CustomInformation>
<PLAN_IT_RESOURCE_DAYS>430</PLAN_IT_RESOURCE_DAYS>
<PLAN_IT_COST>59627</PLAN_IT_COST>
<ANNUAL_EBIT>100351</ANNUAL_EBIT>
<ACTUAL_NPV>3</ACTUAL_NPV>
<ACTUAL_IRR>9.2</ACTUAL_IRR>
</CustomInformation>
<General addedBy="XOG" addedDate="03/26/2005 8:10:09 PM"/>
</Project>
</Projects>
</NikuDataBus>
The following resources script does the transformation.
Figure 15. Resources script resources-ars.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="ars">
<sx:serialize>
<sx:transform>
<sx:content ref="ars"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="ars">
<sx:flatFileReader>
<sx:flatFile ref="arsFlatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="arsToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="arsFlatFile">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType name="detail">
<sx:fieldDelimiter value="^"/>
<sx:delimitedField name="ProjectID"/>
<sx:delimitedField name="Project_Name"/>
<sx:delimitedField name="Description"/>
<sx:delimitedField name="PLAN_IT_RESC_DAYS"/>
<sx:delimitedField name="PLAN_IT_COST"/>
<sx:delimitedField name="ANNUAL_EBIT"/>
<sx:delimitedField name="ACTUAL_NPV"/>
<sx:delimitedField name="ACTUAL_IRR"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="arsToXmlMapping">
<NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Header version="6.0.11" action="write" objectType="project" externalSource="NIKU"/>
<Projects>
<sx:groupBy fields="ProjectID">
<Project>
<sx:fieldAttributeMap field="Project_Name" attribute="name"/>
<sx:fieldAttributeMap field="ProjectID" attribute="projectID"/>
<sx:fieldAttributeMap field="Description" attribute="description"/>
<sx:onRecord>
<CustomInformation>
<sx:fieldElementMap field="PLAN_IT_RESC_DAYS" element="PLAN_IT_RESOURCE_DAYS"/>
<sx:elementMap element="PLAN_IT_COST">
<sx:findAndReplace searchFor ="," replaceWith ="">
<sx:toString value="{PLAN_IT_COST}"/>
</sx:findAndReplace>
</sx:elementMap>
<sx:elementMap element="ANNUAL_EBIT">
<sx:findAndReplace searchFor ="," replaceWith ="">
<sx:toString value="{ANNUAL_EBIT}"/>
</sx:findAndReplace>
</sx:elementMap>
<sx:fieldElementMap field="ACTUAL_NPV" element="ACTUAL_NPV"/>
<sx:fieldElementMap field="ACTUAL_IRR" element="ACTUAL_IRR"/>
</CustomInformation>
<General addedBy="XOG">
<sx:fieldAttributeMap attribute="addedDate">
<sx:formatDateTime format="MM/dd/yyyy h:mm:ss a">
<sx:currentDateTime/>
</sx:formatDateTime>
</sx:fieldAttributeMap>
</General>
</sx:onRecord>
</Project>
</sx:groupBy>
</Projects>
</NikuDataBus>
</sx:recordMapping>
</sx:resources>
Note the following points about this script.
The sx:currentDateTime element evaluates to the current date with a lexical representation as defined for xs:dateTime of XML Schema Part 2: Datatypes .
The sx:formatDateTime formats the XML Schema lexical representation of a date using a format string which must must follow the syntax specified for the JDK SimpleDateFormat class..
You can run this example on the command line by entering
servingxml -r resources-ars.xml -i data/ars.txt -o output/ars.xml ars
The example below shows how to convert a flat file with tab delimiters to XML.
The input file is a tab (0x09) delimited flat file with a header.
Figure 16. Input flat file tab_delimited_employees.txt
employee-no employee-name dept salary
00000001 Smith, Matthew sales 150,000.00
00000002 Brown, Sarah sales 89,000.00
00000003 Oberc, Scott finance 110,000.00
00000004 Scott, Colette sales 75,000.00
The desired output is shown below.
Figure 17. Output XML file employees.xml
<?xml version="1.0" encoding="utf-8"?>
<employees>
<employee>
<employee-no>00000001</employee-no>
<employee-name>Smith, Matthew</employee-name>
<department>sales</department>
<salary>150,000.00</salary>
</employee>
<employee>
<employee-no>00000002</employee-no>
<employee-name>Brown, Sarah</employee-name>
<department>sales</department>
<salary>89,000.00</salary>
</employee>
<employee>
<employee-no>00000003</employee-no>
<employee-name>Oberc, Scott</employee-name>
<department>finance</department>
<salary>110,000.00</salary>
</employee>
<employee>
<employee-no>00000004</employee-no>
<employee-name>Scott, Colette</employee-name>
<department>sales</department>
<salary>75,000.00</salary>
</employee>
</employees>
The following resources script does the transformation.
Figure 18. Resources script resources-tab_delimited_employees.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="employees">
<sx:serialize>
<sx:transform>
<sx:content ref="employee-data"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="employee-data" name="employees">
<sx:flatFileReader>
<sx:urlSource url="data/tab_delimited_employees.txt"/>
<sx:flatFile ref="employee-file"/>
</sx:flatFileReader>
</sx:recordContent>
<sx:flatFile id="employee-file">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:fieldDelimiter value="\t"/>
<!-- Another way of specifying a horizontal tab character -->
<!-- <sx:fieldDelimiter value="	"/> -->
<sx:flatRecordType name="employee">
<sx:delimitedField name="employee-no"/>
<sx:delimitedField name="employee-name"/>
<sx:delimitedField name="department"/>
<sx:delimitedField name="salary"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
Note the following points about this script.
The tab delimiter may be specified either with the
\t
symbol, or, alternatively, with the numerical entity reference
	
numerical entity
You can run this example on the command line by entering
servingxml -o output/employees.xml
-r resources-tab_delimited_employees.xml employees
The example below shows how to convert a pipe delimited flat file with multiple record formats to XM with grouping.
The input file is a pipe delimited file.
Figure 19. Input flat file exotics.txt
SWAP|1234567|FLOAT|PAY|CAD|BA|2010/04/28|10000000.00
Cap|1234567||CAD|SELL|2010/04/28|0.03
SWAP|1234567|FIX|REC|CAD
Cap|1234568||CAD|SELL|2010/04/28|0.05
The desired output is shown below.
Figure 20. Output XML file exotics.xml
<exotics>
<trade tradeId="1234567">
<option>SWAP</option>
<swapLeg style="FLOAT">
<side>PAY</side>
<maturityDate original="2010/04/28">2001/04/29</maturityDate>
<currency>CAD</currency>
<index>BA</index>
<notional>10000000.00</notional>
</swapLeg>
<capSell>
<buySell>SELL</buySell>
<maturityDate original="2010/04/28">2001/04/28</maturityDate>
<strike>0.03</strike>
</capSell>
<swapLeg style="FIX">
<side>REC</side>
</swapLeg>
</trade>
<trade tradeId="1234568">
<option>Cap</option>
<capBuy>
<buySell>BUY</buySell>
<maturityDate original="2010/04/28">2001/04/28</maturityDate>
<strike>0.05</strike>
</capBuy>
</trade>
</exotics>
The following resources script does the transformation.
Figure 21. Resources script resources-exotics.xml
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="exotics">
<sx:serialize>
<sx:transform>
<sx:content ref="exotics"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="exotics">
<sx:flatFileReader>
<sx:fileSource file="data/exotics.txt"/>
<sx:flatFile ref="exoticsFlatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="exoticsToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="exoticsFlatFile">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="recordType"/>
<sx:delimitedField name="tradeId"/>
<sx:delimitedField name="style"/>
<sx:when test="recordType='Cap'">
<sx:flatRecordType name='Cap'>
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="recordType"/>
<sx:delimitedField name="tradeId"/>
<sx:delimitedField name="placeholder1"/>
<sx:delimitedField name="currency"/>
<sx:delimitedField name="buySell"/>
<sx:delimitedField name="maturityDate"/>
<sx:delimitedField name="strike"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="recordType='SWAP' and style='FLOAT'">
<sx:flatRecordType name='SwapFloatingLeg'>
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="recordType"/>
<sx:delimitedField name="tradeId"/>
<sx:delimitedField name="style"/>
<sx:delimitedField name="side"/>
<sx:delimitedField name="currency"/>
<sx:delimitedField name="index"/>
<sx:delimitedField name="maturityDate"/>
<sx:delimitedField name="notional"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="recordType='SWAP' and style='FIX'">
<sx:flatRecordType name='SwapFixedLeg'>
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="recordType"/>
<sx:delimitedField name="tradeId"/>
<sx:delimitedField name="style"/>
<sx:delimitedField name="side"/>
<sx:delimitedField name="placeholder1"/>
<sx:delimitedField name="currency"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="exoticsToXmlMapping">
<exotics>
<sx:groupBy fields="tradeId">
<trade><option>
<sx:choose>
<sx:when test="recordType='Cap' or (recordType='SWAP' and style='FLOAT')">
<sx:toString value="{recordType}"/>
</sx:when>
</sx:choose></option>
<sx:fieldAttributeMap field="tradeId" attribute="tradeId"/>
<sx:onRecord recordType="Cap">
<sx:choose>
<sx:when test="buySell='BUY'">
<capBuy>
<sx:fieldElementMap field="buySell" element="buySell"/>
<sx:elementMap element="maturityDate">
<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
<sx:choose>
<sx:when test="maturityDate='2010/04/28'">
<sx:toString value="2001/04/28"/>
</sx:when>
</sx:choose>
</sx:elementMap>
<sx:fieldElementMap field="strike" element="strike"/>
</capBuy>
</sx:when>
<sx:otherwise>
<capSell>
<sx:fieldElementMap field="buySell" element="buySell"/>
<sx:elementMap element="maturityDate">
<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
<sx:choose>
<sx:when test="maturityDate='2010/04/28'">
<sx:toString value="2001/04/28"/>
</sx:when>
</sx:choose>
</sx:elementMap>
<sx:fieldElementMap field="strike" element="strike"/>
</capSell>
</sx:otherwise>
</sx:choose>
</sx:onRecord>
<sx:onRecord recordType="SwapFloatingLeg">
<swapLeg>
<sx:fieldAttributeMap field="style" attribute="style"/>
<sx:fieldElementMap field="side" element="side"/>
<maturityDate>
<sx:fieldAttributeMap field="maturityDate" attribute="original"/>
<sx:choose>
<sx:when test="maturityDate='2010/04/28'">
<sx:toString value="2001/04/29"/>
</sx:when>
</sx:choose>
</maturityDate>
<sx:fieldElementMap field="currency" element="currency"/>
<sx:fieldElementMap field="index" element="index"/>
<sx:fieldElementMap field="notional" element="notional"/>
</swapLeg>
</sx:onRecord>
<sx:onRecord recordType="SwapFixedLeg">
<swapLeg>
<sx:fieldAttributeMap field="style" attribute="style"/>
<sx:fieldElementMap field="side" element="side"/>
</swapLeg>
</sx:onRecord>
</trade>
</sx:groupBy>
</exotics>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-exotics.xml -o output/exotics.xml exotics
The example below shows how to convert a flat file to XML by applying an XSLT transform to the canonical XML representation of each record.
The input file is shown below.
262025218|John|Smith|657827168||1|Main Street|London|45879|56|||||
262091012|Jane|Bell||000000000000|105|Penny Lane|Oxford|12345|56|1|Main Street|London|45879|56
Each line has an id, followed by a first and last name, followed by a customer address, and optionally a billing address. If the house number and street of the billing address are empty, the rule is that the entire billing address be omitted from the XML output.
The desired output is shown below.
<?xml version="1.0" encoding="utf-8"?>
<mns:Message xmlns:mns="www.abc.com/abc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="www.abc.com/abc abc_v_1.1.xsd ">
<wf_id>262025218</wf_id>
<timeStamp>2006-02-11T22:18:21</timeStamp>
<MessageData>
<addCustomerRequest>
<foreName>John</foreName>
<surName>Smith</surName>
<custAddress>
<houseNumber>1</houseNumber>
<street>Main Street</street>
<town>London</town>
<postCode>45879</postCode>
<country>56</country>
</custAddress>
</addCustomerRequest>
</MessageData>
<wf_id>262091012</wf_id>
<timeStamp>2006-02-11T22:18:21</timeStamp>
<MessageData>
<addCustomerRequest>
<foreName>Jane</foreName>
<surName>Bell</surName>
<custAddress>
<houseNumber>105</houseNumber>
<street>Penny Lane</street>
<town>Oxford</town>
<postCode>12345</postCode>
<country>56</country>
</custAddress>
<billAddress>
<houseNumber>1</houseNumber>
<street>Main Street</street>
<town>London</town>
<postCode>45879</postCode>
<country>56</country>
</billAddress>
</addCustomerRequest>
</MessageData>
</mns:Message>
The following resources script does the transformation.
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="addCustomer">
<sx:serialize>
<sx:xsltSerializer>
<sx:outputProperty name="indent" value="yes"/>
</sx:xsltSerializer>
<sx:transform>
<sx:content ref="addCustomer"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:flatFile id="customerFile">
<sx:flatFileBody>
<sx:flatRecordType name="customer">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="rowid"/>
<sx:delimitedField name="forename"/>
<sx:delimitedField name="surname"/>
<sx:delimitedField name="vatnumber"/>
<sx:delimitedField name="ninumber"/>
<sx:delimitedField name="custaddr_house_nr"/>
<sx:delimitedField name="custaddr_street"/>
<sx:delimitedField name="custaddr_town"/>
<sx:delimitedField name="custaddr_postcode"/>
<sx:delimitedField name="custaddr_country"/>
<sx:delimitedField name="billaddr_house_nr"/>
<sx:delimitedField name="billaddr_street"/>
<sx:delimitedField name="billaddr_town"/>
<sx:delimitedField name="billaddr_postcode"/>
<sx:delimitedField name="billaddr_country"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordContent id="addCustomer">
<sx:flatFileReader>
<sx:fileSource file="data/optionalElements.txt"/>
<sx:flatFile ref="customerFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="addCustomerToXmlMapping"/>
</sx:recordContent>
<sx:recordMapping id="addCustomerToXmlMapping">
<mns:Message xmlns:mns="www.abc.com/abc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="www.abc.com/abc abc_v_1.1.xsd">
<sx:onRecord>
<sx:fieldElementMap field="rowid" element="wf_id"/>
<timeStamp>
<sx:formatDateTime format = "yyyy-MM-dd'T'HH:mm:ss">
<sx:currentDateTime/>
</sx:formatDateTime>
</timeStamp>
<sx:transformRecord>
<sx:xslt>
<xsl:transform version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="customer">
<MessageData>
<addCustomerRequest>
<xsl:element name="foreName">
<xsl:value-of select="forename"/>
</xsl:element>
<xsl:element name="surName">
<xsl:value-of select="surname"/>
</xsl:element>
<custAddress>
<xsl:element name="houseNumber">
<xsl:value-of select="custaddr_house_nr"/>
</xsl:element>
<xsl:element name="street">
<xsl:value-of select="custaddr_street"/>
</xsl:element>
<xsl:element name="town">
<xsl:value-of select="custaddr_town"/>
</xsl:element>
<xsl:element name="postCode">
<xsl:value-of select="custaddr_postcode"/>
</xsl:element>
<xsl:element name="country">
<xsl:value-of select="custaddr_country"/>
</xsl:element>
</custAddress>
<xsl:if test="string(billaddr_house_nr) and string(billaddr_street)">
<billAddress>
<xsl:element name="houseNumber">
<xsl:value-of select="billaddr_house_nr"/>
</xsl:element>
<xsl:element name="street">
<xsl:value-of select="billaddr_street"/>
</xsl:element>
<xsl:element name="town">
<xsl:value-of select="billaddr_town"/>
</xsl:element>
<xsl:element name="postCode">
<xsl:value-of select="billaddr_postcode"/>
</xsl:element>
<xsl:element name="country">
<xsl:value-of select="billaddr_country"/>
</xsl:element>
</billAddress>
</xsl:if>
</addCustomerRequest>
</MessageData>
</xsl:template>
</xsl:transform>
</sx:xslt>
</sx:transformRecord>
</sx:onRecord>
</mns:Message>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-optionalElements.xml
-o output/optionalElements.xml optionalElements
The example below shows how to convert a Java properties file to XML.
The input file is a Java properties file.
Figure 22. Input property file messages.properties
parser.ope.2=')' or '-[' or '+[' or '&[' is expected.
parser.factor.5=A back reference or an anchor or a lookahead or a lookbehind \
is expected in a conditional pattern.
parser.next.2='?' is not expected. '(?:' or '(?=' or '(?!' or '(?<' or '(?#' or '(?>'?
#parser.next.3='(?<=' or '(?<!' is expected.
This file has a number of features.
The entries are key-value pairs separated by the '=' symbol.
The value in the first line contains an '&' symbol, which is a special character in XML, and will be escaped in the output file.
The second line ends with a continuation character.
The value in the third line contains an '=' symbol.
The last line starts with a comment symbol.
The desired output is shown below.
Figure 23. Output XML file messages.xml
<?xml version="1.0" encoding="utf-8"?>
<messages>
<parser.ope.2>')' or '-[' or '+[' or '&[' is expected.</parser.ope.2>
<parser.factor.5>A back reference or an anchor or a lookahead or a lookbehind is expected in a conditional pattern.</parser.factor.5>
<parser.next.2>'?' is not expected. '(?:' or '(?=' or '(?!' or '(?<' or '(?#' or '(?>'?</parser.next.2>
</messages>
The following resources script does the transformation.
Figure 24. Resources script resources-messages.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="messages">
<sx:serialize>
<sx:transform>
<sx:content ref="messages"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="messages">
<sx:flatFileReader>
<sx:urlSource url="data/messages.properties"/>
<sx:flatFile>
<sx:recordDelimiter continuation="\" value="\r\n"/>
<sx:recordDelimiter continuation="\" value="\n"/>
<sx:commentStarter value="#"/>
<sx:flatFileBody>
<sx:flatRecordType name="property">
<sx:delimitedField name="name">
<sx:fieldDelimiter value="="/>
</sx:delimitedField>
<sx:delimitedField name="value"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:flatFileReader>
<sx:recordMapping>
<messages>
<sx:onRecord>
<sx:fieldElementMap field="value" element="{name}"/>
</sx:onRecord>
</messages>
</sx:recordMapping>
</sx:recordContent>
</sx:resources>
Note the following points about this script.
The '=' delimiter is associated only with the
name
field,
not the
value
field.
There is no delimiter specified for the
value
field, nor for the record as a whole.
This means that the
value
field will contain all the text between the first '=' symbol and the
record delimiter, including any more '=' symbols.
You can run this example on the command line by entering
servingxml -r resources-messages.xml -o output/messages.xml
messages
This example shows one way to write a resources script that will convert a Java properties file to XML with nested tags.
The input file is a Java properties file.
Figure 25. Input property file multipart-keys.properties
image.text.info1=hello world 1
image.text.info2=hello \
world 2
image.text.info3=hello world 3
This file has a number of features.
The entries are key-value pairs separated by an '=' symbol.
Each name consists of three parts separated by a '.' symbol
The second line ends with a continuation character.
The desired output is shown below.
Figure 26. Output XML file nested-tags1.xml
<?xml version="1.0" encoding="utf-8"?>
<messages>
<image>
<text>
<info1>hello world 1</info1>
<info2>hello world 2</info2>
<info3>hello world 3</info3>
</text>
</image>
</messages>
Note that each level of nesting corresponds to separate part of the key.
The following resources script does the transformation.
Figure 27. Resources script resources-props2nested1.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="messages">
<sx:serialize>
<sx:transform>
<sx:content ref="messages"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="messages">
<sx:flatFileReader>
<sx:flatFile>
<sx:commentStarter value="#"/>
<sx:commentStarter value="//"/>
<sx:recordDelimiter continuation="\" value="\r\n"/>
<sx:recordDelimiter continuation="\" value="\n"/>
<sx:flatFileBody>
<sx:flatRecordType name="property">
<sx:delimitedField name="key1">
<sx:fieldDelimiter value="."/>
</sx:delimitedField>
<sx:delimitedField name="key2">
<sx:fieldDelimiter value="."/>
</sx:delimitedField>
<sx:delimitedField name="key3">
<sx:fieldDelimiter value="="/>
</sx:delimitedField>
<sx:delimitedField name="value"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:flatFileReader>
<sx:recordMapping>
<messages>
<sx:groupBy fields="key1">
<sx:elementMap element="{key1}">
<sx:groupBy fields="key2 key2">
<sx:elementMap element="{key2}">
<sx:onRecord>
<sx:fieldElementMap field="value" element="{key3}"/>
</sx:onRecord>
</sx:elementMap>
</sx:groupBy>
</sx:elementMap>
</sx:groupBy>
</messages>
</sx:recordMapping>
</sx:recordContent>
</sx:resources>
This script relies on reading each record as four fields.
key1
is terminated with a '.' delimiter
key2
is terminated with a '.' delimiter
key3
is terminated with an '=' delimiter
value
is terminated with a record delimiter
You can run this example on the command line by entering
servingxml -i data/multipart-keys.properties
-o output/nested-tags1.xml -r resources-props2nested1.xml messages
This example shows another way to write a resources script that will convert a Java properties file to XML with nested tags.
The properties input file and the desired XML output file are the same as the previous example.
This time the resources script is written as follows.
Figure 28. Resources script resources-props2nested2.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="messages">
<sx:serialize>
<sx:transform>
<sx:content ref="messages"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="messages">
<sx:flatFileReader>
<sx:flatFile>
<sx:commentStarter value="#"/>
<sx:commentStarter value="//"/>
<sx:recordDelimiter continuation="\" value="\r\n"/>
<sx:recordDelimiter continuation="\" value="\n"/>
<sx:flatFileBody>
<sx:flatRecordType name="property">
<sx:delimitedField name="name">
<sx:fieldDelimiter value="="/>
<sx:subfieldDelimiter value="."/>
</sx:delimitedField>
<sx:delimitedField name="value"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:flatFileReader>
<sx:recordMapping>
<messages>
<sx:groupBy fields="name[1]">
<sx:elementMap element="{name[1]}">
<sx:groupBy fields="name[1] name[2]">
<sx:elementMap element="{name[2]}">
<sx:onRecord>
<sx:fieldElementMap field="value" element="{name[3]}"/>
</sx:onRecord>
</sx:elementMap>
</sx:groupBy>
</sx:elementMap>
</sx:groupBy>
</messages>
</sx:recordMapping>
</sx:recordContent>
</sx:resources>
This script relies on reading each record as two fields, a multi-valued name field and a value field.
name
has three values that may be indexed as 1...3
value
has one value
You can run this example on the command line by entering
servingxml -i data/multipart-keys.properties
-o output/nested-tags2.xml -r resources-props2nested2.xml messages
The example below shows how to reorder a group of records before mapping them to XML.
The input file is shown below.
HEADER
S 001
S 002
B 001
S 003
B 002
TRAILER
Each 'B' record represents a summary of the 'S' records since the last summary..
Suppose you want to turn it into the following:.
<?xml version="1.0" encoding="utf-8"?>
<DOCUMENT>
<HEADER/>
<B value="001">
<S value="001"/>
<S value="002"/>
</B>
<B value="003">
<S value="003"/>
</B>
<TRAILER/>
</DOCUMENT>
If the summary records came before the details, rather than after, the mapping would be straightforward: see the examples with sx:innerGroup grouping elements. This suggests reordering the records as they come in.
The following resources script does the transformation.
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="multipleSummaries">
<sx:serialize>
<sx:transform>
<sx:content ref="multipleSummariesContent"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="multipleSummariesContent">
<sx:flatFileReader>
<sx:urlSource url="data/multipleSummaries.txt"/>
<sx:flatFile ref="multipleSummariesFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="multipleSummariesMapping"/>
</sx:recordContent>
<sx:recordMapping id="multipleSummariesMapping">
<DOCUMENT>
<sx:groupChoice>
<sx:innerGroup startTest="sx:current/Header"
endTest="sx:previous/Header">
<sx:onRecord>
<HEADER/>
</sx:onRecord>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current/Trailer"
endTest="sx:previous/Trailer">
<sx:onRecord>
<TRAILER/>
</sx:onRecord>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current/S"
endTest="sx:previous/B">
<!-- reorder the records in this group - move the B records to the front -->
<sx:reorderRecords recordTypes="B S"/>
<B>
<sx:fieldAttributeMap field="value" attribute="value"/>
<sx:onRecord recordType='S'>
<S>
<sx:fieldAttributeMap field="value" attribute="value"/>
</S>
</sx:onRecord>
</B>
</sx:innerGroup>
</sx:groupChoice>
</DOCUMENT>
</sx:recordMapping>
<sx:flatFile id="multipleSummariesFile">
<sx:flatFileBody>
<sx:fieldDelimiter>
<sx:whitespaceSeparator/>
</sx:fieldDelimiter>
<sx:flatRecordTypeChoice>
<sx:delimitedField name="tag"/>
<sx:when test="tag='HEADER'">
<sx:flatRecordType name="Header">
<sx:delimitedField name="tag"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='TRAILER'">
<sx:flatRecordType name="Trailer">
<sx:delimitedField name="tag"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='S'">
<sx:flatRecordType name="S">
<sx:delimitedField name="tag"/>
<sx:delimitedField name="value"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='B'">
<sx:flatRecordType name="B">
<sx:delimitedField name="tag"/>
<sx:delimitedField name="value"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-multipleSummaries.xml
-o output/multipleSummaries.xml multipleSummaries
Suppose you have the following delimited file of students, their course grades, and their addresses.
The file has the following layout.
| name | the field delimiter ^ defines the end of the field |
| subject-grade | repeating group, repeating segments are separated by the repeat delimiter ~, the segment delimiter | defines the end of the group |
| year-born | the field delimiter ^ defines the end of the field |
| favorite-color | the field delimiter ^ defines the end of the field |
| address | repeating group, repeating segments are separated by the repeat delimiter ~, the segment delimiter | defines the end of the group |
You want the XML output to look as follows.
Figure 30. Output XML file students-fix.xml
<?xml version="1.0" encoding="utf-8"?>
<StudentGrades>
<StudentGrade>
<Name>JANE</Name>
<SubjectGrade>
<Subject>ENGL</Subject>
<Grade>C-</Grade>
</SubjectGrade>
<SubjectGrade>
<Subject>MATH</Subject>
<Grade>A+</Grade>
</SubjectGrade>
<YearBorn>1972</YearBorn>
<FavoriteColor>BLUE</FavoriteColor>
<Address>
<City>CHICAGO</City>
<State>IL</State>
</Address>
<Address>
<City>ATLANTA</City>
<State>GA</State>
</Address>
</StudentGrade>
</StudentGrades>
The required resources script is
Figure 31. Resources script resources-students-delim.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="students">
<sx:serialize>
<sx:transform>
<sx:content ref="students-content"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="students-content">
<sx:flatFileReader>
<sx:flatFile ref="students-file"/>
</sx:flatFileReader>
<sx:recordMapping ref="students-mapping"/>
</sx:recordContent>
<sx:flatFile id="students-file">
<sx:flatFileBody>
<sx:flatRecordType name="student">
<sx:fieldDelimiter value="^"/>
<sx:repeatDelimiter value="~"/>
<sx:segmentDelimiter value="|"/>
<sx:delimitedField name="name"/>
<sx:repeatingGroup name="grades">
<sx:flatRecordType name="subject-grade">
<sx:fieldDelimiter value="^"/>
<sx:delimitedField name="subject"/>
<sx:delimitedField name="grade"/>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:delimitedField name="year-born"/>
<sx:delimitedField name="favorite-color"/>
<sx:repeatingGroup name="addresses">
<sx:flatRecordType name="address">
<sx:fieldDelimiter value="^"/>
<sx:delimitedField name="city"/>
<sx:delimitedField name="state"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="students-mapping">
<StudentGrades>
<sx:onRecord>
<StudentGrade>
<sx:fieldElementMap field="name" element="Name"/>
<sx:subrecordMapping repeatingGroup="grades">
<sx:onRecord>
<SubjectGrade>
<sx:fieldElementMap field="subject" element="Subject"/>
<sx:fieldElementMap field="grade" element="Grade"/>
</SubjectGrade>
</sx:onRecord>
</sx:subrecordMapping>
<sx:fieldElementMap field="year-born" element="YearBorn"/>
<sx:fieldElementMap field="favorite-color" element="FavoriteColor"/>
<sx:subrecordMapping repeatingGroup="addresses">
<sx:onRecord>
<Address>
<sx:fieldElementMap field="city" element="City"/>
<sx:fieldElementMap field="state" element="State"/>
</Address>
</sx:onRecord>
</sx:subrecordMapping>
</StudentGrade>
</sx:onRecord>
</StudentGrades>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-students-delim.xml -i data/students-delim.txt -o output/students-delim.xml
students
Suppose you have the following data file.
Figure 32. Input flat file invoice.txt
12|2007-02-02|DocType|}
SndId|SndName|SndAddress|SndZip|}
RecId|RecName|RecAddress|RecZip|}
~
1|Item 1|2|Kg|150|300|}
2|Item 2|2|Kg|350|700|}
3|Item 3|1|$|50|50|}
4|Item 4|10|Unt|30|100|}
~
1|Ref 1|Doc 1|Text|}
2|Ref 2|Doc 2|Text|}
~
1|Disc 1|Item 1|}
2|Disc 2|Item 2|}
3|Disc 3|Item 3|}
~
Code a|Value 1|}
Code z|Value 3|}
Code x|Value 2|}
Code w|Value 2|}
Code y|Value 1|}
~
!
The file has
n
sections separated by the "~" character. The first section has several non repeating records separated by a "}".
But after that, there are 4 repeating sections separated by the same "~" as above. Each repeating section starts where the previous one ends.
The character "!" as the first character in the last line indicates the end of the data.
You want the XML output to look as follows.
Figure 33. Output XML file invoice.xml
<?xml version="1.0" encoding="utf-8"?>
<InvoiceDocument>
<Header>
<DocId>
<Id>12</Id>
<DocDate>2007-02-02</DocDate>
<DocumentType>DocType</DocumentType>
</DocId>
<Sender>
<SenderPassport>SndId</SenderPassport>
<SenderFullName>SndName</SenderFullName>
<SenderAddress>SndAddress</SenderAddress>
<SenderZip>SndZip</SenderZip>
</Sender>
<Recipient>
<RecipientPassport>RecId</RecipientPassport>
<RecipientFullName>RecName</RecipientFullName>
<RecipientAddress>RecAddress</RecipientAddress>
<RecipientZip>RecZip</RecipientZip>
</Recipient>
</Header>
<Detail>
<Item>1</Item>
<Description>Item 1</Description>
<Quantity>2</Quantity>
<Unit>Kg</Unit>
<ItemPrice>150</ItemPrice>
<TotalPrice>300</TotalPrice>
</Detail>
<Detail>
<Item>2</Item>
<Description>Item 2</Description>
<Quantity>2</Quantity>
<Unit>Kg</Unit>
<ItemPrice>350</ItemPrice>
<TotalPrice>700</TotalPrice>
</Detail>
<Detail>
<Item>3</Item>
<Description>Item 3</Description>
<Quantity>1</Quantity>
<Unit>$</Unit>
<ItemPrice>50</ItemPrice>
<TotalPrice>50</TotalPrice>
</Detail>
<Detail>
<Item>4</Item>
<Description>Item 4</Description>
<Quantity>10</Quantity>
<Unit>Unt</Unit>
<ItemPrice>30</ItemPrice>
<TotalPrice>100</TotalPrice>
</Detail>
<Reference>
<RefId>1</RefId>
<RefType>Ref 1</RefType>
<RefDoc>Doc 1</RefDoc>
<RefText>Text</RefText>
</Reference>
<Reference>
<RefId>2</RefId>
<RefType>Ref 2</RefType>
<RefDoc>Doc 2</RefDoc>
<RefText>Text</RefText>
</Reference>
<Discount>
<DiscId>1</DiscId>
<DiscDescription>Disc 1</DiscDescription>
<DiscItem>Item 1</DiscItem>
</Discount>
<Discount>
<DiscId>2</DiscId>
<DiscDescription>Disc 2</DiscDescription>
<DiscItem>Item 2</DiscItem>
</Discount>
<Discount>
<DiscId>3</DiscId>
<DiscDescription>Disc 3</DiscDescription>
<DiscItem>Item 3</DiscItem>
</Discount>
<Code>
<Code>Code a</Code>
<Value>Value 1</Value>
</Code>
<Code>
<Code>Code z</Code>
<Value>Value 3</Value>
</Code>
<Code>
<Code>Code x</Code>
<Value>Value 2</Value>
</Code>
<Code>
<Code>Code w</Code>
<Value>Value 2</Value>
</Code>
<Code>
<Code>Code y</Code>
<Value>Value 1</Value>
</Code>
</InvoiceDocument>
The required resources script is
Figure 34. Resources script resources-invoice.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="invoice">
<sx:serialize>
<sx:xsltSerializer>
<sx:outputProperty name="indent" value="yes"/>
</sx:xsltSerializer>
<sx:transform>
<sx:content ref="invoice-content"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="invoice-content">
<sx:flatFileReader>
<sx:flatFile ref="invoice-file"/>
</sx:flatFileReader>
<sx:recordMapping ref="invoice-mapping"/>
</sx:recordContent>
<sx:flatFile id="invoice-file">
<sx:recordDelimiter value="\r\n!"/>
<sx:recordDelimiter value="\n!"/>
<sx:flatFileBody>
<sx:flatRecordType name="invoice" omitFinalRepeatDelimiter="false">
<sx:repeatDelimiter value="~"/>
<sx:repeatingGroup name="Header" count="1">
<sx:flatRecordType name="Header">
<sx:fieldDelimiter value="|"/>
<sx:repeatDelimiter value="}"/>
<!--sx:segmentDelimiter value="~"/-->
<sx:repeatingGroup name="Document" count="1">
<sx:flatRecordType name="Doc">
<sx:delimitedField name="Id"/>
<sx:delimitedField name="DocDate"/>
<sx:delimitedField name="DocumentType"/>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:repeatingGroup name="Sender" count="1">
<sx:flatRecordType name="Sender">
<sx:delimitedField name="SenderPassport"/>
<sx:delimitedField name="SenderFullName"/>
<sx:delimitedField name="SenderAddress"/>
<sx:delimitedField name="SenderZip"/>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:repeatingGroup name="Recipient" count="1">
<sx:flatRecordType name="Recipient">
<sx:delimitedField name="RecipientPassport"/>
<sx:delimitedField name="RecipientFullName"/>
<sx:delimitedField name="RecipientAddress"/>
<sx:delimitedField name="RecipientZip"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:repeatingGroup name="Detail" count="1">
<sx:flatRecordType name="Detail">
<sx:fieldDelimiter value="|"/>
<sx:repeatDelimiter value="}"/>
<sx:repeatingGroup name="Item">
<sx:flatRecordType name="Item">
<sx:delimitedField name="Item"/>
<sx:delimitedField name="Description"/>
<sx:delimitedField name="Quantity"/>
<sx:delimitedField name="Unit"/>
<sx:delimitedField name="ItemPrice"/>
<sx:delimitedField name="TotalPrice"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:repeatingGroup name="Reference" count="1">
<sx:flatRecordType name="Reference">
<sx:fieldDelimiter value="|"/>
<sx:repeatDelimiter value="}"/>
<sx:segmentDelimiter value="~"/>
<sx:repeatingGroup name="Ref">
<sx:flatRecordType name="Ref">
<sx:delimitedField name="RefId"/>
<sx:delimitedField name="RefType"/>
<sx:delimitedField name="RefDoc"/>
<sx:delimitedField name="RefText"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:repeatingGroup name="Discount" count="1">
<sx:flatRecordType name="Discount">
<sx:fieldDelimiter value="|"/>
<sx:repeatDelimiter value="}"/>
<sx:segmentDelimiter value="~"/>
<sx:repeatingGroup name="Disc">
<sx:flatRecordType name="Disc">
<sx:delimitedField name="DiscId"/>
<sx:delimitedField name="DiscDescription"/>
<sx:delimitedField name="DiscItem"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:repeatingGroup name="Codes" count="1">
<sx:flatRecordType name="Codes">
<sx:fieldDelimiter value="|"/>
<sx:repeatDelimiter value="}"/>
<sx:segmentDelimiter value="~"/>
<sx:repeatingGroup name="Code">
<sx:flatRecordType name="Code">
<sx:delimitedField name="Code"/>
<sx:delimitedField name="Value"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="invoice-mapping">
<InvoiceDocument>
<sx:onRecord>
<sx:subrecordMapping repeatingGroup="Header">
<sx:onRecord>
<Header>
<sx:subrecordMapping repeatingGroup="Document">
<sx:onRecord>
<DocId>
<sx:fieldElementMap field="Id" element="Id"/>
<sx:fieldElementMap field="DocDate" element="DocDate"/>
<sx:fieldElementMap field="DocumentType" element="DocumentType"/>
</DocId>
</sx:onRecord>
</sx:subrecordMapping>
<sx:subrecordMapping repeatingGroup="Sender">
<sx:onRecord>
<Sender>
<sx:fieldElementMap field="SenderPassport" element="SenderPassport"/>
<sx:fieldElementMap field="SenderFullName" element="SenderFullName"/>
<sx:fieldElementMap field="SenderAddress" element="SenderAddress"/>
<sx:fieldElementMap field="SenderZip" element="SenderZip"/>
</Sender>
</sx:onRecord>
</sx:subrecordMapping>
<sx:subrecordMapping repeatingGroup="Recipient">
<sx:onRecord>
<Recipient>
<sx:fieldElementMap field="RecipientPassport" element="RecipientPassport"/>
<sx:fieldElementMap field="RecipientFullName" element="RecipientFullName"/>
<sx:fieldElementMap field="RecipientAddress" element="RecipientAddress"/>
<sx:fieldElementMap field="RecipientZip" element="RecipientZip"/>
</Recipient>
</sx:onRecord>
</sx:subrecordMapping>
</Header>
</sx:onRecord>
</sx:subrecordMapping>
<sx:subrecordMapping repeatingGroup="Detail">
<sx:onRecord>
<sx:subrecordMapping repeatingGroup="Item">
<sx:onRecord>
<Detail>
<sx:fieldElementMap field="Item" element="Item"/>
<sx:fieldElementMap field="Description" element="Description"/>
<sx:fieldElementMap field="Quantity" element="Quantity"/>
<sx:fieldElementMap field="Unit" element="Unit"/>
<sx:fieldElementMap field="ItemPrice" element="ItemPrice"/>
<sx:fieldElementMap field="TotalPrice" element="TotalPrice"/>
</Detail>
</sx:onRecord>
</sx:subrecordMapping>
</sx:onRecord>
</sx:subrecordMapping>
<sx:subrecordMapping repeatingGroup="Reference">
<sx:onRecord>
<sx:subrecordMapping repeatingGroup="Ref">
<sx:onRecord>
<Reference>
<sx:fieldElementMap field="RefId" element="RefId"/>
<sx:fieldElementMap field="RefType" element="RefType"/>
<sx:fieldElementMap field="RefDoc" element="RefDoc"/>
<sx:fieldElementMap field="RefText" element="RefText"/>
</Reference>
</sx:onRecord>
</sx:subrecordMapping>
</sx:onRecord>
</sx:subrecordMapping>
<sx:subrecordMapping repeatingGroup="Discount">
<sx:onRecord>
<sx:subrecordMapping repeatingGroup="Disc">
<sx:onRecord>
<Discount>
<sx:fieldElementMap field="DiscId" element="DiscId"/>
<sx:fieldElementMap field="DiscDescription" element="DiscDescription"/>
<sx:fieldElementMap field="DiscItem" element="DiscItem"/>
</Discount>
</sx:onRecord>
</sx:subrecordMapping>
</sx:onRecord>
</sx:subrecordMapping>
<sx:subrecordMapping repeatingGroup="Codes">
<sx:onRecord>
<sx:subrecordMapping repeatingGroup="Code">
<sx:onRecord>
<Code>
<sx:fieldElementMap field="Code" element="Code"/>
<sx:fieldElementMap field="Value" element="Value"/>
</Code>
</sx:onRecord>
</sx:subrecordMapping>
</sx:onRecord>
</sx:subrecordMapping>
</sx:onRecord>
</InvoiceDocument>
</sx:recordMapping>
</sx:resources>
The idea is to map all the data up to the terminating "!" as one record, and to structure the record as a sequence of nested repeating groups.
You can run this example on the command line by entering
servingxml -r resources-invoice.xml -i data/invoice.txt -o output/invoice.xml
invoice
The example below shows how to convert a composite field in an edi file to XML.
The input file is an edi file.
This file has a number of features.
The segment delimiter "'" terminates the segment.
The field delimiter "+" terminates a field.
Composite fields are delimited with ":".
The desired output is shown below.
Figure 36. Output XML file messages.xml
<?xml version="1.0" encoding="utf-8"?>
<edifact>
<segment segmentType="BKG">
<flightSegmentNumber>1</flightSegmentNumber>
<accessID/>
<bookingCodeCount>4</bookingCodeCount>
<C704>
<bookingCode>Y</bookingCode>
<seatAvailabilityCount>1</seatAvailabilityCount>
</C704>
<C704>
<bookingCode>B</bookingCode>
<seatAvailabilityCount>0</seatAvailabilityCount>
</C704>
<C704>
<bookingCode>M</bookingCode>
<seatAvailabilityCount>0</seatAvailabilityCount>
</C704>
<C704>
<bookingCode>L</bookingCode>
<seatAvailabilityCount>1</seatAvailabilityCount>
</C704>
</segment>
</edifact>
The following resources script does the transformation.
Figure 37. Resources script resources-comptest.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="edifact">
<sx:serialize>
<sx:transform>
<sx:content ref="edifact"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="edifact">
<sx:flatFileReader>
<sx:flatFile ref="edifactFlatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="edifactToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="edifactFlatFile">
<sx:recordDelimiter value="'"/>
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:fieldDelimiter value="+"/>
<sx:delimitedField name="segmentType"/>
<!-- edifact3 - Body Segments.-->
<!-- BKG BOOKING CODE DATA .-->
<sx:when test="segmentType='BKG'">
<sx:flatRecordType name='BKG'>
<sx:fieldDelimiter value="+"/>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="flightSegmentNumber"/>
<sx:delimitedField name="accessID"/>
<sx:delimitedField name="bookingCodeCount"/>
<sx:repeatingGroup name="booking" count="{bookingCodeCount}">
<sx:flatRecordType name="C704">
<sx:fieldDelimiter value=":"/>
<sx:delimitedField name="bookingCode">
</sx:delimitedField>
<sx:delimitedField name="seatAvailabilityCount">
</sx:delimitedField>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="edifactToXmlMapping">
<edifact>
<!-- edifact3 - Body Segments.-->
<!-- BKG BOOKING CODE DATA.-->
<sx:onRecord recordType="BKG">
<sx:elementMap element="segment">
<sx:fieldAttributeMap field="segmentType" attribute="segmentType"/>
<sx:fieldElementMap field="flightSegmentNumber" element="flightSegmentNumber"/>
<sx:fieldElementMap field="accessID" element="accessID"/>
<sx:fieldElementMap field="bookingCodeCount" element="bookingCodeCount"/>
<sx:subrecordMapping repeatingGroup="booking">
<sx:onRecord recordType="C704">
<C704>
<sx:fieldElementMap field="bookingCode" element="bookingCode"/>
<sx:fieldElementMap field="seatAvailabilityCount" element="seatAvailabilityCount"/>
</C704>
</sx:onRecord>
</sx:subrecordMapping>
</sx:elementMap>
</sx:onRecord>
</edifact>
</sx:recordMapping>
</sx:resources>
Note the following points about this script.
All the values of the composite field are put in the multiple valued field "C704".
The sx:repeatingGroup element allows us to map these values, repeated two at a time,
named as
bookingCode
and
seatAvailabilityCount
.
You can run this example on the command line by entering
servingxml -i data/comptest.txt -r resources-comptest.xml
-o output/comptest.xml edifact
The example below shows how to convert a composite field in an edi file to XML.
The input file is an edi file.
Figure 38. Input edi file invoic96A.txt
UNB+UNOA:3+0000008033253:14+01534730278:ZZ+080130:0000+04000020++++++1'
UNH+1+INVOIC:D:96A:UN:EAN008'
BGM+381+1000322+9'
DTM+137:20080130:102'
NAD+SU+++COMPANY XXXX ?+ YYYY+VIA ROMA, 1+CITTA?' DI?: GENOVA+GE+16163+IT'
RFF+VA:05577790207'
CUX+2:EUR:4+3'
PAT+10E+ZZZ:::BONIFICO BANCARIO 90 GG D.F.+5:1:D:090'
DTM+13:20080430:102'
LIN+1+L02+8026495011408:EN'
IMD+B++TU:::BAGUETTE PREC. GR.280 40PZ'
QTY+47:5.000:PCE'
QTY+59:40.000:PCE'
MOA+203:54.130:EUR:4'
PRI+AAA:10.830::::PCE'
UNS+S'
MOA+125:54.130'
UNT+46+1'
UNZ+1+04000020'
This file has a number of features.
Records are terminated with '
Fields are terminated with +
Subfields are terminated with :
The character ? is used as an escape character. For example,
COMPANY?: XXXX?+YYYY+ will be interpreted as:
COMPANY: XXXX+YYYY
The desired output is shown below.
Figure 39. Output XML file invoic96A.xml
<?xml version="1.0" encoding="UTF-8"?>
<Invoic96A>
<UNB>
<segmentType>UNB</segmentType>
</UNB>
<UNH>
<segmentType>UNH</segmentType>
<DE0062>1</DE0062>
<S009>
<S009>
<DE0065>INVOIC</DE0065>
<DE0052>D</DE0052>
<DE0054>96A</DE0054>
<DE0051>UN</DE0051>
<DE0057>EAN008</DE0057>
</S009>
</S009>
<DE0068/>
</UNH>
<BGM>
<segmentType>BGM</segmentType>
<C002>
<C002>
<DE1001>381</DE1001>
<DE1131/>
<DE3055/>
<DE1000/>
</C002>
</C002>
<DE1004>1000322</DE1004>
<DE1225>9</DE1225>
<DE4343/>
</BGM>
<DTM>
<segmentType>DTM</segmentType>
<C507>
<C507>
<DE2005>137</DE2005>
<DE2380>20080130</DE2380>
<DE2379>102</DE2379>
</C507>
</C507>
</DTM>
<NAD>
<segmentType>NAD</segmentType>
<DE3035>SU</DE3035>
<C080>
<C080>
<DE3036-1>COMPANY XXXX + YYYY</DE3036-1>
<DE3036-2/>
<DE3036-3/>
<DE3036-4/>
<DE3036-5/>
<DE3045/>
</C080>
</C080>
<C059>
<C059>
<DE3042-1>VIA ROMA, 1</DE3042-1>
<DE3042-2/>
<DE3042-3/>
<DE3042-4/>
</C059>
</C059>
<DE3164>CITTA' DI: GENOVA</DE3164>
<DE3229>GE</DE3229>
<DE3251>16163</DE3251>
<DE3207>IT</DE3207>
</NAD>
<RFF>
<segmentType>RFF</segmentType>
<C506>
<C506>
<DE1153>VA</DE1153>
<DE1154>05577790207</DE1154>
<DE1156/>
<DE4000/>
</C506>
</C506>
</RFF>
<CUX>
<segmentType>CUX</segmentType>
<C504>
<C504>
<DE6347>2</DE6347>
<DE6345>EUR</DE6345>
<DE6343>4</DE6343>
<DE6348/>
</C504>
<C504>
<DE6347>3</DE6347>
<DE6345/>
<DE6343/>
<DE6348/>
</C504>
</C504>
<DE5402/>
<DE6341/>
</CUX>
<PAT>
<segmentType>PAT</segmentType>
<DE4279>10E</DE4279>
<C110>
<C110>
<DE4277>ZZZ</DE4277>
<DE1131/>
<DE3055/>
<DE4276-1>BONIFICO BANCARIO 90 GG D.F.</DE4276-1>
<DE4276-2/>
</C110>
</C110>
<C112>
<C112>
<DE2475>5</DE2475>
<DE2009>1</DE2009>
<DE2151>D</DE2151>
<DE2152>090</DE2152>
</C112>
</C112>
</PAT>
<DTM>
<segmentType>DTM</segmentType>
<C507>
<C507>
<DE2005>13</DE2005>
<DE2380>20080430</DE2380>
<DE2379>102</DE2379>
</C507>
</C507>
</DTM>
<LIN>
<segmentType>LIN</segmentType>
<DE1082>1</DE1082>
<DE1229>L02</DE1229>
<C212>
<C212>
<DE7140>8026495011408</DE7140>
<DE7143>EN</DE7143>
<DE1131/>
<DE3055/>
</C212>
</C212>
<DE1222/>
<DE7083/>
</LIN>
<IMD>
<segmentType>IMD</segmentType>
<DE7077>B</DE7077>
<DE7081/>
<C273>
<C273>
<DE7009>TU</DE7009>
<DE1131/>
<DE3055/>
<DE7008-1>BAGUETTE PREC. GR.280 40PZ</DE7008-1>
<DE7008-2/>
<DE3435/>
</C273>
</C273>
<DE7383/>
</IMD>
<QTY>
<segmentType>QTY</segmentType>
<C186>
<C186>
<DE6063>47</DE6063>
<DE6060>5.000</DE6060>
<DE6411>PCE</DE6411>
</C186>
</C186>
</QTY>
<QTY>
<segmentType>QTY</segmentType>
<C186>
<C186>
<DE6063>59</DE6063>
<DE6060>40.000</DE6060>
<DE6411>PCE</DE6411>
</C186>
</C186>
</QTY>
<MOA>
<segmentType>MOA</segmentType>
<C516>
<C516>
<DE5025>203</DE5025>
<DE5004>54.130</DE5004>
<DE6345>EUR</DE6345>
<DE6343>4</DE6343>
<DE4405/>
</C516>
</C516>
</MOA>
<PRI>
<segmentType>PRI</segmentType>
<C509>
<C509>
<DE5125>AAA</DE5125>
<DE5118>10.830</DE5118>
<DE5375/>
<DE5387/>
<DE5284/>
<DE6411>PCE</DE6411>
</C509>
</C509>
<DE5213/>
</PRI>
<UNS>
<segmentType>UNS</segmentType>
<DE0081>S</DE0081>
</UNS>
<MOA>
<segmentType>MOA</segmentType>
<C516>
<C516>
<DE5025>125</DE5025>
<DE5004>54.130</DE5004>
<DE6345/>
<DE6343/>
<DE4405/>
</C516>
</C516>
</MOA>
<UNT>
<segmentType>UNT</segmentType>
<DE0074>46</DE0074>
<DE0062>1</DE0062>
</UNT>
<UNZ>
<segmentType>UNZ</segmentType>
<F0036>1</F0036>
<F0020>04000020</F0020>
</UNZ>
</Invoic96A>
The following resources script does the transformation.
Figure 40. Resources script resources-invoic96A.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv"
xmlns:fn="http://www.w3.org/2005/xpath-functions">
<sx:service id="invoic96A-to-xml">
<sx:serialize>
<sx:xsltSerializer>
<sx:outputProperty name="indent" value="yes"/>
</sx:xsltSerializer>
<sx:transform>
<sx:content ref="Invoic_96A_1"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="Invoic_96A_1" name="Invoic96A">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="edifactFlatFile"/>
</sx:flatFileReader>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
</sx:discardHandler>
</sx:recordStream>
</sx:recordContent>
<sx:flatFile id="edifactFlatFile">
<sx:recordDelimiter value="'" escapeCharacter="?"/>
<!-- If a line is broken with a new line character, strip the new line character -->
<sx:recordDelimiter value="\r\n" continuationSequence="\r\n"/>
<sx:recordDelimiter value="\n" continuationSequence="\n"/>
<!-- if a ' character is found after a ? (?')-->
<!-- it is an escape sequence and should remain the ' character-->
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
<sx:when test="segmentType='UNA' or segmentType='UNA:'">
<sx:flatRecordType name='UNA'>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='UNB'">
<sx:flatRecordType name='UNB'>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='UNH'">
<sx:flatRecordType name='UNH'>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE0062"/>
<sx:nonrepeatingGroup name="S009">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="S009">
<sx:delimitedField name="DE0065"/>
<sx:delimitedField name="DE0052"/>
<sx:delimitedField name="DE0054"/>
<sx:delimitedField name="DE0051"/>
<sx:delimitedField name="DE0057"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE0068"/>
<sx:nonrepeatingGroup name="S010">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="S010">
<sx:delimitedField name="DE0070"/>
<sx:delimitedField name="DE0073"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='BGM'">
<sx:flatRecordType name='BGM'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C002">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C002">
<sx:delimitedField name="DE1001"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
<sx:delimitedField name="DE1000"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE1004"/>
<sx:delimitedField name="DE1225"/>
<sx:delimitedField name="DE4343"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='DTM'">
<sx:flatRecordType name='DTM'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C507">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C507">
<sx:delimitedField name="DE2005"/>
<sx:delimitedField name="DE2380"/>
<sx:delimitedField name="DE2379"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='PAI'">
<sx:flatRecordType name='PAI'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C534">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C534">
<sx:delimitedField name="DE4439"/>
<sx:delimitedField name="DE4431"/>
<sx:delimitedField name="DE4461"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
<sx:delimitedField name="DE4435"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='FTX'">
<sx:flatRecordType name='FTX'>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE4451"/>
<sx:delimitedField name="DE4453"/>
<sx:nonrepeatingGroup name="C107">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C107">
<sx:delimitedField name="DE4441"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:nonrepeatingGroup name="C108">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C108">
<sx:delimitedField name="DE4440-1"/>
<sx:delimitedField name="DE4440-2"/>
<sx:delimitedField name="DE4440-3"/>
<sx:delimitedField name="DE4440-4"/>
<sx:delimitedField name="DE4440-5"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE3453"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='NAD'">
<sx:flatRecordType name='NAD'>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE3035"/>
<sx:nonrepeatingGroup name="C082">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C082">
<sx:delimitedField name="DE3039"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:nonrepeatingGroup name="C058">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C058">
<sx:delimitedField name="DE3124-1"/>
<sx:delimitedField name="DE3124-2"/>
<sx:delimitedField name="DE3124-3"/>
<sx:delimitedField name="DE3124-4"/>
<sx:delimitedField name="DE3124-5"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:nonrepeatingGroup name="C080">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C080">
<sx:delimitedField name="DE3036-1"/>
<sx:delimitedField name="DE3036-2"/>
<sx:delimitedField name="DE3036-3"/>
<sx:delimitedField name="DE3036-4"/>
<sx:delimitedField name="DE3036-5"/>
<sx:delimitedField name="DE3045"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:nonrepeatingGroup name="C059">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C059">
<sx:delimitedField name="DE3042-1"/>
<sx:delimitedField name="DE3042-2"/>
<sx:delimitedField name="DE3042-3"/>
<sx:delimitedField name="DE3042-4"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE3164"/>
<sx:delimitedField name="DE3229"/>
<sx:delimitedField name="DE3251"/>
<sx:delimitedField name="DE3207"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='FII'">
<sx:flatRecordType name='FII'>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE3035"/>
<sx:nonrepeatingGroup name="C078">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C078">
<sx:delimitedField name="DE3194"/>
<sx:delimitedField name="DE3192-1"/>
<sx:delimitedField name="DE3192-2"/>
<sx:delimitedField name="DE6345"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:nonrepeatingGroup name="C088">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C088">
<sx:delimitedField name="DE3433"/>
<sx:delimitedField name="DE1131-1"/>
<sx:delimitedField name="DE3055-1"/>
<sx:delimitedField name="DE3434"/>
<sx:delimitedField name="DE1131-2"/>
<sx:delimitedField name="DE3055-2"/>
<sx:delimitedField name="DE3432"/>
<sx:delimitedField name="DE3436"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE3207"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='RFF'">
<sx:flatRecordType name='RFF'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C506">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C506">
<sx:delimitedField name="DE1153"/>
<sx:delimitedField name="DE1154"/>
<sx:delimitedField name="DE1156"/>
<sx:delimitedField name="DE4000"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='CTA'">
<sx:flatRecordType name='CTA'>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE3139"/>
<sx:nonrepeatingGroup name="C056">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C056">
<sx:delimitedField name="DE3413"/>
<sx:delimitedField name="DE3412"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='COM'">
<sx:flatRecordType name='COM'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C076">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C076">
<sx:delimitedField name="DE3148"/>
<sx:delimitedField name="DE3155"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='CUX'">
<sx:flatRecordType name='CUX'>
<sx:delimitedField name="segmentType"/>
<sx:repeatingGroup name="C504" count="2">
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:repeatDelimiter value="+" escapeCharacter="?"/>
<!--<sx:segmentDelimiter value="+" escapeCharacter="?"/>-->
<sx:flatRecordType name="C504">
<sx:delimitedField name="DE6347"/>
<sx:delimitedField name="DE6345"/>
<sx:delimitedField name="DE6343"/>
<sx:delimitedField name="DE6348"/>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:delimitedField name="DE5402"/>
<sx:delimitedField name="DE6341"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='PAT'">
<sx:flatRecordType name='PAT'>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE4279"/>
<sx:nonrepeatingGroup name="C110">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C110">
<sx:delimitedField name="DE4277"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
<sx:delimitedField name="DE4276-1"/>
<sx:delimitedField name="DE4276-2"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:nonrepeatingGroup name="C112">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C112">
<sx:delimitedField name="DE2475"/>
<sx:delimitedField name="DE2009"/>
<sx:delimitedField name="DE2151"/>
<sx:delimitedField name="DE2152"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='PCD'">
<sx:flatRecordType name='PCD'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C501">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C501">
<sx:delimitedField name="DE5245"/>
<sx:delimitedField name="DE5482"/>
<sx:delimitedField name="DE5249"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='MOA'">
<sx:flatRecordType name='MOA'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C516">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C516">
<sx:delimitedField name="DE5025"/>
<sx:delimitedField name="DE5004"/>
<sx:delimitedField name="DE6345"/>
<sx:delimitedField name="DE6343"/>
<sx:delimitedField name="DE4405"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='ALC'">
<sx:flatRecordType name='ALC'>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE5463"/>
<sx:nonrepeatingGroup name="C552">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C552">
<sx:delimitedField name="DE1230"/>
<sx:delimitedField name="DE5189"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE4471"/>
<sx:delimitedField name="DE1227"/>
<sx:nonrepeatingGroup name="C214">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C214">
<sx:delimitedField name="DE7161"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
<sx:delimitedField name="DE7160-1"/>
<sx:delimitedField name="DE7160-2"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<!-- Detail segments -->
<sx:when test="segmentType='LIN'">
<sx:flatRecordType name='LIN'>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE1082"/>
<sx:delimitedField name="DE1229"/>
<sx:nonrepeatingGroup name="C212">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C212">
<sx:delimitedField name="DE7140"/>
<sx:delimitedField name="DE7143"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:nonrepeatingGroup name="C829">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C829">
<sx:delimitedField name="DE5495"/>
<sx:delimitedField name="DE1082"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE1222"/>
<sx:delimitedField name="DE7083"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='PIA'">
<sx:flatRecordType name='PIA'>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE4347"/>
<sx:nonrepeatingGroup name="C212">
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<!--<sx:segmentDelimiter value="+" escapeCharacter="?"/>-->
<sx:flatRecordType name="C212">
<sx:delimitedField name="DE7140"/>
<sx:delimitedField name="DE7143"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='IMD'">
<sx:flatRecordType name='IMD'>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE7077"/>
<sx:delimitedField name="DE7081"/>
<sx:nonrepeatingGroup name="C273">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C273">
<sx:delimitedField name="DE7009"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
<sx:delimitedField name="DE7008-1"/>
<sx:delimitedField name="DE7008-2"/>
<sx:delimitedField name="DE3435"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE7383"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='QTY'">
<sx:flatRecordType name='QTY'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C186">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C186">
<sx:delimitedField name="DE6063"/>
<sx:delimitedField name="DE6060"/>
<sx:delimitedField name="DE6411"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='PRI'">
<sx:flatRecordType name='PRI'>
<sx:delimitedField name="segmentType"/>
<sx:nonrepeatingGroup name="C509">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C509">
<sx:delimitedField name="DE5125"/>
<sx:delimitedField name="DE5118"/>
<sx:delimitedField name="DE5375"/>
<sx:delimitedField name="DE5387"/>
<sx:delimitedField name="DE5284"/>
<sx:delimitedField name="DE6411"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE5213"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='TAX'">
<sx:flatRecordType name='TAX'>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE5283"/>
<sx:nonrepeatingGroup name="C241">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C241">
<sx:delimitedField name="DE5153"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
<sx:delimitedField name="DE5152"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:nonrepeatingGroup name="C533">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C533">
<sx:delimitedField name="DE5289"/>
<sx:delimitedField name="DE1131"/>
<sx:delimitedField name="DE3055"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE5286"/>
<sx:nonrepeatingGroup name="C243">
<sx:segmentDelimiter value="+" escapeCharacter="?"/>
<sx:fieldDelimiter value=":" escapeCharacter="?"/>
<sx:flatRecordType name="C243">
<sx:delimitedField name="DE5279"/>
<sx:delimitedField name="DE1131-1"/>
<sx:delimitedField name="DE3055-1"/>
<sx:delimitedField name="DE5278"/>
<sx:delimitedField name="DE5273"/>
<sx:delimitedField name="DE1131-2"/>
<sx:delimitedField name="DE3055-2"/>
</sx:flatRecordType>
</sx:nonrepeatingGroup>
<sx:delimitedField name="DE5305"/>
<sx:delimitedField name="DE3446"/>
</sx:flatRecordType>
</sx:when>
<!-- Trailer segments -->
<sx:when test="segmentType='UNS'">
<sx:flatRecordType name='UNS'>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE0081"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='UNT'">
<sx:flatRecordType name='UNT'>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="DE0074"/>
<sx:delimitedField name="DE0062"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="segmentType='UNZ'">
<sx:flatRecordType name='UNZ'>
<sx:fieldDelimiter value="+" escapeCharacter="?"/>
<sx:delimitedField name="segmentType"/>
<sx:delimitedField name="F0036"/>
<sx:delimitedField name="F0020"/>
</sx:flatRecordType>
</sx:when>
<sx:otherwise>
<sx:flatRecordType name='ERROR'>
<sx:delimitedField name="segmentType"/>
</sx:flatRecordType>
</sx:otherwise>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
Note the following points about this script.
Record delimiters delimit the boundaries of a record, segment delimiters delimit areas within a record, and repeat delimiters separate subrecords from each other within segments. An sx:repeatingGroup reads the subrecords separated by repeat delimiters within a segment. A special case is when there are a number of segments, but at most one subrecord per segment, a nonrepeating group. An sx:nonrepeatingGroup reads the single subrecord, if any.
The continuation sequence for a linefeed is defined be the linefeed itself, e.g.
<sx:recordDelimiter value="\r\n" continuationSequence="\r\n"/>
<sx:recordDelimiter value="\n" continuationSequence="\n"/>
This has the effect of stripping out the linefeed characters.
You can run this example on the command line by entering
servingxml -r resources-invoic96A.xml -i data/invoic96A.txt
-o output/invoic96A.xml invoic96A-to-xml
Suppose "Bank A" has the following CSV file of FRA trades. Each FRA trade is represented by one record in the file.
Figure 41. Input flat file fra.csv
product,trade_id,client_id,client_name,trade_date,buy_sell_code,currency,index,tenor,day_count_basis,fixed_rate,start_date,maturity_date,payment_date,notional
FRA,1001,party1,Party 1,2006/12/07,BUY,CAD,BA,3M,ACT/365,0.04, 2008/09/08,2008/12/08,2008/09/08,250000000
FRA,1002,party1,Party 1,2007/03/19,B,CAD,BA,3M,ACT/365,0.04, 2008/09/15,2008/12/15,2008/09/15,110000000
FRA,1003,party1,Party 1,2007/03/19,SELL,CAD,BA,3M,ACT/365,0.04, 2008/12/15,2009/03/16,2008/12/15,70000000
FRA,1004,party2,Party 2,2007/05/02,BUY,USD,LIBOR,3M,ACT/360,0.04,2009/02/02,2009/05/04,2009/02/02,100000000
FRA,1005,party2,Party 2,2007/05/02,BUY,USD,LIBOR,3M,ACT/360,0.04,2009/02/02,2009/05/04,2009/02/02,100000000
FRA,1006,party3,Party 3,2007/08/01,B,USD,LIBOR,3M,ACT/360,0.05,09/05/01,2009/08/04,2009/05/01,50000000
Note that two of the FRA trades have errors. The trade with trade_id 1002 has an invalid buy_sell_code, "B" instead of "BUY". The trade with trade id 1006 has the same problem with the buy_sell_code, and also has an invalid start_date, with a two digit year instead of a four digit year. Bank A wants to reject the trades with errors and write them along with an error message to a discard file.
Bank A wants the XML output to look as follows.
Figure 42. Output XML file fra.xml
<?xml version="1.0" encoding="UTF-8"?>
<dataDocument xmlns="http://www.fpml.org/FpML-5-0/reporting"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" fpmlVersion="5-0"
xsi:schemaLocation="http://www.fpml.org/FpML-5-0/reporting ../fpml-main-5-0.xsd">
<trade>
<tradeHeader>
<partyTradeIdentifier>
<tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">1001</tradeId>
</partyTradeIdentifier>
<tradeDate>2006-12-07</tradeDate>
</tradeHeader>
<fra>
<sellerPartyReference>party1</sellerPartyReference>
<adjustedEffectiveDate>2008-09-08</adjustedEffectiveDate>
<adjustedTerminationDate>2008-12-08</adjustedTerminationDate>
<paymentDate>
<adjustedDate>2008-09-08</adjustedDate>
</paymentDate>
<dayCountFraction>ACT/365</dayCountFraction>
<notional>
<currency>CAD</currency>
<amount>250000000</amount>
</notional>
<fixedRate>0.04</fixedRate>
<floatingRateIndex>BA</floatingRateIndex>
<indexTenor>
<periodMultiplier>3</periodMultiplier>
<period>M</period>
</indexTenor>
</fra>
</trade>
<trade>
<tradeHeader>
<partyTradeIdentifier>
<tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">1003</tradeId>
</partyTradeIdentifier>
<tradeDate>2007-03-19</tradeDate>
</tradeHeader>
<fra>
<buyerPartyReference>party1</buyerPartyReference>
<adjustedEffectiveDate>2008-12-15</adjustedEffectiveDate>
<adjustedTerminationDate>2009-03-16</adjustedTerminationDate>
<paymentDate>
<adjustedDate>2008-12-15</adjustedDate>
</paymentDate>
<dayCountFraction>ACT/365</dayCountFraction>
<notional>
<currency>CAD</currency>
<amount>70000000</amount>
</notional>
<fixedRate>0.04</fixedRate>
<floatingRateIndex>BA</floatingRateIndex>
<indexTenor>
<periodMultiplier>3</periodMultiplier>
<period>M</period>
</indexTenor>
</fra>
</trade>
<trade>
<tradeHeader>
<partyTradeIdentifier>
<tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">1004</tradeId>
</partyTradeIdentifier>
<tradeDate>2007-05-02</tradeDate>
</tradeHeader>
<fra>
<sellerPartyReference>party2</sellerPartyReference>
<adjustedEffectiveDate>2009-02-02</adjustedEffectiveDate>
<adjustedTerminationDate>2009-05-04</adjustedTerminationDate>
<paymentDate>
<adjustedDate>2009-02-02</adjustedDate>
</paymentDate>
<dayCountFraction>ACT/360</dayCountFraction>
<notional>
<currency>USD</currency>
<amount>100000000</amount>
</notional>
<fixedRate>0.04</fixedRate>
<floatingRateIndex>LIBOR</floatingRateIndex>
<indexTenor>
<periodMultiplier>3</periodMultiplier>
<period>M</period>
</indexTenor>
</fra>
</trade>
<trade>
<tradeHeader>
<partyTradeIdentifier>
<tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">1005</tradeId>
</partyTradeIdentifier>
<tradeDate>2007-05-02</tradeDate>
</tradeHeader>
<fra>
<sellerPartyReference>party2</sellerPartyReference>
<adjustedEffectiveDate>2009-02-02</adjustedEffectiveDate>
<adjustedTerminationDate>2009-05-04</adjustedTerminationDate>
<paymentDate>
<adjustedDate>2009-02-02</adjustedDate>
</paymentDate>
<dayCountFraction>ACT/360</dayCountFraction>
<notional>
<currency>USD</currency>
<amount>100000000</amount>
</notional>
<fixedRate>0.04</fixedRate>
<floatingRateIndex>LIBOR</floatingRateIndex>
<indexTenor>
<periodMultiplier>3</periodMultiplier>
<period>M</period>
</indexTenor>
</fra>
</trade>
<party id="bankA">
<partyId>Bank A</partyId>
</party>
<party id="party1">
<partyId>Party 1</partyId>
</party>
<party id="party3">
<partyId>Party 3</partyId>
</party>
<party id="party2">
<partyId>Party 2</partyId>
</party>
</dataDocument>
Note that the document concludes with a unique list of parties that
are referenced by
id
in the
buyerPartyReference
and
sellerPartyReference
elements.
The required resources script is
Figure 43. Resources script resources-fra.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:fn="http://www.w3.org/2005/xpath-functions">
<sx:service id="fra-to-xml">
<sx:serialize>
<sx:transform>
<sx:content ref="fra-xml"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="fra-xml">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="fra-flat-file"/>
</sx:flatFileReader>
<sx:recordValidator message="Error in trade {trade_id}.">
<sx:fieldValidator field="buy_sell_code"
message="Invalid buy_sell_code {buy_sell_code}.">
<sx:or>
<sx:valueRestriction pattern="BUY"/>
<sx:valueRestriction pattern="SELL"/>
</sx:or>
</sx:fieldValidator>
<sx:fieldValidator field="start_date"
message="Invalid start_date {start_date}.">
<sx:valueRestriction pattern="(19|20)\d\d[/](0[1-9]|1[012])[/](0[1-9]|[12][0-9]|3[01])"/>
</sx:fieldValidator>
<sx:fieldValidator field="maturity_date"
message="Invalid maturity_date {maturity_date}.">
<sx:valueRestriction pattern="(20|21)\d\d[/](0[1-9]|1[012])[/](0[1-9]|[12][0-9]|3[01])"/>
</sx:fieldValidator>
</sx:recordValidator>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<sx:modifyRecord>
<sx:newField name="message" value="{$sx:message}"/>
</sx:modifyRecord>
<sx:flatFileWriter>
<sx:fileSink file="output/fra-error.csv"/>
</sx:flatFileWriter>
</sx:discardHandler>
</sx:recordStream>
<sx:recordMapping ref="fra-to-xml-mapping"/>
</sx:recordContent>
<sx:flatFile id="fra-flat-file">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="record_type"/>
<sx:when test="record_type='FRA'">
<sx:flatRecordType name='fra'>
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="record_type"/>
<sx:delimitedField name="trade_id"/>
<sx:delimitedField name="client_id"/>
<sx:delimitedField name="client_name"/>
<sx:delimitedField name="trade_date"/>
<sx:delimitedField name="buy_sell_code"/>
<sx:delimitedField name="currency"/>
<sx:delimitedField name="index"/>
<sx:delimitedField name="tenor"/>
<sx:delimitedField name="day_count_basis"/>
<sx:delimitedField name="fixed_rate"/>
<sx:delimitedField name="start_date"/>
<sx:delimitedField name="maturity_date"/>
<sx:delimitedField name="payment_date"/>
<sx:delimitedField name="notional"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="fra-to-xml-mapping">
<dataDocument xmlns="http://www.fpml.org/FpML-5-0/reporting"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
fpmlVersion="5-0"
xsi:schemaLocation="http://www.fpml.org/FpML-5-0/reporting ../fpml-main-5-0.xsd">
<sx:groupBy fields="trade_id">
<trade>
<tradeHeader>
<partyTradeIdentifier>
<sx:fieldElementMap field="trade_id"
element="tradeId">
<sx:fieldAttributeMap value="http://www.banka.com/ird/trade-id"
attribute="tradeIdScheme"/>
</sx:fieldElementMap>
</partyTradeIdentifier>
<tradeDate>
<sx:toXmlDate fromFormat="yyyy/MM/dd">
<sx:toString value="{trade_date}"/>
</sx:toXmlDate>
</tradeDate>
</tradeHeader>
<sx:onRecord recordType="fra">
<fra>
<sx:choose>
<sx:when test="buy_sell_code='BUY'">
<sx:fieldElementMap field="client_id"
element="sellerPartyReference"/>
</sx:when>
<sx:otherwise>
<sx:fieldElementMap field="client_id"
element="buyerPartyReference"/>
</sx:otherwise>
</sx:choose>
<adjustedEffectiveDate>
<sx:toXmlDate fromFormat="yyyy/MM/dd">
<sx:toString value="{start_date}"/>
</sx:toXmlDate>
</adjustedEffectiveDate>
<adjustedTerminationDate>
<sx:toXmlDate fromFormat="yyyy/MM/dd">
<sx:toString value="{maturity_date}"/>
</sx:toXmlDate>
</adjustedTerminationDate>
<paymentDate>
<adjustedDate>
<sx:toXmlDate fromFormat="yyyy/MM/dd">
<sx:toString value="{payment_date}"/>
</sx:toXmlDate>
</adjustedDate>
</paymentDate>
<sx:fieldElementMap field="day_count_basis"
element="dayCountFraction"/>
<notional>
<sx:fieldElementMap field="currency"
element="currency"/>
<sx:fieldElementMap field="notional"
element="amount"/>
</notional>
<sx:fieldElementMap field="fixed_rate"
element="fixedRate"/>
<sx:fieldElementMap field="index"
element="floatingRateIndex"/>
<indexTenor>
<sx:choose>
<sx:when test="fn:ends-with(tenor,'D')">
<sx:fieldElementMap element="periodMultiplier">
<sx:toString select="fn:substring-before(tenor,'D')"/>
</sx:fieldElementMap>
<period>D</period>
</sx:when>
<sx:when test="fn:ends-with(tenor,'M')">
<sx:fieldElementMap element="periodMultiplier">
<sx:toString select="fn:substring-before(tenor,'M')"/>
</sx:fieldElementMap>
<period>M</period>
</sx:when>
<sx:when test="fn:ends-with(tenor,'Y')">
<sx:fieldElementMap element="periodMultiplier">
<sx:toString select="fn:substring-before(tenor,'Y')"/>
</sx:fieldElementMap>
<period>Y</period>
</sx:when>
</sx:choose>
</indexTenor>
</fra>
</sx:onRecord>
</trade>
</sx:groupBy>
<sx:nestedContent>
<sx:recordContent ref="clients"/>
</sx:nestedContent>
</dataDocument>
</sx:recordMapping>
<sx:recordContent id="clients">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="fra-flat-file"/>
</sx:flatFileReader>
<sx:recordProjection fields="client_id client_name"/>
</sx:recordStream>
<sx:recordMapping>
<party id="bankA">
<partyId>Bank A</partyId>
</party>
<sx:onRecord>
<party>
<sx:fieldAttributeMap field="client_id" attribute="id"/>
<sx:fieldElementMap field="client_name" element="partyId"/>
</party>
</sx:onRecord>
</sx:recordMapping>
</sx:recordContent>
</sx:resources>
We use the
sx:nestedContent
element to insert a list of parties from the
swap.csv
file into the output XML, and we use the
sx:recordProjection
element to eliminate duplicate pairs of client_id and client_name.
You can run this example on the command line by entering
servingxml -i data/fra.csv -o output/fra.xml -r resources-fra.xml fra-to-xml
The following error messages will be written to the console (or log file.)
25-Sep-2008 10:10:47 PM com.servingxml.util.system.DefaultLogger error
SEVERE: Error in record "fra" on line 3. Error in trade 1002. Invalid buy_sell_code B.
25-Sep-2008 10:10:47 PM com.servingxml.util.system.DefaultLogger error
SEVERE: Error in record "fra" on line 7. Error in trade 1006. Invalid buy_sell_code B. Invalid start_date 09/05/01.
Suppose "Bank A" has the following CSV file of SWAP trades. Each swap trade is represented by two records in the file, one for the pay (PAY) side, and one for the receive (REC) side.
Figure 44. Input flat file swap.csv
product,trade_id,client_id,client_name,trade_date,pay_receive_code,currency,index,tenor,day_count_basis,fixed_rate,start_date,maturity_date,reset_date,payment_date,notional,calculation_period
SWAP,2001,party1,Party 1,2008/05/14,PAY,FIXED,CAD,BA,6M,30/360,0.04,2008/07/17,2009/01/17,2008/07/17,2009/01/17,10000000.00,6M
SWAP,2001,party1,Party 1,2008/05/14,REC,FLOAT,CAD,BA,6M,30/360,0.04,2008/07/17,2009/01/17,2008/07/17,2009/01/17,10000000.00,6M
SWAP,2002,party2,Party 2,2008/05/14,PAY,FIXED,CAD,BA,6M,30/360,0.04,2008/07/17,2009/01/17,2008/07/17,2009/01/17,10000000.00,6M
SWAP,2002,party2,Party 2,2008/05/14,R,FLOAT,CAD,BA,6M,30/360,0.04,2008/07/17,2009/01/17,2008/07/17,2009/01/17,10000000.00,6M
Note that one of the records, the receive side for the trade with trade_id 2002, has an invalid value for pay_receive_code, the value has been mistakenly entered as "R" instead of "REC". Bank A wants to validate this field, and if it is wrong for either side of the swap, Bank A wants to reject both sides, and write them both with an error message to a discard file.
Bank A wants the XML output to look as follows.
Figure 45. Output XML file swap.xml
<?xml version="1.0" encoding="UTF-8"?>
<dataDocument xmlns="http://www.fpml.org/FpML-5-0/reporting"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
fpmlVersion="5-0"
xsi:schemaLocation="http://www.fpml.org/FpML-5-0/reporting ../fpml-main-5-0.xsd">
<trade>
<tradeHeader>
<partyTradeIdentifier>
<tradeId tradeIdScheme="http://www.banka.com/ird/trade-id">2001</tradeId>
</partyTradeIdentifier>
<tradeDate>2008-05-14</tradeDate>
</tradeHeader>
<swap>
<swapStream>
<payerPartyReference href="bankA"/>
<calculationPeriodDates id="floatingCalcPeriodDates">
<effectiveDate>
<adjustedDate>2008-07-17</adjustedDate>
</effectiveDate>
<terminationDate>
<adjustedDate>2009-01-17</adjustedDate>
</terminationDate>
<calculationPeriodFrequency>
<periodMultiplier>6</periodMultiplier>
<period>M</period>
</calculationPeriodFrequency>
</calculationPeriodDates>
<calculationPeriodAmount>
<calculation>
<notionalSchedule>
<notionalStepSchedule>
<initialValue>10000000.00</initialValue>
<currency currencyScheme="http://www.fpml.org/ext/iso4217">CAD</currency>
</notionalStepSchedule>
</notionalSchedule>
<fixedRateSchedule>
<initialValue>0.04</initialValue>
</fixedRateSchedule>
<dayCountFraction>30/360</dayCountFraction>
</calculation>
</calculationPeriodAmount>
</swapStream>
<swapStream>
<payerPartyReference href="party1"/>
<calculationPeriodDates id="floatingCalcPeriodDates">
<effectiveDate>
<adjustedDate>2008-07-17</adjustedDate>
</effectiveDate>
<terminationDate>
<adjustedDate>2009-01-17</adjustedDate>
</terminationDate>
<calculationPeriodFrequency>
<periodMultiplier>6</periodMultiplier>
<period>M</period>
</calculationPeriodFrequency>
</calculationPeriodDates>
<calculationPeriodAmount>
<calculation>
<notionalSchedule>
<notionalStepSchedule>
<initialValue>10000000.00</initialValue>
<currency currencyScheme="http://www.fpml.org/ext/iso4217">CAD</currency>
</notionalStepSchedule>
</notionalSchedule>
<floatingRateCalculation>
<floatingRateIndex>BA</floatingRateIndex>
<indexTenor>
<periodMultiplier>6</periodMultiplier>
<period>M</period>
</indexTenor>
</floatingRateCalculation>
<dayCountFraction>30/360</dayCountFraction>
</calculation>
</calculationPeriodAmount>
</swapStream>
</swap>
</trade>
<party id="bankA">
<partyId>Bank A</partyId>
</party>
<party id="party1">
<partyId>Party 1</partyId>
</party>
<party id="party2">
<partyId>Party 2</partyId>
</party>
</dataDocument>
Note that the document concludes with a unique list of parties that
are referenced by
id
in the
payerPartyReference
elements.
The required resources script is
Figure 46. Resources script resources-swap.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:fn="http://www.w3.org/2005/xpath-functions">
<sx:service id="swap-to-xml">
<sx:serialize>
<sx:transform>
<sx:content ref="swap-xml"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="swap-xml">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="swap-flat-file"/>
</sx:flatFileReader>
<sx:modifyRecord>
<sx:newField name="trade_id" select="legs/child::node()[1]/trade_id"/>
<sx:newField name="trade_date" select="legs/child::node()[1]/trade_date"/>
</sx:modifyRecord>
<sx:recordValidator ref="swap-validator"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<sx:splitRecord recordType="swap" repeatingGroup="legs"/>
<sx:flatFileWriter>
<sx:fileSink file="output/swap-error.csv"/>
</sx:flatFileWriter>
</sx:discardHandler>
</sx:recordStream>
<sx:recordMapping ref="swap-to-xml-mapping"/>
</sx:recordContent>
<sx:flatFile id="swap-flat-file">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:compositeFlatRecordType>
<sx:delimitedField name="record_type"/>
<sx:delimitedField name="trade_id"/>
<sx:combinePhysicalRecords recordType="swap"
repeatingGroup="legs"
endTest="sx:current//trade_id != sx:previous//trade_id">
<sx:flatRecordTypeChoice>
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="record_type"/>
<sx:delimitedField name="trade_id"/>
<sx:delimitedField name="client_id"/>
<sx:delimitedField name="client_name"/>
<sx:delimitedField name="trade_date"/>
<sx:delimitedField name="pay_receive_code"/>
<sx:delimitedField name="fixed_float_code"/>
<sx:when test="record_type='SWAP' and fixed_float_code='FLOAT'">
<sx:flatRecordType name='swap_floating_leg'>
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="record_type"/>
<sx:delimitedField name="trade_id"/>
<sx:delimitedField name="client_id"/>
<sx:delimitedField name="client_name"/>
<sx:delimitedField name="trade_date"/>
<sx:delimitedField name="pay_receive_code"/>
<sx:delimitedField name="fixed_float_code"/>
<sx:delimitedField name="currency"/>
<sx:delimitedField name="index"/>
<sx:delimitedField name="tenor"/>
<sx:delimitedField name="day_count_basis"/>
<sx:delimitedField name="fixed_rate"/>
<sx:delimitedField name="start_date"/>
<sx:delimitedField name="maturity_date"/>
<sx:delimitedField name="reset_date"/>
<sx:delimitedField name="payment_date"/>
<sx:delimitedField name="notional"/>
<sx:delimitedField name="calculation_period"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="record_type='SWAP' and fixed_float_code='FIXED'">
<sx:flatRecordType name='swap_fixed_leg'>
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="record_type"/>
<sx:delimitedField name="trade_id"/>
<sx:delimitedField name="client_id"/>
<sx:delimitedField name="client_name"/>
<sx:delimitedField name="trade_date"/>
<sx:delimitedField name="pay_receive_code"/>
<sx:delimitedField name="fixed_float_code"/>
<sx:delimitedField name="currency"/>
<sx:delimitedField name="index"/>
<sx:delimitedField name="tenor"/>
<sx:delimitedField name="day_count_basis"/>
<sx:delimitedField name="fixed_rate"/>
<sx:delimitedField name="start_date"/>
<sx:delimitedField name="maturity_date"/>
<sx:delimitedField name="reset_date"/>
<sx:delimitedField name="payment_date"/>
<sx:delimitedField name="notional"/>
<sx:delimitedField name="calculation_period"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:combinePhysicalRecords>
</sx:compositeFlatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordValidator id="swap-validator"
recordType="swap"
message="Error in trade {trade_id}.">
<sx:fieldValidator field="legs"
message="">
<sx:recordValidator>
<sx:fieldValidator field="pay_receive_code"
message="Invalid pay_receive_code {pay_receive_code}.">
<sx:or>
<sx:valueRestriction pattern="PAY"/>
<sx:valueRestriction pattern="REC"/>
</sx:or>
</sx:fieldValidator>
<sx:fieldValidator field="fixed_float_code"
message="Invalid fixed_float_code {fixed_float_code}.">
<sx:or>
<sx:valueRestriction pattern="FIXED"/>
<sx:valueRestriction pattern="FLOAT"/>
</sx:or>
</sx:fieldValidator>
<sx:fieldValidator field="start_date"
message="Invalid start_date {start_date}.">
<sx:valueRestriction pattern="(19|20)\d\d[/](0[1-9]|1[012])[/](0[1-9]|[12][0-9]|3[01])"/>
</sx:fieldValidator>
<sx:fieldValidator field="maturity_date"
message="Invalid maturity_date {maturity_date}.">
<sx:valueRestriction pattern="(20|21)\d\d[/](0[1-9]|1[012])[/](0[1-9]|[12][0-9]|3[01])"/>
</sx:fieldValidator>
</sx:recordValidator>
</sx:fieldValidator>
</sx:recordValidator>
<sx:recordMapping id="swap-to-xml-mapping">
<dataDocument xmlns="http://www.fpml.org/FpML-5-0/reporting"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
fpmlVersion="5-0"
xsi:schemaLocation="http://www.fpml.org/FpML-5-0/reporting ../fpml-main-5-0.xsd">
<sx:groupBy fields="trade_id">
<trade>
<tradeHeader>
<partyTradeIdentifier>
<sx:fieldElementMap field="trade_id" element="tradeId">
<sx:fieldAttributeMap value="http://www.banka.com/ird/trade-id"
attribute="tradeIdScheme"/>
</sx:fieldElementMap>
</partyTradeIdentifier>
<tradeDate>
<sx:toXmlDate fromFormat="yyyy/MM/dd">
<sx:toString value="{trade_date}"/>
</sx:toXmlDate>
</tradeDate>
</tradeHeader>
<swap>
<sx:subrecordMapping repeatingGroup="legs">
<sx:groupBy fields="trade_id pay_receive_code">
<swapStream>
<sx:choose>
<sx:when test="pay_receive_code='PAY'">
<payerPartyReference>
<sx:fieldAttributeMap value="bankA"
attribute="href"/>
</payerPartyReference>
</sx:when>
<sx:when test="pay_receive_code='REC'">
<payerPartyReference>
<sx:fieldAttributeMap field="client_id"
attribute="href"/>
</payerPartyReference>
</sx:when>
</sx:choose>
<calculationPeriodDates id="floatingCalcPeriodDates">
<effectiveDate>
<adjustedDate>
<sx:toXmlDate fromFormat="yyyy/MM/dd">
<sx:toString value="{start_date}"/>
</sx:toXmlDate>
</adjustedDate>
</effectiveDate>
<terminationDate>
<adjustedDate>
<sx:toXmlDate fromFormat="yyyy/MM/dd">
<sx:toString value="{maturity_date}"/>
</sx:toXmlDate>
</adjustedDate>
</terminationDate>
<calculationPeriodFrequency>
<sx:parameter name="term" value="{calculation_period}"/>
<sx:choose ref="expand-term"/>
</calculationPeriodFrequency>
</calculationPeriodDates>
<calculationPeriodAmount>
<calculation>
<notionalSchedule>
<notionalStepSchedule>
<sx:fieldElementMap field="notional" element="initialValue"/>
<sx:fieldElementMap field="currency" element="currency">
<sx:fieldAttributeMap value="http://www.fpml.org/ext/iso4217"
attribute="currencyScheme"/>
</sx:fieldElementMap>
</notionalStepSchedule>
</notionalSchedule>
<sx:choose>
<sx:when test="fixed_float_code='FLOAT'">
<floatingRateCalculation>
<sx:fieldElementMap field="index"
element="floatingRateIndex"/>
<indexTenor>
<sx:parameter name="term" value="{tenor}"/>
<sx:choose ref="expand-term"/>
</indexTenor>
</floatingRateCalculation>
</sx:when>
<sx:when test="fixed_float_code='FIXED'">
<fixedRateSchedule>
<sx:fieldElementMap field="fixed_rate"
element="initialValue"/>
</fixedRateSchedule>
</sx:when>
</sx:choose>
<sx:fieldElementMap field="day_count_basis"
element="dayCountFraction"/>
</calculation>
</calculationPeriodAmount>
</swapStream>
</sx:groupBy>
</sx:subrecordMapping>
</swap>
</trade>
</sx:groupBy>
<sx:nestedContent>
<sx:recordContent ref="clients"/>
</sx:nestedContent>
</dataDocument>
</sx:recordMapping>
<!-- Expands a parameter named "term" with a value <number><D|M|Y> as
<periodMultiplier>number</periodMultiplier>
<period>D|M|Y</period>
-->
<sx:choose id="expand-term">
<sx:when test="fn:ends-with($term,'D')">
<sx:fieldElementMap element="periodMultiplier">
<sx:toString select="fn:substring-before($term,'D')"/>
</sx:fieldElementMap>
<period>D</period>
</sx:when>
<sx:when test="fn:ends-with($term,'M')">
<sx:fieldElementMap element="periodMultiplier">
<sx:toString select="fn:substring-before($term,'M')"/>
</sx:fieldElementMap>
<period>M</period>
</sx:when>
<sx:when test="fn:ends-with($term,'Y')">
<sx:fieldElementMap element="periodMultiplier">
<sx:toString select="fn:substring-before($term,'Y')"/>
</sx:fieldElementMap>
<period>Y</period>
</sx:when>
</sx:choose>
<sx:recordContent id="clients">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile>
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType>
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="record_type"/>
<sx:delimitedField name="trade_id"/>
<sx:delimitedField name="client_id"/>
<sx:delimitedField name="client_name"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:flatFileReader>
<sx:recordProjection fields="client_id client_name"/>
</sx:recordStream>
<sx:recordMapping>
<party id="bankA">
<partyId>Bank A</partyId>
</party>
<sx:onRecord>
<party>
<sx:fieldAttributeMap field="client_id" attribute="id"/>
<sx:fieldElementMap field="client_name" element="partyId"/>
</party>
</sx:onRecord>
</sx:recordMapping>
</sx:recordContent>
</sx:resources>
Note the following points about this script.
We use the sx:combineRecords element to compose a single record from the two legs of the swap. This way, when we validate the composite record, a failure in either of the legs will result in the entire swap being discarded, not just one of the legs.
We use the
sx:nestedContent
element to insert a list of parties from the
swap.csv
file into the output XML, and we use the
sx:recordProjection
element to eliminate duplicate pairs of client_id and client_name.
Inside the sx:discardHandler element, we use the sx:splitRecord element to decompose the composite swap record back into separate leg records. This allows us to easily use the default record writer to write the legs to a discard file.
We have two fields,
calculation_period
and
tenor
, that have values like
3M
and
30D
, where the numeric value and the trailing letter need to
be mapped separately to the elements
periodMultiplier
and
period
. We use a parameterized
sx:choose
instruction to handle both cases.
You can run this example on the command line by entering
servingxml -i data/swap.csv -o output/swap.xml -r resources-swap.xml swap-to-xml
The following error message will be written to the console (or log file.)
25-Sep-2008 10:10:54 PM com.servingxml.util.system.DefaultLogger error SEVERE: Error in record "swap" on line 5. Error in trade 2002. Invalid pay_receive_code R.
The example below shows how to convert a flat file to XML.
The input file is a positional file.
The requirements are as follows:
The desired output is shown below.
Figure 48. Output XML file Recs.xml
<?xml version="1.0" encoding="utf-8"?>
<Recs>
<Rec>
<First>SOUTH</First>
<Second>TEST</Second>
<Third>01</Third>
<Last>10</Last>
</Rec>
<Rec>
<First>WEST</First>
<Second>TEST</Second>
<Third>01</Third>
<Last>12</Last>
</Rec>
<Rec>
<First>SOUTH</First>
<Second>TEST</Second>
<Third>02</Third>
<Last>80</Last>
</Rec>
</Recs>
The following resources script does the transformation.
Figure 49. Resources script resources-abtest.xml
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="abtest">
<sx:serialize>
<sx:transform>
<sx:content ref="data"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="data">
<sx:flatFileReader>
<sx:urlSource url="data/abtest.txt"/>
<sx:flatFile ref="flatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="abtestToXml"/>
</sx:recordContent>
<sx:flatFile id="flatFile">
<sx:flatFileBody>
<sx:flatRecordType name="match">
<sx:positionalField name="first" width="2"/>
<sx:positionalField name="second" width="4"/>
<sx:positionalField name="third" width="2"/>
<sx:positionalField name="fourth" width="2"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="abtestToXml">
<Recs>
<sx:onRecord>
<Rec>
<sx:fieldElementMap match="first[text()='AB']" select="'SOUTH'" element="First"/>
<sx:fieldElementMap match="first[text()='AC']" select="'WEST'" element="First"/>
<sx:fieldElementMap field="second" element="Second"/>
<sx:fieldElementMap match="third[text()='CD']" select="'01'" element="Third"/>
<sx:fieldElementMap match="third[text()!='CD']" select="'02'" element="Third"/>
<sx:fieldElementMap field="fourth" element="Last"/>
</Rec>
</sx:onRecord>
</Recs>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-abtest.xml -o output/abtest.xml abtest
Suppose you have the following positional file of persons and their addresses.
Figure 50. Input flat file persons.txt
PersonId Name FirstName Street Postcode CityTown
1 Jones Joe 215 Walmer Rd M5R3P7 Toronto
1 Jones Joe 180 Redwood ST 94102-3280 San Francisco
2 Davis Ken 212 Harbord St M5S1H6 Toronto
3 Morris Jane 1100 Danforth Ave M4J1N2 Toronto
3 Morris Jane 6 Green St 94111-1402 San Francisco
Each line containes information about a person and an address. A person may have multiple lines indicating multiple addresses.
You want the XML output to look as follows.
Figure 51. Output XML file persons.xml
<?xml version="1.0" encoding="utf-8"?>
<Persons-Addresses>
<Persons>
<Person>
<PersonId>1</PersonId>
<Name>Jones</Name>
<FirstName>Joe</FirstName>
</Person>
<Person>
<PersonId>2</PersonId>
<Name>Davis</Name>
<FirstName>Ken</FirstName>
</Person>
<Person>
<PersonId>3</PersonId>
<Name>Morris</Name>
<FirstName>Jane</FirstName>
</Person>
</Persons>
<Addresses>
<Address>
<PersonId>1</PersonId>
<Street>215 Walmer Rd</Street>
<PostCode>M5R3P7</PostCode>
<CityTown>Toronto</CityTown>
</Address>
<Address>
<PersonId>1</PersonId>
<Street>180 Redwood ST</Street>
<PostCode>94102-3280</PostCode>
<CityTown>San Francisco</CityTown>
</Address>
<Address>
<PersonId>2</PersonId>
<Street>212 Harbord St</Street>
<PostCode>M5S1H6</PostCode>
<CityTown>Toronto</CityTown>
</Address>
<Address>
<PersonId>3</PersonId>
<Street>1100 Danforth Ave</Street>
<PostCode>M4J1N2</PostCode>
<CityTown>Toronto</CityTown>
</Address>
<Address>
<PersonId>3</PersonId>
<Street>6 Green St</Street>
<PostCode>94111-1402</PostCode>
<CityTown>San Francisco</CityTown>
</Address>
</Addresses>
</Persons-Addresses>
There are two ways to do this:
Use a record mapping section with two sx:groupBy sections. In this case the implementation will buffer the records in memory until the first group is finished, and only then do the second.
Within a record mapping section, read the flat file twice, first inserting the person data content, and then inserting the address content.
Taking the first approach, the resources script looks like this:
Figure 52. Resources script resources-persons1.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="personsAddresses">
<sx:serialize>
<sx:transform>
<sx:content ref="persons"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:flatFile id="personsData">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType ref="persons"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordContent id="persons">
<sx:flatFileReader>
<sx:flatFile ref="personsData"/>
</sx:flatFileReader>
<sx:recordMapping ref="personsAddressesMapping"/>
</sx:recordContent>
<sx:flatRecordType id="persons" name="persons">
<sx:positionalField name="PersonId" width="9"/>
<sx:positionalField name="Name" width="11"/>
<sx:positionalField name="FirstName" width="16"/>
<sx:positionalField name="Street" width="20"/>
<sx:positionalField name="PostCode" width="12"/>
<sx:positionalField name="CityTown" width="20" />
</sx:flatRecordType>
<sx:recordMapping id="personsAddressesMapping">
<Persons-Addresses>
<Persons>
<sx:groupBy fields="PersonId">
<Person>
<sx:fieldElementMap field="PersonId" element="PersonId"/>
<sx:fieldElementMap field="Name" element="Name"/>
<sx:fieldElementMap field="FirstName" element="FirstName"/>
<sx:onRecord/>
</Person>
</sx:groupBy>
</Persons>
<Addresses>
<sx:onRecord>
<Address>
<sx:fieldElementMap field="PersonId" element="PersonId"/>
<sx:fieldElementMap field="Street" element="Street"/>
<sx:fieldElementMap field="PostCode" element="PostCode"/>
<sx:fieldElementMap field="CityTown" element="CityTown"/>
</Address>
</sx:onRecord>
</Addresses>
</Persons-Addresses>
</sx:recordMapping>
</sx:resources>
Taking the second approach, the resources script looks like this:
Figure 53. Resources script resources-persons2.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="personsAddresses">
<sx:serialize>
<sx:transform>
<sx:content ref="persons"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:flatFile id="personsData">
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType ref="persons"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordContent id="persons">
<sx:recordMapping ref="personsAddressesMapping"/>
</sx:recordContent>
<sx:flatRecordType id="persons" name="persons">
<sx:positionalField name="PersonId" width="9"/>
<sx:positionalField name="Name" width="11"/>
<sx:positionalField name="FirstName" width="16"/>
<sx:positionalField name="Street" width="20"/>
<sx:positionalField name="PostCode" width="12"/>
<sx:positionalField name="CityTown" width="20" />
</sx:flatRecordType>
<sx:recordMapping id="personsAddressesMapping">
<Persons-Addresses>
<sx:nestedContent>
<sx:recordContent>
<sx:flatFileReader>
<sx:flatFile ref="personsData"/>
</sx:flatFileReader>
<sx:recordMapping ref="personsMapping"/>
</sx:recordContent>
</sx:nestedContent>
<sx:nestedContent>
<sx:recordContent>
<sx:flatFileReader>
<sx:flatFile ref="personsData"/>
</sx:flatFileReader>
<sx:recordMapping ref="addressesMapping"/>
</sx:recordContent>
</sx:nestedContent>
</Persons-Addresses>
</sx:recordMapping>
<sx:recordMapping id="personsMapping">
<Persons>
<sx:groupBy fields="PersonId">
<Person>
<sx:fieldElementMap field="PersonId" element="PersonId"/>
<sx:fieldElementMap field="Name" element="Name"/>
<sx:fieldElementMap field="FirstName" element="FirstName"/>
<sx:onRecord/>
</Person>
</sx:groupBy>
</Persons>
</sx:recordMapping>
<sx:recordMapping id="addressesMapping">
<Addresses>
<sx:onRecord>
<Address>
<sx:fieldElementMap field="PersonId" element="PersonId"/>
<sx:fieldElementMap field="Street" element="Street"/>
<sx:fieldElementMap field="PostCode" element="PostCode"/>
<sx:fieldElementMap field="CityTown" element="CityTown"/>
</Address>
</sx:onRecord>
</Addresses>
</sx:recordMapping>
</sx:resources>
Note the following:
You can run the two examples on the command line by entering
servingxml -i data/persons.txt -o output/persons1.xml -r resources-persons1.xml
personsAddresses
servingxml -i data/persons.txt -o output/persons2.xml -r resources-persons2.xml
personsAddresses
The example below shows how to process
all the files in the
data
directory matching the regular expression "books.*[.]txt",
convert them from positional flat file formats to XML files, and write them out to the
output
directory.
As before, the input is a directory of files.
Figure 54. Directory containing input files
[.] countries.csv multivalued-field.csv
[..] countries.xsd plans.txt
3545_JH4DA3_4_H_.xml country-record.xsd tasks.csv
bad-countries.csv exotics.txt timesheets.csv
books.20040613.txt hot-record.xsd trades.txt
books.20040802.txt hot-record.xsx
books.txt hot.txt
books.xml messages.properties
We want to take the positional files whose names match the regular expression
"(books.*)[.]txt"
and convert them all to XML files.
The following resources script does the transformation.
Figure 56. Resources script resources-allbooks.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="all-books">
<sx:recordStream>
<sx:directoryReader directory="data">
<sx:fileFilter pattern="books.*[.]txt"/>
</sx:directoryReader>
<sx:processRecord>
<sx:parameter name="output-file">
<sx:findAndReplace searchFor ="(books.*)[.]txt" replaceWith ="$1.xml">
<sx:toString value="{name}"/>
</sx:findAndReplace>
</sx:parameter>
<sx:serialize>
<sx:xsltSerializer>
<sx:fileSink directory="output" file="{$output-file}"/>
</sx:xsltSerializer>
<sx:transform>
<sx:content ref="books"/>
</sx:transform>
</sx:serialize>
</sx:processRecord>
</sx:recordStream>
</sx:service>
<sx:recordContent id="books">
<sx:flatFileReader>
<sx:fileSource directory="{parentDirectory}" file="{name}"/>
<sx:flatFile ref="booksFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="booksToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="booksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="bookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="bookType" name="bookType">
<sx:positionalField name="category" label="Category" width="1"/>
<sx:positionalField name="author" label="Author" width="30"/>
<sx:positionalField name="title" label="Title" width="30"/>
<sx:positionalField name="price" label="Price" width="10" justify="right"/>
</sx:flatRecordType>
<sx:recordMapping id="booksToXmlMapping">
<myns:books xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="url2">
<sx:onRecord>
<myns:book>
<sx:fieldAttributeMap field="category" attribute="categoryCode"/>
<sx:fieldElementMap field="title" element="myns:title"/>
<sx:fieldElementMap field="author" element="myns:author"/>
<sx:fieldElementMap field="price" element="myns:price"/>
</myns:book>
</sx:onRecord>
</myns:books>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-allbooks.xml all-books
The example below shows how to process one flat file and produce multiple XML output files.
The input file is shown below.
Figure 57. Books input file
CAuthor Title Price ISBN
FCharles Bukowski Factotum 22.95 0876852630
FSergei Lukyanenko The Night Watch 17.99 0434014125
FAndrew Crumey Mr Mee 22.00 0312282354
CSteven John Metsker Building Parsers with Java 39.95 0201719622
This is a trailer record
We want to produce multiple files like the one shown below, with filenames differentiated by the isbn number.
Figure 58. Book XML output file book-0876852630.xml.
<?xml version="1.0" encoding="UTF-8"?>
<myns:book xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
categoryCode="F">
<myns:title>Factotum</myns:title>
<myns:author>Charles Bukowski</myns:author>
<myns:price>22.95</myns:price>
<myns:isbn>0876852630</myns:isbn>
</myns:book>
The following resources script does the transformation.
Figure 59. Resources script resources-books_multiple.xml
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/" >
<sx:service id="books">
<sx:transform>
<sx:content ref="books"/>
<sx:processSubtree path="/myns:books/myns:book">
<sx:parameter name="isbn" select="myns:isbn"/>
<sx:serialize>
<sx:xsltSerializer>
<sx:fileSink directory="output" file="book-{$isbn}.xml"/>
</sx:xsltSerializer>
</sx:serialize>
</sx:processSubtree>
</sx:transform>
</sx:service>
<!--
This sx:flatFileReader element does not specify a stream source, so
the source will default to the file specified with the -i option on the command line.
-->
<sx:recordContent id="books">
<sx:flatFileReader>
<sx:flatFile ref="booksFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="booksToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="booksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="bookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="bookType" name="bookType">
<sx:positionalField name="category" label="Category" width="1"/>
<sx:positionalField name="author" label="Author" width="30"/>
<sx:positionalField name="title" label="Title" width="30"/>
<sx:positionalField name="price" label="Price" width="10" justify="right"/>
<sx:positionalField name="isbn" label="ISBN" start="73" width="10"/>
</sx:flatRecordType>
<sx:recordMapping id="booksToXmlMapping">
<myns:books xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="url2">
<sx:onRecord>
<myns:book>
<sx:fieldAttributeMap field="category" attribute="categoryCode"/>
<sx:fieldElementMap field="title" element="myns:title"/>
<sx:fieldElementMap field="author" element="myns:author"/>
<sx:fieldElementMap field="price" element="myns:price"/>
<sx:fieldElementMap field="isbn" element="myns:isbn"/>
</myns:book>
</sx:onRecord>
</myns:books>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-books_multiple.xml -i data/books.txt books
The example below shows how to process one flat file and produce multiple XML documents with default namespace prefixes.
The input file is the same as in the previous example.
Figure 60. Books input file
CAuthor Title Price ISBN
FCharles Bukowski Factotum 22.95 0876852630
FSergei Lukyanenko The Night Watch 17.99 0434014125
FAndrew Crumey Mr Mee 22.00 0312282354
CSteven John Metsker Building Parsers with Java 39.95 0201719622
This is a trailer record
This time we want to produce multiple files like the one shown below, with a default namespace prefix.
Figure 61. Book XML output file book-with-default-ns-0156028972.xml.
<?xml version="1.0" encoding="UTF-8"?>
<book xmlns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
categoryCode="F">
<title>Gun, with Occasional Music</title>
<author>Jonathan Lethem</author>
<price>17.99</price>
<isbn>0156028972</isbn>
</book>
The following resources script does the transformation.
Figure 62. Resources script resources-multiple_default_ns.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="books">
<sx:transform>
<sx:content ref="books"/>
<sx:processSubtree path="/myns:books/myns:book">
<sx:parameter name="isbn" select="myns:isbn"/>
<sx:serialize>
<sx:xsltSerializer>
<sx:fileSink directory="output" file="book-with-default-ns-{$isbn}.xml"/>
<sx:outputProperty name="indent" value="yes"/>
</sx:xsltSerializer>
</sx:serialize>
</sx:processSubtree>
</sx:transform>
</sx:service>
<!--
This sx:flatFileReader element does not specify a stream source, so
the source will default to the file specified with the -i option on the command line.
-->
<sx:recordContent id="books">
<sx:flatFileReader>
<sx:flatFile ref="booksFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="booksToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="booksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="bookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="bookType" name="bookType">
<sx:positionalField name="category" label="Category" width="1"/>
<sx:positionalField name="author" label="Author" width="30"/>
<sx:positionalField name="title" label="Title" width="30"/>
<sx:positionalField name="price" label="Price" width="10" justify="right"/>
<sx:positionalField name="isbn" label="ISBN" start="73" width="10"/>
</sx:flatRecordType>
<sx:recordMapping id="booksToXmlMapping">
<books xmlns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="url2">
<sx:onRecord>
<book>
<sx:fieldAttributeMap field="category" attribute="myns:categoryCode"/>
<sx:fieldElementMap field="title" element="myns:title"/>
<sx:fieldElementMap field="author" element="myns:author"/>
<sx:fieldElementMap field="price" element="myns:price"/>
<sx:fieldElementMap field="isbn" element="myns:isbn"/>
</book>
</sx:onRecord>
</books>
</sx:recordMapping>
</sx:resources>
Note that the root element in the sx:recordMapping section contains a default namespace declaration, which will appear in the output XML, and which will have the effect that XML names belonging to the namespace "http://mycompany.com/mynames/" will be output without an explicit prefix.
Note that the
sx:fieldAttributeMap
and
sx:fieldElementMap
elements use the
myns
prefix to associate attribute and
element names with the namespace
"http://mycompany.com/mynames/". The prefix is resolved
against the nearest in-scope prefix declaration for
myns
, which happens to be the one in the
sx:resources
element.
Since this declaration does not appear inside the record mapping
section, it is used only to associate a name with a namespace, not
for serialization.
If a prefix were omitted in an attribute or element mapping,
say on
title
, that element would belong to no
namespace, all others would belong to the default namespace,
and the output would become
<book xmlns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
categoryCode="F">
<title xmlns="">Gun, with Occasional Music</title>
<author>Jonathan Lethem</author>
<price>17.99</price>
<isbn>0156028972</isbn>
</book>
You can run this example on the command line by entering
servingxml -r resources-multiple_default_ns.xml -i data/books.txt books
The example below shows how to convert a positional flat file with multi-byte encoding to XML.
The input file is a positional flat file containing the following hex UTF-8 byte input.
The two byte sequence C396 represents the single character Ö.
The desired output is shown below.
Figure 64. Output XML file doubleByte.xml
<?xml version="1.0" encoding="UTF-8" ?>
<document>
<documentData>
<param1>AA</param1>
<param2>BB</param2>
<param3>CC</param3>
</documentData>
<documentData>
<param1>AÖ</param1>
<param2>BB</param2>
<param3>CC</param3>
</documentData>
</document>
The following resources script does the transformation.
Figure 65. Resources script resources-doubleByte.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="convert">
<sx:serialize>
<sx:transform>
<sx:content ref="data"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="data">
<sx:flatFileReader>
<sx:flatFile ref="flatfile"/>
<sx:defaultStreamSource encoding="UTF-8"/>
</sx:flatFileReader>
<sx:recordMapping ref="recordmap"/>
</sx:recordContent>
<sx:flatFile id="flatfile">
<sx:flatFileBody>
<sx:flatRecordType ref="recordtype"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="recordtype">
<sx:positionalField name="param1" width="2"/>
<sx:positionalField name="param2" width="2"/>
<sx:positionalField name="param3" width="2"/>
</sx:flatRecordType>
<sx:recordMapping id="recordmap">
<document>
<sx:onRecord>
<documentData>
<sx:fieldElementMap element="param1" field="param1"/>
<sx:fieldElementMap element="param2" field="param2"/>
<sx:fieldElementMap element="param3" field="param3"/>
</documentData>
</sx:onRecord>
</document>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-doubleByte.xml -i data/doubleByte.dat
-o output/doubleByte.xml convert
The example below shows how to convert a flat file with multiple record formats to XML.
The input file is a positional flat file.
Figure 66. Input flat file trades.txt
TR000103/25/2005 1:50:00This is a trade record
TN0002X1234A child transaction
TN0003X1235Another child transaction
The desired output is shown below.
Figure 67. Output XML file trades.xml
<?xml version="1.0" encoding="UTF-8"?>
<trades feed="LONDON">
<trade id="0001">
<trade-date>2005-03-25T01:50:00.000-05:00</trade-date>
<description>This is a trade record</description>
<transaction id="0002">
<reference>X1234</reference>
<description>A child transaction</description>
</transaction>
<transaction id="0003">
<reference>X1235</reference>
<description>Another child transaction</description>
</transaction>
</trade>
</trades>
The following resources script does the transformation.
Figure 68. Resources script resources-trades.xml
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="trades">
<sx:parameter name="feed">
<sx:defaultValue>NY</sx:defaultValue>
</sx:parameter>
<sx:serialize>
<sx:transform>
<sx:content ref="trades"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="trades">
<sx:recordStream>
<sx:flatFileReader>
<sx:urlSource url="data/trades.txt"/>
<sx:flatFile ref="tradesFlatFile"/>
</sx:flatFileReader>
<sx:combineRecords recordType="composite" repeatingGroup="detail"
startTest="sx:current//record_type='TR'"
endTest="sx:current//record_type='TR'">
<sx:newField name="id" select="detail/trade/id"/>
</sx:combineRecords>
</sx:recordStream>
<sx:recordMapping ref="tradesToXmlMapping"/>
</sx:recordContent>
<sx:recordMapping id="tradesToXmlMapping">
<trades>
<sx:fieldAttributeMap value="{$feed}" attribute="feed"/>
<sx:onRecord recordType="composite">
<trade>
<sx:fieldAttributeMap field="id" attribute="id"/>
<sx:subrecordMapping repeatingGroup="detail">
<sx:onRecord recordType="trade">
<sx:elementMap element="trade-date">
<sx:convertToDateTime format="MM/dd/yyyy H:mm:ss">
<sx:concat separator=" ">
<sx:toString value="{trade_date}"/>
<sx:toString value="{trade_time}"/>
</sx:concat>
</sx:convertToDateTime>
</sx:elementMap>
<sx:fieldElementMap field="description" element="description"/>
</sx:onRecord>
<sx:onRecord recordType="transaction">
<transaction>
<sx:fieldAttributeMap field="id" attribute="id"/>
<sx:fieldElementMap field="reference" element="reference"/>
<sx:fieldElementMap field="description" element="description"/>
</transaction>
</sx:onRecord>
</sx:subrecordMapping>
</trade>
</sx:onRecord>
</trades>
</sx:recordMapping>
<sx:flatFile id="tradesFlatFile">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="record_type" width="2"/>
<sx:when test="record_type='TR'">
<sx:flatRecordType name="trade">
<sx:positionalField name="record_type" width="2"/>
<sx:positionalField name="id" width="4"/>
<sx:positionalField name="trade_date" width="10"/>
<sx:positionalField name="trade_time" width="8"/>
<sx:positionalField name="description" width="30"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="record_type='TN'">
<sx:flatRecordType name="transaction">
<sx:positionalField name="record_type" width="2"/>
<sx:positionalField name="id" width="4"/>
<sx:positionalField name="reference" width="5"/>
<sx:positionalField name="description" width="30"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
Note the following points about the content of the<sx:elementMap element="trade-date"> element.
You can run this example on the command line by entering
servingxml -r resources-trades.xml -o output/trades.xml trades feed=LONDON
The example below shows how to convert a flat file with header, detail, and summary records to XML.
The input file is a positional flat file with variable record layouts.
Figure 69. Input flat file CONV_in.dat
90112345678abcdefg1000001
100SURNAME1 GIVEN1 A
100SURNAME2 GIVEN2 B
3011111111AB111111
The desired output is shown below.
Figure 70. Output XML file CONV.xml
<?xml version="1.0" encoding="utf-8"?>
<Submission>
<Header>
<record_type>901</record_type>
<sbmt_ref_id>abcdefg</sbmt_ref_id>
<trnmtr_tcd>1</trnmtr_tcd>
<summ_cnt>000001</summ_cnt>
</Header>
<All>
<Level1>
<Detail>
<snm>SURNAME1</snm>
<gvn_nm>GIVEN1</gvn_nm>
<init>A</init>
</Detail>
<Detail>
<snm>SURNAME2</snm>
<gvn_nm>GIVEN2</gvn_nm>
<init>B</init>
</Detail>
<Summary>
<bn>1111111AB111111</bn>
</Summary>
</Level1>
</All>
</Submission>
The following resources script does the transformation.
Figure 71. Resources script resources-CONV.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="CONV">
<sx:serialize>
<sx:transform>
<sx:content ref="CONV"/>
<sx:removeEmptyElements elements="*"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="CONV">
<sx:flatFileReader>
<sx:urlSource url="data/CONV_in.dat"/>
<sx:flatFile ref="CONVFlatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="CONVToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="CONVFlatFile">
<sx:commentStarter value="#"/>
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="record_type" width="3"/>
<sx:when test="record_type='901'">
<sx:flatRecordType name="header">
<sx:positionalField name="record_type" width="3"/>
<sx:positionalField name="trnmtr_nbr" width="8"/>
<sx:positionalField name="sbmt_ref_id" width="7"/>
<sx:positionalField name="trnmtr_tcd" width="1"/>
<sx:positionalField name="summ_cnt" width="6"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="record_type='100'">
<sx:flatRecordType name="detail">
<sx:positionalField name="record_type" width="3"/>
<sx:positionalField name="snm" width="9"/>
<sx:positionalField name="gvn_nm" width="7"/>
<sx:positionalField name="init" width="1"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="record_type='301'">
<sx:flatRecordType name="summary">
<sx:positionalField name="record_type" width="3"/>
<sx:positionalField name="bn" width="15"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="CONVToXmlMapping">
<Submission>
<sx:onRecord recordType="header">
<Header>
<sx:fieldElementMap field="record_type" element="record_type"/>
<sx:fieldElementMap field="sbmt_ref_id" element="sbmt_ref_id"/>
<sx:fieldElementMap field="trnmtr_tcd" element="trnmtr_tcd"/>
<sx:fieldElementMap field="summ_cnt" element="summ_cnt"/>
</Header>
</sx:onRecord>
<sx:innerGroup startTest="sx:current/detail or sx:current/summary"
endTest="not(sx:current/detail or sx:current/summary)">
<All>
<Level1>
<sx:onRecord recordType='detail'>
<Detail>
<sx:fieldElementMap field="snm" element="snm"/>
<sx:fieldElementMap field="gvn_nm" element="gvn_nm"/>
<sx:fieldElementMap field="init" element="init"/>
</Detail>
</sx:onRecord>
<sx:onRecord recordType="summary">
<Summary>
<sx:fieldElementMap field="bn" element="bn"/>
</Summary>
</sx:onRecord>
</Level1>
</All>
</sx:innerGroup>
</Submission>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -o output/CONV.xml -r resources-CONV.xml CONV
This is the first of two examples that map the
hot.txt
flat file to XML.
The input file is a variable record positional file.
Figure 72. Input flat file hot.txt
BFH01value01
BCH02value02
BOH03value03
BKT06value04issue
BKP84value05A6USD2
BOT93value06
BOT94value07
BOH03value08
BKT06value09issue
BKP84value10A6CAD2
BOT93value11
BKT06value12refund
BKP84value13A6USD2
BOT93value14
BOT94value15
BCT95value16
BFT99value17
This file has variant record types. The first five characters identify the record type, and the next seven characters represent a hypothetical attribute. The BKT06 and BKP84 records have additional fields.
The desired output is shown below.
Figure 73. Output XML file hot1.xml
<?xml version="1.0" encoding="utf-8" ?>
<hot>
<BFH01 attr1="value01"/>
<BCH02 attr1="value02"/>
<BOH03 attr1="value03"/>
<BKT06 attr1="value04"/>
<BKP84 attr1="value05" amount="1.66" currency="USD"/>
<BOT93 attr1="value06"/>
<BOT94 attr1="value07"/>
<BOH03 attr1="value08"/>
<BKT06 attr1="value09"/>
<BKP84 attr1="value10" amount="2.46" currency="CAD"/>
<BOT93 attr1="value11"/>
<BKT06 attr1="value12"/>
<BKP84 attr1="value13" amount="2.27" currency="USD"/>
<BOT93 attr1="value14"/>
<BOT94 attr1="value15"/>
<BCT95 attr1="value16"/>
<BFT99 attr1="value17"/>
</hot>
The following resources script does the transformation.
Figure 74. flatfile-hot.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:flatFile id="hotFlatFile">
<sx:commentStarter value="#"/>
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="record-type" width="5"/>
<sx:when test="record-type='BKP84'">
<sx:flatRecordType name="bkp84">
<sx:positionalField name="record-type-prefix" width="3"/>
<sx:positionalField name="record-type" start="1" width="5"/>
<sx:positionalField name="value" width="7"/>
<sx:positionalField name="amount" width="2"/>
<sx:positionalField name="currency" width="3"/>
<sx:positionalField name="precision" width="1"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="record-type='BKT06'">
<sx:flatRecordType name="bkt06">
<sx:positionalField name="record-type-prefix" width="3"/>
<sx:positionalField name="record-type" start="1" width="5"/>
<sx:positionalField name="value" width="7"/>
<sx:positionalField name="type" width="6"/>
</sx:flatRecordType>
</sx:when>
<sx:otherwise>
<sx:flatRecordType name="other">
<sx:positionalField name="record-type-prefix" width="3"/>
<sx:positionalField name="record-type" start="1" width="5"/>
<sx:positionalField name="value" width="7"/>
</sx:flatRecordType>
</sx:otherwise>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
The values in the
bkp84
record need to be transformed as follows:
The amount field is encoded as a hexadecimal value and needs to be converted to a decimal value.
The resulting decimal value needs to be converted to the specified precision.
In the hot 1 example in this section, we transform the values using a sx:customRecordFilter element to define a custom record filter written in Java. In the hot 2 example in the next section, we take a different approach, transforming the values within an XPath select expression against the record fields, making use of extension calls to static Java functions.
Figure 75. XML Schema file hot-record.xsd
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- This element's name matches the value of the name attribute in the "bkp84" sx:flatFileRecordType. -->
<xs:element name="bkp84" type="bkp84Type"/>
<!-- This element's name matches the value of the name attribute in the "bkt06" sx:flatFileRecordType. -->
<xs:element name="bkt06" type="bkt06Type"/>
<!-- This element's name matches the value of the name attribute in the "other" sx:flatFileRecordType. -->
<xs:element name="other" type="otherType"/>
<xs:complexType name="bkp84Type">
<xs:sequence>
<xs:element name="record-type-prefix" type="xs:string"/>
<xs:element name="record-type" type="xs:string"/>
<xs:element name="value" type="xs:string"/>
<xs:element name="amount" type="xs:hexBinary"/>
<xs:element name="currency" type="xs:string"/>
<xs:element name="precision" type="xs:nonNegativeInteger"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="bkt06Type">
<xs:sequence>
<xs:element name="record-type-prefix" type="xs:string"/>
<xs:element name="record-type" type="xs:string"/>
<xs:element name="value" type="xs:string"/>
<xs:element name="type" type="xs:string"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="otherType">
<xs:sequence>
<xs:element name="record-type-prefix" type="xs:string"/>
<xs:element name="record-type" type="xs:string"/>
<xs:element name="value" type="xs:string"/>
</xs:sequence>
</xs:complexType>
</xs:schema>
Figure 76. Resources script resources-hot1.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:include href="flatfile-hot.xml"/>
<sx:service id="hot1">
<sx:serialize>
<sx:transform>
<sx:content ref="hot1"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="hot1">
<sx:flatFileReader>
<sx:flatFile ref="hotFlatFile"/>
</sx:flatFileReader>
<msv:recordValidator>
<sx:urlSource url="data/hot-record.xsd"/>
</msv:recordValidator>
<sx:customRecordFilter class="flat2xml.HotRecordFilter"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
</sx:discardHandler>
<sx:recordMapping ref="hot1ToXmlMapping"/>
</sx:recordContent>
<sx:recordMapping id="hot1ToXmlMapping">
<hot>
<sx:onRecord recordType="bkp84">
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
<sx:fieldAttributeMap field="calculatedAmount" attribute="amount"/>
<sx:fieldAttributeMap field="currency" attribute="currency"/>
</sx:elementMap>
</sx:onRecord>
<sx:onRecord recordType="bkt06">
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
<sx:onRecord recordType="other">
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
</hot>
</sx:recordMapping>
</sx:resources>
Figure 77. HotRecordFilter class
package flat2xml;
import com.servingxml.app.ServiceContext;
import com.servingxml.components.recordio.AbstractRecordFilter;
import com.servingxml.app.Flow;
import com.servingxml.util.Name;
import com.servingxml.util.QualifiedName;
import com.servingxml.util.ServingXmlException;
import com.servingxml.util.record.Record;
import com.servingxml.util.record.RecordBuilder;
public class HotRecordFilter extends AbstractRecordFilter {
private static final Name BKP84_RECORD_TYPE = new QualifiedName("bkp84");
private static final Name AMOUNT_NAME = new QualifiedName("amount");
private static final Name PRECISION_NAME = new QualifiedName("precision");
private static final Name CALCULATED_AMOUNT_NAME = new QualifiedName("calculatedAmount");
public void writeRecord(ServiceContext context, Flow flow)
{
Record record = flow.getRecord();
Flow newFlow = flow;
if (record.getRecordType().getName().equals(BKP84_RECORD_TYPE)) {
RecordBuilder recordBuilder = new RecordBuilder(record);
String amountString = record.getString(AMOUNT_NAME);
if (amountString == null) {
throw new ServingXmlException("amount is NULL");
}
String precisionString = record.getString(PRECISION_NAME);
if (precisionString == null) {
throw new ServingXmlException("precision is NULL");
}
int amount = Integer.parseInt(amountString,16);
int precision = Integer.parseInt(precisionString);
double calculatedAmount = (double)amount/Math.pow(10.0,(double)precision);
recordBuilder.setDouble(CALCULATED_AMOUNT_NAME,calculatedAmount);
record = recordBuilder.toRecord();
newFlow = flow.replaceRecord(context, record);
}
super.writeRecord(context, newFlow);
}
}
You can run this example on the command line by
HotRecordFilter
and copying
the resulting
.class
file into the
dir/classes
directory
servingxml -r resources-hot1.xml -i data/hot.txt -o output/hot1.xml hot1
This version of the hot example shows how to convert a flat file to XML document with outer and inner grouping. Each row in the flat file is validated with Sun's multi schema validator, and the row is discarded if an error is encountered.
This examples illustrates the
sx:innerGroup
and
sx:outerGroup
elements. Both elements use
the
startTest
and
endTest
attributes to
define the beginning and end of a group through XPath boolean expressions applied to records. They
differ in that
sx:innerGroup
will never allow any descendent
elements to be produced unless a group is recognized at its level, whereas a
sx:outerGroup
will only suppress content between this element and
the next succeeding grouping element.
The
resources-hot2.xml
file defines the record mapping for this version of
the hot example. The
BFH01
record type marks the beginning of a group of level 1,
the
BFT99
s mark the end. The
BCH02
s mark
the beginning of a group of level 2, the
BCT95
s mark the end.
The
BOH03
marks the beginning of a group of level 3, the
BOT94s
mark the end.
The desired output is shown below.
Figure 78. Output XML file hot2.xml
<?xml version="1.0" encoding="utf-8" ?>
<hot>
<BFH01 attr1="value01"/>
<periods>
<period>
<BCH02s>
<BCH02 attr1="value02"/>
</BCH02s>
<agents>
<agent>
<BOH03 attr1="value03"/>
<issues>
<issue>
<BKT06 attr1="value04"/>
<BKP84 attr1="value05" amount="1.66" currency="USD"/>
</issue>
<BOT93s>
<BOT93 attr1="value06"/>
</BOT93s>
</issues>
<BOT94s>
<BOT94 attr1="value07"/>
</BOT94s>
</agent>
<agent>
<BOH03 attr1="value08"/>
<issues>
<issue>
<BKT06 attr1="value09"/>
<BKP84 attr1="value10" amount="2.46" currency="CAD"/>
</issue>
<BOT93s>
<BOT93 attr1="value11"/>
</BOT93s>
</issues>
<refunds>
<refund>
<BKT06 attr1="value12"/>
<BKP84 attr1="value13" amount="2.27" currency="USD"/>
</refund>
<BOT93s>
<BOT93 attr1="value14"/>
</BOT93s>
</refunds>
<BOT94s>
<BOT94 attr1="value15"/>
</BOT94s>
</agent>
</agents>
<BCT95s>
<BCT95 attr1="value16"/>
</BCT95s>
</period>
</periods>
<BFT99s>
<BFT99 attr1="value17"/>
</BFT99s>
</hot>
The following resources script does the transformation.
Figure 79. Resources script resources-hot2.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv"
xmlns:integer="java.lang.Integer" xmlns:math="java.lang.Math">
<sx:include href="flatfile-hot.xml"/>
<sx:service id="hot2">
<sx:serialize>
<sx:transform>
<sx:content ref="hot2"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="hot2">
<sx:flatFileReader>
<sx:flatFile ref="hotFlatFile"/>
</sx:flatFileReader>
<msv:recordValidator>
<sx:urlSource url="data/hot-record.xsd"/>
</msv:recordValidator>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
</sx:discardHandler>
<sx:recordMapping ref="hot2ToXmlMapping"/>
</sx:recordContent>
<sx:recordMapping id="hot2ToXmlMapping">
<hot>
<sx:outerGroup startTest="sx:previous//record-type='BFH01'"
endTest="sx:current//record-type='BFT99'">
<periods>
<sx:outerGroup startTest="sx:current//record-type='BCH02'"
endTest="sx:previous//record-type='BCT95'">
<period>
<sx:outerGroup startTest="sx:previous//record-type='BCH02'"
endTest="sx:current//record-type='BCT95'">
<agents>
<sx:outerGroup startTest="sx:current//record-type='BOH03'"
endTest="sx:previous//record-type='BOT94'">
<agent>
<sx:groupChoice>
<sx:innerGroup startTest="sx:current//record-type-prefix='BFH'"
endTest="sx:previous//record-type-prefix='BFH'">
<sx:onRecord>
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current//record-type-prefix='BOH'"
endTest="sx:previous//record-type-prefix='BOH'">
<sx:onRecord>
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current//record-type-prefix='BFT'"
endTest="sx:previous//record-type-prefix='BFT'">
<sx:elementMap element="{record-type}s">
<sx:onRecord>
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
</sx:elementMap>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current//record-type-prefix='BCT'"
endTest="sx:previous//record-type-prefix='BCT'">
<sx:elementMap element="{record-type}s">
<sx:onRecord>
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
</sx:elementMap>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current//record-type='BOT94'"
endTest="sx:previous//record-type='BOT94'">
<sx:elementMap element="{record-type}s">
<sx:onRecord>
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
</sx:elementMap>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current//record-type='BCH02'"
endTest="sx:previous//record-type='BOH03'">
<sx:elementMap element="{record-type}s">
<sx:onRecord>
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
</sx:elementMap>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current//record-type='BKT06'"
endTest="sx:previous//record-type='BOT93'">
<sx:elementMap element="{type}s">
<sx:groupChoice>
<sx:innerGroup startTest="sx:current//record-type='BKT06'"
endTest="sx:previous//record-type='BKP84'">
<sx:elementMap element="{type}">
<sx:onRecord recordType="bkt06">
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
<sx:onRecord recordType="bkp84">
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
<sx:fieldAttributeMap select="integer:valueOf(amount,16) div math:pow(10,precision)" attribute="amount"/>
<sx:fieldAttributeMap field="currency" attribute="currency"/>
</sx:elementMap>
</sx:onRecord>
</sx:elementMap>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current//record-type-prefix='BOT'"
endTest="sx:previous//record-type-prefix='BOT'">
<sx:elementMap element="{record-type}s">
<sx:onRecord>
<sx:elementMap element="{record-type}">
<sx:fieldAttributeMap field="value" attribute="attr1"/>
</sx:elementMap>
</sx:onRecord>
</sx:elementMap>
</sx:innerGroup>
</sx:groupChoice>
</sx:elementMap>
</sx:innerGroup>
</sx:groupChoice>
</agent>
</sx:outerGroup>
</agents>
</sx:outerGroup>
</period>
</sx:outerGroup>
</periods>
</sx:outerGroup>
</hot>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-hot2.xml -i data/hot.txt -o output/hot2.xml hot2
The example below shows how to convert a flat file with variable field widths to XML.
The input file is a positional flat file.
Figure 80. Input flat file testWidth.txt
TX#00000000010016This is a remarkXXXXXXXXXX
TX#0000000002
TX#0000000003 YYYYYYYYYY
TX#00000000030004TestZZZZZZZZZZ
This file contains an optional positional field called "remarksText". The field size varies and when the field exists it's width is specified using the value of its preceding field called "remarksSize". When no "remarksText" exists, 4 space characters are specified as "remarksSize" in the file directly followed by the value of "followingField".
The desired output is shown below.
Figure 81. Output XML file testWidth.xml
<?xml version="1.0" encoding="UTF-8"?>
<Doc>
<Transaction>
<Reference>0000000001</Reference>
<Remarks>This is a remark</Remarks>
<FollowingField>XXXXXXXXXX</FollowingField>
</Transaction>
<Transaction>
<Reference>0000000002</Reference>
<Remarks/>
<FollowingField/>
</Transaction>
<Transaction>
<Reference>0000000003</Reference>
<Remarks/>
<FollowingField>YYYYYYYYYY</FollowingField>
</Transaction>
<Transaction>
<Reference>0000000003</Reference>
<Remarks>Test</Remarks>
<FollowingField>ZZZZZZZZZZ</FollowingField>
</Transaction>
</Doc>
The following resources script does the transformation.
Figure 82. Resources script resources-testWidth.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="testWidth">
<sx:serialize>
<sx:transform>
<sx:content ref="data"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="data">
<sx:flatFileReader>
<sx:flatFile ref="flatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="segmentsToXml"/>
</sx:recordContent>
<sx:flatFile id="flatFile">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="tag" width="3"/>
<sx:when test="tag='TX#'">
<sx:flatRecordType name="transaction">
<sx:positionalField name="recordType" width="3"/>
<sx:positionalField name="reference" width="10"/>
<sx:positionalField name="remarksSize" width="4">
<sx:defaultValue value="0"/>
</sx:positionalField>
<sx:positionalField name="remarksText" width="{remarksSize}"/>
<sx:positionalField name="followingField" width="10"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="segmentsToXml">
<Doc>
<sx:innerGroup startTest="sx:current/transaction">
<Transaction>
<sx:fieldElementMap field="reference" element="Reference"/>
<sx:fieldElementMap field="remarksText" element="Remarks"/>
<sx:fieldElementMap field="followingField" element="FollowingField"/>
</Transaction>
</sx:innerGroup>
</Doc>
</sx:recordMapping>
</sx:resources>
Note the following.
0
. The "remarksSize" positional field definition therefore
contains a
sx:defaultValue
element, which supplies the default.
You can run this example on the command line by entering
servingxml -r resources-testWidth.xml -i testWidth.txt -o output/testWidth.xml testWidth
The example below shows how to convert a flat file to XML with header information repeated when processing the other records.
The record input is
The desired output is shown below.
Figure 84. Output XML file repeated_header_info.xml
<doc>
<b attr="B">
<c attr="C">A</c>
</b>
<b attr="D">
<c attr="E">A</c>
</b>
</doc>
The following resources script does the transformation.
Figure 85. Resources script resources-repeatHeader.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="hut">
<sx:serialize>
<sx:transform>
<sx:content ref="HutContent"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="HutContent">
<sx:flatFileReader ref="HutFlatFileReader"/>
<sx:recordMapping ref="hut2ToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="HutFlatFile">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="record-type" start="1" width="1"/>
<sx:when test='record-type="1"'>
<sx:flatRecordType id="R1" name="R1">
<sx:positionalField name="TOTO" start="2" width="1"/>
</sx:flatRecordType>
</sx:when>
<sx:when test='record-type="2"'>
<sx:flatRecordType id="R2" name="R2">
<sx:positionalField name="TATA" start="2" width="1"/>
</sx:flatRecordType>
</sx:when>
<sx:when test='record-type="3"'>
<sx:flatRecordType id="R3" name="R3">
<sx:positionalField name="TITI" start="2" width="1"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatFileReader id="HutFlatFileReader">
<sx:inlineSource>1A
2B
3C
2D
3E</sx:inlineSource>
<sx:flatFile ref="HutFlatFile"/>
</sx:flatFileReader>
<sx:recordMapping id="hut2ToXmlMapping">
<doc>
<sx:parameter name="temp" value="{TOTO}"/>
<sx:outerGroup startTest="sx:current/R2">
<b>
<sx:fieldAttributeMap value="{TATA}" attribute="attr"/>
<sx:onRecord recordType="R3">
<sx:fieldElementMap value="{$temp}" element="c">
<sx:fieldAttributeMap value="{TITI}" attribute="attr"/>
</sx:fieldElementMap>
</sx:onRecord>
</b>
</sx:outerGroup>
</doc>
</sx:recordMapping>
</sx:resources>
Note the parameter declaration for "temp" in the record mapping
section. This parameter declaration occurs immediately below the
document element, which represents a grouping of all records. Within a
group, ServingXML ServingXML binds the parameter name to
the record that begins the grouping section, in this case the first
header record in the input.
You can run this example on the command line by entering
servingxml -r resources-repeatheader.xml -o output/repeated_header_info.xml hut
This shows how to convert fixed position records that contain repeatable/optional fixed parts identified by tags.
The input file is a positional file.
Note the following:
The desired output is shown below.
Figure 87. Output XML file segments.xml
<?xml version="1.0" encoding="utf-8"?>
<Recs>
<Rec>
<Z01>
<xxx>xxx</xxx>
<yyy>yyy</yyy>
</Z01>
<Z02>
<aaaa>aaaa</aaaa>
<bb>bb</bb>
<cc>cc</cc>
</Z02>
<Z02>
<aaaa>aaaa</aaaa>
<bb>bb</bb>
<cc>cc</cc>
</Z02>
<Z01>
<xxx>xxx</xxx>
<yyy>yyy</yyy>
</Z01>
</Rec>
</Recs>
The following resources script does the transformation.
Figure 88. Resources script resources-segments.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="segments">
<sx:serialize>
<sx:transform>
<sx:content ref="data"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="data">
<sx:flatFileReader>
<sx:urlSource url="data/segments.txt"/>
<sx:flatFile ref="flatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="segmentsToXml"/>
</sx:recordContent>
<sx:flatFile id="flatFile">
<sx:flatFileBody>
<sx:flatRecordType name="rec">
<sx:repeatingGroup name="segments">
<sx:flatRecordTypeChoice>
<sx:positionalField name="tag" width="3"/>
<sx:when test="tag='Z01'">
<sx:flatRecordType name="Z01">
<sx:positionalField name="tag" width="3"/>
<sx:positionalField name="xxx" width="3"/>
<sx:positionalField name="yyy" width="3"/>
</sx:flatRecordType>
</sx:when>
<sx:otherwise>
<sx:flatRecordType name="Z02">
<sx:positionalField name="tag" width="3"/>
<sx:positionalField name="aaaa" width="4"/>
<sx:positionalField name="bb" width="2"/>
<sx:positionalField name="cc" width="2"/>
</sx:flatRecordType>
</sx:otherwise>
</sx:flatRecordTypeChoice>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="segmentsToXml">
<Recs>
<sx:onRecord>
<Rec>
<sx:subrecordMapping repeatingGroup="segments">
<sx:onRecord recordType="Z01">
<Z01>
<sx:fieldElementMap field="xxx" element="xxx"/>
<sx:fieldElementMap field="yyy" element="yyy"/>
</Z01>
</sx:onRecord>
<sx:onRecord recordType="Z02">
<Z02>
<sx:fieldElementMap field="aaaa" element="aaaa"/>
<sx:fieldElementMap field="bb" element="bb"/>
<sx:fieldElementMap field="cc" element="cc"/>
</Z02>
</sx:onRecord>
</sx:subrecordMapping>
</Rec>
</sx:onRecord>
</Recs>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -o output/segments.xml -r resources-segments.xml
segments
Suppose you have the following positional file of students, their course grades, and their addresses.
The file has the following layout.
| name | 4 characters |
| subject-grade | repeating group, repeats twice |
| year-born | 4 characters |
| favorite-color | 4 characters |
| address | repeating group, repeats twice |
You want the XML output to look as follows.
Figure 90. Output XML file students-fix.xml
<?xml version="1.0" encoding="utf-8"?>
<StudentGrades>
<StudentGrade>
<Name>JANE</Name>
<SubjectGrade>
<Subject>ENGL</Subject>
<Grade>C-</Grade>
</SubjectGrade>
<SubjectGrade>
<Subject>MATH</Subject>
<Grade>A+</Grade>
</SubjectGrade>
<YearBorn>1972</YearBorn>
<FavoriteColor>BLUE</FavoriteColor>
<Address>
<City>CHICAGO</City>
<State>IL</State>
</Address>
<Address>
<City>ATLANTA</City>
<State>GA</State>
</Address>
</StudentGrade>
</StudentGrades>
The required resources script is
Figure 91. Resources script resources-students-fix.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="students">
<sx:serialize>
<sx:transform>
<sx:content ref="students-content"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="students-content">
<sx:flatFileReader>
<sx:flatFile ref="students-file"/>
</sx:flatFileReader>
<sx:recordMapping ref="students-mapping"/>
</sx:recordContent>
<sx:flatFile id="students-file">
<sx:flatFileBody>
<sx:flatRecordType name="student">
<sx:positionalField name="name" width="4"/>
<sx:repeatingGroup name="grades" count="2">
<sx:flatRecordType name="subject-grade">
<sx:positionalField name="subject" width="4"/>
<sx:positionalField name="grade" width="2"/>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:positionalField name="year-born" width="4"/>
<sx:positionalField name="favorite-color" width="4"/>
<sx:repeatingGroup name="addresses" count="2">
<sx:flatRecordType name="address">
<sx:positionalField name="city" width="7"/>
<sx:positionalField name="state" width="2"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="students-mapping">
<StudentGrades>
<sx:onRecord>
<StudentGrade>
<sx:fieldElementMap field="name" element="Name"/>
<sx:subrecordMapping repeatingGroup="grades">
<sx:onRecord>
<SubjectGrade>
<sx:fieldElementMap field="subject" element="Subject"/>
<sx:fieldElementMap field="grade" element="Grade"/>
</SubjectGrade>
</sx:onRecord>
</sx:subrecordMapping>
<sx:fieldElementMap field="year-born" element="YearBorn"/>
<sx:fieldElementMap field="favorite-color" element="FavoriteColor"/>
<sx:subrecordMapping repeatingGroup="addresses">
<sx:onRecord>
<Address>
<sx:fieldElementMap field="city" element="City"/>
<sx:fieldElementMap field="state" element="State"/>
</Address>
</sx:onRecord>
</sx:subrecordMapping>
</StudentGrade>
</sx:onRecord>
</StudentGrades>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-students-fix.xml -i data/students-fix.txt -o output/students-fix.xml
students
The example below shows how to convert a flat file with a single record format, but no end-of-line delimiters, to XML.
The input file is a fixed field width flat file.
Figure 92. Input flat file non_delimited_book_orders.txt
CAuthor Title Price InvoiceNo InvoiceDate FCharles Bukowski Factotum 22.95 001 12/Mar/2005 FSergei Lukyanenko The Night Watch 17.99 002 13/Mar/2005 FAndrew Crumey Mr Mee 22.00 003 14/June/2005 CSteven John Metsker Building Parsers with Java 39.95 004 15/June/2005 This is a trailer record
The desired output is shown below.
Figure 93. Output XML file book_orders.xml
<?xml version="1.0" encoding="utf-8"?>
<book-orders>
<book-type>
<category>F</category>
<author>Charles Bukowski</author>
<title>Factotum</title>
<price>22.95</price>
<invoiceno>001</invoiceno>
<invoicedate>12/Mar/2005</invoicedate>
<filler/>
</book-type>
<book-type>
<category>F</category>
<author>Sergei Lukyanenko</author>
<title>The Night Watch</title>
<price>17.99</price>
<invoiceno>002</invoiceno>
<invoicedate>13/Mar/2005</invoicedate>
<filler/>
</book-type>
<book-type>
<category>F</category>
<author>Andrew Crumey</author>
<title>Mr Mee</title>
<price>22.00</price>
<invoiceno>003</invoiceno>
<invoicedate>14/June/2005</invoicedate>
<filler/>
</book-type>
<book-type>
<category>C</category>
<author>Steven John Metsker</author>
<title>Building Parsers with Java</title>
<price>39.95</price>
<invoiceno>004</invoiceno>
<invoicedate>15/June/2005</invoicedate>
<filler/>
</book-type>
</book-orders>
The following resources script does the transformation.
Figure 94. Resources script resources-non_delimited_book_orders.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="book-orders">
<sx:serialize>
<sx:transform>
<sx:content ref="book-orders"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="book-orders" name="book-orders">
<sx:flatFileReader>
<sx:flatFile ref="books-flatfile"/>
</sx:flatFileReader>
</sx:recordContent>
<sx:flatFile id="books-flatfile" lineDelimited="false">
<sx:flatFileHeader>
<sx:flatRecordType ref="book-type"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="book-type"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord width="100">This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="book-type" name="book-type">
<sx:positionalField name="category" width="1"/>
<sx:positionalField name="author" width="30"/>
<sx:positionalField name="title" width="30"/>
<sx:positionalField name="price" width="10" justify="right"/>
<sx:positionalField name="invoiceno" width="4" justify="right"/>
<sx:positionalField name="invoicedate" width="13" justify="right"/>
<sx:positionalField name="filler" width="12"/>
</sx:flatRecordType>
</sx:resources>
Note the following points.
lineDelimited
attribute set to
false
.
You can run this example on the command line by entering
servingxml -r resources-non_delimited_book_orders.xml
-i data/non_delimited_book_orders.txt
-o output/book_orders.xml book-orders
The example below shows how to convert a flat file with multiple record formats, but no end-of-line delimiters, to XML.
The input file is a fixed field width flat file.
Figure 95. Input flat file non_delimited_variant_trades.txt
TR000103/25/2005 1:50:00This is a trade record TN0002X1234A child transaction TN0003X1235Another child transaction
The desired output is shown below.
Figure 96. Output XML file trades.xml
<?xml version="1.0" encoding="utf-8"?>
<trades>
<trade id="0001">
<trade-date>2005-03-25T01:50:00.000-05:00</trade-date>
<description>This is a trade record</description>
</trade>
<transaction id="0002">
<reference>X1234</reference>
<description>A child transaction</description>
</transaction>
<transaction id="0003">
<reference>X1235</reference>
<description>Another child transaction</description>
</transaction>
</trades>
The following resources script does the transformation.
Figure 97. Resources script resources-non_delimited_variant_trades.xm
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="trades">
<sx:serialize>
<sx:transform>
<sx:content ref="trades"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="trades">
<sx:flatFileReader>
<sx:urlSource url="data/non_delimited_variant_trades.txt"/>
<sx:flatFile ref="trades-flatfile"/>
</sx:flatFileReader>
<sx:recordMapping ref="trades-xml-mapping"/>
</sx:recordContent>
<sx:flatFile id="trades-flatfile" name="trades" lineDelimited="false">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="record_type" width="2"/>
<sx:when test="record_type='TR'">
<sx:flatRecordType name="trade">
<sx:positionalField name="record_type" width="2"/>
<sx:positionalField name="id" width="4"/>
<sx:positionalField name="trade_date" width="10"/>
<sx:positionalField name="trade_time" width="8"/>
<sx:positionalField name="description" width="30"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="record_type='TN'">
<sx:flatRecordType name="transaction">
<sx:positionalField name="record_type" width="2"/>
<sx:positionalField name="id" width="4"/>
<sx:positionalField name="reference" width="5"/>
<sx:positionalField name="description" width="30"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="trades-xml-mapping">
<trades>
<sx:onRecord recordType="trade">
<trade>
<sx:fieldAttributeMap field="id" attribute="id"/>
<sx:elementMap element="trade-date">
<sx:convertToDateTime format="MM/dd/yyyy H:mm:ss">
<sx:concat separator=" ">
<sx:toString value="{trade_date}"/>
<sx:toString value="{trade_time}"/>
</sx:concat>
</sx:convertToDateTime>
</sx:elementMap>
<sx:fieldElementMap field="description" element="description"/>
</trade>
</sx:onRecord>
<sx:onRecord recordType="transaction">
<transaction>
<sx:fieldAttributeMap field="id" attribute="id"/>
<sx:fieldElementMap field="reference" element="reference"/>
<sx:fieldElementMap field="description" element="description"/>
</transaction>
</sx:onRecord>
</trades>
</sx:recordMapping>
</sx:resources>
Note the following points.
lineDelimited
attribute set to
false
.
You can run this example on the command line by entering
servingxml -r resources-non_delimited_variant_trades.xml
-o output/trades.xml trades
The example below shows how to convert a flat file with special characters to XML.
The input file is a flat file with two adjacent records (no line delimiters):
A record beginning with the hexadecimal value 03, followed by the text "1111111"
A record beginning with the hexadecimal value 04, followed by the text "John Smith"
The two hexadecimal values are needed to differentiate the record type, but these valued cannot be written as  and in XML, as these are not valid XML characters.
The desired output is shown below.
Figure 98. Output XML file specialChar.xml
<?xml version="1.0" encoding="utf-8"?>
<transaction>
<R03>
<firstField>03</firstField>
<CLIENUM>1111111</CLIENUM>
</R03>
<R04>
<firstField>04</firstField>
<NAME>John Smith</NAME>
</R04>
</transaction>
The following resources script does the transformation.
Figure 99. Resources script resources-specialChar.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="transaction">
<sx:serialize>
<sx:transform>
<sx:content ref="transactionDoc"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="transactionDoc" name="transaction">
<sx:flatFileReader>
<sx:flatFile ref="transactionFile"/>
</sx:flatFileReader>
</sx:recordContent>
<sx:flatFile id="transactionFile" lineDelimited="false">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:binaryField name="firstField" width="1"/>
<sx:when test="firstField='03'">
<sx:flatRecordType id="R03" name="R03">
<sx:binaryField name="firstField" label="firstField" width="1" />
<sx:positionalField name="CLIENUM" label="CLIENUM" width="007" />
</sx:flatRecordType>
</sx:when>
<sx:when test="firstField='04'">
<sx:flatRecordType id="R04" name="R04">
<sx:binaryField name="firstField" label="firstField" width="1" />
<sx:positionalField name="NAME" label="NAME" width="020" />
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
Note that the first field of each record is declared as an sx:binaryField . The string rendering of this field is as an hexidecimal value.
You can run this example on the command line by entering
servingxml -r resources-specialChar.xml -i data/specialChar.txt
-o output/specialChar.xml transaction
The example below shows how to convert a non-delimited flat file containing Cobol packed decimal data to XML.
The input file, shown below, is encoded in ebcdic, except for the price field,
which is expressed as a Cobol packed decimal number
99999999.99
.
Figure 100. Input positional flat file books.txt
CAuthor Title Price
FCharles Bukowski Factotum 22.95
FSergei Lukyanenko The Night Watch 17.99
FAndrew Crumey Mr Mee 22.00
CSteven John Metsker Building Parsers with Java 39.95
This is a trailer record
The desired output is shown below.
Figure 101. Output XML file books.xml
<?xml version="1.0" encoding="utf-8"?>
<myns:books xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="url2">
<myns:book categoryCode="F">
<myns:title>Factotum</myns:title>
<myns:author>Charles Bukowski</myns:author>
<myns:price>22.95</myns:price>
</myns:book>
<myns:book categoryCode="F">
<myns:title>The Night Watch</myns:title>
<myns:author>Sergei Lukyanenko</myns:author>
<myns:price>17.99</myns:price>
</myns:book>
<myns:book categoryCode="F">
<myns:title>Mr Mee</myns:title>
<myns:author>Andrew Crumey</myns:author>
<myns:price>22.00</myns:price>
</myns:book>
<myns:book categoryCode="C">
<myns:title>Building Parsers with Java</myns:title>
<myns:author>Steven John Metsker</myns:author>
<myns:price>39.95</myns:price>
</myns:book>
</myns:books>
The following resources script does the transformation.
Figure 102. Resources script resources-books_ebcdic_packed.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="books-ebcdic-packed2xml">
<sx:serialize>
<sx:transform>
<sx:content ref="books"/>
</sx:transform>
</sx:serialize>
</sx:service>
<!--
This sx:flatFileReader element does not specify a stream source, so
the source will default to the file specified with the -i option on the command line.
-->
<sx:recordContent id="books">
<sx:flatFileReader>
<sx:flatFile ref="booksFile"/>
<sx:defaultStreamSource encoding="Cp1047"/>
</sx:flatFileReader>
<sx:recordMapping ref="booksToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="booksFile" lineDelimited="false">
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="bookType" name="bookType">
<sx:positionalField name="category" label="Category" width="1"/>
<sx:positionalField name="author" label="Author" width="30"/>
<sx:positionalField name="title" label="Title" width="30"/>
<sx:packedDecimalField name="price" label="Price" digitCount="10" decimalPlaces="2"/>
</sx:flatRecordType>
<sx:recordMapping id="booksToXmlMapping">
<myns:books xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="url2">
<sx:onRecord>
<myns:book>
<sx:fieldAttributeMap field="category" attribute="categoryCode"/>
<sx:fieldElementMap field="title" element="myns:title"/>
<sx:fieldElementMap field="author" element="myns:author"/>
<sx:fieldElementMap field="price" element="myns:price"/>
</myns:book>
</sx:onRecord>
</myns:books>
</sx:recordMapping>
</sx:resources>
Note that the last field of each record is declared as an sx:packedDecimalField .
You can run this example on the command line by entering
servingxml -r resources-books_ebcdic_packed.xml
-i data/books_ebcdic_packed.dat
-o output/books.xml books-ebcdic-packed2xml
The example below shows how to convert a System Management Facilities (SMF) file to XML. The example includes support for variable block spanned (VBS) records. The format of an SMF file is described in SMF records.
The desired output is shown below.
Figure 103. Output XML file smf.xml
<?xml version="1.0" encoding="UTF-8"?>
<smf>
<smfRecord>
<systemIndicator>1E</systemIndicator>
<type>2</type>
</smfRecord>
<smfRecord>
<systemIndicator>DE</systemIndicator>
<type>110</type>
</smfRecord>
<smfRecord>
<systemIndicator>1E</systemIndicator>
<type>64</type>
<jobName>SMF</jobName>
</smfRecord>
<smfRecord>
<systemIndicator>5E</systemIndicator>
<type>42</type>
<subType>6</subType>
</smfRecord>
<smfRecord>
<systemIndicator>1E</systemIndicator>
<type>64</type>
<jobName>GPL1</jobName>
</smfRecord>
<smfRecord>
<systemIndicator>1E</systemIndicator>
<type>64</type>
<jobName>GPL1</jobName>
</smfRecord>
<smfRecord>
<systemIndicator>1E</systemIndicator>
<type>64</type>
<jobName>GPL1</jobName>
</smfRecord>
<smfRecord>
<systemIndicator>1E</systemIndicator>
<type>3</type>
</smfRecord>
</smf>
The following resources script does the transformation.
Figure 104. Resources script resources-smf.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:fn="http://www.w3.org/2005/xpath-functions">
<sx:service id="smf-xml">
<sx:serialize>
<sx:xsltSerializer>
<sx:outputProperty name="indent" value="yes"/>
</sx:xsltSerializer>
<sx:transform>
<sx:content ref="smf"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="smf">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="smfFile"/>
<sx:defaultStreamSource encoding="Cp1047"/>
</sx:flatFileReader>
</sx:recordStream>
<sx:recordMapping ref="smfToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="smfFile" lineDelimited="false" countPositionsInBytes="true">
<sx:flatFileBody>
<sx:flatRecordType ref="smfType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:vbsFlatRecordType id="smfType">
<sx:integerField name="recordLength" width="2"/>
<sx:integerField name="segmentControlCode" width="1"/>
<sx:integerField name="reserved" width="1"/>
<sx:mergePhysicalSegments startTest = "number(sx:current//segmentControlCode) = 0
or number(sx:current//segmentControlCode) = 1"
segmentLength="{recordLength}">
<sx:flatRecordTypeChoice>
<sx:integerField name="recordLength" width="2"/>
<sx:integerField name="segmentControlCode" width="1"/>
<sx:integerField name="reserved" width="1"/>
<sx:binaryField name="systemIndicator" width="1"/>
<sx:integerField name="recordType" width="1"/>
<sx:when test="recordType=64">
<sx:flatRecordType name="smfType64">
<sx:integerField name="recordLength" width="2"/>
<sx:integerField name="segmentControlCode" width="1"/>
<sx:integerField name="reserved" width="1"/>
<sx:binaryField name="systemIndicator" label="systemIndicator" width="1"/>
<sx:integerField name="recordType" label="recordType" width="1"/>
<sx:positionalField name="jobName" label="Job Name" start="19" width="8"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="recordType=42">
<sx:flatRecordType name="smfType42">
<sx:integerField name="recordLength" width="2"/>
<sx:integerField name="segmentControlCode" width="1"/>
<sx:integerField name="reserved" width="1"/>
<sx:binaryField name="systemIndicator" label="systemIndicator" width="1"/>
<sx:integerField name="recordType" label="recordType" width="1"/>
<sx:integerField name="subType" label="Sub Type" start="23" width="2"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="recordType=110">
<sx:flatRecordType name="smfType110">
<sx:integerField name="recordLength" width="2"/>
<sx:integerField name="segmentControlCode" width="1"/>
<sx:integerField name="reserved" width="1"/>
<sx:binaryField name="systemIndicator" label="systemIndicator" width="1"/>
<sx:integerField name="recordType" label="recordType" width="1"/>
</sx:flatRecordType>
</sx:when>
<sx:otherwise>
<sx:flatRecordType name="other">
<sx:integerField name="recordLength" width="2"/>
<sx:integerField name="segmentControlCode" width="1"/>
<sx:integerField name="reserved" width="1"/>
<sx:binaryField name="systemIndicator" label="systemIndicator" width="1"/>
<sx:integerField name="recordType" label="recordType" width="1"/>
</sx:flatRecordType>
</sx:otherwise>
</sx:flatRecordTypeChoice>
</sx:mergePhysicalSegments>
</sx:vbsFlatRecordType>
<sx:recordMapping id="smfToXmlMapping">
<smf>
<sx:onRecord recordType="smfType64">
<smfRecord>
<sx:fieldElementMap field="systemIndicator" element="systemIndicator"/>
<sx:fieldElementMap field="recordType" element="type"/>
<sx:fieldElementMap field="jobName" element="jobName"/>
</smfRecord>
</sx:onRecord>
<sx:onRecord recordType="smfType42">
<smfRecord>
<sx:fieldElementMap field="systemIndicator" element="systemIndicator"/>
<sx:fieldElementMap field="recordType" element="type"/>
<sx:fieldElementMap field="subType" element="subType"/>
</smfRecord>
</sx:onRecord>
<sx:onRecord recordType="smfType110">
<smfRecord>
<sx:fieldElementMap field="systemIndicator" element="systemIndicator"/>
<sx:fieldElementMap field="recordType" element="type"/>
</smfRecord>
</sx:onRecord>
<sx:onRecord recordType="other">
<smfRecord>
<sx:fieldElementMap field="systemIndicator" element="systemIndicator"/>
<sx:fieldElementMap field="recordType" element="type"/>
</smfRecord>
</sx:onRecord>
</smf>
</sx:recordMapping>
</sx:resources> The field definitions that appear as children of
sx:vbsFlatRecordType
comprise the 4-byte segment descriptor word (SDW), which describes the segment.
The first 2 bytes contain the length of the segment, including the 4-byte SDW.
The third byte of the SDW contains the segment control code, a value of
00
indicates that the segment is a complete logical record, and
01
indicates that the segment is the first segment of a multisegment
record. For more details, see
segment descriptor word.
The
sx:mergePhysicalSegments
element controls the composition of the mulitple segments into a
single logical record. Note that a segmentControlCode
of either
00 or 01 marks the beginning of a
logical record. By default, the first segment descriptor
word goes at the front of the logical record, in the place of a
record descriptor word (RDW),
subsequent segment descriptor words are discarded. If the optional
suppressRDW attribute is set to "true", the
first segment descriptor word is discarded as well, the SDW fields
must be removed from the content of
sx:mergePhysicalSegments,
and four bytes need to be subtracted from any positions specified
in the record definitions.
Note that ServingXML positions are one-based, while the offsets in the IBM documentation are zero-based.
You can run this example on the command line by entering
servingxml -r resources-smf.xml -i data/G6744V00.smf -o output/smf.xml smf-xml
The input file is a fixed field width flat file.
Figure 105. Input flat file composite-records.txt
ODANIEL 356996699001
A233ServingXML132433SAS
B343443DD2322ARDE021101
CDANIEL 029299839907
B344233DD2322PREE021100
D0000040000002300000001
A233ServingXML132433AGS
B3434432323000022021101
CDANIEL 029299839907
B3433132323000022021101
CDANIEL 029299839907
D0000040000002300000002
E0030000000100009803600
Note the following properties of this data:
The desired output is shown below.
Figure 106. Output XML file composite-records.xml
<?xml version="1.0" encoding="UTF-8"?>
<document>
<header>
<O>
<recordType>O</recordType>
<name>DANIEL</name>
<id>356</id>
<code>996699</code>
<tracking>001</tracking>
</O>
</header>
<body>
<A>
<recordType>A</recordType>
<code>233</code>
<book>ServingXML</book>
<isbn>132433</isbn>
<identifier>SAS</identifier>
</A>
<B-SAS>
<recordType>B</recordType>
<library>343443</library>
<shelf>DD2322ARDE</shelf>
<ebook>021</ebook>
<code>101</code>
</B-SAS>
<C>
<recordType>C</recordType>
<author>DANIEL</author>
<publisher>0292998399</publisher>
<flag>07</flag>
</C>
<B-SAS>
<recordType>B</recordType>
<library>344233</library>
<shelf>DD2322PREE</shelf>
<ebook>021</ebook>
<code>100</code>
</B-SAS>
<D>
<recordType>D</recordType>
<count>000004</count>
<amount>0000002300</amount>
<tracking>000001</tracking>
</D>
<A>
<recordType>A</recordType>
<code>233</code>
<book>ServingXML</book>
<isbn>132433</isbn>
<identifier>AGS</identifier>
</A>
<B-AGS>
<recordType>B</recordType>
<store>34344323</store>
<cost>23000022</cost>
<code>021</code>
<make>101</make>
</B-AGS>
<C>
<recordType>C</recordType>
<author>DANIEL</author>
<publisher>0292998399</publisher>
<flag>07</flag>
</C>
<B-AGS>
<recordType>B</recordType>
<store>34331323</store>
<cost>23000022</cost>
<code>021</code>
<make>101</make>
</B-AGS>
<C>
<recordType>C</recordType>
<author>DANIEL</author>
<publisher>0292998399</publisher>
<flag>07</flag>
</C>
<D>
<recordType>D</recordType>
<count>000004</count>
<amount>0000002300</amount>
<tracking>000002</tracking>
</D>
</body>
<trailer>
<E>
<recordType>E</recordType>
<total>00300</total>
<amount>0000010000</amount>
<count>9803</count>
<tracking>600</tracking>
</E>
</trailer>
</document>
The following resources script does the transformation.
Figure 107. Resources script resources-recordComposition.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:fn="http://www.w3.org/2005/xpath-functions">
<sx:service id="composite-records-to-xml">
<sx:task ref="serialized-composite-records"/>
</sx:service>
<sx:serialize id="serialized-composite-records">
<sx:transform>
<sx:recordContent ref="composite-record-content"/>
</sx:transform>
</sx:serialize>
<sx:recordContent id="composite-record-content">
<sx:flatFileReader ref="composite-record-reader"/>
<sx:recordMapping ref="composite-record-mapping"/>
</sx:recordContent>
<sx:flatFileReader id="composite-record-reader">
<sx:flatFile>
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="recordType" width="1"/>
<sx:when test="recordType='O'">
<sx:flatRecordType ref="O"/>
</sx:when>
<sx:when test="recordType='E'">
<sx:flatRecordType ref="E"/>
</sx:when>
<sx:otherwise>
<sx:compositeFlatRecordType>
<sx:positionalField name="recordType" width="1"/>
<sx:positionalField name="code" width="3"/>
<sx:positionalField name="book" width="10"/>
<sx:positionalField name="isbn" width="6"/>
<sx:positionalField name="identifier" width="3"/>
<sx:combinePhysicalRecords recordType="composite" repeatingGroup="group"
startTest = "sx:current//recordType = 'A'
and sx:current//identifier = 'SAS'"
endTest = "sx:current//recordType = 'A'">
<sx:flatRecordTypeChoice>
<sx:positionalField name="recordType" width="1"/>
<sx:when test="recordType='A'">
<sx:flatRecordType ref="A"/>
</sx:when>
<sx:when test="recordType='B'">
<sx:flatRecordType ref="B-SAS"/>
</sx:when>
<sx:when test="recordType='C'">
<sx:flatRecordType ref="C"/>
</sx:when>
<sx:when test="recordType='D'">
<sx:flatRecordType ref="D"/>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:combinePhysicalRecords>
<sx:combinePhysicalRecords recordType="composite" repeatingGroup="group"
startTest = "sx:current//recordType = 'A'
and sx:current//identifier = 'AGS'"
endTest = "sx:current//recordType = 'A'">
<sx:flatRecordTypeChoice>
<sx:positionalField name="recordType" width="1"/>
<sx:when test="recordType='A'">
<sx:flatRecordType ref="A"/>
</sx:when>
<sx:when test="recordType='B'">
<sx:flatRecordType ref="B-AGS"/>
</sx:when>
<sx:when test="recordType='C'">
<sx:flatRecordType ref="C"/>
</sx:when>
<sx:when test="recordType='D'">
<sx:flatRecordType ref="D"/>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:combinePhysicalRecords>
</sx:compositeFlatRecordType>
</sx:otherwise>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
</sx:flatFileReader>
<sx:flatRecordType id="A" name="A">
<sx:positionalField name="recordType" width="1"/>
<sx:positionalField name="code" width="3"/>
<sx:positionalField name="book" width="10"/>
<sx:positionalField name="isbn" width="6"/>
<sx:positionalField name="identifier" width="3"/>
</sx:flatRecordType>
<sx:flatRecordType id="C" name="C">
<sx:positionalField name="recordType" width="1"/>
<sx:positionalField name="author" width="10"/>
<sx:positionalField name="publisher" width="10"/>
<sx:positionalField name="flag" width="2"/>
</sx:flatRecordType>
<sx:flatRecordType id="D" name="D">
<sx:positionalField name="recordType" width="1"/>
<sx:positionalField name="count" width="6"/>
<sx:positionalField name="amount" width="10"/>
<sx:positionalField name="tracking" width="6"/>
</sx:flatRecordType>
<sx:flatRecordType id="E" name="E">
<sx:positionalField name="recordType" width="1"/>
<sx:positionalField name="total" width="5"/>
<sx:positionalField name="amount" width="10"/>
<sx:positionalField name="count" width="4"/>
<sx:positionalField name="tracking" width="3"/>
</sx:flatRecordType>
<sx:flatRecordType id="B-SAS" name="B-SAS">
<sx:positionalField name="recordType" width="1"/>
<sx:positionalField name="library" width="6"/>
<sx:positionalField name="shelf" width="10"/>
<sx:positionalField name="ebook" width="3"/>
<sx:positionalField name="code" width="3"/>
</sx:flatRecordType>
<sx:flatRecordType id="B-AGS" name="B-AGS">
<sx:positionalField name="recordType" width="1"/>
<sx:positionalField name="store" width="8"/>
<sx:positionalField name="cost" width="8"/>
<sx:positionalField name="code" width="3"/>
<sx:positionalField name="make" width="3"/>
</sx:flatRecordType>
<sx:flatRecordType id="O" name="O">
<sx:positionalField name="recordType" width="1"/>
<sx:positionalField name="name" width="10"/>
<sx:positionalField name="id" width="3"/>
<sx:positionalField name="code" width="6"/>
<sx:positionalField name="tracking" width="3"/>
</sx:flatRecordType>
<sx:recordMapping id="composite-record-mapping">
<document>
<header>
<sx:onRecord recordType="O">
<O>
<sx:defaultFieldElementMap fields="*"/>
</O>
</sx:onRecord>
</header>
<body>
<sx:onRecord recordType="composite">
<sx:subrecordMapping repeatingGroup="group">
<sx:onRecord recordType="A">
<A>
<sx:defaultFieldElementMap fields="*"/>
</A>
</sx:onRecord>
<sx:onRecord recordType="B-SAS">
<B-SAS>
<sx:defaultFieldElementMap fields="*"/>
</B-SAS>
</sx:onRecord>
<sx:onRecord recordType="B-AGS">
<B-AGS>
<sx:defaultFieldElementMap fields="*"/>
</B-AGS>
</sx:onRecord>
<sx:onRecord recordType="C">
<C>
<sx:defaultFieldElementMap fields="*"/>
</C>
</sx:onRecord>
<sx:onRecord recordType="D">
<D>
<sx:defaultFieldElementMap fields="*"/>
</D>
</sx:onRecord>
<sx:onRecord recordType="O">
<O>
<sx:defaultFieldElementMap fields="*"/>
</O>
</sx:onRecord>
</sx:subrecordMapping>
</sx:onRecord>
</body>
<trailer>
<sx:onRecord recordType="E">
<E>
<sx:defaultFieldElementMap fields="*"/>
</E>
</sx:onRecord>
</trailer>
</document>
</sx:recordMapping>
</sx:resources>
The field definitions that appear as children of sx:compositeFlatRecordType belong to the header of a group of related records.
The sx:combinePhysicalRecords element controls the aggregation of a group of physical records into a composite logical record.
You can run this example on the command line by entering
servingxml -r resources-recordComposition.xml -i data/composite-records.txt
-o output/composite-records.xml composite-records-to-xml
This example shows a resources script that will concatenate the values of several records into a single XML element value. It illustrates the use of the sx:combineRecords element.
The input file is
Figure 108. Input flat file orders.txt
HEADER
ORDER/CUS/CON/USD/
PO1/PAR1/MFR FADD1/5/
REM AAAAA
REM BBBBB
REM CCCCC
PO2/PAR2/MFR 30066/1/
PO3/PAR3/MFR PAGT2/2/
REM CCCCC
REM DDDDD
TRAILER
The desired output is
Figure 109. Output XML file order.xml
<?xml version="1.0" encoding="UTF-8"?>
<Order>
<OrderHeader>
<CUS>CUS</CUS>
<CON>CON</CON>
<CUR>USD</CUR>
</OrderHeader>
<OrderPosition>
<ORN>PO1</ORN>
<PAR>PAR1</PAR>
<QTY>5</QTY>
<MFR>FADD1</MFR>
<REM>AAAAA BBBBB CCCCC</REM>
</OrderPosition>
<OrderPosition>
<ORN>PO2</ORN>
<PAR>PAR2</PAR>
<QTY>1</QTY>
<MFR>30066</MFR>
</OrderPosition>
<OrderPosition>
<ORN>PO3</ORN>
<PAR>PAR3</PAR>
<QTY>2</QTY>
<MFR>PAGT2</MFR>
<REM>CCCCC DDDDD</REM>
</OrderPosition>
</Order>
Note that the REM values are concatenated in a space separated list.
The following resources script does the transformation.
Figure 110. Resources script resources-order.xml
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:fn="http://www.w3.org/2005/xpath-functions">
<sx:service id="Order">
<sx:serialize>
<sx:transform>
<sx:content ref="Order"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="Order" name="Order">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="OrderFile"/>
</sx:flatFileReader>
<sx:combineRecords recordType="composite" repeatingGroup="detail"
startTest="sx:current/orderRemarks"
endTest="not(sx:current/orderRemarks)">
<sx:newField name="remarks" select="detail/orderRemarks/REM"/>
</sx:combineRecords>
</sx:recordStream>
<sx:recordMapping ref="OrderToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="OrderFile">
<sx:commentStarter value="#"/>
<sx:fieldDelimiter value="/"/>
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="tag" width="3"/>
<sx:when test="tag='ORD'">
<sx:flatRecordType name="orderHeader">
<sx:delimitedField name="CMD" label="Command Code"/>
<sx:delimitedField name="CUS" label="Customer"/>
<sx:delimitedField name="CON" label="Contractor"/>
<sx:delimitedField name="CUR" label="Currency Code"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='REM'">
<sx:flatRecordType name="orderRemarks">
<sx:positionalField name="REMTEI" label="Remark" width="3"/>
<sx:delimitedField name="REM" label="Remark"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='TRA'">
<sx:flatRecordType name="orderTrailer">
</sx:flatRecordType>
</sx:when>
<sx:otherwise>
<sx:flatRecordType name="orderPosition">
<sx:delimitedField name="ORN" label="Order Number"/>
<sx:delimitedField name="PAR" label="Part Number"/>
<sx:delimitedField name="MFRTEI" maxWidth="3"/>
<sx:delimitedField name="MFR" label="Manufacturer"/>
<sx:delimitedField name="QTY" label="Order Quantity"/>
</sx:flatRecordType>
</sx:otherwise>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="OrderToXmlMapping">
<Order>
<sx:innerGroup startTest="sx:current/orderHeader">
<OrderHeader>
<sx:fieldElementMap field="CUS" element="CUS"/>
<sx:fieldElementMap field="CON" element="CON"/>
<sx:fieldElementMap field="CUR" element="CUR"/>
</OrderHeader>
<sx:innerGroup startTest="sx:current/orderPosition">
<OrderPosition>
<sx:fieldElementMap field="ORN" element="ORN" minOccurs="1"/>
<sx:fieldElementMap field="PAR" element="PAR"/>
<sx:fieldElementMap field="QTY" element="QTY"/>
<sx:fieldElementMap field="MFR" element="MFR" minOccurs="0"/>
<sx:onRecord recordType="composite">
<sx:choose>
<sx:when test="remarks != ''">
<sx:fieldElementMap select="remarks" element="REM"/>
</sx:when>
</sx:choose>
</sx:onRecord>
</OrderPosition>
</sx:innerGroup>
</sx:innerGroup>
</Order>
</sx:recordMapping>
</sx:resources>
The effect of
sx:combineRecords
is to collect all the
adjacent
orderRemarks
records, where they become sub-records
belonging to a field called "detail", and create one record of type
"composite" that has the repeating group field "detail", plus another field
"remarks" that is computed from "detail".
The
orderRemarks
records flowing into
sx:combineRecords
look like, in their
XML representation,
<orderRemarks><REMTEI>REM</REMTEI><REM>AAAA</REM></orderRemarks>
<orderRemarks><REMTEI>REM</REMTEI><REM>BBBB</REM></orderRemarks>
<orderRemarks><REMTEI>REM</REMTEI><REM>CCCC</REM></orderRemarks>
The composite record flowing out looks like
<composite>
<detail>
<orderRemarks><REMTEI>REM</REMTEI><REM>AAAA</REM></orderRemarks>
<orderRemarks><REMTEI>REM</REMTEI><REM>BBBB</REM></orderRemarks>
<orderRemarks><REMTEI>REM</REMTEI><REM>CCCC</REM></orderRemarks>
</detail>
<remarks>AAAA BBBB CCCC</remarks>
</composite>
You can run this example on the command line by entering
servingxml -i data/order.txt -r resources-order.xml -o output/order.xml Order
This example shows a resources script that maps two records at the top of a file to a header element in the XML output. To map fields from multiple records to the attributes of a single element, you first need to compose an aggregate of the two records with the sx:combineRecords instruction. You can then select against the composite record with an sx:fieldAttributeMap and populate the attributes of the parent element.
The input file is
Figure 111. Input flat file files.txt
DP1 458293048kd02
*****2LV 2008070132 EX9
E0000015498273
F0000014938294
G000001E7839201
D0000024839305
The rules for mapping the third through last records are as follows.
If a "D" record is found, start a new
file
element, but "D" records are optional.
If an "E" record is found, start a new
file
element, unless
one has already been started because of a
"D" record.
The desired output is
Figure 112. Output XML file files.xml
<?xml version="1.0" encoding="UTF-8"?>
<output date="20080701" sequence="32"
code="EX9" reference="458293048kd0">
<file filenumber="000001"/>
<file filenumber="000002"/>
</output>
The following resources script does the transformation.
Figure 113. Resources script resources-files.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="files">
<sx:serialize>
<sx:transform>
<sx:content ref="filesContent"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="filesContent">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="fileFlatFile"/>
</sx:flatFileReader>
<sx:combineRecords recordType="header" repeatingGroup="headerRecords"
startTest="sx:current/header1" endTest="sx:previous/header2">
</sx:combineRecords>
</sx:recordStream>
<sx:recordMapping ref="fileToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="fileFlatFile">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="tag" width="1"/>
<sx:positionalField name="tag2" width="2"/>
<sx:when test="tag='D' and tag2='P1'">
<sx:flatRecordType name="header1">
<sx:positionalField name="reference" start="5" width="13"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='*'">
<sx:flatRecordType name="header2">
<sx:positionalField name="date" start="10" width="8"/>
<sx:positionalField name="sequence" width="2"/>
<sx:positionalField name="code" start="21" width="3"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='E'">
<sx:flatRecordType name="E">
<sx:positionalField name="filenumber" start="2" width="6"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='D'">
<sx:flatRecordType name="D">
<sx:positionalField name="filenumber" start="2" width="6"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="fileToXmlMapping">
<output>
<sx:fieldAttributeMap select="headerRecords/header2/date" attribute="date"/>
<sx:fieldAttributeMap select="headerRecords/header2/sequence" attribute="sequence"/>
<sx:fieldAttributeMap select="headerRecords/header2/code" attribute="code"/>
<sx:fieldAttributeMap select="headerRecords/header1/reference" attribute="reference"/>
<sx:groupChoice>
<sx:innerGroup startTest="sx:current/D" endTest="sx:current/D">
<file>
<sx:fieldAttributeMap field="filenumber" attribute="filenumber"/>
</file>
</sx:innerGroup>
<sx:innerGroup startTest="sx:current/E" endTest="sx:current/D or sx:current/E">
<file>
<sx:fieldAttributeMap field="filenumber" attribute="filenumber"/>
</file>
</sx:innerGroup>
</sx:groupChoice>
</output>
</sx:recordMapping>
</sx:resources>
The effect of
sx:combineRecords
is to collect the top two header records,
header1
and
header2
, where they become sub-records belonging to a
field called "headerRecords", and create one record of type "header" that
has the repeating group field "headerRecords".
The
header
record has the following
XML representation,
<header>
<headerRecords>
<header1>
<reference>458293048kd0</reference>
</header1>
<header2>
<date>20080701</date>
<sequence>32</sequence>
<code>EX9</code>
</header2>
</headerRecords>
</header>
You can run this example on the command line by entering
servingxml -i data/files.txt -r resources-files.xml -o output/files.xml files
The example below shows how to convert a flat file with start/end record delimiters.
The input file is shown below.
Figure 114. Input file opdef.txt
1 20050505 121205 xtyzzya
{:HJSED
:20:886452311980706
:23:CREDDFED
:32:030612USD5443,99
:33:USD5443,99
:50K:GIAN
NNPLLS
:52A:B0
:53A:B33
:54A:I3N
:57A:BRE
:59:/20041010050500001M02606
KKL S.A
GNRBBL
:70:/FF/INV 559661
:71:SAH
-}
3 20050505 121205 bahthy
This file has a number of features.
The first and last lines are header and footers, delimited by new line symbols.
The middle part must be taken as a whole chunk, starting from the opening
{
and ending at the closing
}
.
The desired output is shown below.
Figure 115. Output XML file opdef.xml
<?xml version="1.0" encoding="utf-8"?>
<opdef>
<start>
<opdate>20050505</opdate>
<optime>121205</optime>
<optarget>xtyzzya</optarget>
</start>
:HJSED
:20:886452311980706
:23:CREDDFED
:32:030612USD5443,99
:33:USD5443,99
:50K:GIAN
NNPLLS
:52A:B0
:53A:B33
:54A:I3N
:57A:BRE
:59:/20041010050500001M02606
KKL S.A
GNRBBL
:70:/FF/INV 559661
:71:SAH
-
<end>
<tpdate>20050505</tpdate>
<tptime>121205</tptime>
<tptarget>bahthy</tptarget>
</end>
</opdef>
The following resources script does the transformation.
Figure 116. Resources script resources-opdef.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="opdef">
<sx:serialize>
<sx:transform>
<sx:content ref="opdef"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="opdef">
<sx:flatFileReader>
<sx:flatFile ref="opdefFlatFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="opdefRecordMapping"/>
</sx:recordContent>
<sx:flatFile id="opdefFlatFile" name="opdef">
<sx:recordDelimiter value="\r\n"/>
<sx:recordDelimiter value="\n"/>
<sx:recordDelimiter start="{" end="}"/>
<sx:flatFileBody trim="true">
<sx:flatRecordTypeChoice>
<sx:positionalField name="tag" width="1"/>
<sx:when test="tag='1'">
<sx:flatRecordType name="start">
<sx:positionalField name="tag" width="2"/>
<sx:positionalField name="opdate" width="9"/>
<sx:positionalField name="optime" width="7"/>
<sx:positionalField name="optarget" width="7"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='3'">
<sx:flatRecordType name="end">
<sx:positionalField name="tag" width="2"/>
<sx:positionalField name="tpdate" width="9"/>
<sx:positionalField name="tptime" width="7"/>
<sx:positionalField name="tptarget" width="7"/>
</sx:flatRecordType>
</sx:when>
<sx:otherwise>
<sx:flatRecordType name="data">
<sx:delimitedField name="field1"/>
</sx:flatRecordType>
</sx:otherwise>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="opdefRecordMapping">
<opdef>
<sx:onRecord recordType="start">
<start>
<sx:defaultFieldElementMap fields="opdate optime optarget"/>
</start>
</sx:onRecord>
<sx:onRecord recordType="end">
<end>
<sx:defaultFieldElementMap fields="tpdate tptime tptarget"/>
</end>
</sx:onRecord>
<sx:onRecord recordType="data">
<sx:cdata>
<sx:toString value="{field1}"/>
</sx:cdata>
</sx:onRecord>
</opdef>
</sx:recordMapping>
</sx:resources>
Note the following points about this script.
There are three record delimiters, which are checked sequentially. The third record delimiter specifies both start and end values. When the start value is found, the matching end value is looked for, taking account of nesting, and the whole returned as one record. The chunk returned may contain occurances of the other two delimiters.
The data is mapped to a single field by using a
sx:delimitedField
element, taking the default field delimiter, which is the
end of the record. The data is enclosed in a
CDATA
section, using the
sx:cdata
element.
You can run this example on the command line by entering
servingxml -i data/opdef.txt -r resources-opdef.xml
-o output/opdef.xml opdef
The example below shows how to convert a flat file with nested start/end record and segment delimiters and name/value pairs.
The input file is shown below.
Figure 117. Input file transaction.txt
{TRAN:trantype=PRIN#origin=Toronto#
{TRANREF:reftype=TREF#ref=1234567890123456B#}
{TRANREF:reftype=FOTR#ref=1234567890123456#}
{DRIVER:drivertype=OPER#drivercode=SELL#}
{DRIVER:drivertype=STRM#drivercode=FOP#}
{INSTR:instrtype=PINS#instrreftype=INT#instrref=PTEQTY1#qty=1000000#}
{INSTR:instrtype=TCCY#instrreftype=ISO#instrref=USD#}
{INSTR:instrtype=SINS#instrreftype=ISO#instrref=USD#}
{PARTY:partytype=COMP#partyreftype=X3#partyref=CMP1#}
{PARTY:partytype=SECP#partyreftype=X3#partyref=PTSECP1#}
{DATE:datetype=TDAT#date=01-Jan-2007#}
{DATE:datetype=VDAT#date=01-Jan-2007#}
{CHARGE:chargetype=COMM#calctype=NONE#chargeamount=50.00#instrreftype=ISO#instrref=USD}
{PRICE:ratetype=TPRC#price=4.9#multdiv=X1#pricetype=YP#}
}
This file has a number of features.
The entire record is contained inside "{" - "}" record delimiters.
The first field in the record is delimited by a ":" and identifies the type of the record.
The record contains a number of name/value pairs, with names and values separated by the character "=", and fields terminated by the character "#".
The record contains a number of segments enclosed in "{" - "}" segment delimiters.
The first field in each segment is delimited by a ":" and identifies the type of the segment.
Each segment contains a number of name/value pairs, with names and values separated by the character "=", and fields terminated by the character "#".
The desired output is shown below.
Figure 118. Output XML file transaction.xml
<?xml version="1.0" encoding="utf-8"?>
<transaction>
<TRAN>
<record-type>TRAN</record-type>
<trantype>PRIN</trantype>
<origin>Toronto</origin>
<segments>
<TRANREF>
<record-type>TRANREF</record-type>
<reftype>TREF</reftype>
<ref>1234567890123456B</ref>
</TRANREF>
<TRANREF>
<record-type>TRANREF</record-type>
<reftype>FOTR</reftype>
<ref>1234567890123456</ref>
</TRANREF>
<DRIVER>
<record-type>DRIVER</record-type>
<drivertype>OPER</drivertype>
<drivercode>SELL</drivercode>
</DRIVER>
<DRIVER>
<record-type>DRIVER</record-type>
<drivertype>STRM</drivertype>
<drivercode>FOP</drivercode>
</DRIVER>
<INSTR>
<record-type>INSTR</record-type>
<instrtype>PINS</instrtype>
<instrreftype>INT</instrreftype>
<instrref>PTEQTY1</instrref>
<qty>1000000</qty>
</INSTR>
<INSTR>
<record-type>INSTR</record-type>
<instrtype>TCCY</instrtype>
<instrreftype>ISO</instrreftype>
<instrref>USD</instrref>
</INSTR>
<INSTR>
<record-type>INSTR</record-type>
<instrtype>SINS</instrtype>
<instrreftype>ISO</instrreftype>
<instrref>USD</instrref>
</INSTR>
<PARTY>
<record-type>PARTY</record-type>
<partytype>COMP</partytype>
<partyreftype>X3</partyreftype>
<partyref>CMP1</partyref>
</PARTY>
<PARTY>
<record-type>PARTY</record-type>
<partytype>SECP</partytype>
<partyreftype>X3</partyreftype>
<partyref>PTSECP1</partyref>
</PARTY>
<DATE>
<record-type>DATE</record-type>
<datetype>TDAT</datetype>
<date>01-Jan-2007</date>
</DATE>
<DATE>
<record-type>DATE</record-type>
<datetype>VDAT</datetype>
<date>01-Jan-2007</date>
</DATE>
<CHARGE>
<record-type>CHARGE</record-type>
<chargetype>COMM</chargetype>
<calctype>NONE</calctype>
<chargeamount>50.00</chargeamount>
<instrreftype>ISO</instrreftype>
<instrref>USD</instrref>
</CHARGE>
<PRICE>
<record-type>PRICE</record-type>
<ratetype>TPRC</ratetype>
<price>4.9</price>
<multdiv>X1</multdiv>
<pricetype>YP</pricetype>
</PRICE>
</segments>
</TRAN>
</transaction>
The following resources script does the transformation.
Figure 119. Resources script resources-transaction.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="transaction">
<sx:serialize>
<sx:transform>
<sx:content ref="transaction-content"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="transaction-content" name="transaction">
<sx:flatFileReader>
<sx:flatFile ref="transaction-flat-file"/>
</sx:flatFileReader>
</sx:recordContent>
<sx:flatFile id="transaction-flat-file">
<sx:recordDelimiter start="{" end="}"/>
<sx:flatFileBody>
<sx:repeatDelimiter start="{" end="}"/>
<sx:nameDelimiter value="="/>
<sx:flatRecordType name="TRAN">
<sx:delimitedField name="record-type">
<sx:fieldDelimiter value=":"/>
</sx:delimitedField>
<sx:repeatingField>
<sx:fieldDelimiter value="#"/>
<sx:segmentDelimiter value="{"/>
<sx:delimitedNamedField/>
</sx:repeatingField>
<sx:repeatingGroup name="segments">
<sx:flatRecordTypeChoice>
<sx:delimitedField name="record-type">
<sx:fieldDelimiter value=":"/>
</sx:delimitedField>
<sx:when test="1=1">
<sx:flatRecordType name="{record-type}">
<sx:delimitedField name="record-type">
<sx:fieldDelimiter value=":"/>
</sx:delimitedField>
<sx:repeatingField>
<sx:delimitedNamedField/>
<sx:fieldDelimiter value="#"/>
</sx:repeatingField>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
Note the following points about this script.
The value of the first field in the segment, terminated by ":", is assigned as the record type name of the segment.
All the name/value pairs within a segment are read with a single sx:repeatingField element.
You can run this example on the command line by entering
servingxml -i data/transaction.txt -r resources-transaction.xml
-o output/transaction.xml transaction
The example below shows how to convert the the output of a custom record reader to XML.
The input records are generated by the following Java class.
Figure 120. TradeRecordReader Java class
package flat2xml;
import com.servingxml.app.ServiceContext;
import com.servingxml.components.recordio.AbstractRecordReader;
import com.servingxml.app.Flow;
import com.servingxml.util.Name;
import com.servingxml.util.QualifiedName;
import com.servingxml.util.ServingXmlException;
import com.servingxml.util.record.Record;
import com.servingxml.util.record.RecordBuilder;
public class TradeRecordReader extends AbstractRecordReader {
private static final Name TRADE_RECORD_TYPE = new QualifiedName("trade");
private static final Name TRANSACTION_RECORD_TYPE = new QualifiedName("transaction");
private static final Name RECORD_TYPE_NAME = new QualifiedName("record_type");
private static final Name ID_NAME = new QualifiedName("id");
private static final Name TRADE_DATE_NAME = new QualifiedName("trade_date");
private static final Name TRADE_TIME_NAME = new QualifiedName("trade_time");
private static final Name DESCRIPTION_NAME = new QualifiedName("description");
private static final Name REFERENCE_NAME = new QualifiedName("reference");
public void readRecords(ServiceContext context, Flow flow)
{
// Start the record stream
startRecordStream(context, flow);
RecordBuilder trRecordBuilder = new RecordBuilder(TRADE_RECORD_TYPE);
RecordBuilder tnRecordBuilder = new RecordBuilder(TRANSACTION_RECORD_TYPE);
Record record;
trRecordBuilder.setString(RECORD_TYPE_NAME,"TR");
trRecordBuilder.setString(ID_NAME,"0001");
trRecordBuilder.setString(TRADE_DATE_NAME,"03/25/2005");
trRecordBuilder.setString(TRADE_TIME_NAME,"01:50:00");
trRecordBuilder.setString(DESCRIPTION_NAME,"This is a trade record");
record = trRecordBuilder.toRecord();
trRecordBuilder.clear();
// Write the "trade" record
writeRecord(context, flow, record);
tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
tnRecordBuilder.setString(ID_NAME,"0002");
tnRecordBuilder.setString(REFERENCE_NAME,"X1234");
tnRecordBuilder.setString(DESCRIPTION_NAME,"A child transaction");
record = tnRecordBuilder.toRecord();
tnRecordBuilder.clear();
// Write the first "transaction" record
writeRecord(context, flow, record);
tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
tnRecordBuilder.setString(ID_NAME,"0003");
tnRecordBuilder.setString(REFERENCE_NAME,"X1235");
tnRecordBuilder.setString(DESCRIPTION_NAME,"Another child transaction");
record = tnRecordBuilder.toRecord();
tnRecordBuilder.clear();
// Write the second "transaction" record
writeRecord(context, flow, record);
// End the record stream
endRecordStream(context, flow);
}
}
The
TradeRecordReader
class will generate the following stream of records.
| record_type | id | trade_date | trade_time | description |
|---|---|---|---|---|
| TR | 0001 | 03/25/2005 | 1:50:00 | This is a trade record |
| record_type | id | reference | description |
|---|---|---|---|
| TN | 0002 | X1234 | A child transaction |
| TN | 0003 | X1235 | Another child transaction |
The desired output is the same as for the trades example.
Figure 121. Output XML file trades.xml
<?xml version="1.0" encoding="utf-8"?>
<trades feed="LONDON">
<trade id="0001">
<trade-date>2005-03-25T01:50:00.000-05:00</trade-date>
<description>This is a trade record</description>
</trade>
<transaction id="0002">
<reference>X1234</reference>
<description>A child transaction</description>
</transaction>
<transaction id="0003">
<reference>X1235</reference>
<description>Another child transaction</description>
</transaction>
</trades>
The following resources script does the transformation.
Figure 122. Resources script resources-custom_trade_reader.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="trades">
<sx:parameter name="feed">
<sx:defaultValue>NY</sx:defaultValue>
</sx:parameter>
<sx:serialize>
<sx:transform>
<sx:content ref="trades"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="trades">
<sx:customRecordReader class="flat2xml.TradeRecordReader"/>
<sx:recordMapping ref="tradesToXmlMapping"/>
</sx:recordContent>
<sx:recordMapping id="tradesToXmlMapping">
<trades>
<sx:fieldAttributeMap value="{$feed}" attribute="feed"/>
<sx:onRecord recordType="trade">
<trade>
<sx:fieldAttributeMap field="id" attribute="id"/>
<sx:elementMap element="trade-date">
<sx:convertToDateTime format="MM/dd/yyyy H:mm:ss">
<sx:concat separator=" ">
<sx:toString value="{trade_date}"/>
<sx:toString value="{trade_time}"/>
</sx:concat>
</sx:convertToDateTime>
</sx:elementMap>
<sx:fieldElementMap field="description" element="description"/>
</trade>
</sx:onRecord>
<sx:onRecord recordType="transaction">
<transaction>
<sx:fieldAttributeMap field="id" attribute="id"/>
<sx:fieldElementMap field="reference" element="reference"/>
<sx:fieldElementMap field="description" element="description"/>
</transaction>
</sx:onRecord>
</trades>
</sx:recordMapping>
</sx:resources>
Note the following points.
TradeRecordReader
.
You can run this example on the command line by
TradeRecordReader
and copying
the resulting
.class
file into the
dir/classes
directory
servingxml -r resources-custom_trade_reader.xml
-o output/trades.xml trades feed=LONDON
This example shows how to serialize an XML document in batches. It illustrates the use of the sx:batchedSerializer element.
Suppose you want to read the CSV file countries.csv and serialize it to XML, but you also want the following:
Country subtrees to be split over multiple documents, no more than 50 to any one file.
06/12/2007 07:37 PM <DIR> .
06/12/2007 07:37 PM <DIR> ..
06/12/2007 07:37 PM 3,773 countries-1.xml
06/12/2007 07:37 PM 3,758 countries-2.xml
06/12/2007 07:37 PM 3,747 countries-3.xml
06/12/2007 07:37 PM 3,753 countries-4.xml
06/12/2007 07:37 PM 3,416 countries-5.xml
Each document to begin with all the XML nodes up to the occurance of the first country subtree.
The following resources script produces the desired output.
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="countries">
<sx:serialize>
<sx:transform>
<sx:content ref="countries"/>
</sx:transform>
<sx:batchedSerializer path="country" batchSize="50">
<sx:xsltSerializer>
<sx:fileSink file="output/countries-{$sx:batchSequenceNumber}.xml"/>
</sx:xsltSerializer>
</sx:batchedSerializer>
</sx:serialize>
</sx:service>
<sx:recordContent id="countries" name="countries">
<sx:recordStream>
<sx:flatFileReader>
<sx:urlSource url="data/countries.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileReader>
<msv:recordValidator>
<sx:urlSource url="data/country-record.xsd"/>
</msv:recordValidator>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
</sx:discardHandler>
</sx:recordStream>
<sx:recordMapping ref="countriesToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="countriesFile">
<sx:commentStarter value="#"/>
<sx:flatFileHeader lineCount="1"/>
<sx:flatFileBody>
<sx:flatRecordType name="country">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="code"/>
<sx:delimitedField name="name" trimLeading="true"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="countriesToXmlMapping">
<countries>
<sx:onRecord>
<country>
<sx:fieldElementMap field="name" element="countryName"/>
<sx:fieldAttributeMap field="code" attribute="countryCode"/>
</country>
</sx:onRecord>
</countries>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-batches.xml countries
rootName=countries
The example illustrates an alternative way of producing XML in batches, in two steps:
First, split a large flat file into a number of smaller flat files.
Second, convert each of the smaller flat files to XML.
Suppose you want to convert the countries.csv file containing 245 country records to XML, each country record mapped to one country subtree, and the output spilt over multiple XML files.
The first step is to produce the batched csv files and output them to an intermediate directory.
Figure 123. Batched intermediate CSV files.
06/12/2007 07:24 PM <DIR> .
06/12/2007 07:24 PM <DIR> ..
06/12/2007 07:24 PM 814 countries-1.csv
06/12/2007 07:24 PM 795 countries-2.csv
06/12/2007 07:24 PM 786 countries-3.csv
06/12/2007 07:24 PM 784 countries-4.csv
06/12/2007 07:24 PM 793 countries-5.csv
The second step is to read all the csv files in the intermediate directory and serialize them to XML.
Figure 124. Batched XML files.
06/12/2007 07:37 PM <DIR> .
06/12/2007 07:37 PM <DIR> ..
06/12/2007 07:37 PM 3,773 countries-1.xml
06/12/2007 07:37 PM 3,758 countries-2.xml
06/12/2007 07:37 PM 3,747 countries-3.xml
06/12/2007 07:37 PM 3,753 countries-4.xml
06/12/2007 07:37 PM 3,416 countries-5.xml
The following resources script performs both steps.
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="countries">
<sx:recordStream>
<sx:flatFileReader>
<sx:fileSource file="data/{$rootName}.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileReader>
<sx:batchedRecordWriter batchSize="50">
<sx:flatFileWriter>
<sx:fileSink file="intermediate/{$rootName}-{$sx:batchSequenceNumber}.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileWriter>
</sx:batchedRecordWriter>
</sx:recordStream>
<sx:recordStream>
<sx:directoryReader directory="intermediate">
<sx:fileFilter pattern="countries.*[.]csv"/>
</sx:directoryReader>
<sx:processRecord>
<sx:parameter name="output-file">
<sx:findAndReplace searchFor ="(countries.*)[.]csv" replaceWith ="$1.xml">
<sx:toString value="{name}"/>
</sx:findAndReplace>
</sx:parameter>
<sx:serialize>
<sx:xsltSerializer>
<sx:fileSink directory="output" file="{$output-file}"/>
</sx:xsltSerializer>
<sx:transform>
<sx:content ref="countries"/>
</sx:transform>
</sx:serialize>
</sx:processRecord>
</sx:recordStream>
</sx:service>
<sx:recordContent id="countries">
<sx:flatFileReader>
<sx:fileSource directory="{parentDirectory}" file="{name}"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="countriesToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="countriesFile">
<sx:commentStarter value="#"/>
<sx:flatFileBody>
<sx:flatRecordType name="country">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="code"/>
<sx:delimitedField name="name" trimLeading="true"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="countriesToXmlMapping">
<countries>
<sx:onRecord>
<country>
<sx:fieldElementMap field="name" element="countryName"/>
<sx:fieldAttributeMap field="code" attribute="countryCode"/>
</country>
</sx:onRecord>
</countries>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-batchRecords.xml countries
rootName=countries
This example requires reading the flat file
ADM2301EOBS
HDR06/13/08200815501300101MEMBER01 BISHOP
ADR1234 TEST DR ARVADA
HDR06/13/08200815501300102MEMBER02 BISHOP
ADR5678 TEST DR ARVADA
TLRTOTAL EOBS GENERATED: 0000001000000000000010000018
and producing this output:
One summary XML document capturing the header ("ADM") and trailer ("TLR") records
eobinputoutput.xml
Separate XML documents for each pair of "HDR" and "ADR" records
eobinputoutput-061308200815501300101.xml
eobinputoutput-061308200815501300102.xml
The summary XML document is to look like
<?xml version="1.0" encoding="UTF-8"?>
<eobs>
<ADMIN_RECTYPE record_type="ADM">
<ADM_0001_TEXT>2301EOBS</ADM_0001_TEXT>
</ADMIN_RECTYPE>
<TLR-REC-TYPE record_type="TLR">
<TLR_0001_TEXT>TOTAL EOBS GENERATED: 0</TLR_0001_TEXT>
<TLR_0001_COUNT>0000010</TLR_0001_COUNT>
<FILLER>00000000000010000018</FILLER>
</TLR-REC-TYPE>
</eobs>
and the first detail documents as
<?xml version="1.0" encoding="UTF-8"?>
<eob>
<HEADER_RECTYPE record_type="HDR">
<HDR_STMT_DATE>06/13/08</HDR_STMT_DATE>
<HDR_CLAIM_NUMBER>200815501300101</HDR_CLAIM_NUMBER>
<HDR_PATIENT_NAME_FIRST>MEMBER01</HDR_PATIENT_NAME_FIRST>
<HDR_PATIENT_NAME_MIDDLE/>
<HDR_PATIENT_NAME_LAST>BISHOP</HDR_PATIENT_NAME_LAST>
</HEADER_RECTYPE>
<ADR_REC_TYPE record_type="ADR">
<ADR_8512_ATTN>1234 TEST DR</ADR_8512_ATTN>
<ADR_8513_STREET1/>
<ADR_8513_STREET2/>
<ADR_8514_CITY>ARVADA</ADR_8514_CITY>
</ADR_REC_TYPE>
</eob>
Below is the required resources script. Since the structure of the record input will be shared with another example, we'll collect that in a separate script and include it in the main one.
Figure 125. Record structure resource-eobclaims-flatfile.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:flatFile id="eobclaims-file">
<sx:flatFileBody>
<sx:flatRecordType ref="eobclaims-record"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordTypeChoice id="eobclaims-record">
<sx:positionalField name="record_type" width="3"/>
<sx:when test="record_type='ADM'">
<sx:flatRecordType ref="adm"/>
</sx:when>
<sx:when test="record_type='HDR'">
<sx:flatRecordType ref="hdr"/>
</sx:when>
<sx:when test="record_type='ADR'">
<sx:flatRecordType ref="adr"/>
</sx:when>
<sx:when test="record_type='TLR'">
<sx:flatRecordType ref="tlr"/>
</sx:when>
</sx:flatRecordTypeChoice>
<sx:flatRecordType id="adm" name="ADM">
<sx:positionalField name="record_type" width="3"/>
<sx:positionalField name="ADM_0001_TEXT" width="20"/>
<sx:positionalField name="ADM_0001_RUN_DATE" width="249"/>
<sx:positionalField name="FILLER" width="8"/>
</sx:flatRecordType>
<sx:flatRecordType id="tlr" name="TLR">
<sx:positionalField name="record_type" width="3"/>
<sx:positionalField name="TLR_0001_TEXT" width="23"/>
<sx:positionalField name="TLR_0001_COUNT" width="7"/>
<sx:positionalField name="FILLER" width="268"/>
</sx:flatRecordType>
<sx:flatRecordType id="hdr" name="HDR">
<sx:positionalField name="record_type" width="3"/>
<sx:positionalField name="HDR_STMT_DATE" width="8"/>
<sx:positionalField name="HDR_CLAIM_NUMBER" width="15"/>
<sx:positionalField name="HDR_PATIENT_NAME_FIRST" width="18"/>
<sx:positionalField name="HDR_PATIENT_NAME_LAST" width="15"/>
</sx:flatRecordType>
<sx:flatRecordType id="adr" name="ADR">
<sx:positionalField name="record_type" width="3"/>
<sx:positionalField name="ADR_8512_ATTN" width="99"/>
<sx:positionalField name="ADR_8514_CITY" start="103" width="18"/>
</sx:flatRecordType>
</sx:resources>
Figure 126. Main resources script resource-eobclaims.xml
<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:include href="resources-eobclaims-flatfile.xml"/>
<sx:service id="eobclaims-to-xml">
<!-- Do the header/trailer first -->
<sx:serialize>
<sx:xsltSerializer>
<sx:fileSink directory="output" file="eobinputoutput.xml"/>
<sx:outputProperty name="indent" value="yes"/>
</sx:xsltSerializer>
<sx:transform>
<sx:recordContent>
<sx:flatFileReader>
<sx:flatFile ref="eobclaims-file"/>
</sx:flatFileReader>
<sx:recordMapping ref="eobclaims-header-trailer-mapping"/>
</sx:recordContent>
</sx:transform>
</sx:serialize>
<!-- Now do the body -->
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="eobclaims-file"/>
</sx:flatFileReader>
<!-- Compose a composite record out of each HDR-ADR pair -->
<sx:combineRecords recordType="composite" repeatingGroup="eob"
startTest="sx:current/HDR"
endTest="sx:previous/ADR">
<sx:newField name="HDR_STMT_DATE" select="eob/HDR/HDR_STMT_DATE"/>
<sx:newField name="HDR_CLAIM_NUMBER" select="eob/HDR/HDR_CLAIM_NUMBER"/>
</sx:combineRecords>
<!-- Discard the records that are not HDR-ADR pairs -->
<sx:restrictRecordFilter>
<sx:recordRestriction recordType="composite"/>
</sx:restrictRecordFilter>
<!-- Process each HDR-ADR composite record -->
<sx:processRecord>
<!-- Construct and store the output filename in the parameter "output-file" -->
<sx:parameter name="output-file">
<sx:findAndReplace searchFor="/" replaceWith ="">
<sx:toString value="eobinputoutput-{HDR_STMT_DATE}{HDR_CLAIM_NUMBER}.xml"/>
</sx:findAndReplace>
</sx:parameter>
<sx:serialize>
<sx:xsltSerializer>
<sx:fileSink directory="output" file="{$output-file}"/>
<sx:outputProperty name="indent" value="yes"/>
</sx:xsltSerializer>
<sx:transform>
<sx:recordContent>
<sx:recordMapping ref="eobclaims-body-mapping"/>
</sx:recordContent>
</sx:transform>
</sx:serialize>
</sx:processRecord>
</sx:recordStream>
</sx:service>
<sx:recordMapping id="eobclaims-header-trailer-mapping">
<eobs>
<sx:onRecord recordType="ADM">
<ADMIN_RECTYPE record_type="ADM">
<sx:fieldElementMap field="ADM_0001_TEXT" element="ADM_0001_TEXT"/>
</ADMIN_RECTYPE>
</sx:onRecord>
<sx:onRecord recordType="TLR">
<TLR-REC-TYPE record_type="TLR">
<sx:fieldElementMap field="TLR_0001_TEXT" element="TLR_0001_TEXT"/>
<sx:fieldElementMap field="TLR_0001_COUNT" element="TLR_0001_COUNT"/>
<sx:fieldElementMap field="FILLER" element="FILLER"/>
</TLR-REC-TYPE>
</sx:onRecord>
</eobs>
</sx:recordMapping>
<sx:recordMapping id="eobclaims-body-mapping">
<eob>
<!-- Our composite record has a repeating group field, "eob",
that has two sub-records, "HDR" and "ADR" -->
<sx:onRecord recordType="composite">
<sx:subrecordMapping repeatingGroup="eob">
<sx:onRecord recordType="HDR">
<HEADER_RECTYPE>
<sx:fieldAttributeMap field="record_type" attribute="record_type"/>
<sx:fieldElementMap field="HDR_STMT_DATE" element="HDR_STMT_DATE"/>
<sx:fieldElementMap field="HDR_CLAIM_NUMBER" element="HDR_CLAIM_NUMBER"/>
<sx:fieldElementMap field="HDR_PATIENT_NAME_FIRST" element="HDR_PATIENT_NAME_FIRST"/>
<sx:fieldElementMap field="HDR_PATIENT_NAME_MIDDLE" element="HDR_PATIENT_NAME_MIDDLE"/>
<sx:fieldElementMap field="HDR_PATIENT_NAME_LAST" element="HDR_PATIENT_NAME_LAST"/>
</HEADER_RECTYPE>
</sx:onRecord>
<sx:onRecord recordType="ADR">
<ADR_REC_TYPE>
<sx:fieldAttributeMap field="record_type" attribute="record_type"/>
<sx:fieldElementMap field="ADR_8512_ATTN" element="ADR_8512_ATTN"/>
<sx:fieldElementMap field="ADR_8513_STREET1" element="ADR_8513_STREET1"/>
<sx:fieldElementMap field="ADR_8513_STREET2" element="ADR_8513_STREET2"/>
<sx:fieldElementMap field="ADR_8514_CITY" element="ADR_8514_CITY"/>
</ADR_REC_TYPE>
</sx:onRecord>
</sx:subrecordMapping>
</sx:onRecord>
</eob>
</sx:recordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -i data/eobclaims.txt -r resources-eobclaims.xml eobclaims-to-xml
This example shows a resources script that maps a primary input file to XML, with lookups from a secondary file.
The primary input file is the pipe delimited file,
primary-input.txt.
HDR|a|b
DET|a|b
DET|a|b
MSG|1|a
MSG|3|a
END
The secondary input file is the pipe delimited file,
messages.txt,
1|test message 1
2|test message 2
3|test message 3
You need to read the primary input file and replace the id
tag values under
message with the message corresponding to the id. The expected output is
<?xml version="1.0" encoding="UTF-8"?>
<mytest>
<field1>a</field1>
<field2>b</field2>
<detail>
<field1>a</field1>
<field2>b</field2>
</detail>
<detail>
<field1>a</field1>
<field2>b</field2>
</detail>
<message>
<message>test message 1</message>
<field2>a</field2>
</message>
<message>
<message>test message 3</message>
<field2>a</field2>
</message>
</mytest>
The required resources script is
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="multiple-reader-test">
<sx:serialize>
<sx:xsltSerializer>
<sx:outputProperty name="indent" value="yes"/>
</sx:xsltSerializer>
<sx:transform>
<sx:content ref="multiple-reader-content"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:recordContent id="multiple-reader-content">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="primary-file"/>
</sx:flatFileReader>
</sx:recordStream>
<sx:recordMapping ref="primary-file-to-xml"/>
</sx:recordContent>
<sx:flatFile id="primary-file">
<sx:commentStarter value="#"/>
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="tag"/>
<sx:when test="tag='HDR'">
<sx:flatRecordType name="header">
<sx:delimitedField name="tag"/>
<sx:delimitedField name="field1"/>
<sx:delimitedField name="field2"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='DET'">
<sx:flatRecordType name="detail">
<sx:delimitedField name="tag"/>
<sx:delimitedField name="field1"/>
<sx:delimitedField name="field2"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="tag='MSG'">
<sx:flatRecordType name="message">
<sx:delimitedField name="tag"/>
<sx:delimitedField name="field1"/>
<sx:delimitedField name="field2"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:recordMapping id="primary-file-to-xml">
<mytest>
<sx:onRecord recordType="header">
<sx:fieldElementMap field="field1" element="field1"/>
<sx:fieldElementMap field="field2" element="field2"/>
</sx:onRecord>
<sx:onRecord recordType="detail">
<detail>
<sx:fieldElementMap field="field1" element="field1"/>
<sx:fieldElementMap field="field2" element="field2"/>
</detail>
</sx:onRecord>
<sx:onRecord recordType="message">
<message>
<sx:parameter name="my-id" value="{field1}"/>
<sx:fieldElementMap select="document('messageContent')/messages/message[id=$my-id]/description" element="message"/>
<sx:fieldElementMap field="field2" element="field2"/>
</message>
</sx:onRecord>
</mytest>
</sx:recordMapping>
<sx:recordContent id="messageContent" name="messages">
<sx:recordStream>
<sx:flatFileReader>
<sx:urlSource url="messages.txt"/>
<sx:flatFile ref="messagesFile"/>
</sx:flatFileReader>
</sx:recordStream>
</sx:recordContent>
<sx:flatFile id="messagesFile" name="messages">
<sx:flatFileBody>
<sx:fieldDelimiter value="|"/>
<sx:flatRecordType name="message">
<sx:delimitedField name="id"/>
<sx:delimitedField name="description"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
You can run this example on the command line by entering
servingxml -i primary-input.txt -r resources.xml -o output.xml multiple-reader-test
The example below illustrates how to prepare a resources script that will convert an XML document into a positional flat file, and include the record count in a trailer section.
The input file is an XML file.
Figure 127. Input XML file books.xml
<myns:books xmlns:myns="http://mycompany.com/mynames/">
<myns:book categoryCode="F">
<myns:title>Kafka on the Shore</myns:title>
<myns:author>Haruki Murakami</myns:author>
<myns:price>25.17</myns:price>
</myns:book>
<myns:book categoryCode="C">
<myns:title>Concurrent Programming in Java</myns:title>
<myns:author>Doug Lea</myns:author>
<myns:price>77.13</myns:price>
</myns:book>
</myns:books>
The desired output is shown below.
Figure 128. Output positional flat file books.txt
CAuthor Title Price FHaruki Murakami Kafka on the Shore 25.17 CDoug Lea Concurrent Programming in Java 77.13 Number of records: 2
The following resources script does the transformation.
Figure 129. Resources script resources-books.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="books2pos">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="booksToFileMapping"/>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="booksFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="booksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="bookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:flatRecordType ref="trailer"/>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="bookType" name="bookType">
<sx:positionalField name="category" label="Category" width="1"/>
<sx:positionalField name="author" label="Author" width="30"/>
<sx:positionalField name="title" label="Title" width="30"/>
<sx:positionalField name="price" label="Price" width="10" justify="right"/>
</sx:flatRecordType>
<sx:flatRecordType id="trailer" name="trailer">
<sx:positionalField name="recordCountLabel" width="19" label="Number of records: "/>
<sx:positionalField name="recordCount" width="10" label="{$sx:recordCount}"/>
</sx:flatRecordType>
<sx:inverseRecordMapping id="booksToFileMapping">
<sx:onSubtree path="/myns:books/myns:book">
<sx:flattenSubtree recordType="book">
<sx:subtreeFieldMap select="myns:title" field="title"/>
<sx:subtreeFieldMap select="@categoryCode" field="category"/>
<sx:subtreeFieldMap select="myns:author" field="author"/>
<sx:subtreeFieldMap select="myns:price" field="price"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
Note the following:
In header and trailer sections, the label attributes of fields are treated as data values.
In a sx:positionalField element, the value of the label
attribute can contain references to parameters, in
particular to $sx:recordCount, which is a parameter that the file record writer adds after finishing writing records.
You can run this example on the command line by entering
servingxml -r resources-books2pos.xml -i data/books.xml
-o output/books.txt books2pos
This example shows one way to prepare a resources script that will convert an XML document having multiple occurances of an element into a CSV file.
The input file is an XML file with multiple occurances of the element myns:review.
Figure 130. Input XML file books.xml
<myns:books xmlns:myns="http://mycompany.com/mynames/">
<myns:book categoryCode="F">
<myns:title>Factotum</myns:title>
<myns:author>Charles Bukowski</myns:author>
<myns:price/>
<myns:reviews>
<myns:review>*****</myns:review>
<myns:review>*</myns:review>
</myns:reviews>
</myns:book>
<myns:book categoryCode="F">
<myns:title>Kafka on the Shore</myns:title>
<myns:author>Haruki Murakami</myns:author>
<myns:price>25.17</myns:price>
<myns:reviews>
<myns:review>*****</myns:review>
<myns:review>****</myns:review>
</myns:reviews>
</myns:book>
<myns:book categoryCode="F">
<myns:title>Mr Mee</myns:title>
<myns:author>Andrew Crumey</myns:author>
<myns:price>22.00</myns:price>
<myns:reviews>
<myns:review>*****</myns:review>
<myns:review>*</myns:review>
</myns:reviews>
</myns:book>
<myns:book categoryCode="C">
<myns:title>Concurrent Programming in Java</myns:title>
<myns:author>Doug Lea</myns:author>
<myns:price>77.13</myns:price>
<myns:reviews>
<myns:review>****</myns:review>
</myns:reviews>
</myns:book>
<myns:book categoryCode="B">
<myns:title>Sex, Lies, and Headlocks</myns:title>
<myns:author>Shaun Assael</myns:author>
<myns:price>7.24 </myns:price>
<myns:reviews>
<myns:review>****</myns:review>
</myns:reviews>
</myns:book>
</myns:books>
The desired output is shown below.
Figure 131. Output CSV file books.csv
Category,Author,Title,Price,Review 1,Review 2 F,Charles Bukowski,Factotum,,*****,* F,Haruki Murakami,Kafka on the Shore,25.17,*****,**** F,Andrew Crumey,Mr Mee,22.00,*****,* C,Doug Lea,Concurrent Programming in Java,77.13,****, B,Shaun Assael,"Sex, Lies, and Headlocks",7.24 ,****,
Note that, because it contains embedded commas, the title of the last book must be enclosed in quote marks.
The following resources script does the transformation.
Figure 132. Resources script resources-books2csv.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="books2csv">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="booksToFileMapping"/>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="booksFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="booksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="bookType" name="bookType">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="category" label="Category"/>
<sx:delimitedField name="author" label="Author"/>
<sx:delimitedField name="title" label="Title"/>
<sx:delimitedField name="price" label="Price"/>
<sx:delimitedField name="review1" label="Review 1"/>
<sx:delimitedField name="review2" label="Review 2"/>
</sx:flatRecordType>
<sx:inverseRecordMapping id="booksToFileMapping">
<sx:onSubtree path="/myns:books/myns:book">
<sx:flattenSubtree recordType="book">
<sx:subtreeFieldMap select="myns:title" field="title"/>
<sx:subtreeFieldMap select="@categoryCode" field="category"/>
<sx:subtreeFieldMap select="myns:author" field="author"/>
<sx:subtreeFieldMap select="myns:price" field="price"/>
<sx:subtreeFieldMap select="myns:reviews/myns:review[1]" field="review1"/>
<sx:subtreeFieldMap select="myns:reviews/myns:review[2]" field="review2"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -o output/persons.xml -r resources-persons.xml
personsAddresses
This example shows another way of writing a resources script that will convert an XML document having multiple occurances of an element into a delimited flat file.
The XML input file is the same as the previous example. The desired output, though, is slightly different.
Figure 133. Output pipe delimited flat file books-pipe.txt
Category|Author|Title|Price|review Fiction|Charles Bukowski|Factotum||*****;* Fiction|Haruki Murakami|Kafka on the Shore|25.17|*****;**** Fiction|Andrew Crumey|Mr Mee|22.00|*****;* Computers|Doug Lea|Concurrent Programming in Java|77.13|**** Biography|Shaun Assael|Sex, Lies, and Headlocks|7.24 |****
Note the following features of this output.
Fields are separated by the pipe delimiter.
Ratings appear in a multivalued field separated by semi-colon subfield delimiters.
"Fiction" has been substituted for "F", "Computers" for "C", etc.
This time the resources script looks like this.
Figure 134. Resources script resources-books2pipe.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="books2pipe">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="booksToFileMapping"/>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
<sx:choose>
<sx:when test="category='F'">
<sx:modifyRecord>
<sx:newField name="category" value="Fiction"/>
</sx:modifyRecord>
</sx:when>
<sx:when test="category='C'">
<sx:modifyRecord>
<sx:newField name="category" value="Computers"/>
</sx:modifyRecord>
</sx:when>
<sx:when test="category='B'">
<sx:modifyRecord>
<sx:newField name="category" value="Biography"/>
</sx:modifyRecord>
</sx:when>
</sx:choose>
<sx:flatFileWriter>
<sx:flatFile ref="booksFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="booksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="bookType" name="bookType">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="category" label="Category"/>
<sx:delimitedField name="author" label="Author"/>
<sx:delimitedField name="title" label="Title"/>
<sx:delimitedField name="price" label="Price"/>
<sx:delimitedField name="review" label="review">
<sx:subfieldDelimiter value=";"/>
</sx:delimitedField>
</sx:flatRecordType>
<sx:inverseRecordMapping id="booksToFileMapping">
<sx:onSubtree path="/myns:books/myns:book">
<sx:flattenSubtree recordType="book">
<sx:subtreeFieldMap select="myns:title" field="title"/>
<sx:subtreeFieldMap select="@categoryCode" field="category"/>
<sx:subtreeFieldMap select="myns:author" field="author"/>
<sx:subtreeFieldMap select="myns:price" field="price"/>
<sx:subtreeFieldMap match="myns:reviews/myns:review" select="text()" field="review"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-books2pipe.xml -i data/books.xml
-o output/books-pipe.txt books2pipe
This example shows how to write a resources script that will convert each book entry
in the books.xml document into a list of three records, the
first comma separated, and the last two space separated.
The XML input file is the same as the previous example.
The desired output is follows.
Figure 135. Output flat file book_lisiting.txt
"Factotum", Charles Bukowski Category Fiction Price $ "Kafka on the Shore", Haruki Murakami Category Fiction Price $25.17 "Mr Mee", Andrew Crumey Category Fiction Price $22.00 "Concurrent Programming in Java", Doug Lea Category Computers Price $77.13 "Sex, Lies, and Headlocks", Shaun Assael Category Biography Price $7.24
Note the following features of this output.
The categoryCode 'F' is mapped to "Fiction", 'C' to "Computers", 'B' to "Biography".
A dollar sign '$' is prepended to the price (even if the price is empty.)
Each book title is enclosed in quotation marks.
Each book subtree is output as three records; the first delimited by ", ", the second delimited by a space character.
This time the resources script looks like this.
Figure 136. Resources script resources-book_listing.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="book-listing">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="booksToFileMapping"/>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
<!-- Convert category code to label -->
<sx:choose>
<sx:when test="category='F'">
<sx:modifyRecord>
<sx:newField name="category" value="Fiction"/>
</sx:modifyRecord>
</sx:when>
<sx:when test="category='C'">
<sx:modifyRecord>
<sx:newField name="category" value="Computers"/>
</sx:modifyRecord>
</sx:when>
<sx:when test="category='B'">
<sx:modifyRecord>
<sx:newField name="category" value="Biography"/>
</sx:modifyRecord>
</sx:when>
</sx:choose>
<sx:flatFileWriter>
<sx:flatFile ref="booksFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="booksFile">
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordTypeChoice id="bookType">
<sx:delimitedField name="xxx"/> <!-- unused here, used for input only -->
<sx:when test="/book"> <!-- matches against record type "book" -->
<sx:flatRecordType name="book">
<sx:fieldDelimiter value=", "/>
<sx:delimitedField name="title" quote="always"/>
<sx:delimitedField name="author"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="/bookCategory"> <!-- matches against record type "bookCategory" -->
<sx:flatRecordType name="book">
<sx:fieldDelimiter value=" "/>
<sx:delimitedField name="label"/>
<sx:delimitedField name="category"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="/bookPrice"> <!-- matches against record type "bookPrice" -->
<sx:flatRecordType name="book">
<sx:fieldDelimiter value=" "/>
<sx:delimitedField name="label"/>
<sx:delimitedField name="price"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
<sx:inverseRecordMapping id="booksToFileMapping">
<sx:onSubtree path="/myns:books/myns:book">
<sx:flattenSubtree recordType="book">
<sx:subtreeFieldMap select="myns:title" field="title"/>
<sx:subtreeFieldMap select="myns:author" field="author"/>
</sx:flattenSubtree>
<sx:flattenSubtree recordType="bookCategory">
<sx:subtreeFieldMap select="'Category'" field="label"/>
<sx:subtreeFieldMap select="@categoryCode" field="category"/>
</sx:flattenSubtree>
<sx:flattenSubtree recordType="bookPrice">
<sx:subtreeFieldMap select="'Price'" field="label"/>
<!-- Append dollar sign to price -->
<sx:subtreeFieldMap select="concat('$',myns:price)" field="price"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-book_listing.xml -i data/books.xml
-o output/book_listing.txt book-listing
This example shows another way of writing a resources script that will convert an XML document into flat file records with multi-valued fields.
The input file is an XML file with multiple occurances of the element GeoPoint inside a Swath element.
Figure 137. Input XML file swath.xml
<Swath>
<Shape>Polygon</Shape>
<GeoPoint>
<lat>1</lat>
<lon>2</lon>
</GeoPoint>
<GeoPoint>
<lat>3</lat>
<lon>4</lon>
</GeoPoint>
<GeoPoint>
<lat>5</lat>
<lon>6</lon>
</GeoPoint>
</Swath>
The desired output is shown below.
This time the resources script looks like this.
Figure 139. Resources script resources-swath2pipe.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="swath">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping >
<sx:onSubtree path="/Swath">
<sx:flattenSubtree recordType="hdrType">
<sx:subtreeFieldMap select="Shape" field="Shape"/>
<sx:subtreeFieldMap match="GeoPoint" select="concat(lat,' ',lon)" field="GeoPoint"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="swathFlatFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="swathFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="hdrType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="hdrType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="hdrType" name="hdrType">
<sx:delimitedField name="Shape">
<sx:fieldDelimiter value="|"/>
</sx:delimitedField>
<sx:delimitedField name="GeoPoint">
<sx:subfieldDelimiter value=";"/>
</sx:delimitedField>
</sx:flatRecordType>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-swath.xml -i data/swath.xml
-o output/swath.txt swath
This example shows how to write a resources script that will convert an XML document into a flat file records with header and multiple detail records.
The input file is an XML file with multiple occurances of the element myns:review inside a myns:book/myns:reviews path.
Figure 140. Input XML file bookReviews.xml
<?xml version="1.0" encoding="utf-8"?>
<myns:books xmlns:myns="http://mycompany.com/mynames/">
<myns:book id="002" categoryCode="F">
<myns:title>Kafka on the Shore</myns:title>
<myns:author>Haruki Murakami</myns:author>
<myns:price>25.17</myns:price>
<myns:reviews>
<myns:review>
<myns:reviewer>Curley</myns:reviewer>
<myns:rating>*****</myns:rating>
</myns:review>
<myns:review>
<myns:reviewer>Larry</myns:reviewer>
<myns:rating>***</myns:rating>
</myns:review>
<myns:review>
<myns:reviewer>Moe</myns:reviewer>
<myns:rating>*</myns:rating>
</myns:review>
</myns:reviews>
</myns:book>
</myns:books>
The desired output is shown below.
Figure 141. Output positional flat file bookReviews.txt
TID CAuthor Title Price H002FHaruki Murakami Kafka on the Shore 25.17 L002Curley ***** L002Larry *** L002Moe * This is a trailer record
The following resources script does the job.
Figure 142. Resources script resources-bookReviews.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="bookReviews">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="booksToFileMapping"/>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
</sx:recordStream>
<sx:flatFileWriter>
<sx:flatFile ref="booksFile"/>
</sx:flatFileWriter>
</sx:service>
<sx:flatFile id="booksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="book"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="tag" width="1"/>
<sx:when test="tag='H'">
<sx:flatRecordType ref="book"/>
</sx:when>
<sx:when test="tag='L'">
<sx:flatRecordType ref="review"/>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="book" name="book">
<sx:positionalField name="tag" label="Tag" width="1"/>
<sx:positionalField name="id" label="ID" width="3"/>
<sx:positionalField name="category" label="Category" width="1"/>
<sx:positionalField name="author" label="Author" width="30"/>
<sx:positionalField name="title" label="Title" width="30"/>
<sx:positionalField name="price" label="Price" width="10" justify="right"/>
</sx:flatRecordType>
<sx:flatRecordType id="review" name="review">
<sx:positionalField name="tag" label="Tag" width="1"/>
<sx:positionalField name="id" label="ID" width="3"/>
<sx:positionalField name="reviewer" label="Reviewer" width="10"/>
<sx:positionalField name="rating" label="Rating" width="5"/>
</sx:flatRecordType>
<sx:inverseRecordMapping id="booksToFileMapping">
<sx:onSubtree path="/myns:books/myns:book">
<sx:parameter name="id" select="@id"/>
<sx:flattenSubtree recordType="book">
<sx:subtreeFieldMap select="'H'" field="tag"/>
<sx:subtreeFieldMap select="$id" field="id"/>
<sx:subtreeFieldMap select="myns:title" field="title"/>
<sx:subtreeFieldMap select="@categoryCode" field="category"/>
<sx:subtreeFieldMap select="myns:author" field="author"/>
<sx:subtreeFieldMap select="myns:price" field="price"/>
</sx:flattenSubtree>
<sx:flattenSubtree match="myns:reviews/myns:review" recordType="review">
<sx:subtreeFieldMap select="'L'" field="tag"/>
<sx:subtreeFieldMap select="$id" field="id"/>
<sx:subtreeFieldMap select="myns:reviewer" field="reviewer"/>
<sx:subtreeFieldMap select="myns:rating" field="rating"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-bookReviews.xml -i data/bookReviews.xml
-o output/bookReviews.txt bookReviews
The example below illustrates how to use an XSLT stylesheet in an XML-flat pipeline to transform the XML into a form that is easier to map.
The input file is an XML file containing a sequence of elements.
Figure 143. Input XML file sequence.xml
<root> <first>1111</first> <second>AD</second> <third>1111</third> <fourth>1111</fourth> <first>2222</first> <second>MO</second> <third>2222</third> <fourth>2222</fourth> </root>
The desired output file is shown below.
The following resources script does the transformation.
Figure 145. Resources script resources-sequence.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="sequence-csv">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:transform>
<sx:document>
<sx:fileSource file="data/sequence.xml"/>
</sx:document>
<sx:xslt ref="sequence-hierarchy"/>
</sx:transform>
<sx:inverseRecordMapping ref="sequenceToFileMapping"/>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="sequenceFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:xslt id="sequence-hierarchy">
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/root">
<xsl:copy>
<xsl:apply-templates select="first"/>
</xsl:copy>
</xsl:template>
<xsl:template match="first">
<PivotNode>
<first>
<xsl:value-of select="."/>
</first>
<second>
<xsl:value-of select="following-sibling::second[1]"/>
</second>
<third>
<xsl:value-of select="following-sibling::third[1]"/>
</third>
<fourth>
<xsl:value-of select="following-sibling::fourth[1]"/>
</fourth>
</PivotNode>
</xsl:template>
</xsl:stylesheet>
</sx:xslt>
<sx:flatFile id="sequenceFile">
<sx:flatFileBody>
<sx:flatRecordType name="sequence">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="first"/>
<sx:delimitedField name="second"/>
<sx:delimitedField name="third"/>
<sx:delimitedField name="fourth"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:inverseRecordMapping id="sequenceToFileMapping">
<sx:onSubtree path="/root/PivotNode">
<sx:flattenSubtree recordType="sequence">
<sx:subtreeFieldMap select="first" field="first"/>
<sx:subtreeFieldMap select="second" field="second"/>
<sx:subtreeFieldMap select="third" field="third"/>
<sx:subtreeFieldMap select="fourth" field="fourth"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -o output/sequence.csv -r resources-sequence.xml sequence-csv
The example below illustrates how to prepare a resources script that will convert an XML document with hexadecimal encoded values to a flat file with special characters.
The input XML document is shown below.
Figure 146. Input XML file transaction.xml
<?xml version="1.0" encoding="utf-8"?>
<transaction>
<R03>
<firstField>03</firstField>
<CLIENUM>1111111</CLIENUM>
</R03>
<R04>
<firstField>04</firstField>
<NAME>John Smith</NAME>
</R04>
</transaction>
The desired output file is a flat file with two adjacent records (no line delimiters):
A record beginning with the hexadecimal value 03, followed by the text "1111111"
A record beginning with the hexadecimal value 04, followed by the text "John Smith"
The following resources script does the transformation.
Figure 147. Resources script resources-specialChar.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="transaction">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="fieldsMapping"/>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
<sx:flatFileWriter >
<sx:flatFile ref="transactionFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="transactionFile" lineDelimited="false">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:binaryField name="firstField" width="1"/>
<sx:when test="firstField='03'">
<sx:flatRecordType id="R03" name="R03">
<sx:binaryField name="firstField" label="firstField" width="1" />
<sx:positionalField name="CLIENUM" label="CLIENUM" width="007" />
</sx:flatRecordType>
</sx:when>
<sx:when test="firstField='04'">
<sx:flatRecordType id="R04" name="R04">
<sx:binaryField name="firstField" label="firstField" width="1" />
<sx:positionalField name="NAME" label="NAME" width="020" />
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:inverseRecordMapping id="fieldsMapping">
<sx:onSubtree path="/transaction/R03">
<sx:flattenSubtree match="/R03" recordType="R03">
<sx:subtreeFieldMap select="firstField" field="firstField"/>
<sx:subtreeFieldMap select="CLIENUM" field="CLIENUM"/>
</sx:flattenSubtree>
</sx:onSubtree>
<sx:onSubtree path="/transaction/R04">
<sx:flattenSubtree match="/R04" recordType="R04">
<sx:subtreeFieldMap select="firstField" field="firstField"/>
<sx:subtreeFieldMap select="NAME" field="NAME"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
Note that the first field of each record is declared as an sx:binaryField. The string rendering of this field is as an hexidecimal value.
You can run this example on the command line by entering
servingxml -r resources-specialChar.xml -i data/transaction.xml -o output/transaction.txt transaction
The example below illustrates how to prepare a resources script that will convert an XML document into a positional flat file and ungroup an XML element.
The input file is an XML file.
Figure 148. Input XML file grouped.xml
<RT_STLMT>
<OPERATING_DATE>03/04/2003</OPERATING_DATE>
<SETTLEMENT_CODE>S14</SETTLEMENT_CODE>
<LINE_ITEMS>
<CHG_TYP>
<CHG_TYP_ID>RT_ADMIN1</CHG_TYP_ID>
<CHG_TYP_NM>Market Administration Amount1</CHG_TYP_NM>
<TOTAL>
<AMT>288.81</AMT>
<INT>
<INT_NUM>1</INT_NUM>
<VAL>54.4</VAL>
</INT>
<INT>
<INT_NUM>2</INT_NUM>
<VAL>31.16</VAL>
</INT>
<INT>
<INT_NUM>3</INT_NUM>
<VAL>81.48</VAL>
</INT>
<INT>
<INT_NUM>4</INT_NUM>
<VAL>121.77</VAL>
</INT>
</TOTAL>
</CHG_TYP>
<CHG_TYP>
<CHG_TYP_ID>RT_ADMIN2</CHG_TYP_ID>
<CHG_TYP_NM>Market Administration Amount2</CHG_TYP_NM>
<TOTAL>
<AMT>288.81</AMT>
<INT>
<INT_NUM>1</INT_NUM>
<VAL>54.4</VAL>
</INT>
<INT>
<INT_NUM>2</INT_NUM>
<VAL>31.16</VAL>
</INT>
<INT>
<INT_NUM>3</INT_NUM>
<VAL>81.48</VAL>
</INT>
<INT>
<INT_NUM>4</INT_NUM>
<VAL>121.77</VAL>
</INT>
</TOTAL>
</CHG_TYP>
</LINE_ITEMS>
</RT_STLMT>
The desired output is shown below.
Figure 149. Output positional flat file ungrouped.csv
OPERATING_DATE,SETTLEMENT_CODE,DETERMINANT_ID,DETERMINANT_NAME,TOTAL_AMT,INT01,INT02,INT03,INT04 03/04/2003,S14,RT_ADMIN1,Market Administration Amount1,288.81,54.4,31.16,81.48,121.77 03/04/2003,S14,RT_ADMIN2,Market Administration Amount2,288.81,54.4,31.16,81.48,121.77
The following resources script does the transformation.
Figure 150. Resources script resources-ungrouping.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core" xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="settlements2csv">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:parameter name="foo">bar</sx:parameter>
<sx:inverseRecordMapping ref="settlementsToFileMapping"/>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="settlementsFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="settlementsFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="settlementType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="settlementType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="settlementType" name="settlementType">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="OPERATING_DATE" label="OPERATING_DATE"/>
<sx:delimitedField name="SETTLEMENT_CODE" label="SETTLEMENT_CODE"/>
<sx:delimitedField name="DETERMINANT_ID" label="DETERMINANT_ID"/>
<sx:delimitedField name="DETERMINANT_NAME" label="DETERMINANT_NAME"/>
<sx:delimitedField name="TOTAL_AMT" label="TOTAL_AMT"/>
<sx:delimitedField name="INT01" label="INT01"/>
<sx:delimitedField name="INT02" label="INT02"/>
<sx:delimitedField name="INT03" label="INT03"/>
<sx:delimitedField name="INT04" label="INT04"/>
</sx:flatRecordType>
<sx:inverseRecordMapping id="settlementsToFileMapping">
<sx:onSubtree path="/RT_STLMT">
<sx:parameter name="OPERATING_DATE" select="OPERATING_DATE"/>
<sx:parameter name="SETTLEMENT_CODE" select="SETTLEMENT_CODE"/>
<sx:parameter name="id" select="@id"/>
<sx:onSubtree path="LINE_ITEMS/CHG_TYP">
<sx:flattenSubtree recordType="settlement">
<sx:subtreeFieldMap select="$OPERATING_DATE" field="OPERATING_DATE"/>
<sx:subtreeFieldMap select="$SETTLEMENT_CODE" field="SETTLEMENT_CODE"/>
<sx:subtreeFieldMap select="'RT'" field="STLMT_TYP"/>
<sx:subtreeFieldMap select="CHG_TYP_ID" field="DETERMINANT_ID"/>
<sx:subtreeFieldMap select="CHG_TYP_NM" field="DETERMINANT_NAME"/>
<sx:subtreeFieldMap select="TOTAL/AMT" field="TOTAL_AMT"/>
<sx:subtreeFieldMap select="TOTAL/INT[1]/VAL" field="INT01"/>
<sx:subtreeFieldMap select="TOTAL/INT[2]/VAL" field="INT02"/>
<sx:subtreeFieldMap select="TOTAL/INT[3]/VAL" field="INT03"/>
<sx:subtreeFieldMap select="TOTAL/INT[4]/VAL" field="INT04"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
Note the use of nested sx:onSubtree elements, and the sx:parameter
elements appearing in the outermost sx:onSubtree. The select
attribute in these parameter declarations can refer to the outer tags in the XML document under "/RT_STLMT", but not the
inner tags handled by the inner sx:onSubtree elements.
An alternative inverse record mapping, which will produce the same output, is shown below.
<sx:inverseRecordMapping id="settlementsToFileMapping">
<sx:onSubtree path="/RT_STLMT">
<sx:parameter name="OPERATING_DATE" select="OPERATING_DATE"/>
<sx:parameter name="SETTLEMENT_CODE" select="SETTLEMENT_CODE"/>
<sx:parameter name="id" select="@id"/>
<sx:flattenSubtree match="LINE_ITEMS/CHG_TYP" recordType="settlement">
<sx:subtreeFieldMap select="$OPERATING_DATE" field="OPERATING_DATE"/>
<sx:subtreeFieldMap select="$SETTLEMENT_CODE" field="SETTLEMENT_CODE"/>
<sx:subtreeFieldMap select="'RT'" field="STLMT_TYP"/>
<sx:subtreeFieldMap select="CHG_TYP_ID" field="DETERMINANT_ID"/>
<sx:subtreeFieldMap select="CHG_TYP_NM" field="DETERMINANT_NAME"/>
<sx:subtreeFieldMap select="TOTAL/AMT" field="TOTAL_AMT"/>
<sx:subtreeFieldMap select="TOTAL/INT[1]/VAL" field="INT01"/>
<sx:subtreeFieldMap select="TOTAL/INT[2]/VAL" field="INT02"/>
<sx:subtreeFieldMap select="TOTAL/INT[3]/VAL" field="INT03"/>
<sx:subtreeFieldMap select="TOTAL/INT[4]/VAL" field="INT04"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
This mapping allows for more flexibility in the sx:parameter declarations, as the
select attribute can now refer to any content in the XML document
under "/RT_STLMT", but at the expense of keeping more of the document in memory.
You can run this example on the command line by entering
servingxml -r resources-ungrouping.xml -i data/grouped.xml
-o output/ungrouped.csv settlements2csv
The example below illustrates how to prepare a resources script that will convert an XML document into a positional flat file with repeating groups.
The input file is an XML file.
Figure 151. Input XML file all.xml
<all>
<field1>val1</field1>
<field2>val2</field2>
<field3>val3</field3>
<compositeA attr1="av11" attr2="av21" attr3="av31"/>
<compositeA attr1="av21" attr2="av22" attr3="av23"/>
<compositeA attr1="av31" attr2="av32" attr3="av33"/>
<compositeB attrx="avx" attry="avy"/>
</all>
The desired output is shown below.
Figure 152. Output positional flat file all.txt
val1 val2 val3 av11 av21 av31 av21 av22 av23 av31 av32 av33 avx avy
The following resources script does the transformation.
Figure 153. Resources script resources-all.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="all2flat">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="allToFileMapping"/>
<sx:transform>
<sx:document/>
</sx:transform>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="allFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="allFile">
<sx:flatFileBody>
<sx:flatRecordType ref="allType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="allType" name="allType">
<sx:positionalField name="f1" label="F1" width="10"/>
<sx:positionalField name="f2" label="F2" width="10"/>
<sx:positionalField name="f3" label="F3" width="10"/>
<sx:repeatingGroup name="compA">
<sx:flatRecordType name="compARecord">
<sx:positionalField name="ca1" width="5"/>
<sx:positionalField name="ca2" width="5"/>
<sx:positionalField name="ca3" width="5"/>
</sx:flatRecordType>
</sx:repeatingGroup>
<sx:repeatingGroup name="compB">
<sx:flatRecordType name="compBRecord">
<sx:positionalField name="cb1" width="5"/>
<sx:positionalField name="cb2" width="5"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
<sx:inverseRecordMapping id="allToFileMapping">
<sx:onSubtree path="/all">
<sx:flattenSubtree recordType="allType">
<sx:subtreeFieldMap select="field1" field="f1"/>
<sx:subtreeFieldMap select="field2" field="f2"/>
<sx:subtreeFieldMap select="field3" field="f3"/>
<sx:subtreeFieldMap match="compositeA" field="compA">
<sx:flattenSubtree recordType="compARecord">
<sx:subtreeFieldMap select="@attr1" field="ca1"/>
<sx:subtreeFieldMap select="@attr2" field="ca2"/>
<sx:subtreeFieldMap select="@attr3" field="ca3"/>
</sx:flattenSubtree>
</sx:subtreeFieldMap>
<sx:subtreeFieldMap match="compositeB" field="compB">
<sx:flattenSubtree recordType="compBRecord">
<sx:subtreeFieldMap select="@attrx" field="cb1"/>
<sx:subtreeFieldMap select="@attry" field="cb2"/>
</sx:flattenSubtree>
</sx:subtreeFieldMap>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-all.xml -i data/all.xml
-o output/all.txt all2flat
The example below illustrates how to prepare a resources script that will convert an XML document into a positional flat file with different record types corresponding to different XML subtrees.
The input file is an XML file.
Figure 154. Input XML file blocks.xml
<BLOCKS>
<BLOCK1>
<FIELD1>XXXX</FIELD1>
<FIELD2>YYYY</FIELD2>
<FIELD3>ZZZZ</FIELD3>
<FIELD4>WWWW</FIELD4>
</BLOCK1>
<BLOCK2>
<TEXT1>AAAA</TEXT1>
<TEXT2>BBBB</TEXT2>
<TEXT3>CCCC</TEXT3>
<TEXT4>DDDD</TEXT4>
</BLOCK2>
<BLOCK3>
<INTEGER1>1111</INTEGER1>
<INTEGER2>2222</INTEGER2>
<INTEGER3>3333</INTEGER3>
</BLOCK3>
</BLOCKS>
The desired output is shown below.
The following resources script does the transformation.
Figure 156. Resources script resources-blocks.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="blocks">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:transform>
<sx:document/>
</sx:transform>
<sx:inverseRecordMapping ref="blocksToFileMapping"/>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile>
<sx:flatFileBody>
<sx:flatRecordType ref="blockType"/>
</sx:flatFileBody>
</sx:flatFile>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatRecordTypeChoice id="blockType" name="blockType">
<sx:positionalField name="dummy" width="1"/>
<sx:when test="/Block1Record">
<sx:flatRecordType name="BLOCK1">
<sx:positionalField name="Field1" width="4"/>
<sx:positionalField name="Field2" width="4"/>
<sx:positionalField name="Field3" width="4"/>
<sx:positionalField name="Field4" width="4"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="/Block2Record">
<sx:flatRecordType name="BLOCK2">
<sx:positionalField name="Text1" width="4"/>
<sx:positionalField name="Text2" width="4"/>
<sx:positionalField name="Text3" width="4"/>
<sx:positionalField name="Text4" width="4"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="/Block3Record">
<sx:flatRecordType name="BLOCK3">
<sx:positionalField name="Integer1" width="4"/>
<sx:positionalField name="Integer2" width="4"/>
<sx:positionalField name="Integer3" width="4"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
<sx:inverseRecordMapping id="blocksToFileMapping">
<sx:onSubtree path="/BLOCKS">
<sx:onSubtree path="BLOCK1">
<sx:flattenSubtree recordType="Block1Record">
<sx:subtreeFieldMap select="FIELD1" field="Field1"/>
<sx:subtreeFieldMap select="FIELD2" field="Field2"/>
<sx:subtreeFieldMap select="FIELD3" field="Field3"/>
<sx:subtreeFieldMap select="FIELD4" field="Field4"/>
</sx:flattenSubtree>
</sx:onSubtree>
<sx:onSubtree path="BLOCK2">
<sx:flattenSubtree recordType="Block2Record">
<sx:subtreeFieldMap select="TEXT1" field="Text1"/>
<sx:subtreeFieldMap select="TEXT2" field="Text2"/>
<sx:subtreeFieldMap select="TEXT3" field="Text3"/>
<sx:subtreeFieldMap select="TEXT4" field="Text4"/>
</sx:flattenSubtree>
</sx:onSubtree>
<sx:onSubtree path="BLOCK3">
<sx:flattenSubtree recordType="Block3Record">
<sx:subtreeFieldMap select="INTEGER1" field="Integer1"/>
<sx:subtreeFieldMap select="INTEGER2" field="Integer2"/>
<sx:subtreeFieldMap select="INTEGER3" field="Integer3"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-blocks.xml -i data/blocks.xml
-o output/blocks.txt blocks
This example illustrates a resources script that will map subsequences of blocks in the XML to a single line in the flat file.
The input file is an XML file.
Figure 157. Input XML file block_subsequences.xml
<BLOCKS>
<BLOCK1>
<FIELD1>XXXX1</FIELD1>
<FIELD2>YYYY1</FIELD2>
<FIELD3>ZZZZ1</FIELD3>
<FIELD4>WWWW1</FIELD4>
</BLOCK1>
<BLOCK2>
<TEXT1>AAAA1</TEXT1>
<TEXT2>BBBB1</TEXT2>
<TEXT3>CCCC1</TEXT3>
<TEXT4>DDDD1</TEXT4>
</BLOCK2>
<BLOCK1>
<FIELD1>XXXX2</FIELD1>
<FIELD2>YYYY2</FIELD2>
<FIELD3>ZZZZ2</FIELD3>
<FIELD4>WWWW2</FIELD4>
</BLOCK1>
<BLOCK2>
<TEXT1>AAAA2</TEXT1>
<TEXT2>BBBB2</TEXT2>
<TEXT3>CCCC2</TEXT3>
<TEXT4>DDDD2</TEXT4>
</BLOCK2>
</BLOCKS>
Note that BLOCK1 and BLOCK2 repeat.
The desired output is shown below.
Figure 158. Output positional flat file block_subsequences.txt
XXXX1YYYY1ZZZZ1WWWW1AAAA1BBBB1CCCC1DDDD1 XXXX2YYYY2ZZZZ2WWWW2AAAA2BBBB2CCCC2DDDD2
Note that BLOCK1 and BLOCK2 form a single line in the Flat output file, and then BLOCK1 and BLOCK2 repeat.
The following resources script does the transformation.
Figure 159. Resources script resources-subsequences.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="blocks">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:transform>
<sx:document/>
</sx:transform>
<sx:inverseRecordMapping ref="blocksToFileMapping"/>
</sx:subtreeRecordReader>
<sx:combineRecords recordType="BLOCKS" repeatingGroup="blocks"
beginTest="sx:current/BLOCK1"
endTest="sx:current/BLOCK1">
</sx:combineRecords>
<sx:flatFileWriter>
<sx:flatFile>
<sx:flatFileBody>
<sx:flatRecordType ref="blockType"/>
</sx:flatFileBody>
</sx:flatFile>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatRecordType id="blockType" name="blockType">
<sx:repeatingGroup name="blocks">
<sx:flatRecordTypeChoice>
<sx:when test="/BLOCK1">
<sx:flatRecordType name="BLOCK1">
<sx:positionalField name="Field1" width="5"/>
<sx:positionalField name="Field2" width="5"/>
<sx:positionalField name="Field3" width="5"/>
<sx:positionalField name="Field4" width="5"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="/BLOCK2">
<sx:flatRecordType name="BLOCK2">
<sx:positionalField name="Text1" width="5"/>
<sx:positionalField name="Text2" width="5"/>
<sx:positionalField name="Text3" width="5"/>
<sx:positionalField name="Text4" width="5"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:repeatingGroup>
</sx:flatRecordType>
<sx:inverseRecordMapping id="blocksToFileMapping">
<sx:onSubtree path="/BLOCKS">
<sx:onSubtree path="BLOCK1">
<sx:flattenSubtree recordType="BLOCK1">
<sx:subtreeFieldMap select="FIELD1" field="Field1"/>
<sx:subtreeFieldMap select="FIELD2" field="Field2"/>
<sx:subtreeFieldMap select="FIELD3" field="Field3"/>
<sx:subtreeFieldMap select="FIELD4" field="Field4"/>
</sx:flattenSubtree>
</sx:onSubtree>
<sx:onSubtree path="BLOCK2">
<sx:flattenSubtree recordType="BLOCK2">
<sx:subtreeFieldMap select="TEXT1" field="Text1"/>
<sx:subtreeFieldMap select="TEXT2" field="Text2"/>
<sx:subtreeFieldMap select="TEXT3" field="Text3"/>
<sx:subtreeFieldMap select="TEXT4" field="Text4"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
The strategy is to map the BLOCK1 and BLOCK2 subtrees to BLOCK1 and BLOCK2 records, compose the BLOCK1 and BLOCK2 records into a composite record, and to map the composite record to a single line in the flat file.
You can run this example on the command line by entering
servingxml -r resources-subsequences.xml -i data/block_subsequences.xml
-o output/block_subsequences.txt blocks
The example below illustrates how to prepare a resources script that will convert an XML document into a delimited flat file with different record types corresponding to different XML subtrees.
The input file is an XML file.
Figure 160. Input XML file book_hierarchy.xml
<myns:rootNode xmlns:myns="http://mycompany.com/mynames/">
<myns:name>root</myns:name>
<attribute>
<myns:name>NONE</myns:name>
<myns:value>NONE</myns:value>
</attribute>
<myns:childNode>
<myns:name>Entity.LONDON</myns:name>
<attribute>
<myns:name>Entity</myns:name>
<myns:value>LONDON</myns:value>
</attribute>
<myns:childNode>
<myns:name>BookName.EQUITIES_NY</myns:name>
<attribute>
<myns:name>BookName</myns:name>
<myns:value>EQUITIES_NY</myns:value>
</attribute>
<myns:childNode>
<myns:name>BookName.EQUITIES_NY_TR_JAYME</myns:name>
<attribute>
<myns:name>BookName</myns:name>
<myns:value>EQUITIES_NY_TR_JAYME</myns:value>
</attribute>
<myns:childNode>
<myns:name>BookName.PR_EQUITY_NY</myns:name>
<attribute>
<myns:name>BookName</myns:name>
<myns:value>PR_EQUITY_NY</myns:value>
</attribute>
</myns:childNode>
<myns:childNode>
<myns:name>BookName.PR_EQUITY_NY1</myns:name>
<attribute>
<myns:name>BookName</myns:name>
<myns:value>PR_EQUITY_NY1</myns:value>
</attribute>
</myns:childNode>
</myns:childNode>
</myns:childNode>
<myns:childNode>
<myns:name>BookName.EQUITIES_LONDON</myns:name>
<attribute>
<myns:name>BookName</myns:name>
<myns:value>EQUITIES_LONDON</myns:value>
</attribute>
<myns:childNode>
<myns:name>BookName.EQUITIES_NY_TR_JAYME</myns:name>
<attribute>
<myns:name>BookName</myns:name>
<myns:value>EQUITIES_NY_TR_JAYME</myns:value>
</attribute>
<myns:childNode>
<myns:name>BookName.PR_EQUITY_NY</myns:name>
<attribute>
<myns:name>BookName</myns:name>
<myns:value>PR_EQUITY_NY</myns:value>
</attribute>
</myns:childNode>
</myns:childNode>
</myns:childNode>
</myns:childNode>
</myns:rootNode>
The desired output is shown below.
Figure 161. Output delimited flat file book_hierarchy.txt
root,NONE,,Entity.LONDON,Entity,LONDON,,BookName,EQUITIES_NY,BookName,EQUITIES_NY_TR_JAYME,BookName,PR_EQUITY_NY,BookName,PR_EQUITY_NY1 root,NONE,,Entity.LONDON,Entity,LONDON,,BookName,EQUITIES_LONDON,BookName,EQUITIES_NY_TR_JAYME,BookName,PR_EQUITY_NY
The following resources script does the transformation.
Figure 162. Resources script resources-bookHierarchy.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="bookHierarchyToFile">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:transform>
<sx:document/>
</sx:transform>
<sx:inverseRecordMapping ref="bookHierarchyToFileMapping"/>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile>
<sx:flatFileBody>
<sx:flatRecordType ref="settlementType"/>
</sx:flatFileBody>
</sx:flatFile>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatRecordType id="settlementType" name="settlementType">
<sx:fieldDelimiter value=","/>
<sx:repeatDelimiter value=","/>
<sx:segmentDelimiter value="|"/>
<sx:delimitedField name="root" label="root"/>
<sx:delimitedField name="bookCode" label="bookCode"/>
<sx:delimitedField name="bookValue" label="bookValue"/>
<sx:delimitedField name="level1" label="level1"/>
<sx:delimitedField name="level1Code" label="level1Code"/>
<sx:delimitedField name="level1Value" label="level1Value"/>
<sx:delimitedField name="level2" label="level2"/>
<sx:delimitedField name="level2Code" label="level2Code"/>
<sx:delimitedField name="level2Value" label="level2Value"/>
<sx:repeatingGroup name="level3" label="level3">
<sx:flatRecordType name="level3Record">
<sx:delimitedField name="code" label="code"/>
<sx:delimitedField name="value" label="value"/>
<sx:repeatingGroup name="level4" label="level4">
<sx:flatRecordType name="level4Record">
<sx:delimitedField name="code" label="code"/>
<sx:delimitedField name="value" label="value"/>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
</sx:repeatingGroup>
</sx:flatRecordType>
<sx:inverseRecordMapping id="bookHierarchyToFileMapping">
<sx:onSubtree path="/myns:rootNode">
<sx:parameter name="root" select="myns:name"/>
<sx:parameter name="bookCode" select="attribute/myns:name"/>
<sx:parameter name="bookValue" select="attribute/value"/>
<sx:onSubtree path="/myns:rootNode/myns:childNode">
<sx:parameter name="level1" select="myns:name"/>
<sx:parameter name="level1Code" select="attribute/myns:name"/>
<sx:parameter name="level1Value" select="attribute/myns:value"/>
<sx:onSubtree path="/myns:childNode/myns:childNode">
<sx:flattenSubtree recordType="settlement">
<sx:subtreeFieldMap select="$root" field="root"/>
<sx:subtreeFieldMap select="$bookCode" field="bookCode"/>
<sx:subtreeFieldMap select="$bookValue" field="bookValue"/>
<sx:subtreeFieldMap select="$level1" field="level1"/>
<sx:subtreeFieldMap select="$level1Code" field="level1Code"/>
<sx:subtreeFieldMap select="$level1Value" field="level1Value"/>
<sx:subtreeFieldMap select="name" field="level2"/>
<sx:subtreeFieldMap select="attribute/myns:name" field="level2Code"/>
<sx:subtreeFieldMap select="attribute/myns:value" field="level2Value"/>
<sx:subtreeFieldMap match="myns:childNode" field="level3">
<sx:flattenSubtree recordType="level3Record">
<sx:subtreeFieldMap select="attribute/myns:name" field="code"/>
<sx:subtreeFieldMap select="attribute/myns:value" field="value"/>
<sx:subtreeFieldMap match="myns:childNode" field="level4">
<sx:flattenSubtree recordType="level4Record">
<sx:subtreeFieldMap select="attribute/myns:name" field="code"/>
<sx:subtreeFieldMap select="attribute/myns:value" field="value"/>
</sx:flattenSubtree>
</sx:subtreeFieldMap>
</sx:flattenSubtree>
</sx:subtreeFieldMap>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:onSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-bookHierarchy.xml -i data/book_hierarchy.xml
-o output/book_hierarchy.txt bookHierarchyToFile
The example below illustrates how to prepare a resources script that will convert an XML representation of an X12 file into a positional flat file. Different record types correspond to different subtrees in the XML document.
The input file is an XML file.
Figure 163. Input file Edi517M2xml.xml
<?xml version="1.0" encoding="utf-8"?>
<X12>
<!-- Reapting group with in the x12 document -->
<ST>
<ST01>517</ST01>
<ST02>0006</ST02>
</ST>
<BR>
<BR01>00</BR01>
<BR02>AN</BR02>
<BR03>20071019</BR03>
<BR04/>
<BR05/>
<BR06/>
<BR07>X7</BR07>
<BR08>0101</BR08>
<BR09>150401</BR09>
<BR10>XM</BR10>
<BR11>003</BR11>
<BR12>0101003</BR12>
</BR>
<G62>
<G6201>CA</G6201>
<G6202>20070720</G6202>
</G62>
<G62>
<G6201>64</G6201>
<G6202>20070903</G6202>
</G62>
<LM>
<LM01>DF</LM01>
</LM>
<LQ>
<LQ01>0</LQ01>
<LQ02>AN9</LQ02>
</LQ>
<N1>
<N101>Z4</N101>
<N102/>
<N103>M4</N103>
<N104>GSA</N104>
<N105/>
<N106>FR</N106>
</N1>
<N1>
<N101>ZR</N101>
<N102/>
<N103>10</N103>
<N104>N23194</N104>
<N105/>
<N106>TO</N106>
</N1>
<!-- Repating group with in ST, nmber of times based on value in BR11( current value =3) -->
<QTY>
<QTY01>63</QTY01>
<QTY02>4</QTY02>
<QTY03>BX</QTY03>
<QTY04>BX4</QTY04>
</QTY>
<N9>
<N901>TN</N901>
<N902>N2319471430196</N902>
</N9>
<N9>
<N901>NS</N901>
<N902>7920006199162</N902>
</N9>
<G62>
<G6201>17</G6201>
<G6202>20070718</G6202>
</G62>
<LM>
<LM01>DF</LM01>
</LM>
<LQ>
<LQ01>0</LQ01>
<LQ02>AN1</LQ02>
</LQ>
<LQ>
<LQ01>81</LQ01>
<LQ02>BV</LQ02>
</LQ>
<LQ>
<LQ01>COG</LQ01>
<LQ02>9Q</LQ02>
</LQ>
<LQ>
<LQ01>DE</LQ01>
<LQ02>C</LQ02>
</LQ>
<LQ>
<LQ01>DF</LQ01>
<LQ02>S</LQ02>
</LQ>
<LQ>
<LQ01>A9</LQ01>
<LQ02>N23194</LQ02>
</LQ>
<LQ>
<LQ01>78</LQ01>
<LQ02>ZQ0</LQ02>
</LQ>
<LQ>
<LQ01>79</LQ01>
<LQ02>03</LQ02>
</LQ>
<FA1>
<FA101>DN</FA101>
<FA102>D340</FA102>
</FA1>
<FA2>
<FA201>B5</FA201>
<FA202>RC</FA202>
</FA2>
<QTY>
<QTY01>63</QTY01>
<QTY02>1</QTY02>
<QTY03>EA</QTY03>
<QTY04>EA1</QTY04>
</QTY>
<N9>
<N901>TN</N901>
<N902>N2319471430158</N902>
</N9>
<N9>
<N901>NS</N901>
<N902>5140003694927</N902>
</N9>
<G62>
<G6201>17</G6201>
<G6202>20070725</G6202>
</G62>
<LM>
<LM01>DF</LM01>
</LM>
<LQ>
<LQ01>0</LQ01>
<LQ02>AN1</LQ02>
</LQ>
<LQ>
<LQ01>81</LQ01>
<LQ02>BB</LQ02>
</LQ>
<LQ>
<LQ01>COG</LQ01>
<LQ02>9Q</LQ02>
</LQ>
<LQ>
<LQ01>DE</LQ01>
<LQ02>C</LQ02>
</LQ>
<LQ>
<LQ01>DF</LQ01>
<LQ02>S</LQ02>
</LQ>
<LQ>
<LQ01>A9</LQ01>
<LQ02>N23194</LQ02>
</LQ>
<LQ>
<LQ01>78</LQ01>
<LQ02>ZQ0</LQ02>
</LQ>
<LQ>
<LQ01>79</LQ01>
<LQ02>03</LQ02>
</LQ>
<FA1>
<FA101>DN</FA101>
<FA102>D340</FA102>
</FA1>
<FA2>
<FA201>B5</FA201>
<FA202>RC</FA202>
</FA2>
<QTY>
<QTY01>63</QTY01>
<QTY02>6</QTY02>
<QTY03>BG</QTY03>
<QTY04>BG6</QTY04>
</QTY>
<N9>
<N901>TN</N901>
<N902>N2319471430272</N902>
</N9>
<N9>
<N901>NS</N901>
<N902>5610009851800</N902>
</N9>
<G62>
<G6201>17</G6201>
<G6202>20070703</G6202>
</G62>
<LM>
<LM01>DF</LM01>
</LM>
<LQ>
<LQ01>0</LQ01>
<LQ02>AN1</LQ02>
</LQ>
<LQ>
<LQ01>81</LQ01>
<LQ02>BV</LQ02>
</LQ>
<LQ>
<LQ01>COG</LQ01>
<LQ02>9Q</LQ02>
</LQ>
<LQ>
<LQ01>DE</LQ01>
<LQ02>C</LQ02>
</LQ>
<LQ>
<LQ01>DF</LQ01>
<LQ02>S</LQ02>
</LQ>
<LQ>
<LQ01>A9</LQ01>
<LQ02>N23194</LQ02>
</LQ>
<LQ>
<LQ01>78</LQ01>
<LQ02>ZQ0</LQ02>
</LQ>
<LQ>
<LQ01>79</LQ01>
<LQ02>03</LQ02>
</LQ>
<FA1>
<FA101>DN</FA101>
<FA102>D340</FA102>
</FA1>
<FA2>
<FA201>B5</FA201>
<FA202>RC</FA202>
</FA2>
<SE>
<SE01>54</SE01>
<SE02>0006</SE02>
</SE>
</X12>
The desired output is shown below.
Figure 164. Output delimited flat file Edi517M.txt
AN9GSA0101003 N23194 AN1 7920006199162 CBX4N2319471430196N23194SRC9QZQ003 BV AN1 5140003694927 CEA1N2319471430158N23194SRC9QZQ003 BB AN1 5610009851800 CBG6N2319471430272N23194SRC9QZQ003 BV
The following resources script does the transformation.
Figure 165. Resources script resources-xml-517M.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="xmltodlss">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:transform>
<sx:document/>
<sx:xslt ref="dlss-layout"/>
</sx:transform>
<sx:inverseRecordMapping ref="dlssToFileMapping"/>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="dlssFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:xslt id="dlss-layout">
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:strip-space elements="*"/>
<xsl:template match="/X12">
<xsl:copy>
<xsl:apply-templates/>
</xsl:copy>
</xsl:template>
<xsl:template match="ST">
<PivotNode1>
<ST>
<xsl:value-of select="."/>
</ST>
<BR>
<xsl:value-of select="following-sibling::BR/BR08"/>
</BR>
<BR01>
<xsl:value-of select="following-sibling::BR/BR11"/>
</BR01>
<G62Sub1>
<xsl:value-of select="following-sibling::G62[1]/G6203"/>
</G62Sub1>
<G62Sub2>
<xsl:value-of select="following-sibling::G62[1]/G6204"/>
</G62Sub2>
<G62Sub3>
<xsl:value-of select="following-sibling::G62[2]/G6203"/>
</G62Sub3>
<G62Sub4>
<xsl:value-of select="following-sibling::G62[2]/G6204"/>
</G62Sub4>
<LMSub1>
<xsl:value-of select="following-sibling::LM[1]/LM01"/>
</LMSub1>
<LQSub1>
<xsl:value-of select="following-sibling::LQ[1]/LQ02"/>
</LQSub1>
<N1Sub1>
<xsl:value-of select="following-sibling::N1[1]/N104"/>
</N1Sub1>
<N1Sub2>
<xsl:value-of select="following-sibling::N1[2]/N104"/>
</N1Sub2>
<SE>
<xsl:value-of select="following-sibling::SE/SE01"/>
</SE>
<xsl:apply-templates />
</PivotNode1>
</xsl:template>
<xsl:template match="QTY">
<PivotNode>
<QTY>
<xsl:value-of select="."/>
</QTY>
<QTY01>
<xsl:value-of select="QTY04"/>
</QTY01>
<N104Sub>
<xsl:value-of select="QTY05"/>
</N104Sub>
<N9Sub1>
<xsl:value-of select="following-sibling::N9[1]/N902"/>
</N9Sub1>
<N9Sub2>
<xsl:value-of select="following-sibling::N9[2]/N902"/>
</N9Sub2>
<G62Sub5>
<xsl:value-of select="following-sibling::G62/G6203"/>
</G62Sub5>
<G62Sub6>
<xsl:value-of select="following-sibling::G62/G6204"/>
</G62Sub6>
<G62Sub7>
<xsl:value-of select="following-sibling::G62/G6205"/>
</G62Sub7>
<G62Sub8>
<xsl:value-of select="following-sibling::G62/G6206"/>
</G62Sub8>
<LMSub2>
<xsl:value-of select="following-sibling::LM[1]/LM01"/>
</LMSub2>
<LQSub2>
<xsl:value-of select="following-sibling::LQ[1]/LQ02"/>
</LQSub2>
<LQSub3>
<xsl:value-of select="following-sibling::LQ[2]/LQ02"/>
</LQSub3>
<LQSub4>
<xsl:value-of select="following-sibling::LQ[3]/LQ02"/>
</LQSub4>
<LQSub5>
<xsl:value-of select="following-sibling::LQ[4]/LQ02"/>
</LQSub5>
<LQSub6>
<xsl:value-of select="following-sibling::LQ[5]/LQ02"/>
</LQSub6>
<LQSub7>
<xsl:value-of select="following-sibling::LQ[6]/LQ02"/>
</LQSub7>
<LQSub8>
<xsl:value-of select="following-sibling::LQ[7]/LQ02"/>
</LQSub8>
<LQSub9>
<xsl:value-of select="following-sibling::LQ[8]/LQ02"/>
</LQSub9>
<FA1>
<xsl:value-of select="following-sibling::FA1/FA102"/>
</FA1>
<FA2>
<xsl:value-of select="following-sibling::FA2/FA202"/>
</FA2>
</PivotNode>
</xsl:template>
</xsl:stylesheet>
</sx:xslt>
<sx:flatFile id="dlssFile">
<sx:flatFileBody>
<sx:flatRecordTypeChoice>
<sx:positionalField name="placeholder" width="1"/>
<sx:when test="/dlss1">
<sx:flatRecordType name="dlss1">
<sx:positionalField name="LQSub1" width="3"/>
<sx:positionalField name="N1Sub1" width="3"/>
<sx:positionalField name="BR" width="4"/>
<sx:positionalField name="BR01" width="3"/>
<sx:positionalField name="spaces1" width="15"/>
<sx:positionalField name="N1Sub2" width="6"/>
<sx:positionalField name="G62Sub1" width="2"/>
<sx:positionalField name="G62Sub2" width="3"/>
<sx:positionalField name="G62Sub3" width="4"/>
<sx:positionalField name="G62Sub4" width="2"/>
</sx:flatRecordType>
</sx:when>
<sx:when test="/dlss2">
<sx:flatRecordType name="dlss2">
<sx:positionalField name="LQSub2" width="3"/>
<sx:positionalField name="N104Sub" width="2"/>
<sx:positionalField name="N9Sub2" width="14"/>
<sx:positionalField name="LQSub5" width="1"/>
<sx:positionalField name="QTY01" width="3"/>
<sx:positionalField name="N9Sub1" width="14"/>
<sx:positionalField name="LQSub7" width="6"/>
<sx:positionalField name="LQSub6" width="1"/>
<sx:positionalField name="FA2" width="2"/>
<sx:positionalField name="LQSub4" width="2"/>
<sx:positionalField name="LQSub8" width="3"/>
<sx:positionalField name="LQSub9" width="2"/>
<sx:positionalField name="G62Sub6" width="2"/>
<sx:positionalField name="LQSub3" width="3"/>
<sx:positionalField name="G62Sub7" width="3"/>
<sx:positionalField name="G62Sub8" width="3"/>
</sx:flatRecordType>
</sx:when>
</sx:flatRecordTypeChoice>
</sx:flatFileBody>
</sx:flatFile>
<sx:inverseRecordMapping id="dlssToFileMapping">
<sx:onSubtree path="/X12/PivotNode1">
<sx:flattenSubtree recordType="dlss1">
<sx:subtreeFieldMap select="LQSub1" field="LQSub1"/>
<sx:subtreeFieldMap select="N1Sub1" field="N1Sub1"/>
<sx:subtreeFieldMap select="BR" field="BR"/>
<sx:subtreeFieldMap select="BR01" field="BR01"/>
<sx:subtreeFieldMap select="N1Sub2" field="N1Sub2"/>
<sx:subtreeFieldMap select="G62Sub1" field="G62Sub1"/>
<sx:subtreeFieldMap select="G62Sub2" field="G62Sub2"/>
<sx:subtreeFieldMap select="G62Sub3" field="G62Sub3"/>
<sx:subtreeFieldMap select="G62Sub4" field="G62Sub4"/>
</sx:flattenSubtree>
</sx:onSubtree>
<sx:onSubtree path="/X12/PivotNode">
<sx:flattenSubtree recordType="dlss2">
<sx:subtreeFieldMap select="LQSub2" field="LQSub2"/>
<sx:subtreeFieldMap select="N104Sub" field="N104Sub"/>
<sx:subtreeFieldMap select="N9Sub2" field="N9Sub2"/>
<sx:subtreeFieldMap select="LQSub5" field="LQSub5"/>
<sx:subtreeFieldMap select="QTY01" field="QTY01"/>
<sx:subtreeFieldMap select="N9Sub1" field="N9Sub1"/>
<sx:subtreeFieldMap select="LQSub7" field="LQSub7"/>
<sx:subtreeFieldMap select="LQSub6" field="LQSub6"/>
<sx:subtreeFieldMap select="FA2" field="FA2"/>
<sx:subtreeFieldMap select="LQSub4" field="LQSub4"/>
<sx:subtreeFieldMap select="LQSub8" field="LQSub8"/>
<sx:subtreeFieldMap select="LQSub9" field="LQSub9"/>
<sx:subtreeFieldMap select="G62Sub6" field="G62Sub6"/>
<sx:subtreeFieldMap select="LQSub3" field="LQSub3"/>
<sx:subtreeFieldMap select="G62Sub7" field="G62Sub7"/>
<sx:subtreeFieldMap select="G62Sub8" field="G62Sub8"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -i Edi517M2xml.xml -r resources-xml-517M.xml -o Edi517M.txt xmltodlss
This example shows a resources script that will read a number of XML documents in a directory and transform them into a single CSV file. It illustrates the use of the sx:documentSequence element.
The input is a directory of XML documents.
countries-1.xml countries-2.xml countries-3.xml countries-4.xml countries-5.xml
The desired output is one CSV file containing all of the country entries.
country_code,country_name ABW,ARUBA ADH,UNITED ARAB EMIRATES ... ZWE,ZIMBABWE
The resources script resources-countrySequence.xml
does the transformation.
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="countries-to-csv">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="countriesToFileMapping"/>
<sx:transform>
<sx:documentSequence wrapWith="result">
<sx:recordStream>
<sx:directoryReader directory="data">
<sx:fileFilter pattern="countries.*[.]xml"/>
</sx:directoryReader>
</sx:recordStream>
<sx:transform>
<sx:document>
<sx:fileSource directory="{parentDirectory}" file="{name}"/>
</sx:document>
<msv:schemaValidator>
<sx:urlSource url="data/countries.xsd"/>
</msv:schemaValidator>
</sx:transform>
</sx:documentSequence>
</sx:transform>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="countriesFile">
<sx:commentStarter value="#"/>
<sx:flatFileHeader>
<sx:flatRecordType ref="countryType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="countryType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType name="country" id="countryType">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="code" label="country_code"/>
<sx:delimitedField name="name" label="country_name"/>
</sx:flatRecordType>
<sx:inverseRecordMapping id="countriesToFileMapping">
<sx:onSubtree path="/result/countries/country">
<sx:flattenSubtree recordType="country">
<sx:subtreeFieldMap select="countryName" field="name"/>
<sx:subtreeFieldMap select="@countryCode" field="code"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
The effect of the sx:documentSequence
instruction is to read all the documents in the data
directory whose file names match the regular expression "countries.*[.]xml".
The content of these documents is wrapped in the root element
result.
You can run this example on the command line by entering
servingxml -r resources-countrySequence.xml -o output/countries.csv countries-to-csv
This example shows a resources script that will read a number of XML documents in a directory and transform them into a single CSV file. It illustrates the use of the sx:documentSequence and saxon:xquery, elements.
The input is a directory of XML documents.
eobclaims1.xml eobclaims2.xml eobclaimshdrtlr.xml
The desired output is one CSV file containing all of the country entries.
ADM2301EOBS GENERATED ON 06/1 3/08 HDR06/13/08200815501300101MEMBER01 BISHOP ADR1234 TEST DR ARVADA HDR06/13/08200815501300102MEMBER02 BISHOP ADR5678 TEST DR ARVADA TLRTOTAL EOBS GENERATED: 0000001000000000000010000018
The resources script resources-eobclaims.xml
does the transformation. Note that this script includes the shared eobclaims record structure declarations.
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:saxon="http://www.servingxml.com/extensions/saxon">
<sx:include href="resources-eobclaims-flatfile.xml"/>
<sx:service id="eobclaims-to-flat">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:inverseRecordMapping ref="eobclaims-to-flat-mapping"/>
<sx:transform>
<sx:documentSequence wrapWith="result">
<sx:content ref="eobclaims-document-header"/>
<sx:content ref="eobclaims-document-body"/>
<sx:content ref="eobclaims-document-trailer"/>
</sx:documentSequence>
</sx:transform>
</sx:subtreeRecordReader>
<sx:flatFileWriter>
<sx:flatFile ref="eobclaims-file"/>
<sx:fileSink file="output/eobclaims.txt"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<saxon:xquery id="eobclaims-document-header">
<sx:preserveMarkup>
<header>
{doc('data/eobclaimshdrtlr.xml')/headertrailer/ADMIN_RECTYPE}
</header>
</sx:preserveMarkup>
</saxon:xquery>
<saxon:xquery id="eobclaims-document-trailer">
<sx:preserveMarkup>
<trailer>
{doc('data/eobclaimshdrtlr.xml')/headertrailer/TLR-REC-TYPE}
</trailer>
</sx:preserveMarkup>
</saxon:xquery>
<sx:documentSequence id="eobclaims-document-body">
<sx:directoryReader directory="data">
<sx:fileFilter pattern=".*[1-9].xml"/>
</sx:directoryReader>
<sx:document>
<sx:fileSource directory="{parentDirectory}" file="{name}"/>
</sx:document>
</sx:documentSequence>
<sx:inverseRecordMapping id="eobclaims-to-flat-mapping">
<sx:onSubtree path="header/ADMIN_RECTYPE">
<sx:flattenSubtree recordType="ADM">
<sx:subtreeFieldMap select="@record_type" field="record_type"/>
<sx:subtreeFieldMap select="ADM_0001_TEXT" field="ADM_0001_TEXT"/>
<sx:subtreeFieldMap select="ADM_0001_RUN_DATE" field="ADM_0001_RUN_DATE"/>
<sx:subtreeFieldMap select="FILLER" field="FILLER"/>
</sx:flattenSubtree>
</sx:onSubtree>
<sx:onSubtree path="eob/HEADER_RECTYPE">
<sx:flattenSubtree recordType="HDR">
<sx:subtreeFieldMap select="@record_type" field="record_type"/>
<sx:subtreeFieldMap select="HDR_STMT_DATE" field="HDR_STMT_DATE"/>
<sx:subtreeFieldMap select="HDR_CLAIM_NUMBER" field="HDR_CLAIM_NUMBER"/>
<sx:subtreeFieldMap select="HDR_PATIENT_NAME_FIRST" field="HDR_PATIENT_NAME_FIRST"/>
<sx:subtreeFieldMap select="HDR_PATIENT_NAME_LAST" field="HDR_PATIENT_NAME_LAST"/>
</sx:flattenSubtree>
</sx:onSubtree>
<sx:onSubtree path="eob/ADR_REC_TYPE">
<sx:flattenSubtree recordType="HDR">
<sx:subtreeFieldMap select="@record_type" field="record_type"/>
<sx:subtreeFieldMap select="ADR_8512_ATTN" field="ADR_8512_ATTN"/>
<sx:subtreeFieldMap select="HDR_CLAIM_NUMBER" field="HDR_CLAIM_NUMBER"/>
<sx:subtreeFieldMap select="ADR_8514_CITY" field="ADR_8514_CITY"/>
</sx:flattenSubtree>
</sx:onSubtree>
<sx:onSubtree path="trailer/TLR-REC-TYPE">
<sx:flattenSubtree recordType="TLR">
<sx:subtreeFieldMap select="@record_type" field="record_type"/>
<sx:subtreeFieldMap select="TLR_0001_TEXT" field="TLR_0001_TEXT"/>
<sx:subtreeFieldMap select="TLR_0001_COUNT" field="TLR_0001_COUNT"/>
<sx:subtreeFieldMap select="FILLER" field="FILLER"/>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-eobclaims.xml -o output/eobclaims.txt eobclaims-to-flat
The example below illustrates how to prepare a resources script that will convert the
positional flat file books.txt into the delimited flat file
new-books.txt.
The input file is a positional file.
Figure 166. Input positional flat file books.txt
CAuthor Title Price FCharles Bukowski Factotum 22.95 FSergei Lukyanenko The Night Watch 17.99 FAndrew Crumey Mr Mee 22.00 CSteven John Metsker Building Parsers with Java 39.95 This is a trailer record
The desired output file is a pipe delimited flat file.
Figure 167. Output pipe delimited flat file new-books.txt
Author|Category|Title|Price Charles Bukowski|F|Factotum|22.95 Sergei Lukyanenko|F|The Night Watch|17.99 Andrew Crumey|F|Mr Mee|22.00 Steven John Metsker|C|Building Parsers with Java|39.95
The following resources script does the transformation.
Figure 168. Resources script resources-conversion.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="new-books">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="oldBooksFlatFile"/>
</sx:flatFileReader>
<sx:flatFileWriter>
<sx:flatFile ref="newBooksFlatFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="newBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="newBookType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="newBookType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="newBookType" name="newBookType">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="author" label="Author"/>
<sx:delimitedField name="category" label="Category"/>
<sx:delimitedField name="title" label= "Title"/>
<sx:delimitedField name="price" label="Price"/>
</sx:flatRecordType>
<sx:flatFile id="oldBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="oldBookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="oldBookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="oldBookType" name="oldBookType">
<sx:positionalField name="category" width="1"/>
<sx:positionalField name="author" width="30"/>
<sx:positionalField name="title" width="30"/>
<sx:positionalField name="price" width="10" justify="right"/>
</sx:flatRecordType>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-conversion.xml -i data/books.txt
-o output/new-books.txt new-books
The example below illustrates how to prepare a resources script that will convert the
positional flat file books.txt into the new positional flat file
book-defaults.txt.
The input file is a positional file.
Figure 169. Input positional flat file books.txt
CAuthor Title Price FCharles Bukowski Factotum 22.95 FSergei Lukyanenko The Night Watch 17.99 FAndrew Crumey Mr Mee 22.00 CSteven John Metsker Building Parsers with Java 39.95 This is a trailer record
The desired output file is a new positional flat file with a different field ordering, a comma instead of a period in "Price", a new "Pub Date" field initialized to spaces, and a new Publisher field initialized to "Acme".
Figure 170. Output positional flat file books-defaults.txt
Author Title CPub Date Publisher Price Charles Bukowski Factotum F Acme 22,95 Sergei Lukyanenko The Night Watch F Acme 17,99 Andrew Crumey Mr Mee F Acme 22,00 Steven John Metsker Building Parsers with Java C Acme 39,95
This can be accomplished with the following resources script..
Figure 171. Resources script resources-bookDefaults.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="book-defaults">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="oldBooksFlatFile"/>
</sx:flatFileReader>
<sx:flatFileWriter>
<sx:flatFile ref="newBooksFlatFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="newBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="newBookType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="newBookType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="newBookType" name="newBookType">
<sx:positionalField name="author" width="30" label="Author"/>
<sx:positionalField name="title" width="30" label="Title"/>
<sx:positionalField name="category" width="1" label="C"/>
<sx:positionalField name="pubDate" width="10" label="Pub Date"/>
<sx:positionalField name="publisher" width="15" label="Publisher">
<sx:defaultValue>Acme</sx:defaultValue>
</sx:positionalField>
<sx:positionalField name="price2" width="10" justify="right" label="Price">
<sx:defaultValue>
<sx:findAndReplace searchFor ="." replaceWith="," useRegex="false">
<sx:toString value="{price}"/>
</sx:findAndReplace>
</sx:defaultValue>
</sx:positionalField>
</sx:flatRecordType>
<sx:flatFile id="oldBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="oldBookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="oldBookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="oldBookType" name="oldBookType">
<sx:positionalField name="category" width="1"/>
<sx:positionalField name="author" width="30"/>
<sx:positionalField name="title" width="30"/>
<sx:positionalField name="price" width="10" justify="right"/>
</sx:flatRecordType>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-bookDefaults.xml -i data/books.txt
-o output/book-defaults.txt book-defaults
The example below illustrates how to prepare a resources script that will convert the ASCII books.txt file of the last example to an EBCDIC file, and vice versa.
The following resources script does the transformation.
Figure 172. Resources script resources-ascii-ebcdic.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="ascii2ebcdic">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="asciiBooksFile"/>
</sx:flatFileReader>
<sx:flatFileWriter>
<sx:flatFile ref="ebcdicBooksFile"/>
<sx:defaultStreamSink encoding="Cp1047"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:service id="ebcdic2ascii">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="ebcdicBooksFile"/>
<sx:defaultStreamSource encoding="Cp1047"/>
</sx:flatFileReader>
<sx:flatFileWriter>
<sx:flatFile ref="asciiBooksFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="ebcdicBooksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="ebcdicBookType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="ebcdicBookType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="ebcdicBookType" name="ebcdicBookType">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="author" label="Author"/>
<sx:delimitedField name="category" label="Category"/>
<sx:delimitedField name="title" label= "Title"/>
<sx:delimitedField name="price" label="Price"/>
</sx:flatRecordType>
<sx:flatFile id="asciiBooksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="asciiBookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="asciiBookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="asciiBookType" name="asciiBookType">
<sx:positionalField name="category" width="1"/>
<sx:positionalField name="author" width="30"/>
<sx:positionalField name="title" width="30"/>
<sx:positionalField name="price" width="10" justify="right"/>
</sx:flatRecordType>
</sx:resources>
You can run convert the ASCII file books.txt to the EBCDIC file books-ebcdic.dat with the command
servingxml -r resources-ascii-ebcdic.xml -i data/books.txt
-o output/books-ebcdic.dat ascii2ebcdic
You can run convert the EBCDIC file books-ebcdic.dat to the ASCII file books-ascii.txt with the command
servingxml -r resources-ascii-ebcdic.xml -i data/books-ebcdic.dat
-o output/books-ascii.txt ebcdic2ascii
The example below illustrates how to prepare a resources script that will convert the ASCII books.txt file to an EBCDIC positional file with a Cobol packed decimal field, and vice versa.
The following resources script does the transformation.
Figure 173. Resources script resources-ascii-ebcdic.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="books2ebcdic-packed">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="booksFile"/>
</sx:flatFileReader>
<sx:flatFileWriter>
<sx:flatFile ref="booksFile2"/>
<sx:defaultStreamSink encoding="Cp1047"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:service id="ebcdic-packed2books">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="booksFile2"/>
<sx:defaultStreamSource encoding="Cp1047"/>
</sx:flatFileReader>
<sx:flatFileWriter>
<sx:flatFile ref="booksFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="booksFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="bookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="bookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatFile id="booksFile2" lineDelimited="false">
<sx:flatFileBody>
<sx:flatRecordType ref="bookType2"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="bookType" name="bookType">
<sx:positionalField name="category" label="Category" width="1"/>
<sx:positionalField name="author" label="Author" width="30"/>
<sx:positionalField name="title" label="Title" width="30"/>
<sx:positionalField name="price" label="Price" width="10" justify="right"/>
</sx:flatRecordType>
<sx:flatRecordType id="bookType2" name="bookType">
<sx:positionalField name="category" label="Category" width="1"/>
<sx:positionalField name="author" label="Author" width="30"/>
<sx:positionalField name="title" label="Title" width="30"/>
<sx:packedDecimalField name="price" label="Price" digitCount="10" decimalPlaces="2"/>
</sx:flatRecordType>
</sx:resources>
You can run convert the ASCII file, books.txt, to the EBCDIC file with the packed decimal price field, books-ebcdic.dat, with the command
servingxml -r resources-books_ebcdic_packed.xml -i data/books.txt
-o output/books_ebcdic_packed.dat books2ebcdic-packed
You can run convert the EBCDIC file with the packed decimal price field, books_ebcdic_packed.dat, to the ASCII file, books-ascii_unpacked.txt, with the command
servingxml -r resources-books_ebcdic_packed.xml -i data/books_ebcdic_packed.dat
-o output/books_ascii_unpacked.txt ebcdic-packed2books
The example below illustrates how to prepare a resources script that will convert the
positional flat file bookorders-pos.txt into the pipe-delimited flat file
bookorders-delim.txt.
The input file is a positional file.
Figure 174. Input positional flat file bookorders-pos.txt
CAuthor Title Price InvoiceNo InvoiceDate FCharles Bukowski Factotum 22.95 001 12/Mar/2005 FSergei Lukyanenko The Night Watch 17.99 002 13/Mar/2005 FAndrew Crumey Mr Mee 22.00 003 14/June/2005 CSteven John Metsker Building Parsers with Java 39.95 004 15/June/2005 This is a trailer record
The desired output file is a pipe delimited flat file.
Figure 175. Output pipe delimited flat file bookorders-delim.txt
Author|Category|Title|Price|InvoiceNo|InvoiceDate Charles Bukowski|F|Factotum|22.95|INV001|12/03/2005 Sergei Lukyanenko|F|The Night Watch|17.99|INV002|13/03/2005 Andrew Crumey|F|Mr Mee|22.00|INV003|14/06/2005 Steven John Metsker|C|Building Parsers with Java|39.95|INV004|15/06/2005
Note the following:
The invoice number has the text "INV" prepended, e.g. "001" becomes "INV001"
The month labels Mar, June etc. have been converted to numbers.
The following resources script does the transformation.
Figure 176. Resources script resources-bookorders.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="orders-pos-delim">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="posBooksFlatFile"/>
</sx:flatFileReader>
<sx:replaceRecord ref="changeBookOrder"/>
<sx:flatFileWriter>
<sx:flatFile ref="delimBooksFlatFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:replaceRecord id="changeBookOrder">
<sx:newRecord fields="*">
<sx:newField name="invoiceno">
<sx:findAndReplace searchFor ="([0-9]{3})" replaceWith ="INV$1"><sx:toString value="{invoiceno}"/></sx:findAndReplace>
</sx:newField>
<sx:newField name="invoicedate">
<sx:convertDate fromFormat="dd/MMM/yyyy"
toFormat="dd/MM/yyyy">
<sx:toString value="{invoicedate}"/>
</sx:convertDate>
</sx:newField>
</sx:newRecord>
</sx:replaceRecord>
<sx:flatFile id="delimBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="delimBookType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="delimBookType"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType id="delimBookType" name="delimBookType">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="author" label="Author"/>
<sx:delimitedField name="category" label="Category"/>
<sx:delimitedField name="title" label= "Title"/>
<sx:delimitedField name="price" label="Price"/>
<sx:delimitedField name="invoiceno" label="InvoiceNo"/>
<sx:delimitedField name="invoicedate" label="InvoiceDate"/>
</sx:flatRecordType>
<sx:flatFile id="posBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="posBookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="posBookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="posBookType" name="posBookType">
<sx:positionalField name="category" width="1"/>
<sx:positionalField name="author" width="30"/>
<sx:positionalField name="title" width="30"/>
<sx:positionalField name="price" width="10" justify="right"/>
<sx:positionalField name="invoiceno" width="4" justify="right"/>
<sx:positionalField name="invoicedate" width="13" justify="right"/>
</sx:flatRecordType>
</sx:resources>
You can run this example on the command line by entering
servingxml -i data/bookorders-pos.txt -r resources-bookorders.xml
-o output/bookorders-delim.txt orders-pos-delim
The example below illustrates how to prepare a resources script for checking and signing flat files
with CRC values and byte counts. Here, the input file is the
positional flat file checkedBooks.txt, to be converted into the pipe-delimited flat file
signedBooks.txt.
The input file is a positional file.
Figure 177. Input positional flat file checkedBooks.txt
356 2360012630 FCharles Bukowski Factotum 22.95 001 12/Mar/2005 FSergei Lukyanenko The Night Watch 17.99 002 13/Mar/2005 FAndrew Crumey Mr Mee 22.00 003 14/Jun/2005 CSteven John Metsker Building Parsers with Java 39.95 004 15/Jun/2005 This is a trailer record
Note the following.
The file has a two line header section, followed by a data section, followed by a two line trailer section.
The first line of the header contains two positional fields, the size of the data section in bytes, followed by the CRC for the data section.
The desired output file is a pipe delimited flat file.
Figure 178. Output pipe delimited flat file signedBooks.txt
Author|Category|Title|Price|InvoiceNo|InvoiceDate 237|1661800843 Charles Bukowski|F|Factotum|22.95|001|12/Mar/2005 Sergei Lukyanenko|F|The Night Watch|17.99|002|13/Mar/2005 Andrew Crumey|F|Mr Mee|22.00|003|14/Jun/2005 Steven John Metsker|C|Building Parsers with Java|39.95|004|15/Jun/2005
The output file has a two line header section. The two delimited fields in the second line store the size and CRC of the data section below.
The following resources script does the transformation.
Figure 179. Resources script resources-checkedBooks.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="books-pos-delim">
<sx:recordStream>
<sx:flatFileReader>
<sx:flatFile ref="posBooksFlatFile"/>
</sx:flatFileReader>
<sx:flatFileWriter>
<sx:flatFile ref="delimBooksFlatFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="delimBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="delimBookType"/>
<sx:flatRecordType ref="delimBookHeaderType"/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="delimBookType"/>
</sx:flatFileBody>
<sx:flatFileSignature recordType="bookHeaderType" field="filecrc" method="crc"/>
<sx:flatFileSignature recordType="bookHeaderType" field="filesize" method="size"/>
</sx:flatFile>
<sx:flatRecordType id="delimBookType" name="delimBookType">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="author" label="Author"/>
<sx:delimitedField name="category" label="Category"/>
<sx:delimitedField name="title" label= "Title"/>
<sx:delimitedField name="price" label="Price"/>
<sx:delimitedField name="invoiceno" label="InvoiceNo"/>
<sx:delimitedField name="invoicedate" label="InvoiceDate"/>
</sx:flatRecordType>
<sx:flatRecordType id="delimBookHeaderType" name="bookHeaderType">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="filesize"/>
<sx:delimitedField name="filecrc"/>
</sx:flatRecordType>
<sx:flatFile id="posBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="posBookHeaderType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="posBookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
<sx:flatFileSignature recordType="bookHeaderType" field="filesize" method="size"/>
<sx:flatFileSignature recordType="bookHeaderType" field="filecrc" method="crc"/>
</sx:flatFile>
<sx:flatRecordType id="posBookHeaderType" name="bookHeaderType">
<sx:positionalField name="filesize" width="9"/>
<sx:positionalField name="filecrc" width="10"/>
</sx:flatRecordType>
<sx:flatRecordType id="posBookType" name="posBookType">
<sx:positionalField name="category" width="1"/>
<sx:positionalField name="author" width="30"/>
<sx:positionalField name="title" width="30"/>
<sx:positionalField name="price" width="10" justify="right"/>
<sx:positionalField name="invoiceno" width="4" justify="right"/>
<sx:positionalField name="invoicedate" width="13" justify="right"/>
</sx:flatRecordType>
</sx:resources>
Note the following.
The sx:recordContent element contains two sx:flatFileSignature elements, which define the rules for checking the CRC and size when reading the file, and signing the header or trailer with a CRC and size when writing the file.
The sx:flatFileSignature element has attributes
recordType and field, which
identify where in the header or trailer the signature value belongs.
You can run this example on the command line by entering
servingxml -i data/checkedBooks.txt -r resources-checkedBooks.xml
-o output/checkedBooks-delim.txt books-pos-delim
If the CRC value in the header does not match the computed CRC value for the data, processing will be stopped and a message will be written to the log:
CRC integrity check failed. Expected 2360012630, found 252430032.
The example below illustrates how to prepare a resources script that will process
all the files in the data directory matching the regular expression "books.*[.]txt",
convert them from positional flat file formats to pipe delimited formats, and write them out to the
output directory.
This time the input is a directory of files.
Figure 180. Directory containing books fixed record length input files
[.] countries.csv multivalued-field.csv [..] countries.xsd plans.txt 3545_JH4DA3_4_H_.xml country-record.xsd tasks.csv bad-countries.csv exotics.txt timesheets.csv books.20040613.txt hot-record.xsd trades.txt books.20040802.txt hot-record.xsx books.txt hot.txt books.xml messages.properties
We want to take the positional files whose names match the regular expression "(books.*)[.]txt"
and convert them all into pipe delimited files.
Figure 181. Books pipe delimited output files.
books-new.txt books.20040802-new.txt books.20040613-new.txt
The following resources script does the transformation.
Figure 182. Resources script resources-books2csv.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="all-books">
<sx:recordStream>
<sx:directoryReader directory="data">
<sx:fileFilter pattern="books.*[.]txt"/>
</sx:directoryReader>
<sx:processRecord>
<sx:parameter name="output-file">
<sx:findAndReplace searchFor ="(books.*)[.]txt" replaceWith ="$1-new.txt"><sx:toString value="{name}"/></sx:findAndReplace>
</sx:parameter>
<sx:recordStream>
<sx:flatFileReader>
<sx:fileSource directory="{parentDirectory}" file="{name}"/>
<sx:flatFile ref="oldBooksFlatFile"/>
</sx:flatFileReader>
<sx:flatFileWriter>
<sx:fileSink directory="output" file="{$output-file}"/>
<sx:flatFile ref="newBooksFlatFile"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:processRecord>
</sx:recordStream>
</sx:service>
<sx:flatRecordType name="newBookType">
<sx:fieldDelimiter value="|"/>
<sx:delimitedField name="author" label="Author"/>
<sx:delimitedField name="category" label="Category"/>
<sx:delimitedField name="title" label= "Title"/>
<sx:delimitedField name="price" label="Price"/>
</sx:flatRecordType>
<sx:flatFile id="oldBooksFlatFile">
<sx:flatFileHeader>
<sx:flatRecordType ref="oldBookType"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="oldBookType"/>
</sx:flatFileBody>
<sx:flatFileTrailer>
<sx:annotationRecord></sx:annotationRecord>
<sx:annotationRecord>This is a trailer record</sx:annotationRecord>
</sx:flatFileTrailer>
</sx:flatFile>
<sx:flatRecordType id="oldBookType" name="oldBookType">
<sx:positionalField name="category" width="1"/>
<sx:positionalField name="author" width="30"/>
<sx:positionalField name="title" width="30"/>
<sx:positionalField name="price" width="10" justify="right"/>
</sx:flatRecordType>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-conversion.xml all-books
The example below illustrates how to prepare a resources script that will send the the output of a custom record reader to multiple output files, depending on parameter and record field values supplied by the custom record reader.
The input records are generated by the following Java class.
Figure 183. TradeRecordReader Java class
package flat2flat;
import com.servingxml.app.ServiceContext;
import com.servingxml.components.recordio.AbstractRecordReader;
import com.servingxml.app.Flow;
import com.servingxml.util.Name;
import com.servingxml.util.QualifiedName;
import com.servingxml.util.ServingXmlException;
import com.servingxml.util.record.Record;
import com.servingxml.util.record.RecordBuilder;
import com.servingxml.util.record.ParameterBuilder;
public class TradeRecordReader extends AbstractRecordReader {
private static final Name FEED_NAME = new QualifiedName("feed");
private static final Name TRADE_RECORD_TYPE = new QualifiedName("trade");
private static final Name TRANSACTION_RECORD_TYPE = new QualifiedName("transaction");
private static final Name RECORD_TYPE_NAME = new QualifiedName("record_type");
private static final Name ID_NAME = new QualifiedName("id");
private static final Name TRADE_DATE_NAME = new QualifiedName("trade_date");
private static final Name TRADE_TIME_NAME = new QualifiedName("trade_time");
private static final Name DESCRIPTION_NAME = new QualifiedName("description");
private static final Name REFERENCE_NAME = new QualifiedName("reference");
public void readRecords(ServiceContext context, Flow flow)
{
// Start the record stream
startRecordStream(context, flow);
ParameterBuilder parameterBuilder = new ParameterBuilder(flow.getParameters());
RecordBuilder trRecordBuilder = new RecordBuilder(TRADE_RECORD_TYPE);
RecordBuilder tnRecordBuilder = new RecordBuilder(TRANSACTION_RECORD_TYPE);
Record newParameters;
Record record;
// Load London trades
// Set the parameter feed=LONDON
parameterBuilder.setString(FEED_NAME,"LONDON");
newParameters = parameterBuilder.toRecord();
trRecordBuilder.setString(RECORD_TYPE_NAME,"TR");
trRecordBuilder.setString(ID_NAME,"0001");
trRecordBuilder.setString(TRADE_DATE_NAME,"03/25/2005");
trRecordBuilder.setString(TRADE_TIME_NAME,"01:50:00");
trRecordBuilder.setString(DESCRIPTION_NAME,"This is a trade record");
record = trRecordBuilder.toRecord();
trRecordBuilder.clear();
// Write the London "trade" record
Flow newFlow = flow.replaceParameters(context, newParameters).replaceRecord(context, record);
writeRecord(context, newFlow);
tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
tnRecordBuilder.setString(ID_NAME,"0002");
tnRecordBuilder.setString(REFERENCE_NAME,"X1234");
tnRecordBuilder.setString(DESCRIPTION_NAME,"A child transaction");
record = tnRecordBuilder.toRecord();
tnRecordBuilder.clear();
// Write the first London "transaction" record
newFlow = flow.replaceRecord(context, record);
writeRecord(context, newFlow);
tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
tnRecordBuilder.setString(ID_NAME,"0003");
tnRecordBuilder.setString(REFERENCE_NAME,"X1235");
tnRecordBuilder.setString(DESCRIPTION_NAME,"Another child transaction");
record = tnRecordBuilder.toRecord();
tnRecordBuilder.clear();
// Write the second London "transaction" record
newFlow = flow.replaceRecord(context, record);
writeRecord(context, newFlow);
// Load Toronto trades
// Set the parameter feed=TORONTO
parameterBuilder.setString(FEED_NAME,"TORONTO");
newParameters = parameterBuilder.toRecord();
trRecordBuilder.setString(RECORD_TYPE_NAME,"TR");
trRecordBuilder.setString(ID_NAME,"0004");
trRecordBuilder.setString(TRADE_DATE_NAME,"03/25/2005");
trRecordBuilder.setString(TRADE_TIME_NAME,"04:50:00");
trRecordBuilder.setString(DESCRIPTION_NAME,"This is a trade record");
record = trRecordBuilder.toRecord();
trRecordBuilder.clear();
// Write the Toronto "trade" record
newFlow = flow.replaceParameters(context, newParameters).replaceRecord(context, record);
writeRecord(context, newFlow);
tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
tnRecordBuilder.setString(ID_NAME,"0005");
tnRecordBuilder.setString(REFERENCE_NAME,"X1236");
tnRecordBuilder.setString(DESCRIPTION_NAME,"A child transaction");
record = tnRecordBuilder.toRecord();
tnRecordBuilder.clear();
// Write the first Tronto "transaction" record
newFlow = flow.replaceRecord(context, record);
writeRecord(context, newFlow);
tnRecordBuilder.setString(RECORD_TYPE_NAME,"TN");
tnRecordBuilder.setString(ID_NAME,"0006");
tnRecordBuilder.setString(REFERENCE_NAME,"X1237");
tnRecordBuilder.setString(DESCRIPTION_NAME,"Another child transaction");
record = tnRecordBuilder.toRecord();
tnRecordBuilder.clear();
// Write the second Toronto "transaction" record
newFlow = flow.replaceRecord(context, record);
writeRecord(context, newFlow);
// End the record stream
endRecordStream(context, flow);
}
}
With parameter feed=LONDON,
the TradeRecordReader class will generate the following stream of records.
| record_type | id | |||
|---|---|---|---|---|
| TR | 0001 | 03/25/2005 | 1:50:00 | This is a trade record |
| TN | 0002 | X1234 | A child transaction | |
| TN | 0003 | X1235 | Another child transaction |
With parameter feed=TORONTO,
the TradeRecordReader class will generate the following stream of records.
| record_type | id | |||
|---|---|---|---|---|
| TR | 0004 | 03/25/2005 | 4:50:00 | This is a trade record |
| TN | 0005 | X1236 | A child transaction | |
| TN | 0006 | X1237 | Another child transaction |
The LONDON master trades need to be written to the file trades-London.txt.
Figure 184. Output flat file trades-London.txt
000103/25/200501:50:00This is a trade record
The LONDON transaction details need to be written to the file transaction-London.txt.
Figure 185. Output flat file transactions-London.txt
0002X1234A child transaction 0003X1235Another child transaction
The TORONTO master trades need to be written to the file trades-Toronto.txt.
Figure 186. Output flat file trades-Toronto.txt
000403/25/200504:50:00This is a trade record
The TORONTO transaction details need to be written to the file transaction-Toronto.txt.
Figure 187. Output flat file transactions-Toronto.txt
0005X1236A child transaction 0006X1237Another child transaction
The following resources script does the transformation.
Figure 188. Resources script resources-multiple_output_files.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="trades">
<sx:recordStream>
<sx:customRecordReader class="flat2flat.TradeRecordReader"/>
<!-- Which output file is determined by the parameter "feed"
and the record field "record_type", which are both
supplied in the custom record reader.
-->
<sx:choose>
<sx:when test="$feed='LONDON' and record_type = 'TR'">
<sx:flatFileWriter>
<sx:fileSink directory="output" file="trades-London.txt"/>
<sx:flatFile ref="trades"/>
</sx:flatFileWriter>
</sx:when>
<sx:when test="$feed = 'LONDON' and record_type = 'TN'">
<sx:flatFileWriter>
<sx:fileSink directory="output" file="transactions-London.txt"/>
<sx:flatFile ref="transactions"/>
</sx:flatFileWriter>
</sx:when>
<sx:when test="$feed='TORONTO' and record_type = 'TR'">
<sx:flatFileWriter>
<sx:fileSink directory="output" file="trades-Toronto.txt"/>
<sx:flatFile ref="trades"/>
</sx:flatFileWriter>
</sx:when>
<sx:when test="$feed = 'TORONTO' and record_type = 'TN'">
<sx:flatFileWriter>
<sx:fileSink directory="output" file="transactions-Toronto.txt"/>
<sx:flatFile ref="transactions"/>
</sx:flatFileWriter>
</sx:when>
</sx:choose>
</sx:recordStream>
</sx:service>
<sx:flatFile id="trades">
<sx:flatFileBody>
<sx:flatRecordType name="trade">
<sx:positionalField name="id" width="4"/>
<sx:positionalField name="trade_date" width="10"/>
<sx:positionalField name="trade_time" width="8"/>
<sx:positionalField name="description" width="30"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatFile id="transactions">
<sx:flatFileBody>
<sx:flatRecordType name="transaction">
<sx:positionalField name="id" width="4"/>
<sx:positionalField name="reference" width="5"/>
<sx:positionalField name="description" width="30"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
Note the following points.
TradeRecordReader.
You can run this example on the command line by
TradeRecordReader and copying
the resulting .class file into the dir/classes directory
servingxml -r resources-multiple_output_files.xml trades
This example shows how to read a remote directory and write a listing of its contents to a flat file.
The desired output is a positional flat file showing a remote directory listing.
Figure 189. Output flat file with directory listing
Parent Name Size Date Modified / incoming 28672 Mon Dec 20 03:27:00 PST 2004 / lost+found 16384 Wed Feb 07 00:00:00 PST 2001 / pub 4096 Sun Oct 17 23:45:00 PDT 2004 /pub sf-overflow 17 Sun Oct 17 23:45:00 PDT 2004 /pub sourceforge 581632 Mon Nov 15 18:34:00 PST 2004 /pub/sourceforge a 28672 Fri Mar 19 00:00:00 PST 2004 /pub/sourceforge/a a- 4096 Wed Oct 06 18:36:00 PDT 2004 /pub/sourceforge/a/a- a-4 4096 Tue Dec 16 00:00:00 PST 2003 /pub/sourceforge/a/a-/a-4 a4-0.01777.tar.gz 173990 Thu Feb 13 00:00:00 PST 2003 /pub/sourceforge/a/a-/a-4 a4-0.03109.tar.gz 175327 Tue May 06 00:00:00 PDT 2003
The following resources script does the transformation.
Figure 190. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:edt="http://www.servingxml.com/extensions/edtftp">
<edt:ftpClient id="sourceforge" host="upload.sourceforge.net"
user="anonymous" password="xxx"/>
<sx:service id="remote-dir-listing">
<sx:recordStream>
<edt:ftpDirectoryReader remoteDirectory="." recurse="true" maxItems="10">
<edt:ftpClient ref="sourceforge"/>
</edt:ftpDirectoryReader>
<sx:flatFileWriter>
<sx:flatFile ref="directory-flat-file"/>
</sx:flatFileWriter>
</sx:recordStream>
</sx:service>
<sx:flatRecordType id="directory-record-type" name="directory-record-type">
<sx:positionalField name="parentDirectory" label="Parent" width="30"/>
<sx:positionalField name="name" label="Name" width="30"/>
<sx:positionalField name="size" label="Size" width="10"/>
<sx:positionalField name="lastModified" label="Date Modified" width="30"/>
</sx:flatRecordType>
<sx:flatFile id="directory-flat-file">
<sx:flatFileHeader>
<sx:flatRecordType ref="directory-record-type"/>
<sx:annotationRecord/>
</sx:flatFileHeader>
<sx:flatFileBody>
<sx:flatRecordType ref="directory-record-type"/>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources.xml -o output/remote-dir-listing.txt remote-dir-listing
The example below illustrates how to prepare a resources script that wil break up a stream of records into multiple batches.
We want to read the CSV file countries.csv containing 245 country records and produce five files containing 50, 50, 50, 50 and 45 country records.
Figure 191. Batched countries files.
06/12/2007 07:24 PM <DIR> . 06/12/2007 07:24 PM <DIR> .. 06/12/2007 07:24 PM 814 countries-1.csv 06/12/2007 07:24 PM 795 countries-2.csv 06/12/2007 07:24 PM 786 countries-3.csv 06/12/2007 07:24 PM 784 countries-4.csv 06/12/2007 07:24 PM 793 countries-5.csv
The following resources script does the transformation.
Figure 192. Resources script resources-batchRecords.xml
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="countries">
<sx:recordStream>
<sx:flatFileReader>
<sx:fileSource file="data/{$rootName}.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileReader>
<sx:batchedRecordWriter batchSize="50">
<sx:flatFileWriter>
<sx:fileSink file="output/{$rootName}-{$sx:batchSequenceNumber}.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileWriter>
</sx:batchedRecordWriter>
</sx:recordStream>
</sx:service>
<sx:flatFile id="countriesFile">
<sx:commentStarter value="#"/>
<sx:flatFileBody>
<sx:flatRecordType name="country">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="code"/>
<sx:delimitedField name="name" trimLeading="true"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-batchRecords.xml countries rootName=countries
The example below illustrates how to prepare a resources script that wil break up a flat file into smaller batches.
Suppose you want to read the CSV file countries.csv containing 245 country records and write it out as five files containing 50, 50, 50, 50 and 45 records.
Figure 193. Batched CSV files.
06/12/2007 07:24 PM <DIR> . 06/12/2007 07:24 PM <DIR> .. 06/12/2007 07:24 PM 814 countries-1.csv 06/12/2007 07:24 PM 795 countries-2.csv 06/12/2007 07:24 PM 786 countries-3.csv 06/12/2007 07:24 PM 784 countries-4.csv 06/12/2007 07:24 PM 793 countries-5.csv
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="countries">
<sx:recordStream>
<sx:flatFileReader>
<sx:fileSource file="data/{$rootName}.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileReader>
<sx:batchedRecordWriter batchSize="50">
<sx:flatFileWriter>
<sx:fileSink file="intermediate/{$rootName}-{$sx:batchSequenceNumber}.csv"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileWriter>
</sx:batchedRecordWriter>
</sx:recordStream>
</sx:service>
<sx:recordContent id="countries">
<sx:flatFileReader>
<sx:fileSource directory="{parentDirectory}" file="{name}"/>
<sx:flatFile ref="countriesFile"/>
</sx:flatFileReader>
<sx:recordMapping ref="countriesToXmlMapping"/>
</sx:recordContent>
<sx:flatFile id="countriesFile">
<sx:commentStarter value="#"/>
<sx:flatFileBody>
<sx:flatRecordType name="country">
<sx:fieldDelimiter value=","/>
<sx:delimitedField name="code"/>
<sx:delimitedField name="name" trimLeading="true"/>
</sx:flatRecordType>
</sx:flatFileBody>
</sx:flatFile>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-batchRecords.xml countries rootName=countries
This example shows how to extract a set of subtrees from an XML document according to some criteria and wrap them in containing tags.
The input file is an XML document.
Figure 194. Input XML document books.xml
<?xml version="1.0" encoding="UTF-8"?>
<myns:books xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="url2">
<myns:book categoryCode="F">
<myns:title>Factotum</myns:title>
<myns:author>Charles Bukowski</myns:author>
<myns:price>22.95</myns:price>
<myns:isbn>0876852630</myns:isbn>
</myns:book>
<myns:book categoryCode="F">
<myns:title>The Night Watch</myns:title>
<myns:author>Sergei Lukyanenko</myns:author>
<myns:price>17.99</myns:price>
<myns:isbn>0434014125</myns:isbn>
</myns:book>
<myns:book categoryCode="F">
<myns:title>Mr Mee</myns:title>
<myns:author>Andrew Crumey</myns:author>
<myns:price>22.00</myns:price>
<myns:isbn>0312282354</myns:isbn>
</myns:book>
<myns:book categoryCode="C">
<myns:title>Building Parsers with Java</myns:title>
<myns:author>Steven John Metsker</myns:author>
<myns:price>39.95</myns:price>
<myns:isbn>0201719622</myns:isbn>
</myns:book>
</myns:books>
The desired output is.
<?xml version="1.0" encoding="UTF-8"?>
<envelope>
<header mode="xml">
<creationDate>2007-12-03T21:39:48.265-05:00</creationDate>
</header>
<body>
<myns:books xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="url2">
<myns:book categoryCode="F">
<myns:title>Factotum</myns:title>
<myns:author>Charles Bukowski</myns:author>
<myns:price>22.95</myns:price>
<myns:isbn>0876852630</myns:isbn>
</myns:book>
<myns:book categoryCode="F">
<myns:title>The Night Watch</myns:title>
<myns:author>Sergei Lukyanenko</myns:author>
<myns:price>17.99</myns:price>
<myns:isbn>0434014125</myns:isbn>
</myns:book>
<myns:book categoryCode="F">
<myns:title>Mr Mee</myns:title>
<myns:author>Andrew Crumey</myns:author>
<myns:price>22.00</myns:price>
<myns:isbn>0312282354</myns:isbn>
</myns:book>
</myns:books>
</body>
<trailer>This is a trailer element</trailer>
</envelope>
The resources script below does the transformation.
Figure 195. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/">
<sx:service id="wrap">
<sx:wrap>
<sx:xsltSerializer/>
<envelope>
<header>
<sx:attribute name="mode" value="xml"/>
<creationDate>
<sx:currentDateTime/>
</creationDate>
</header>
<body>
<myns:books xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="url2">
<sx:transform>
<sx:document/>
<sx:processSubtree path="/myns:books/myns:book">
<sx:choose>
<sx:when test="@categoryCode='F'">
<sx:transform/>
</sx:when>
</sx:choose>
</sx:processSubtree>
</sx:transform>
</myns:books>
</body>
<trailer>This is a trailer element</trailer>
</envelope>
</sx:wrap>
</sx:service>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-wrap.xml -i documents/books.xml -o output/wrap.xml wrap
Continuing from the previous example, the input file is the same, but you want to produce multiple files like the one shown below, with filenames differentiated by the isbn number.
Figure 196. Book XML output file book-0876852630.xml.
<?xml version="1.0" encoding="UTF-8"?>
<myns:book xmlns:myns="http://mycompany.com/mynames/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
categoryCode="F">
<myns:title>Factotum</myns:title>
<myns:author>Charles Bukowski</myns:author>
<myns:price>22.95</myns:price>
<myns:isbn>0876852630</myns:isbn>
</myns:book>
The resources script below does the transformation.
Figure 197. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:myns="http://mycompany.com/mynames/" >
<sx:service id="unwrap">
<sx:transform>
<sx:document/>
<sx:processSubtree path="/myns:books/myns:book">
<sx:choose>
<sx:when test="@categoryCode='F'">
<sx:parameter name="isbn" select="myns:isbn"/>
<sx:transform>
<sx:xsltSerializer>
<sx:fileSink file="output/book-{$isbn}.xml"/>
</sx:xsltSerializer>
</sx:transform>
</sx:when>
</sx:choose>
</sx:processSubtree>
</sx:transform>
</sx:service>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-unwrap.xml -i documents/books.xml unwrap
This example shows an implementation of the SAX Pipeline example in Michael Kay's XSLT 2nd Edition Programmer's Reference, Appendix F, Example 5.
Figure 198. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="pipeline">
<sx:serialize>
<sx:transform>
<sx:saxFilter class="PreFilter"/>
<sx:xslt><sx:urlSource url="styles/filter.xsl"/></sx:xslt>
<sx:saxFilter class="PostFilter"/>
</sx:transform>
</sx:serialize>
</sx:service>
</sx:resources>
PreFilter and PostFilter are the names of Java classes that
implement the org.xml.sax.XMLFilter interface. The listings below show the
Java code for the PreFilter and PostFilter classes
(from Kay's book.)
Figure 199. PreFilter class
public class PreFilter extends XMLFilterImpl {
public void startElement (String uri, String localName, String qName,
Attributes atts) throws SAXException {
String newLocalName = localName.toLowerCase();
String newQName = qName.toUpperCase();
AttributesImpl newAtts =
(atts.getLength()>0 ?
new AttributesImpl(atts) :
new AttributesImpl());
newAtts.addAttribute("", "old-local-name", "old-local-name", "CDATA", localName);
newAtts.addAttribute("", "old-qname", "old-qname", "CDATA", qName);
super.startElement(uri, newLocalName, newQName, newAtts);
}
public void endElement (String uri, String localName, String qName)
throws SAXException {
String newLocalName = localName.toLowerCase();
String newQName = qName.toUpperCase();
super.endElement(uri, newLocalName, newQName);
}
}
Figure 200. PostFilter class
public class PostFilter extends XMLFilterImpl {
public Stack stack;
public void startDocument() throws SAXException {
stack = new Stack();
super.startDocument();
}
public void startElement (String uri, String localName, String qName,
Attributes atts)
throws SAXException {
String originalLocalName = localName;
String originalQName = qName;
AttributesImpl newAtts = new AttributesImpl();
for (int i=0; i<atts.getLength(); i++) {
String name = atts.getQName(i);
String val = atts.getValue(i);
if (name.equals("old-local-name")) {
originalLocalName = val;
} else if (name.equals("old-qname")) {
originalQName = val;
} else {
newAtts.addAttribute(atts.getURI(i),
atts.getLocalName(i),
name,
atts.getType(i),
val);
}
}
super.startElement(uri, originalLocalName, originalQName, newAtts);
stack.push(originalLocalName);
stack.push(originalQName);
}
public void endElement (String uri, String localName, String qName)
throws SAXException {
String originalQName = (String)stack.pop();
String originalLocalName = (String)stack.pop();
super.endElement(uri, originalLocalName, originalQName);
}
}
You can run this example on the command line by
XMLFilter classes and copy them into
the dir/classes directory.
servingxml -r resources-file.xml -i documents/mixed-up.xml
-o output/original-file.html pipeline
This example produces the same output as the previous, but this time, instead of reading an XML document from a file, we generate the SAX events dynamically in a custom SAX reader.
Figure 201. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="pipeline">
<sx:serialize>
<sx:transform>
<sx:document>
<sx:saxReader class="MySaxReader"/>
</sx:document>
<sx:saxFilter class="PreFilter"/>
<sx:xslt><sx:urlSource url="styles/filter.xsl"/></sx:xslt>
<sx:saxFilter class="PostFilter"/>
</sx:transform>
</sx:serialize>
</sx:service>
</sx:resources>
The Java class MySaxReader looks like this.
public class MySaxReader implements XMLReader ...
private static final AttributesImpl NO_ATTRIBUTES = new AttributesImpl();
public void parse(String systemId)
throws IOException, SAXException {
if (contentHandler != null) {
contentHandler.startDocument();
contentHandler.startElement("","HTML","HTML",NO_ATTRIBUTES);
contentHandler.startElement("","Head","Head",NO_ATTRIBUTES);
contentHandler.startElement("","TITLE","TITLE",NO_ATTRIBUTES);
char[] title = "This is a mixed-case HTML-like XML document".toCharArray();
contentHandler.characters(title,0,title.length);
contentHandler.endElement("","TITLE","TITLE");
contentHandler.endElement("","Head","Head");
contentHandler.startElement("","Body","Body");
...
contentHandler.endElement("","Body","Body");
contentHandler.endElement("","HTML","HTML");
contentHandler.endDocument();
}
}
You can run this example on the command line by
XMLFilter classes and copy them into
the dir/classes directory.
servingxml -r resources-sax.xml -o output/original-sax.html pipeline
This example shows how to serialize individual document subtrees to separate html and pdf files.
The input file is an XML document containing a number of invoices.
Figure 202. Input XML file invoices.xml
<?xml version="1.0" encoding="iso-8859-1"?>
<inv:invoices xmlns:inv="http://www.telio.be/ns/2002/invoice">
<inv:invoice currency="EUR" id="200302-01" paid="false">
<inv:client id="test">
<inv:email>john@abc.com</inv:email>
</inv:client>
<inv:date>2003-02-03</inv:date>
<inv:description>Test description</inv:description>
<inv:item>
<inv:quantity>67.5</inv:quantity>
<inv:unit>h</inv:unit>
<inv:description>January 2004</inv:description>
<inv:unitprice>400</inv:unitprice>
<inv:tax>21.00</inv:tax>
</inv:item>
</inv:invoice>
<inv:invoice currency="EUR" id="200302-02" paid="true">
<inv:client id="test">
<inv:email>jane@efg.com</inv:email>
</inv:client>
<inv:date>2003-02-03</inv:date>
<inv:description>Test description</inv:description>
<inv:item>
<inv:quantity>67.5</inv:quantity>
<inv:unit>h</inv:unit>
<inv:description>January 2004</inv:description>
<inv:unitprice>400</inv:unitprice>
<inv:tax>21.00</inv:tax>
</inv:item>
</inv:invoice>
</inv:invoices>
The desired output is a set of HTML and PDF invoice documents, one pair for each inv:invoice element in the master XML document.
Figure 203. Directory listing of output files
invoice200302-01.html invoice200302-01.pdf invoice200302-02.html invoice200302-02.pdf
The following resources script does the transformation.
Figure 204. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:fop="http://www.servingxml.com/extensions/fop"
xmlns:inv="http://www.telio.be/ns/2002/invoice">
<sx:service id="invoices">
<sx:transform>
<sx:document/>
<!-- Here we extract a document subtree from the SAX stream -->
<sx:processSubtree path="/inv:invoices/inv:invoice">
<!-- Main pipeline - invoice document subtree to pdf-->
<sx:transform>
<!-- We initialize a parameter with an XPATH expression
applied to the document subtree -->
<sx:parameter name="invoice-name" select="@id"/>
<fop:foSerializer>
<sx:fileSink file="output/invoice{$invoice-name}.pdf"/>
</fop:foSerializer>
<sx:transform>
<sx:transform ref="steps1-4"/>
<!-- Tee - invoice document subtree to html-->
<sx:tagTee>
<sx:xslt documentBase="documents/">
<sx:urlSource url="styles/invoice2html.xsl"/>
</sx:xslt>
<sx:xsltSerializer>
<sx:fileSink file="output/invoice{$invoice-name}.html"/>
</sx:xsltSerializer>
</sx:tagTee>
<sx:xslt documentBase="documents/">
<sx:urlSource url="styles/invoice2fo.xsl"/>
</sx:xslt>
</sx:transform>
</sx:transform>
</sx:processSubtree>
</sx:transform>
</sx:service>
<sx:transform id="steps1-4">
<sx:xslt><sx:urlSource url="styles/step1.xsl"/></sx:xslt>
<sx:xslt><sx:urlSource url="styles/step2.xsl"/></sx:xslt>
<sx:xslt><sx:urlSource url="styles/step3.xsl"/></sx:xslt>
<sx:xslt documentBase="documents/">
<sx:urlSource url="styles/step4.xsl"/>
</sx:xslt>
</sx:transform>
</sx:resources>
Not the sx:tagTee element, which forks the SAX event stream, and serializes this stream as HTML.
You can run this example on the command line by entering
servingxml -r resources-invoices.xml -i documents/invoices.xml invoices
Continuing with the previous example, suppose you want to send the invoices as pdf mail attachments. Then you need the following resources script.
Figure 205. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:fop="http://www.servingxml.com/extensions/fop"
xmlns:inv="http://www.telio.be/ns/2002/invoice"
xmlns:jm="http://www.servingxml.com/extensions/javamail">
<sx:service id="invoices">
<sx:transform>
<sx:document/>
<!-- Here we extract a document subtree from the SAX stream -->
<sx:processSubtree path="/inv:invoices/inv:invoice">
<sx:parameter name="email" select="inv:client/inv:email"/>
<!-- We want to send an email with the PDF invoice as an attachment -->
<jm:sendMail subject="Invoice" to="{$email}">
<sx:parameter name="invoice-name" select="@id"/>
<jm:mailAccount ref="myMailAccount"/>
<jm:message>
<jm:attachment filename="{$invoice-name}.pdf">
<sx:serialize>
<!-- We initialize a parameter with an XPATH expression
applied to the document subtree -->
<fop:foSerializer/>
<sx:transform>
<sx:transform ref="steps1-4"/>
<sx:xslt documentBase="documents/">
<sx:urlSource url="styles/invoice2fo.xsl"/>
</sx:xslt>
</sx:transform>
</sx:serialize>
</jm:attachment>
</jm:message>
<sx:onError>
<!-- If send mail fails (e.g. no smtp host), just proceed with the next action -->
<!-- (the error will still be logged) -->
</sx:onError>
</jm:sendMail>
</sx:processSubtree>
</sx:transform>
</sx:service>
<sx:transform id="steps1-4">
<sx:xslt>
<sx:urlSource url="styles/step1.xsl"/>
</sx:xslt>
<sx:xslt>
<sx:urlSource url="styles/step2.xsl"/>
</sx:xslt>
<sx:xslt>
<sx:urlSource url="styles/step3.xsl"/>
</sx:xslt>
<sx:xslt documentBase="documents/">
<sx:urlSource url="styles/step4.xsl"/>
</sx:xslt>
</sx:transform>
<!-- Set up this mail account, substituting your smtp host, display name, and email address -->
<!-- You will also need to enter the 'to' and 'cc' recipient addresses in the jm:sendMail element -->
<jm:mailAccount id="myMailAccount"
smtpHost="smtp1.abc.ca">
<jm:sender displayName="Daniel"
emailAddress="danielaparker@abc.ca"/>
</jm:mailAccount>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources-mailInvoices.xml -i documents/invoices.xml invoices
This example shows an implementation of dynamic content handlers.
Running this example requires the following steps:
Run the sample build script at the command line, to compile the DynamicContentHandler classes and copy them into the distribution classes directory.
Enter the following command to produce a page of books in the fiction (F) category,
servingxml -r resources.xml -o output/books.html books category=F
Figure 206. Resources script
<sx:resources>
<sx:service id="swath">
<sx:serialize>
<sx:transform>
<sx:content ref="swath"/>
<sx:xslt ref="swath"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:xslt id="swath">
<sx:urlSource url="styles/books.xsl"/>
<sx:parameter name="category"/>
</sx:xslt>
<!-- BookCatalog is the name of a Java class used to produce dynamic content for the primary document -->
<sx:dynamicContent name="swath" class="dynamiccontent.BookCatalog"/>
<!-- Categories is the name of a Java class used to produce dynamic content in a
document function in the stylesheet -->
<sx:dynamicContent name="categories" class="dynamiccontent.Categories"/>
</sx:resources>
BookCatalog and Categories are the names of Java classes that
implement the com.servingxml.components.content.dynamic.DynamicContentHandler interface.
The listing below shows the Java code for the Categories class.
Figure 207. Categories
package dynamiccontent;
public class Categories implements DynamicContentHandler {
static class Category {
private final String code;
private final String description;
public Category(String code, String description) {
this.code = code;
this.description = description;
}
public String getCode() {
return code;
}
public String getDescription() {
return description;
}
}
private final HashMap categoryMap;
private final String namespace = "";
public Categories() {
categoryMap = new HashMap();
categoryMap.put("S", new Category("S","Science"));
categoryMap.put("C", new Category("C","Computing"));
categoryMap.put("CHILD", new Category("CHILD","Children"));
categoryMap.put("F", new Category("F","Fiction"));
categoryMap.put("P", new Category("P","Philospohy"));
categoryMap.put("R", new Category("R","Religion"));
}
// The onRequest method must be reentrant
public void onRequest(ServiceContext context, Book parameters,
ContentWriter contentWriter) {
contentWriter.startElement(namespace,"categories");
String id = parameters.getCategory();
if (id != null && id.length() != 0) {
if (id.equals("all")) {
Collection values = categoryMap.values();
Iterator iter = values.iterator();
while (iter.hasNext()) {
Category category = (Category)iter.next();
writeCategory(category,contentWriter);
}
} else {
Category category = (Category)categoryMap.get(id);
writeCategory(category,contentWriter);
}
}
contentWriter.endElement(namespace,"categories");
}
private void writeCategory(Category category, ContentWriter contentWriter)
{
AttributeSet attributes = contentWriter.newAttributeSet();
attributes.addAttribute(namespace,"code",category.getCode());
attributes.addAttribute(namespace,"desc",category.getDescription());
contentWriter.startEndElement(namespace,"category",attributes);
}
}
You can run this example on the command line by
servingxml -r resources.xml -o output/books.html swath category=F
The example below illustrates how to prepare a resources script that will transform an XML document to XSL-FO, format it to PDF, and send the result as an attachment in a mail message.
This example requires the mail module.
Once you have everything configured, rebuild the ServingXML jar file in the servingxml directory.
Figure 208. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:fop="http://www.servingxml.com/extensions/fop"
xmlns:jm="http://www.servingxml.com/extensions/javamail">
<sx:service id="pulp">
<!-- Our first action is to send an email with the pulp novel as an attachment -->
<jm:sendMail subject="test" to="john@yyy.com" cc="linda@zzz.com">
<jm:mailAccount ref="myMailAccount"/>
<jm:message>
<!-- Put some HTML in the message body -->
<jm:part type="alternative">
<sx:transform>
<sx:document><xxx/></sx:document>
<sx:xslt ref="welcome/html"/>
</sx:transform>
</jm:part>
<jm:attachment filename="pulp.pdf">
<fop:foSerializer/>
<sx:transform>
<sx:content ref="pulp"/>
<sx:xslt ref="novel/fo"/>
</sx:transform>
</jm:attachment>
</jm:message>
<sx:onError>
<!-- If send mail fails (e.g. no smtp host), just proceed with the next action -->
<!-- (the error will still be logged) -->
</sx:onError>
</jm:sendMail>
<!-- Our second action is to write the pulp novel to a file -->
<sx:serialize>
<fop:foSerializer/>
<sx:transform>
<sx:content ref="pulp"/>
<sx:xslt ref="novel/fo"/>
</sx:transform>
</sx:serialize>
</sx:service>
<!-- Set up this mail account, substituting your smtp host, display name, and email address -->
<!-- You will also need to enter the 'to' and 'cc' recipient addresses in the jm:sendMail element -->
<jm:mailAccount id="myMailAccount"
smtpHost="smtp1.abc.ca">
<jm:sender displayName="DP"
emailAddress="danielaparker@abc.ca"/>
</jm:mailAccount>
<sx:document id="pulp">
<sx:urlSource url="data/pulp.xml"/>
</sx:document>
<sx:xslt id="novel/fo">
<sx:urlSource url="data/novel.xsl"/>
</sx:xslt>
<sx:xslt id="welcome/html">
<sx:urlSource url="data/welcome.xsl"/>
</sx:xslt>
</sx:resources>
This example produces two outputs:
A mail message with an HTML body part displaying "Welcome to Presenting Books" and a PDF attachment.
A PDF file pulp.pdf written to the output directory.
You can run this example on the command line by entering
servingxml -r resources.xml -o output/pulp.pdf pulp
This example shows how to send stream output to an FTP host.
Figure 209. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:edt="http://www.servingxml.com/extensions/edtftp">
<edt:ftpClient id="myFtpSite" host="myFtpHost"
user="anonymous" password="xxx"/>
<sx:service id="books">
<sx:serialize>
<sx:xsltSerializer>
<edt:ftpSink remoteDirectory="incoming" remoteFile="books{$version}.xml">
<edt:ftpClient ref="myFtpSite"/>
</edt:ftpSink>
</sx:xsltSerializer/>
<sx:transform>
<sx:content ref="books"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:document id="books">
<sx:urlSource url="documents/books.xml"/>
</sx:document>
</sx:resources>
You can run this example on the command line by
Updating the resources.xml file with a valid FTP host
Entering the following command to send an XML file to the FTP host,
servingxml -r resources.xml books version=1.0
The example below shows how to compare two XML documents for equality with the XPATH fn:deep-equal function.
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:fn="http://www.w3.org/2005/xpath-functions">
<sx:service id="compare-documents">
<sx:task ref="test-documents-are-equal"/>
</sx:service>
<sx:document id="odd-element-document">
<document>
<title>Title</title>
<para>First paragraph.</para>
</document>
</sx:document>
<sx:document id="another-odd-element-document">
<document>
<title>Title</title>
<para>First paragraph.</para>
</document>
</sx:document>
<sx:serialize id="test-documents-are-equal">
<sx:transform>
<sx:choose>
<sx:when test="fn:deep-equal(fn:document('odd-element-document'),
fn:document('another-odd-element-document'))">
<sx:document>
<result>equal</result>
</sx:document>
</sx:when>
<sx:otherwise>
<sx:document>
<result>unequal</result>
</sx:document>
</sx:otherwise>
</sx:choose>
</sx:transform>
</sx:serialize>
</sx:resources>
The documents are equal, so the script produces the output <result>equal</result>
You can run this example on the command line by entering
servingxml -r resources-compare.xml -o output/compare-documents.xml compare-documents
The java run time needs to be able to find the database driver.
The examples in this section are set up for the Oracle driver,
oracle.jdbc.driver.OracleDriver. To configure ServingXML to recognize
this driver, copy the Oracle classes12.jar file (not distributed) to
the lib/local directory, and then rebuild. Of course, you can specify a different driver,
making the appropriate changes in the sample resources script and copying the right
jar file to the lib/local directory.
This example shows how to prepare a resources script that will write the results of a SQL query to a CSV file, using a default flat file writer.
The desired CSV file is shown below.
Figure 210. Output XML file employees.csv
EMPNO,NAME,JOB 7902,FORD,ANALYST 7788,SCOTT,ANALYST 7876,ADAMS,CLERK 7900,JAMES,CLERK 7934,MILLER,CLERK 7369,SMITH,CLERK
The following resources script does the transformation.
Figure 211. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="employees-to-csv">
<sx:recordStream>
<sx:recordReader ref="employees"/>
<sx:flatFileWriter/>
</sx:recordStream>
</sx:service>
<sx:sqlReader id="employees">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
<sx:choose>
<sx:when test="$jobList">
AND JOB IN ({$jobList})
</sx:when>
</sx:choose>
ORDER BY JOB,ENAME
</sx:sqlQuery>
</sx:sqlReader>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="{$user}"
password="{$password}"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Executing the employees service with two job parameters, like this
servingxml -r resources-employees_csv.xml -o output/employees.csv employees-to-csv
user=scott password=spring job=ANALYST job=CLERK
results in the following SQL statement being sent to the Oracle server:
SELECT EMPNO, ENAME AS NAME,JOB
FROM EMP
WHERE 1=1 AND JOB IN ('ANALYST','CLERK')
ORDER BY JOB,ENAME
The attribute names EMPNO, NAME and JOB become the headings of the CSV file, and the values in each row are written out below.
The example below illustrates how to prepare a resources script that will convert the results of a SQL query into an XML document.
The desired output is shown below.
Figure 212. Output XML file employees.xml
<?xml version="1.0" encoding="utf-8"?>
<employees>
<employee employee-no="7788">
<name>SCOTT</name>
<job>ANALYST</job>
</employee>
<employee employee-no="7902">
<name>FORD</name>
<job>ANALYST</job>
</employee>
</employees>
The following resources script does the transformation.
Figure 213. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="employees">
<sx:serialize>
<sx:content ref="employees"/>
</sx:serialize>
</sx:service>
<sx:recordContent id="employees">
<sx:sqlReader>
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE JOB = '{$job}'
</sx:sqlQuery>
</sx:sqlReader>
<sx:recordMapping ref="employeesToXml"/>
</sx:recordContent>
<sx:recordMapping id="employeesToXml">
<employees>
<sx:groupBy fields="JOB">
<sx:elementMap element="{JOB}">
<sx:onRecord>
<employee>
<sx:fieldAttributeMap field="EMPNO" attribute="employee-no"/>
<sx:fieldElementMap field="NAME" element="name"/>
</employee>
</sx:onRecord>
</sx:elementMap>
</sx:groupBy>
</employees>
</sx:recordMapping>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="{$user}"
password="{$password}"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources.xml employees -o output/employees.txt employees
user=scott password=spring job=ANALYST
The example below illustrates how to prepare a resources script that will generate and execute a different SQL statement depending on passed parameters.
The desired output is shown below.
Figure 214. Output XML file employees.xml
<?xml version="1.0" encoding="utf-8"?>
<employees>
<ANALYST>
<employee employee-no="7902">
<name>FORD</name>
</employee>
<employee employee-no="7788">
<name>SCOTT</name>
</employee>
</ANALYST>
<CLERK>
<employee employee-no="7876">
<name>ADAMS</name>
</employee>
<employee employee-no="7900">
<name>JAMES</name>
</employee>
<employee employee-no="7934">
<name>MILLER</name>
</employee>
<employee employee-no="7369">
<name>SMITH</name>
</employee>
</CLERK>
</employees>
The following resources script does the transformation.
Figure 215. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="employees">
<sx:serialize>
<sx:content ref="employeesDoc"/>
</sx:serialize>
</sx:service>
<sx:recordContent id="employeesDoc">
<sx:sqlReader>
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
<sx:choose>
<sx:when test="$jobList">
AND JOB IN ({$jobList})
</sx:when>
</sx:choose>
ORDER BY JOB,ENAME
</sx:sqlQuery>
</sx:sqlReader>
<sx:recordMapping ref="employeesToXml"/>
</sx:recordContent>
<sx:recordMapping id="employeesToXml">
<employees>
<sx:groupBy fields="JOB">
<sx:elementMap element="{JOB}">
<sx:onRecord>
<employee>
<sx:fieldAttributeMap field="EMPNO" attribute="employee-no"/>
<sx:fieldElementMap field="NAME" element="name"/>
</employee>
</sx:onRecord>
</sx:elementMap>
</sx:groupBy>
</employees>
</sx:recordMapping>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Executing the employees service with two job parameters, like this
servingxml -r resources.xml employees -o output/employees.xml employees
job=ANALYST job=CLERK
results in the following SQL statement being sent to the Oracle server:
SELECT EMPNO, ENAME AS NAME,JOB
FROM EMP
WHERE 1=1 AND JOB IN ('ANALYST','CLERK')
ORDER BY JOB,ENAME
But executing the employees service with no parameters results in a SQL select
statement that returns all employees for all job categories.
The example below illustrates how to prepare a resources script that will execute two queries to two different databases, where the second is joined with the first.
The desired output is shown below.
Figure 216. Output XML file deptEmployees.xml
<?xml version="1.0" encoding="utf-8"?>
<employees>
<department dept-no="10" dept-name="ACCOUNTING">
<employee employee-no="7782">
<name>CLARK</name>
</employee>
<employee employee-no="7839">
<name>KING</name>
</employee>
</department>
<department dept-no="30" dept-name="SALES">
<employee employee-no="7499">
<name>ALLEN</name>
</employee>
<employee employee-no="7698">
<name>BLAKE</name>
</employee>
</department>
</employees>
The following resources script does the transformation.
Figure 217. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="employees">
<sx:serialize>
<sx:content ref="employeesDoc"/>
</sx:serialize>
</sx:service>
<sx:recordContent id="employeesDoc">
<sx:recordStream>
<sx:sqlReader>
<sx:sqlConnectionPool ref="devPool"/>
<sx:sqlQuery recordType = "employee">
SELECT DEPTNO, DNAME FROM DEPT ORDER BY DNAME
</sx:sqlQuery>
</sx:sqlReader>
<!-- for each row from the previous reader, run this reader -->
<sx:sqlReader>
<sx:sqlConnectionPool ref="dev2Pool"/>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME,JOB, {DEPTNO} AS DEPTNO, '{DNAME}' AS DEPT FROM EMP WHERE DEPTNO = {DEPTNO}
ORDER BY ENAME
</sx:sqlQuery>
</sx:sqlReader>
</sx:recordStream>
<sx:recordMapping ref="employeesToXml"/>
</sx:recordContent>
<sx:recordMapping id="employeesToXml">
<employees>
<sx:groupBy fields="DEPT">
<department>
<sx:fieldAttributeMap field="DEPTNO" attribute="dept-no"/>
<sx:fieldAttributeMap field="DEPT" attribute="dept-name"/>
<sx:onRecord>
<employee>
<sx:fieldAttributeMap field="EMPNO" attribute="employee-no"/>
<sx:fieldElementMap field="NAME" element="name"/>
</employee>
</sx:onRecord>
</department>
</sx:groupBy>
</employees>
</sx:recordMapping>
<sx:sqlConnectionPool id="devPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
<sx:sqlConnectionPool id="dev2Pool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev2"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Executing the employees service, like this
servingxml -r resources-multipleReaders.xml employees -o output/deptEmployees.xml employees
results in the following two SQL statements being sent to the Oracle "dev2" database by the second reader:
SELECT EMPNO, ENAME AS NAME, 10 AS DEPTNO, 'ACCOUNTING' AS DEPT FROM EMP WHERE DEPTNO = 10 ORDER ENAME SELECT EMPNO, ENAME AS NAME, 30 AS DEPTNO, 'SALES' AS DEPT FROM EMP WHERE DEPTNO = 30 ORDER ENAME
This example shows how to prepare a resources script that will insert master/detail records into separate master and detail tables as a single transaction.
The input file is an XML file.
Figure 218. Input XML file masterDetail.xml
<?xml version="1.0" encoding="utf-8"?>
<masterDetail>
<master id="1">
<name>aName</name>
<detail id="11">
<date>2008-09-03</date>
<type>aType</type>
</detail>
<detail id="12">
<date>2008-09-06</date>
<type>aType</type>
</detail>
<detail id="13">
<date>2008-09-10</date>
<type>aType</type>
</detail>
</master>
<master id="2">
<name>anotherName</name>
<detail id="21">
<date>2008-10-03</date>
<type>aType</type>
</detail>
<detail id="22">
<date>2008-10-06</date>
<type>aType</type>
</detail>
<detail id="23">
<date>2008-10-07</date>
<type>anotherType</type>
</detail>
</master>
</masterDetail>
The following resources script performs the update.
Figure 219. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="loadMasterDetail">
<sx:recordStream>
<sx:subtreeRecordReader>
<sx:transform>
<sx:document/>
</sx:transform>
<sx:inverseRecordMapping ref="masterDetailMapping"/>
</sx:subtreeRecordReader>
<!-- You should validate the record with an sx:recordValidator here -->
<sx:sqlWriter ref="masterDetailWriter"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<!-- You can set up a record pipeline here to write bad records to a file or database table -->
</sx:discardHandler>
</sx:recordStream>
</sx:service>
<sx:inverseRecordMapping id="masterDetailMapping">
<sx:onSubtree path="/masterDetail/master">
<sx:flattenSubtree recordType="master">
<sx:subtreeFieldMap select="@id" field="masterId"/>
<sx:subtreeFieldMap select="name" field="name"/>
<sx:subtreeFieldMap match="detail" field="details">
<sx:flattenSubtree recordType="detail">
<sx:subtreeFieldMap select="@id" field="detailId"/>
<sx:subtreeFieldMap select="date" field="date"/>
<sx:subtreeFieldMap select="type" field="type"/>
</sx:flattenSubtree>
</sx:subtreeFieldMap>
</sx:flattenSubtree>
</sx:onSubtree>
</sx:inverseRecordMapping>
<sx:sqlWriter id="masterDetailWriter">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdate>
INSERT INTO master(master_id,name) VALUES({masterId},'{name}')
<sx:sqlUpdateDetail field="details">
<sx:sqlUpdate>
INSERT INTO detail(detail_id,detail_date,detail_type)
VALUES({detailId},to_date('{date}','yyyy-mm-dd'),'{type}')
</sx:sqlUpdate>
</sx:sqlUpdateDetail>
</sx:sqlUpdate>
</sx:sqlWriter>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
The inverse record mapping instructions in the script will produce a stream of composite records of type "master" that contain a repeating group field named "details" having subrecords of type "detail".
Executing the loadMasterDetail service, like this,
servingxml -i data/masterDetail.xml -r resources-masterDetail.xml loadMasterDetail
results in the following update statements being sent to the RDMS and committed to the database:
INSERT INTO master(master_id,name) VALUES(1,'aName')
INSERT INTO detail(detail_id,detail_date,detail_type)
VALUES(11,to_date('2008-09-03','yyyy-mm-dd'),'aType')
INSERT INTO detail(detail_id,detail_date,detail_type)
VALUES(12,to_date('2008-09-06','yyyy-mm-dd'),'aType')
INSERT INTO detail(detail_id,detail_date,detail_type)
VALUES(13,to_date('2008-09-10','yyyy-mm-dd'),'aType')
The second set of master/detail records fails, however,
because the last detail has a value for detail_type,
"anotherType", that
exceeds the length of the column, 10. The following error message
is written to the log:
SEVERE: ORA-12899: value too large for column "SCOTT"."DETAIL"."DETAIL_TYPE" (actual: 11, maximum: 10)
This example shows how to prepare a resources script that will read rows from one database table and write them to another.
The following resources script does the transformation.
Figure 220. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="loadEmployees">
<sx:recordStream>
<sx:sqlReader ref="employeesReader"/>
<msv:recordValidator>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
<xsd:element name="employee" type="EmployeeType"/>
<xsd:complexType name="EmployeeType">
<xsd:sequence>
<xsd:element name="EMPNO" type="xsd:integer"/>
<xsd:element name="NAME" type="xsd:string"/>
<xsd:element name="JOB" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:schema>
</msv:recordValidator>
<sx:sqlWriter ref="employeesWriter"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<!-- You can include a record pipeline here to write bad records to a file or database table -->
</sx:discardHandler>
</sx:recordStream>
</sx:service>
<sx:sqlWriter id="employeesWriter">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdate>
INSERT INTO EMP_HISTORY(EMPNO, ENAME) VALUES({EMPNO},'{NAME}')
</sx:sqlUpdate>
</sx:sqlWriter>
<sx:sqlReader id="employeesReader">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
<sx:choose>
<sx:when test="$jobList">
AND JOB IN ({$jobList})
</sx:when>
</sx:choose>
ORDER BY JOB,ENAME
</sx:sqlQuery>
</sx:sqlReader>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Executing the loadEmployees service with two job parameters, like this
servingxml -r resources-insert_employees.xml loadEmployees
job=ANALYST job=CLERK
results in the following SQL statement being sent to the Oracle server:
SELECT EMPNO, ENAME AS NAME,JOB
FROM EMP
WHERE 1=1 AND JOB IN ('ANALYST','CLERK')
ORDER BY JOB,ENAME
These results are then written to the database table EMP_HISTORY.
This example shows how to prepare a resources script that will read rows from one database table and write them in batches to another.
The following resources script does the transformation.
Figure 221. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="loadEmployees">
<sx:recordStream>
<sx:sqlReader ref="employeesReader"/>
<msv:recordValidator>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
<xsd:element name="employee" type="EmployeeType"/>
<xsd:complexType name="EmployeeType">
<xsd:sequence>
<xsd:element name="EMPNO" type="xsd:integer"/>
<xsd:element name="NAME" type="xsd:string"/>
<xsd:element name="JOB" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:schema>
</msv:recordValidator>
<sx:sqlWriter ref="employeesWriter"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<!-- You can include a record pipeline here to write bad records to a file or database table -->
</sx:discardHandler>
</sx:recordStream>
</sx:service>
<sx:sqlBatchWriter id="employeesWriter" batchSize="4">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdate>
INSERT INTO EMP_HISTORY(EMPNO, ENAME) VALUES( {EMPNO}, '{NAME}')
</sx:sqlUpdate>
</sx:sqlBatchWriter>
<sx:sqlReader id="employeesReader">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
<sx:choose>
<sx:when test="$jobList">
AND JOB IN ({$jobList})
</sx:when>
</sx:choose>
ORDER BY JOB,ENAME
</sx:sqlQuery>
</sx:sqlReader>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Executing the loadEmployees service with two job parameters, like this
servingxml -r resources-batch_insert_employees.xml loadEmployees
job=ANALYST job=CLERK
results in the following SQL statement being sent to the Oracle server:
SELECT EMPNO, ENAME AS NAME,JOB
FROM EMP
WHERE 1=1 AND JOB IN ('ANALYST','CLERK')
ORDER BY JOB,ENAME
These results are then written in batches of 4 to the database table EMP_HISTORY.
This example shows how to prepare a resources script that will read rows from one database table and write them using a prepared SQL statement to another.
The following resources script does the transformation.
Figure 222. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="loadEmployees">
<sx:recordStream>
<sx:sqlReader ref="employeesReader"/>
<msv:recordValidator>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
<xs:element name="employee" type="EmployeeType"/>
<xs:complexType name="EmployeeType">
<xs:sequence>
<xs:element name="EMPNO" type="xs:integer"/>
<xs:element name="NAME" type="xs:string"/>
<xs:element name="JOB" type="xs:string"/>
<xs:element name="MGR" type="xs:integer"/>
<xs:element name="HIREDATE" type="xs:date"/>
<xs:element name="SAL" type="xs:decimal"/>
<xs:element name="COMM" type="xs:decimal" minOccurs="0"/>
<xs:element name="DEPTNO" type="xs:integer"/>
</xs:sequence>
</xs:complexType>
</xs:schema>
</msv:recordValidator>
<sx:sqlWriter ref="employeesWriter"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<!-- You can include a record pipeline here to write bad records to a database table -->
<sx:sqlWriter ref="discardWriter"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
</sx:discardHandler>
</sx:discardHandler>
</sx:recordStream>
</sx:service>
<sx:sqlWriter id="employeesWriter" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdate>
<!-- Because the SQL insert executes a prepared statement, values must be
associated with appropriate types. In this example, because these values are
coming from a SQL query source, they will already be associated with SQL types,
and no further action is really necessary. In the case of other data sources, however,
values by default will be associated with string types, and may need
to be recast in parameter assignments, as shown below.
-->
<sx:parameter name="employee-no" value="{EMPNO}" type="xs:long"/>
<sx:parameter name="mgr" value="{MGR}" type="xs:long"/>
<sx:parameter name="hiredate" value="{HIREDATE}" type="xs:date"/>
<sx:parameter name="salary" value="{SAL}" type="xs:decimal"/>
<sx:parameter name="commission" value="{COMM}" type="xs:decimal"/>
<sx:sqlPrepare>
INSERT INTO EMP_HISTORY(EMPNO, ENAME, JOB, MGR, HIREDATE, SAL, COMM)
VALUES({$employee-no},{NAME},{JOB}, {$mgr}, {$hiredate}, {$salary}, {$commission})
</sx:sqlPrepare>
</sx:sqlUpdate>
</sx:sqlWriter>
<sx:sqlWriter id="discardWriter" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdate>
<sx:parameter name="employee-no" value="{EMPNO}" type="xs:long"/>
<sx:parameter name="mgr" value="{MGR}" type="xs:long"/>
<sx:parameter name="hiredate" value="{HIREDATE}" type="xs:date"/>
<sx:parameter name="salary" value="{SAL}" type="xs:decimal"/>
<sx:parameter name="commission" value="{COMM}" type="xs:decimal"/>
<sx:sqlPrepare>
INSERT INTO EMP_DISCARD(MESSAGE, EMPNO, ENAME, JOB, MGR, HIREDATE, SAL,
COMM)
VALUES({$message}, {$employee-no}, {NAME}, {JOB}, {$mgr}, {$hiredate},
{$salary}, {$commission})
</sx:sqlPrepare>
</sx:sqlUpdate>
</sx:sqlWriter>
<sx:sqlReader id="employeesReader">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO FROM EMP WHERE 1=1
<sx:choose>
<sx:when test="$jobList">
AND JOB IN ({$jobList})
</sx:when>
</sx:choose>
ORDER BY JOB,ENAME
</sx:sqlQuery>
</sx:sqlReader>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Executing the loadEmployees service with two job parameters, like this
servingxml -r resources-prepare_insert_employees.xml loadEmployees
job=SALESMAN job=MANAGER
results in the following SQL statement being sent to the Oracle server:
SELECT EMPNO, ENAME AS NAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO
FROM EMP
WHERE 1=1 AND JOB IN ('SALESMAN','MANAGER')
ORDER BY JOB,ENAME
These results are then written in batches of 4 to the database table EMP_HISTORY.
This example shows how to prepare a resources script that will read rows from one database table and write them in batches, using a prepared SQL statement, to another.
The following resources script does the transformation.
Figure 223. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="loadEmployees">
<sx:recordStream>
<sx:sqlReader ref="employeesReader"/>
<msv:recordValidator>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
<xs:element name="employee" type="EmployeeType"/>
<xs:complexType name="EmployeeType">
<xs:sequence>
<xs:element name="EMPNO" type="xs:integer"/>
<xs:element name="NAME" type="xs:string"/>
<xs:element name="JOB" type="xs:string"/>
<xs:element name="MGR" type="xs:integer"/>
<xs:element name="HIREDATE" type="xs:date"/>
<xs:element name="SAL" type="xs:decimal"/>
<xs:element name="COMM" type="xs:decimal" minOccurs="0"/>
<xs:element name="DEPTNO" type="xs:integer"/>
</xs:sequence>
</xs:complexType>
</xs:schema>
</msv:recordValidator>
<sx:sqlWriter ref="employeesWriter"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<!-- You can include a record pipeline here to write bad records to a file or database table -->
</sx:discardHandler>
</sx:recordStream>
</sx:service>
<sx:sqlBatchWriter id="employeesWriter" xmlns:xs="http://www.w3.org/2001/XMLSchema" batchSize="4">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdate>
<!-- Because the SQL insert executes a prepared statement, values must be
associated with appropriate types. In this example, because these values are
coming from a SQL query source, they will already be associated with SQL types,
and no further action is really necessary. In the case of other data sources, however,
values by default will be associated with string types, and may need
to be recast in parameter assignments, as shown below.
-->
<sx:parameter name="employee-no" value="{EMPNO}" type="xs:long"/>
<sx:parameter name="mgr" value="{MGR}" type="xs:long"/>
<sx:parameter name="hiredate" value="{HIREDATE}" type="xs:date"/>
<sx:parameter name="salary" value="{SAL}" type="xs:decimal"/>
<sx:parameter name="commission" value="{COMM}" type="xs:decimal"/>
<sx:sqlPrepare>
INSERT INTO EMP_HISTORY(EMPNO, ENAME, JOB, MGR, HIREDATE, SAL, COMM)
VALUES({$employee-no},{NAME},{JOB}, {$mgr}, {$hiredate}, {$salary}, {$commission})
</sx:sqlPrepare>
</sx:sqlUpdate>
</sx:sqlBatchWriter>
<sx:sqlReader id="employeesReader">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO FROM EMP WHERE 1=1
<sx:choose>
<sx:when test="$jobList">
AND JOB IN ({$jobList})
</sx:when>
</sx:choose>
ORDER BY JOB,ENAME
</sx:sqlQuery>
</sx:sqlReader>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Executing the loadEmployees service with two job parameters, like this
servingxml -r resources-batch_prepare_insert_employees.xml loadEmployees
job=SALESMAN job=MANAGER
results in the following SQL statement being sent to the Oracle server:
SELECT EMPNO, ENAME AS NAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO
FROM EMP
WHERE 1=1 AND JOB IN ('SALESMAN','MANAGER')
ORDER BY JOB,ENAME
These results are then written in batches of 4 to the database table EMP_HISTORY, using prepared SQL statements.
The example below illustrates how to prepare a resources script that will change an employee job categorization using command line parameters.
Figure 224. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="updateEmployee">
<sx:recordStream>
<sx:parameterReader recordType="employee"/>
<sx:sqlWriter>
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdate>
UPDATE EMP SET JOB = '{$job}' WHERE EMPNO='{$empNo}'
</sx:sqlUpdate>
</sx:sqlWriter>
</sx:recordStream>
</sx:service>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
You can run this example on the command line by entering
servingxml -r resources.xml -o updateEmployee
empNo=7499 job=ANALYST
This example shows how to prepare a resources script that will read rows from one database table and write them to another, inserting if the record does not already exist, updating otherwise.
The following resources script does the transformation.
Figure 225. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core"
xmlns:msv="http://www.servingxml.com/extensions/msv">
<sx:service id="loadEmployees">
<sx:recordStream>
<sx:sqlReader ref="employeesReader"/>
<msv:recordValidator>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- This element's name matches the value of the name attribute in the sx:sqlQuery element. -->
<xs:element name="employee" type="EmployeeType"/>
<xs:complexType name="EmployeeType">
<xs:sequence>
<xs:element name="EMPNO" type="xs:integer"/>
<xs:element name="NAME" type="xs:string"/>
<xs:element name="JOB" type="xs:string"/>
</xs:sequence>
</xs:complexType>
</xs:schema>
</msv:recordValidator>
<sx:sqlWriter ref="employeesWriter"/>
<sx:discardHandler>
<sx:log message="{$sx:message}"/>
<!-- You can include a record pipeline here to write bad records to a file or database table -->
</sx:discardHandler>
</sx:recordStream>
</sx:service>
<sx:sqlWriter id="employeesWriter">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdateChoice>
<sx:sqlQuery>
SELECT EMPNO FROM EMP_HISTORY WHERE EMPNO = {EMPNO}
</sx:sqlQuery>
<sx:recordNotFound>
<sx:sqlUpdate>
INSERT INTO EMP_HISTORY(EMPNO, ENAME) VALUES({EMPNO},'{NAME}')
</sx:sqlUpdate>
</sx:recordNotFound>
<sx:recordFound>
<sx:sqlUpdate>
UPDATE EMP_HISTORY SET ENAME = '{NAME}' WHERE EMPNO = {EMPNO}
</sx:sqlUpdate>
</sx:recordFound>
</sx:sqlUpdateChoice>
</sx:sqlWriter>
<sx:sqlReader id="employeesReader">
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
<sx:choose>
<sx:when test="$jobList">
AND JOB IN ({$jobList})
</sx:when>
</sx:choose>
ORDER BY JOB,ENAME
</sx:sqlQuery>
</sx:sqlReader>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Execute the loadEmployees service with two job parameters, like this
servingxml -r resources-insertUpdate.xml loadEmployees
job=ANALYST job=CLERK
This example shows how to prepare a resources script that will read rows from an RDBMS, write them to a database table as they pass through the pipe, and finally produce an XML file.
The desired XML file is shown below.
Figure 226. Output XML file employees.xml
<?xml version="1.0" encoding="utf-8"?>
<employees>
<ANALYST>
<employee employee-no="7902">
<name>FORD</name>
</employee>
<employee employee-no="7788">
<name>SCOTT</name>
</employee>
</ANALYST>
<CLERK>
<employee employee-no="7876">
<name>ADAMS</name>
</employee>
<employee employee-no="7900">
<name>JAMES</name>
</employee>
<employee employee-no="7934">
<name>MILLER</name>
</employee>
<employee employee-no="7369">
<name>SMITH</name>
</employee>
</CLERK>
</employees>
The following resources script does the transformation.
Figure 227. Resources script
<sx:resources xmlns:sx="http://www.servingxml.com/core">
<sx:service id="employees">
<sx:serialize>
<sx:content ref="employeesDoc"/>
</sx:serialize>
</sx:service>
<sx:recordContent id="employeesDoc">
<sx:sqlReader>
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:parameter name="jobList">
<sx:toString value="{$job}" separator=",">
<sx:quoteSymbol character="'" escapeCharacter="'"/>
</sx:toString>
</sx:parameter>
<sx:sqlQuery recordType = "employee">
SELECT EMPNO, ENAME AS NAME,JOB FROM EMP WHERE 1=1
<sx:choose>
<sx:when test="$jobList">
AND JOB IN ({$jobList})
</sx:when>
</sx:choose>
ORDER BY JOB,ENAME
</sx:sqlQuery>
</sx:sqlReader>
<sx:recordTee>
<sx:sqlWriter>
<sx:sqlConnectionPool ref="jdbcPool"/>
<sx:sqlUpdate>
INSERT INTO EMP_LOG(EMPNO, ENAME) VALUES({EMPNO},'{NAME}')
</sx:sqlUpdate>
</sx:sqlWriter>
</sx:recordTee>
<sx:recordMapping ref="employeesToXml"/>
</sx:recordContent>
<sx:recordMapping id="employeesToXml">
<employees>
<sx:groupBy fields="JOB">
<sx:elementMap element="{JOB}">
<sx:onRecord>
<employee>
<sx:fieldAttributeMap field="EMPNO" attribute="employee-no"/>
<sx:fieldElementMap field="NAME" element="name"/>
</employee>
</sx:onRecord>
</sx:elementMap>
</sx:groupBy>
</employees>
</sx:recordMapping>
<sx:sqlConnectionPool id="jdbcPool"
driver="oracle.jdbc.driver.OracleDriver"
databaseUrl="jdbc:oracle:thin:@127.0.0.1:1521:dev"
user="scott"
password="spring"
minConnections="2"
testStatement="SELECT * FROM DUAL"/>
</sx:resources>
Executing the employees service with two job parameters, like this
servingxml -r resources-employeesTee.xml employees -o output/employees.xml employees
job=ANALYST job=CLERK
results in the following SQL statement being sent to the Oracle server:
SELECT EMPNO, ENAME AS NAME,JOB
FROM EMP
WHERE 1=1 AND JOB IN ('ANALYST','CLERK')
ORDER BY JOB,ENAME
Each EMPNO and ENAME pair is written to the database table EMP_LOG, and all rows are
ultimately mapped to the XML file employees.xml.