I'm writing a FireMonkey application which must run in Windows as well in OSX. Until now I've used MSXML for my Windows apps, but of course this library is not compatible with OSX. So the question is, does Delphi XE2 include any XML library compatible with OSX, or is there a third-party XML library compatible with OSX?
In unit Xml.XMLDoc; you have a TXMLDocument;
If you choose DOMVendor := ADOM XML v4, it's available for Windows and OSX.
It's directly available from the Tool palette.
I'm currently starting to look into using Scala's XML support for business critical processes. In that regard I would like to know what the current state of the standard XML library in Scala is.
I have read some "old" articles relating to Scala 2.7 and the 2.8 trunk, where it was stated that the XML handling would be threadsafe for scala 2.8, and that there was several bugs in version 2.7.x. Some critique has also been giving on the XML class hierachy, but I guess that's a matter of taste.
So I would like if somebody could answer the question, what is the current state of the Scala XML library for version 2.9.x?
Thanks in advance.
Let me answer this way:
Or, putting it into words, many people are not satisfied. After all, people are writing full blown alternatives instead of trying to "fix" the library. And, speaking of fixing, I had a fix which turned an operation from O(n^2) to O(n) submitted for so long that, when someone opened the very same issue again, I didn't even remember having opened it before.
Mind you, Lift uses standard library XML, and, as far as I know, so do most of the other web frameworks (I suspect Play doesn't), so it's not like it's unusable.
I have an error "Out of memory" while parsing large (100 Mb) XML file
use strict;
use warnings;
use XML::Twig;
my $twig=XML::Twig->new();
my $data = XML::Twig->new
->parsefile("divisionhouserooms-v3.xml")
->simplify( keyattr => []);
my @good_division_numbers = qw( 30 31 32 35 38 );
foreach my $property ( @{ $data->{DivisionHouseRoom}}) {
my $house_code = $property->{HouseCode};
print $house_code, "\n";
my $amount_of_bedrooms = 0;
foreach my $division ( @{ $property->{Divisions}->{Division} } ) {
next unless grep { $_ eq $division->{DivisionNumber} } @good_division_numbers;
$amount_of_bedrooms += $division->{DivisionQuantity};
}
open my $fh, ">>", "Result.csv" or die $!;
print $fh join("\t", $house_code, $amount_of_bedrooms), "\n";
close $fh;
}
What i can do to fix this error issue?
Handling large XML files that don't fit in memory is something that XML::Twig advertises:
One of the strengths of XML::Twig is that it let you work with files that do not fit in memory (BTW storing an XML document in memory as a tree is quite memory-expensive, the expansion factor being often around 10).
To do this you can define handlers, that will be called once a specific element has been completely parsed. In these handlers you can access the element and process it as you see fit (...)
The code posted in the question isn't making use of the strength of XML::Twig at all (using the simplify method doesn't make it much better than XML::Simple).
What's missing from the code are the 'twig_handlers' or 'twig_roots', which essentially cause the parser to focus on relevant portions of the XML document memory-efficiently.
It's difficult to say without seeing the XML whether processing the document chunk-by-chunk or just selected parts is the way to go, but either one should solve this issue.
So the code should look something like the following (chunk-by-chunk demo):
use strict;
use warnings;
use XML::Twig;
use List::Util 'sum'; # To make life easier
use Data::Dump 'dump'; # To see what's going on
my %bedrooms; # Data structure to store the wanted info
my $xml = XML::Twig->new (
twig_roots => {
DivisionHouseRoom => \&count_bedrooms,
}
);
$xml->parsefile( 'divisionhouserooms-v3.xml');
sub count_bedrooms {
my ( $twig, $element ) = @_;
my @divParents = $element->children( 'Divisions' );
my $id = $element->first_child_text( 'HouseCode' );
for my $divParent ( @divParents ) {
my @divisions = $divParent->children( 'Division' );
my $total = sum map { $_->text } @divisions;
$bedrooms{$id} = $total;
}
$element->purge; # Free up memory
}
dump \%bedrooms;
I have a legacy HTTP/XML service that I need to interact with for various features in my application.
I have to create a wide range of request messages for the service, so to avoid a lot of magic strings littered around the code, I've decided to create xml XElement fragments to create a rudimentary DSL.
For example.
Instead of...
new XElement("root",
new XElement("request",
new XElement("messageData", ...)));
I'm intended to use:
Root( Request( MessageData(...) ) );
With Root, Request and MessageData (of course, these are for illustrative purposes) defined as static methods which all do something similar to:
private static XElement Root(params object[] content)
{
return new XElement("root", content);
}
This gives me a pseudo functional composition style, which I like for this sort of task.
My ultimate question is really one of sanity / best practices, so it's probably too subjective, however I'd appreciate the opportunity to get some feedback regardless.
I'm intending to move these private methods over to public static class, so that they are easily accessible for any class that wants to compose a message for the service.
I'm also intending to have different features of the service have their messages created by specific message building classes, for improved maintainability.
Is this a good way to implement this simple DSL, or am I missing some special sauce that will let me do this better?
The thing that leads me to doubt, is the fact that as soon as I move these methods to another class I increase the length of these method calls (of course I do still retain the initial goal of removing the large volume magic strings.) Should I be more concerned about the size (loc) of the DSL language class, than I am about syntax brevity?
Note that in this instance the remote service poorly implemented, and doesn't conform to any general messaging standards, e.g. WSDL, SOAP, XML/RPC, WCF etc.
In those cases, it would obviously not be wise to create hand built messages.
In the rare cases where you do have to deal with a service like the one in question here, and it cannot be re-engineered for whatever reason, the answers below provide some possible ways of dealing with the situation.
Have you noticed that all the System.Linq.Xml classes are not sealed?
public class Root : XElement
{
public Request Request { get { return this.Element("Request") as Request; } }
public Response Response { get { return this.Element("Response") as Response; } }
public bool IsRequest { get { return Request != null; } }
/// <summary>
/// Initializes a new instance of the <see cref="Root"/> class.
/// </summary>
public Root(RootChild child) : base("Root", child) { }
}
public abstract class RootChild : XElement { }
public class Request : RootChild { }
public class Response : RootChild { }
var doc = new Root(new Request());
Remember this won't work for 'reading' scenarios, you will only have the strong-typed graph from the XML that your application creates via code.
I tried parsing this huge XML document using XML minidom. While it worked fine on a sample file, it choked the system when trying to process the real file (about 400 MB).
I tried adapting code (it processes data in a streaming fashion rather than in-memory load at once) from codereview for my xml file, I am having trouble isolating the datasets due to the nested nature of the elements. I worked on simple XML files before but not on a memory intensive task like this.
Is this the right approach? How do I associate the Inventory and Publisher IDs to each book? That is how I am planning to relate the 2 tables eventually.
Any feedback is much appreciated.
book.xml
<BookDatabase>
<BookHeader>
<Name>BookData</Name>
<BookUniverse>All</BookUniverse>
<AsOfDate>2010-05-02</AsOfDate>
<Version>1.1</Version>
</BookHeader>
<InventoryBody>
<Inventory ID="12">
<PublisherClass ID="34">
<Publisher>
<PublisherDetails>
<Name>Microsoft Press</Name>
<Type>Tech</Type>
<ID>7462</ID>
</PublisherDetails>
</Publisher>
</PublisherClass>
<BookList>
<Listing>
<BookListSummary>
<Date>2009-01-30</Date>
</BookListSummary>
<Book>
<BookDetail ID="67">
<BookName>Code Complete 2</BookName>
<Author>Steve McConnell</Author>
<Pages>960</Pages>
<ISBN>0735619670</ISBN>
</BookDetail>
<BookDetail ID="78">
<BookName>Application Architecture Guide 2</BookName>
<Author>Microsoft Team</Author>
<Pages>496</Pages>
<ISBN>073562710X</ISBN>
</BookDetail>
</Book>
</Listing>
</BookList>
</Inventory>
<Inventory ID="64">
<PublisherClass ID="154">
<Publisher>
<PublisherDetails>
<Name>O'Reilly Media</Name>
<Type>Tech</Type>
<ID>7484</ID>
</PublisherDetails>
</Publisher>
</PublisherClass>
<BookList>
<Listing>
<BookListSummary>
<Date>2009-03-30</Date>
</BookListSummary>
<Book>
<BookDetail ID="98">
<BookName>Head First Design Patterns</BookName>
<Author>Kathy Sierra</Author>
<Pages>688</Pages>
<ISBN>0596007124</ISBN>
</BookDetail>
</Book>
</Listing>
</BookList>
</Inventory>
</InventoryBody>
</BookDatabase>
Python Code :
import sys
import os
#import MySQLdb
from lxml import etree
CATEGORIES = set(['BookHeader', 'Inventory', 'PublisherClass', 'PublisherDetails', 'BookDetail'])
SKIP_CATEGORIES = set(['BookHeader'])
DATA_ITEMS = ["Name", "Type", "ID", "BookName", "Author", "Pages", "ISBN"]
def clear_element(element):
element.clear()
while element.getprevious() is not None:
del element.getparent()[0]
def extract_book_elements(context):
for event, element in context:
if element.tag in CATEGORIES:
yield element
clear_element(element)
def fast_iter2(context):
for bookCounter, element in enumerate(extract_book_elements(context)):
books = [book.text for book in element.findall("BookDetail")]
bookdetail = {
'element' : element.tag,
'ID' : element.get('ID')
}
for data_item in DATA_ITEMS:
data = element.find(data_item)
if data is not None:
bookdetail[data_item] = data
if bookdetail['element'] not in SKIP_CATEGORIES:
#populate_database(bookdetail, books, cursor)
print bookdetail, books
print "========>", bookCounter , "<======="
def main():
#cursor = connectToDatabase()
#cursor.execute("""SET NAMES utf8""")
context = etree.iterparse("book.xml", events=("start", "end"))
#fast_iter(context, cursor)
fast_iter2(context)
#cursor.close()
if __name__ == '__main__':
main()
Python Output :
$ python lxmletree_book.py
========> 0 <=======
========> 1 <=======
{'ID': '12', 'element': 'Inventory'} []
========> 2 <=======
{'ID': '34', 'element': 'PublisherClass'} []
========> 3 <=======
{'Name': <Element Name at 0x105140af0>, 'Type': <Element Type at 0x105140b40>, 'ID': <Element ID at 0x105140b90>, 'element': 'PublisherDetails'} []
========> 4 <=======
{'ID': None, 'element': 'PublisherDetails'} []
========> 5 <=======
{'ID': None, 'element': 'PublisherClass'} []
========> 6 <=======
{'ISBN': <Element ISBN at 0x105140eb0>, 'Name': <Element Name at 0x105140dc0>, 'Author': <Element Author at 0x105140e10>, 'ID': '67', 'element': 'BookDetail', 'Pages': <Element Pages at 0x105140e60>} []
========> 7 <=======
{'ID': None, 'element': 'BookDetail'} []
========> 8 <=======
{'ISBN': <Element ISBN at 0x1051460a0>, 'Name': <Element Name at 0x105140f50>, 'Author': <Element Author at 0x105140fa0>, 'ID': '78', 'element': 'BookDetail', 'Pages': <Element Pages at 0x105146050>} []
========> 9 <=======
{'ID': None, 'element': 'BookDetail'} []
========> 10 <=======
{'ID': None, 'element': 'Inventory'} []
========> 11 <=======
{'ID': '64', 'element': 'Inventory'} []
========> 12 <=======
{'ID': '154', 'element': 'PublisherClass'} []
========> 13 <=======
{'Name': <Element Name at 0x105146230>, 'Type': <Element Type at 0x105146280>, 'ID': <Element ID at 0x1051462d0>, 'element': 'PublisherDetails'} []
========> 14 <=======
{'ID': None, 'element': 'PublisherDetails'} []
========> 15 <=======
{'ID': None, 'element': 'PublisherClass'} []
========> 16 <=======
{'ISBN': <Element ISBN at 0x1051465f0>, 'Name': <Element Name at 0x105146500>, 'Author': <Element Author at 0x105146550>, 'ID': '98', 'element': 'BookDetail', 'Pages': <Element Pages at 0x1051465a0>} []
========> 17 <=======
{'ID': None, 'element': 'BookDetail'} []
========> 18 <=======
{'ID': None, 'element': 'Inventory'} []
========> 19 <=======
Desired Output (eventually stored in MySQL - for now a List in Python):
Publishers
InventoryID PublisherClassID Name Type ID
12 34 Microsoft Press Tech 7462
64 154 O'Reilly Media Tech 7484
Books
PublisherID BookDetailID Name Author Pages ISBN
7462 67 Code Complete 2 Steve McConnell 960 0735619670
7462 78 Application Architecture Guide 2 Microsoft Team 496 073562710X
7484 98 Head First Design Patterns Kathy Sierra 688 0596007124
You might try something like this:
import MySQLdb
from lxml import etree
import config
def fast_iter(context, func, args=[], kwargs={}):
# http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
# Author: Liza Daly
for event, elem in context:
func(elem, *args, **kwargs)
elem.clear()
while elem.getprevious() is not None:
del elem.getparent()[0]
del context
def extract_paper_elements(element,cursor):
pub={}
pub['InventoryID']=element.attrib['ID']
try:
pub['PublisherClassID']=element.xpath('PublisherClass/@ID')[0]
except IndexError:
pub['PublisherClassID']=None
pub['PublisherClassID']=element.xpath('PublisherClass/@ID')[0]
for key in ('Name','Type','ID'):
try:
pub[key]=element.xpath(
'PublisherClass/Publisher/PublisherDetails/{k}/text()'.format(k=key))[0]
except IndexError:
pub[key]=None
sql='''INSERT INTO Publishers (InventoryID, PublisherClassID, Name, Type, ID)
VALUES (%s, %s, %s, %s, %s)
'''
args=[pub.get(key) for key in
('InventoryID', 'PublisherClassID', 'Name', 'Type', 'ID')]
print(args)
# cursor.execute(sql,args)
for bookdetail in element.xpath('descendant::BookList/Listing/Book/BookDetail'):
pub['BookDetailID']=bookdetail.attrib['ID']
for key in ('BookName', 'Author', 'Pages', 'ISBN'):
try:
pub[key]=bookdetail.xpath('{k}/text()'.format(k=key))[0]
except IndexError:
pub[key]=None
sql='''INSERT INTO Books
(PublisherID, BookDetailID, Name, Author, Pages, ISBN)
VALUES (%s, %s, %s, %s, %s, %s)
'''
args=[pub.get(key) for key in
('ID', 'BookDetailID', 'BookName', 'Author', 'Pages', 'ISBN')]
# cursor.execute(sql,args)
print(args)
def main():
context = etree.iterparse("book.xml", events=("end",), tag='Inventory')
connection=MySQLdb.connect(
host=config.HOST,user=config.USER,
passwd=config.PASS,db=config.MYDB)
cursor=connection.cursor()
fast_iter(context,extract_paper_elements,args=(cursor,))
cursor.close()
connection.commit()
connection.close()
if __name__ == '__main__':
main()
fast_iter2. The original fast_iter separates the
useful utility from the specific processing function
(extract_paper_elements). fast_iter2 mixes the two together
leaving you with no repeatable code.tag parameter in etree.iterparse("book.xml",
events=("end",), tag='Inventory') then your processing function
extract_paper_elements will only see Inventory elements.xpath method to burrow
down and scrape the desired data.args and kwargs parameters were added to fast_iter so cursor
can be passed to extract_paper_elements.I am using the svn log --xml response to look at the changes for a particular revision. But in case of an invalid revision or path, I want to just to get a blank but valid xml response. Is it possible to do that?
So instead of this:
<?xml version="1.0"?>
<log>
svn: Unable to find repository location for 'http://subversion.ny.jpmorgan.com /svn/repos/IM_RPS_CORE/rps_deploy_tools_content_test/branches/rol-201106-content-test' in revision 1556
I could get:
<?xml version="1.0"?>
<log>
</log>
Thanks!
No can do.
However, if you check the exit status code of your svn log command, you can handle the issue yourself. For example, in a Bash shell script:
if ! svn log --xml $url > $xmlOutput 2> /dev/null
then
cat > $xmlOutput <<EOF
<?xml version="1.0"?>
<log>
</log>
EOF
fi
I'm creating a very large XML file (700mb +) that process large amounts of data via batch. The program serves as an interface between a extremely large sybase database and an application. I currently have the xsd schema bound to classes. I need a way of being able to write the XML with restart logic in mind.
I.E. being able to know where I left off. Or in other words, if the program fails, I need to be able to see what the was last wrote to the XML file so it can pick up where I left off. Here's an exmaple.
<root>
<WorkSet>
<Work>
<Customer>
<Work>
<Customer>
<WorkSet>
<Work>
.....
<root>
Say the program fails after writing a write 'work' or 'workset' node. Is there a way to pick up where I left off processing? I'm trying to avoid reading the XML file back into memory due to the shear size of the XML file (Say it finishes 500mb of XML and fails).
Thanks for the help.
If you could split your data to independent WorkSet elements you can write them out one at a time with JAXB's fragment mode (when JAXB does not write the headers). Later simply concatenate the files and add the missing XML declaration, opening end closing tags.
It's is possible that you have to modify your generated classes for this. I mean adding @XmlRootElement to the WorkSet java class. If one WorkSet is still big for one step you can do this with Work too, but you have to generate somehow the missing tags.
I have a C# .NET 3.5 application where I would like to serialize a class containing a List<> to XML. My class looks like this:
[XmlRoot("Foo")]
class Foo
{
private List<Bar> bar_ = new List<Bar>();
private string something_ = "My String";
[XmlElement("Something")]
public string Something { get { return something_; } }
[XmlElement("Bar")]
public ICollection<Bar> Bars
{
get { return bar_; }
}
}
If I populate it like this:
Bar b1 = new Bar();
// populate b1 with interesting data
Bar b2 = new Bar();
// populate b2 with interesting data
Foo f = new Foo();
f.Bars.Add(b1);
f.Bars.Add(b2);
And then serialize it like this:
using (System.IO.TextWriter textWriter = new System.IO.StreamWriter(@"C:\foo.xml"))
{
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(typeof(Foo));
serializer.Serialize(textWriter, f);
}
I get a file that looks like this:
<Foo>
<Something>My String</Something>
</Foo>
But, what I want is XML that looks like this:
<Foo>
<Something>My String</Something>
<Bar>
<!-- Data from first Bar -->
</Bar>
<Bar>
<!-- Data from second Bar -->
</Bar>
</Foo>
What do I need to do to get the List<> to appear in the XML?
The XmlSerializer requires that serializable properties have a setter. Besides that, the XmlSerializer can not serialize interface properties. The following code will work:
[XmlElement("Bar")]
public List<Bar> Bars
{
get { return bar_; }
set { throw new NotSupportedException("This property 'Bars' cannot be set. This property is readonly."); }
}
If you don't like this solution (the exception is kinda ugly) then you could implement IXmlSerializable and write your own custom serialization.
Edit: Artur Mustafin is right, members that implement IEnumerable or ICollection don't need a setter, as is explained on this msdn page:
The XmlSerializer gives special treatment to classes that implement
IEnumerableorICollection. A class that implementsIEnumerablemust implement a publicAddmethod that takes a single parameter. The Add method's parameter must be of the same type as is returned from theCurrentproperty on the value returned fromGetEnumerator, or one of that type's bases. A class that implementsICollection(such as CollectionBase) in addition toIEnumerablemust have a publicItemindexed property (indexer in C#) that takes an integer, and it must have a publicCountproperty of type integer. The parameter to theAddmethod must be the same type as is returned from theItemproperty, or one of that type's bases. For classes that implementICollection, values to be serialized are retrieved from the indexedItemproperty, not by callingGetEnumerator.
First time posting and first time working on Android, so go easy if I am breaking any rules :)
Anyways, I just got into making an Android app and I'm trying to create a simple trivia game. I plan to have many questions (hopefully about 5000+ questions) made. No data manipulation made, just straight up reading the questions and presenting it to the user. I am now faced with the dilema in how to store the questions.
I have two choices:
1.) Bundle the questions with the app: Possibly store the information in SQLite. Originally, for demo purposes, I placed the questions in an XML file, but I quickly realized how inefficient it will be once the questions start piling up. First, I am concerned if opening up such a huge XML file would suck up Android's memory. Secondly, I am worried how large the app would be if it contains 5000+ questions. I read here about the pros of XML vs SQLite. In that example, the user has 70,000 entries so maybe my 5,000 questions would be enough?
OR
2.) Host the questions on a server: I believe the upside is the app wouldn't need to be bundled with a vast amount of questions and wouldn't need to worry about the logic of opening and assembling the questions. It would just hit a PHP page and depending on the parameters sent, the PHP page would return the questions in XML format. The downside is the user would need to be online (to retrieve the questions) in order to play the game and my server would need to be up and running 24/7.
Has anyone encountered this design issue of how and where to store vast amount of data in an Android app?
Thanks, any help would be much appreciated!
From a marketing perspective, I think the right approach is #1. Just looking at the top app list, it's filled with apps that work offline. I don't know why most people fail to mention this one important criteria when they talk about marketing apps... especially in Android, where lots of people don't have a data plan. Plus, a trivia app sounds like something someone would probably use during a commute in a train as opposed to something like Facebook Chat that they'd use when they're online.
From a technical perspective, storing 5000+ questions really won't take a huge amount of space. There's this app called "MyFitnessPal". It stores maybe over 30,000 foods in a SQlite database, with nutritional information. So don't overestimate how much space it will take.
The advantage of going with #2 is if the questions or answers change often, you might want to go with that approach.
I am trying to read a file to xml with the following code:
import scala.xml._
object HebrewToEnglishCityTranslator {
val data = XML.loadFile("cities_hebrew_utf.xml");
for(val entry <- data \\ "city") {
val hebrewName = (entry \\ "hebrew_name").text
val englishName = (entry \\ "english_name").text
println(hebrewName + "=" + englishName) }
However, my file is encoded in UTF-8 (hebrew chars) and XML encoding is val encoding = "ISO-8859-1"
what should I do?
You should use XML.load(reader: java.io.Reader), which allows you to specify the file encoding:
XML.load(new java.io.InputStreamReader(new java.io.FileInputStream("cities_hebrew_utf.xml"), "UTF-8"))
I can pass a variable as a multivalued parameter:
scala> <b/>
res26: scala.xml.Elem = <b></b>
scala> Elem(null,"a",Null,TopScope,res26)
res27: scala.xml.Elem = <a><b></b></a>
But I can't pass an XML literal as a multivalued parameter:
scala> Elem(null,"a",Null,TopScope,<b/>)
<console>:12: error: not found: value <
Elem(null,"a",Null,TopScope,<b/>)
But I can pass an XML literal as a simple parameter
scala> def bar(s:String,n:Elem) = s+n.toString
bar: (s: String, n: scala.xml.Elem)java.lang.String
scala> bar("super ", <a/>)
res30: java.lang.String = super <a></a>
?
Thanks
Adding a space before the XML element makes it work:
scala> Elem(null, "a", Null, TopScope, <b/>)
resN: scala.xml.Elem = <a><b></b></a>
From the Scala Language Specification, Section 1.5:
In order to allow literal inclusion of XML fragments, lexical analysis switches from Scala mode to XML mode when encountering an opening angle bracket ’<’ in the following circumstance: The ’<’ must be preceded either by whitespace, an opening parenthesis or an opening brace and immediately followed by a character starting an XML name
for example is the following xpath string valid:
/web:input_text[@value=null]
which means the element doest not has a value attribute.
This XPath returns web:input_text which don't have value attribute:
/web:input_text[not(@value)]
XPath /web:input_text[@value=null] selects web:input_text which have value of value attribute = null node.