Regular Expressions: Still Two Problems?

You sometimes stumble upon regular expressions which are difficult to understand, e.g. what exactly means:

?:(?!(foo1|foo2)).)*

The most common problem is to get the structure out of deeply nested regular expressions.  
Using the online service http://www.regexper.com/ makes it much easier to see the structure:

Now you see that this regular expression matches all content beside "foo1" or "foo2" using the pattern explained here: http://trent-intovalue.blogspot.de/2010/01/negating-regexes.html

Using the example taken from Regular Expressions: Now You Have Two Problems:

<\/?p>|<br\s?\/?>|<\/?b>|<\/?strong>|<\/?i>|<\/?em>|   <\/?s>|<\/?strike>|<\/?blockquote>|<\/?sub>|<\/?super>|   <\/?h(1|2|3)>|<\/?pre>|<hr\s?\/?>|<\/?code>|<\/?ul>|   <\/?ol>|<\/?li>|<\/a>|<a[^>]+>|<img[^>]+\/?>

Try it out - you probably see what the author try to achieve.
Note: you have to use regular expressions in JavaScript style to use the service.

Search and Replace on multiple files

Search & Replace is a common task in data processing environments. You cannot avoid to build process your data to replace or add a word, syntax or even multiple lines of text in several different resources.

If the task can be fully automated, means there is a unique algorithm to transform a resource A to A' based on the content of A than you will look for available methods and tools supporting you to do this kind of operation.

Methods

Regular expressions are very powerful rules to express not only finding common pattern in text based resources but also a good foundation to replace or extend existing content.
Compared to simple phrase based pattern most imaginable rules can be expressed and used as a source for the required transformation.
But regular expressions come with high cost of complexity. It is very likely to defines rules which results in "false positives", means matches that you didn't want to match.

Tools

Doing Search & Replace in the file system on multiple resources (files) is easy for IT people using linux tools like grep, ....
On Windows you also can install those tools and make them a powerful foundation for those kind of operations (see http://gnuwin32.sourceforge.net/packages/grep.htm).

TextCrawler

But not all people like to become an IT expert for simple replacing the term "foo" with "delicious" . On Windows you can use TextCrawler for this. One of the best UI based tools I'm aware of.

It provides

• simple phrase based operation "Replace phrase A with B" on multiplier files
• more complex regular expression based operation 
• and in addition a fuzzy search operation for more advanced search operations

It also supports the use of Unicode characters to search and replace and the processing of files encoded in Unicode (utf-8, utf-16).

To avoid false positives you can

• preview the hits before actual performing the replace operation
• use a dedicated regular expression tester to see what exactly match and what will replace

 Search and Replace is something you have to consider harmful but in case you have to do it on a Windows Desktop using this tool is something I can recommend.

Negate expressions using regex

a not so rare question using regex is "match all strings doesn't contain word foo". that isn't something regex is made for. by the way searching the web shows that the expression

(?:(?!REGEX).)*

where the expression REGEX must be replace with expression must be negated, e.g. (?:(?!(foo1|foo2)).)* returns true for all strings doesn't contain foo1 OR foo2.

more details and background can be found here: http://www.perlmonks.org/?node_id=588315#588368

identify content: powerful but tricky

regular expression and xpath are two approaches to identify a matching subset of content within a given amount of content for further usage. the first one based on plain text the second one based on xml.

both are powerful but if you not using it on a day by day basis are tricky and error prone. error prone means that you have to avoid all edge cases where a given expression shouldn't match but it does or the other way around where a given expression should match but it doesn't.

if you work in XML related world you often need both approaches e.g. using XSLT or XQuery.

there are few commercial IDE's which helping out to develop the right expression for the required use case but there are two which i'm often use dedicated to help you dealing either with regular expressions or xpath

SketchPath

powerful standalone tool to create and verify xpath expression. support for xpath 2.0 and most features you ask for in this context. this tool is based on well-known Saxon for xpath 2.0 and .net subsystem for xpath 1.0

home: http://pgfearo.googlepages.com/

Regulator

powerful standalone tool to create and verify regular expression. with buildin regex analyser and on-the-fly validation, code generation (for .net and vb only), ....

download: http://downloads.sourceforge.net/regulator/Regulator20Bin.zip?modtime=1189740236&big_mirror=0