Ruby and Natural Language Processing

Just a quick list of natural language processing resources for Ruby:

* MIT has a list of [AI-related Ruby extensions](http://web.media.mit.edu/~dustin/papers/ai_ruby_plugins/).
* Jason Adams, who does “opinion mining for a startup in Atlanta”, has a list of [NLP Resources for Ruby](http://mendicantbug.com/2009/09/13/nlp-resources-for-ruby/).
* Nick Sieger has a post on [RubyConf: Natural language generation and processing in Ruby](http://blog.nicksieger.com/articles/2006/10/22/rubyconf-natural-language-generation-and-processing-in-ruby).
* Finally, there are a couple of papers that mention Ruby and NLP: [“Trust Region Newton Method for Large-Scale Logistic Regression”](http://ntucsu.csie.ntu.edu.tw/~cjlin/papers/logistic.pdf) and [“Natural language question answering: the view from here”](http://www.loria.fr/~gardent/applicationsTAL/papers/jnle-qa.pdf).

Gollum is a Ruby Gem that creates a small wiki-server that runs locally. [Check it out on GitHub(https://github.com/github/gollum/).

And while, yes, I am setting out to learn Python, I am finding Ruby in general so much easier for me to grok. Gems et alum.

An Easy Rails Server on Your Mac

The good folks at 37 Signals have done it again: produced not only a nice piece of software but also open sourced it for everyone else to try, use, and modify:

Pow is a clean, self-contained server that runs as your user without root privileges. There are no Apache extensions to compile, no gems to install. A single command installs and upgrades Pow automatically. And your system files are left pristine and untouched.

Pow is available here. (Here’s a direct link to the GitHub repository.)

{less} extends CSS

While CSS3 continues to evolve in interesting and powerful ways, it still clings to some conventions that make it more like a set of markup conventions and less like a programming language. That is, while you can describe a lot of things in CSS, you typically have to do so verbosely and somewhat redundantly. You cannot, like most modern programming languages, compress your efforts, pass variables, create something like objects.

For those who either run Ruby or, like me, are confined to admiring it and continue to try to learn it, [`less`](http://lesscss.org/) is available for installation and use and what it does to, or for, CSS is nothing short of amazing.

For one thing, you can set up variables for things used site-wide that allow you to establish, as in the example below, all your colors in one place:

@brand_color: #4D926F;

#header {
color: @brand_color;
}

h2 {
color: @brand_color;
}

Or you can use something they call *mixins* which operate like some sort of powerful hybrid of variables and functions — note how you can pass an argument to what looks like a variable:

.rounded_corners (@radius: 5px) {
-moz-border-radius: @radius;
-webkit-border-radius: @radius;
border-radius: @radius;
}

#header {
.rounded_corners;
}

#footer {
.rounded_corners(10px);
}

Finally, for me at least, there is the opportunity to nest selector names in a way to make inheritance more obvious:

#header {
color: red;
a {
font-weight: bold;
text-decoration: none;
}
}

(Nothing kills me more than trying to figure out inheritance of an item that just won’t style for me because I’ve missed some piece of the CSS flow in my analysis.)

Check out [`less`](http://lesscss.org/) for more.

Ruby 1.9.1 Released

More news is available at [Ruby Inside](http://www.rubyinside.com/ruby-191-released-first-production-release-of-the-fastest-ruby-ever-1480.html). With all the changes, maybe it was a good thing I put learning Ruby on hold. Maybe?

Updating Gems

I wasn’t sure if I had ever updated Rails on my MBP, and so, as I begin developing my first real project, I thought it was time. A regular `gem update` didn’t work. I had to use `sudo`:

sudo gem update rails –include-dependencies

The same applied for `gem cleanup`, which deleted the following items — after asking me for confirmation:

Successfully uninstalled rails-1.2.6
Successfully uninstalled rake-0.7.3
Successfully uninstalled actionwebservice-1.2.3
Successfully uninstalled activerecord-1.15.3
Successfully uninstalled actionmailer-1.3.3
Successfully uninstalled actionpack-1.13.3
Successfully uninstalled activesupport-1.4.2

I now have to do the same on the iMac.

Oniguruma Regular Expressions

Version 5.3.0 2006/11/20

syntax: ONIG_SYNTAX_RUBY (default)

1. Syntax elements

\ escape (enable or disable meta character meaning)
| alternation
(…) group
[…] character class

2. Characters

\t horizontal tab (0x09)
\v vertical tab (0x0B)
\n newline (0x0A)
\r return (0x0D)
\b back space (0x08)
\f form feed (0x0C)
\a bell (0x07)
\e escape (0x1B)
\nnn octal char (encoded byte value)
\xHH hexadecimal char (encoded byte value)
\x{7HHHHHHH} wide hexadecimal char (character code point value)
\cx control char (character code point value)
\C-x control char (character code point value)
\M-x meta (x|0x80) (character code point value)
\M-\C-x meta control char (character code point value)

(* \b is effective in character class […] only)

3. Character types

. any character (except newline)

\w word character

Not Unicode:
alphanumeric, “_” and multibyte char.

Unicode:
General_Category — (Letter|Mark|Number|Connector_Punctuation)

\W non word char

\s whitespace char

Not Unicode:
\t, \n, \v, \f, \r, \x20

Unicode:
0009, 000A, 000B, 000C, 000D, 0085(NEL),
General_Category — Line_Separator
— Paragraph_Separator
— Space_Separator

\S non whitespace char

\d decimal digit char

Unicode: General_Category — Decimal_Number

\D non decimal digit char

\h hexadecimal digit char [0-9a-fA-F]

\H non hexadecimal digit char

Character Property

* \p{property-name}
* \p{^property-name} (negative)
* \P{property-name} (negative)

property-name:

+ works on all encodings
Alnum, Alpha, Blank, Cntrl, Digit, Graph, Lower,
Print, Punct, Space, Upper, XDigit, Word, ASCII,

+ works on EUC_JP, Shift_JIS
Hiragana, Katakana

+ works on UTF8, UTF16, UTF32
Any, Assigned, C, Cc, Cf, Cn, Co, Cs, L, Ll, Lm, Lo, Lt, Lu,
M, Mc, Me, Mn, N, Nd, Nl, No, P, Pc, Pd, Pe, Pf, Pi, Po, Ps,
S, Sc, Sk, Sm, So, Z, Zl, Zp, Zs,
Arabic, Armenian, Bengali, Bopomofo, Braille, Buginese,
Buhid, Canadian_Aboriginal, Cherokee, Common, Coptic,
Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian,
Glagolitic, Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul,
Hanunoo, Hebrew, Hiragana, Inherited, Kannada, Katakana,
Kharoshthi, Khmer, Lao, Latin, Limbu, Linear_B, Malayalam,
Mongolian, Myanmar, New_Tai_Lue, Ogham, Old_Italic, Old_Persian,
Oriya, Osmanya, Runic, Shavian, Sinhala, Syloti_Nagri, Syriac,
Tagalog, Tagbanwa, Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan,
Tifinagh, Ugaritic, Yi

4. Quantifier

greedy

? 1 or 0 times
* 0 or more times
+ 1 or more times
{n,m} at least n but not more than m times
{n,} at least n times
{,n} at least 0 but not more than n times ({0,n})
{n} n times

reluctant

?? 1 or 0 times
*? 0 or more times
+? 1 or more times
{n,m}? at least n but not more than m times
{n,}? at least n times
{,n}? at least 0 but not more than n times (== {0,n}?)

possessive (greedy and does not backtrack after repeated)

?+ 1 or 0 times
*+ 0 or more times
++ 1 or more times

({n,m}+, {n,}+, {n}+ are possessive op. in ONIG_SYNTAX_JAVA only)

ex. /a*+/ === /(?>a*)/

5. Anchors

^ beginning of the line
$ end of the line
\b word boundary
\B not word boundary
\A beginning of string
\Z end of string, or before newline at the end
\z end of string
\G matching start position

6. Character class

^… negative class (lowest precedence operator)
x-y range from x to y
[…] set (character class in character class)
..&&.. intersection (low precedence at the next of ^)

ex. [a-w&&[^c-g]z] ==> ([a-w] AND ([^c-g] OR z)) ==> [abh-w]

* If you want to use ‘[‘, ‘-‘, ‘]’ as a normal character
in a character class, you should escape these characters by ‘\’.

POSIX bracket ([:xxxxx:], negate [:^xxxxx:])

Not Unicode Case:

alnum alphabet or digit char
alpha alphabet
ascii code value: [0 – 127]
blank \t, \x20
cntrl
digit 0-9
graph include all of multibyte encoded characters
lower
print include all of multibyte encoded characters
punct
space \t, \n, \v, \f, \r, \x20
upper
xdigit 0-9, a-f, A-F
word alphanumeric, “_” and multibyte characters

Unicode Case:

alnum Letter | Mark | Decimal_Number
alpha Letter | Mark
ascii 0000 – 007F
blank Space_Separator | 0009
cntrl Control | Format | Unassigned | Private_Use | Surrogate
digit Decimal_Number
graph [[:^space:]] && ^Control && ^Unassigned && ^Surrogate
lower Lowercase_Letter
print [[:graph:]] | [[:space:]]
punct Connector_Punctuation | Dash_Punctuation | Close_Punctuation |
Final_Punctuation | Initial_Punctuation | Other_Punctuation |
Open_Punctuation
space Space_Separator | Line_Separator | Paragraph_Separator |
0009 | 000A | 000B | 000C | 000D | 0085
upper Uppercase_Letter
xdigit 0030 – 0039 | 0041 – 0046 | 0061 – 0066
(0-9, a-f, A-F)
word Letter | Mark | Decimal_Number | Connector_Punctuation

7. Extended groups

(?#…) comment

(?imx-imx) option on/off
i: ignore case
m: multi-line (dot(.) match newline)
x: extended form
(?imx-imx:subexp) option on/off for subexp

(?:subexp) not captured group
(subexp) captured group

(?=subexp) look-ahead
(?!subexp) negative look-ahead
(?<=subexp) look-behind (?subexp) atomic group
don’t backtrack in subexp.

(?subexp) define named group
(All characters of the name must be a word character.)

Not only a name but a number is assigned like a captured
group.

Assigning the same name as two or more subexps is allowed.
In this case, a subexp call can not be performed although
the back reference is possible.

8. Back reference

\n back reference by group number (n >= 1)
\k back reference by group name

In the back reference by the multiplex definition name,
a subexp with a large number is referred to preferentially.
(When not matched, a group of the small number is referred to.)

* Back reference by group number is forbidden if named group is defined
in the pattern and ONIG_OPTION_CAPTURE_GROUP is not setted.

back reference with nest level

\k n: 0, 1, 2, …
\k n: 0, 1, 2, …

Destinate relative nest level from back reference position.

ex 1.

/\A(?|.|(?:(?.)\g\k))\z/.match(“reer”)

ex 2.

r = Regexp.compile(<<'__REGEXP__'.strip, Regexp::EXTENDED) (? \g \g* \g ){0}
(? < \g \s* > ){0}
(? [a-zA-Z_:]+ ){0}
(? [^<&]+ (\g | [^<&]+)* ){0} (? >){0}
\g
__REGEXP__

p r.match(‘fbbbf‘).captures

9. Subexp call (“Tanaka Akira special”)

\g call by group name
\g call by group number (n >= 1)

* left-most recursive call is not allowed.
ex. (?a|\gb) => error
(?a|b\gc) => OK

* Call by group number is forbidden if named group is defined in the pattern
and ONIG_OPTION_CAPTURE_GROUP is not setted.

* If the option status of called group is different from calling position
then the group’s option is effective.

ex. (?-i:\g)(?i:(?a)){0} match to “A”

10. Captured group

Behavior of the no-named group (…) changes with the following conditions.
(But named group is not changed.)

case 1. /…/ (named group is not used, no option)

(…) is treated as a captured group.

case 2. /…/g (named group is not used, ‘g’ option)

(…) is treated as a no-captured group (?:…).

case 3. /..(?..)../ (named group is used, no option)

(…) is treated as a no-captured group (?:…).
numbered-backref/call is not allowed.

case 4. /..(?..)../G (named group is used, ‘G’ option)

(…) is treated as a captured group.
numbered-backref/call is allowed.

where
g: ONIG_OPTION_DONT_CAPTURE_GROUP
G: ONIG_OPTION_CAPTURE_GROUP

(‘g’ and ‘G’ options are argued in ruby-dev ML)

—————————–
A-1. Syntax depend options

+ ONIG_SYNTAX_RUBY
(?m): dot(.) match newline

+ ONIG_SYNTAX_PERL and ONIG_SYNTAX_JAVA
(?s): dot(.) match newline
(?m): ^ match after newline, $ match before newline

A-2. Original extensions

+ hexadecimal digit char type \h, \H
+ named group (?…)
+ named backref \k
+ subexp call \g, \g

A-3. Lacked features compare with perl 5.8.0

+ \N{name}
+ \l,\u,\L,\U, \X, \C
+ (?{code})
+ (??{code})
+ (?(condition)yes-pat|no-pat)

* \Q…\E
This is effective on ONIG_SYNTAX_PERL and ONIG_SYNTAX_JAVA.

A-4. Differences with Japanized GNU regex(version 0.12) of Ruby 1.8

+ add character property (\p{property}, \P{property})
+ add hexadecimal digit char type (\h, \H)
+ add look-behind
(?<=fixed-char-length-pattern), (?>]/ in EUC-JP encoding.
+ effect range of isolated option is to next ‘)’.
ex. (?:(?i)a|b) is interpreted as (?:(?i:a|b)), not (?:(?i:a)|b).
+ isolated option is not transparent to previous pattern.
ex. a(?i)* is a syntax error pattern.
+ allowed incompleted left brace as an usual string.
ex. /{/, /({)/, /a{2,3/ etc…
+ negative POSIX bracket [:^xxxx:] is supported.
+ POSIX bracket [:ascii:] is added.
+ repeat of look-ahead is not allowed.
ex. /(?=a)*/, /(?!b){5}/
+ Ignore case option is effective to numbered character.
ex. /\x61/i =~ “A”
+ In the range quantifier, the number of the minimum is omissible.
/a{,n}/ == /a{0,n}/
The simultanious abbreviation of the number of times of the minimum
and the maximum is not allowed. (/a{,}/)
+ /a{n}?/ is not a non-greedy operator.
/a{n}?/ == /(?:a{n})?/
+ invalid back reference is checked and cause error.
/\1/, /(a)\2/
+ Zero-length match in infinite repeat stops the repeat,
then changes of the capture group status are checked as stop condition.
/(?:()|())*\1\2/ =~ “”
/(?:\1a|())*/ =~ “a”

A-5. Disabled functions by default syntax

+ capture history

(?@…) and (?@…)

ex. /(?@a)*/.match(“aaa”) ==> [<0-1>, <1-2>, <2-3>]

see sample/listcap.c file.

A-6. Problems

+ Invalid encoding byte sequence is not checked in UTF-8.

* Invalid first byte is treated as a character.
/./u =~ “\xa3”

* Incomplete byte sequence is not checked.
/\w+/ =~ “a\xf3\x8ec”

// END