Lexers Menu Help

Name:	Lexers Menu
Version:	0.1
Last update:	2019-12-17 16:57:24

Introduction
Activation
Basics
Lexers location
Description of the menu items
Default lexers
Key Bindings
Lexers vs. Filters
Notes for developers
- Using lexers from a script
- Registering new lexers in a package
Known problems
License and Disclaimer

Introduction

The Lexers Menu allows you to apply a lexical analyzer (aka lexer) to a document window or to a chunk of text.
Alpha uses the term lexer in a somewhat restrictive sense: throughout this document, it designates an executable command line tool that takes a string as argument and outputs another string. Typically, lexers are designed to process a string byte by byte and apply transformation rules depending of what it finds at the current position. The lexer parses the string from start to end and advances its position after a rule was applied.
Lexers can also be used to gather information while processing a string and display the results.
Lexers are usually written using the Flex program. But this is not a requirement: any command line tool that takes a string argument and returns a string may be used.

Activation

To activate the Lexers Menu, go to Alpha ↣ Global Setup ↣ Features and select the Lexers Menu checkbox.
Once it is activated, you have a new submenu called Lexers in the Tools menu.

Basics

In order to process a piece of text, you have to select an appropriate lexer with the Choose A Lexer… menu item.
In order to process the current window, select the Apply To Window… menu item. If you want to process only a part of your current window, hilite the corresponding region and choose the Apply To Selection… menu item.
Alpha comes with some predefined lexers found inside the application bundle but you can also declare your own lexers as explained in the next section.

Lexers location

All third party or user defined lexers should be stored in one of the Application Support folders, either in the user domain or in the local domain:

/Users/your-login/Library/Application Support/Alpha/Lexers
/Library/Application Support/Alpha/Lexers

Use the Add A Lexer… and Remove A Lexer… commands described below to manage third party lexers.
Alternatively it is also possible to put lexers in Alpha's preferences folder (designated by the PREFS variable):

$PREFS/Lexers

Description of the menu items

Here is the description of all the menu items found in the Lexers Menu.

Choose A Lexer…: Display a list of all the currently available lexers to let you select the one you want to apply. The name of the selected lexer is displayed at the bottom of the Lexers Menu.
Apply Lexer To Selection: Apply the lexer to a selected region in the front window.
Apply Lexer To Window: Apply the lexer to the current window.
Add A Lexer…: Lets you add a new lexer. New lexers may be added in one of Alpha's Application Support folders.
Remove A Lexer…: Lets you remove a lexer from the Alpha's Application Support folders. Note that lexers registered by a package for its own use can not be removed through this menu item (see Registering new lexers in a package).
Reveal Lexers Folder…: Lets you open in the Finder one of Alpha's Lexers folders (found in the Application Support folders of the application).
Apply Temp To Folder…: Prompts you for a folder to apply the Temporary Lexer to. This is not undoable.
Current Lexer: Display a window with some information.

Default lexers

Alpha ships with some default lexers included in the application bundle. Here is a short description of the default lexers as of version 9.1.2 of Alpha:

EscapeHtmlSymbols: Convert HTML basic symbols (&,<,>,',") to entities.
Rot13: Apply Rot13 conversion, shifting the alphabet letters circularly by 13 positions.
Rot47: Apply Rot47 conversion shifting the 7-bit printable characters, excluding space, from decimal 33 '!' through 126 '~', 94 in total, taken in the order of their ASCII codes, by 47 positions.
StripAccents: Replace accented letters by their unaccented equivalent.
UnescapeHtmlEntities: Convert HTML basic entities to the corresponding symbols.

Many other lexers may be available since users can add their own lexers (see Add A Lexer) and developers of Alpha packages can register lexers to implement some functionalities (see the Notes for developers section).

Key Bindings

There are a few easy to remember key bindings à la emacs. They are composite bindings starting with ⌃X. Click here to display all the currently available bindings.

Lexers vs. Filters

Alpha has a notion of filters that are provided by the Filters Menu. Despite apparent similarity, filters and lexers are completely different objects.
Both can be applied to a selection or a window, but they differ conceptually. A filter is just a set of Search & Replace operations that are applied sequentially: each Search & Replace operation is performed on the entire text resulting from the previous operation. Filters are a handy way of storing one or several Search & Replace commands and reuse them later.
On the contrary, a lexer considers the text as a stream and, at each position reached, it tests a series of rules to find the one that best matches the following characters. At each position, only one rule is applied, then the lexer moves to the position after the last match and proceeds.

Notes for developers

Using lexers from a script

The AlphaTcl library defines several procs which let you apply a lexer from a Tcl script. The syntax is:

lexer::applyToSelection ?-w win? lexer
lexer::applyToWindow ?-w win? lexer
lexer::applyToFile lexer file ?backup?

For more information, see the comments in the file lexers.tcl.

Registering new lexers in a package

A package can provide its own lexers in order to implement some functionality and register them so that they are available for use from the Lexers Menu. Two procs let you easily register or deregister a lexer, with the following syntax:

lexer::register path ?name?
lexer::deregister name

For instance, a package could register, during its activation phase, a lexer /usr/local/bin/somelexer under the name "Some Lexer" like this:

lexer::register /usr/local/bin/somelexer "Some Lexer"

During the deactivation phase, the package could deregister this lexer like this:

lexer::deregister "Some Lexer"

That's all there is to it. Note that it is not possible to remove a registered lexer using the Remove A Lexer… menu item.

Known problems

Please report any problem or bug you encounter in Alpha's Bug Tracker.

License and Disclaimer

Copyright (c) 2019, Bernard Desgraupes.
All rights reserved.
The Lexers Menu package is free software and distributed under the terms of the new BSD license:
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Bernard Desgraupes nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL BERNARD DESGRAUPES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.