felix rosencrantz Generating Clearcase Completions
process
 Using perl and XML I've been attempting to automatically (or semi-automatically) generate ZSH completion functions.  As a test case for a complex command that has sub-commands I wanted to try generating completions for the Clearcase cleartool command.

It should be possible to automatically generate completion functions that can complete flags.  However, it will still require a person to create functions that complete different types of arguments (e.g. a listing of label types, or branch types).

I use to work with Clearcase.  I had some rudimentary completion functions in tcsh and zsh. The problem I had was that as new versions of Clearcase were released the functions had to change since there would be new subcommands, new flags, or changed flags.  Also, depending on the licenses, some machines might have some subcommands that other hosts don't.

Being able to generate completion functions from Clearcase documentation would make a significant difference.

Here is the process I used to create these functions. Ideally it would be possible for do this for

step 1
 Create an output file of Clearcase help commands.

cleartool help > ct_help.out

The help command without any arguments will list help for all subcommands. The help output contains basic command line usage descriptions without flag descriptions.  I made this file when I had access to Clearcase. 

step 2

 Parse the help output to create an xml file.  To do this I wrote a perl script.  This script is fairly specific to this help output.  Though, hopefully, it should be possible to make this more general, to be useful for other tools.  (I needed to install the Text::Balanced package.)

perl parse_ct.pl ct_help.out > ct_helpA.xml

The generated xml file is a format that should be easy to create from the above perl script, and then via various XSLT transformations generate an XML file that can be used with args.xsl.

So from the chview command usage:

cleartool chview {[-cachesize size] [-shareable_dos | -nshareable_dos]  [-readonly | -readwrite]}  {-cview | view-tag}

The perl script generates output that looks like:

   <subcommand cmd='chview'>
 <!-- chview {[-cachesize size] [-shareable_dos | -nshareable_dos] [-readonly | -readwrite]} {-cview | view-tag} -->
    <brace>
    <brace>
     <flag name='-cachesize'/>
     <word name='size'/>
    </brace>
    <brace>
     <flag name='-shareable_dos'/>
     <or/>
     <flag name='-nshareable_dos'/>
    </brace>
    <brace>
     <flag name='-readonly'/>
     <or/>
     <flag name='-readwrite'/>
    </brace>
    </brace>
    <brace>
     <flag name='-cview'/>
     <or/>
     <word name='view-tag'/>
    </brace>
 <!-- mapping ((fw)(f|f)(f|f))(f|w) -->
     </subcommand>

 

step 3

This step transforms the XML generated in step2 to something that is a little easier to generate the completion XML format.  The basic transforms are:

  1. Add a <flagset> node to each <subcommand>. <flagset> holds a list of all the flags that this command has.  This is useful for performing iterations when generating the _arguments call.
  2. Wrap the contents in a <content> node.  This is useful since we added the <flagset> node to <subcommand>.
  3. Convert the <or> tags into a hierarchy.  With a root of <group>, that only has <or> children.

saxon ct_helpA.xml s1.xsl > ct_helpB.xml

So the XML output from the previous step on the chview subcommand now looks like:

   <subcommand cmd="chview">
      <flagset>
         <flag name="-cachesize"/>
         <flag name="-cview"/>
         <flag name="-nshareable_dos"/>
         <flag name="-readonly"/>
         <flag name="-readwrite"/>
         <flag name="-shareable_dos"/>
      </flagset>
      <contents>
         <brace>
            <brace>
               <flag name="-cachesize"/>
               <word name="size"/>
            </brace>
            <brace>
               <group>
                  <or end="2">
                     <flag name="-shareable_dos"/>
                  </or>
                  <or start="2" end="4">
                     <flag name="-nshareable_dos"/>
                  </or>
               </group>
            </brace>
            <brace>
               <group>
                  <or end="2">
                     <flag name="-readonly"/>
                  </or>
                  <or start="2" end="4">
                     <flag name="-readwrite"/>
                  </or>
               </group>
            </brace>
         </brace>
         <brace>
            <group>
               <or end="2">
                  <flag name="-cview"/>
               </or>
               <or start="2" end="4">
                  <word name="view-tag"/>
               </or>
            </group>
         </brace>
      </contents>
   </subcommand>

step 4

 Another transformation to get closer to the completion format.  The basic transformations are:
  1. Make <flagset> have a unique set of flags.  Some commands had the same flag listed multiple times.
  2. Group flags with their arguments. In ct_helpB.xml, <flag> nodes were leave nodes.  Now they have children for their arguments.  (There are some problems with this transform, that causes some flags to get extra arguments.)
  3. Look for the pattern -flag {const1|const2|const3}, and convert this to constant word set. (This is not shown in the example.  To see an example look at the "find -type" flag.)

saxon ct_helpB.xml s2.xsl > ct_helpC.xml

So the chview command now looks like:

   <subcommand cmd="chview">
      <flagset>
         <flag name="-cachesize"/>
         <flag name="-cview"/>
         <flag name="-nshareable_dos"/>
         <flag name="-readonly"/>
         <flag name="-readwrite"/>
         <flag name="-shareable_dos"/>
      </flagset>
      <contents>
         <brace>
            <brace>
               <flag name="-cachesize">
                  <flag_word name="size"/>
               </flag>
            </brace>
            <brace>
               <group>
                  <or end="2">
                     <flag name="-shareable_dos"/>
                  </or>
                  <or start="2" end="4">
                     <flag name="-nshareable_dos"/>
                  </or>
               </group>
            </brace>
            <brace>
               <group>
                  <or end="2">
                     <flag name="-readonly"/>
                  </or>
                  <or start="2" end="4">
                     <flag name="-readwrite"/>
                  </or>
               </group>
            </brace>
         </brace>
         <brace>
            <group>
               <or end="2">
                  <flag name="-cview"/>
               </or>
               <or start="2" end="4">
                  <word name="view-tag"/>
               </or>
            </group>
         </brace>
      </contents>
   </subcommand>

step 5

 We now have a format that is easier to generate thecompletion XML format.  This file contains header information that is copied directly to the ct_args.xml file.  The rest is generated from the input of ct_helpC.xml. 

saxon ct_helpC.xml s3.xsl > ct_args.xml

Here is the start of chview:

<subcommand name="chview">
         <service>
            <arguments/>
            <argspecs>
               <argspec>
                  <optspec>
                     <flagarg name="-cachesize">
                        <explanation/>
                     </flagarg>
                  </optspec>
                  <description>
                     <message/>
                     <action tag="size"/>
                  </description>
               </argspec>
               <argspec>
                  <optspec>
                     <exclusionlist/>
                     <flagarg name="-cview">
                        <explanation/>
                     </flagarg>
                  </optspec>
                  <description>
                     <message/>
                  </description>
               </argspec>
               <argspec>
                  <optspec>
                     <exclusionlist>
                        <flag>-shareable_dos</flag>
                     </exclusionlist>
                     <flagarg name="-nshareable_dos">
                        <explanation/>
                     </flagarg>
                  </optspec>
                  <description>
                     <message/>

 

step 6

 The final step to generate the completion function.  However, before we can do that we need to generate an include file that says how to map action tags to actual actions.  For example, the chview command's flag -cachesize has an action with the tag 'size'.   The include file can map that to an action.

The next few steps are an attempt to generate this include file  based on data on the ct_args.xml file.  The first xsl script does the following:

  1. Extract all the action elements, and converts them to actiontemplates, which is close to the final format.

saxon ct_args.xml gen_comdefs.xsl > defsA.xml

 

Some of the output looks like:

   <actiontemplate tag="command-name">
      <action> </action>
   </actiontemplate>
   <actiontemplate tag="command-name">
      <action> </action>
   </actiontemplate>
   <actiontemplate tag="topic">
      <action> </action>
   </actiontemplate>

step 7

 This step removes duplicates. This should have been done in the previous step.

saxon defsA.xml unique_actions.xsl > defsB.xml

step 8

 The purpose of this step is to insert actions into the  generated include file.  The action.xup script has hand-coded mappings of tags to action. This step uses a transform language other than XSLT, called xupdate,.  XSLT could be used, though I found xupdate did more of what I wanted for this situation.   The perl script cleans up some of the resulting XML, removing a bogus doctype instruction.

xupdate actions.xup defsB.xml > defsC.xml

perl -p -e 's/<!DOCTYPE root>//;' defsC.xml > clearcase_defines.xml

Xupdate information can be found here.  I had to make changes to the XUpdateQueryImpl.java to be able to use xupdate on the command line.

step 9

 The final step.  We've generated the ct_args.xml file, and the includes file.  Now we can generate the actual completion function.

saxon ct_args.xml arg.xsl > _cleartool

Now the completion for chview:

(( $+functions[_cc_chview] )) ||
_cc_chview () {
        _arguments \
                '-cachesize:Handling Tag size' \
                '()-cview' \
                '(-shareable_dos)-nshareable_dos' \
                '(-readwrite)-readonly' \
                '(-readonly)-readwrite' \
                '(-nshareable_dos)-shareable_dos' \
                '1:Handling Tag view-tag:_cc_views'
}
 

problems

 This does a reasonable first draft.  Though it still has a lot of problems.
  1. No way to attach help messages/explanations
    • to flags
    • to fixed sets
  2. Problem with dealing with commands that have multiple formats for example mkview. Currently generate multiple functions.  So the last function is the one that is used.
  3. Problem determining command args vs. flag args. (Look at "get" command. flag is followed by 2 words...)
  4. Generating sets that are a list, -types A,B,C (is there an example?)
     
  5. Find command has a problem with multiple anypositions.
     
  6. Find command has problem in that flags will only be allowed once.
  7. The clearcase command has some flags, these are exclusive of clearcase commands.  But currently that is not how it works.
  8. This does not handle flag abbreviations.  Clearcase allow users to abbreviate flag names.
  9. Problem generating exclusion list for flags version anypostion.  See chview command with -cview flag.
  10. Sure there are dozens of other problems...

files
File (in html) Original Format Explanation
actions.xup actions.xup
arg.xsl arg.xsl
clearcase_defines.xml clearcase_defines.xml
_cleartool _cleartool
command_defines.dtd command_defines.dtd
completion.dtd completion.dtd
ct_args.xml ct_args.xml
ct_helpA.xml ct_helpA.xml
ct_helpB.xml ct_helpB.xml
ct_helpC.xml ct_helpC.xml
ct_help.out ct_help.out
defsA.xml defsA.xml
defsB.xml defsB.xml
defsC.xml defsC.xml
_find _find
find.xml find.xml
gen_comdefs.xsl gen_comdefs.xsl
parse_ct.pl parse_ct.pl
s1.xsl s1.xsl
s2.xsl s2.xsl
s3.xsl s3.xsl
unique_actions.xsl unique_actions.xsl
XUpdateQueryImpl.java XUpdateQueryImpl.java
XUpdateQueryImpl.diff XUpdateQueryImpl.diff
zip zip
zip.xml zip.xml

Web site Copyright ©  2001 Felix Rosencrantz, f_rosencrantz@yahoo.com