vim

Fork of vim, an acceptable text editor
git clone git://git.laack.co/vim.git
Log | Files | Refs | README | LICENSE

README.txt (10205B)


      1 Tests for syntax highlighting plugins
      2 =====================================
      3 
      4 Summary: Files in the "input" directory are edited by Vim with syntax
      5 highlighting enabled.  Screendumps are generated and compared with the
      6 expected screendumps in the "dumps" directory.  This will uncover any
      7 character attributes that differ.
      8 
      9 The dumps are normally 20 screen lines tall.  Without any further setup
     10 a screendump is made at the top of the file (using _00.dump) and another
     11 screendump is made if there are more lines (using _01.dump), and so on.
     12 
     13 When the screendumps are OK an empty "done/{name}" file is created.  This
     14 avoids running the test again until "make clean" is used.  Thus you can run
     15 "make test", see one test fail, try to fix the problem, then run "make test"
     16 again to only repeat the failing test.
     17 
     18 When a screendump differs it is stored in the "failed" directory.  This allows
     19 for comparing it with the expected screendump, using a command like:
     20 
     21 	let fname = '{name}_00.dump'
     22 	call term_dumpdiff('failed/' .. fname, 'dumps/' .. fname)
     23 
     24 
     25 Creating a syntax plugin test
     26 -----------------------------
     27 
     28 Create a source file in the language you want to test in the "input"
     29 directory.  Use the filetype name as the base and a filename extension
     30 matching the filetype.  Let's use Java as an example.  The file would then be
     31 "input/java.java".  As another example, consider Gitignore.  Since there is no
     32 established filename extension for this filetype, use the filetype name for
     33 its base AND filename extension, "input/gitignore.gitignore", as the first
     34 step.
     35 
     36 Make sure to include some interesting constructs with plenty of complicated
     37 highlighting.  Optionally, pre-configure the testing environment by including
     38 setup commands at the top of the input file.  The format for these lines is:
     39 
     40 	VIM_TEST_SETUP {command}
     41 
     42 where {command} is any valid Ex command, which extends to the end of the line.
     43 The first 20 lines of the input file are ALWAYS scanned for setup commands and
     44 these will be executed before the syntax highlighting is enabled.  Typically,
     45 these lines would be included as comments so as not to introduce any syntax
     46 errors in the input file but this is not required.
     47 
     48 Continuing the Java example:
     49 
     50 	// VIM_TEST_SETUP let g:java_space_errors = 1
     51 	// VIM_TEST_SETUP let g:java_minlines = 5
     52 	class Test { }
     53 
     54 Continuing the Gitignore example, give the file a valid name as the last step:
     55 
     56 	# VIM_TEST_SETUP call DeferRenamingTestFileTo('.gitignore')
     57 
     58 As an alternative, setup commands can be included in an external Vim script
     59 file in the "input/setup" directory.  This script file must have the same base
     60 name as the input file.
     61 
     62 So, the equivalent example configuration using this method would be to create
     63 an "input/setup/java.vim" script file with the following lines:
     64 
     65 	let g:java_space_errors = 1
     66 	let g:java_minlines = 5
     67 
     68 Both inline setup commands and setup scripts may be used at the same time, the
     69 script file will be sourced before any VIM_TEST_SETUP commands are executed.
     70 
     71 Every line of a source file must not be longer than 1425 (19 x 75) characters.
     72 
     73 If there is no further setup required, you can now run all tests:
     74 
     75 	make test
     76 
     77 Or you can run the tests for a filetype only by passing its name as another
     78 target, e.g. "java", before "test":
     79 
     80 	make java test
     81 
     82 Or you can run a test or two by passing their filenames as extra targets, e.g.
     83 "java_string.java" and "java_numbers.java", before "test", after listing all
     84 available syntax tests for Java:
     85 
     86 	ls testdir/input/java*
     87 	make java_string.java java_numbers.java test
     88 
     89 (Some interactive shells may attempt to perform word completion on arbitrary
     90 command arguments when you press certain keys, e.g. Tab or Ctrl-i.)
     91 
     92 As an alternative, you can specify a subset of test filenames for running as
     93 a regular expression and assign it to a VIM_SYNTAX_TEST_FILTER environment
     94 variable; e.g. to run all tests whose base names contain "fold", use any of:
     95 
     96 	make test -e 'VIM_SYNTAX_TEST_FILTER = fold.*\..\+'
     97 	make test VIM_SYNTAX_TEST_FILTER='fold.*\..\+'
     98 	VIM_SYNTAX_TEST_FILTER='fold.*\..\+' make test
     99 
    100 Consider quoting the variable value to avoid any interpretation by the shell.
    101 
    102 Both Make targets and the variable may be used at the same time, the target
    103 names will be tried for matching before the variable value.
    104 
    105 The first time testing "input/java.java" will fail with an error for a missing
    106 screendump.  The newly created screendumps will be "failed/java_00.dump",
    107 "failed/java_01.dump", etc.  You can inspect each with:
    108 
    109 	call term_dumpload('failed/java_00.dump')
    110 	call term_dumpload('failed/java_01.dump')
    111 	...
    112 
    113 If they look OK, move them to the "dumps" directory:
    114 
    115 	:!mv failed/java_00.dump dumps
    116 	:!mv failed/java_01.dump dumps
    117 	...
    118 
    119 If you now run the test again, it will succeed.
    120 
    121 
    122 Adjusting a syntax plugin test
    123 ------------------------------
    124 
    125 If you make changes to the syntax plugin, you should add code to the input
    126 file to see the effect of these changes.  So that the effect of the changes
    127 is covered by the test.  You can follow these steps:
    128 
    129 1. Edit the syntax plugin somewhere in your personal setup.  Use a file
    130    somewhere to try out the changes.
    131 2. Go to the directory where you have the Vim code checked out and replace the
    132    syntax plugin.  Run the tests: "make test".  Usually the tests will still
    133    pass, but if you fixed syntax highlighting that was already visible in the
    134    input file, carefully check that the changes in the screendump are
    135    intentional:
    136 
    137 	let fname = '{name}_00.dump'
    138 	call term_dumpdiff('failed/' .. fname, 'dumps/' .. fname)
    139 
    140    Fix the syntax plugin until the result is good.
    141 2. Edit the input file for your language to add the items you have improved.
    142    (TODO: how to add another screendump?).
    143    Run the tests and you should get failures.  (You may opt for faster failure
    144    by assigning a small number, e.g. "1", to a VIM_SYNTAX_TEST_WAIT_TIME
    145    environment variable and gambling away an "uncertain" possibility of
    146    success.)  Like with the previous step, carefully check that the new
    147    screendumps in the "failed" directory are good.  Update the syntax plugin
    148    and the input file until the highlighting is good and you can see the
    149    effect of the syntax plugin improvements.  Then move the screendumps from
    150    the "failed" to the "dumps" directory.  Now "make test" should succeed.
    151 3. Prepare a pull request with the modified files:
    152 	- syntax plugin:    syntax/{name}.vim
    153 	- Vim setup file:   syntax/testdir/input/setup/{name}.vim (if any)
    154 	- test input file:  syntax/testdir/input/{name}.{ext}
    155 	- test dump files:  syntax/testdir/dumps/{name}_*.dump
    156    Since no input file is ever executed when you run the tests, review and
    157    revoke each previously granted permission to execute such a file before
    158    publishing it; e.g. "chmod -x input/java.java; git add input/java.java".
    159 
    160 As an extra check you can temporarily put back the old syntax plugin and
    161 verify that the tests fail.  Then you know your changes are covered by the
    162 test.
    163 
    164 
    165 Viewing generated screendumps (local)
    166 -------------------------------------
    167 
    168 You may also wish to look at the whole batch of failed screendumps after
    169 running "make test".  Source the "viewdumps.vim" script for this task:
    170 
    171 	[VIMRUNTIME=../..] \
    172 	../../src/vim --clean -S testdir/viewdumps.vim \
    173 				[testdir/dumps/java_*.dump ...]
    174 
    175 By default, all screendumps found in the "failed" directory will be added to
    176 the argument list and then the first one will be loaded.  Loaded screendumps
    177 that bear filenames of screendumps found in the "dumps" directory will be
    178 rendering the contents of any such pair of files and the difference between
    179 them (:help term_dumpdiff()); otherwise, they will be rendering own contents
    180 (:help term_dumpload()).  Remember to execute :edit when occasionally you see
    181 raw file contents instead of rendered.
    182 
    183 At any time, you can add, list, and abandon other screendumps:
    184 
    185 	:$argedit testdir/dumps/java_*.dump
    186 	:args
    187 	:qall
    188 
    189 The listing of argument commands can be found under :help buffer-list.
    190 
    191 
    192 Viewing generated screendumps (from a CI-uploaded artifact)
    193 -----------------------------------------------------------
    194 
    195 After you have downloaded an artifact archive containing failed screendumps
    196 and extracted its files in a temporary directory, you need to set up a "dumps"
    197 directory by creating a symlink:
    198 
    199 	cd /path/to/fork
    200 	ln -s $(pwd)/runtime/syntax/testdir/dumps \
    201 				/tmp/runtime/syntax/testdir/dumps
    202 
    203 You can now examine the extracted screendumps:
    204 
    205 	./src/vim --clean -S runtime/syntax/testdir/viewdumps.vim \
    206 				/tmp/runtime/syntax/testdir/failed/*.dump
    207 
    208 
    209 Viewing generated screendumps (submitted for a pull request)
    210 ------------------------------------------------------------
    211 
    212 Note: There is also a "git difftool" extension described in
    213       src/testdir/commondumps.vim
    214 
    215 First, you need to check out the topic branch with the proposed changes and
    216 write down a difference list between the HEAD commit (index) and its parent
    217 commit with respect to the changed "dumps" filenames:
    218 
    219 	cd /path/to/fork
    220 	git switch prs/1234
    221 	git diff-index --relative=runtime/syntax/testdir/dumps/ \
    222 				--name-only prs/1234~1 > /tmp/filelist
    223 
    224 Then, you need to check out the master branch, change the current working
    225 directory to reconcile relative filepaths written in the filenames list, copy
    226 in the "failed" directory the old "dumps" files, whose names are on the same
    227 list, and follow it by checking out the topic branch:
    228 
    229 	git switch master
    230 	cd runtime/syntax/testdir/dumps
    231 	cp -t ../failed $(cat /tmp/filelist)
    232 	git switch prs/1234
    233 
    234 Make note of any missing new screendumps.  Please remember about the
    235 introduced INVERTED relation between "dumps" and "failed", i.e. the files to
    236 be committed are in "dumps" already and their old versions are in "failed".
    237 Therefore, you need to copy the missing new screendumps from "dumps" to
    238 "failed":
    239 
    240 	cp -t ../failed foo_10.dump foo_11.dump foo_12.dump
    241 
    242 After you have changed the current working directory to its parent directory,
    243 you can now examine the screendumps from the "failed" directory (note that new
    244 screendumps will be shown with no difference between their versions):
    245 
    246 	cd ..
    247 	../../../src/vim --clean -S viewdumps.vim
    248 
    249 
    250 TODO: test syncing by jumping around