README.txt (10205B)
1 Tests for syntax highlighting plugins 2 ===================================== 3 4 Summary: Files in the "input" directory are edited by Vim with syntax 5 highlighting enabled. Screendumps are generated and compared with the 6 expected screendumps in the "dumps" directory. This will uncover any 7 character attributes that differ. 8 9 The dumps are normally 20 screen lines tall. Without any further setup 10 a screendump is made at the top of the file (using _00.dump) and another 11 screendump is made if there are more lines (using _01.dump), and so on. 12 13 When the screendumps are OK an empty "done/{name}" file is created. This 14 avoids running the test again until "make clean" is used. Thus you can run 15 "make test", see one test fail, try to fix the problem, then run "make test" 16 again to only repeat the failing test. 17 18 When a screendump differs it is stored in the "failed" directory. This allows 19 for comparing it with the expected screendump, using a command like: 20 21 let fname = '{name}_00.dump' 22 call term_dumpdiff('failed/' .. fname, 'dumps/' .. fname) 23 24 25 Creating a syntax plugin test 26 ----------------------------- 27 28 Create a source file in the language you want to test in the "input" 29 directory. Use the filetype name as the base and a filename extension 30 matching the filetype. Let's use Java as an example. The file would then be 31 "input/java.java". As another example, consider Gitignore. Since there is no 32 established filename extension for this filetype, use the filetype name for 33 its base AND filename extension, "input/gitignore.gitignore", as the first 34 step. 35 36 Make sure to include some interesting constructs with plenty of complicated 37 highlighting. Optionally, pre-configure the testing environment by including 38 setup commands at the top of the input file. The format for these lines is: 39 40 VIM_TEST_SETUP {command} 41 42 where {command} is any valid Ex command, which extends to the end of the line. 43 The first 20 lines of the input file are ALWAYS scanned for setup commands and 44 these will be executed before the syntax highlighting is enabled. Typically, 45 these lines would be included as comments so as not to introduce any syntax 46 errors in the input file but this is not required. 47 48 Continuing the Java example: 49 50 // VIM_TEST_SETUP let g:java_space_errors = 1 51 // VIM_TEST_SETUP let g:java_minlines = 5 52 class Test { } 53 54 Continuing the Gitignore example, give the file a valid name as the last step: 55 56 # VIM_TEST_SETUP call DeferRenamingTestFileTo('.gitignore') 57 58 As an alternative, setup commands can be included in an external Vim script 59 file in the "input/setup" directory. This script file must have the same base 60 name as the input file. 61 62 So, the equivalent example configuration using this method would be to create 63 an "input/setup/java.vim" script file with the following lines: 64 65 let g:java_space_errors = 1 66 let g:java_minlines = 5 67 68 Both inline setup commands and setup scripts may be used at the same time, the 69 script file will be sourced before any VIM_TEST_SETUP commands are executed. 70 71 Every line of a source file must not be longer than 1425 (19 x 75) characters. 72 73 If there is no further setup required, you can now run all tests: 74 75 make test 76 77 Or you can run the tests for a filetype only by passing its name as another 78 target, e.g. "java", before "test": 79 80 make java test 81 82 Or you can run a test or two by passing their filenames as extra targets, e.g. 83 "java_string.java" and "java_numbers.java", before "test", after listing all 84 available syntax tests for Java: 85 86 ls testdir/input/java* 87 make java_string.java java_numbers.java test 88 89 (Some interactive shells may attempt to perform word completion on arbitrary 90 command arguments when you press certain keys, e.g. Tab or Ctrl-i.) 91 92 As an alternative, you can specify a subset of test filenames for running as 93 a regular expression and assign it to a VIM_SYNTAX_TEST_FILTER environment 94 variable; e.g. to run all tests whose base names contain "fold", use any of: 95 96 make test -e 'VIM_SYNTAX_TEST_FILTER = fold.*\..\+' 97 make test VIM_SYNTAX_TEST_FILTER='fold.*\..\+' 98 VIM_SYNTAX_TEST_FILTER='fold.*\..\+' make test 99 100 Consider quoting the variable value to avoid any interpretation by the shell. 101 102 Both Make targets and the variable may be used at the same time, the target 103 names will be tried for matching before the variable value. 104 105 The first time testing "input/java.java" will fail with an error for a missing 106 screendump. The newly created screendumps will be "failed/java_00.dump", 107 "failed/java_01.dump", etc. You can inspect each with: 108 109 call term_dumpload('failed/java_00.dump') 110 call term_dumpload('failed/java_01.dump') 111 ... 112 113 If they look OK, move them to the "dumps" directory: 114 115 :!mv failed/java_00.dump dumps 116 :!mv failed/java_01.dump dumps 117 ... 118 119 If you now run the test again, it will succeed. 120 121 122 Adjusting a syntax plugin test 123 ------------------------------ 124 125 If you make changes to the syntax plugin, you should add code to the input 126 file to see the effect of these changes. So that the effect of the changes 127 is covered by the test. You can follow these steps: 128 129 1. Edit the syntax plugin somewhere in your personal setup. Use a file 130 somewhere to try out the changes. 131 2. Go to the directory where you have the Vim code checked out and replace the 132 syntax plugin. Run the tests: "make test". Usually the tests will still 133 pass, but if you fixed syntax highlighting that was already visible in the 134 input file, carefully check that the changes in the screendump are 135 intentional: 136 137 let fname = '{name}_00.dump' 138 call term_dumpdiff('failed/' .. fname, 'dumps/' .. fname) 139 140 Fix the syntax plugin until the result is good. 141 2. Edit the input file for your language to add the items you have improved. 142 (TODO: how to add another screendump?). 143 Run the tests and you should get failures. (You may opt for faster failure 144 by assigning a small number, e.g. "1", to a VIM_SYNTAX_TEST_WAIT_TIME 145 environment variable and gambling away an "uncertain" possibility of 146 success.) Like with the previous step, carefully check that the new 147 screendumps in the "failed" directory are good. Update the syntax plugin 148 and the input file until the highlighting is good and you can see the 149 effect of the syntax plugin improvements. Then move the screendumps from 150 the "failed" to the "dumps" directory. Now "make test" should succeed. 151 3. Prepare a pull request with the modified files: 152 - syntax plugin: syntax/{name}.vim 153 - Vim setup file: syntax/testdir/input/setup/{name}.vim (if any) 154 - test input file: syntax/testdir/input/{name}.{ext} 155 - test dump files: syntax/testdir/dumps/{name}_*.dump 156 Since no input file is ever executed when you run the tests, review and 157 revoke each previously granted permission to execute such a file before 158 publishing it; e.g. "chmod -x input/java.java; git add input/java.java". 159 160 As an extra check you can temporarily put back the old syntax plugin and 161 verify that the tests fail. Then you know your changes are covered by the 162 test. 163 164 165 Viewing generated screendumps (local) 166 ------------------------------------- 167 168 You may also wish to look at the whole batch of failed screendumps after 169 running "make test". Source the "viewdumps.vim" script for this task: 170 171 [VIMRUNTIME=../..] \ 172 ../../src/vim --clean -S testdir/viewdumps.vim \ 173 [testdir/dumps/java_*.dump ...] 174 175 By default, all screendumps found in the "failed" directory will be added to 176 the argument list and then the first one will be loaded. Loaded screendumps 177 that bear filenames of screendumps found in the "dumps" directory will be 178 rendering the contents of any such pair of files and the difference between 179 them (:help term_dumpdiff()); otherwise, they will be rendering own contents 180 (:help term_dumpload()). Remember to execute :edit when occasionally you see 181 raw file contents instead of rendered. 182 183 At any time, you can add, list, and abandon other screendumps: 184 185 :$argedit testdir/dumps/java_*.dump 186 :args 187 :qall 188 189 The listing of argument commands can be found under :help buffer-list. 190 191 192 Viewing generated screendumps (from a CI-uploaded artifact) 193 ----------------------------------------------------------- 194 195 After you have downloaded an artifact archive containing failed screendumps 196 and extracted its files in a temporary directory, you need to set up a "dumps" 197 directory by creating a symlink: 198 199 cd /path/to/fork 200 ln -s $(pwd)/runtime/syntax/testdir/dumps \ 201 /tmp/runtime/syntax/testdir/dumps 202 203 You can now examine the extracted screendumps: 204 205 ./src/vim --clean -S runtime/syntax/testdir/viewdumps.vim \ 206 /tmp/runtime/syntax/testdir/failed/*.dump 207 208 209 Viewing generated screendumps (submitted for a pull request) 210 ------------------------------------------------------------ 211 212 Note: There is also a "git difftool" extension described in 213 src/testdir/commondumps.vim 214 215 First, you need to check out the topic branch with the proposed changes and 216 write down a difference list between the HEAD commit (index) and its parent 217 commit with respect to the changed "dumps" filenames: 218 219 cd /path/to/fork 220 git switch prs/1234 221 git diff-index --relative=runtime/syntax/testdir/dumps/ \ 222 --name-only prs/1234~1 > /tmp/filelist 223 224 Then, you need to check out the master branch, change the current working 225 directory to reconcile relative filepaths written in the filenames list, copy 226 in the "failed" directory the old "dumps" files, whose names are on the same 227 list, and follow it by checking out the topic branch: 228 229 git switch master 230 cd runtime/syntax/testdir/dumps 231 cp -t ../failed $(cat /tmp/filelist) 232 git switch prs/1234 233 234 Make note of any missing new screendumps. Please remember about the 235 introduced INVERTED relation between "dumps" and "failed", i.e. the files to 236 be committed are in "dumps" already and their old versions are in "failed". 237 Therefore, you need to copy the missing new screendumps from "dumps" to 238 "failed": 239 240 cp -t ../failed foo_10.dump foo_11.dump foo_12.dump 241 242 After you have changed the current working directory to its parent directory, 243 you can now examine the screendumps from the "failed" directory (note that new 244 screendumps will be shown with no difference between their versions): 245 246 cd .. 247 ../../../src/vim --clean -S viewdumps.vim 248 249 250 TODO: test syncing by jumping around