README_VIM9.md (10833B)
1  2 3 # What is Vim9? 4 5 This is a new syntax for Vim script that was introduced with Vim 9.0. 6 It intends making Vim script faster and better. 7 8 9 # Why Vim9? 10 11 ## 1. FASTER VIM SCRIPT 12 13 The third item on the poll results of 2018, after popup windows and text 14 properties, both of which have been implemented, is faster Vim script. 15 So how do we do that? 16 17 I have been throwing some ideas around, and soon came to the conclusion 18 that the current way functions are called and executed, with 19 dictionaries for the arguments and local variables, is never going to be 20 very fast. We're lucky if we can make it twice as fast. The overhead 21 of a function call and executing every line is just too high. 22 23 So what then? We can only make something fast by having a new way of 24 defining a function, with similar but different properties of the old 25 way: 26 * Arguments are only available by name, not through the a: dictionary or 27 the a:000 list. 28 * Local variables are not available in an l: dictionary. 29 * A few more things that slow us down, such as exception handling details. 30 31 I implemented a "proof of concept" and measured the time to run a simple 32 for loop with an addition (Justin used this example in his presentation, 33 full code is below): 34 35 ``` vim 36 let sum = 0 37 for i in range(1, 2999999) 38 let sum += i 39 endfor 40 ``` 41 42 | how | time in sec | 43 | --------| -------- | 44 | Vim old | 5.018541 | 45 | Python | 0.369598 | 46 | Lua | 0.078817 | 47 | LuaJit | 0.004245 | 48 | Vim new | 0.073595 | 49 50 That looks very promising! It's just one example, but it shows how much 51 we can gain, and also that Vim script can be faster than builtin 52 interfaces. 53 54 LuaJit is much faster at Lua-only instructions. In practice the script would 55 not do something useless counting, but change the text. For example, 56 reindent all the lines: 57 58 ``` vim 59 let totallen = 0 60 for i in range(1, 100000) 61 call setline(i, ' ' .. getline(i)) 62 let totallen += len(getline(i)) 63 endfor 64 ``` 65 66 | how | time in sec | 67 | --------| -------- | 68 | Vim old | 0.578598 | 69 | Python | 0.152040 | 70 | Lua | 0.164917 | 71 | LuaJit | 0.128400 | 72 | Vim new | 0.079692 | 73 74 [These times were measured on a different system by Dominique Pelle] 75 76 The differences are smaller, but Vim 9 script is clearly the fastest. 77 Using LuaJIT is only a little bit faster than plain Lua here, clearly the call 78 back to the Vim code is costly. 79 80 How does Vim9 script work? The function is first compiled into a sequence of 81 instructions. Each instruction has one or two parameters and a stack is 82 used to store intermediate results. Local variables are also on the 83 stack, space is reserved during compilation. This is a fairly normal 84 way of compilation into an intermediate format, specialized for Vim, 85 e.g. each stack item is a typeval_T. And one of the instructions is 86 "execute Ex command", for commands that are not compiled. 87 88 89 ## 2. DEPRIORITIZE INTERFACES 90 91 Attempts have been made to implement functionality with built-in script 92 languages such as Python, Perl, Lua, Tcl and Ruby. This never gained much 93 foothold, for various reasons. 94 95 Instead of using script language support in Vim: 96 * Encourage implementing external tools in any language and communicate 97 with them. The job and channel support already makes this possible. 98 Really any language can be used, also Java and Go, which are not 99 available built-in. 100 * No priority for the built-in language interfaces. They will have to be kept 101 for backwards compatibility, but many users won't need a Vim build with these 102 interfaces. 103 * Improve the Vim script language, it is used to communicate with the external 104 tool and implements the Vim side of the interface. Also, it can be used when 105 an external tool is undesired. 106 107 Altogether this creates a clear situation: Vim with the +eval feature 108 will be sufficient for most plugins, while some plugins require 109 installing a tool that can be written in any language. No confusion 110 about having Vim but the plugin not working because some specific 111 language is missing. This is a good long term goal. 112 113 Rationale: Why is it better to run a tool separately from Vim than using a 114 built-in interface and interpreter? Take for example something that is 115 written in Python: 116 * The built-in interface uses the embedded python interpreter. This is less 117 well maintained than the python command. Building Vim with it requires 118 installing developer packages. If loaded dynamically there can be a version 119 mismatch. 120 * When running the tool externally the standard python command can be used, 121 which is quite often available by default or can be easily installed. 122 * The built-in interface has an API that is unique for Vim with Python. This is 123 an extra API to learn. 124 * A .py file can be compiled into a .pyc file and execute much faster. 125 * Inside Vim multi-threading can cause problems, since the Vim core is single 126 threaded. In an external tool there are no such problems. 127 * The Vim part is written in .vim files, the Python part is in .py files, this 128 is nicely separated. 129 * Disadvantage: An interface needs to be made between Vim and Python. 130 JSON is available for this, and it's fairly easy to use. But it still 131 requires implementing asynchronous communication. 132 133 134 ## 3. BETTER VIM SCRIPT 135 136 To make Vim faster a new way of defining a function needs to be added. 137 While we are doing that, since the lines in this function won't be fully 138 backwards compatible anyway, we can also make Vim script easier to use. 139 In other words: "less weird". Making it work more like modern 140 programming languages will help. No surprises. 141 142 A good example is how in a function the arguments are prefixed with 143 "a:". No other language I know does that, so let's drop it. 144 145 Taking this one step further is also dropping "s:" for script-local variables; 146 everything at the script level is script-local by default. Since this is not 147 backwards compatible it requires a new script style: Vim9 script! 148 149 To avoid having more variations, the syntax inside a compiled function is the 150 same as in Vim9 script. Thus you have legacy syntax and Vim9 syntax. 151 152 It should be possible to convert code from other languages to Vim 153 script. We can add functionality to make this easier. This still needs 154 to be discussed, but we can consider adding type checking and a simple 155 form of classes. If you look at JavaScript for example, it has gone 156 through these stages over time, adding real class support and now 157 TypeScript adds type checking. But we'll have to see how much of that 158 we actually want to include in Vim script. Ideally a conversion tool 159 can take Python, JavaScript or TypeScript code and convert it to Vim 160 script, with only some things that cannot be converted. 161 162 Vim script won't work the same as any specific language, but we can use 163 mechanisms that are commonly known, ideally with the same syntax. One 164 thing I have been thinking of is assignments without ":let". I often 165 make that mistake (after writing JavaScript especially). I think it is 166 possible, if we make local variables shadow commands. That should be OK, 167 if you shadow a command you want to use, just rename the variable. 168 Using "var" and "const" to declare a variable, like in JavaScript and 169 TypeScript, can work: 170 171 172 ``` vim 173 def MyFunction(arg: number): number 174 var local = 1 175 var todo = arg 176 const ADD = 88 177 while todo > 0 178 local += ADD 179 todo -= 1 180 endwhile 181 return local 182 enddef 183 ``` 184 185 The similarity with JavaScript/TypeScript can also be used for dependencies 186 between files. Vim currently uses the `:source` command, which has several 187 disadvantages: 188 * In the sourced script, is not clear what it provides. By default all 189 functions are global and can be used elsewhere. 190 * In a script that sources other scripts, it is not clear what function comes 191 from what sourced script. Finding the implementation is a hassle. 192 * Prevention of loading the whole script twice must be manually implemented. 193 194 We can use the `:import` and `:export` commands from the JavaScript standard to 195 make this much better. For example, in script "myfunction.vim" define a 196 function and export it: 197 198 ``` vim 199 vim9script " Vim9 script syntax used here 200 201 var local = 'local variable is not exported, script-local' 202 203 export def MyFunction() " exported function 204 ... 205 206 def LocalFunction() " not exported, script-local 207 ... 208 ``` 209 210 And in another script import the function: 211 212 ``` vim 213 vim9script " Vim9 script syntax used here 214 215 import MyFunction from 'myfunction.vim' 216 ``` 217 218 This looks like JavaScript/TypeScript, thus many users will understand the 219 syntax. 220 221 These are ideas, this will take time to design, discuss and implement. 222 Eventually this will lead to Vim 9! 223 224 225 ## Code for sum time measurements 226 227 Vim was built with -O2. 228 229 ``` vim 230 func VimOld() 231 let sum = 0 232 for i in range(1, 2999999) 233 let sum += i 234 endfor 235 return sum 236 endfunc 237 238 func Python() 239 py3 << END 240 sum = 0 241 for i in range(1, 3000000): 242 sum += i 243 END 244 return py3eval('sum') 245 endfunc 246 247 func Lua() 248 lua << END 249 sum = 0 250 for i = 1, 2999999 do 251 sum = sum + i 252 end 253 END 254 return luaeval('sum') 255 endfunc 256 257 def VimNew(): number 258 var sum = 0 259 for i in range(1, 2999999) 260 sum += i 261 endfor 262 return sum 263 enddef 264 265 let start = reltime() 266 echo VimOld() 267 echo 'Vim old: ' .. reltimestr(reltime(start)) 268 269 let start = reltime() 270 echo Python() 271 echo 'Python: ' .. reltimestr(reltime(start)) 272 273 let start = reltime() 274 echo Lua() 275 echo 'Lua: ' .. reltimestr(reltime(start)) 276 277 let start = reltime() 278 echo VimNew() 279 echo 'Vim new: ' .. reltimestr(reltime(start)) 280 ``` 281 282 ## Code for indent time measurements 283 284 ``` vim 285 def VimNew(): number 286 var totallen = 0 287 for i in range(1, 100000) 288 setline(i, ' ' .. getline(i)) 289 totallen += len(getline(i)) 290 endfor 291 return totallen 292 enddef 293 294 func VimOld() 295 let totallen = 0 296 for i in range(1, 100000) 297 call setline(i, ' ' .. getline(i)) 298 let totallen += len(getline(i)) 299 endfor 300 return totallen 301 endfunc 302 303 func Lua() 304 lua << END 305 b = vim.buffer() 306 totallen = 0 307 for i = 1, 100000 do 308 b[i] = " " .. b[i] 309 totallen = totallen + string.len(b[i]) 310 end 311 END 312 return luaeval('totallen') 313 endfunc 314 315 func Python() 316 py3 << END 317 cb = vim.current.buffer 318 totallen = 0 319 for i in range(0, 100000): 320 cb[i] = ' ' + cb[i] 321 totallen += len(cb[i]) 322 END 323 return py3eval('totallen') 324 endfunc 325 326 new 327 call setline(1, range(100000)) 328 let start = reltime() 329 echo VimOld() 330 echo 'Vim old: ' .. reltimestr(reltime(start)) 331 bwipe! 332 333 new 334 call setline(1, range(100000)) 335 let start = reltime() 336 echo Python() 337 echo 'Python: ' .. reltimestr(reltime(start)) 338 bwipe! 339 340 new 341 call setline(1, range(100000)) 342 let start = reltime() 343 echo Lua() 344 echo 'Lua: ' .. reltimestr(reltime(start)) 345 bwipe! 346 347 new 348 call setline(1, range(100000)) 349 let start = reltime() 350 echo VimNew() 351 echo 'Vim new: ' .. reltimestr(reltime(start)) 352 bwipe! 353 ```