vim

Fork of vim, an acceptable text editor
git clone git://git.laack.co/vim.git
Log | Files | Refs | README | LICENSE

README_VIM9.md (10833B)


      1 ![Vim Logo](https://github.com/vim/vim/blob/master/runtime/vimlogo.gif)
      2 
      3 # What is Vim9?
      4 
      5 This is a new syntax for Vim script that was introduced with Vim 9.0.
      6 It intends making Vim script faster and better.
      7 
      8 
      9 # Why Vim9?
     10 
     11 ## 1. FASTER VIM SCRIPT
     12 
     13 The third item on the poll results of 2018, after popup windows and text
     14 properties, both of which have been implemented, is faster Vim script.
     15 So how do we do that?
     16 
     17 I have been throwing some ideas around, and soon came to the conclusion
     18 that the current way functions are called and executed, with
     19 dictionaries for the arguments and local variables, is never going to be
     20 very fast.  We're lucky if we can make it twice as fast.  The overhead
     21 of a function call and executing every line is just too high.
     22 
     23 So what then?  We can only make something fast by having a new way of
     24 defining a function, with similar but different properties of the old
     25 way:
     26 * Arguments are only available by name, not through the a: dictionary or
     27   the a:000 list.
     28 * Local variables are not available in an l: dictionary.
     29 * A few more things that slow us down, such as exception handling details.
     30 
     31 I implemented a "proof of concept" and measured the time to run a simple
     32 for loop with an addition (Justin used this example in his presentation,
     33 full code is below):
     34 
     35 ``` vim
     36   let sum = 0
     37   for i in range(1, 2999999)
     38     let sum += i
     39   endfor
     40 ```
     41 
     42 | how     | time in sec |
     43 | --------| -------- |
     44 | Vim old | 5.018541 |
     45 | Python  | 0.369598 |
     46 | Lua     | 0.078817 |
     47 | LuaJit  | 0.004245 |
     48 | Vim new | 0.073595 |
     49 
     50 That looks very promising!  It's just one example, but it shows how much
     51 we can gain, and also that Vim script can be faster than builtin
     52 interfaces.
     53 
     54 LuaJit is much faster at Lua-only instructions.  In practice the script would
     55 not do something useless counting, but change the text.  For example,
     56 reindent all the lines:
     57 
     58 ``` vim
     59   let totallen = 0
     60   for i in range(1, 100000)
     61     call setline(i, '    ' .. getline(i))
     62     let totallen += len(getline(i))
     63   endfor
     64 ```
     65 
     66 | how     | time in sec |
     67 | --------| -------- |
     68 | Vim old | 0.578598 |
     69 | Python  | 0.152040 |
     70 | Lua     | 0.164917 |
     71 | LuaJit  | 0.128400 |
     72 | Vim new | 0.079692 |
     73 
     74 [These times were measured on a different system by Dominique Pelle]
     75 
     76 The differences are smaller, but Vim 9 script is clearly the fastest.
     77 Using LuaJIT is only a little bit faster than plain Lua here, clearly the call
     78 back to the Vim code is costly.
     79 
     80 How does Vim9 script work?  The function is first compiled into a sequence of
     81 instructions.  Each instruction has one or two parameters and a stack is
     82 used to store intermediate results.  Local variables are also on the
     83 stack, space is reserved during compilation.  This is a fairly normal
     84 way of compilation into an intermediate format, specialized for Vim,
     85 e.g. each stack item is a typeval_T.  And one of the instructions is
     86 "execute Ex command", for commands that are not compiled.
     87 
     88 
     89 ## 2. DEPRIORITIZE INTERFACES
     90 
     91 Attempts have been made to implement functionality with built-in script
     92 languages such as Python, Perl, Lua, Tcl and Ruby.  This never gained much
     93 foothold, for various reasons.
     94 
     95 Instead of using script language support in Vim:
     96 * Encourage implementing external tools in any language and communicate
     97   with them.  The job and channel support already makes this possible.
     98   Really any language can be used, also Java and Go, which are not
     99   available built-in.
    100 * No priority for the built-in language interfaces.  They will have to be kept
    101   for backwards compatibility, but many users won't need a Vim build with these
    102   interfaces.
    103 * Improve the Vim script language, it is used to communicate with the external
    104   tool and implements the Vim side of the interface.  Also, it can be used when
    105   an external tool is undesired.
    106 
    107 Altogether this creates a clear situation: Vim with the +eval feature
    108 will be sufficient for most plugins, while some plugins require
    109 installing a tool that can be written in any language.  No confusion
    110 about having Vim but the plugin not working because some specific
    111 language is missing.  This is a good long term goal.
    112 
    113 Rationale: Why is it better to run a tool separately from Vim than using a
    114 built-in interface and interpreter?  Take for example something that is
    115 written in Python:
    116 * The built-in interface uses the embedded python interpreter.  This is less
    117   well maintained than the python command.  Building Vim with it requires
    118   installing developer packages.  If loaded dynamically there can be a version
    119   mismatch.
    120 * When running the tool externally the standard python command can be used,
    121   which is quite often available by default or can be easily installed.
    122 * The built-in interface has an API that is unique for Vim with Python. This is
    123   an extra API to learn.
    124 * A .py file can be compiled into a .pyc file and execute much faster.
    125 * Inside Vim multi-threading can cause problems, since the Vim core is single
    126   threaded.  In an external tool there are no such problems.
    127 * The Vim part is written in .vim files, the Python part is in .py files, this
    128   is nicely separated.
    129 * Disadvantage: An interface needs to be made between Vim and Python.
    130   JSON is available for this, and it's fairly easy to use.  But it still
    131   requires implementing asynchronous communication.
    132 
    133 
    134 ## 3. BETTER VIM SCRIPT
    135 
    136 To make Vim faster a new way of defining a function needs to be added.
    137 While we are doing that, since the lines in this function won't be fully
    138 backwards compatible anyway, we can also make Vim script easier to use.
    139 In other words: "less weird".  Making it work more like modern
    140 programming languages will help.  No surprises.
    141 
    142 A good example is how in a function the arguments are prefixed with
    143 "a:". No other language I know does that, so let's drop it.
    144 
    145 Taking this one step further is also dropping "s:" for script-local variables;
    146 everything at the script level is script-local by default.  Since this is not
    147 backwards compatible it requires a new script style: Vim9 script!
    148 
    149 To avoid having more variations, the syntax inside a compiled function is the
    150 same as in Vim9 script.  Thus you have legacy syntax and Vim9 syntax.
    151 
    152 It should be possible to convert code from other languages to Vim
    153 script.  We can add functionality to make this easier.  This still needs
    154 to be discussed, but we can consider adding type checking and a simple
    155 form of classes.  If you look at JavaScript for example, it has gone
    156 through these stages over time, adding real class support and now
    157 TypeScript adds type checking.  But we'll have to see how much of that
    158 we actually want to include in Vim script.  Ideally a conversion tool
    159 can take Python, JavaScript or TypeScript code and convert it to Vim
    160 script, with only some things that cannot be converted.
    161 
    162 Vim script won't work the same as any specific language, but we can use
    163 mechanisms that are commonly known, ideally with the same syntax.  One
    164 thing I have been thinking of is assignments without ":let".  I often
    165 make that mistake (after writing JavaScript especially).  I think it is
    166 possible, if we make local variables shadow commands.  That should be OK,
    167 if you shadow a command you want to use, just rename the variable.
    168 Using "var" and "const" to declare a variable, like in JavaScript and
    169 TypeScript, can work:
    170 
    171 
    172 ``` vim
    173 def MyFunction(arg: number): number
    174    var local = 1
    175    var todo = arg
    176    const ADD = 88
    177    while todo > 0
    178       local += ADD
    179       todo -= 1
    180    endwhile
    181    return local
    182 enddef
    183 ```
    184 
    185 The similarity with JavaScript/TypeScript can also be used for dependencies
    186 between files.  Vim currently uses the `:source` command, which has several
    187 disadvantages:
    188 *   In the sourced script, is not clear what it provides.  By default all
    189     functions are global and can be used elsewhere.
    190 *   In a script that sources other scripts, it is not clear what function comes
    191     from what sourced script.  Finding the implementation is a hassle.
    192 *   Prevention of loading the whole script twice must be manually implemented.
    193 
    194 We can use the `:import` and `:export` commands from the JavaScript standard to
    195 make this much better.  For example, in script "myfunction.vim" define a
    196 function and export it:
    197 
    198 ``` vim
    199 vim9script  " Vim9 script syntax used here
    200 
    201 var local = 'local variable is not exported, script-local'
    202 
    203 export def MyFunction()  " exported function
    204 ...
    205 
    206 def LocalFunction() " not exported, script-local
    207 ...
    208 ```
    209 
    210 And in another script import the function:
    211 
    212 ``` vim
    213 vim9script  " Vim9 script syntax used here
    214 
    215 import MyFunction from 'myfunction.vim'
    216 ```
    217 
    218 This looks like JavaScript/TypeScript, thus many users will understand the
    219 syntax.
    220 
    221 These are ideas, this will take time to design, discuss and implement.
    222 Eventually this will lead to Vim 9!
    223 
    224 
    225 ## Code for sum time measurements
    226 
    227 Vim was built with -O2.
    228 
    229 ``` vim
    230 func VimOld()
    231   let sum = 0
    232   for i in range(1, 2999999)
    233     let sum += i
    234   endfor
    235   return sum
    236 endfunc
    237 
    238 func Python()
    239   py3 << END
    240 sum = 0
    241 for i in range(1, 3000000):
    242   sum += i
    243 END
    244   return py3eval('sum')
    245 endfunc
    246 
    247 func Lua()
    248   lua << END
    249     sum = 0
    250     for i = 1, 2999999 do
    251       sum = sum + i
    252     end
    253 END
    254   return luaeval('sum')
    255 endfunc
    256 
    257 def VimNew(): number
    258   var sum = 0
    259   for i in range(1, 2999999)
    260     sum += i
    261   endfor
    262   return sum
    263 enddef
    264 
    265 let start = reltime()
    266 echo VimOld()
    267 echo 'Vim old: ' .. reltimestr(reltime(start))
    268 
    269 let start = reltime()
    270 echo Python()
    271 echo 'Python: ' .. reltimestr(reltime(start))
    272 
    273 let start = reltime()
    274 echo Lua()
    275 echo 'Lua: ' .. reltimestr(reltime(start))
    276 
    277 let start = reltime()
    278 echo VimNew()
    279 echo 'Vim new: ' .. reltimestr(reltime(start))
    280 ```
    281 
    282 ## Code for indent time measurements
    283 
    284 ``` vim
    285 def VimNew(): number
    286   var totallen = 0
    287   for i in range(1, 100000)
    288     setline(i, '    ' .. getline(i))
    289     totallen += len(getline(i))
    290   endfor
    291   return totallen
    292 enddef
    293 
    294 func VimOld()
    295   let totallen = 0
    296   for i in range(1, 100000)
    297     call setline(i, '    ' .. getline(i))
    298     let totallen += len(getline(i))
    299   endfor
    300   return totallen
    301 endfunc
    302 
    303 func Lua()
    304   lua << END
    305     b = vim.buffer()
    306     totallen = 0
    307     for i = 1, 100000 do
    308       b[i] = "    " .. b[i]
    309       totallen = totallen + string.len(b[i])
    310     end
    311 END
    312   return luaeval('totallen')
    313 endfunc
    314 
    315 func Python()
    316   py3 << END
    317 cb = vim.current.buffer
    318 totallen = 0
    319 for i in range(0, 100000):
    320   cb[i] = '    ' + cb[i]
    321   totallen += len(cb[i])
    322 END
    323   return py3eval('totallen')
    324 endfunc
    325 
    326 new
    327 call setline(1, range(100000))
    328 let start = reltime()
    329 echo VimOld()
    330 echo 'Vim old: ' .. reltimestr(reltime(start))
    331 bwipe!
    332 
    333 new
    334 call setline(1, range(100000))
    335 let start = reltime()
    336 echo Python()
    337 echo 'Python: ' .. reltimestr(reltime(start))
    338 bwipe!
    339 
    340 new
    341 call setline(1, range(100000))
    342 let start = reltime()
    343 echo Lua()
    344 echo 'Lua: ' .. reltimestr(reltime(start))
    345 bwipe!
    346 
    347 new
    348 call setline(1, range(100000))
    349 let start = reltime()
    350 echo VimNew()
    351 echo 'Vim new: ' .. reltimestr(reltime(start))
    352 bwipe!
    353 ```