e0259b9466
Problem: getregionpos() wrong with blockwise mode and multibyte.
Solution: Use textcol and textlen instead of start_vcol and end_vcol.
Handle coladd properly (zeertzjq).
Also remove unnecessary buflist_findnr() in add_regionpos_range(), as
getregionpos() has already switched buffer.
closes: vim/vim#14805
|
||
---|---|---|
.. | ||
api | ||
eval | ||
event | ||
generators | ||
lib | ||
lua | ||
msgpack_rpc | ||
os | ||
po | ||
tui | ||
viml/parser | ||
CMakeLists.txt | ||
README.md | ||
arabic.c | ||
arabic.h | ||
arglist.c | ||
arglist.h | ||
arglist_defs.h | ||
ascii_defs.h | ||
assert_defs.h | ||
auevents.lua | ||
autocmd.c | ||
autocmd.h | ||
autocmd_defs.h | ||
base64.c | ||
base64.h | ||
buffer.c | ||
buffer.h | ||
buffer_defs.h | ||
buffer_updates.c | ||
buffer_updates.h | ||
bufwrite.c | ||
bufwrite.h | ||
change.c | ||
change.h | ||
channel.c | ||
channel.h | ||
channel_defs.h | ||
charset.c | ||
charset.h | ||
cmdexpand.c | ||
cmdexpand.h | ||
cmdexpand_defs.h | ||
cmdhist.c | ||
cmdhist.h | ||
context.c | ||
context.h | ||
cursor.c | ||
cursor.h | ||
cursor_shape.c | ||
cursor_shape.h | ||
debugger.c | ||
debugger.h | ||
decoration.c | ||
decoration.h | ||
decoration_defs.h | ||
decoration_provider.c | ||
decoration_provider.h | ||
diff.c | ||
diff.h | ||
digraph.c | ||
digraph.h | ||
drawline.c | ||
drawline.h | ||
drawscreen.c | ||
drawscreen.h | ||
edit.c | ||
edit.h | ||
eval.c | ||
eval.h | ||
eval.lua | ||
eval_defs.h | ||
ex_cmds.c | ||
ex_cmds.h | ||
ex_cmds.lua | ||
ex_cmds2.c | ||
ex_cmds2.h | ||
ex_cmds_defs.h | ||
ex_docmd.c | ||
ex_docmd.h | ||
ex_eval.c | ||
ex_eval.h | ||
ex_eval_defs.h | ||
ex_getln.c | ||
ex_getln.h | ||
ex_getln_defs.h | ||
ex_session.c | ||
ex_session.h | ||
extmark.c | ||
extmark.h | ||
extmark_defs.h | ||
file_search.c | ||
file_search.h | ||
fileio.c | ||
fileio.h | ||
fold.c | ||
fold.h | ||
fold_defs.h | ||
func_attr.h | ||
garray.c | ||
garray.h | ||
garray_defs.h | ||
getchar.c | ||
getchar.h | ||
getchar_defs.h | ||
gettext_defs.h | ||
globals.h | ||
grid.c | ||
grid.h | ||
grid_defs.h | ||
hashtab.c | ||
hashtab.h | ||
hashtab_defs.h | ||
help.c | ||
help.h | ||
highlight.c | ||
highlight.h | ||
highlight_defs.h | ||
highlight_group.c | ||
highlight_group.h | ||
iconv_defs.h | ||
indent.c | ||
indent.h | ||
indent_c.c | ||
indent_c.h | ||
input.c | ||
input.h | ||
insexpand.c | ||
insexpand.h | ||
keycodes.c | ||
keycodes.h | ||
linematch.c | ||
linematch.h | ||
log.c | ||
log.h | ||
macros_defs.h | ||
main.c | ||
main.h | ||
map.c | ||
map_defs.h | ||
map_glyph_cache.c | ||
map_key_impl.c.h | ||
map_value_impl.c.h | ||
mapping.c | ||
mapping.h | ||
mapping_defs.h | ||
mark.c | ||
mark.h | ||
mark_defs.h | ||
marktree.c | ||
marktree.h | ||
marktree_defs.h | ||
match.c | ||
match.h | ||
math.c | ||
math.h | ||
mbyte.c | ||
mbyte.h | ||
mbyte_defs.h | ||
memfile.c | ||
memfile.h | ||
memfile_defs.h | ||
memline.c | ||
memline.h | ||
memline_defs.h | ||
memory.c | ||
memory.h | ||
memory_defs.h | ||
menu.c | ||
menu.h | ||
menu_defs.h | ||
message.c | ||
message.h | ||
message_defs.h | ||
mouse.c | ||
mouse.h | ||
move.c | ||
move.h | ||
normal.c | ||
normal.h | ||
normal_defs.h | ||
ops.c | ||
ops.h | ||
option.c | ||
option.h | ||
option_defs.h | ||
option_vars.h | ||
options.lua | ||
optionstr.c | ||
optionstr.h | ||
path.c | ||
path.h | ||
plines.c | ||
plines.h | ||
popupmenu.c | ||
popupmenu.h | ||
pos_defs.h | ||
profile.c | ||
profile.h | ||
quickfix.c | ||
quickfix.h | ||
rbuffer.c | ||
rbuffer.h | ||
rbuffer_defs.h | ||
regexp.c | ||
regexp.h | ||
regexp_defs.h | ||
runtime.c | ||
runtime.h | ||
runtime_defs.h | ||
search.c | ||
search.h | ||
sha256.c | ||
sha256.h | ||
shada.c | ||
shada.h | ||
sign.c | ||
sign.h | ||
sign_defs.h | ||
spell.c | ||
spell.h | ||
spell_defs.h | ||
spellfile.c | ||
spellfile.h | ||
spellsuggest.c | ||
spellsuggest.h | ||
state.c | ||
state.h | ||
state_defs.h | ||
statusline.c | ||
statusline.h | ||
statusline_defs.h | ||
strings.c | ||
strings.h | ||
syntax.c | ||
syntax.h | ||
syntax_defs.h | ||
tag.c | ||
tag.h | ||
terminal.c | ||
terminal.h | ||
testing.c | ||
testing.h | ||
textformat.c | ||
textformat.h | ||
textobject.c | ||
textobject.h | ||
types_defs.h | ||
ugrid.c | ||
ugrid.h | ||
ui.c | ||
ui.h | ||
ui_client.c | ||
ui_client.h | ||
ui_compositor.c | ||
ui_compositor.h | ||
ui_defs.h | ||
undo.c | ||
undo.h | ||
undo_defs.h | ||
usercmd.c | ||
usercmd.h | ||
version.c | ||
version.h | ||
vim_defs.h | ||
vvars.lua | ||
window.c | ||
window.h | ||
winfloat.c | ||
winfloat.h |
README.md
Nvim core
Module-specific details are documented at the top of each module (terminal.c
, undo.c
, …).
See :help dev
for guidelines.
Filename conventions
The source files use extensions to hint about their purpose.
*.c
,*.generated.c
- full C files, with all includes, etc.*.c.h
- parametrized C files, contain all necessary includes, but require defining macros before actually using. Example:typval_encode.c.h
*.h
- full headers, with all includes. Does not apply to*.generated.h
.*.h.generated.h
- exported functions’ declarations.*.c.generated.h
- static functions’ declarations.
Common structures
- StringBuilder
- kvec or garray.c for dynamic lists / vectors (use StringBuilder for strings)
Logs
Low-level log messages sink to $NVIM_LOG_FILE
.
UI events are logged at DEBUG level.
rm -rf build/
make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG"
Use LOG_CALLSTACK()
(Linux only) to log the current stacktrace. To log to an
alternate file (e.g. stderr) use LOG_CALLSTACK_TO_FILE(FILE*)
. Requires
-no-pie
(ref):
rm -rf build/
make CMAKE_EXTRA_FLAGS="-DLOG_DEBUG -DCMAKE_C_FLAGS=-no-pie"
Many log messages have a shared prefix, such as "UI" or "RPC". Use the shell to filter the log, e.g. at DEBUG level you might want to exclude UI messages:
tail -F ~/.local/state/nvim/log | cat -v | stdbuf -o0 grep -v UI | stdbuf -o0 tee -a log
Build with ASAN
Building Nvim with Clang sanitizers (Address Sanitizer: ASan, Undefined Behavior Sanitizer: UBSan, Memory Sanitizer: MSan, Thread Sanitizer: TSan) is a good way to catch undefined behavior, leaks and other errors as soon as they happen. It's significantly faster than Valgrind.
Requires clang 3.4 or later, and llvm-symbolizer
must be in $PATH
:
clang --version
Build Nvim with sanitizer instrumentation (choose one):
CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_ASAN_UBSAN=ON"
CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_MSAN=ON"
CC=clang make CMAKE_EXTRA_FLAGS="-DENABLE_TSAN=ON"
Create a directory to store logs:
mkdir -p "$HOME/logs"
Configure the sanitizer(s) via these environment variables:
# Change to detect_leaks=1 to detect memory leaks (slower, noisier).
export ASAN_OPTIONS="detect_leaks=0:log_path=$HOME/logs/asan"
# Show backtraces in the logs.
export MSAN_OPTIONS="log_path=${HOME}/logs/msan"
export TSAN_OPTIONS="log_path=${HOME}/logs/tsan"
Logs will be written to ${HOME}/logs/*san.PID
then.
For more information: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags
Reproducible build
To make a reproducible build of Nvim, set cmake variable LUA_GEN_PRG
to
a LuaJIT binary built with LUAJIT_SECURITY_PRN=0
. See commit
cb757f2663
.
Debug: Performance
Profiling (easy)
For debugging performance bottlenecks in any code, there is a simple (and very effective) approach:
- Run the slow code in a loop.
- Break execution ~5 times and save the stacktrace.
- The cause of the bottleneck will (almost always) appear in most of the stacktraces.
Profiling (fancy)
For more advanced profiling, consider perf
+ flamegraph
.
USDT profiling (powerful)
Or you can use USDT probes via NVIM_PROBE
(#12036).
USDT is basically a way to define stable probe points in userland binaries. The benefit of bcc is the ability to define logic to go along with the probe points.
Tools:
- bpftrace provides an awk-like language to the kernel bytecode, BPF.
- BCC provides a subset of C. Provides more complex logic than bpftrace, but takes a bit more effort.
Example using bpftrace to track slow vim functions, and print out any files that were opened during the trace. At the end, it prints a histogram of function timing:
#!/usr/bin/env bpftrace
BEGIN {
@depth = -1;
}
tracepoint:sched:sched_process_fork /@pidmap[args->parent_pid]/ {
@pidmap[args->child_pid] = 1;
}
tracepoint:sched:sched_process_exit /@pidmap[args->pid]/ {
delete(@pidmap[args->pid]);
}
usdt:build/bin/nvim:neovim:eval__call_func__entry {
@pidmap[pid] = 1;
@depth++;
@funcentry[@depth] = nsecs;
}
usdt:build/bin/nvim:neovim:eval__call_func__return {
$func = str(arg0);
$msecs = (nsecs - @funcentry[@depth]) / 1000000;
@time_histo = hist($msecs);
if ($msecs >= 1000) {
printf("%u ms for %s\n", $msecs, $func);
print(@files);
}
clear(@files);
delete(@funcentry[@depth]);
@depth--;
}
tracepoint:syscalls:sys_enter_open,
tracepoint:syscalls:sys_enter_openat {
if (@pidmap[pid] == 1 && @depth >= 0) {
@files[str(args->filename)] = count();
}
}
END {
clear(@depth);
}
$ sudo bpftrace funcslower.bt
1527 ms for Slower
@files[/usr/lib/libstdc++.so.6]: 2
@files[/etc/fish/config.fish]: 2
<snip>
^C
@time_histo:
[0] 71430 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1] 346 | |
[2, 4) 208 | |
[4, 8) 91 | |
[8, 16) 22 | |
[16, 32) 85 | |
[32, 64) 7 | |
[64, 128) 0 | |
[128, 256) 0 | |
[256, 512) 6 | |
[512, 1K) 1 | |
[1K, 2K) 5 | |
Debug: TUI
TUI troubleshoot
Nvim logs its internal terminfo state at 'verbose' level 3. This makes it possible to see exactly what terminfo values Nvim is using on any system.
nvim -V3log
TUI Debugging with gdb/lldb
Launching the nvim TUI involves two processes, one for main editor state and one for rendering the TUI. Both of these processes use the nvim binary, so somewhat confusingly setting a breakpoint in either will generally succeed but may not be hit depending on which process the breakpoints were set in.
To debug the main process, you can debug the nvim binary with the --headless
flag which does not launch the TUI and will allow you to set breakpoints in code
not related to TUI rendering like so:
lldb -- ./build/bin/nvim --headless --listen ~/.cache/nvim/debug-server.pipe
While in lldb, enter run
. You can then attach to the headless process in a
new terminal window to interact with the editor like so:
./build/bin/nvim --remote-ui --server ~/.cache/nvim/debug-server.pipe
Conversely for debugging TUI rendering, you can start a headless process and debug the remote-ui process multiple times without losing editor state.
For details on using nvim-dap and automatically debugging the child (main) process, see here
TUI trace
The ancient script
command is still the "state of the art" for tracing
terminal behavior. The libvterm vterm-dump
utility formats the result for
human-readability.
Record a Nvim terminal session and format it with vterm-dump
:
script foo
./build/bin/nvim -u NONE
# Exit the script session with CTRL-d
# Use `vterm-dump` utility to format the result.
./.deps/usr/bin/vterm-dump foo > bar
Then you can compare bar
with another session, to debug TUI behavior.
TUI redraw
Set the 'writedelay' and 'redrawdebug' options to see where and when the UI is painted.
:set writedelay=50 rdb=compositor
Note: neovim uses an internal screenbuffer to only send minimal updates even if a large region is repainted internally. To also highlight excess internal redraws, use
:set writedelay=50 rdb=compositor,nodelta
Terminal reference
man terminfo
- http://bazaar.launchpad.net/~libvterm/libvterm/trunk/view/head:/doc/seqs.txt
- http://invisible-island.net/xterm/ctlseqs/ctlseqs.html
Data structures
Buffer text is stored as a tree of line segments, defined in memline.c. The central idea is found in ml_find_line.
Nvim lifecycle
Following describes how Nvim processes input.
Consider a typical Vim-like editing session:
- Vim displays the welcome screen
- User types:
:
- Vim enters command-line mode
- User types:
edit README.txt<CR>
- Vim opens the file and returns to normal mode
- User types:
G
- Vim navigates to the end of the file
- User types:
5
- Vim enters count-pending mode
- User types:
d
- Vim enters operator-pending mode
- User types:
w
- Vim deletes 5 words
- User types:
g
- Vim enters the "g command mode"
- User types:
g
- Vim goes to the beginning of the file
- User types:
i
- Vim enters insert mode
- User types:
word<ESC>
- Vim inserts "word" at the beginning and returns to normal mode
Note that we split user actions into sequences of inputs that change the state of the editor. While there's no documentation about a "g command mode" (step 16), internally it is implemented similarly to "operator-pending mode".
From this we can see that Vim has the behavior of an input-driven state machine (more specifically, a pushdown automaton since it requires a stack for transitioning back from states). Assuming each state has a callback responsible for handling keys, this pseudocode represents the main program loop:
def state_enter(state_callback, data):
do
key = readkey() # read a key from the user
while state_callback(data, key) # invoke the callback for the current state
That is, each state is entered by calling state_enter
and passing a
state-specific callback and data. Here is a high-level pseudocode for a program
that implements something like the workflow described above:
def main()
state_enter(normal_state, {}):
def normal_state(data, key):
if key == ':':
state_enter(command_line_state, {})
elif key == 'i':
state_enter(insert_state, {})
elif key == 'd':
state_enter(delete_operator_state, {})
elif key == 'g':
state_enter(g_command_state, {})
elif is_number(key):
state_enter(get_operator_count_state, {'count': key})
elif key == 'G'
jump_to_eof()
return true
def command_line_state(data, key):
if key == '<cr>':
if data['input']:
execute_ex_command(data['input'])
return false
elif key == '<esc>'
return false
if not data['input']:
data['input'] = ''
data['input'] += key
return true
def delete_operator_state(data, key):
count = data['count'] or 1
if key == 'w':
delete_word(count)
elif key == '$':
delete_to_eol(count)
return false # return to normal mode
def g_command_state(data, key):
if key == 'g':
go_top()
elif key == 'v':
reselect()
return false # return to normal mode
def get_operator_count_state(data, key):
if is_number(key):
data['count'] += key
return true
unshift_key(key) # return key to the input buffer
state_enter(delete_operator_state, data)
return false
def insert_state(data, key):
if key == '<esc>':
return false # exit insert mode
self_insert(key)
return true
The above gives an idea of how Nvim is organized internally. Some states like
the g_command_state
or get_operator_count_state
do not have a dedicated
state_enter
callback, but are implicitly embedded into other states (this
will change later as we continue the refactoring effort). To start reading the
actual code, here's the recommended order:
state_enter()
function (state.c). This is the actual program loop, note that aVimState
structure is used, which contains function pointers for the callback and state data.main()
function (main.c). After all startup,normal_enter
is called at the end of function to enter normal mode.normal_enter()
function (normal.c) is a small wrapper for setting up the NormalState structure and callingstate_enter
.normal_check()
function (normal.c) is called before each iteration of normal mode.normal_execute()
function (normal.c) is called when a key is read in normal mode.
The basic structure described for normal mode in 3, 4 and 5 is used for other
modes managed by the state_enter
loop:
- command-line mode:
command_line_{enter,check,execute}()
(ex_getln.c
) - insert mode:
insert_{enter,check,execute}()
(edit.c
) - terminal mode:
terminal_{enter,execute}()
(terminal.c
)
Important variables
The current mode is stored in State
. The values it can have are MODE_NORMAL
,
MODE_INSERT
, MODE_CMDLINE
, and a few others.
The current window is curwin
. The current buffer is curbuf
. These point
to structures with the cursor position in the window, option values, the file
name, etc.
All the global variables are declared in globals.h
.
The main loop
The main loop is implemented in state_enter. The basic idea is that Vim waits
for the user to type a character and processes it until another character is
needed. Thus there are several places where Vim waits for a character to be
typed. The vgetc()
function is used for this. It also handles mapping.
Updating the screen is mostly postponed until a command or a sequence of
commands has finished. The work is done by update_screen()
, which calls
win_update()
for every window, which calls win_line()
for every line.
See the start of drawscreen.c for more explanations.
Command-line mode
When typing a :
, normal_cmd()
will call getcmdline()
to obtain a line with
an Ex command. getcmdline()
calls a loop that will handle each typed
character. It returns when hitting <CR>
or <Esc>
or some other character that
ends the command line mode.
Ex commands
Ex commands are handled by the function do_cmdline()
. It does the generic
parsing of the :
command line and calls do_one_cmd()
for each separate
command. It also takes care of while loops.
do_one_cmd()
parses the range and generic arguments and puts them in the
exarg_t and passes it to the function that handles the command.
The :
commands are listed in ex_cmds.lua.
Normal mode commands
The Normal mode commands are handled by the normal_cmd()
function. It also
handles the optional count and an extra character for some commands. These
are passed in a cmdarg_T
to the function that handles the command.
There is a table nv_cmds
in normal.c which
lists the first character of every
command. The second entry of each item is the name of the function that
handles the command.
Insert mode commands
When doing an i
or a
command, normal_cmd()
will call the edit()
function.
It contains a loop that waits for the next character and handles it. It
returns when leaving Insert mode.
Options
There is a list with all option names in options.lua.
Async event support
One of the features Nvim added is the support for handling arbitrary asynchronous events, which can include:
- RPC requests
- job control callbacks
- timers
Nvim implements this functionality by entering another event loop while waiting for characters, so instead of:
def state_enter(state_callback, data):
do
key = readkey() # read a key from the user
while state_callback(data, key) # invoke the callback for the current state
Nvim program loop is more like:
def state_enter(state_callback, data):
do
event = read_next_event() # read an event from the operating system
while state_callback(data, event) # invoke the callback for the current state
where event
is something the operating system delivers to us, including (but
not limited to) user input. The read_next_event()
part is internally
implemented by libuv, the platform layer used by Nvim.
Since Nvim inherited its code from Vim, the states are not prepared to receive "arbitrary events", so we use a special key to represent those (When a state receives an "arbitrary event", it normally doesn't do anything other than update the screen).
Main loop
The Loop
structure (which describes main_loop
) abstracts multiple queues
into one loop:
uv_loop_t uv;
MultiQueue *events;
MultiQueue *thread_events;
MultiQueue *fast_events;
loop_poll_events
checks Loop.uv
and Loop.fast_events
whenever Nvim is
idle, and also at os_breakcheck
intervals.
MultiQueue is cool because you can attach throw-away "child queues" trivially.
For example do_os_system()
does this (for every spawned process!) to
automatically route events onto the main_loop
:
Process *proc = &uvproc.process;
MultiQueue *events = multiqueue_new_child(main_loop.events);
proc->events = events;