The Rabbit Hole of Multi-line URLs
I'm a long time user of weechat. I don't really use IRC anymore, but I use it to connect to bitlbee for IMs, slack, and so on.
I really like weechat. But for a long time, I've had strange screen corruption problems, such as in the second to last line of:
That's a mild example. Sometimes weechat is all but unusable. Now, I can press Ctrl+L to redraw the screen and make the corruption go away. But it really bugged me one day, so I started investigating.
It took me a while, but eventually I realized that running weechat with a fresh configuration did not exhibit the problem. From there, it was straight-forward (but annoying) to isolate the problematic configuration option. It turned out to be an option called eat_newline_glitch.
According to the weechat user manual:
if set, the eat_newline_glitch will be set to 0; this is used to not add new line char at end of each line, and then not break text when you copy/paste text from WeeChat to another application (this option is disabled by default because it can cause serious display bugs)
If eat_newline_glitch is not turned on, then links to wrap from one line to another are not properly detected by gnome-terminal. It will detect the portion on the first line as a link only. In the modern days of long, automatically generated URLs, this is really annoying to say the least.
Anyway, at some point I must have turned eat_newline_glitch on and
forgot about it, and never associated it with the screen corruption.
I could have just turned the option off, but I obsess over things
like this like to know why things don't work. ๐
The first problem was being able to replicate the problem on demand. I eventually realized that the problem seemed to be related to lines that are exactly the width of the terminal. After some fighting with weechat scripting, I managed to write the following script which prints a command-line that reliably triggers the problem for me:
#!/usr/bin/env python
import subprocess
import time
prefixlen = len("13:52:08 | ")
collen = int(subprocess.check_output("tput cols", shell=True))
n = collen - prefixlen
s = "A" * n
commands = []
commands.append("/set weechat.look.eat_newline_glitch on")
commands.append("/bar hide buflist")
commands.append("/print %s" % s)
commands.append("/print %s" % s)
for _ in range(20):
commands.append("/exec -buffer new echo This text should not be visible in buffer 1")
commands.append("/buffer exec.new")
commands.append("/wait 1 /buffer 1")
print ("weechat -d /tmp -r '%s'" % "; ".join(commands))The problem is pretty easy to spot. If you're in buffer 1 and you see "This text should not be visible in buffer 1", the bug was present! Here's an example:
Now that I could reliably replicate the problem, how to figure out what was going on? Weechat uses the ncurses library to write the UI to the screen. And interestingly, eat_newline_glitch corresponds to a capability name in terminfo, which is used by ncurses to draw the screen. The official description is not very helpful:
Newline ignored after 80 columns (Concept)
After perusing the ncurses source code and documentation, I found that it has a trace mode. Since ncurses maintains an internal representation of the terminal, it can print out what it thinks the screen should look like.
newscr[ 0] -1 -1 ='WeeChat 2.9-dev (C) 2003-2020 - https://weechat.org/ '
colors[ 0] ='222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222'
newscr[ 1] -1 -1 ='11:11:30 | ___ __ ______________ _____ '
colors[ 1] ='11611611177kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk '
newscr[ 2] -1 -1 ='11:11:30 | __ | / /___________ ____/__ /_______ __ /_ '
colors[ 2] ='11611611177kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk '
newscr[ 3] -1 -1 ='11:11:30 | __ | /| / /_ _ \ _ \ / __ __ \ __ `/ __/ '
colors[ 3] ='11611611177kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk '
newscr[ 4] -1 -1 ='11:11:30 | __ |/ |/ / / __/ __/ /___ _ / / / /_/ // /_ '
colors[ 4] ='11611611177kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk '
newscr[ 5] -1 -1 ='11:11:30 | ____/|__/ \___/\___/\____/ /_/ /_/\__,_/ \__/ '
colors[ 5] ='11611611177kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk '
newscr[ 6] -1 -1 ='11:11:30 | WeeChat 2.9-dev [compiled on Apr 1 2020 10:32:22] '
colors[ 6] ='1161161117799999999999999997888888888888888888888888888888887 '
newscr[ 7] -1 -1 ='11:11:30 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - '
colors[ 7] ='11611611177111111111111111111111111111111111111111111111111111111111111111 '
newscr[ 8] -1 -1 ='11:11:30 | Plugins loaded: alias, buflist, charset, exec, fifo, fset, irc, logger, python, relay, script, trigger, xfer '
colors[ 8] ='11611611177111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 '
newscr[ 9] -1 -1 ='11:11:30 | WARNING: this option can cause serious display bugs, if you have such problems, you must turn off this option. '
colors[ 9] ='1161161117711111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 '
newscr[10] -1 -1 ='11:11:30 | Option changed: weechat.look.eat_newline_glitch = on (default: off) '
colors[10] ='1161161117711111111111111111111111111111111111111111111111777887771111111118887 '
newscr[11] -1 -1 ='11:11:30 | AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
colors[11] ='116116111771111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[12] -1 -1 ='11:11:30 | AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
colors[12] ='116116111771111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[13] -1 -1 =' '
newscr[14] -1 -1 =' '
colors[14] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[15] -1 -1 =' '
colors[15] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[16] -1 -1 =' '
colors[16] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[17] -1 -1 =' '
colors[17] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[18] -1 -1 =' '
colors[18] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[19] -1 -1 =' '
colors[19] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[20] -1 -1 =' '
colors[20] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[21] -1 -1 =' '
colors[21] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[22] -1 -1 =' '
colors[22] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[23] -1 -1 =' '
colors[23] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[24] -1 -1 =' '
colors[24] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[25] -1 -1 =' '
colors[25] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[26] -1 -1 =' '
colors[26] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[27] -1 -1 =' '
colors[27] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[28] -1 -1 =' '
colors[28] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[29] -1 -1 =' '
colors[29] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[30] -1 -1 =' '
colors[30] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[31] -1 -1 =' '
colors[31] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[32] -1 -1 =' '
colors[32] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'
newscr[33] 5 5 ='[11:14] [2] [core] 1:weechat '
colors[33] ='322222323232322223243555555522222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222'
newscr[34] -1 -1 =' '
colors[34] ='111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111'Ncurses' internal representation of the screen does not show the "This text should not be visible" messages, which implies that something ncurses is doing to update the screen is not having the desired effect.
After digging through the ncurses code some more, I found a very useful define called POSITION_DEBUG:
Enable checking to see if doupdate and friends are tracking the true cursor position correctly. NOTE: this is a debugging hack which will work ONLY on ANSI-compatible terminals!
Since eat_newline_glitch was related to the cursor position, this sounded promising! And indeed, with this option turned on, the trace log contained the following messages:
position seen (10, 188) doesn't match expected one (11, 0) in wrap_cursor
position seen (11, 188) doesn't match expected one (12, 0) in wrap_cursorNot only does it tell us that something is wrong, but also the function it is in, wrap_cursor (with some debug code removed):
/*
* Wrap the cursor position, i.e., advance to the beginning of the next line.
*/
static void
wrap_cursor(NCURSES_SP_DCL0)
{
if (eat_newline_glitch) {
/*
* xenl can manifest two different ways. The vt100 way is that, when
* you'd expect the cursor to wrap, it stays hung at the right margin
* (on top of the character just emitted) and doesn't wrap until the
* *next* graphic char is emitted. The c100 way is to ignore LF
* received just after an am wrap.
*
* An aggressive way to handle this would be to emit CR/LF after the
* char and then assume the wrap is done, you're on the first position
* of the next line, and the terminal out of its weird state. Here
* it's safe to just tell the code that the cursor is in hyperspace and
* let the next mvcur() call straighten things out.
*/
SP_PARM->_curscol = -1;
SP_PARM->_cursrow = -1;
} else if (auto_right_margin) {
SP_PARM->_curscol = 0;
SP_PARM->_cursrow++;
/*
* We've actually moved - but may have to work around problems with
* video attributes not working.
*/
if (!move_standout_mode && AttrOf(SCREEN_ATTRS(SP_PARM))) {
VIDPUTS(SP_PARM, A_NORMAL, 0);
} else {
SP_PARM->_curscol--;
}
position_check(NCURSES_SP_ARGx
SP_PARM->_cursrow,
SP_PARM->_curscol,
"wrap_cursor");
}The explanation of xenl (the capability name for eat_newline_glitch) is the best I've ever seen. Basically, it means that the cursor's location is unknown as soon as the right-most column is output. Ncurses handles this by marking the cursor location as unknown, which then forces a conservative mechanism to reset the cursor location the next time it is moved. OK, that makes sense, but then why are we getting screen corruption when we set eat_newline_glitch in weechat?
Because setting weechat's eat_newline_glitch to x sets ncurses'
eat_newline_glitch to !x. I'm embarrassed to admit this took me
quite a long time to notice.
Ok, so to refresh, I set eat_newline_glitch to true in weechat,
which sets it to false in ncurses. So when we execute wrap_cursor,
we take the branch for else if (auto_right_margin), which assumes
that the terminal moves the cursor to the left-most column of the next
row. Since this is not what gnome-terminal does, screen corruption
results!
So where is the bug? Weechat surely has a bug, in that screen corruption can occur when you set eat_newline_glitch. But they do warn you about this in a very vague way. In my opinion, ncurses does not have a bug. My terminal has weird behavior when reaching the right-most column, and thus it is no surprise that when we effectively tell ncurses the glitch does not exist, it results in screen corruption.
That being said, I think there is some room for improvement. The problem with eat_newline_glitch is that the behavior is ambiguous by definition; it describes two types of behaviors. If ncurses knew the actual behavior of the terminal, it wouldn't need to manually move the cursor or insert a line break, which wouldn't interfere with the terminal's ability to detect a wrapped URL. So one solution would be to create two capabilities based on the two different behaviors that comprise eat_newline_glitch.
Another option would be to query for the cursor's location the first time wrap_cursor is invoked, and use the result to predict the cursor location in the future.
Anyway, that is my long trip down the rabbit hole of multi-line URLs. Unlike most trips of this nature, this one did not have a very satisfying ending. Hopefully I can save at least one other person from going down the same rabbit hole.
Powered with by Gatsby 5.0