bash history

[mick.crane]

In debian bookworm, xfce desktop, different virtual terminals have a
different history if same user presses "up key" in different virtual
terminals ?
Is this something that can be changed so history is shared between
virtual terminals?
mick

[Greg Wooledge]

Post by mick.crane
In debian bookworm, xfce desktop, different virtual terminals have a
different history if same user presses "up key" in different virtual
terminals ?

As your subject says, this is "bash history". And yes, each instance of
bash has its own separate history. It has nothing to do with your
terminal emulator or your desktop environment.

Bash reads its history from a file (~/.bash_history by default) at
startup time, manages it in memory while running, and writes it back
out to the history file upon exiting.

Post by mick.crane
Is this something that can be changed so history is shared between virtual
terminals?

No.

I consider this a *good* thing, personally. I used commerical Unix with
ksh for a very long time, and ksh shares history between instances in
real time. This is a *nightmare*. I really, really disliked it.

In the worst case scenario, you and a coworker might both be logged in
as root (where root's shell is ksh) simultaneously. Your histories
would then intermix in real time. You might press ESC k ENTER in your
terminal, expecting to re-run the command you had just executed a
moment ago, and unexpectedly run your coworker's command.

Even in a "normal" scenario where I'm logged in as myself, and nobody
else has my password, I would not want all of my terminal windows to have
a shared history. Each of them is doing something different. I want
each terminal's workflow to be isolated to that terminal and what I'm
doing in it. I don't want them contaminating each other.

Of course, that's my opinion, and if you want a shared history across
concurrent shell instances, there are probably shells that can do that.
I've never looked for it, because having lived with it, I already know
I don't want it again.

[mick.crane]

Post by Greg Wooledge

Post by mick.crane
In debian bookworm, xfce desktop, different virtual terminals have a
different history if same user presses "up key" in different virtual
terminals ?

As your subject says, this is "bash history". And yes, each instance of
bash has its own separate history. It has nothing to do with your
terminal emulator or your desktop environment.

Where is it if not in ~/.bash_history?

Post by Greg Wooledge
Bash reads its history from a file (~/.bash_history by default) at
startup time, manages it in memory while running, and writes it back
out to the history file upon exiting.

Does this separate history get written to ~/.bash_history when terminal
emulation thing is closed?

[Greg Wooledge]

Post by mick.crane

Post by Greg Wooledge

Post by mick.crane
In debian bookworm, xfce desktop, different virtual terminals have a
different history if same user presses "up key" in different virtual
terminals ?

As your subject says, this is "bash history". And yes, each instance of
bash has its own separate history. It has nothing to do with your
terminal emulator or your desktop environment.

Where is it if not in ~/.bash_history?

Then it's wherever you changed it to.

Post by mick.crane

Post by Greg Wooledge
Bash reads its history from a file (~/.bash_history by default) at
startup time, manages it in memory while running, and writes it back
out to the history file upon exiting.

Does this separate history get written to ~/.bash_history when terminal
emulation thing is closed?

You need to specify *exactly* what you're doing.

If you close a terminal emulator in such a way that it kills the shell
without letting the shell write history to disk, then the history will
not be saved.

If you want history to be saved, you should exit from the *shell*
(e.g. by pressing Ctrl-D), rather than closing the terminal and counting
on that to terminate the shell in a way that will preserve history.

If you *don't* want history to be saved, there are many ways to make
that (not) happen, depending on your terminal emulator and so on. You'll
need to experiment, though, to see exactly what happens when you do
whatever it is you do.

Whenever I need to reboot my computer (kernel update or the like), I
decide which shells I want to retain history from, and I exit from those
cleanly (^D). I leave the others running, and exit from FVWM, which
kills them in a way that causes their history not to be written.

If you want a particular shell window's history to be AGGRESSIVELY
written to disk every time you run a command, you can arrange for that
as well. See <https://mywiki.wooledge.org/BashFAQ/088> for example.

[Greg Wooledge]

Post by Greg Wooledge
You need to specify *exactly* what you're doing.

Sometimes I forget where I was after closing a virtual terminal and it
would be handy to see the history
in a new terminal, where I "cd'd" to for example.
stuff like that.

No, I mean *how did you close the terminal*? And what terminal is it?

In the ideal scenario, if you closed the terminal by pressing Ctrl-D
in the shell, the history should be written to disk. Then, opening a
new terminal+shell should read that history into memory. You should be
able to search the previous shell's history in that new shell.

There are lots of ways this could fail. For example, you might be
closing the terminal window by using a window manager control button,
or by running the xkill(1) program and then clicking the window.

Or, you might have too many history lines in memory at the time you
exit the shell, and the cd command that you wanted to find might be
dropped because it was too far in the past. (Bash only stores 500
lines of history by default.)

[Mike Castle]

If I've "su'd" I type "exit".
To close the terminal I click that X in the virtual terminal's top right
hand corner.

Depending on settings, that may or may not save that invocation's
history. You'll likely want to test to verify that it does what you
want. But, as Greg pointed out, C-d or "exit" will generally ensure
the right thing happens. If you weren't aware, C-d is the traditional
terminal setting for end-of-file (EOF). It is a system level setting
to indicates no further input is coming into this terminal, which
causes the shell to exit.

mrc

[Max Nikulin]

Sometimes I forget where I was after closing a virtual terminal  and it
would be handy to see the history
in a new terminal, where I "cd'd" to for example.

help history
less ~/.bash_history

[mick.crane]

Post by Greg Wooledge
You need to specify *exactly* what you're doing.

"exactly" is at mickiwiki.com
I can take the ridicule of my coding understanding.

Post by Greg Wooledge
Whenever I need to reboot my computer (kernel update or the like), I
decide which shells I want to retain history from, and I exit from those
cleanly (^D).

that's useful

[The Wanderer]

Post by mick.crane

Post by Greg Wooledge

Post by mick.crane
In debian bookworm, xfce desktop, different virtual terminals
have a different history if same user presses "up key" in
different virtual terminals ?

As your subject says, this is "bash history". And yes, each
instance of bash has its own separate history. It has nothing to
do with your terminal emulator or your desktop environment.

Where is it if not in ~/.bash_history?

In memory.

Post by mick.crane

Post by Greg Wooledge
Bash reads its history from a file (~/.bash_history by default) at
startup time, manages it in memory while running, and writes it
back out to the history file upon exiting.

As Greg wrote: bash reads it in at startup time, then manages it in
memory, then writes it back out when exiting.

Unless you've configured it otherwise, the place it reads the history in
from and writes it out to is ~/.bash_history.

In between those, each and every separate instance of the shell keeps
the history in memory. Yes, if I'm not mistaken that *does* mean that if
you have 20 terminals open you have 20 copies of (some version of) the
history in memory.

Post by mick.crane
Does this separate history get written to ~/.bash_history when
terminal emulation thing is closed?

That depends on how you have things configured. It's not really
"separate", however; it's just that history is only written out when
bash exits (sufficiently cleanly), and that's just as true when you have
multiple instances of bash as when you have only one.

[Andy Smith]

Hi,

Post by mick.crane
Is this something that can be changed so history is shared between virtual
terminals?

You may be interested in "atuin" to aggregate shell history from
multiple logins and machines in a searchable interface. It can be
self-hosted.

Thanks,
Andy

[Karl Vogel]

This is how I keep a long-term record of bash commands from different
sessions:
https://www.reddit.com/r/bash/comments/ak9c3r/

HTH

[Vincent Lefevre]

Post by mick.crane
In debian bookworm, xfce desktop, different virtual terminals have a
different history if same user presses "up key" in different virtual
terminals ?
Is this something that can be changed so history is shared between virtual
terminals?

This is possible with zsh (option SHARE_HISTORY). But I do not use
this option: if I want to retrieve a command from another terminal,
I simply re-execute the shell to get the whole history of commands
typed so far (but there are other ways to manipulate the history).

[Mike Castle]

Post by mick.crane
Is this something that can be changed so history is shared between
virtual terminals?

Yes.

There are all sorts of settings that can control how shells save
history. Most shells are capable of doing whatever you want, but the
default configuration is likely different across each one (bash vs ksh
vs zsh vs csh), and what values you need to change to achieve what you
want varies between them as well.

Everything you need should be available through the install manual
pages, accessible via the command "man bash". Searching for "history"
will turn up a lot of information. The same information, organized
differently and likely different examples, is available via "info
bash" if you install the "bash-doc" package, and that same
documentation is available online at
https://www.gnu.org/software/bash/manual/ . All forms have a section

If I'm not mistaken, by default bash has the following settings:
* Each invocation bash loads the existing history file into memory
* Each invocation keeps the last 500 commands while open
* As each invocation closes, it writes out its own history,
overwriting this existing history (e.g., the last one to exit wins)

You can verify the above yourself by opening two terminal and running
commands different command in them like:
# In one terminal
echo I am shell one

# In the other term
echo I am shell two

Then exit the shells and open up a new one, and running the command:
history | tail

to see what was saved and in what order.

You should experiment with the various settings listed in the
documentation and see what works for you for day-to-day usage.

For me, I see up bash with the following features:
* Unbounded history
* History is immediately saved to disk after each command finishes
* I keep history under source control (currently git) and regularly
(well, for some definition of "regularly"), merge them across machines

For a while, I did experiment with "immediately sharing history
between running shells." That lasted about three months, then I gave
up on the immediate-sharing, but kept the immediate-saving.
# No limit on running shell history size
HISTSIZE=-1
# No limit on lines saved in history file
HISTFILESIZE=-1
# Enable timestamps in bash_history
HISTTIMEFORMAT="[%F %T %z] "
# Stop history being clobbered if there are multiple shells open
shopt -s histappend
# Immediately append history
PROMPT_COMMAND='history -a'

If you really want to try sharing history immediately between shells, use this:
PROMPT_COMMAND='history -a; history -n'

Note that PROMPT_COMMAND executes before printing a prompt, which
means after executing a command. So, if using "history -n", then
other shells will not load the shared history until a new prompt is
printed (e.g., hitting the ENTER key to display a new prompt).

For the record, I deal with the expected conflicts when merging
history files across machines by using a simple python program that
parses the history file (that includes the timestamps), discards the
conflict markers, orders by timestamps, and writes it back out. It is
by no means perfect, but "good enough for me".

For those worried about the unbounded history, I started doing that
about ten years ago and my work history is currently just shy of
180,000 commands. It would likely be less if I turned on the
"erasedups" feature, but I like to keep the context. And I've seen
comments about folks who have multiple decades of shell history. On
modern machines, it simply isn't an issue.

mrc

[Jeffrey Walton]

Post by Mike Castle

Post by mick.crane
Is this something that can be changed so history is shared between
virtual terminals?

[...]
* Unbounded history
* History is immediately saved to disk after each command finishes
* I keep history under source control (currently git) and regularly
(well, for some definition of "regularly"), merge them across machines

This is an unusual use case (to me). Why do you save history in a
version control system?

Post by Mike Castle
[...]
For the record, I deal with the expected conflicts when merging
history files across machines by using a simple python program that
parses the history file (that includes the timestamps), discards the
conflict markers, orders by timestamps, and writes it back out. It is
by no means perfect, but "good enough for me".
For those worried about the unbounded history, I started doing that
about ten years ago and my work history is currently just shy of
180,000 commands. It would likely be less if I turned on the
"erasedups" feature, but I like to keep the context. And I've seen
comments about folks who have multiple decades of shell history. On
modern machines, it simply isn't an issue.

Out of curiosity, do you scrub the file regularly for credentials,
like usernames and passwords, and remove entries?

Jeff

[Mike Castle]

Post by Jeffrey Walton

Post by Mike Castle
* I keep history under source control (currently git) and regularly
(well, for some definition of "regularly"), merge them across machines

This is an unusual use case (to me). Why do you save history in a
version control system?

I rarely have raw data that needs backing up, so I don't do it on a
system level. At work, all important data is in SCM, and various
managed systems. At home, everything I care about is in SCM
(including history) and that repo is backed up.

Also, it allows merging between the machines. Back when almost every
machine I used had $HOME on NFS, this was less of an issue.

Post by Jeffrey Walton
Out of curiosity, do you scrub the file regularly for credentials,
like usernames and passwords, and remove entries?

I never enter credentials via the command line. This is something I
picked up before Linux even existed.

In addition to history, they are viewable via ps(1). While many
modern systems may allow programs to scrub their command line, few
programs do it, and a wrapper script that just passes it along likely
can't.

But, I have trimmed out the occasional cat-walking-across-the-keyboard
and falling-asleep-faceplant-and-drooling-on-the-keyboard type of
history.

mrc

[Yassine Chaouche]

Post by Mike Castle

Post by mick.crane
Is this something that can be changed so history is shared between
virtual terminals?

Yes.
[...]
# No limit on running shell history size
HISTSIZE=-1
# No limit on lines saved in history file
HISTFILESIZE=-1
# Enable timestamps in bash_history
HISTTIMEFORMAT="[%F %T %z] "
# Stop history being clobbered if there are multiple shells open
shopt -s histappend
# Immediately append history
PROMPT_COMMAND='history -a'
PROMPT_COMMAND='history -a; history -n'
Note that PROMPT_COMMAND executes before printing a prompt, which
means after executing a command. So, if using "history -n", then
other shells will not load the shared history until a new prompt is
printed (e.g., hitting the ENTER key to display a new prompt).
For the record, I deal with the expected conflicts when merging
history files across machines by using a simple python program that
parses the history file (that includes the timestamps), discards the
conflict markers, orders by timestamps, and writes it back out. It is
by no means perfect, but "good enough for me".
For those worried about the unbounded history, I started doing that
about ten years ago and my work history is currently just shy of
180,000 commands. It would likely be less if I turned on the
"erasedups" feature, but I like to keep the context. And I've seen
comments about folks who have multiple decades of shell history. On
modern machines, it simply isn't an issue.
mrc

I second Mike Castle on this.

I have more or less the same config for my bash:

export HISTSIZE=-1 # write all commands to history file
export HISTTIMEFORMAT="%Y/%m/%d %H:%M:%S "
export HISTFILESIZE=-1 #don't truncate
export PROMPT_COMMAND="history -a; history -n" # [a]ppend commands to histfile immedatly,
# and read [n]ew entries (could be written from other terminals)

The PROMPT_COMMAND is where the magic happens:
Every time you press Enter,
your command is instantly recorded to the history file with history -a,
instead of waiting until bash exits properly.
This ensures you never lose track of any command,
even if your bash session ends unexpectedly.
Additionally, history -n allows your shell to read the bash history again,
updating it with any new entries.

Imagine executing foo in terminal 1,
pressing Enter,
then switching to terminal 2.
By simply pressing Enter again and hitting the Up arrow,
you'll get the command foo.

This seamless synchronization across terminals enhances my productivity by ensuring all terminals are always in sync.

Best,

[songbird]

Post by mick.crane
In debian bookworm, xfce desktop, different virtual terminals have a
different history if same user presses "up key" in different virtual
terminals ?
Is this something that can be changed so history is shared between
virtual terminals?
mick

not that i would want that,

but it would be possible for various terminals to save to
their own unique history files based upon terminal pty or
tty or anything else you'd like and to reload those upon
starting up again.

to keep my own setup consistent and to not keep certain
things in history i actually do the opposite of what you
want because i want certain commands already preloaded in
my history for all windows when i start up and then i adjust
my environment based upon which pty or directory i'm in.

i also consider it a security or privacy leak to keep
some information in history files of any kind so those get
cleaned up upon normal shutdowns or start ups.


songbird

[Yassine Chaouche]

Post by songbird
[...]
to keep my own setup consistent and to not keep certain
things in history i actually do the opposite of what you
want because i want certain commands already preloaded in
my history for all windows when i start up and then i adjust
my environment based upon which pty or directory i'm in.

I'm scratching my head here.
Can anyone provide a solid example or reason why preloading commands in history is necessary?

Best,

[The Wanderer]

Post by Yassine Chaouche

to keep my own setup consistent and to not keep certain things in
history i actually do the opposite of what you want because i want
certain commands already preloaded in my history for all windows
when i start up and then i adjust my environment based upon which
pty or directory i'm in.

I'm scratching my head here.
Can anyone provide a solid example or reason why preloading commands
in history is necessary?

It's never *necessary*, AFAIK, but it *is* often convenient. (And in
many things, convenience is *the* deciding factor.)

It's often *far* easier to repeat a lengthy command from history (e.g.
via Ctrl+R and typing part of the command), either verbatim or with
modifications, than by retyping it from scratch. I rely on this
extensively - more in my terminal-based SQL-database interaction
sessions than in the shell itself, because the commands there are far
more complex than those in the shell, but I certainly do also rely on it
in the shell.

If you expect to want to use similar commands repeatedly over multiple
shell sessions, why wouldn't you want them there in the history for
ready access?

[Mike Castle]

Post by songbird
not that i would want that,
but it would be possible for various terminals to save to
their own unique history files based upon terminal pty or
tty or anything else you'd like and to reload those upon
starting up again.

Yes.

Setting HISTFILE is one way to control that. It is not uncommon for
folks to do something like:
HISTFILE=~/.history.${HOSTNAME}

when using a shared file system (e.g., NFS).

In bash, the history built-in takes a filename for most options. So
you can do something like:
history -r ~/.history.project

to load a curated set of helpful commands (though at that point, shell
scripts may start to be more useful)

mrc