Strange syslog behaviour [Solved]

[Jeremy Ardley]

I have systems (armbian) that had anomalous behaviour.

This included sometimes writing to /var/log/syslog.1 rather than to
/var/log/syslog (which was created, but zero size)

Additionally the logrotate was happening daily or twice daily when
seemingly configured for weekly rotates

Anyway long story short, at some stage some package updates must have
written an extra file into /etc/logrotate.d that had duplicate entries
to the normal files.

This was interpreted by the logrotate process as well as the intended
files such as /etc/logrotated.d/rsyslog

On one system this unexpected file was called rsyslog.dpkg-old on
another system it was rsyslog.dpkg-dist

Removing these files ( but not /etc/logrotate.d/dpkg ) now has a
correctly configured log rotation

[Jeremy Ardley]

There are tabooext and taboopat directives for ignoring files in
logrotated.d, and I would have thought it reasonable to exclude
these sorts of housekeeping files by default, because they're very
likely to contain some duplication. I would file a bug against
logrotate.

Keywords tabooext and taboopat don't appear in /etc/*

I did get a hit in binary file /usr/sbin/rsyslogd

Jeremy

[Jeremy Ardley]

Post by Jeremy Ardley

There are tabooext and taboopat directives for ignoring files in
logrotated.d, and I would have thought it reasonable to exclude
these sorts of housekeeping files by default, because they're very
likely to contain some duplication. I would file a bug against
logrotate.

Keywords tabooext and taboopat don't appear in /etc/*
I did get a hit in binary file /usr/sbin/rsyslogd

Further to resolving the problem, on one system I ran

logrotate --debug /etc/logrotate.conf

And discovered that /etc/logrotate.d/inetutils-syslogd was also being
loaded. It had duplicates of many entries in /etc/logrotate.d/rsyslog

I have no idea why inetutils did this. I recall inetutils also did some
other bad stuff I had to disable.

All I wanted was ping but I got a world of hurt as well!

[Jeremy Ardley]

Post by Jeremy Ardley
And discovered that /etc/logrotate.d/inetutils-syslogd was also being
loaded. It had duplicates of many entries in /etc/logrotate.d/rsyslog
All I wanted was ping _traceroute_ but I got a world of hurt as well!

*this* seems likely to solve the problems on that machine and still keep
traceroute

rm -f /etc/logrotate.d/inetutils-syslogd
systemctl stop inetutils-syslogd.service
systemctl disable inetutils-syslogd.service
systemctl mask inetutils-syslogd.service
systemctl stop inetutils-inetd.service
systemctl disable inetutils-inetd.service
systemctl mask inetutils-inetd.service

[David Wright]

While I was composing my second post in this thread,

Post by Jeremy Ardley

Post by Jeremy Ardley

There are tabooext and taboopat directives for ignoring files in
logrotated.d, and I would have thought it reasonable to exclude
these sorts of housekeeping files by default, because they're very
likely to contain some duplication. I would file a bug against
logrotate.

Keywords tabooext and taboopat don't appear in /etc/*
I did get a hit in binary file /usr/sbin/rsyslogd

Further to resolving the problem, on one system I ran
logrotate --debug /etc/logrotate.conf
And discovered that /etc/logrotate.d/inetutils-syslogd was also being
loaded. It had duplicates of many entries in /etc/logrotate.d/rsyslog

If both those file exist, then I would think that's the cause of your
double rotation problem. As for the .dpkg-{old,dist} files, they just
reflect different responses for the upgrading of /etc/logrotate.d/dpkg
when dpkg was being upgraded on the two systems.

Post by Jeremy Ardley
I have no idea why inetutils did this. I recall inetutils also did
some other bad stuff I had to disable.

AFAICT there's no package "inetutils", but eleven inetutils-* separate
packages, amongst them inetutils-ping, inetutils-traceroute and
inetutils-syslogd.

Post by Jeremy Ardley
All I wanted was ping but I got a world of hurt as well!

Most people, I suspect, have iputils-ping installed. On my bullseye
systems, it gets installed by the d-i a little after netbase and a
little before logrotate. That's in the early debootstrap stage, and
before the in-target stages.

Post by Jeremy Ardley
From the follow-up to your post, I get the impression that you've

pulled in some inetutils-* stuff. AFAICT inetutils-ping kicks out
iputils-ping, and inetutils-traceroute kicks out traceroute without
any complications, but you appear to have installed inetutils-syslogd,
perhaps on account of inetutils-inetd, but without removing rsyslogd.
In the follow-up, you appear to prevent -syslogd and -inetd from
doing anything, so one might ask why they're installed at all. Have
I missed spotting some dependency?

Cheers,
David.

[David Wright]

Post by Jeremy Ardley

There are tabooext and taboopat directives for ignoring files in
logrotated.d, and I would have thought it reasonable to exclude
these sorts of housekeeping files by default, because they're very
likely to contain some duplication. I would file a bug against
logrotate.

Keywords tabooext and taboopat don't appear in /etc/*

Looking further into this, I see that tabooext has a sizeable default
list of excluded extensions, and this list is built into logrotate, so
a tabooext directive doesn't have to be specified in logrotate.conf.
The default includes the two extensions you mentioned.

The tabooext list can be appended to, but can also be overwritten if
the necessary + is forgotten. Can we assume you haven't done that?

Post by Jeremy Ardley
I did get a hit in binary file /usr/sbin/rsyslogd

If you mean rsyslogd and not logrotate, that's presumably in the
≥testing version (ie not buster or bullseye).

$ strings /usr/sbin/rsyslogd | grep -n -e dpkg -e rpm -e ucf
$ strings /usr/sbin/logrotate | grep -n -e dpkg -e rpm -e ucf
982:.dpkg-bak
983:.dpkg-del
984:.dpkg-dist
985:.dpkg-new
986:.dpkg-old
987:.dpkg-tmp
989:.rpmnew
990:.rpmorig
991:.rpmsave
993:.ucf-dist
994:.ucf-new
995:.ucf-old
$

Cheers,
David.

[Wayne Sallee]

Thanks for pointing this out.

I just noticed this problem a few days ago.

Wayne Sallee
***@WayneSallee.com
http://www.WayneSallee.com


-------- Original Message --------
*Subject: *  Strange syslog behaviour [Solved]
*From: *     Jeremy Ardley <***@ardley.org>
*To: *         Debian-user <debian-***@lists.debian.org>
*CC: *
*Date: *      2022-4-10  10:07 PM

Post by Jeremy Ardley
I have systems (armbian) that had anomalous behaviour.
This included sometimes writing to /var/log/syslog.1 rather than to /var/log/syslog (which was created, but zero size)
Additionally the logrotate was happening daily or twice daily when seemingly configured for weekly rotates
Anyway long story short, at some stage some package updates must have written an extra file into /etc/logrotate.d that
had duplicate entries to the normal files.
This was interpreted by the logrotate process as well as the intended files such as /etc/logrotated.d/rsyslog
On one system this unexpected file was called rsyslog.dpkg-old on another system it was rsyslog.dpkg-dist
Removing these files ( but not /etc/logrotate.d/dpkg ) now has a correctly configured log rotation

[Greg Wooledge]

Post by Wayne Sallee
-------- Original Message --------
*Subject: *  Strange syslog behaviour [Solved]

Post by Jeremy Ardley
Anyway long story short, at some stage some package updates must have
written an extra file into /etc/logrotate.d that had duplicate entries
to the normal files.
This was interpreted by the logrotate process as well as the intended files such as /etc/logrotated.d/rsyslog
On one system this unexpected file was called rsyslog.dpkg-old on another system it was rsyslog.dpkg-dist
Removing these files ( but not /etc/logrotate.d/dpkg ) now has a correctly configured log rotation

Files with extensions *.dpkg-old and *.dpkg-dist are created when a
package is updated, and its configuration file has been modified. The
current (modified) configuration file is left alone, and the files that
*would* have been installed by the package are placed with these
extensions.

Such files *should not* be operational. Programs are supposed to ignore
them.

In logrotate(8) there is this paragraph:

tabooext [+] list
The current taboo extension list is changed (see the include di‐
rective for information on the taboo extensions). If a + pre‐
cedes the list of extensions, the current taboo extension list
is augmented, otherwise it is replaced. At startup, the taboo
extension list ,v, .cfsaved, .disabled, .dpkg-bak, .dpkg-del,
.dpkg-dist, .dpkg-new, .dpkg-old, .rhn-cfg-tmp-*, .rpmnew, .rp‐
morig, .rpmsave, .swp, .ucf-dist, .ucf-new, .ucf-old, ~

So, unless your /etc/logrotate.conf has been changed, or your logrotate
is not the one provided by Debian, the *.dpkg-old and *.dpkg-dist files
should do nothing. They should be ignored.

Post by Wayne Sallee

Post by Jeremy Ardley
I have systems (armbian) that had anomalous behaviour.

If you're *not on Debian* then we can't tell you what to do. Ask your
operating system support people.

[Greg Wooledge]

edit /etc/logrotate.d/rsyslog
Change from
                 invoke-rc.d rsyslog rotate > /dev/null
to
                 /usr/lib/rsyslog/rsyslog-rotate

Hmm.

In /etc/init.d/rsyslog (which is what the old command calls upon), the
rotate action calls this function:

do_rotate() {
start-stop-daemon --stop --signal HUP --quiet --pidfile $PIDFILE --exec $DAEMON
}

where PIDFILE is defined to be /run/rsyslogd.pid .

This file does not exist on my (bullseye) system, where rsyslog is
started by systemd (/lib/systemd/system/rsyslog.service) with the "-iNONE"
argument, which suppresses the creation of *any* PID file. So, that
explains why the start-stop-daemon command doesn't work.

Therefore, what you've done looks like the correct fix. Deleting the
*.dpkg-dist files shouldn't matter, so long as /etc/logrotate.d/rsyslog
gets updated.

However, this looks like a bug in rsyslog to me.

The rsyslog sysv-rc script, if it's going to exist, should duplicate
the /usr/lib/rsyslog/rsyslog-rotate program in the do_rotate function.
It's clearly designed to handle both systemd and sysvinit:

unicorn:~$ less /usr/lib/rsyslog/rsyslog-rotate
#!/bin/sh

if [ -d /run/systemd/system ]; then
systemctl kill -s HUP rsyslog.service
else
invoke-rc.d rsyslog rotate > /dev/null
fi

So, the init.d script should be using something like that. I would do it
like this:

do_rotate() {
if [ -d /run/systemd/system ]; then
systemctl kill -s HUP rsyslog.service
else
start-stop-daemon --stop --signal HUP --quiet \
--pidfile "$PIDFILE" --exec "$DAEMON"
fi
}

[David Wright]

Jeremy Ardley, did you update your code from " invoke-rc.d rsyslog rotate > /dev/null" to
"/usr/lib/rsyslog/rsyslog-rotate"?

I've added a new post to that part of the thread. I think your
problem concerns changing to systemd, whereas his concerns mixing
inetdutils-* packages with the more usual equivalent ones.

So I got my servers straitened out, I think. I will know tomorrow.
For anyone else running into this problem, the problem was caused from modifying
/etc/logrotate.d/rsyslog then upgrading to Buster.
diff /etc/logrotate.d/rsyslog.dpkg-dist /etc/logrotate.d/rsyslog
edit /etc/logrotate.d/rsyslog
Change from
invoke-rc.d rsyslog rotate > /dev/null
to
/usr/lib/rsyslog/rsyslog-rotate
Then delete /etc/logrotate.d/rsyslog.dpkg-dist
I then rebooted my servers to get the syslog used.
Then
ls /var/log | grep syslog
To see that it was working. I will know tomorrow if it is still working.

I think the point you've missed is that at some stage, you upgraded
from SystemV to systemd. Whenever that happened, the upgrade to
rsyslog would have supplied a new /etc/logrotate.d/rsyslog file
containing "/usr/lib/rsyslog/rsyslog-rotate", which knows how to
handle both SystemV and systemd systems.

You rejected the new file, which is why it was instead written to
/etc/logrotate.d/rsyslog.dpkg-dist (which you could have safely
left or removed—it's harmless).

Effectively, your edit has merged the contents of both files,
whatever changes you made earlier before the Buster upgrade,
and the vital change that would have been made for you if
you'd accepted the new version.

Cheers,
David.

[Wayne Sallee]

-------- Original Message --------
*Subject: *  Re: Strange syslog behaviour [Solved]
*From: *     David Wright <***@lionunicorn.co.uk>
*To: *         Debian-user <debian-***@lists.debian.org>
*CC: *
*Date: *      2022-10-16  01:27 AM

Post by David Wright

Jeremy Ardley, did you update your code from " invoke-rc.d rsyslog rotate > /dev/null" to
"/usr/lib/rsyslog/rsyslog-rotate"?

I've added a new post to that part of the thread. I think your
problem concerns changing to systemd, whereas his concerns mixing
inetdutils-* packages with the more usual equivalent ones.

So I got my servers straitened out, I think. I will know tomorrow.
For anyone else running into this problem, the problem was caused from modifying
/etc/logrotate.d/rsyslog then upgrading to Buster.
diff /etc/logrotate.d/rsyslog.dpkg-dist /etc/logrotate.d/rsyslog
edit /etc/logrotate.d/rsyslog
Change from
invoke-rc.d rsyslog rotate > /dev/null
to
/usr/lib/rsyslog/rsyslog-rotate
Then delete /etc/logrotate.d/rsyslog.dpkg-dist
I then rebooted my servers to get the syslog used.
Then
ls /var/log | grep syslog
To see that it was working. I will know tomorrow if it is still working.

I think the point you've missed is that at some stage, you upgraded
from SystemV to systemd. Whenever that happened, the upgrade to
rsyslog would have supplied a new /etc/logrotate.d/rsyslog file
containing "/usr/lib/rsyslog/rsyslog-rotate", which knows how to
handle both SystemV and systemd systems.

My situation is now solved. It's working as it should. I think Debian 9 is systemd by default, so I don't think it was a
change from systemv to systemd. But it was definitely a change in programs used that required the change to
"rsyslog-rotate".

Post by David Wright
You rejected the new file, which is why it was instead written to
/etc/logrotate.d/rsyslog.dpkg-dist (which you could have safely
left or removed—it's harmless).

Yes, I rejected the new file, causing the other file to be there. It would be nice if updates told what needed to be
updated in the old file when updates are done.

Post by David Wright
Effectively, your edit has merged the contents of both files,
whatever changes you made earlier before the Buster upgrade,
and the vital change that would have been made for you if
you'd accepted the new version.

But If I had accepted the new file, it would probably have discarded the changes that I had made.

It would be nice if updates presented "old file", "new file", "combined file"; choice: (1), (2), (3).

Wayne Sallee
***@WayneSallee.com
http://www.WayneSallee.com

Post by David Wright
Cheers,
David.

[Charles Curley]

On Sun, 16 Oct 2022 09:09:36 -0400

Post by Wayne Sallee

Post by David Wright
You rejected the new file, which is why it was instead written to
/etc/logrotate.d/rsyslog.dpkg-dist (which you could have safely
left or removed—it's harmless).

Yes, I rejected the new file, causing the other file to be there. It
would be nice if updates told what needed to be updated in the old
file when updates are done.

You can get that by accepting or rejecting the new file, then diffing
the resulting file with the one dpkg writes.

Post by Wayne Sallee

Post by David Wright
Effectively, your edit has merged the contents of both files,
whatever changes you made earlier before the Buster upgrade,
and the vital change that would have been made for you if
you'd accepted the new version.

But If I had accepted the new file, it would probably have discarded
the changes that I had made.

It would have. But it preserves the old file, and you can then copy
from the old file to the new as appropriate. Again, diff is your
friend. I use Emacs' ediff mode to do this.

Post by Wayne Sallee
It would be nice if updates presented "old file", "new file",
"combined file"; choice: (1), (2), (3).

There is no way the updates could figure out what the appropriate
combined file would be. It would have no idea what your changes to the
old file do, or which changes are appropriate to the new situation. You
have to make that decision.

When I do an upgrade I run the upgrade in one terminal, and have Emacs
running in another. As these differences pop up, I accept one of the
options, then edit away as necessary. It takes a while to do an upgrade,
but I usually get what I want and don't have to go back and make more
edits.

And I also make sure I have recent backups of /etc and a few other
places so I can restore my original files if need be.

[Wayne Sallee]

-------- Original Message --------
*Subject: *  Re: Strange syslog behaviour [Solved]
*From: *     Charles Curley <***@charlescurley.com>
*To: *         Debian Users <debian-***@lists.debian.org>
*CC: *
*Date: *      2022-10-16  11:09 AM

Post by Charles Curley
When I do an upgrade I run the upgrade in one terminal, and have Emacs
running in another. As these differences pop up, I accept one of the
options, then edit away as necessary. It takes a while to do an upgrade,
but I usually get what I want and don't have to go back and make more
edits.

That's a good idea to use 2 terminals like that.

Wayne Sallee
***@WayneSallee.com
http://www.WayneSallee.com

[Tixy]

Post by Wayne Sallee
It would be nice if updates presented "old file", "new file", "combined file"; choice: (1), (2), (3).

It does offer several choices, one of them is to show a 'diff' of the
old and new files. Only you can know what changes you made and want to
keep, and just automatically merging files is very likely just to
produce a broken config.

Personally, I do a diff first, which usually acts of a reminder of what
I've changed, then I accept the new file and in another terminal edit
that to reapply my changes.

[Wayne Sallee]

-------- Original Message --------
*Subject: *  Re: Strange syslog behaviour [Solved]
*From: *     Tixy <***@yxit.co.uk>
*To: *         Debian-user <debian-***@lists.debian.org>
*CC: *
*Date: *      2022-10-16  12:40 PM

Post by Tixy

Post by Wayne Sallee
It would be nice if updates presented "old file", "new file", "combined file"; choice: (1), (2), (3).

It does offer several choices, one of them is to show a 'diff' of the
old and new files. Only you can know what changes you made and want to
keep, and just automatically merging files is very likely just to
produce a broken config.
Personally, I do a diff first, which usually acts of a reminder of what
I've changed, then I accept the new file and in another terminal edit
that to reapply my changes.

Yes, I did do the diff, but I was in the middle of the update. I like Charles Curley's idea of having another terminal
to make the needed changes as they come up.

Wayne Sallee
***@WayneSallee.com
http://www.WayneSallee.com

[David Wright]

Post by Wayne Sallee

Post by David Wright

Jeremy Ardley, did you update your code from " invoke-rc.d rsyslog rotate > /dev/null" to
"/usr/lib/rsyslog/rsyslog-rotate"?

I've added a new post to that part of the thread. I think your
problem concerns changing to systemd, whereas his concerns mixing
inetdutils-* packages with the more usual equivalent ones.

So I got my servers straitened out, I think. I will know tomorrow.
For anyone else running into this problem, the problem was caused from modifying
/etc/logrotate.d/rsyslog then upgrading to Buster.
diff /etc/logrotate.d/rsyslog.dpkg-dist /etc/logrotate.d/rsyslog
edit /etc/logrotate.d/rsyslog
Change from
invoke-rc.d rsyslog rotate > /dev/null
to
/usr/lib/rsyslog/rsyslog-rotate
Then delete /etc/logrotate.d/rsyslog.dpkg-dist
I then rebooted my servers to get the syslog used.
Then
ls /var/log | grep syslog
To see that it was working. I will know tomorrow if it is still working.

I think the point you've missed is that at some stage, you upgraded
from SystemV to systemd. Whenever that happened, the upgrade to
rsyslog would have supplied a new /etc/logrotate.d/rsyslog file
containing "/usr/lib/rsyslog/rsyslog-rotate", which knows how to
handle both SystemV and systemd systems.

My situation is now solved. It's working as it should. I think Debian
9 is systemd by default, so I don't think it was a change from systemv
to systemd. But it was definitely a change in programs used that
required the change to "rsyslog-rotate".

That's right: AIUI buster is when rsyslog started using systemd to
rotate the logs, rather than the compatibility script that's still
present in /etc/init.d/ for non-systemd users. The decision is now
made in /usr/lib/rsyslog/rsyslog-rotate.

Post by Wayne Sallee

Post by David Wright
You rejected the new file, which is why it was instead written to
/etc/logrotate.d/rsyslog.dpkg-dist (which you could have safely
left or removed—it's harmless).

Yes, I rejected the new file, causing the other file to be there. It
would be nice if updates told what needed to be updated in the old
file when updates are done.

If you reject the file, it falls to you to look at what changes were
made. All the system knows is that your file's message digest doesn't
match what dpkg expects as the old Debian version.

Post by Wayne Sallee

Post by David Wright
Effectively, your edit has merged the contents of both files,
whatever changes you made earlier before the Buster upgrade,
and the vital change that would have been made for you if
you'd accepted the new version.

But If I had accepted the new file, it would probably have discarded the changes that I had made.

Yes. It's a while since I did this, but I'm almost certain that
your own version would be renamed as *.dpkg-old were you to accept
the new version.

Post by Wayne Sallee
It would be nice if updates presented "old file", "new file", "combined file"; choice: (1), (2), (3).

(3) isn't really practicable, and you couldn't really set that
as Policy unless it could be more or less guaranteed to work.
To mis-quote Tolstoy, “Happy updates are all alike; every
unhappy update is unhappy in its own way.”

You could use patch to generate a .diff between your version and
its original, then apply the patch to the new Debian version.
Fine when it works, but there's no guarantee. It requires a
certain discipline when making any changes. So it's safer to
leave the choice of process to the system administrator.

Cheers,
David.