Problem with cryptsetup-initramfs

[Ian Molton]

Hi all,

I'm having problems with one of my machines. it's a Pine RockPro64.

Debian bookworm has been running very stably on it for some time. I
rebooted it a couple of weeks ago for maintainance, having applied
updates, after 108 days up. I have an encrypted LVM volume, containing
root, swap, and data LVs, with /boot on a MMC card, and I use
cryptsetup-initramfs to allow me to log in and unlock the volume at boot
time, via ssh.

I rebooted it this morning, due to a crash, and it didn't come back up.

I can still connect to dropbear (running in the initramfs context), and
the MoTD (as always) prompts me to run cryptroot-unlock, which appears
to do its job (ie. the LVM volume is unlocked / decrypted), however it
does not proceed to switch root (and drop the ssh connection), as it
used to, with complete reliability.

At first, I suspected a problem with the rootfs, however this appears
not to be the issue - the volume is present at the expected location,
and can be manually mounted.

Executing the following allows me to enter a chroot on the rootfs:

mount -text4 /dev/vg0/root /root
mount --bind /dev /root/dev
mount --bind /proc /root/proc
mount --bind /sys /root/sys
chroot /root /bin/bash --login

and following this I can run:

mount /boot

which correctly mounts the MMC card containing the /boot partition in
the chrooted environment.

Inspecting /boot, everything appears to be in order. I have issued
update-initramfs a few times, even completely removing the existing
initramfs and recreating it.  I have also inspected the initramfs built
by update-initramfs, and can see nothing out of the ordinary. crypttab
is copied from the host, and the UUID matches that displayed by lsblk -f
- which is not surprising given that executing cryptroot-unlock does, in
fact, decrypt the volume.

Once chrooted, I can see that /sbin/init is a symlink to
/lib/systemd/systemd, which exists and is executable, but obviously
cannot be executed as anything other than PID 1. Attempting to execute
it results in it complaining of a missing argument, or, if one is
provided, an error that it is ignoring the request due to running in a
chroot.

The kernel command line (cat /proc/cmdline) contains a correct root=
entry, which points at /dev/mapper/vg0-root

I'm stumped - I cannot see why the initramfs environment fails to mount
the rootfs and execute init.

I have run an additional apt update / upgrade / dist-upgrade, whilst
under chroot, in the hope that it will magically fix everything, but to
no avail.

I was using a custom dtb that enabled PCIe x4 on the board, but have
removed that and reverted to the debian-supplied .dtb file just in case.

Any ideas? I have several machines using this configuration, both arm64
and amd64, and I'm now a little uneasy about rebooting any of them, in
case there has been a breaking change somewhere which they, too, are
likely to fall afoul of.

Any thoughts? I've never really had to debug the init process (ie. PID1)
and am not sure how to proceed.

Thanks,

-Ian

[Michel Verdier]

I'm stumped - I cannot see why the initramfs environment fails to mount the
rootfs and execute init.

You could run with kernel parameter "debug=vc" if you have a console. Else
with "debug" you get logs in /run/initramfs.
Also initramfs uses busybox (if you installed it). Some busybox commands
are slightly different from normal ones. Although I don't see differences
for cryptsetup.

[Ian Molton]

Hi all,


I have solved the boot issue. I got physical access to the machine, and
connected a monitor and keyboard to it.

As usual, it booted the initramfs, and started dropbear, but this time,
I entered the disk key on the keyboard.

The machine attempted to boot, but failed, dropping back to the
initramfs prompt - however, crucially, the video console logged a useful
error - the rootfs could not be mounted (despite mounting just fine,
without errors, in read write mode, over the ssh connection previously).

I ran a fsck in the initramfs environment, and rebooted it.

The machine booted and I again unlocked it via the (keyboard) console.

This resulted in a clean boot with no untoward errors.

I shut it down, and rebooted, and now it boots successfully even if I
unlock it via the ssh connection.

I feel like this is a bug in cryptroot-initramfs - surely it would be
critically important for the aforementioned filesystem error to have
been logged to the ssh console, rather than it silently failing to boot?
afterall - it DOES log the errors to the video console.

Or have I been daft and somehow misconfigured it? I don't think I
deviated from the standard configuration.

Is there a simple way to have errors logged to the console AND to the
ssh session? Have I been thick?

Thanks,

-Ian

Post by Ian Molton
Hi all,
I'm having problems with one of my machines. it's a Pine RockPro64.
Debian bookworm has been running very stably on it for some time. I
rebooted it a couple of weeks ago for maintainance, having applied
updates, after 108 days up. I have an encrypted LVM volume, containing
root, swap, and data LVs, with /boot on a MMC card, and I use
cryptsetup-initramfs to allow me to log in and unlock the volume at
boot time, via ssh.
I rebooted it this morning, due to a crash, and it didn't come back up.
I can still connect to dropbear (running in the initramfs context),
and the MoTD (as always) prompts me to run cryptroot-unlock, which
appears to do its job (ie. the LVM volume is unlocked / decrypted),
however it does not proceed to switch root (and drop the ssh
connection), as it used to, with complete reliability.
At first, I suspected a problem with the rootfs, however this appears
not to be the issue - the volume is present at the expected location,
and can be manually mounted.
mount -text4 /dev/vg0/root /root
mount --bind /dev /root/dev
mount --bind /proc /root/proc
mount --bind /sys /root/sys
chroot /root /bin/bash --login
mount /boot
which correctly mounts the MMC card containing the /boot partition in
the chrooted environment.
Inspecting /boot, everything appears to be in order. I have issued
update-initramfs a few times, even completely removing the existing
initramfs and recreating it.  I have also inspected the initramfs
built by update-initramfs, and can see nothing out of the ordinary.
crypttab is copied from the host, and the UUID matches that displayed
by lsblk -f - which is not surprising given that executing
cryptroot-unlock does, in fact, decrypt the volume.
Once chrooted, I can see that /sbin/init is a symlink to
/lib/systemd/systemd, which exists and is executable, but obviously
cannot be executed as anything other than PID 1. Attempting to execute
it results in it complaining of a missing argument, or, if one is
provided, an error that it is ignoring the request due to running in a
chroot.
The kernel command line (cat /proc/cmdline) contains a correct root=
entry, which points at /dev/mapper/vg0-root
I'm stumped - I cannot see why the initramfs environment fails to
mount the rootfs and execute init.
I have run an additional apt update / upgrade / dist-upgrade, whilst
under chroot, in the hope that it will magically fix everything, but
to no avail.
I was using a custom dtb that enabled PCIe x4 on the board, but have
removed that and reverted to the debian-supplied .dtb file just in case.
Any ideas? I have several machines using this configuration, both
arm64 and amd64, and I'm now a little uneasy about rebooting any of
them, in case there has been a breaking change somewhere which they,
too, are likely to fall afoul of.
Any thoughts? I've never really had to debug the init process (ie.
PID1) and am not sure how to proceed.
Thanks,
-Ian

[Ian Molton]

Now what?

Should this be reported somewhere else? Any devs reading this?

[Nicolas George]

Post by Ian Molton
Now what?

Now maybe you fix your MUA configuration to not omit “Re: ”.

Regards,

[Ian Molton]

Really helpful. Thanks for that.

Really makes one feel part of the community.

[t***@tuxteam.de]

Post by Ian Molton
Really helpful. Thanks for that.

C'm on Ian.

On another post you complained about the community falling to pieces.
There are many causes for it. This is one.

I know -- the poster you're replying to can be very caustic (sometimes,
in my opinion, uselessly so), but they are pretty helpful as well.

Just "hitting back" at the first impulse, as you did, turns this place
into a mud slinging fest at every little, silly misunderstanding.

Post by Ian Molton
Really makes one feel part of the community.

Then please help us to make it so.

Cheers

[Jonathan Dowland]

Post by Nicolas George
Now maybe you fix your MUA configuration to not omit “Re: ”.

What would that achieve?

[Andy Smith]

Hi,

Post by Ian Molton
Should this be reported somewhere else? Any devs reading this?

Yes, if you think there is a bug it should be reported in
bugs.debian.org. The "reportbug" program may help.

This mailing list is mostly for user support by other users, not for
reporting bugs or getting the attention of package maintainers.
Filing a bug report would have more chance of getting the attention
of the package maintainer.

Thanks,
Andy

[Jonathan Dowland]

I'm glad you've solved your issue. I read your original (very detailed)
mail and I had nothing to contribute with respect to a fix; but I was
interested to follow it, as I rely upon remote decryption of the root
filesystem myself.

From what you write, I think you are correct that some component

(perhaps cryptroot-initramfs) should be enhanced to handle this
scenario better. I'm not sure of exactly what change would be required,
nor where that change should be made.

There are very few Debian developers monitoring this list. For there to
be any chance at all of this enhancement being made, someone will need
to spend the time to write up a summary and submit it as a bug against
the cryptroot-initramfs package. (perhaps the cryptroot-initramfs
maintainers decide the bug better belongs to another component: bugs can
be re-assigned after filing with ease).

From there, with luck and a following wind, it may get resolved.

Good luck,

[Ian Molton]

Hi,

Thanks for your reply,

I'm afraid I've always found the bug tracker a very inaccessible thing.
Despite having a decent amount of technical knowledge,  I find it
daunting, and a jarring thing to cope with mentally.

This is why i come to the mailing lists to seek help, but as you note,
devs do not seem to inhabit/monitor this place, and i would not like this
bug to be overlooked.

Perhaps someone can help me with the bug tracker?

Long ago, i was part of the LFS community,  which was always a place
where devs were excellent at interacting with users.

I attended debconf 11, but found it hard to make connections there.

Much of my experience of using / contributing to FOSS projects was in the
period from 2000-2008, and (to my eyes) it is shocking how, in general,
FOSS project communities have split into hierarchical Dev/user systems.

I dont think this is a good sign of the times, and to my mind it is
evidence of a corporate sickness that threatens to destroy FOSS community
that would have been Microsoft's wet dream back then, and something which
used to be strongly resisted, not encouraged.

Users are one of the pillars of FOSS. without them, projects wither and
die, whilst the corporate world subsumes them...

Maybe its just one persons opinion though.  I hope not!

[Michel Verdier]

Post by Ian Molton
Perhaps someone can help me with the bug tracker?

Install the package reportbug. It's as easy as writing a mail.

[Thomas Schmitt]

Hi,

Post by Ian Molton
Perhaps someone can help me with the bug tracker?

If i have to submit a bug, then i use the e-mail way.
See:

https://www.debian.org/Bugs/Reporting
"Sending the bug report via e-mail"
(about 30 lines down the page)
"An Example Bug Report"
(another 30 lines down the page)

The first three lines of the example are real mail headers for your
mail client. The others lines are mail body text.

--------------------------------------------------------------------
In particular i'd do for submitting a bug about xorriso:

- Compose a mail for ***@bugs.debian.org

- Begin the mail body text by

Package: xorriso

Maybe after having searched the package name by
apt-file search ...program.name...

- Next comes the package version header with the version string

Version: 1.5.4-4

The internet proposes a tangible command line for this:

dpkg -s xorriso | grep '^Version'

- After an empty line comes the bug description text.

A checklist for information to be included is at
https://www.debian.org/Bugs/Reporting
under
"Please include in your report:"
Usually you can provide only a part of this list. Just leave out
what would stop you from submitting the bug report.

- Read what you wrote and check whether it would be understandable
for somebody who does not know more about your situation than you
wrote in the mail body.
(You may of course assume that the reader is an expert with the
package in question.)

- Send the mail (as said, to ***@bugs.debian.org)

- Wait for the acknowledgement mail which will tell you the bug number.

Since i am upstream of xorriso i never submitted a Debian bug for it.
Here is the acknowledgement for one of my earliest bug reports:

Date: Fri, 07 Aug 2015 13:03:05 +0000
From: Debian Bug Tracking System <***@bugs.debian.org>
Reply-To: ***@bugs.debian.org
To: Thomas Schmitt <***@gmx.net>
Subject: Bug#794868: Acknowledgement (dvd+rw-tools: Burn failure of
growisofs on DVD-R[W] with write type DAO)

- Subscribe to your new bug report (which is behaving like a little
mailing list).
Send a mail with arbitrary subject and body text to the bug number's
subscribe address

794868-***@bugs.debian.org

Or go to the bug report page
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794868
and click the "subscribe" link about 10 lines down from the page top.

- If you get replies, send answers to the bug report's own mail address:

***@bugs.debian.org

It might be a good idea to Cc: the mail address from where the answer
came. Just in case it is not subscribed to the bug report.

--------------------------------------------------------------------

Have a nice day :)

Thomas

[Michel Verdier]

Post by Thomas Schmitt
https://www.debian.org/Bugs/Reporting
"Sending the bug report via e-mail"
(about 30 lines down the page)
"An Example Bug Report"
(another 30 lines down the page)

Still the first and recommended way is to use the package reportbug which
do all you describe.