I applied the patch through the 9443 interface.  I had no issues and all appears to be working OK.  I would open an another SR.. 

I agree with Robin, my update was without any issues too ...

Another hitch-free upgrade, here (large installation)

However, a thought: it may be that kafka is the first package that tries to write to somewhere else than "/", i.e. "/var" or "/vastorage". Before running zypper manually, you could check if those are writable and not full.
We are currently debugging some Filr boot issues at a customer site that seem related to virtual disk availability, and they are running a VMware vSAN. are you also on one, by any chance?

HTH!

Hello,

I found out that there is a loop in the filesystem snapshots. "File system loop detected; /.snapshots/1/snapshots is part of the same file system loop as / 

That must be the cause of the problem. Restoring a backup of a couple of weeks ago shows the same. So I have no roll-back. I do not have a clue how to repair this. I already tried a btrfs repair. I repaired some inodes, but the main problem insists.

Like Christian wrote, full disk is one of the first things

But readonly filesystem is another thing

I have seen this often in unpatched vmware enviroments with vSAN

Somethings happens with disks in the vSAN enviroment and Linux boxes put their filesystem in readonly

often reboot fix this, sometimes not

Then you have to fix / repair the filesystem. The system show this after reboot.

I run the update without any problems on a single box, but after the upgrade filr is very slow

It looks like, that btrfs has to do a lot after the upgrade

btrfs-cleaner

btrfs-transaction

catch up to 100% of one CPU

I wait that these processes finish and give an update

Thomas

See my other remark

Try to see the snapshots

snapper list

never come back, kill it

I wait 30 minutes and after that I reboot the box

then snapper finsed the work and I can see the number of snapshots, more then 40

But after this, I can work on the system.

try to remove btrfs snapshots an let see what happens

Thomas

I can list the snaphots. There are a lot of them 'timeline'. But they are showing in less than a minute.

In response to you previous question: I was not on a vSAN when things happened. Odd enough I moved it to a vSAN yesterday to gain more speed. However the backup I restored is still on traditional storage and is having the same problem.

Rebooting does not solve the problem.

I was able to reproduce the loop in the snapshots by installing a brand new Filr 5 ova. And then do the updates, including 23.2. The update succeeded. But after a reboot there was the same error and several services failed to start. I am on vSphere 8.0.1. Maybe it is something with the appliance updates and this version of vSphere. I will investigate further.

Confirmed. The third thime after the updates is the same. I tried the installs on: 1. local storage, iSCSI (san) storage and vSAN. And all three act the same. When I install the updates and reboot, the snapshots are in a loop.

Did anyone check the filesystem after the upgrade? E.g. do a find anywhere on the filesystem. I the snapshots are in a loop it will be reported then.