Systems upgrading

[Wesley]

Hello

We have 2000+ dedi servers planning to upgrade from debian 11 to Debian
12.

The main tech stacks on them are apache, mysql, redis, resin/tomcat,
ceph and some hdfs nodes.

We are looking for a consultancy for the implementation. If there is any
who have the interest, please contact me.

Thanks.

[Andy Smith]

Hi Wesley,

We have 2000+ dedi servers planning to upgrade from debian 11 to Debian 12.

I should think that the typical debian-user subscriber manages only
a small number of systems with all of them for personal/hobby use,
with the rest of us being extreme outliers.

I think your use case is vastly outside of the experience of almost
everyone here, so here may not be a great place to find a
consultant.

This page may help in your search:

https://www.debian.org/consultants/

I can recommend some in my country (UK) but maybe that is not of
interest to you.

I've never used them, but my company pays Freexian at the bronze
tier to do Debian LTS packaging work and I understand they do wider
Debian consultancy, so maybe that is a good option.

https://www.freexian.com/services/debian-support/

There's the debian-consultants mailing list but it gets almost no
traffic. I don't think it will aid you in finding a consultancy any
better than the previous links, but just in case:

https://lists.debian.org/debian-consultants/

Just as some free advice though…

1. I find it hard to believe you have more than 2000 Debian installs
without some sort of existing automation / configuration
management

2. Given (1), I would approach the task by learning your config
management and modifying it to deploy a Debian 12 version of each
kind of Debian 11 server you already have.

3. I'd then do a rolling deploy that slowly takes Debian 11 servers
out of service and re-provisions them as Debian 12. I would not
try to upgrade anything in place. Although Debian supports that,
at scale I find it harder to account for all variables than with
a clean install, and if you already have automation to deploy and
configure a host then an in-lace upgrade also takes longer in my
experience.

Something to chat to your consultancy about, anyway.

It would be really interesting if you or your consultant would write
up what you ended up doing.

Thanks,
Andy

[Wesley]

And yes we just experienced a p0 accident which affected 100 millions
end users. (There is maybe already the news on internet.) So we are
taking serious consideration on hiring a consultant for our system
upgrading.

Thank you.

Post by Andy Smith
Hi Wesley,

We have 2000+ dedi servers planning to upgrade from debian 11 to Debian 12.

I should think that the typical debian-user subscriber manages only
a small number of systems with all of them for personal/hobby use,
with the rest of us being extreme outliers.
I think your use case is vastly outside of the experience of almost
everyone here, so here may not be a great place to find a
consultant.
https://www.debian.org/consultants/
I can recommend some in my country (UK) but maybe that is not of
interest to you.
I've never used them, but my company pays Freexian at the bronze
tier to do Debian LTS packaging work and I understand they do wider
Debian consultancy, so maybe that is a good option.
https://www.freexian.com/services/debian-support/
There's the debian-consultants mailing list but it gets almost no
traffic. I don't think it will aid you in finding a consultancy any
https://lists.debian.org/debian-consultants/
Just as some free advice though…
1. I find it hard to believe you have more than 2000 Debian installs
without some sort of existing automation / configuration
management
2. Given (1), I would approach the task by learning your config
management and modifying it to deploy a Debian 12 version of each
kind of Debian 11 server you already have.
3. I'd then do a rolling deploy that slowly takes Debian 11 servers
out of service and re-provisions them as Debian 12. I would not
try to upgrade anything in place. Although Debian supports that,
at scale I find it harder to account for all variables than with
a clean install, and if you already have automation to deploy and
configure a host then an in-lace upgrade also takes longer in my
experience.
Something to chat to your consultancy about, anyway.
It would be really interesting if you or your consultant would write
up what you ended up doing.
Thanks,
Andy

[Dan Ritter]

Post by Andy Smith
Just as some free advice though…
1. I find it hard to believe you have more than 2000 Debian installs
without some sort of existing automation / configuration
management
2. Given (1), I would approach the task by learning your config
management and modifying it to deploy a Debian 12 version of each
kind of Debian 11 server you already have.
3. I'd then do a rolling deploy that slowly takes Debian 11 servers
out of service and re-provisions them as Debian 12. I would not
try to upgrade anything in place. Although Debian supports that,
at scale I find it harder to account for all variables than with
a clean install, and if you already have automation to deploy and
configure a host then an in-lace upgrade also takes longer in my
experience.

We do hundreds rather than thousands, but we do them:

- with an existing configuration automation system (chef/cinc)

- in-place upgrades

- in tiers, where a given function (e.g. web servers) will have
representative machines in each tier, starting with a very
small proof-of-concept upgrade, followed by corrections; then
a somewhat larger upgrade group, followed by all the rest of
the machines.

Automated monitoring, too.

-dsr-