[Spacewalk-list] Ubuntu 18.04 package management in Spacewalk 2.8

[philippe bidault]

Hi all,

I am trying to make our Spacewalk 2.8 working with Ubuntu 18.04, but as
well described here :
https://github.com/spacewalkproject/spacewalk/wiki/DebianUbuntuSupportIn27

... there are 3 main issues :

"1- The current version comparison logic does not distinct a dot from an
hyphen, a tilde or a plus character. This leads to some packages wrongly
shown as an update for a client in spacewalk when the package is actually a
downgrade. The client however uses a correct comparison and handles the
package upgrade correctly

2- The deb importer does not import all the package header information into
the database and the repository-writer will not write the missing
information to the repository metadata served by spacewalk. This will lead
to problems on the client in case of the missing Multi-Arch header: Clients
will try to reinstall the same package over and over again when this header
is missing.

3- A deb repository provided by spacewalk is not GPG signed and thus will
not work without disabling secure-apt. Spacewalk imports and recreates the
repository based on the imported package catalogue, this will destroy the
GPG signing of the repository vendor."

I did manage to solve the point 2 and 3. The point 3 thanks to this method
:
http://www.devops-blog.net/spacewalk/gpg-signing-apt-repository-in-spacewalk

Regarding the point 2, I did have to customize the script
https://github.com/rpasche/spacewalk-debian-sync/tree/add-multiarch-header ,
as it seems that in any case the add of the Multi-Arch header in
Packages.gz is not enough for Ubuntu 18.04. I did only succeed in not
ending in an infinite loop of packages flagged to be upgraded as the result
of "apt upgrade" by adding more headers in Packages.gz :

p, v, a, multi, breaks, predeps = line.rstrip().split("||")
if (packages.has_key(p + v + a) and (multi is not "#")) :
packages[p + v + a]['Multi-Arch'] = multi
if (packages.has_key(p + v + a) and (breaks is not "#")) :
packages[p + v + a]['Breaks'] = breaks
if (packages.has_key(p + v + a) and (predeps is not "#")) :
packages[p + v + a]['Pre-Depends'] = predeps

Right now, what is really causing me much more trouble, is the point 1,
which is making Spacewalk not really reliable for deb package management ,
as I have a permanent list of upgradable packages in the Spacewalk console,
whereas the installed versions are already the latest.

Example :


Latest Package
Installed Package
gcc-8-base-8-20180414-1ubuntu2.amd64-deb
<https://space01/rhn/software/packages/Details.do?sid=1000010058&id_combo=21120|7544|145>
gcc-8-base-8.2.0-1ubuntu2~18.04.amd64-deb
lib32gcc1-8-20180414-1ubuntu2:1.amd64-deb
<https://space01/rhn/software/packages/Details.do?sid=1000010058&id_combo=21933|7796|145>
lib32gcc1-8.2.0-1ubuntu2~18.04:1.amd64-deb


So in resume, I have 2 questions :

- Regarding the point 1, is there already an existing solution I could use,
or at least some clues ?
- Regarding the point 2 : Did I miss something, or do we really need to add
some extra headers (Breaks and Pre-depends) to have the Ubuntu 18.04 client
servers correctly listing packages to be upgraded ?

Regards,
Philippe.

[Robert Paschedag]

Am 5. Dezember 2018 00:44:25 MEZ schrieb philippe bidault <***@gmail.com>:
>Hi all,
>
>I am trying to make our Spacewalk 2.8 working with Ubuntu 18.04, but as
>well described here :
>https://github.com/spacewalkproject/spacewalk/wiki/DebianUbuntuSupportIn27
>
>... there are 3 main issues :
>
>"1- The current version comparison logic does not distinct a dot from
>an
>hyphen, a tilde or a plus character. This leads to some packages
>wrongly
>shown as an update for a client in spacewalk when the package is
>actually a
>downgrade. The client however uses a correct comparison and handles the
>package upgrade correctly
>
>2- The deb importer does not import all the package header information
>into
>the database and the repository-writer will not write the missing
>information to the repository metadata served by spacewalk. This will
>lead
>to problems on the client in case of the missing Multi-Arch header:
>Clients
>will try to reinstall the same package over and over again when this
>header
>is missing.
>
>3- A deb repository provided by spacewalk is not GPG signed and thus
>will
>not work without disabling secure-apt. Spacewalk imports and recreates
>the
>repository based on the imported package catalogue, this will destroy
>the
>GPG signing of the repository vendor."
>
>I did manage to solve the point 2 and 3. The point 3 thanks to this
>method
>:
>http://www.devops-blog.net/spacewalk/gpg-signing-apt-repository-in-spacewalk
>
>Regarding the point 2, I did have to customize the script
>https://github.com/rpasche/spacewalk-debian-sync/tree/add-multiarch-header
>,
>as it seems that in any case the add of the Multi-Arch header in
>Packages.gz is not enough for Ubuntu 18.04. I did only succeed in not
>ending in an infinite loop of packages flagged to be upgraded as the
>result
>of "apt upgrade" by adding more headers in Packages.gz :
>
> p, v, a, multi, breaks, predeps = line.rstrip().split("||")
> if (packages.has_key(p + v + a) and (multi is not "#")) :
> packages[p + v + a]['Multi-Arch'] = multi
> if (packages.has_key(p + v + a) and (breaks is not "#")) :
> packages[p + v + a]['Breaks'] = breaks
> if (packages.has_key(p + v + a) and (predeps is not "#")) :
> packages[p + v + a]['Pre-Depends'] = predeps

I did improve the script (still testing). Basically it just adds all missing headers from the original file to that generated by spacewalk. The repo is not yet updated.

Robert
>
>Right now, what is really causing me much more trouble, is the point 1,
>which is making Spacewalk not really reliable for deb package
>management ,
>as I have a permanent list of upgradable packages in the Spacewalk
>console,
>whereas the installed versions are already the latest.
>
>Example :
>
>
>Latest Package
>Installed Package
>gcc-8-base-8-20180414-1ubuntu2.amd64-deb
><https://space01/rhn/software/packages/Details.do?sid=1000010058&id_combo=21120|7544|145>
>gcc-8-base-8.2.0-1ubuntu2~18.04.amd64-deb
>lib32gcc1-8-20180414-1ubuntu2:1.amd64-deb
><https://space01/rhn/software/packages/Details.do?sid=1000010058&id_combo=21933|7796|145>
>lib32gcc1-8.2.0-1ubuntu2~18.04:1.amd64-deb
>
>
>So in resume, I have 2 questions :
>
>- Regarding the point 1, is there already an existing solution I could
>use,
>or at least some clues ?
>- Regarding the point 2 : Did I miss something, or do we really need to
>add
>some extra headers (Breaks and Pre-depends) to have the Ubuntu 18.04
>client
>servers correctly listing packages to be upgraded ?
>
>Regards,
>Philippe.


--
sent from my mobile device

[philippe bidault]

Hi Robert,

Good to know, thanks !!

Regarding the version comparison logic issue for .deb packages, is there a
solution that could be implemented ?

Regards,
Philippe.

On Wed, 5 Dec 2018 at 20:18, Robert Paschedag <***@web.de>
wrote:

> Am 5. Dezember 2018 00:44:25 MEZ schrieb philippe bidault <
> ***@gmail.com>:
> >Hi all,
> >
> >I am trying to make our Spacewalk 2.8 working with Ubuntu 18.04, but as
> >well described here :
> >
> https://github.com/spacewalkproject/spacewalk/wiki/DebianUbuntuSupportIn27
> >
> >... there are 3 main issues :
> >
> >"1- The current version comparison logic does not distinct a dot from
> >an
> >hyphen, a tilde or a plus character. This leads to some packages
> >wrongly
> >shown as an update for a client in spacewalk when the package is
> >actually a
> >downgrade. The client however uses a correct comparison and handles the
> >package upgrade correctly
> >
> >2- The deb importer does not import all the package header information
> >into
> >the database and the repository-writer will not write the missing
> >information to the repository metadata served by spacewalk. This will
> >lead
> >to problems on the client in case of the missing Multi-Arch header:
> >Clients
> >will try to reinstall the same package over and over again when this
> >header
> >is missing.
> >
> >3- A deb repository provided by spacewalk is not GPG signed and thus
> >will
> >not work without disabling secure-apt. Spacewalk imports and recreates
> >the
> >repository based on the imported package catalogue, this will destroy
> >the
> >GPG signing of the repository vendor."
> >
> >I did manage to solve the point 2 and 3. The point 3 thanks to this
> >method
> >:
> >
> http://www.devops-blog.net/spacewalk/gpg-signing-apt-repository-in-spacewalk
> >
> >Regarding the point 2, I did have to customize the script
> >
> https://github.com/rpasche/spacewalk-debian-sync/tree/add-multiarch-header
> >,
> >as it seems that in any case the add of the Multi-Arch header in
> >Packages.gz is not enough for Ubuntu 18.04. I did only succeed in not
> >ending in an infinite loop of packages flagged to be upgraded as the
> >result
> >of "apt upgrade" by adding more headers in Packages.gz :
> >
> > p, v, a, multi, breaks, predeps = line.rstrip().split("||")
> > if (packages.has_key(p + v + a) and (multi is not "#")) :
> > packages[p + v + a]['Multi-Arch'] = multi
> > if (packages.has_key(p + v + a) and (breaks is not "#")) :
> > packages[p + v + a]['Breaks'] = breaks
> > if (packages.has_key(p + v + a) and (predeps is not "#")) :
> > packages[p + v + a]['Pre-Depends'] = predeps
>
> I did improve the script (still testing). Basically it just adds all
> missing headers from the original file to that generated by spacewalk. The
> repo is not yet updated.
>
> Robert
> >
> >Right now, what is really causing me much more trouble, is the point 1,
> >which is making Spacewalk not really reliable for deb package
> >management ,
> >as I have a permanent list of upgradable packages in the Spacewalk
> >console,
> >whereas the installed versions are already the latest.
> >
> >Example :
> >
> >
> >Latest Package
> >Installed Package
> >gcc-8-base-8-20180414-1ubuntu2.amd64-deb
> ><
> https://space01/rhn/software/packages/Details.do?sid=1000010058&id_combo=21120|7544|145
> >
> >gcc-8-base-8.2.0-1ubuntu2~18.04.amd64-deb
> >lib32gcc1-8-20180414-1ubuntu2:1.amd64-deb
> ><
> https://space01/rhn/software/packages/Details.do?sid=1000010058&id_combo=21933|7796|145
> >
> >lib32gcc1-8.2.0-1ubuntu2~18.04:1.amd64-deb
> >
> >
> >So in resume, I have 2 questions :
> >
> >- Regarding the point 1, is there already an existing solution I could
> >use,
> >or at least some clues ?
> >- Regarding the point 2 : Did I miss something, or do we really need to
> >add
> >some extra headers (Breaks and Pre-depends) to have the Ubuntu 18.04
> >client
> >servers correctly listing packages to be upgraded ?
> >
> >Regards,
> >Philippe.
>
>
> --
> sent from my mobile device
>

[Robert Paschedag]

_______________________________________________
Spacewalk-list mailing list
Spacewalk-***@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

[philippe bidault]

Hi Robert,

Thanks for the script, really appreciated. Already implemented in my
Spacewalk and for the moment, working like a charm.
In fact it even solved an issue I had with the "netplan.io" package always
flagged as upgradable (I guess because of a missing header).

Regarding the fake-upgradable Debian packages appearing on Spacewalk, I
have 12 of them for the moment :

 gcc-8-base-8-20180414-1ubuntu2.amd64-deb
gcc-8-base-8.2.0-1ubuntu2~18.04.amd64-deb

lib32gcc1-8-20180414-1ubuntu2:1.amd64-deb
lib32gcc1-8.2.0-1ubuntu2~18.04:1.amd64-deb

libatomic1-8-20180414-1ubuntu2.amd64-deb
libatomic1-8.2.0-1ubuntu2~18.04.amd64-deb

libcc1-0-8-20180414-1ubuntu2.amd64-deb libcc1-0-8.2.0-1ubuntu2~18.04.amd64-deb

libgcc1-8-20180414-1ubuntu2:1.amd64-deb
libgcc1-8.2.0-1ubuntu2~18.04:1.amd64-deb

libgomp1-8-20180414-1ubuntu2.amd64-deb libgomp1-8.2.0-1ubuntu2~18.04.amd64-deb

libitm1-8-20180414-1ubuntu2.amd64-deb libitm1-8.2.0-1ubuntu2~18.04.amd64-deb

liblsan0-8-20180414-1ubuntu2.amd64-deb liblsan0-8.2.0-1ubuntu2~18.04.amd64-deb

libmpx2-8-20180414-1ubuntu2.amd64-deb libmpx2-8.2.0-1ubuntu2~18.04.amd64-deb

libquadmath0-8-20180414-1ubuntu2.amd64-deb
libquadmath0-8.2.0-1ubuntu2~18.04.amd64-deb

libstdc++6-8-20180414-1ubuntu2.amd64-deb
libstdc++6-8.2.0-1ubuntu2~18.04.amd64-deb

libtsan0-8-20180414-1ubuntu2.amd64-deb
libtsan0-8.2.0-1ubuntu2~18.04.amd64-deb

Will try to dig, but indeed if the versioning motor is managed by Java and
not Python, will be hard to debug ...

Regards,
Philippe.

On Wed, 5 Dec 2018 at 22:08, Robert Paschedag <***@web.de>
wrote:

> Hi Philippe,
>
> Well....until now, I only have very few packages that are shown as
> upgradable for "Debian" systems. I only found some version comparision
> problem while testing Uyuni (and testing to get Debian system registered as
> Salt client). I think this could be further tuned to use the python-apt or
> python-dpkg package...but I did not have time to dig into that.
>
> Within spacewalk, I currently do not know, where the comparision takes
> place (within java or within python).
>
> The "modified" script is a "new" one. It does not depend on the - from
> myself - modified sync script. What it needs is the original Packages.gz
> file for the channel you sync.
>
> Just throwing this here so you can test yourself. Basically, it should add
> all "missing" headers for every package.
>
> import sys
> import os
> from debian.deb822 import *
> if len(sys.argv) < 3:
> print "Usage: %s <path_to_unzipped_Packages_from_Spacewalk_Uyuni>
> <path_to_original_Debian_repo>" % sys.argv[0]
> sys.exit(1)
> spacewalk_file = sys.argv[1]
> original_file = sys.argv[2]
> if not os.path.isfile(spacewalk_file):
> print "Error: Inputfile '%s' not available." % spacewalk_file
> sys.exit(1)
> if not os.path.isfile(original_file):
> print "Error: Inputfile '%s' not available." % original_file
> sys.exit(1)
> spacewalk_packages = {}
> original_packages = {}
> with open(spacewalk_file, 'r') as pkgs:
> for pkg in Packages.iter_paragraphs(pkgs):
> spacewalk_packages[pkg['Package'] + pkg['Version'] +
> pkg['Architecture']] = pkg
> with open(original_file, 'r') as orig_file:
> for pkg in Packages.iter_paragraphs(orig_file):
> p = pkg['Package']
> v = pkg['Version']
> a = pkg['Architecture']
> if spacewalk_packages.has_key(p + v + a):
> # found package. Check for missing headers
> for header in pkg.keys():
> if not header in spacewalk_packages[p + v + a].keys():
> spacewalk_packages[p + v + a][header] = pkg[header]
> # open new file
> new_package = open(spacewalk_file + '.new', 'w')
> for pkg in spacewalk_packages.values():
> pkg.dump(new_package)
> new_package.write("\n")
> new_package.close()
> sys.exit(0)
>
> Cheers,
> Robert
>
> *Gesendet:* Mittwoch, 05. Dezember 2018 um 21:10 Uhr
> *Von:* "philippe bidault" <***@gmail.com>
> *An:* ***@web.de
> *Cc:* spacewalk-***@redhat.com
> *Betreff:* Re: [Spacewalk-list] Ubuntu 18.04 package management in
> Spacewalk 2.8
> Hi Robert,
>
> Good to know, thanks !!
>
> Regarding the version comparison logic issue for .deb packages, is there a
> solution that could be implemented ?
>
> Regards,
> Philippe.
>
> On Wed, 5 Dec 2018 at 20:18, Robert Paschedag <***@web.de>
> wrote:
>
>> Am 5. Dezember 2018 00:44:25 MEZ schrieb philippe bidault <
>> ***@gmail.com>:
>> >Hi all,
>> >
>> >I am trying to make our Spacewalk 2.8 working with Ubuntu 18.04, but as
>> >well described here :
>> >
>> https://github.com/spacewalkproject/spacewalk/wiki/DebianUbuntuSupportIn27
>> >
>> >... there are 3 main issues :
>> >
>> >"1- The current version comparison logic does not distinct a dot from
>> >an
>> >hyphen, a tilde or a plus character. This leads to some packages
>> >wrongly
>> >shown as an update for a client in spacewalk when the package is
>> >actually a
>> >downgrade. The client however uses a correct comparison and handles the
>> >package upgrade correctly
>> >
>> >2- The deb importer does not import all the package header information
>> >into
>> >the database and the repository-writer will not write the missing
>> >information to the repository metadata served by spacewalk. This will
>> >lead
>> >to problems on the client in case of the missing Multi-Arch header:
>> >Clients
>> >will try to reinstall the same package over and over again when this
>> >header
>> >is missing.
>> >
>> >3- A deb repository provided by spacewalk is not GPG signed and thus
>> >will
>> >not work without disabling secure-apt. Spacewalk imports and recreates
>> >the
>> >repository based on the imported package catalogue, this will destroy
>> >the
>> >GPG signing of the repository vendor."
>> >
>> >I did manage to solve the point 2 and 3. The point 3 thanks to this
>> >method
>> >:
>> >
>> http://www.devops-blog.net/spacewalk/gpg-signing-apt-repository-in-spacewalk
>> >
>> >Regarding the point 2, I did have to customize the script
>> >
>> https://github.com/rpasche/spacewalk-debian-sync/tree/add-multiarch-header
>> >,
>> >as it seems that in any case the add of the Multi-Arch header in
>> >Packages.gz is not enough for Ubuntu 18.04. I did only succeed in not
>> >ending in an infinite loop of packages flagged to be upgraded as the
>> >result
>> >of "apt upgrade" by adding more headers in Packages.gz :
>> >
>> > p, v, a, multi, breaks, predeps = line.rstrip().split("||")
>> > if (packages.has_key(p + v + a) and (multi is not "#")) :
>> > packages[p + v + a]['Multi-Arch'] = multi
>> > if (packages.has_key(p + v + a) and (breaks is not "#")) :
>> > packages[p + v + a]['Breaks'] = breaks
>> > if (packages.has_key(p + v + a) and (predeps is not "#")) :
>> > packages[p + v + a]['Pre-Depends'] = predeps
>>
>> I did improve the script (still testing). Basically it just adds all
>> missing headers from the original file to that generated by spacewalk. The
>> repo is not yet updated.
>>
>> Robert
>> >
>> >Right now, what is really causing me much more trouble, is the point 1,
>> >which is making Spacewalk not really reliable for deb package
>> >management ,
>> >as I have a permanent list of upgradable packages in the Spacewalk
>> >console,
>> >whereas the installed versions are already the latest.
>> >
>> >Example :
>> >
>> >
>> >Latest Package
>> >Installed Package
>> >gcc-8-base-8-20180414-1ubuntu2.amd64-deb
>> ><
>> https://space01/rhn/software/packages/Details.do?sid=1000010058&id_combo=21120|7544|145
>> >
>> >gcc-8-base-8.2.0-1ubuntu2~18.04.amd64-deb
>> >lib32gcc1-8-20180414-1ubuntu2:1.amd64-deb
>> ><
>> https://space01/rhn/software/packages/Details.do?sid=1000010058&id_combo=21933|7796|145
>> >
>> >lib32gcc1-8.2.0-1ubuntu2~18.04:1.amd64-deb
>> >
>> >
>> >So in resume, I have 2 questions :
>> >
>> >- Regarding the point 1, is there already an existing solution I could
>> >use,
>> >or at least some clues ?
>> >- Regarding the point 2 : Did I miss something, or do we really need to
>> >add
>> >some extra headers (Breaks and Pre-depends) to have the Ubuntu 18.04
>> >client
>> >servers correctly listing packages to be upgraded ?
>> >
>> >Regards,
>> >Philippe.
>>
>>
>> --
>> sent from my mobile device
>
>

[Paul-Andre Panon]

This post might be inappropriate. Click to display it.

[Paul-Andre Panon]

Earlier today I wrote about the changes in PR500. That parsing actually
seems to be OK after all because the EVR entries in the database are OK.

That said, it looks like the query that generates the list of upgradeable
packages in the Spacewalk GUI is system_upgradable_package_list in
spacewalk/java/code/src/com/redhat/rhn/common/db/datasource/xml/Package_qu
eries.xml

<mode name="system_upgradable_package_list"
class="com.redhat.rhn.frontend.dto.UpgradablePackageListItem">
<query params="sid">
SELECT n.id AS id,
n.id AS name_id,
lookup_evr(((latest.evr)).epoch, (latest.evr).version,
(latest.evr).release) AS evr_id,
latest.package_arch_id AS arch_id,
(latest.evr).epoch AS epoch,
(latest.evr).version AS version,
(latest.evr).release AS release,
n.name AS name,
n.name ||'-'|| evr_t_as_vre_simple(latest.evr) || '.' ||
latest_pa.label AS nvrea,
n.name ||'-'|| evr_t_as_vre_simple(spe.evr) || '.' || spa.label AS
installed_package,
n.id || '|' || lookup_evr((latest.evr).epoch,
(latest.evr).version, (latest.evr).release)|| '|' ||
latest.package_arch_id AS id_combo
FROM
rhnServerPackage sp
join rhnPackageName n
on n.id = sp.name_id
join rhnPackageArch spa
on spa.id = sp.package_arch_id
join rhnPackageEvr spe
on spe.id = sp.evr_id
join (
select sop.package_name_id,
sop.package_arch_id,
max(PE.evr) evr
from rhnServerOutdatedPackages sop
join rhnPackageEVR pe
on sop.package_evr_id = pe.id
where sop.server_id = :sid
group by sop.package_name_id, sop.package_arch_id) latest
on latest.package_name_id = sp.name_id
join rhnPackageArch latest_pa
on latest_pa.id = latest.package_arch_id
join rhnPackageUpgradeArchCompat puac
on puac.package_arch_id = sp.package_arch_id
and puac.package_upgrade_arch_id = latest.package_arch_id
where sp.server_id = :sid
order by upper(n.name)
</query>
<elaborator multiple="t" params="sid">
SELECT PN.id AS id,
SOP.errata_id AS errata_id,
SOP.errata_advisory AS errata_advisory,
E.advisory_type AS errata_advisory_type,
E.severity_id AS errata_severity_id
FROM rhnServerOutdatedPackages SOP
INNER JOIN rhnPackageName PN on SOP.package_name_id = PN.id
INNER JOIN rhnErrata E on SOP.errata_id = E.id
WHERE PN.id IN (%s)
AND SOP.server_id = :sid
</elaborator>
</mode>

The sub-clause

join (
select sop.package_name_id,
sop.package_arch_id,
max(PE.evr) evr
from rhnServerOutdatedPackages sop
join rhnPackageEVR pe
on sop.package_evr_id = pe.id
where sop.server_id = :sid
group by sop.package_name_id, sop.package_arch_id) latest

is returning the gcc_8-related packages because they are provided by the
rhnServerOutdatedPackages view

SELECT DISTINCT SNC.server_id,
P.name_id,
P.evr_id,
P.package_arch_id,
PN.name || '-' || evr_t_as_vre_simple( PE.evr ),
E.id,
E.advisory
FROM rhnPackageName PN,
rhnPackageEVR PE,
rhnPackage P,
rhnServerNeededCache SNC
left outer join
rhnErrata E
on SNC.errata_id = E.id
WHERE SNC.package_id = P.id
AND P.name_id = PN.id
AND P.evr_id = PE.id

and the view is listing those packages because of entries for those
packages in the rhnServerNeededCache table. So the question is what is
populating the rhnServerNeededCache table incorrectly, and why?