Home

Linked-list debugging in gdb

Jeremy Kerr - 3 min 38 sec ago

Quite a few of the codebases I've worked on have some form of linked list implementation; generally a list node type:

struct list_node {
    struct list_node *prev;
    struct list_node *next;
};

... and sometimes (for type-safety) a separate type for the lists themselves:

struct list {
    struct list_node head;
};

Debugging these lists often means a lot of manual pointer-chasing, which can be error prone. If your list iterators are causing segfaults, it'd be nice to be able to track down invalid list entries.

python gdb extensions

It turns out we can write a little code to automate this, using the gdb module. Amongst other things, we can define our own commands to use in the standard gdb interpreter.

The skeleton code to implement a gdb command in python goes something like this:

import gdb

class MyNewCommand(gdb.Command):

    def __init__(self):
        super(MyNewCommand, self).__init__("my-new-command",
                gdb.COMMAND_DATA, gdb.COMPLETE_SYMBOL)

    def invoke(self, argument, from_tty):
        args = gdb.string_to_argv(argument)
        expr = args[0]
        list = gdb.parse_and_eval(expr)

        # command implementation goes here ...

MyNewCommand()

In a nutshell:

  • We define a new subclass of gdb.Command
  • Our new class' __init__() function will register the command name ("my-new-command" in this example) with the gdb interpreter, and pass a few hints about the command's usage.
  • The invoke() function is called when the user invokes the command at the gdb interpreter interface.
  • The invoke() implementation can access the debugger's state thorugh the gdb python module.
  • Finally, we create an instance of the new class to get it registered with the interpreter.

The class' docstring is used by gdb's inline help command, which makes documenting your new command a total piece of cake.

Back to our list example, I've written a little python gdb extension to iterate and print a list:

#!/usr/bin/env python

import gdb

class ListPrintCommand(gdb.Command):
    """Iterate and print a list.

list-print <EXPR> [MAX]

Given a list EXPR, iterate though the list nodes' ->next pointers, printing
each node iterated. We will iterate thorugh MAX list nodes, to prevent
infinite loops with corrupt lists. If MAX is zero, we will iterate the
entire list.

List nodes types are expected to have a member named "next". List types
may be the same as node types, or a separate type with an explicit
head node, called "head"."""

    MAX_ITER = 20

    def __init__(self):
        super(ListPrintCommand, self).__init__("list-print",
                gdb.COMMAND_DATA, gdb.COMPLETE_SYMBOL)

    def invoke(self, argument, from_tty):
        args = gdb.string_to_argv(argument)
        if len(args) == 0:
            print "Argument required (list to iterate)"
            return

        expr = args[0]

        if len(args) == 2:
            max_iter = int(args[1])
        else:
            max_iter = self.MAX_ITER

        list = gdb.parse_and_eval(expr)

        fnames = [ f.name for f in list.type.fields() ]

        # handle lists with a separate list type....
        if 'head' in fnames:
            head = list['head']

        # ...and those with the head as a regular node
        elif 'next' in fnames:
            head = list

        else:
            print "Unknown list head type"
            return

        # if the type has a 'prev' member, we check for validity as we walk
        # the list
        check_prev = 'prev' in [ f.name for f in head.type.fields() ]

        print "list@%s: %s" % (head.address, head)
        node = head['next']
        prev = head.address
        i = 1

        while node != head.address:
            print "node@%s: %s #%d" % (node, node.dereference(), i)

            if check_prev and prev != node['prev']:
                    print " - invalid prev pointer!"

            if i == max_iter:
                print " ... (max iterations reached)"
                break

            prev = node
            node = node['next']
            i += 1

        if check_prev and i != max_iter and head['prev'] != prev:
            print "list has invalid prev pointer!"

ListPrintCommand()

This defines a new function, list-print, which takes an expression as an argument, and iterates through the list nodes:

(gdb) list-print handler->devices[0]->device->boot_options
list@0x6128f0: {prev = 0x6129f8, next = 0x613918}
node@0x613918: {prev = 0x6128f0, next = 0x613568} #1
node@0x613568: {prev = 0x613918, next = 0x6131b8} #2
node@0x6131b8: {prev = 0x613568, next = 0x612e08} #3
node@0x612e08: {prev = 0x6131b8, next = 0x6129f8} #4
node@0x6129f8: {prev = 0x612e08, next = 0x6128f0} #5
(gdb) 

To use the gdb-list macro: download gdb-list.py (2.2 kB), and source it into your gdb session:

(gdb) source ~/devel/gdb-list/gdb-list.py 

Then you should be able to invoke list-print <symbol> to debug your list structures. Typing help list-print will show the command's docstring:

(gdb) help list-print 
Iterate and print a list.

list-print <EXPR> [MAX]

Given a list EXPR, iterate though the list nodes' ->next pointers, printing
each node iterated. We will iterate thorugh MAX list nodes, to prevent
infinite loops with corrupt lists. If MAX is zero, we will iterate the
entire list.

List nodes types are expected to have a member named "next". List types
may be the same as node types, or a separate type with an explicit
head node, called "head".
(gdb) 

UDS-R wrapup

Jeremy Kerr - 3 min 38 sec ago

The Ubuntu Developer Summit was held in Copenhagen last week, to discuss plans for the next six-month cycle of Ubuntu. This was the most productive UDS that I've been to — maybe it was the shorter four-day schedule, or the overlap with Linaro Connect, but it sure felt like a whirlwind of activity.

I thought I'd share some details about some of the sessions that cover areas I'm working on at the moment. In no particular order:

Improving cross-compilation

Blueprint: foundations-r-improve-cross-compilation

This plan is a part of a mutli-cycle effort to improve cross-compilation support in Ubuntu. Progress is generally going well — the consensus from the session was that the components are fairly close to complete, but we still need some work to pull those parts together into something usable.

So, this cycle we'll be working on getting that done. While we have a few bugfixes and infrastructure updates to do, one significant part of this cycle’s work will be to document the “best-practices” for cross builds in Ubuntu, on wiki.ubuntu.com. This process will be heavily based on existing pages on the Linaro wiki. Because most of the support for cross-building is already done, the actual process for cross-building should be fairly straightforward, but needs to be defined somewhere.

I'll post an update when we have a working draft on the Ubuntu wiki, stay tuned for details.

Rapid archive bringup for new hardware

Blueprint: foundations-r-rapid-archive-bringup

I'd really like for there to be a way to get an Ubuntu archive built “from scratch”, to enable custom toolchain/libc/other system components to be built and tested. This is typically useful when bringing up new hardware, or testing rebuilds with new compiler settings. Because we may be dealing with new hardware, doing this bootstrap through cross-compilation is something we'd like too.

Eventually, it would be great to have something as straightforward as the OpenEmbedded or OpenWRT build process to construct a repository with a core set of Ubuntu packages (say, minbase), for previously-unsupported hardware.

The archive bootstrap process isn't done often, and can require a large amount of manual intervention. At present, there's only a couple of folks who know how to get it working. The plan here is to document the bootstrap process in this cycle, so that others can replicate the process, and possibly improve the bits that are a little too janky for general consumption.

ARM64 / ARMv8 / aarch64 support

Blueprint: foundations-r-aarch64

This session is an update for progress on the support for ARMv8 processors in Ubuntu. While no general-purpose hardware exists at the moment, we want to have all the pieces ready for when we start seeing initial implementations. Because we don't have hardware yet, this work has to be done in a cross-build environment; another reason to keep on with the foundations-r-improve-cross-compilation plan!

So far, toolchain progress is going well, with initial cross toolchains available for quantal.

Although kernel support isn’t urgent at the moment, we’ll be building an initial kernel-headers package for aarch64. There's also a plan to get a page listing the aarch64-cross build status of core packages, so we'll know what is blocked for 64-bit ARM enablement.

We’ve also got a bunch of workitems for volunteers to fix cross-build issues as they arise. If you're interested, add a workitem in the blueprint, and keep an eye on it for updates.

Secure boot support in Ubuntu

Blueprint: foundations-r-secure-boot

This session covered progress of secure boot support as at the 12.10 Quantal Quetzal release, items that are planned for 13.04, and backports for 12.04.2.

As for 12.10, we’ve got the significant components of secure boot support into the release — the signed boot chain. The one part that hasn't hit 12.10 yet is the certificate management & update infrastructure, but that is planned to reach 12.10 by way of a not-too-distant-future update.

The foundations team also mentioned that they were starting the 12.04.2 backport right after UDS, which will bring secure boot support to our current “Long Term Support” (LTS) release. Since the LTS release is often preferred Ubuntu preinstall situations, this may be used as a base for hardware enablement on secure boot machines. Combined with the certificate management tools (described at sbkeysync & maintaining uefi key databases), and the requirement for “custom mode” in general-purpose hardware, this will allow for user-defined trust configuration in an LTS release.

As for 13.04, we're planning to update the shim package to a more recent version, which will have Matthew Garrett's work on the Machine Owner Key plan from SuSE.

We're also planning to figure out support for signed kernel modules, for users who wish to verify all kernel-level code. Of course, this will mean some changes to things like DKMS, which run custom module builds outside of the normal Ubuntu packages.

Netboot with secure boot is still in progress, and will require some fixes to GRUB2.

And finally, the sbsigntools codebase could do with some new testcases, particularly for the PE/COFF parsing code. If you're interested in contributing, please contact me at jeremy.kerr@canonical.com.

aarch64 cross compiler packages for Ubuntu & Debian

Jeremy Kerr - 3 min 38 sec ago

Cross-compile-o-naut wookey has recently posted a set of Debian & Ubuntu packages for an aarch64 (ie, 64-bit ARMv8) cross compiler:

The repositories here contain sources and binaries for the arm64 bootstrap in Debian (unstable) and Ubuntu (quantal). There are both toolchain and tools packages for amd64 build machines and arm64 binaries built with them. And corresponding sources.

For the lazy:

[jk@pecola ~]$ wget -O - http://people.debian.org/~wookey/bootstrap/bootstrap-archive.key |
    sudo apt-key add -
[jk@pecola ~]$ sudo apt-add-repository 'deb http://people.debian.org/~wookey/bootstrap/ubunturepo/ quantal-bootstrap main universe'
[jk@pecola ~]$ sudo apt-get update
[jk@pecola ~]$ sudo apt-get install --install-recommends gcc-4.7-aarch64-linux-gnu

... which gives you a cross compiler for aarch64:

[jk@pecola ~]$ cat helloworld.c 
#include <stdio.h>
int main(void) { printf("hello world\n"); return 0; }
[jk@pecola ~]$ aarch64-linux-gnu-gcc-4.7 -Wall -O2 -o helloworld helloworld.c
[jk@pecola ~]$ aarch64-linux-gnu-objdump -f helloworld 
helloworld:     file format elf64-littleaarch64
architecture: aarch64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000400450

aarch64 support in upstream binutils

Jeremy Kerr - 3 min 38 sec ago

Yufeng Zhang has recently posted that support for the aarch64 architecture (ie., 64-bit ARMv8 processors) has now landed in the official binutils git repository. This code should make its way into the next release of binutils, version 2.23. The relevant commit is e7dff39c, the ChangeLog update is here.

I've updated the ARM64 toolchain document to refer to the upstream sources, and a test build runs without problems. Nice work, ARM team!

sbkeysync & maintaining uefi key databases

Jeremy Kerr - 3 min 38 sec ago

Over the last couple of weeks I've been working on a set of secure-boot tools. Originally, this project was intended as a quick implementation of an EFI-image-signing utility, but it has since grown a little. I've now added code to help maintain the UEFI signature databases from within a running OS.

A new utility, sbkeysync, reads the current EFI signature databases from firmware, and reads a set of keys from a standard location in the filesystem - for example, /etc/secureboot/keys. It then updates the firmware key databases with any keys that are not already present.

A filesystem keystore will look something like this:

  • /etc/secureboot/keys/PK/<pk-file>
  • /etc/secureboot/keys/KEK/<kek-file-1>
  • /etc/secureboot/keys/KEK/<kek-file-2>
  • /etc/secureboot/keys/KEK/…
  • /etc/secureboot/keys/db/<db-file-1>
  • /etc/secureboot/keys/db/<db-file-2>
  • /etc/secureboot/keys/db/…
  • /etc/secureboot/keys/dbx/<dbx-file-1>
  • /etc/secureboot/keys/dbx/<dbx-file-2>
  • /etc/secureboot/keys/dbx/…

These files need to be in a certain format: signed EFI_SIGNATURE_LIST data. There's two other utilities in the sbtools tree to help to create the key files: sbsiglist and sbvarsign. The following example shows how you'd use these tools to do a basic secure boot key configuration.

An example key setup

If you're interested in trying sbkeysync, the following guide should get you set up. To start, you'll need:

  • A build of the secure boot tools (git repository information below);
  • A kernel with the efivars filesystem. Either build your own kernel with efivars-1f087c6.patch (from Matthew Garrett, with some minor changes), or use linux-image-3.5.0-13-generic_3.5.0-13.14~efivars1_amd64.deb, which should work on Ubuntu 12.04 or 12.10; and
  • A machine with firmware that implements UEFI secure boot, configured to be in setup mode (ie, no PK installed).

Be warned that you're playing with three different layers of development code here: the secure boot tools are new, the efivars implementation hasn't had a lot of review yet, and firmware secure boot implementations are still fairly recent too. I'd recommend against doing this testing on a production machine.

generating keys

We'll generate a test key, and a self-signed certificate:

[jk@pecola ~]$ openssl genrsa -out test-key.rsa 2048
[jk@pecola ~]$ openssl req -new -x509 -sha256 \
        -subj '/CN=test-key' -key test-key.rsa -out test-cert.pem
[jk@pecola ~]$ openssl x509 -in test-cert.pem -inform PEM \
        -out test-cert.der -outform DER

We'll also need a GUID to represent the "key owner". Just generate one with uuidgen.

[jk@pecola ~]$ guid=$(uuidgen)
[jk@pecola ~]$ echo $guid

generating key updates

In order to install this key into the firmware signature databases, we need to create an EFI_SIGNATURE_LIST container for the key, and provide an EFI_VARIABLE_AUTHENTICATION_2 descriptor. The update data will be self-signed, to keep things simple.

First, we create the EFI_SIGNATURE_LIST containing the certificate:

[jk@pecola ~]$ sbsiglist --owner $guid --type x509 --output test-cert.der.siglist test-cert.der

Next, we create a signed update for the EFI signature databases. The signed update consists of the certificate, prefixed with an EFI_VARIABLE_AUTHENTICATION_2 descriptor. The authentication descriptor signs the key data, plus the variable name and attributes. Becuase the variable name is included, we need to generate a separate signed update for each variable (PK, KEK and db):

[jk@pecola ~]$ for n in PK KEK db
> do
>   sbvarsign --key test-key.rsa --cert test-cert.pem \
>     --output test-cert.der.siglist.$n.signed \
>     $n test-cert.der.siglist
> done

creating a keystore

Next up, we'll put our keys into standard locations for sbkeysync to find:

[jk@pecola ~]$ sudo mkdir -p /etc/secureboot/keys/{PK,KEK,db,dbx}
[jk@pecola ~]$ sudo cp *.PK.signed /etc/secureboot/keys/PK/
[jk@pecola ~]$ sudo cp *.KEK.signed /etc/secureboot/keys/KEK/
[jk@pecola ~]$ sudo cp *.db.signed /etc/secureboot/keys/db/

If you'd rather use a different location for the keystore, just use the --keystore and/or --no-default-keystores arguments to the sbkeysync commands that follow.

using sbkeysync

We can now use sbkeysync to synchronise the firmware key databases with the keystore we just created. Do a dry-run first to make sure all is OK:

[jk@pecola ~]$ sbkeysync --verbose --pk --dry-run
Filesystem keystore:
  /etc/secureboot/keys/db/test-cert.der.siglist.db.signed [2116 bytes]
  /etc/secureboot/keys/KEK/test-cert.der.siglist.KEK.signed [2116 bytes]
  /etc/secureboot/keys/PK/test-cert.der.siglist.PK.signed [2116 bytes]
firmware keys:
  PK:
  KEK:
  db:
  dbx:
filesystem keys:
  PK:
    /CN=test-key
     from /etc/secureboot/keys/PK/test-cert.der.siglist.PK.signed
  KEK:
    /CN=test-key
     from /etc/secureboot/keys/KEK/test-cert.der.siglist.KEK.signed
  db:
    /CN=test-key
     from /etc/secureboot/keys/db/test-cert.der.siglist.db.signed
  dbx:
New keys in filesystem:
 /etc/secureboot/keys/db/test-cert.der.siglist.db.signed
 /etc/secureboot/keys/KEK/test-cert.der.siglist.KEK.signed
 /etc/secureboot/keys/PK/test-cert.der.siglist.PK.signed

The output will list the keys were found in the EFI key databases, keys found in the filesystem keystore, and which keys should be inserted to the EFI key databases to bring them in sync with the keystore.

If all looks good, we can remove the --dry-run argument to actually update the firmware key databases. However, be careful here - once a PK is enrolled, the machine is no longer in setup mode, and secure boot is enforced. At the very least, ensure that your firmware setup screens have a facility for returning your machine to setup mode, and/or removing the PK.

Note that some firmware implementations may require a reboot for the changes to take effect in the EFI variables. Before you do this though, you'll probably want to sign your bootloader, so that you can actually boot something!

signing a bootloader

Now that secure boot is enabled, it'd be nice if we actually had something to boot. Although it isn't recommended for production systems, we'll just sign the GRUB2 binary that's already there:

[jk@pecola ~]$ sbsign --key test-key.rsa --cert test-cert.pem \
        --output grubx64.efi /boot/efi/efi/ubuntu/grubx64.efi
[jk@pecola ~]$ sudo cp /boot/efi/efi/ubuntu/grubx64.efi{,.bak}
[jk@pecola ~]$ sudo cp grubx64.efi /boot/efi/efi/ubuntu/

reverting to setup mode

Theoretically, since we have the private-key component of PK, we can revert the machine from user-mode to setup mode. This requires writing an empty signed update to the PK variable:

[jk@pecola ~]$ : > empty
[jk@pecola ~]$ sbvarsign --key test-key.rsa --cert test-cert.pem \
        --attrs NON_VOLATILE,BOOTSERVICE_ACCESS,RUNTIME_ACCESS \
        --include-attrs --output empty.PK.signed PK empty
[jk@pecola ~]$ sudo dd bs=4k if=empty.PK.signed \
        of=/sys/firmware/efi/vars/PK-8be4df61-93ca-11d2-aa0d-00e098032b8c

However, I have not been able to reset the PK on all firmware implementations so far; there may be bugs in the signing tools (or firmware) that prevent the update from being properly verified. Because of this, I strongly suggest checking for the facility to clear the PK through your firmware setup screens before attempting to set the PK.

secure boot tools resources

If you'd like to check out the code, the following links may be useful:

Power ISA Version 2.07 - the latest on the Power Instruction Set Architecture Comment Entries Likes

The PowerLinux Community - Wed, 05/22/2013 - 10:08
On Power.org, the latest revision of the Power Instruction Set Architecture has been published. https://www.power.org/documentation/power-isa-version-2-07/ In this latest update, there's a page and a half summary of the architectural changes being... 0 1 197

POWER development hosting - OSU open source lab Comment Entries Likes

The PowerLinux Community - Wed, 05/22/2013 - 08:53
Over on Google+ , Jeff Scheel pointed out how easy it is to get access to a partition for PowerLinux testing in the Open Source Lab at Oregon State University (OSU).   Check it out! Jeff followed that post with another open question for... 0 0 156

Virtualization@IBM: KVM: The logical choice for Linux Comment Entries Likes

The PowerLinux Community - Tue, 05/21/2013 - 08:39
For an interesting cross-post from IBM DeveloperWorks, check out a ... 0 0 218

KVM: The logical choice for Linux Comment Entries Likes

Virtualization@IBM - Mon, 05/20/2013 - 09:10
The open source hypervisor KVM (Kernel-based Virtual Machine) is gaining ground in the enterprise. KVM adoption echoes the early days of Linux since organizations, by now familiar with server virtualization, are evaluating not only hypervisors from the current... 0 0 13

Novos repositórios do IBM Advance Toolchain for PowerLinux Entradas de Comentários Curtir

PowerLinux Community - Brazil - Fri, 05/17/2013 - 09:54
Por: Tulio Magno Quites Machado Filho Os repositórios do IBM Advance Toolchain for PowerLinux estão mudando para um novo servidor. Usuários do IBM POWER Linux Tools Repository serão automaticamente migrados para os novos... 0 0 0

IBM Advance Toolchain for PowerLinux 6.0-4 released Comment Entries Likes

The PowerLinux Community - Fri, 05/17/2013 - 09:41
By: Tulio Magno Quites Machado Filho The IBM Advance Toolchain for PowerLinux is a set of open source development tools and runtime libraries that allow users to take leading edge advantage of IBM's latest POWER hardware features on Linux. A new update... 0 0 240

IBM Advance Toolchain for PowerLinux repositories moving Comment Entries Likes

The PowerLinux Community - Thu, 05/16/2013 - 15:34
By: Tulio Magno Quites Machado Filho The IBM Advance Toolchain for PowerLinux repositories are moving to a new host. Users of the IBM POWER Linux Tools Repository will be automatically moved to the new repositories and don't need to change their... 0 0 344

Generalized Crypto Interfaces and OpenSSL Comment Entries Likes

Smarter Security for a Smarter Planet - Mon, 05/06/2013 - 13:49
One often repeated piece of advice with respect to adding crypto to your app is to never roll your own crypto. That is, never reimplement a crypto algorithm from scratch. This is very good advice. There are already several different libraries to choose from... 0 0 28

Confessions of a Recovering Proprietary Programmer, Part X

Paul McKenney: Multi-core Linux - Sun, 04/21/2013 - 17:45
I have been using xfig for a very long time, almost as long as I have been using gnuplot. But xfig has been getting a bit cranky lately, mostly in terms of font handling. I suspect that it is possible to make it handle fonts like it used to, but I decided to take this as a hint to try something that might actually be younger than the typical Linux kernel hacker. (Yes, I am getting a bit old to engage in ageism, but there you have it!)

I had tried inkscape some years back, but at the time it was not ready for prime time, at least not from the perspective of a long-time xfig user. But I had recently received a .svg, and had installed inkscape in order to be able to work with it. Besides, some of the more recent browsers can render directly from .svg, which may in the fullness of time remove the need to generate bitmap files for HTML documents.

So I gave inkscape a try.

The first pleasant surprise is that inkscape is able to import xfig's .fig file format. This import process is not perfect, for example, the fonts do not match exactly and arrowheads are sometimes imported as objects separate from the line that they are supposed to be attached to, but it is much nicer than recreating the diagram from scratch. In addition, in many cases, the import imperfections are not a problem, such as when the goal is simply to add something to the figure.

Of course, the menu layout is completely different than that of xfig, but this is not always a bad thing. For example, even given long familiarity with xfig, I found inkscape's object rotation to be much more powerful and easier to use than that of xfig. Object alignment and distribution is also much nicer in xfig. The manual canvas configuration in inkscape is a step back from xfig's automation, but it might well be that I just haven't yet found the corresponding inkscape setting. Finally, the ability to directly generate .pdf files works more smoothly with pdflatex, which I use heavily. The fact that they get rotated 90 degrees was a bit surprising at first, but the \rotatebox{270} directive in Latex takes care of that.

So who knows? After more years than I care to recall, it might finally be time to bid xfig a fond farewell.

Linux Certification LPIC-1 Consists of Two Separate Exams

As I mentioned in the introductory entry for our PowerLinux Blog, the Junior Level Linux Certification (LPIC-1) from the Linux Professional Institute (LPI) is composed of two separate exams, exam 101 and exam 102. Passing the level 1 certification exam requires the ability to work at the Linux command line, perform operating system maintenance tasks, and install and configure a workstation, including graphics configuration and network configuration.

Introducing Our New Power Linux Blog

We're starting a new POWER IT Pro blog targeted specifically at those who are responsible for the administration of Linux instances or those POWER-focused pros who want to gain deeper insight into how to work with the Linux operating system. Plus, the blog will prep you to take the Linux Professional Institute level 1 exams (LPIC-1). There are a total of 42 topics split across the 10 objectives outlined above, and our plan is to present a blog entry on each topic. The intent is that by the time we reach the end of the blog entries, a complete reference will have been created that covers all of the LPIC-1 topics . . . and provides both a comprehensive study guide for the LPIC-1 exams as well as a reasonably complete basic Linux reference guide that will be handy for POWER-focused IT professionals.

Introducing Our New PowerLinux Blog

We're starting a new POWER IT Pro blog targeted specifically at those who are responsible for the administration of Linux instances or those POWER-focused pros who want to gain deeper insight into how to work with the Linux operating system. Plus, the blog will prep you to take the Linux Professional Institute level 1 exams (LPIC-1). There are a total of 42 topics split across the 10 objectives outlined above, and our plan is to present a blog entry on each topic. The intent is that by the time we reach the end of the blog entries, a complete reference will have been created that covers all of the LPIC-1 topics . . . and provides both a comprehensive study guide for the LPIC-1 exams as well as a reasonably complete basic Linux reference guide that will be handy for POWER-focused IT professionals.
Syndicate content

Buy steroids - Winstrol, Anavar, Omnadren