Marat BN — Marat Borisovich Nepomnyashy's home page

Background:

Upgrading a software package on a typical popular Linux distro such as Fedora or Ubuntu can cause the package manager to also update one or more of the package’s configuration files in the ‘/etc/’ directory. The way software updates are usually conducted upgrades multiple packages all at once, causing lots of configuration files to get changed all at once. With all these changes mixed together, it gets impossible to effectively discern between them, to track their history, or to tell which change came from which software update or manual edit.

Configuration changes are inherently risky from a system administration and security perspective, as an incorrect setting can break functionality and/or increase vulnerability to an attack. Blindly accepting configuration changes is therefore especially risky as the nature and extent of the changes is not known. Besides coming from the package maintainers via software updates, silent configuration changes can also be introduced in an attack by a malicious program or user, or more often by a regular non-malicious program or user if nobody is notified of the change, or if the change is simply forgotten.

For all these reasons, the ability to quickly detect and review configuration changes in ‘/etc/’ as they occur, and to know, and track over time, which configuration files have changed, when, how, and why, comes in very handy for keeping a Linux or any Unix-based system stable, secure, and reliable for years and years.

Some default ways package managers protect custom configuration:

The mainstream package managers have some built in functionality to prevent overwriting and/or to do a backup, of customized configuration files.

• The Red Hat Package Manager (RPM) refuses to overwrite certain configuration files (depending on package configuration), and instead saves the package maintainer’s version in a separate file with the extension ‘.rpmnew’. Conversely, when it does overwrite a customized configuration file, it saves a backup of the system’s previous version in a separate file with the extension ‘.rpmsave’.

• The Debian Package Manager (DPKG) uses a very similar scheme with the extension ‘.dpkg-dist’ for the package maintainer’s version, and ‘.dpkg-old’ for the system’s previous version.

• The Gentoo package manager (Portage) does not overwrite any customized configuration file, and requires that the administrator manually runs a configuration merging utility such as ‘etc-update’, to approve each configuration change line by line.

These package managers’ default safety features lack the following:

• Custom configuration changes not coming from the package manager can not be detected at the time they occur, but may only be discovered later when the associated package is upgraded, and only when the package upgrade also involves a configuration update. Configuration changes from malware infestations would fall under this category.

• Package maintainer’s updates to non-customized configuration files can not be detected.

• No detailed logs are kept.

The limited functionality of most package managers is not sufficient to be able to detect and review all configuration changes as they occur and to track them over time.

Git as a solution for keeping track of ‘/etc/’ changes:

Git is a robust revision control system originally developed to manage source code revision history. Since its introduction; however, it has been utilized in all sorts of other ways for which it has been discovered to be useful. And one of those utilizations is to keep track of ‘/etc/’ changes, as fundamentally, an ‘/etc/’ configuration tree is identical to any source code tree from the perspective that it is yet another hierarchy of periodically revised text files.

Initializing and maintaining a git repo in the root of an ‘/etc/’ configuration tree provides the following conveniences:

• It is now possible to quickly and at any time detect any new change to any configuration file with the ‘git status’ command. This is especially useful to see what changed after a software installation or upgrade.

• It is now possible to log a timestamp and a detailed description of any configuration revision with the ‘git commit’ command.

• It is now possible to track the revision history of any one configuration file, or all files under any subdirectory, or all of the ‘/etc/’ hierarchy as a whole, with the ‘git log’ command to see the revision timestamps and descriptions, and the ‘git diff’ command to see the actual revision changes.

• It is now possible to quickly revert undesired configuration changes, and to recall previous versions of revised configuration files with the ‘git reset’ and ‘git checkout’ commands.

• Advanced usage: It is now possible to rapidly switch between different configuration profiles by keeping them in and switching across distinct git branches with the ‘git branch’ and ‘git checkout’ commands.

• Advanced usage: It is now possible to quickly compare configurations across distinct profile branches just like between different revisions.

• Advanced usage: It is now possible to migrate configuration changes from one distinct configuration profile branch to another with the ‘git merge’ and ‘git cherry-pick’ commands.

• Advanced usage: It is now possible to share, compare, and migrate configuration changes across multiple systems just like across profile branches by cloning and updating the systems’ ‘/etc/’ git repos with the ‘git clone’, ‘git push’, and ‘git pull’ commands.

Instructions for deploying git to keep track of ‘/etc/’ changes:

1. Make sure that git is installed on your system.

   1. Open a regular bash shell.

   2. Type ‘git –version’ into the bash shell.

      If the response is something like “git version 1.7.9.5”, then git is already properly installed.

      If the response is something like “No command ‘git’ found…”, then git is not yet installed.

   3. The procedure to install git differs from one Linux distribution to another. However, regardless of the flavor of Linux or Unix running, either sudo or root privileges are required.

      Installation instructions are available on the official git website at http://git-scm.com/downloads and http://git-scm.com/download/linux.

2. Start a bash session as the root user.

   This can be done either with the command “su -” with the root password, or the command “sudo bash” with the sudo privileges.

3. Switch / cd to the ‘/etc/’ directory in the shell.

4. Initialize an empty git repo in the root of the ‘/etc/’ directory.

   Issue the command “git init”. The response should be “Initialized empty Git repository in /etc/.git/”.

5. Prepare to add all the files and subdirectories in the ‘/etc/’ directory to the git repo as an initial commit.

   1. Issue the command “git add .” making sure to include the dot after the ‘add’. This dot represents the ‘/etc/’ directory and everything in it. The command should run silently, and it can take several seconds on systems using traditional hard drives. This time will be used by git to scan the whole ‘/etc/’ hierarchy.

   2. Issue the command “git status” to verify that all files have been added. The response should now be a long list of all the file paths under the ‘/etc/’ directory, with each path prefixed with the text “new file:”.

6. Exclude certain special files from tracking by git.

   Certain files kept in ‘/etc/’ are not part of system configuration, but are auto-generated in there for house-keeping purposes. There is no need to track these files, and they should be excluded from the git repo. These files are:

   • ld.so.cache

     An auto-generated binary file used by the dynamic linker system.

   • ld.so.conf

     An auto-generated text file also used by the dynamic linker system.

   • mtab

     An auto-generated text file which lists out the currently mounted file systems.

   These backup files (with names trailed by dash) should not be tracked either:

   • group-

   • gshadow-

   • passwd-

   • shadow-

   To exclude all these files from the initial commit to the git repo, do the following:

   1. Issue the command “git rm –cached ld.so.cache ld.so.conf mtab group- gshadow- passwd- shadow-“.

      The ‘–cached’ option passed to ‘git rm’ removes the files from tracking, but prevents git from deleting them from the filesystem. The response should be a list of files excluded each prefixed by the “rm” token.

      If the response is instead “fatal: pathspec ‘<FILE>’ did not match any files”, that means that the file <FILE> was not tracked in the first place. In such case just reissue the command without that file in the argument list.

   2. Issue the command “git status” to verify that the files are now excluded. The response should now include a heading “Untracked files:” with the excluded files listed under it.

7. Make the initial commit.

   1. Issue the command “git commit -m ‘Initial commit.'”. The response should be a long list of all the file paths under the ‘/etc/’ directory, with each path prefixed with the text “create mode”.

   2. Issue the command “git log” to verify that the log entry for the initial commit is there with the timestamp and the description. The author of the commit should be listed as the root user.

8. Prepare and commit the ‘.gitignore’ file.

   1. Issue the command “git status”, and verify that the files excluded in step (6) are listed under the heading “Untracked files”. No other files should be listed there.

   2. Using vim or some other text editor, create a new file called ‘.gitignore’ with the following contents:

      ld.so.cache
      ld.so.conf
      mtab
      group-
      gshadow-
      passwd-
      shadow-
      

   3. Issue the “git status” command again, and verify that now only the file “.gitignore” is listed as untracked.

   4. Issue the command “git add .gitignore ; git commit -m ‘Added .gitignore'” to track the history of the ‘.gitignore’ file as well.

   5. Issue the “git log” command to verify that the output now lists 2 log entries — the first one for the initial commit, and the second one for the ‘.gitignore’ commit.

   6. Issue the “git status” command again to verify that its output is now the following:

      # On branch master
      nothing to commit (working directory clean)
      

      At this point the git repo is now ready to keep track of all future ‘/etc/’ changes.

Using git to keep track of ‘/etc/’ changes:

• The ‘/etc/’-tracking git repo can now be used in the same way as any other git repo.

• Do not neglect the repo and check it regularly with the “git status” command. The output for a properly maintained repo should always be “nothing to commit (working directory clean)”. Use “git diff” to review any new changes and ‘cat’ to view newly added untracked files. Examine new changes carefully.

• Always make sure to commit any previously uncommitted changes before making any new configuration changes or performing software upgrades. Then make additional commits for the newly added changes.

• Give the commits good descriptions so that it will be easy to find and refer to them later.

• Use the command “git log” to track how all configuration has changed over time. Use the command “git log <PATH>” to track how the configuration under a specific path, be it a directory or a file, has changed over time.

Troubleshooting:

Git versions 1.7.12.4 and above can emit an error like this on the first commit attempt:

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: unable to auto-detect email address (got 'root@<hostname>.(none)')

If you see an error like that, then issue the following command:

git config user.email "root@<hostname>"

The original commit command can now be reissued, and that error message should not appear again.

Conclusion:

All configuration changes, intended or unintended, can now be detected and reviewed as they occur, and tracked over time.

See also:

• Git – Documentation