Simple testing of Salt modules

by

After spending some time trying to learn testing and later how to test Salt modules I felt that I need to summarise what I have done and what I missed.

Testing Salt modules seems to require the following python packages:

    pytest
    pytest-salt-factories

When I tried to run tests without the salt-factories plugin I only received the following error:

FAILED _modules/test_interface_freebsd.py::test_parse_rc_conf - NameError: name '__salt__' is not defined

Since I started using Tox for testing I had forgotten to add pytest-salt-factories as a dependency for the test python environment.

def test_parse_rc_conf() -> None:
    rc_contents = 'ifconfig_vtnet0="inet 192.168.1.1/24"\nifconfig_vlan123="vlandev vtnet0 vlan 123 192.168.2.1/24 description \'UnitTesting\''
    with patch.dict(
        interfacemod.__salt__,
        {
            "file.replace": MagicMock(return_value=True),
            "file.read": mock_open(read_data=rc_contents),
            "file.grep": MagicMock(return_value={"stdout": rc_contents})
        },
    ):
        expected = {
            'vlan123': '"vlandev vtnet0 vlan 123 192.168.2.1/24 description \'UnitTesting\'', 
            'vtnet0': '"inet 192.168.1.1/24"'
        }
        assert interfacemod.parse_rc_conf() == expected

Newer FreeBSD version for package X bypass

by

Newer FreeBSD version for package bypass

Newer FreeBSD version for package pkg:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1301000
- running kernel: 1300139

Dont agree,

Do the following change and lock your FreeBSD version to latest release for that version.

root@freebsd:/usr/local/etc/pkg/repos # cat FreeBSD.conf
FreeBSD: {
  url: "pkg+http://pkg0.bme.freebsd.org/FreeBSD:13:amd64/release_1/"
}

Install Salt 3005.1 FreeBSD 13.0

Follow above steps first

pkg install python3.X
mkdir salt_env
python3.x -m venv env
source env/bin/activate.csh
pip install --upgrade pip

pip install salt==3005.1 WILL FAIL

setenv C_INCLUDE_PATH /usr/local/include/
setenv CPLUS_INCLUDE_PATH /usr/local/include/
pkg install cppzmq

might need
pkg install libunwind

This solved errors like:

      /usr/include/string.h:95:9: note: previous declaration is here
      size_t   strlcpy(char * __restrict, const char * __restrict, size_t);
               ^
      1 error generated.
      error: command 'c++' failed with exit status 1
      #include <libunwind.h>
               ^~~~~~~~~~~~~
      1 error generated.
      error: command 'c++' failed with exit status 1

FreeBSD mfsbsd pxe

by

https://github.com/mmatuska/mfsbsd

https://forums.freebsd.org/threads/freebsd-iso-bootable.52444/

https://docs.freebsd.org/en/books/handbook/disks/#disks-virtual

fetch http://ftp4.se.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/13.0/FreeBSD-13.0-RELEASE-amd64-disc1.iso

mdconfig -a -t vnode -f FreeBSD-13.0-RELEASE-amd64-disc1.iso
mount_cd9660 /dev/md0 /cdrom/

git clone https://github.com/mmatuska/mfsbsd
cd mfsbsd

#Customize wanted files in conf/ specially rc.local that kickstarts everything

make iso BASE=/cdrom/usr/freebsd-dist

rc.local

fetch http://<servername>/mfsbsd/installerconfig -o /etc/installerconfig
tail -n 7 /etc/rc.local > /tmp/start.sh
chmod +x /tmp/start.sh
/tmp/start.sh 
exit 0

#!/bin/csh
setenv DISTRIBUTIONS "kernel.txz base.txz"
setenv BSDINSTALL_DISTDIR /tmp
setenv BSDINSTALL_DISTSITE http://ftp4.se.freebsd.org/pub/FreeBSD/releases/amd64/13.0-RELEASE

bsdinstall distfetch 
bsdinstall script /etc/installerconfig

installerconfig, ensure that gpart edits correct harddrive

PARTITIONS=ada0
DISTRIBUTIONS="kernel.txz base.txz"
BSDINSTALL_DISTDIR=/tmp
BSDINSTALL_DISTSITE=http://ftp4.se.freebsd.org/pub/FreeBSD/releases/amd64/13.0-RELEASE

#!/bin/sh
gpart bootcode -b /boot/pmbr    -p /boot/gptboot -i 1 ada0
sysrc ifconfig_em0=DHCP
sysrc sshd_enable=YES
echo "Installation complete, running in host system"
echo "hostname=\"FreeBSD\"" >> /etc/rc.conf
echo "autoboot_delay=\"5\"" >> /boot/loader.conf
echo "sshd_enable=YES" >> /etc/rc.conf
echo "Setup done" >> /tmp/log.txt
echo "Setup done."
poweroff

ipxe file

#!ipxe
sanboot http://XXXXXX/mfsbsd-13.0-RELEASE-amd64.iso
boot

Mirror a OpenBSD repo

by

http://www.openbsd.org/ftp.html#prospective

rsync -rv --progress --delete-delay --delay-updates --fuzzy rsync://openbsd.ipacct.com/OpenBSD/6.2/packages/amd64/ /var/www/pub/OpenBSD/6.2/packages

OpenBSd httpd config:

http://www.openbsd.org/httpd.conf

Source: https://daulton.ca/2018/10/openbsd-create-private-mirror/

https://webhome.phy.duke.edu/~rgb/General/yum_article/yum_article/node15.html

Only to list the files on the remote hosts since the rsync and http endpoints may contains different files:

rsync --list-only rsync://openbsd.ipacct.com/OpenBSD/6.2/packages/amd64/

https://stackoverflow.com/questions/13414086/how-to-use-rsync-list-only-source-to-list-all-the-files-in-that-directory

Build Python 3.X on OpenBSD

by 
cd /usr/ports/lang/python/3.X/
make plist
make package

Maybe
env PKG_CREATE_NO_CHECKS=yes make package

Ensure that you dont have a current Python version that may conflict

#thisisugly

OpenBSD VMD example

by

vmctl – https://man.openbsd.org/vmctl
vmd – https://man.openbsd.org/vmd

Start by downloading the desired OpenBSD version you want to run. In my case I wanted to run OpenBSD 6.2 which is EOL.

I found a Swedish repo that has all OpenBSD versions available so I downloaded it from here:
https://ftp.lysator.liu.se/pub/OpenBSD/

I had some issues booting only from the install62.iso so I downloaded the bsd.rd and booted from it and used the iso just to install all sets.

vmctl create -s 12G openbsd62.immg
vmctl start -m 4G -L -i 1 -b bsd.rd -r install62.iso -d /home/abc/vms/openbsd62.img -c openbsd62

After the install finishes you can boot directly from you virtual disk.

vmctl start -m 4G -L -i 1 -d /home/abc/vms/openbsd62.img -c openbsd62

Salt 2019.2.7 on OpenBSD 6.6

by

This is probably not the recommended way but it works.

Do everything below in a OpenBSD 6.6 setup. If you run some other version you can do this using OpenBSD VMM

cd /tmp
ftp https://cdn.openbsd.org/pub/OpenBSD/$(uname -r)/{ports.tar.gz,SHA256.sig}
signify -Cp /etc/signify/openbsd-$(uname -r | cut -c 1,3)-base.pub -x SHA256.sig ports.tar.gz

cd /usr
tar xzf /tmp/ports.tar.gz
cd /usr/ports/sysutils/
cp -r salt salt2019.2.7
cd salt2019.2.7

Edit distinfo

SHA256 (salt-2019.2.7.tar.gz) = d648bcfb7f6c3f2e13b5dfd67dec1043fc25cc72ebebec94b00b1dc5b2e1b873
SIZE (salt-2019.2.7.tar.gz) = 14576515

Edit Makefile

# $OpenBSD: Makefile,v 1.156 2020/10/05 19:46:18 jca Exp $

# optional dependencies
# https://github.com/saltstack/salt/blob/develop/doc/conf.py
# libvirt-python
# py-GitPython
# py-croniter
# py-django
# py-libcloud (cloud-requirements.txt)
# py-mako
# py-mongo
# py-mysql
# py-pyinotify
# py-openssl
# py-redis
# rabbitmq

COMMENT =               remote execution and configuration management system

MODPY_EGG_VERSION =     2019.2.7
REVISION =              0
DISTNAME =              salt-${MODPY_EGG_VERSION}

CATEGORIES =            sysutils net devel

HOMEPAGE =              https://community.saltstack.com/

MAINTAINER =            Jasper Lievisse Adriaanse <[email protected]>

# Apache 2.0
PERMIT_PACKAGE =        Yes

MODPY_PI =              Yes
MODPY_VERSION =         ${MODPY_DEFAULT_VERSION_3}
MODPY_SETUPTOOLS =      No

MODULES =               lang/python

BUILD_DEPENDS =         textproc/py-yaml${MODPY_FLAVOR} \
                        www/py-jinja2${MODPY_FLAVOR} \
                        www/py-requests${MODPY_FLAVOR}

# requirements/base.txt
RUN_DEPENDS =           net/py-msgpack${MODPY_FLAVOR}>=0.5.6 \
                        textproc/py-MarkupSafe${MODPY_FLAVOR} \
                        textproc/py-yaml${MODPY_FLAVOR} \
                        www/py-jinja2${MODPY_FLAVOR} \
                        www/py-requests${MODPY_FLAVOR} \
                        www/py-tornado${MODPY_FLAVOR} \
                        sysutils/py-distro

# requirements/zeromq.txt
RUN_DEPENDS +=          net/py-zmq${MODPY_FLAVOR} \
                        security/py-cryptodome${MODPY_FLAVOR}

# salt/output/progress.py
RUN_DEPENDS +=          devel/py-progressbar${MODPY_FLAVOR}

# salt/modules/x509.py
RUN_DEPENDS +=          security/py-M2Crypto${MODPY_FLAVOR}

# salt/{beacons,modules}/ps.py
RUN_DEPENDS +=          sysutils/py-psutil${MODPY_FLAVOR}

# max openfiles, soft: 3072, hard: 4096; DBus system session running...
TEST_IS_INTERACTIVE =   Yes
PORTHOME =              ${WRKDIST}
TEST_DEPENDS =          databases/py-mysqlclient${MODPY_FLAVOR} \
                        devel/git \
                        devel/py-gitpython${MODPY_FLAVOR} \
                        devel/py-pip${MODPY_FLAVOR} \
                        devel/py-six${MODPY_FLAVOR} \
                        devel/py-virtualenv${MODPY_FLAVOR} \
                        devel/subversion \
                        net/py-libcloud${MODPY_FLAVOR} \
                        net/rabbitmq \
                        sysutils/salt-testing \
                        www/py-CherryPy${MODPY_FLAVOR}

pre-configure:
        ${SUBST_CMD} ${WRKSRC}/salt/returners/zabbix_return.py
        sed -i 's,share/man,man,g' ${WRKSRC}/setup.py
        sed -i 's,^#user: root,user: _salt,' ${WRKSRC}/conf/{master,proxy}

post-install:
        ${INSTALL_DATA_DIR} ${PREFIX}/share/examples
        cp -R ${WRKSRC}/conf ${PREFIX}/share/examples/salt

do-test:
        cd ${WRKSRC} && \
                ${MODPY_BIN} tests/runtests.py -v

.include <bsd.port.mk>

Now we bake the package

make plist
make package

If errors

rm /usr/ports/plist/amd64/salt*

in /usr/ports/sysutils/salt2019.2.7/ run:
make clean=dist
make clean

try again

Saltstack 2019.2.5 on OpenBSD 6.6

by

Download ports

This will not always work.
This is not the recommended way and its not a clean way.
This method is only to make it work.

Maybe remove option –single-version-externally-managed not recognized from /usr/ports/lang/python/python.mk

Search for “single-version-externally-managed” in /usr/ports/lang/python/python.port.mk


cp -r /usr/ports/sysutil/salt /usr/ports/sysutil/salt2019.2.5

cd /usr/ports/sysutil/salt2019.2.5
rm -rf patches
Edit distinfo with the new version and correct hash/size
Edit Makefile to the following

# $OpenBSD: Makefile,v 1.135 2019/07/12 20:49:51 sthen Exp $

# optional dependencies
# https://github.com/saltstack/salt/blob/develop/doc/conf.py#L54
# libvirt-python
# py-GitPython
# py-croniter
# py-django
# py-libcloud (cloud-requirements.txt)
# py-mako
# py-mongo
# py-mysql
# py-pyinotify
# py-openssl
# py-redis
# rabbitmq

COMMENT =               remote execution and configuration management system

MODPY_EGG_VERSION =     2019.2.5
DISTNAME =              salt-${MODPY_EGG_VERSION}
REVISION =              0

CATEGORIES =            sysutils net devel

HOMEPAGE =              http://saltstack.org/

MAINTAINER =            Jasper Lievisse Adriaanse <[email protected]>

# Apache 2.0
PERMIT_PACKAGE =        Yes

MODPY_PI =              Yes
MODPY_VERSION =         ${MODPY_DEFAULT_VERSION_3}
MODPY_SETUPTOOLS =      No

MODULES =               lang/python

BUILD_DEPENDS =         textproc/py-yaml${MODPY_FLAVOR} \
                        www/py-jinja2${MODPY_FLAVOR} \
                        www/py-requests${MODPY_FLAVOR}

# requirements/base.txt
RUN_DEPENDS =           net/py-msgpack${MODPY_FLAVOR} \
                        net/py-msgpack${MODPY_FLAVOR} \
                        textproc/py-MarkupSafe${MODPY_FLAVOR} \
                        textproc/py-yaml${MODPY_FLAVOR} \
                        www/py-jinja2${MODPY_FLAVOR} \
                        www/py-requests${MODPY_FLAVOR} \
                        www/py-tornado${MODPY_FLAVOR}

# requirements/zeromq.txt
RUN_DEPENDS +=          net/py-zmq${MODPY_FLAVOR} \
                        security/py-cryptodome${MODPY_FLAVOR}

# salt/output/progress.py
RUN_DEPENDS +=          devel/py-progressbar${MODPY_FLAVOR}

# salt/modules/x509.py
RUN_DEPENDS +=          security/py-M2Crypto${MODPY_FLAVOR}

# max openfiles, soft: 3072, hard: 4096; DBus system session running...
TEST_IS_INTERACTIVE =   Yes
PORTHOME =              ${WRKDIST}
TEST_DEPENDS =          databases/py-mysql${MODPY_FLAVOR} \
                        devel/git \
                        devel/py-gitpython${MODPY_FLAVOR} \
                        devel/py-pip${MODPY_FLAVOR} \
                        devel/py-six${MODPY_FLAVOR} \
                        devel/py-virtualenv${MODPY_FLAVOR} \
                        devel/subversion \
                        net/py-libcloud${MODPY_FLAVOR} \
                        net/rabbitmq \
                        sysutils/salt-testing \
                        www/py-CherryPy${MODPY_FLAVOR}

# https://github.com/saltstack/salt/pull/45164
post-extract:
        cp ${FILESDIR}/{pf,vmctl}.py ${WRKSRC}/salt/modules/

pre-configure:
        ${SUBST_CMD} ${WRKSRC}/salt/returners/zabbix_return.py
        sed -i 's,share/man,man,g' ${WRKSRC}/setup.py
        sed -i 's,^#user: root,user: _salt,' ${WRKSRC}/conf/{master,proxy}

post-install:
        ${INSTALL_DATA_DIR} ${PREFIX}/share/examples
        cp -R ${WRKSRC}/conf ${PREFIX}/share/examples/salt

do-test:
        cd ${WRKSRC} && \
                ${MODPY_BIN} tests/runtests.py -v

.include <bsd.port.mk>

Then run

make plist

make package
/usr/ports/packages/amd64/all/salt-2019.2.5p0.tgz

The package is unsigned and when installing it the option “-D unsigned” must be used. (https://man.openbsd.org/pkg_add)

Done

Salt Capirca for BSD

by
Firewall

Use Salt to generate firewall rules for Open/Free-BSD
Google Capirca
Salt capirca_acl

This is not tested yet.
Since the syntax of pf.conf is not exactly the same between Free/Open-BSD some functions/attributes may not work(?)

Pillar-data:

acl:
  - my-filter:
      terms:
        - my-term:
            source_port: [1234, 1235]
            action: reject
        - my-other-term:
            source_port:
              - [5678, 5680]
            protocol: tcp
            action: accept
sudo salt freebsd* capirca.get_policy_config packetfilter
freebsd-lab:
    # Packetfilter my-filter Policy
    # $Date: 2020/08/19 $
    # inet
    
    # term my-term
    block return quick inet from { any } port { 1235 1234 } to { any } flags S/SA
    
    # term my-other-term
    pass quick inet proto { tcp } from { any } port { 5678:5680 } to { any } flags S/SA keep state

How to apply to pf.conf

/etc/pf.conf:
  file.managed:
    - contents: {{ salt.capirca.get_policy_config('packetfilter') }}