Profile cover photo
Profile photo
Daemon Security
Security solutions to simplify the business process.
Security solutions to simplify the business process.


[01/10/2018] Running CentOS with Bhyve

With the addition of UEFI in FreeBSD (since version 11), users of bhyve can use the UEFI boot loader instead of the grub2-bhyve port for booting operating systems such as Microsoft Windows, Linux and OpenBSD. The following page provides information necessary for setting up bhyve with UEFI boot loader support:

Features have been added to to make it easier to setup the UEFI boot loader, but the following is required to install the UEFI firmware pkg:
pkg install -y uefi-edk2-bhyve
With graphical support, you can use a vnc client like tigervnc, which can be installed with the following command:
# pkg install -y tigervnc
In the case of most corporate or government environments, the Linux of choice is RHEL, or CentOS. Utilizing bhyve, you can test and install CentOS in a bhyve VM the same way you would deploy a Linux VM in production. The first step is to download the CentOS iso (for this tutorial I used the CentOS minimal ISO):

I normally use a ZFS Volume (zvol) when running bhyve VMs. Run the following commands to create a zvol (ensure you have enough disk space to perform these operations):
# zfs create -V20G -o volmode=dev zroot/centos0
(zroot in this case is the zpool I am using)
Similar to my previous post about, you need certain items to be configured on FreeBSD in order to use bhyve. The following commands are necessary to get things running:
# echo "vfs.zfs.vol.mode=2" >> /boot/loader.conf
# kldload vmm
# ifconfig tap0 create
# sysctl 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm em0 addm tap0
# ifconfig bridge0 up
(replace em0 with whatever your physical interface is).
There are a number of utilities that can be used to manage bhyve VMs, and I am sure there is a way to use to run Linux VMs, but since all of the HowTos for running Linux use the bhyve command line, the following script is what I use for running CentOS with bhyve.
# General bhyve install/run script for CentOS
# Based on scripts from pr1ntf and lattera


if [ "$1" == "install" ];

#Kill it before starting it
bhyvectl --destroy --vm=$VMNAME

bhyve -c $CPU -m $RAM -H -P -A \
-s 0,hostbridge \
-s 2,virtio-net,$TAP \
-s 3,ahci-cd,$ISO \
-s 4,virtio-blk,/dev/zvol/zroot/$ZVOL \
-s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
-s 30,xhci,tablet \
-s 31,lpc -l com1,/dev/$SERIAL \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \

#kill it after
bhyvectl --destroy --vm=$VMNAME

elif [ "$1" == "run" ];

#Kill it before starting it
bhyvectl --destroy --vm=centos

bhyve -c $CPU -m $RAM -w -H \
-s 0,hostbridge \
-s 2,virtio-net,$TAP \
-s 4,virtio-blk,/dev/zvol/zroot/$ZVOL \
-s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
-s 30,xhci,tablet \
-s 31,lpc -l com1,/dev/$SERIAL \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \

echo "Please type install or run";
The variables at the top of the script can be adjusted to fit your own needs. With the addition of the graphics output protocol in UEFI (or UEFI-GOP), a VNC console is launched and hosted with the HOST and PORT setting. There is a password option available for the VNC service, but the connection should be treated as insecure. It is advised to only listen on localhost with the VNC console and tunnel into the host of the bhyve VM. Now with the ISO copied to /tmp/centos.iso, and the script saved as you can run the following command to start the install:
# ./ install
At this point, using vncviewer (on the local machine, or over an SSH tunnel), you should be able to bring up the console and run the CentOS installer as normal. The absolutely most critical item is to resolve an issue with the booting of UEFI after the installation has completed. Because of the path used in bhyve, you need to run the following to be able to boot CentOS after the installation:
# cp -f /mnt/sysimage/boot/efi/EFI/centos/grubx64.efi /mnt/sysimage/boot/efi/EFI/BOOT
With this setting changed, the same script can be used to launch your CentOS VM as needed:
# ./ run
If you are interested in a better solution for managing your Linux VM, take a look at the various bhyve management ports in the FreeBSD ports tree.
Add a comment...

Post has attachment
[08/04/2017] From BSDCan to vBSDCon

As always, BSDCan 2017 was another great conference. I recommend BSDCan to anyone I speak to as one of the best BSD conferences to attend in this part of the world. I gave my talk on the State of Network Security Monitoring (NSM) tools on the BSD operating systems. The talk was well received, but I will be updating my slides when I give this talk again at vBSDCon with updated information and allow time for more questions. In addition to being a speaker at the conference, Daemon Security once again is a "Silver Sponsor" of vBSDCon and I am looking forward to the conference venue that will be in the Reston Town Center, at the Hyatt Regency Hotel. Registration is now available from the conference website:

I look forward to seeing everyone there.

Author: Michael Shirk
Add a comment...

Post has attachment
[01/18/2017] Running Bro in a FreeBSD Jail

A few weeks ago, a user on the Bro IDS mailing list was looking for a way to run Bro in a FreeBSD jail. FreeBSD jails provide the foundation of operating system-level virtualization, later utilized and enhanced by Solaris zones, and those containers that everyone thinks are something new. To avoid going on a complete rant, I recommend the following write-up as an overview of FreeBSD jails:

The purpose of this howto is to document that basic steps necessary to use Bro within a FreeBSD jail. For jail management, ezjail is normally the recommended way to setup jails. With a recent copy of FreeBSD (FreeBSD 11), run the following commands to install the ezjail package and setup your Bro jail (Note: vtnet0 is my network interface on the host, which could also be em0, or re0 in your case):

# pkg install -y ezjail
# ezjail-admin install -p
# ezjail-admin create bro 'vtnet0|'
# cat << EOF > /usr/jail/bro/etc/rc.conf
cron_flags="$cron_flags -J 15"
# echo 'nameserver' > /usr/jails/bro/etc/resolv.conf
# sysrc ezjail_enable=yes
# ezjail-admin start bro

The settings for rc.conf are recommended for jails, but they can be adjusted as needed, since Bro can be configured to send out email alerts. Now with the jail running, the Bro package can be installed within the jail using pkg with the following command:

# pkg -j bro install -y bro

Before Bro can be used, the jail configuration needs to be updated to allow the bro jail access to bpf, the berkley packet filter used for reading packets from a network device. By design, the jail does not allow access to listen on a network interface. Modifying the devfs rules loaded by the jail will allow for reading of packets from the interface used to create the jail (which matches the host interface). Run the following commands to create a special devfs rule that will allow the jail access to bpf device:

# cat << EOF >> /etc/devfs.conf
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_bpf
# sed -i '' -e 's/devfsrules_jail/7/' /usr/local/etc/ezjail/bro

The jail is now setup to work with Bro, and the necessary Bro configuration updates can be made. The following commands are the bare minimum to get Bro running, please see the Bro documentation for additional configuration settings (Note: replace vtnet0 with the same interface used to create the jail):

# sed -i '' -e 's/^*/ bro localhost/' /usr/jails/bro/etc/hosts
# sed -i '' -e 's/^interface=.*/interface=vtnet0/' /usr/jails/bro/usr/local/etc/node.cfg
# sed -i '' -e 's/^host=.*/host=' /usr/jails/bro/usr/local/etc/node.cfg

Now after restarting the jail, you can start Bro using the 'broctl' script:

# ezjail-admin restart bro
# jexec bro /usr/local/bin/broctl deploy

Bro will now be running within the jail, and from the host you can analyze the logs in /usr/jails/bro/usr/local/spool/bro.

There are additional configurations that can be made to this Bro jail, including the usage of a non-root user for Bro. With this configuration, if there was a vulnerability within Bro that could be exploited, the attacker could only break out into the jail file tree structure. Another caveat is that the jail is setup to start when the FreeBSD host starts, but in order to make Bro start automatically, you can add the following to rc.local within the jail

echo '/usr/local/bin/broctl deploy' >> /usr/jails/bro/etc/rc.local

Author: Michael Shirk
Add a comment...

Post has attachment
[11/28/2016] Recap of BroCon and SuriCon 2016

In September, I gave a talk about running Bro NSM on BSD operating systems. The talk was well received and stirred interest in the BSD operating systems and their use for network security monitoring. The slides (and at the some point the video) for my talk are posted here:

One of the interesting things I learned at this conference, was the important role that FreeBSD plays in regards to Bro. FreeBSD and Linux are treated as the tier 1 operating systems for which Bro must work on before a software update is released. After my talk was given, the updated netmap code was merged into the FreeBSD 12-CURRENT tree to add better support for packet I/O on FreeBSD which Bro can be configured to use. For those interested in running Bro, Bro 2.5 is now available for download from here:

The port/pkg updates should be available soon for FreeBSD. I will be working to get 2.5 into OpenBSD 6.1, as the Bro port was updated to 2.4.1 in September.
Not as much BSD related, but SuriCon 2016 was held in Washington, DC and was a great conference discussing the open source IDS/IPS engine Suricata. Users of Suricata on FreeBSD can compile in support for netmap, to provide fast packet I/O for use with IDS. There are configurations with netmap-fwd that can be used with ipfw to provide fast IPS capabilities that I am looking to further test. I gave a lighting talk on pulledpork, the signature update script that works with Snort and Suricata. A lot of people have forked pulledpork to suit their own needs and there seems to be some common themes that could be incorporated into pulledpork to provide value for everyone. I fully recommend these conferences to anyone interested in network security monitoring and open source security tools as I really enjoyed the content.

Author: Michael Shirk
Add a comment...

Post has attachment
[07/13/2016] - The default way to use bhyve on FreeBSD.
bhyve is a type-2 hypervisor that is installed by default in FreeBSD 10+. One of its greatest features is how simple the interface is to create and run virtual machines on FreeBSD. Since bhyve first appeared in FreeBSD 10, the operating systems support has expanded beyond FreeBSD and OpenBSD to include most Linux distributions and Microsoft Windows. In FreeBSD 11, bhyve will feature graphical support (UEFI-GOP) allowing for graphical UEFI installations. There are several tools that have been created to make the managing of bhyve VMs as easy as the managing of FreeBSD jails.
iohyve - bhyve management with ZFS support
vmrc - VM rc script for managing bhyve VMs
In addition to these management tools, the FreeBSD Handbook provides details for a script that is provided with the base OS which makes it easy to use bhyve VMs. The script is called, and is provided at the following location:
Before using this script, there are some necessary steps to setup networking and storage for use with bhyve VMs. These steps are fully documented in the FreeBSD Handbook, but here are the necessary commands to load the vmm kernel module, and setup networking to allow for the tap interface to be used by bhyve VMs:
# kldload vmm
# ifconfig tap0 create
# sysctl 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm re0 addm tap0
# ifconfig bridge0 up
In this example, re0 is the interface of the host, which is added to bridge0 with a tap0 interface added for the bhyve VM. If you would like this to be a persistent configuration, take a look at the FreeBSD Handbook for the specific configurations you will need. Once you have the tap0 interface available, you will need to create a virtual disk to be used by the VM. The FreeBSD Handbook details the way to create a disk image file (.img) for the virtual disk. For this howto, a ZFS Volume (zvol) will be used. Run the following commands to create a zvol (ensure you have enough disk space to perform these operations):
# zfs create -V20G -o volmode=dev zroot/freebsdvm0
(zroot in this case is the zpool I am using)
If you are using UFS as your filesystem, and would like to test out ZFS, you can format a USB key with ZFS and use it to test out using bhyve VMs. Once you have the zvol created, you will need an install image to use the script to install a bhyve VM. For this tutorial, FreeBSD-11-BETA1 will be used as the OS for the VM. Run the following command to download the FreeBSD-11-BETA1 install iso:
# fetch…/…/FreeBSD-11.0-BETA1-amd64-disc1.iso
With the installation iso, we can now run with some parameters to startup the bhyve VM and to install an operating system.
# sh /usr/share/examples/bhyve/ -c 1 -m 2048M -t tap0 -d /dev/zvol/zroot/freebsdvm0 -i -I FreeBSD-11.0-BETA1-amd64-disc1.iso freebsdvm
This command will startup the VM with the console output showing in the same terminal. If using a terminal multiplexer like tmux, you can open a new tab and run this command so that you still have shell access. The -c option is used to set the number of CPUs the VM will have assigned to it, -m sets the amount of memory to be assigned to the VM, and the -t option sets the virtio-net tap interface to use with the VM. The -i option forces to boot from an installation CDROM, where -I sets the location of the iso file.
Once the VM is started, everything from this point forward is the same as a standard FreeBSD installation. The only caveat is that you will want to shutdown the VM so you can remove the iso file from the command line to startup the VM. Once the OS is installed, you can start your bhyve VM with the following command:
# sh /usr/share/examples/bhyve/ -c 1 -m 2048M -t tap0 -d /dev/zvol/zroot/freebsdvm0 freebsdvm
This setup provides a simple method to manage multiple VMs from a terminal using the script, and tmux. For additional information on features that are currently supported or planned for bhyve, or additional configuration options, refer to the following FreeBSD links:…/ha…/virtualization-host-bhyve.html
Add a comment...

Post has attachment
[09/08/2015] Daemon Security, a Silver Sponsor of vBSDCon 2015

Daemon Security is a "Silver Sponsor" of vBSDCon 2015, the biennial BSD conference hosted by Verisign, Inc. The conference will bring together members of the BSD community in a series of round-table discussions including presentations on various BSD topics including system administration, networking and security. Daemon Security is proud to be sponsoring this event for a second time to help solidify the BSD operating systems as the only choice for deploying security tools and solutions. The conference is only days away, so be sure to register as soon as possible. Hope to see everyone at the Hacker Lounge to discuss Network Security with BSD, HardenedBSD and the MetaBoF.

vBSDCon 2015 at the Sheraton in Reston, VA.
Add a comment...

Post has attachment
[07/27/2015] Hunter NSM - A modular platform for deploying network sensors.

Hunter NSM is a simple install script for Snort or Bro IDS with JSON logging configured for FreeBSD. This is a simplified version of the snorby install script, as the goal is to provide a modular platform to plug into any existing security architecture. The current version has been tested on FreeBSD 10.1 and HardenedBSD.
The script is available on github:
Add a comment...

Post has attachment
[05/29/2015] zfscron - A great idea from the BSDNow podcast to backup your home directory.

First off, if you are interested in all of the latest news and information on the BSD operating systems, you should checkout the BSD Now podcast. In the segment where Allan Jude and Kris Moore discuss viewer's questions, Allan was talking about creating zfs snapshots of your home directory every 30 minutes or so. This seemed like a great idea to capture changes that may have occurred since the last daily backup in your user home directory. has been added to the zfsbackup scripts and only needs to be setup as a cronjob for a user account that has privilege to perform zfs snapshots.
$ crontab -e 
(Then add the following to setup he cronjob for the user):
∗/30 ∗ ∗ ∗ ∗ /usr/home/test/
Now as you work throughout the day, snapshots will be rolled every 30 minutes, allowing you to go back if you have accidentally deleted files or directories from your user account within the past hour. on github:
Add a comment...

Post has attachment
[05/05/2015] Mumblehard - Malware that affects Linux and BSD Systems.

Several websites linked to this writeup by Marc-Etienne M.Leveille of ESET in regards to the Mumblehard malware he discovered when working with a customer's issue. Though Linux malware (just like OSX malware) is nothing new, this software included a very interesting packer that actually detects BSD systems. The attack vector was by way of Joomla and Wordpress exploits, and an illegal copy of DirectMailer, which installs the backdoor once the software is loaded (M.Leveille, 2015). 

The malware is packed with perl code inside of an ELF binary (the Linux file format and a compatible binary file format on BSD systems). Using specific system calls, the malware can determine whether the binary is executing on Linux or BSD. The following is the specific disassembled code from the M.Leveille report:

mov eax, SYS_time; //BSD_fchdir
push ebx; //Set to NULL or 0
push eax
int 80h;//syscall 13
//saves EAX and compares
cmp eax, 0 
//jumps to a specific location for BSD systems if the value is less than 0 (negative) 
//Or jumps to specific location for Linux systems when EAX is set to current number of seconds since the UNIX EPOCH

(M.Leveille, 2015, p. 6)

There is no specific data on the number of BSD systems there were compromised, except for the compromised systems showing up in the ESET sinkholes. The key thing from this report is that even BSD systems may be unpatched, or misconfigured and as vulnerable as Linux systems when care is not taken to keep systems up-to-date, and to promptly patch web applications when vulnerabilities are discovered. To check your BSD systems, look for binaries running from /var/tmp or /tmp. The malware also sets $0 to httpd to hide itself, and it will place a cronjob to run every 15 minutes:

∗/15 ∗ ∗ ∗ ∗ /var/tmp/qCVwOWA >/dev/null 2>&1

(M.Leveille, 2015, p. 6)

Make sure you are monitoring your BSD systems and keeping your applications up-to-date.

M.Leveille, M.E. (2015). Unboxing Linux/Mumblehard: Muttering spam from your servers. Retrieved from
Add a comment...

Post has attachment
[04/29/2015] jail.conf hack when upgrading from FreeBSD 9.x to 10.

If you are still using FreeBSD 9.x, you will want to migrate your jails to the new jail.conf format when you upgrade to FreeBSD 10. The new jail.conf format has been around since FreeBSD 9.1:

jail.conf manpage

In an effort to assist with migrating to the new jail.conf format, a template file is created based on the configuration of the jails within your rc.conf file. In the following example, a jail called "testjail" is configured in rc.conf then started on a FreeBSD 10.1 system:
If you run the jail, you will receive the following output:
# service jail start testjail
Starting jails:/etc/rc.d/jail: WARNING: /var/run/jail.testjail.conf is created and used for jail testjail.
/etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables is obsolete. Please consider to migrate to /etc/jail.conf
When you use the old rc.conf variables, the jail service script will create the new format for you, in this case /var/run/jail.testjail.conf. This file can be copied to /etc/jail.conf and used to start your jail with the new format. The following is the contents of the converted jail.testjail.conf:
# Generated by rc.d/jail at 2015-04-28 13:23:43
testjail {
host.hostname = "testjail";
path = "/usr/jails/testjail";
ip4.addr += "";
allow.raw_sockets = 0;
exec.system_user = "root";
exec.jail_user = "root";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_testjail_console.log";
mount.fstab = "/etc/fstab.testjail";
allow.set_hostname = 0;
allow.sysvipc = 0;
The generated jail.conf files can be consolidated into a single /etc/jail.conf file as documented by Dan Rue (2014):
cat /var/run/jail*.conf >> /etc/jail.conf
If you do not want to run the jail, you can use the "config" option with the service script and it will create the jail.conf file based on the content of your rc.conf file:
# service jail config 
testjail/etc/rc.d/jail: WARNING: /var/run/jail.testjail.conf is created and used for jail testjail.
testjail: parameters are in /var/run/jail.testjail.conf.
If you are supporting a number of customers (and jails), you can simply copy all of the generated configs into a single /etc/jail.conf file. Tools like ezjail handle the updating of the jail.conf for you when creating or modifying FreeBSD jails. With the "config" option, you can avoid having to run the jail in order to generate the proper jail.conf file for your jails.

Rue, D. (2014) Convert FreeBSD 10 jails from rc.conf to jail.conf. Retrieved from
Add a comment...
Wait while more posts are being loaded