Running a home lab with public facing services, you run into the problem of Split DNS. Any name, like my home.stovallhut.online webpage, needs to be registered with a public IP address to reach it over the internet. Problem is, if you’re at home then you should be contacting a local address on your network (some routers let you use reflection/hairpinning to get around that but this has its own issues). My OPNsense router makes this pretty easy to manage with its Unbound DNS service and dns overrides.

That all works pretty good but the icing on the cake came when I figured out (with the help of a Great Guide) how to forward queries to my local DNS even when the client software specifically requested a different DNS server. Like if the client sends DNS queries to google’s public DNS at 8.8.8.8, then my router will now STILL handle the request if it can locally without contacting google. And if it does contact a public server, it won’t be google, and it will go out using DNS over TLS so my searches are private (at least to third parties like my ISP).

Amazing device

The post Split DNS done right using opnsense appeared first on Walter's Little World.

I’m not running a github server in my lab, and I’d rather not store this on the public github server even if my account is supposedly secure (anything can be compromised). The configuration is very sensitive, including security certificates, user accounts and passwords, firewall rules and more. I’d prefer to backup the configuration to my Synology NAS via SFTP. But the opnsense Command prompt offers no such selection. Github is it!

Stuck right? No! While it looks like I’m captive to what’s available in the gui, I figured out how to pull this off after a good bit of hunting and educating myself on some internals of opnsense and the FreeBSD OS it runs on top of. One avenue (that is in fact a dead-end) is to make a SSH connection with the opnsense router and setup a cron job using the command line crontab -e. This will appear successful at first, but if you modify ANY cron job settings in the opnsense gui, your cron job will be removed!

There’s actually a way to pretty easily extend the picklist of Commands you see in the opnsense GUI to include a custom job that does exactly what I want. This post describes the steps to do that.

Objectives

The steps I show below will first configure the Synology to accept SFTP connections for the purpose of sending it XML configuration backups from opnsense. We’ll setup a new user on the NAS called opnsense_backup. This user has limited privileges, allowing it only read/write access to a new folder where the backups will be stored. For security, the SFTP login that opnsense executes will authenticate thru SSH keys (password prompts would not work, there’s nobody to type it in). The backup is done by a script that you can install on opnsense and register as a new cron-job type that the opnsense gui recognizes. Finally, in the opnsense gui we’ll setup the cron job to execute the backup with the desired frequency.

Skills you need

To execute the steps in this post you need be just basically familiar with the DSM gui at the NAS, and also the opnsense gui for the router. You need to be able to make a SSH connection to the NAS with admin privileges, and also a SSH connection to the opnsense router with root access. You need just a basic understanding of using vi (no editing of files, just create new ones).

Create NAS backup destination and user account

In DSM at the NAS, go to Control Panel -> Shared Folder and create a new shared folder called opnsense_backup. This folder will collect the XML backup files from opnsense. As shown later, the files are all time-stamped with unique names that identify the date/time of the backup, and we’ll setup file-rotation so backups older than 30 days get deleted. Remember to add this new share to your HyperBackup/other backup software to further protect these files.

At Control Panel -> File Services -> FTP make sure the SFTP service is enabled. At Control Panel -> Users and Groups make sure user homes are enabled, then create a new opnsense_backup user account. Give the user a strong password. For privileges, give this account access to its own home folder, the opnsense_backup share created above, and the SFTP application itself. This will all “limit the damage” should this login ever be compromised.

At a SSH prompt on the NAS, execute the following commands to create a destination for SSH keys.

# Create location to store opnsense_backup user's keys
mkdir /var/services/homes/opnsense_backup/.ssh
touch /var/services/homes/opnsense_backup/.ssh/authorized_keys

Configure opnsense to make secure SFTP connections

At a SSH prompt on opnsense, generate a new SSH key using the keygen command.

# Generate a key
ssh-keygen
# accept default filename /root/.ssh/id_rsa at prompt, supply no passphrase

When keygen runs it will prompt you for a filename to store the key in. I accepted the default of /root/.ssh/id_rsa. Take note of the name and use it below if yours is different. You’ll also be prompted to enter a passphrase. Do NOT do that – just press enter. Passphrases are generally a good idea, but not in this case where the job is automated, with nobody to anwer a passphrase-prompt.

Now enter the following command to copy the generated key to the NAS. You’ll be prompted for a password – supply the password created on the NAS for the opnsense_backup user. Note my use of port 22 below – if your SFTP is configured with a different port then use that. Replace <your-NAS-hostname> with the hostname or IP address of your NAS.

# Copy key to NAS
scp -P 22 /root/.ssh/id_rsa.pub opnsense_backup@<your-NAS-hostname>:/home/

When that command completes, you should find the id_rsa.pub file in the opnsense_backup user’s home folder on the NAS.

Register SSH key at the NAS

We’ve stored the id_rsa.pub on the NAS above, now it needs to be properly registered as a SSH key for this user on the NAS.

In a SSH session on the NAS, login as admin and execute the following commands to append the new key to the authorized_keys file we created earlier. Also, it’s critically important to set the file permissions and ownership (chmod/chown below). SFTP will NOT accept the connection if you don’t do this!

# Save the key into authorized_keys file then remove and set permissions
cat /volume1/homes/opnsense_backup/id_rsa.pub >> /volume1/homes/opnsense_backup/.ssh/authorized_keys
rm /volume1/homes/opnsense_backup/id_rsa.pub
chmod 700 /volume1/homes/opnsense_backup/.ssh
chmod 600 /volume1/homes/opnsense_backup/.ssh/authorized_keys
chown -R opnsense_backup:users /volume1/homes/opnsense_backup/.ssh

Note that similar steps are shown in the Synology knowledge-base article for managing SSH keys where you just use File Station and no SSH work like I have above. The problem is the documented steps only work for SFTP logins by a admin user! Do what I show above for preparing .ssh and authorized_keys, and it will work for any user.

Test doing the backup

At a SSH prompt in opnsense, store the following script into a backup_opnsense.sh file. The location of the script is not important yet – just put it in your user home.

#!/bin/sh
DATE=$(date +%Y-%m-%d-%H%M)
BACKUP_FILE="/root/config-backup-$DATE.xml"
cp /conf/config.xml $BACKUP_FILE
scp $BACKUP_FILE opnsense_backup@stovallhut.online:/opnsense_backup/
rm $BACKUP_FILE

Remember to make the script executable:

chmod +x backup_opnsense.sh

Execute the script as a test.

./backup_opnsense.sh

On this first run you’ll probably see a warning about the unknown ssh signature – press enter to ignore the warning and accept the SSH connection. This won’t show when you run the script again in the future.

You should NOT be prompted for a password. If you are, then something is not right with the above steps when it comes to properly registering the key on the NAS. Review and debug before moving forward.

If all went well, you’ll find a new XML backup in the opnsense_backup share on your NAS!

Add new cron job type recognized by the opnsense gui

We have a script that can backup the configuration now, but it does not run automatically on a schedule. Let’s fix that. We’ll use the above script as the basis for a new type of cron command we can schedule in the opnsense gui.

At a SSH connection as root on opnsense, execute the steps below. This will relocate the script in the FreeBSD file system to the /usr/local/etc folder so it’s part of the opnsense environment. File permissions are set as required. Then we’ll create an actions_sftp_backup.conf file to register our script as a cron job.

# Move script...
mv backup_opnsense.sh /usr/local/etc/backup_opnsense.sh
# Set file permissions
chmod 700 /usr/local/etc/backup_opnsense.sh
# Add as opnsense action
vi /usr/local/opnsense/service/conf/actions.d/actions_sftp_backup.conf

Paste the following content into the actions_sftp_backup.conf at the vi prompt.

[sftp_backup]
command:/usr/local/etc/backup_opnsense.sh
parameters:
type:script
message:Starting backup script
description:Backup config to NAS

Finally, restart the configd service so the new cron job will be recognized in the gui.

# Restart configd service to expose new config
service configd restart

Create cron job to periodically backup opnsense configuration

At this point we’re nearly done. In the opnsense web console, go to System -> Settings -> Cron and press the plus (+) button to add a new job.

Fill in the time to run the job. In my example I’m executing the script at 11:28 every night.

On the above Command picklist, you should see a new Backup config to NAS you can pick. Choose that and it will execute the SFTP backup script we prepared above.

To test the cron job, see in the gui how you can clone your job – this is an easy way to create a job that will run in about one minute so you can make sure this works. Let that happen, confirm you get a new backup in the opnsense_backup network share on the NAS. Then just delete your clone when you’ve seen it work.

Trim the backups so only the last 30 days are saved

With backups happening every day, the opnsense_backup share on the NAS will consume more and more space over time. See steps below that will fix that by running a script on the NAS each day that deletes backups that are older than 30 days.

On the NAS go to Control Panel -> Task Scheduler. Create a new task based on a user-defined script. Select to execute the job as root. On the schedule tab, set the frequency you want to run the task such as daily. On the task tab, paste the following content:

#!/bin/sh
# Path to the backup directory
BACKUP_DIR="/volume1/opnsense_backup"

# Delete files older than 30 days
find $BACKUP_DIR -name "*.xml" -mtime +30 -exec rm {} \;

We’re done! Like anything, test this for proper behavior. You should see backups getting saved every day into the new opnsense_backup share on the NAS. Make a point to come back later and make sure old ones are getting deleted as expected.

The post Automatically backup your opnsense router to a Synology NAS via SFTP appeared first on Walter's Little World.

In the early ’80s after a few years of obsessing at home writing video games and the like, I managed to get a job at Scientific Atlanta (2nd job) through a lab-tech friend of mine that I had been interacting with, learning more about digital electronics. At the time, Scientific Atlanta (and most of the world) was early in the process of using microprocessors in the devices they manufactured. I was tasked with writing the firmware for a new antenna position tracking device. Like the name implies, this was a seemingly simple device with a digital display that showed the angle of rotation for a microwave antenna. Position indicators were used primarily during construction of new microwave antennas, where prototypes would receive signals while being positioned at virtually all possible azimuth and elevation angles, while recording equipment would store amplitude modulation data that was also correlated with the position in an antenna test range.

The work was led primarily by Bob Hyers who was the principal engineer. Beyond Bob’s detailed work on the high level design, the real work was done by two engineers Steve and Charles. We quickly became good friends, and they taught me how to work with the prototypes they were building in the lab, as we worked on getting things working a little more every day. Steve and Charles were responsible for defining exactly what components to put in the unit, and they laid out circuit boards and all interconnections. Then came me – the unit was not going to do anything at all in the way of indicating or transmitting antenna positions without somebody to write the necessary software. I learned from the ground-up how to debug code in a lab environment with no user interface, just an oscilloscope and some pretty amazing debugging tools (logic analyzer) that were more powerful than anything I used before or since on traditional computers. My software handled every button press, changing display modes, actually calculating the position of the antenna as a decimal angle of degrees, and displaying the position or transmitting it to other test-range equipment in a timely manner.

It was on the last point of having timely position that was at the heart of most of the complexity in the design. Scientific Atlanta already had a position indictor at the time and used it heavily. The positions it indicated were perfectly accurate too. The problem was that the position was not calculated frequently enough. Since the antenna in the test range is moving, by the time you get position data it’s already out of date due to group delay. So to make a good graph of position vs. amplitude modulation, the antenna had to be moved very very slowly as thousands of data points were recorded. This took a long time to do. Then the antenna might undergo some changes to make improvements, and the whole process had to be repeated. The problem was further complicated when the signals being transmitted were distant from the antenna receiving the data. The new position indicator would radically improve on these problems.

None of the engineers knew a thing about writing computer code. So in that sense I had a blank slate. But the algorithm was still well conceived before I came onboard. The goal was to generate accurate position data every 200 nanoseconds (ns). When I first heard this, it sounded like an impossible problem for a microprocessor of the day. The hardware design was based around an Intel CPU with a hardware clock ticking at 5 MHZ, which happens to be every 200 ns. But it was in fact impossible for the processor to calculate anything in just one tick of the clock! Even a single microcode instruction can’t execute in a single tick. But learning more, I discovered that the design was not as crazy as it sounded. It was also an opportunity for me to let my programming abilities expand a lot given the nontrivial behavior that would be called for to pull it off.

The hardware of the 1885/86 Antenna Position Indicator as this model was known, included a “rate counter”. This was a device that could indicate a changing angle, and steadily increment that value by perhaps a few thousands of a degree every 200 ns. – all with literally no action taken by the microprocessor. As intelligent as the rate counter was (including wrapping a 359.9999 position to 0.0000 at the next increment), the rate counter did not know what angle to display and did not know what increment to apply to it at the next clock tick. Telling the rate counter what to do was the job of the firmware I was to write. But in order to save manufacturing costs, my ability to control the rate counter was very limited. I could in fact not even tell the counter what angle to display! All my software could do was to change the increment that the rate counter would apply at each clock tick. Essentially, I could find out what position the counter was indicating “now” (more on that later). And I knew what increment it was currently applying at each tick (by remembering what I last told it to use). And finally, I knew the current position of the antenna. So I have these three inputs: the angle being displayed, the actual angle of the antenna, and the increment being used currently. It was up to my software to now generate one little piece of information – what should be the new increment the rate counter was using? That was my single means of controlling the position output. So it was up to me to see that the indicated position would reasonably quickly converge on an accurate indication of antenna angle.

While the hardware seemingly tied my hands behind my back, this was my favorite kind of thing to code. I want nothing more than a seemingly impossible problem that in fact has a solution. Don’t get me wrong though, the problem had been figured out already, including very detailed mathematics that my software should apply in the solution. But as always, there’s a million miles between a concept and a working device. A successful project takes competent engineers building the device and for a reasonable cost that turns a profit for the manufacturer.

The firmware I designed to do this was organized around a core software routine that was interrupt driven. Every 16 milliseconds (ms) the hardware would raise the interrupt-line on the CPU. Doing so would cause the CPU to execute an INT instruction. This meant the CPU would save its current instruction address (i.e. the address of the next instruction it plans to execute) on the system stack. Then it would immediately jump to the address of the interrupt handler (pointed to by a table in the first page of RAM). There, was a tight little piece of code I wrote in assembly language (but most of the code was written in C-language). The function of the interrupt handler was to schedule the most high priority task defined in my self-designed SAMOS (Scientific Atlanta Multitasking Operating System). There were various tasks for the software to regularly perform, including updating the display, responding to button presses by the operator, transmitting position output to the front panel or other devices on the serial I/O bus, and miscellaneous maintenance and diagnostics tasks.

As its last step the interrupt handler transferred control to the SAMOS scheduler to execute the current highest-priority task and that happened to pretty much always be updating the position. To calculate the angle, the code would read a couple values from the synchro that was attached to the base of the antenna. These values were not in constant motion, like the antenna itself. Instead they were latched at the time of the interrupt immediately beforehand. By applying some trigonometry to ratios from the synchro, it was possible to generate a digital angle in degrees. Like many things though, even that was a lot harder than it sounds given the real-time requirements of the device. The trigonometry calculations could naturally be done with floating point math. But to do that would mean using a floating point library, given the limits of the instruction set on the 8088 processor. That would have been way too slow. If, in the execution of this task, the software takes too long to calculate the angle and the new rate and along comes another interrupt signal then you’ve failed! That’s an overspeed-condition. It’s absolutely essential to prevent that. Downrange measurements will be wrong to a degree that’s supposed to be impossible.

Various techniques were used to multiply or divide without using floating point arthmetic. In some cases the code just looked up a number in say a 1KB table stored in ROM (truncating index bits as necessary, but also often using them to skew the looked-up table entry). Another simple example would be to use the processor’s shift-left instruction to shift a variable by a small number of bits (where each shift is a multiply-by-2), then multiply or divide integer variables (usually by a pre-calculated constant that’s also been shifted). That generates an integer result, then finally use the shift-right (divide by 2) to get the result back in the intended scale. It takes a lot of care with the scale of the numbers to avoid overflow conditions, while still achieving accuracy requirements.

As always, latching was key to synchronizing the calculations done in the firmware with the time of the moving antenna. Input latching means that the software is always calculating using numbers that are now obsolete given the moving antenna, but were known to be accurate at a specific moment in the recent past. Output latching means the software calculates updated rate values the equipment should now apply, but the change is only applied at a well defined moment in the near future. All this makes it possible to make precise measurements and control, in spite of the variable response times in the software.

As a point of real pride, I was named on the United States Patent 4,853,839 for Antenna Position Tracking Apparatus and Methods. The basis for calling this a new invention, is that for the first time this results in a position indicator with zero group delay for an antenna moving at a constant velocity. It does that essentially by predicting where the antenna will be in the near future, rather than always showing obsolete position data. I’m named as one of four inventors – Steven Nichols, Robert Hyers, Walter Stovall, and Charles Trawick. Good times I’ll always remember fondly.

(use the tool-bar for full-screen and zoom/turn pages, how it all works is on pages 20 & 21)

The post My early days grokking the wonder of microprocessors appeared first on Walter's Little World.

It takes a lot of engines to power the StarShip – 33 engines in the first stage and then six more in the second stage. Producing an engine per day is a big confidence booster when it comes to the risks associated with using the StarShip as planned. For perspective on what that rate means, compare this to just a few years ago when production of a comparable engine was targeted at four per year.

In 2015, NASA gave Aerojet Rocketdyne a contract worth $1.16 billion to “restart the production line” for the RS-25 engine. Again, that was money just to reestablish manufacturing facilities, not actually build the engines. NASA is paying more than $100 million for each of those. With this startup funding, the goal was for Aerojet Rocketdyne to produce four of these engines per year.

arstechnica.com

Moving ahead with StarShip testing and development

Production capabilities aside, the StarShip is far from proven. There’s yet to be an orbital launch. Although there have been successful static fire tests of the second stage, there’s yet to be a full static fire of all 33 engines in the first stage. When that will happen is guesswork, but any day now. A few weeks ago, it was going to happen in a few weeks.

I can’t wait. Seeing the static fire and orbital launch of StarShip, a rocket big enough to pack cargo to Mars, that will be sweet!

Late breaking news 11/3/2022…. SpaceX is pushing for an orbital test before the end of 2022 and NASA plans are in keeping with that!

The post SpaceX is churning out a Raptor per day now appeared first on Walter's Little World.

I’m going detail below some of how I’m managing this with a couple VMs I have deployed on a cluster of (two) servers. The details of how I do this on a Synology NAS are pretty specific to that hardware – the concepts are not.

Highlights of this framework include:

• Packaging services in a VM contains the scope of the damage when the “server” is compromised.
• Clustered hosts make it easy to move VMs to a new host or failover the VM if its host server is down.
• Snapshots of VMs can be created instantly as scheduled and then replicated to other hosts in the cluster.
• VMs can be exported to an external file system for off-site backup

How I put the pieces together

So much for the abstract, see below I’ll show you how I put this architecture together on my home network, clustering two servers that share two virtual machines.

The purpose of the virtual machines is not hugely relevant but as you’ll see in the screenshots here, the two virtual computers I have are hutbuddy_websites and Quicken_WindowsServer. The first is a virtual computer that runs a copy of Virtual DSM and hosts a few websites on my network. Websites can be notoriously vulnerable to attack. While I’m careful with security at those sites, it’s good to know that if the whole server went down it would still be only those websites and not my whole network. The second VM is something I use for running Quicken on a virtual Windows machine.

Now let’s start with VMs that exist, but they aren’t protected like I’ll outline. On a Synology server and many others, backing up virtual computers can get tricky and some of it gets downright philosophical with certain camps touting that you should just backup the VM from within the VM itself. Yeah that’s possible but recovering from a disaster requires rebuilding that VM from scratch starting by installing an operating system. It’s going to take hours with anything complex, and maybe days. I’m not settling for that because I don’t have to…

Clustering virtual computers

The redundancy starts by clustering hosts that each share the same virtual machines. Only one host at a time is designated to be the one that runs a given VM. But with a simple action in the Protection Plan it is possible to move the VM to another host, either for better loading or because a host is down. Note that on a Synology clustering requires a Virtual Machine Manager Pro license.

The key to redundancy is in the Protection Plan you choose for the VM. By clicking on Protection you get to this console.

In the protection plan you’ll schedule snapshots. A snapshot is a complete copy of the state of the virtual computer. Snapshots can be taken while the VM runs as filesystem-consistent snapshots at a point in time. Then you define a Retention Policy that says exactly when you want to release the space for old snapshots.

In the example policy above, the system retains snapshots for the last week and then keeps one snapshot per week for the last month.

Now that sounds like a lot of diskspace. My websites VM takes up about 250GB and I’m storing 15 or so copies of that?? Not really. Snapshots take advantage of the BTRFS file system and only store deltas. What it does mean is (unless you manually delete snapshots which you can do) if you delete a bunch of stuff it doesn’t go away immediately. That’s usually a good thing!

The outcome of clustering hosts like this is that if a host goes down, I can failover its VMs to the other host in just a few minutes. And if the VM crashes/other then I can restore from a snapshot made at various times that day, or less frequently for up to a month.

What’s missing?

OK so now we have two host servers that can each separately run the very same virtual machines. Not just sort of the same, but the same all the way down to the full content of the file system, the MAC address, everything. If a server goes down then I can almost instantly boot the VMs it hosted and they’re completely back in operation.

The only remaining problem is what if I lose both servers?? The two servers are in physical proximity. Theft, fire, or other might mean that both servers go down perhaps permanently. Obviously I won’t recover from that in just a few minutes, but the real problem is the fact that the servers were replicating snapshots to each other so now ALL the snapshots are gone!

One solution to this problem would be to periodically export the VM to a file. This is NOT a “snapshot” with only deltas, it’s a great big file that’s the whole state of the VM and everything in its internal file system.

The problems here are two-fold. For one thing, the export of a big VM might take several hours and the whole time it exports, you have to have your VM shutdown/offline. The other problem is that this is a manual action! I’m loath to have manual procedures that I can automate. But can I?

At first it seems like we’re stuck here – and that’s indeed where I stayed for months. But ultimately I got some help from a friend at reddit and found this German website that details a solution that includes using an internal utility we find in DSM (good thing google translates).

(I’m fine with using this even though not publically documented – be your own judge)

See my SSH session below (run it with root privilege i.e. sudo -i)

/volume1/@appstore/Virtualization/bin/vmm_backup_ova --help

Usage: /volume1/@appstore/Virtualization/bin/vmm_backup_ova [--dst] [--batch] [--host] [--guests] [--retent] [--retry]
        backup VM to shared folder on VMM

Options:
        --default       use default options to backup
        --dst           shared folder path for storing backup OVA
        --batch         the number of VMs exporting at a time (default: 5)
        --host|--guests mutually exclusive options
                        '--host' only backup VMs which repository is on the specified host (default: all)
                        '--guests' only backup specified VMs (default: not specified, use | for seperator if there are multiple targets)
        --retent        the number of backups for retention (default: 3)
        --retry         the number of times for backup retrying (default: 3)

Examples:
        Run backup script by default
                ./vmm_backup_ova --default
        Backup all guests which repository is on the host and store OVAs in certain shared folder
                ./vmm_backup_ova --dst=<share-name> --host="<host-name>"
        Backup all guests which repository is on the host and limit the number of VMs exporting at a time to avoid affecting performance
                ./vmm_backup_ova --batch=2 --host="<host-name>"
        Backup certain guests and store the last two OVAs per VM
                ./vmm_backup_ova --guests="<guest_name_1>|<guest_name_2>" --retent=2
root@HomeNAS2602:~#

The vmm_backup_ova utility is the cat’s meow here. I launch the program with a ssh script that reads as follows:

# clone/export VMs on this host for disaster recovery
#!/bin/bash
set -e
/volume1/@appstore/Virtualization/bin/vmm_backup_ova --dst=VMBackups --host="HomeNAS2602" --retent=1

In this case I’m telling vmm_backup_ova to export every VM running on that host and store the export in a shared folder called VMBackups and retain only one backup. A key advantage of this utility is that we do NOT have to shutdown the VM! Instead, vmm_backup_ova starts by making a temporary clone of the running VM, which happens in nearly an instant. Then it proceeds to export that clone (which is never run) while the real VM continues to run. The export of a large VM might take several hours, but it runs in the background while everything else continues to function and then the clone VM is automatically deleted.

Tip: Avoid spaces in your virtual computer names. My experience is the utility creates destination directories with the wrong names and then can’t populate them. See my use of underbars instead.

In practice I run a script like that on each of the two hosts. It’s nice that in the GUI of Virtual Machine Manager I can see and monitor the snapshot/export process even though I didn’t initiate it there. And although each NAS exports to its own file system, the VMBackups shared folder is replicated to the other host too via ShareSync, and the Hyper Backup program is used to make off-site copies of VMBackups. Finally, the VM backups share itself gets snapshot retaining content for up to a month (I snapshot nearly everything to protect it from ransomware if nothing else).

I’m currently exporting once per month as scheduled in the Task Scheduler. So if I lost BOTH hosts then I can still recover the VM from the latest export (with some hardware of course), then restore VM files from within the VM itself, as I’ll typically have made more recent file backups and not have to revert all the way back to the last export once I’m all done.

The post The Security and Redundancy of Clustered Virtual Machines appeared first on Walter's Little World.

What are these protections and why are they necessary? First let me cover a little background on how docker containers work. The code in the container executes as part of the docker engine. The docker engine by necessity, executes with root privilege and can therefore read or write any data in the file system whatsoever. To cause damage, malware in the container need only successfully submit a request to delete critical system files etc.

In addition to damaging the file system, containers can also carry out network attacks on other containers on your server. Containers normally run in the default bridge network. Being on the same subnet, the docker engine makes the containers visible to each other by name. So containers can discover other containers and get their IP address thru DNS. The requests they send to each other may be malicious and won’t be blocked by any firewall since they occur within the same docker subnet (which is not a real network – it’s a virtual LAN in the engine).

I recently went digging in this area when I got interested in installing the wiki.js container on my system to hold a wiki site. Wiki.js is a fully fledged web site/web publishing framework. Its JavaScript architecture and interfaces make it particularly susceptible to injection attacks. There’s also a history of quite a few bugs, and I’m not sure the codebase is clean of malware or poor security practices. That might be a reason to have second thoughts about using it all, but IMO that’s a little drastic if things are managed well.

But these concerns did spur me to learn about some controls that can be put in place, and how to use them. What I’m looking for here is to see that the attack surface within the docker engine, is limited to the wiki.js website itself – not my whole server. This means that an attacker might bring down wiki.js and might gain access to any information that’s been published there. But potentially numerous other services like my password manager, websites/sql databases, financial software, online movies, etc. remain unaffected.

Isolate Docker Containers

Docker provides a couple of ways to manage container security. You just have to make a point to use them when you have reason to be concerned about what a container might do (like uh…all the time I should have been doing this all along).

• Give the container its own network. Most people install containers on the default bridge network by simply not specifying otherwise. So usually examples you find on the web try to keep things simple and leave this out. Alternatively, you can isolate your container on its on network and this means it has its own subnet. Now, even if your container magically knew the IP address of another container, it would not be able to send it anything. The docker runtime would not route the request. This is why docker has this facility and why you should use it.
• Limit the logical file-system privilege of the container. As mentioned, the docker runtime runs with root privilege. That would seem to drive a nail into the coffin, for any goal of seeing your container have limited privileges as it executes file system code. But docker has a facility to address just this concern, that being the PUID/PGID arguments that tell docker to execute container requests as-though the request were executed by a specific user. So barring some kind of zero-day vulnerability in the runtime, this goes a long way to limiting the damage done by ill-formed or ill-intent code. Again, you don’t usually see these arguments getting used. They won’t protect you unless you use them.

How I went about container isolation by example

The details of applying the above docker facilities are system specific when you look at the details. But similar steps will apply regardless. In a broad sense, the problem is that of creating a dedicated bridge network for the container and then use that. Then also limit the file system privileges.

These are the specific steps I took to deploy the wiki.js container on my DS-1520+ NAS. There are lots of ways of doing the equivalent things, this is just a by-example for the steps I took based on what’s easy and familiar to me.

The first thing I’m going to do is create a network that I’ll call “wiki” where I’ll isolate the wiki.js container. I do that by running portainer, select my host and go to Networks and click on Add. Fill in the name of the network as “wiki”. Confirm the Driver is “bridge” and accept defaults on everything else and save this as a new network.

Now you can see your new network that’s setup on its own subnet.

With the network ready, I’ll now setup a user account for limiting the container’s privileges.

Start by creating a system user. On my system I just went to the Control Panel and setup a new user I call “docker_wikijs”. This user has file system privileges where the only directory it has any access to whatsoever, is the shared folder where the wiki.js maintains all its settings and data.

Getting the PUID/PGID takes executing the linux id command. If you’re comfortable with using SSH and you have SSH enabled on your server etc. then you can open a SSH prompt and get the output as shown by this example where I execute “id docker_wikijs”.

So what if you’re NOT so comfortable with SSH and you don’t have it setup? Well on a Synology don’t despair. You can actually execute the id command by setting up a task to do that in the Control Panel. The output will come to you as email. See this easy guide on doing that. (by the way you can use this same trick to execute any task such as docker run as root, just know that you need to take proper care doing so)

So take the uid and gid values that come from the id command and that’s all you need for making PUID and PGID arguments for the docker run command.

Having prepared the shared folders that wiki.js specifically wants, now I’m ready to execute docker run to install the container. See the following docker run command with highlighted arguments that isolate the container.

docker run -d --name=wikijs \
--network=wiki \
-e PUID=<uid value> \
-e PGID=<gid value> \
-p 3540:3000 \
-e TZ=America/New_York \
-v /volume1/docker/wikijs/config:/config \
-v /volume1/docker/wikijs/data:/data \
--restart always \
ghcr.io/linuxserver/wikijs

The network argument naturally puts the container on that bridge instead of the default. The PUID and PGID arguments look just like simple environment variables, but the docker runtime picks up on these and quietly applies those privileges.

Like anything though, test it out. For example reduce the user to read-only privilege and observe the wiki website failing to save files when you tell it to.

I execute the above docker run and then go to portainer and find wikijs installed as requested.

The post How to limit the possible damage done by docker container malware appeared first on Walter's Little World.

With my DS-1520+ server with its cloud and other functions I keep using more, I’ve managed to put together a house of cards. I’ve added one service after another including various docker containers and applications for about a year now. That included lots of trials and tribulations. I don’t want to lose it all in a hardware failure. I keep backups, but they haven’t been validated and I know for a fact there are various things that won’t get restored by Hyper Backup. So I need a procedure for doing a rebuild, and I need to be able to validate that it works when the time comes, and know what it takes to get a new system functional. And wouldn’t it be great if I could do that without buying new systems to try it out!?! I don’t want to break what I have.

It dawned on me recently that I have all the tools at my disposal to develop and test backup recovery, and without buying any new hardware. On either of my NASes I can run virtual computers using the Virtual Machine Manager. On that virtual computer, I can install a variety of operating systems including Virtual DSM. This lets met build a NAS within a NAS. Now I can start with a fresh installation of DSM and try to restore functionality from a backup. Upon finding and fixing problems with my procedure or limitations in the scope of the backup, I can make various changes and then just throw that Virtual DSM away and start over.

So I’ve been doing exactly that, and stumbled on an interesting thing to try out that I suspect substantially improves the security of my site and is otherwise just cool. In some of my testing I created a Virtual DSM and restored a backup of this web site on it (naturally including databases etc that it needs to function). I got that working and that’s why you can see this web page…the original site has been shut down at least for a while.

The reason this is more secure is this: If at attacker manages to infect the walterstovall.online host with malware, the damage that malware can cause is limited to the virtual computer. If it did get compromised somehow then I could just restore that virtual computer from a saved snapshot. If for example the malware were to somehow delete every file on the hard disk and render the operating system unbootable, this would still be limited to the virtual computer that runs the site. The “hard disk” that got wiped is just a virtual hard disk that’s a small part of the storage on the hosting server.

How I moved this site into a virtual computer

I went thru the process of creating a VM for my site several times before I ended up with a smooth and consolidated set of steps. I wanted to boil it down to the essentials so I can easily move things as I see fit. I think it worked out very nicely and I thought I’d record what I came up with here.

To sum it up here’s what I’ll do below. Setup a virtual NAS on my DS-918 using Virtual Machine Manager. Point that virtual NAS to a backup of my DS-1520 server where walterstovall.online is currently deployed. Restore from that backup, the services necessary for running the walterstovall.online blog along with the site files and SQL database etc. Shutdown the new site and startup the new one in the VM and make it accessible over the internet.

Let’s get into it!

Create Virtual DSM Computer to host my blog

On the DS-918 I’ll visit Virtual Machine Manager and tell it to create a new virtual computer.

Stepping thru a few simple dialogs to allocate hardware resources, the VM gets created and I can connect to it at its assigned IP address. Now I give the computer a unique host name and login account.

That’s about all there is to that…now I login with the new admin account and I’m at a new DSM desktop.

Install Web Backends

After downloading DSM updates I now go to the Package Center and tell it to download the backend services that my site needs. These are mostly provided by third parties and won’t be part of the backup we’re going to restore.

By inspecting the Web Station setup on the source NAS where walterstovall.online resides, I see the following capabilities there.

• Apache HTTP 2.4
• PHP 7.4
• phpMyAdmin

Restore system configuration and web sites

Now I go to the Package Center and tell it to install Hyper Backup which I then launch and tell it to restore a data backup. I’ll point to a backup of the whole NAS.

Now I step thru a few dialogs telling it just what to restore from the backup and get down to this summary to confirm.

That kicks off and runs for a couple hours, loading a few hundred GB of applications and data into this virtual server.

After running, that restores ALL network settings and user accounts from the DS-1520 server. This is great in most respects except that it also gives the server the same network and domain name as the stovallhut.online server I restored this backup from. So now it’s important to visit the network settings and login portal in the Virtual DSM and restore the correct settings. Also important, is to configure my local DNS so the IP address of the server is permanently reserved.

Finally, I can visit https://walterstovall.online and view the site! But not done quite yet…

Fix inability to login at site

Even though I can view the site anonymously, I can’t login. The reason being that my site is setup for a 2FA login and after moving the site, I can’t seem to make that work (even though it is a TOTP code, so it seems like it should work based on the same secret key).

In any case I came up with the following solution that will let me login and get things working.

Open a SSH session and navigate to the wordpress directory where I’ll temporarily hide the wordfence plugin which handles the 2FA login (and a lot of other security issues).

The mv wordfence wordfence.bak just renames the plugin directory and makes it unable to participate in the login which then becomes based only on username & password.

Now that I’m logged in I can then rename wordfence.bak, giving it the original name. This makes it possible to now get to wordfence and turn off 2FA logins. Now I can log out, log in, etc. And turning on 2FA makes that work again too.

Secure internet access

At this point the site is fully functional except for one glaring problem. Security certificates are not installed, so access in a web brower includes bypassing stern warnings about the site not being secure. It’s also not possible to reach the VM over the internet. I’ll solve both of those problems below.

To reach the VM over the internet I’ll setup a reverse proxy on stovallhut.online that will send walterstovall.online traffic to the right place on the LAN. The reverse proxy tells stovallhut.online that when it receives HTTPS traffic directed to walterstovall.online, that it should send that traffic to the IP address of the Virtual DSM where we’ve placed the site.

(this reverse proxy needs to be associated to the walterstovall.online certificate too at the certificates settings page or browsers will still complain regardless of the certificate at the final destination)

Now I’ll just pick the export-action on this certificate so I can import it at my new hutbuddy.online virtual server where the web site has been moved.

With the certificate imported, I’ll then associate it with the walterstovall.online virtual host.

And that’s about it! If that seems like a lot of stuff to do, you should have seen my steps the first time

The post My site has moved – hopefully to a more secure home appeared first on Walter's Little World.

Initially my login showed a new People folder to me with nothing in it. The server worked on indexing my thousands of images for at least an hour or so. Then I came back and now, when I view the people folder I see lots of sub-folders, one for each person the package thinks are the same individual. In each case I see a sample picture of the person. But instead of a person’s name, I see “who’s this?”.

Fill in the person’s name, and hit enter. Repeat with each one. In a few cases I would come to somebody that I’ve already identified, but the software apparently thinks these are two different people. In that case I just type the same name again (auto-completed for me) and I’ll be prompted to merge the two into one.

I’ve also seen where I can remove people from a folder if they’ve been wrongly recognized, but I’ve yet to witness that. What I’m on the lookout for, is a way for me to identify somebody that was not recognized at all, and say who they are. That may be missing feature? Or I’m too much of a newbie.

Even though some people don’t get recognized, it’s often very surprising who it does recognize. Like people in the background of the image. It also appears to me that it correlates separate images, combining clues from each. For example I have a case where it recognizes an image of my nephew. Then in another picture that it also recognizes of him (taken on the same day at the same gathering), there’s a clear shot of his body but he’s looking at the floor and instead of his face, the picture just shows the top his head. When I look at the pictures I know who it is because of what he’s wearing.

Now when I wish I could find some pictures of ??? I can do that. And with the NAS just sitting here anyway, this comes at no extra cost

The post The face detection in Synology Photos is too cool! appeared first on Walter's Little World.

Let’s say I want to watch a movie on my smart tv and the internet is down. My tv is setup to let me play movies from my home server residing at hostname video.stovallhut.online. If I pickup my roku and go to the DS Video app to watch a movie, it will fail right off the bat because it can’t resolve the hostname to a IP address. Dead in the water.

In my case the solution to this problem comes all in configuring my main router for my local network by adding a DNS Server that knows the names of my home servers and will then resolve the name without contacting a server on the internet. This also means the names will resolve much faster here at my house.

While this is a quick setup, it didn’t just fall in my lap. I was immediately hit with unfamiliar entities to setup and not knowing what zones are much less master zones vs. slave zones and why I might want one. But just resolving names on your local network is really pretty easy at least on my RT2600ac router. I captured the steps below in case this helps anybody else (or me later) dodge the complexity and just handle this simple scenario.

Start by logging into the Synology router and go to the Package Center. Install the DNS Server package. Open DNS Server and create a Master Zone. You need one master zone for each domain you want to resolve. So in my case I have a server I get to as stovallhut.online and fill out the dialog like this.

To me it was not intuitive to think of the name I want to resolve to as a “Master DNS server”. Don’t think that you need to setup something on your server for that. This is just the IP of the computer you want to reach using that name. That computer does not participate in resolving the name (I think the master-server concept is for the rare case where you would run your own nameserver).

Accept defaults for other choices and save the new Master Zone. To resolve the name though, you’re not quite done…

Assuming you have a Windows computer connected to your router (and using the DNS server i.e. no manual override of DHCP settings), open a command prompt and try to resolve the name you just setup.

See above that no address at all shows when trying to resolve the new domain. Compare that to resolving cnn.com which comes back with several IPs.

To resolve the name you need to add a “A” record to the master zone you just created. Visit the zone and double-click to edit.

Fill in just the IP address and save the rule – that’s all there is to it. Now go back to your dos prompt and you should see the name resolve correctly now.

Handle subdomains with more “A Records”

If you have any subdomains you’ll need A Records for each of them too. See below how I setup a few subdomains for my stovallhut.online domain.

Now my smart tv can go find movies at video.stovallhut.online when my internet is disconnected

Visit this site for more information on the Synology DNS Server. Note that I could have used CNAME records for my subdomains. That might be a little less maintenance especially if you change around your IP addresses. But I elected not to because of at least a minor performance issue in that CNAME records include the overhead of an additional query to the DNS Server during name resolution.

Going further with DNS Records

My above focus on basic address resolution leaves a lot out of what you can do with a DNS Server. See this good reference for what the various records in your DNS Server can do to help you manage a local network.

The post Keep your “online” life alive when the internet is down appeared first on Walter's Little World.

I’ve had a burr in my butt around the need for multifactor authentication on my server, especially for an admin account that if stolen, would be awful needless to say. It’s equally important for other accounts too depending on usage. But I come along kicking and screaming. I don’t login for good luck. I’m trying to do something and logging in is a distraction and a frustration if it takes much time or data entry. After setting up 2FA on my Synology DS-1520+ I’ve been real pleased with the outcome.

I’ll start with the outcome…let’s say I’m logging onto my NAS (directly or using one of my apps) and I’m prompted to authenticate. I enter my User Id and password and press enter. Now my web page tells me to approve the login at my phone.

I open the notice on my phone and hit Approve and I’m logged in

While I don’t welcome the interruption of another prompt, with two factor authentication being the goal I can’t see how it gets much easier than this.

The Secure Signin will also let you login using OTP codes.

It’s interesting to note that the security code can be generated and used even if your phone has no cell or internet connection. In fact you don’t even need a LAN connection to your NAS. The code is just good by itself since both the NAS and the phone have synchronized times and both of them know the secret key used to generate the code.

What if you don’t care about 2FA and just want a easier login?

Do you want the security of some authentication, but don’t want to mess around finding and typing in your long cryptic password all the time? You can setup your account for what’s called passwordless login. This will enable you, while holding your unlocked phone, to authenticate with literally one button press (avoid giving the phone an unlock code by setting up biometrics on the phone).

To see how easy this is let’s say I’m logging into the DS-1520 and I’m prompted to authenticate. I DO have to type in my user name.

Now when I click on the arrow I get this message in my browser window…

Now I look at my previously sleeping and locked phone…

So I tap on that message, my phone unlocks based on facial recognition, and I can approve or deny my waiting browser login…

At this point I’m done

OK so it works good but I bet setting it up is a nightmare right?

Nope – it was actually painless. As an administrator I didn’t have to do anything more than the basic server configuration I had in place where the server knows its domain name, is configured with security certificates, etc. as need for normal operation.

After that the setup of 2FA or passwordless login is all done by the user that wants that capability.

At the DSM desktop the user edits their account settings and has the option of setting up these login options.

With either passwordless or 2FA, now you point your phone at the QR code to download the app.

There’s one more code to scan in order to point the app to your server and then you’re done.

More to think about

I personally think if you have the option of getting an approve prompt on your phone and just pressing it is nearly as secure as true “2FA” where you type in a code you see on the phone, assuming your phone doesn’t stay unlocked of course. I’m no expert. It does mean that somebody that knows your user id and has your unlocked phone can login without your password. It also means somebody that knows your user id and password can login, which is what 2FA is there to prevent in the first place. Be your own judge.

• See PC Mag for a great review of authenticator apps. Using the Synology Secure Login app, I may not need any of these (not sure yet but the codes it generates should be usable anywhere given the standards).
• If you decide to set this up then don’t lock yourself out! Consider what happens when your phone is broken or dead. Various 2FA systems have ways around this. Synology will let you get codes via e-mail. And what if your e-mail is down or inaccessible too? In my case I created another admin account that I don’t use for anything but recovering another account (I can login as a different admin and go turn off 2FA for my real account). You never lock yourself out with passwordless login since you can always use your password without having the phone.
• If you use a verification code, understand using an authenticator app is distinctly better than getting sent a SMS text code. The SMS text is not secure, it takes time for the text message to reach you, and you need to have cell coverage for one thing.
• For Synology servers this is great article on making the server secure in general not just as related to authentication.

The post Do you want two factor authentication or passwordless login? appeared first on Walter's Little World.