In my recent network configuration on Proxmox, I encountered an issue that had me scratching my head. I was settuping vmbr5 on bond0, expecting to access the VLANs that were already declared in the network configuration. However, despite everything looking correct on the surface, I couldn't access the VLAN.

Below is an example of the initial setup I used:

I configured vmbr5 as a bridge on bond0, and tried to set up each vmbr to handle a specific VLAN, expecting them to work as intended. But despite this setup, I had no access to the VLANs.

The Solution

After some trial and error, I discovered the solution. The trick is to create vmbr5 on bond0, but then adapt the configuration of each vmbr to include as the bridge port: vmbr5.<VLAN number>.

Here’s how the configuration looked after applying the fix:

Now, each vmbr is correctly pointing to its corresponding VLAN via the vmbr5.<VLAN number> interface. With this setup, everything worked perfectly.

Final Thoughts

It’s one of those things that makes you say : "How dumb, yet so logical." It’s a solution that works, but it's not exactly intuitive. Sometimes, the most straightforward solutions can be the most elusive.

By organizing the network bridges in this way, Proxmox can correctly manage and route the VLAN traffic, and now I can access the VLANs as intended. While the solution is simple in hindsight, the lack of intuition in the initial approach makes this a learning point for anyone setting up VLANs in Proxmox.

After expanding my rack from 2U to 12U, I needed a simple solution that would allow me to remotely switch off outlets, monitor my consumption, and integrate it into Grafana. After a few minutes of research, I stumbled upon the "Intertech SW-0816" PDU. The quality/function/price ratio seemed perfect, so I decided to purchase it.

A week later, I received it and began testing. Everything worked as I imagined except, unfortunately, SNMP was not enabled, and I couldn't link it to my Prometheus... This is why I thought the simplest solution would be to hack it. Looking at the interface, it seemed likely possible.

Chapter 2 - How?

Once logged in with the admin account of the PDU, I saw that all the data were displayed on the web page, so I had two options:

• The first: write a script that makes a GET request with basic auth, retrieves the page, and extracts the data.
• The second: Find a vulnerability and see how the data is sent to WEB clients.

I chose the second option because it seemed more fun, simple, and faster than the first.

And I'm "sorry" to say that it was ridiculously easy to steal the data. By pressing F12 on my browser, then going to the "Network" tab, I saw that the application made 1 request per second to the URL "http:///status.xml". I tested this page in incognito mode to see if, by any chance, I could access it without authentication, and it worked perfectly...

I ended up processing this information: Temperature, Amperage, Humidity, and a boolean on/off for the 8 outlets.

Chapter 3 - Implementation

After finding a data source, I started to exploit it. I armed myself with my WebStorm IDE and began a NodeJS application, which is kind of my go-to language. It's worth noting that I'm not a developer by any means, but that doesn't stop me from learning this language.

I started by making a request to the status.xml URL, retrieving the data, installing a prometheus-client module, and then creating all the fields I wanted to export. A very simple app that links the PDU data to a Prometheus server.

Once the data was on the Prometheus server, all that was left was to link this data to a panel. So, I created a simple panel with the graphs and data I wanted.

And that's it, the setup is complete, and now it's time to show the result!

Chapter 3 - Result

As you can see below, the result is, in my opinion, perfect.

This now allows me to have a consumption history as well as the real-time state of my electrical circuit!

To allow everyone to use the script and my findings, you'll find all the code and Grafana panel on my GitHub below!
GitHub Link

Replicating a VM on Proxmox is still a challenging task, requiring several prerequisites, such as:

• VM storage must use ZFS.
• The ZFS storage name must be identical on both nodes.
• Nodes must be part of a cluster.

After setting up my new nodes in a hyper-converged configuration, I was unable to maintain consistent naming across the board. Consequently, replicating my VMs through the GUI became impossible, leading me to create the tutorial below.

It's important to note that this feature was first requested in 2017, and as of the current date (March 17, 2024), Proxmox has yet to fully develop this functionality. Here is the bug tracker link: Proxmox Bug Tracker Issue #2087

Prerequisites for Replicating Your VMs on Proxmox Using My Script

• ZFS storage
• SSH keys between nodes for passwordless authentication

Script for VM Replication

Below is the Bash script I use for VM replication:

#!/bin/bash

# Define the array containing the VM IDs
ids=(100 101)

# Define the variable for the destination host
destinationHost="h2"

# Define the variable for the destination ZFS pool
destinationPool="rpool/Replica"

# Loop through the array of IDs
for id in "${ids[@]}"; do
    # Execute the qm config command to retrieve specific configuration and extract desired information
    while IFS= read -r line; do
        # Create the ZFS snapshot using the specified pool and the result of the previous command
        eval "zfs destroy ${line}@Replication"
        eval "zfs snapshot ${line}@Replication"
        # Variable to store the same name as local storage
        diskName=$(echo "${line}" | awk -F'/' '{print $2}')
        eval "zfs send ${line}@Replication | ssh ${destinationHost} zfs receive ${destinationPool}/${diskName} -F"
    done < <(qm config "$id" | grep -E 'scsi[0-9]+:|virtio[0-9]+:' | awk -F': |,' '{print $2}' | tr : /)
done

The ids variable contains the IDs of the VMs I wish to replicate. The destinationHost variable specifies the replication host, and the destinationPool variable contains the destination ZFS pool (which can be created using zfs create rpool/<pool name>).

This script retrieves the configuration of each node, lists all your disks, creates a snapshot, and then sends it to the destination host. It's a simple script that replicates storage as Proxmox would.

To mount your VM on the replication host, you can either:

• Copy the VM's configuration from the source host and add its disks.
• Recreate the configuration and import the replicated disks.

I opted for the second option for simplicity and scalability over time. In the event of losing the source host, it takes a maximum of 5 minutes per VM to reconfigure and import the disks.

Note: This script destroys and recreates the snapshot each time to save space. However, this means the snapshot is fully sent to the destination server each time, generating significant network/disk traffic. In my case, this isn't an issue due to having 20GB/s symmetric and NVME storage.

Two weeks ago, I stumbled upon a "Craft Computing" video discussing a 2-node Quanta server. This was perfect timing as I was looking to expand my infrastructure but was constrained by the space in my half rack. I saw this as my only option.

In this blog, I will detail all the challenges I faced and the numerous trips back and forth to the data center.

Chapter 1 - Ordering the Server

After extensive research, I found the Quanta T42S-2U, which houses 4 independent nodes, each with relatively recent processors and 128GB of RAM. I found several options on eBay and opted for the cheapest one. Since I already had a large server, it didn't make sense to go for something much larger.

€300 for the server and €200 for shipping later (ouch), the server arrived a week later.

Chapter 2 - The Troubles Begin

Upon receiving the server, I noticed two things: the transport protection foam had expanded and opened, filling the connector of 2 of the 4 nodes with plastic. It wasn't too serious; I managed to clean it with alcohol. The second issue was the absence of rails in the boxes, which was inconvenient but didn't stop me from installing it in the data center. After contacting the supplier, my rail set was dispatched the following week.

Once the server was plugged in, and after losing a set of ear, I booted into the BIOS and began configuring the server. I set up the BMC IP to access the server remotely without a monitor and attempted to log in to the interface with the generic admin/admin credentials, which didn't work. After trying all combinations and searching online, I decided the easiest solution was to boot from an ISO containing ipmitool. After flashing a USB stick with a rescue Linux distribution, I accessed the IPMI tool, reset the password, and saw everything was functioning well before shutting down and unplugging the server.

Chapter 3 - BMC Configuration

The following weekend, I installed it at the data center, restarted it, and found that my BMC configuration had disappeared. After searching for documentation on the manufacturer's site, I realized they had decided not to provide any...

I ended up navigating through all the menus and finally found a button to preserve the configuration on the flash disk. However, it's crucial not to forget to click this button—a mistake I would make later...

Chapter 4 - BMC Crash

Once the server was set up and the 4 BMCs configured, I went home to finalize the configuration, a big mistake...

Back home, I started configuring everything. Things went smoothly for 2 of the 4 nodes, but as I was loading the ISO through the online console for nodes 3 & 4, the BMC crashed and rebooted. Unfortunately, on these nodes, I hadn't clicked the "preserve config" button hidden in two submenus...
I lost control over these nodes and faced two options: drive 1.5 hours back to the data center or find the administrator account password.

Chapter 5 - BMC - Default Creds & Brute Force

As mentioned, the server manufacturer doesn't seem to like documentation, which was problematic since I desperately needed the default password. Fortunately, having access to two other nodes allowed me to find the default username, "administrator".

Now, I just had to find the password... After much searching, no passwords worked. I found an exploit with IPMItool that exposed the HMAC buffer and auth key. So, I wrote a script to generate an HMAC from a database of passwords and compared the result with the server's output, hoping for a match...

Unfortunately, after 18 million passwords, none matched. I had no choice but to hit the road back to the data center...

Script here if you need it

Chapter 6 - BMC Watchdog

Once my OS (Proxmox) was installed, I noticed the server rebooted every 5 minutes. After some investigation, I discovered the BMC's watchdog was active, and my OS wasn't communicating with it. The watchdog checks if the OS has crashed, and since the OS wasn't signaling that everything was fine, it assumed the server had crashed and thus restarted it. After correcting this in Proxmox, everything worked well.

Chapter 7 - Up & Running

After many struggles and three round trips to the data center, the server is finally up and running! If only the manufacturer had provided documentation... it would have likely saved me a lot of timeand gas...

If you have any questions about your Quanta server, I probably have the answer, so don't hesitate to write me an email!

Hello, I'm Baptiste, a Network and System Engineer working for a service provider. I've embarked on a journey to deploy BGP for fun in my personal infrastructure. This post marks the beginning of many to come, as I plan to use this blog as an archive to document my project. If, along the way, I can assist others, that's even better!

English is not my primary language, so please bear with me and feel free to reach out with any guidance or questions. :)

This post assumes that you have already set up your VyOS with the following minimum requirements:
1.One interface designated as the WAN connection.
2.One interface dedicated to your IPv4 prefix.
3.An Autonomous System Number (ASN).
4.A BGP tunnel ready for connection or a prepared BGP session peer.
5.Basic familiarity with VyOS and proficiency in using the Command Line Interface (CLI)."

My setup :

• Proxmox as hypervisor.
• VM with 4 cores, 8GB of RAM, 32GB of disk space.
• My WAN is one on my ISP's attributed IPv4.
• VyOS Version 1.5 (commands are different between last version !)
• I'm using BGP-Tunnel as my peer with GRE Tunnel.

Step 1 : GRE Tunnel /w BGPTunnel.com

In my current setup, I am utilizing BGPTunnel.com as my primary peer. However, in the future, my datacenter housing provider will enable me to establish an eBGP session directly with them. This future enhancement will replace BGPTunnel.com as my BGP peer.

Note 20.02.2024 : BGP Session established with VTX.

To create the tunnel with BGPTunnel.com follow as procede :

1. Create an account on their website and link your ASN.
2. Once they verified your ASN, click on the "Tunnels" tab.
3. To create a new tunnel, select the type 'GRE' and choose the server location that is closest to you. Enter your WAN IPv4 address, ensuring this IP is reachable by a ping. Finally, click on "Create Tunnel" to complete the process..
4. Now click on "config" and then enter these commande into VyOs

configuration
set interfaces tunnel tun0 encapsulation "gre"
set interfaces tunnel tun0 source-address <Your WAN IP>
set interfaces tunnel tun0 remote <Remote IP of the Tunnel>
set interfaces tunnel tun0 address <Your attributed local address, should be a /30>

5. Now that the tunnel is created, you should be able to ping the other end of the tunnel (Your attributed local address minus 1).
6. Now click on the "BGP Sessions" tab and then "New BGP Session".
7. Select your ASN, then your tunnel and click on "Create new BGP Session".
8. The peering partner is now ready.

Step 2 : BGP Setup

Now that you have someone to peer to we need to configure our router to export your prefix, in this example I'm using my own IPv4 prefix and ASN.

If your provider can provide full BGP table, like mine, the route map will be different, so be aware of the risk.

First we need to put one of your address on the interface that will be used with your prefix, this IP need to be on one of the prefix you are exporting.

set interfaces ethernet eth1 address <IP Address>

Enter these commands to configure BGP on your router :

set protocols bgp system-as <Your ASN>
set protocols bgp neighbor <Peer Gateway> remote-as <Remote ASN>
set protocols bgp neighbor <Peer Gateway> update-source <Your Peer Address>
set protocols bgp address-family ipv4-unicast network <Your IPv4 prefix>
set protocols bgp neighbor <Peer Gateway> interface source-interface <Your tunnel interface>
set protocols bgp parameters router-id <The IP of your prefix's interface>

Now that we have setup the base parameter for BGP we need to make some policy to only export/import our prefix.
To do that enter these commands :

#Create a permitted prefix list
set policy prefix-list <Prefix-Rule-Name>-OUT rule <Prefix Rule ID> action 'permit'
set policy prefix-list <Prefix-Rule-Name>-OUT rule <Prefix Rule ID> prefix <Your prefix>

#Create a permited route map with your prefix.
set policy route-map <Route-Map-Name>-IN rule <Map Rule ID> action 'permit'
set policy route-map <Route-Map-Name>-OUT rule <Map Rule ID> match ip address prefix-list <Prefix-List-Name>-OUT

#Deny everything else (99 is a random ID number, should be higher than your other rules)
set policy route-map <Route-Map-Name>-OUT rule 99 action 'deny'

#Set the Route map you just created on the BGP config.
set protocols bgp neighbor <Peer Gateway> address-family ipv4-unicast route-map import <Route-Map-Name>-INT
set protocols bgp neighbor <Peer Gateway> address-family ipv4-unicast route-map export <Route-Map-Name>-OUT

#Create a default gateway policy route for the interfaces that contain the prefix, if you are provided with full BGP table, create static route only for the neighbor's IP.
set policy route <Policy-Route-Name> rule <Route Rule ID> destination address 0.0.0.0/0
set policy route <Policy-Route-Name> rule <Route Rule ID> set table 100
set protocols static table <Route Table Rule ID> route 0.0.0.0/0 next-hop <Peer Gateway>
set policy route <Policy-Route-Name> interface eth1

#Commit and save your change !
commit
save

Now everything should be working, to verify the BGP status enter the command :

show bgp neighbors <Peer Gateway>

Search for "BGP state" and it should be established !

Step 3 : Test, End & Configuration

To ensure that your prefix is correctly exported, you have multiple options. The simplest one is to ping the IP address associated with your interface's prefix. If the ping is successful, everything is functioning as expected; otherwise, there might be a configuration issue.

Alternatively, you can set up another virtual machine (VM) on the same LAN as your prefix's interface. Configure an IP for this VM and attempt to ping 1.1.1.1. If the ping is successful, you can further verify the routing by checking your own IP. To do this, enter:

curl ifconfig.me

If the response is the IP of the VM interface, you've done everything correctly !

If nothing is working, you will find bellow my entire config !

interfaces {
    ethernet eth0 {
        address 212.147.79.246/29
        hw-id bc:24:11:4f:03:79
    }
    ethernet eth1 {
        address 82.115.209.1/24
        hw-id bc:24:11:fe:a7:52
    }
    loopback lo {
    }
    tunnel tun0 {
        address 10.249.4.10/30
        encapsulation gre
        remote 154.57.85.10
        source-address 212.147.79.246
    }
}
nat {
}
policy {
    prefix-list AS215825-OUT {
        rule 10 {
            action permit
            prefix 82.115.209.0/24
        }
    }
    route-map AS215825-IN {
        rule 10 {
            action permit
        }
    }
    route-map AS215825-OUT {
        rule 10 {
            action permit
            match {
                ip {
                    address {
                        prefix-list AS215825-OUT
                    }
                }
            }
        }
        rule 20 {
            action deny
        }
    }
}
protocols {
    bgp {
        address-family {
            ipv4-unicast {
                network 82.115.209.0/24 {
                }
                redistribute {
                    connected {
                    }
                }
            }
        }
        neighbor 10.249.4.9 {
            address-family {
                ipv4-unicast {
                    route-map {
                        export AS215825-OUT
                        import AS215825-IN
                    }
                }
            }
            interface {
                source-interface tun0
            }
            remote-as 209533
            update-source 10.249.4.10
        }
        parameters {
            router-id 82.115.209.1
        }
        system-as 215825
    }
    static {
        route 0.0.0.0/0 {
            next-hop 212.147.79.241 {
                distance 2
            }
        }
        table 100 {
            route 0.0.0.0/0 {
                next-hop 10.249.4.9 {
                }
            }
        }
    }
}

If you have any questions, don't hesitate to contact me !
By email : baptiste(at)baptiste.it
On Twitter : @Verttigo_